Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

ArXiv: 2407.01906

🎯 Pitch

Fine-tuning only 5–15% of the most task-relevant experts in mixture-of-experts LLMs matches or exceeds full fine-tuning performance while cutting trainable parameters by up to 90%. ESFT exploits the discovery that expert routing is highly concentrated within a task but differs dramatically across tasks, turning MoE specialization into a built-in parameter efficiency mechanism.

1. Executive Summary

This paper proposes Expert-Specialized Fine-Tuning (ESFT), a parameter-efficient fine-tuning method for Mixture-of-Experts (MoE) large language models that selectively trains only the experts most relevant to a downstream task while freezing all other parameters. Using DeepSeek-V2-Lite (a fine-grained MoE with 66 experts per layer) across six diverse tasks spanning math, code, intent recognition, summarization, legal judgment, and low-resource translation, the authors demonstrate that ESFT matches or surpasses full-parameter fine-tuning performance while reducing trainable parameters by up to 90% and training time by up to 30% (e.g., achieving 50.2 average specialized task score versus FFT's 51.0, with only 1.4B–1.85B trainable parameters compared to FFT's 15.7B). The method operates through two complementary relevance scoring functions—average gate score (mean affinity of an expert to sampled task tokens) and token selection ratio (fraction of tokens for which the expert is selected)—which exploit the paper's central empirical finding that MoE routing distributions are highly concentrated within a task yet vary significantly across tasks, establishing that task-specific expert specialization enables efficient adaptation only when the underlying model architecture supports fine-grained expert segmentation.

2. Context and Motivation

The Core Problem: PEFT Has Neglected Sparse Architectures

The fundamental gap this paper addresses is straightforward to state but has been largely overlooked by the research community: parameter-efficient fine-tuning (PEFT) methods have been designed and evaluated almost exclusively for dense-architecture LLMs, leaving sparse Mixture-of-Experts (MoE) architectures with no principled PEFT strategy that exploits their unique structure.

This matters because MoE architectures have become one of the dominant paradigms for scaling LLMs to enormous parameter counts while maintaining tractable inference costs. Models like Mixtral 8×7B (Mistral, 2024b), DBRX (Databricks, 2024), Grok-V1 (XAI, 2024), and DeepSeek-V2 (DeepSeek, 2024) all use MoE designs where only a subset of "experts" (specialized feed-forward sub-networks) are activated for any given input token. In a dense model, every parameter participates in every forward pass. In an MoE model, the router dynamically selects which experts process each token, meaning different parts of the model specialize in different types of computation. This is not a niche architectural curiosity—it is how many of the largest deployed models work.

Yet when practitioners want to adapt these models to downstream tasks with limited compute, they are forced to use PEFT methods designed for dense architectures: LoRA (Hu et al., 2021), adapters (Houlsby et al., 2019), or prefix/prompt tuning (Li and Liang, 2021; Liu et al., 2021). These methods operate by either adding small trainable components (LoRA injects low-rank matrices alongside existing weights; adapters insert bottleneck layers between transformer blocks) or selecting parameters to train based on criteria that are architecture-agnostic (structured pruning, sparse masks). None of them exploit the fact that in an MoE model, different experts already handle different types of input—a structural property that could, in principle, be leveraged to identify which parameters are most relevant to a given downstream task before training begins.

Why This Gap Matters: Practical and Theoretical Significance

Practical urgency: the customization bottleneck. As LLMs grow into the hundreds of billions of parameters, full-parameter fine-tuning (FFT) becomes infeasible for most practitioners. Storing a full fine-tuned copy of a 16B-parameter model (~60 GB in FP32) per downstream task is prohibitively expensive when an organization needs to support dozens or hundreds of customized deployments. PEFT methods address this by producing compact task-specific parameter updates (LoRA adapters can be mere megabytes). But applying LoRA to an MoE model treats it as a black-box dense architecture, missing the opportunity to reduce trainable parameters further by exploiting expert specialization. If, as the paper hypothesizes, only a small fraction of experts are genuinely relevant to any given downstream task, then training even LoRA's low-rank matrices across all layers and all experts wastes compute on parameters that contribute negligibly to task performance.

The paper quantifies this waste directly: FFT on DeepSeek-V2-Lite requires training 15.7B parameters and 28.5 minutes per task; LoRA trains fewer parameters (via low-rank decomposition) but still modifies the forward pass of every expert indirectly. ESFT trains only 1.4B to 1.85B parameters (Table 3) while matching FFT's specialized task performance, representing a 70–90% reduction in trainable parameters with no performance degradation. For organizations running many fine-tuning jobs, this translates directly to reduced GPU memory costs, faster iteration cycles, and lower storage requirements for deployed adapters.

Theoretical significance: understanding expert specialization. Beyond practical efficiency, the paper addresses a deeper scientific question: what do MoE experts actually learn, and how does that learning manifest in routing behavior? Prior work like DeepSeekMoE (Dai et al., 2024) had demonstrated that fine-grained expert segmentation enables specialization—experts become more focused on specific knowledge types when they are smaller and more numerous—but this specialization had not been systematically characterized across diverse downstream tasks. The paper's probing experiments in Section 3.2 provide empirical evidence for two claims that, if true, have significant implications for how we think about MoE model adaptation:

Within-task routing concentration: A small subset of experts handles the majority of routing weight for tokens from a single task. Figure 2 shows that normalized gate values follow a sharply decreasing distribution—the top few experts per layer account for most of the routing mass, while most experts contribute near-zero gate values. This means the model already uses a sparse subset of its capacity for any given task type, even without fine-tuning.
Cross-task routing divergence: The sets of highly-activated experts differ substantially across tasks. Figure 3 visualizes this as a near-diagonal heatmap: the average overlap of Top-6 experts between two samples of the same task is close to 6 (high overlap), while between samples of different tasks it is near 0 (minimal overlap). This means expert assignments are not random or interchangeable—they reflect genuine task-specific specialization.

If both claims hold, then the identity of which experts to fine-tune for a downstream task is not something that must be learned during training through gradient descent—it can be read off from the model's routing behavior on a small sample of task data before training begins. This is the conceptual leap that motivates ESFT and distinguishes it from all prior PEFT work: rather than asking "which parameters should we add or modify to adapt this model?", ESFT asks "which experts does the model already use for this type of task, and can we improve only those?"

Where Existing PEFT Methods Fall Short for MoE

The paper categorizes existing PEFT methods into three families (Section 2.1) and identifies how each is suboptimal when applied to MoE architectures:

Adding new parameters (Adapters, Soft Prompts). Methods like Adapter (Houlsby et al., 2019) insert small bottleneck layers into the transformer block, while Soft Prompt methods (prefix tuning, P-tuning v2) prepend learnable continuous vectors to the input or hidden states. Both families keep the base model frozen and train only the injected components. The problem for MoE: these methods are architecture-agnostic—they insert parameters at fixed positions (e.g., after attention, after FFN) regardless of whether the underlying FFN is an MoE layer with expert-specific processing. An adapter placed after an MoE layer processes the output of the expert system, but it doesn't influence which experts are activated or how individual experts process their assigned tokens. This means the adapter cannot leverage expert specialization—it operates on the aggregated output, treating the MoE as a black box.

Selecting existing parameters (structured/unstructured). These methods fine-tune a subset of existing model parameters while keeping others frozen. Structured approaches select entire modules (e.g., "fine-tune layers 20–24 only"), while unstructured approaches learn sparse binary masks over individual weights (Liao et al., 2023; Ansell et al., 2021). The problem: these methods select parameters based on magnitude (which weights are largest?), gradient (which weights respond most to task data?), or random initialization (sparse masks learned during training), but none of these criteria correspond to the functional organization of an MoE model. In a dense model, there is no inherent grouping of parameters by "task affinity"—every weight participates in every forward pass, so selecting by gradient magnitude is a reasonable proxy for importance. In an MoE model, however, the model already partitions its computation by task through the routing mechanism. Ignoring this structural information and falling back to weight-level or layer-level selection criteria wastes the most valuable signal the architecture provides.

Low-rank adaptation (LoRA). LoRA (Hu et al., 2021) decomposes weight updates into low-rank matrices $\mathbf{B}\mathbf{A}$ such that $\mathbf{W}' = \mathbf{W} + \mathbf{B}\mathbf{A}$ , where $\mathbf{B} \in \mathbb{R}^{d \times r}$ and $\mathbf{A} \in \mathbb{R}^{r \times k}$ with $r \ll \min(d, k)$ . It is the dominant PEFT method and the primary baseline this paper compares against. LoRA's limitations for MoE are subtle but consequential:

LoRA modifies all experts uniformly. When applied to an MoE layer, LoRA is typically attached to each expert's FFN weight matrices (or to the attention projections shared across the layer). This means every expert receives the same low-rank update structure, regardless of whether that expert is relevant to the downstream task. If only 5 out of 66 experts in a layer are actually used for the target task, LoRA still stores and computes low-rank updates for all 66—wasting parameters and compute.
LoRA's rank is a global hyperparameter. The rank $r$ controls the expressivity of the adaptation uniformly across all layers and all experts. But the paper's Figure 4 shows that the number of task-relevant experts varies dramatically across layers (ranging from 2 to 15 out of 66) and across tasks. A fixed rank cannot capture this heterogeneity: layers with highly concentrated routing might need fewer trainable parameters, while layers with diffuse routing might need more. LoRA has no mechanism to allocate capacity differentially based on task-specific routing patterns.
LoRA does not maintain expert specialization. Because LoRA modifies every expert's forward pass (even if the low-rank update is small), it dilutes the specialization that makes MoE models efficient in the first place. An expert that previously specialized in, say, legal terminology might receive LoRA updates during fine-tuning on a math task, causing its representations to drift toward math-relevant features. This is one mechanism by which FFT and LoRA degrade general-task performance after specialized fine-tuning—something ESFT explicitly avoids by freezing non-relevant experts (Table 2 shows ESFT retaining 61.5 vs. FFT's 58.8 average general-task performance).

The deeper issue: no existing method uses routing as a selection signal. All prior PEFT methods treat the decision of which parameters to train as independent of the model's own internal organizational structure. In an MoE model, the router's assignment of tokens to experts is the most direct signal available about which parts of the model are functionally relevant to a given input. ESFT is, to the authors' knowledge, the first method to use this signal as the sole criterion for selecting trainable parameters, making it a natural fit for MoE architectures in a way that no prior method achieves.

Prior Work on MoE Expert Specialization

The paper builds on a specific lineage of MoE research that established the groundwork for expert specialization:

Coarse-grained MoE (GShard, Switch Transformer, Mixtral). Early large-scale MoE models like GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) used relatively few experts (typically 8–16) with top-1 or top-2 routing. Because each expert handles a large fraction of the total token distribution, individual experts must learn to process diverse inputs spanning multiple domains. For instance, in Mixtral 8×7B (Mistral, 2024b), only 2 out of 8 experts are activated per token, meaning each expert sees roughly 25% of all tokens. This forces experts to be generalists—they cannot specialize deeply in any single task type because their routing distribution is too broad. The paper argues (Section 2.2) that this lack of specialization makes coarse-grained MoE models poor candidates for expert-selective fine-tuning: if every expert is a generalist, selecting a subset of experts based on routing patterns is unlikely to capture all the knowledge relevant to a task.

Fine-grained MoE (DeepSeekMoE). The key architectural innovation that enables ESFT is fine-grained expert segmentation, introduced by Dai et al. (2024) in DeepSeekMoE. The core idea is deceptively simple: instead of having $N$ large experts with top- $K$ routing, segment each expert into $m$ smaller ones, yielding $mN$ total experts with top- $mK$ routing. This maintains the same total computational cost (the same fraction of total parameters is activated per token) while allowing each expert to specialize more narrowly. DeepSeek-V2-Lite, the model used in this paper, takes this to an extreme: 66 non-shared experts per layer with top-6 routing (plus 2 shared experts, for a total of 68 per layer). This means each expert processes a much narrower slice of the token distribution, enabling genuine specialization.

The paper's contribution is not the fine-grained architecture itself (that is DeepSeekMoE's contribution) but rather demonstrating that fine-grained specialization enables expert-selective fine-tuning and that coarse-grained models do not support this. Figure 7 shows this clearly: when the authors simulate coarse-grained models by grouping experts (using a greedy similarity-based clustering described in Appendix B), ESFT's performance degrades more severely than FFT's. At a group size of 4 (simulating roughly 16 experts instead of 66), ESFT's MATH score drops from ~23 to ~18 while FFT drops only from ~23 to ~21. This is a crucial finding: ESFT is not a universal PEFT method for any MoE model—it relies critically on the expert specialization that fine-grained segmentation provides.

The shared expert distinction. DeepSeekMoE also introduced shared experts (the $K_s$ experts in Equation 4 that process all tokens regardless of routing). These experts are intended to capture common knowledge that applies across all input types, reducing redundancy among the non-shared (routed) experts. This architectural detail becomes important for ESFT's design: the paper finds (Section 6.3, Table 3) that training shared experts degrades general-task performance (from 61.5 to 60.3 when adding shared expert training) without substantially improving specialized-task performance. This is consistent with the interpretation that shared experts encode universal knowledge that should not be overwritten during task-specific adaptation. ESFT's default configuration—training only task-relevant non-shared experts—emerges from this empirical finding as the optimal balance of specialized performance, general performance, and parameter efficiency.

How This Paper Positions Itself

The paper frames ESFT not as an incremental improvement to existing PEFT methods but as a new category of PEFT that is uniquely enabled by MoE architectures with fine-grained expert specialization. This positioning is explicit in the paper's structure:

Section 2.1 surveys existing PEFT methods and notes they "primarily focused on dense architectural LLMs," with research on sparse architectures being "markedly insufficient."
Section 2.2 establishes that fine-grained MoE is the prerequisite for expert specialization, distinguishing ESFT from what would be possible with coarse-grained models like Mixtral.
Section 3.2 provides the empirical motivation through probing experiments, demonstrating that the routing patterns ESFT exploits are real and robust.
Section 3.3 positions ESFT as the natural consequence: if the model already routes different tasks to different experts, fine-tuning should selectively update those experts.

The paper does not claim that ESFT replaces LoRA or FFT in all contexts. The experimental results in Table 1 show ESFT-Gate achieving 50.2 average specialized score versus FFT's 51.0—a small gap, but FFT is definitionally an upper bound on what training a subset of parameters can achieve. More importantly, the paper demonstrates scenarios where ESFT outperforms FFT on general-task maintenance (61.5 vs. 58.8 in Table 2) while using a fraction of the parameters. This suggests a fundamental tradeoff: FFT maximizes specialized performance at the cost of general ability (via catastrophic forgetting and dilution of expert specialization), while ESFT preserves general ability by restricting updates to the subset of experts that are genuinely task-relevant.

The paper also positions its contribution as analyzing the mechanism behind expert-selective fine-tuning, not just proposing the method. Sections 6.1–6.4 systematically investigate: how many experts are selected per layer (Figure 4), how ESFT compares to LoRA under varying compute budgets (Section 6.2, Figure 6), which types of parameters should be trained (Section 6.3, Table 3), and whether the relevance scoring functions actually matter (Section 6.4, Table 4—random expert selection degrades performance by 2.8–4.4 points on average). This analysis serves to establish ESFT not as an ad-hoc trick but as a principled method grounded in the functional organization of fine-grained MoE models.

3. Technical Approach

3.1 Reader Orientation

This is primarily a method paper that proposes Expert-Specialized Fine-Tuning (ESFT), a parameter-efficient fine-tuning strategy for Mixture-of-Experts language models. The core idea is to pre-select which experts to train based on their routing affinity to a small sample of task data, rather than training all parameters (FFT), adding low-rank adapters to all weights (LoRA), or selecting parameters based on gradient signals learned during training. The system solves the problem of efficiently customizing large MoE models for downstream tasks by exploiting a property the authors demonstrate empirically: that MoE routers already concentrate task-specific computation onto a sparse subset of experts, and this subset can be identified from routing statistics before any gradient-based training begins.

3.2 Big-Picture Architecture (Diagram in Words)

The ESFT system has four main components:

A pre-trained fine-grained MoE backbone (DeepSeek-V2-Lite with 66 non-shared experts per layer, top-6 routing, plus 2 shared experts) — this is the base model whose parameters will be selectively updated.
A data sampling mechanism that draws a small subset of training examples (32 sequences of length 4096) and runs a forward pass to collect expert routing statistics — no gradient computation is needed at this stage.
An expert relevance scoring function (one of two variants) that computes a per-expert, per-layer scalar from the routing statistics, quantifying how important each expert is for the target task.
A threshold-based selection and fine-tuning procedure that identifies the smallest set of experts whose cumulative relevance exceeds a hyperparameter $p$ , marks only those experts as trainable (freezing all other experts, shared experts, attention parameters, and embeddings), and then performs standard supervised fine-tuning on the task data.

Information flows as follows: sample task data → forward pass through frozen MoE model → collect gate values per token per expert → aggregate into per-expert relevance scores → threshold to select trainable experts → freeze all non-selected parameters → fine-tune selected experts on full task training data.

3.3 Roadmap for the Deep Dive

First, the MoE layer formulation (Equations 1–5), since ESFT's selection decisions operate on the gate values these equations define, and understanding what a "gate value" physically represents is prerequisite.
Second, the data sampling protocol and the two relevance scoring functions (Equations 6–7), because these convert raw routing statistics into the selection signal that drives the entire method.
Third, the expert selection threshold mechanism (Equation 8), which bridges from continuous relevance scores to a discrete set of trainable experts.
Fourth, the fine-tuning procedure itself — what gets trained, what stays frozen, and the critical design choices around shared experts and non-expert parameters.
Fifth, the relationship between ESFT and the fine-grained MoE architecture, including why coarse-grained models don't support this approach and how the paper validates this claim experimentally.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a method paper whose central claim is that expert relevance for downstream tasks in fine-grained MoE models can be read off from routing statistics without gradient-based training, and that fine-tuning only these pre-identified experts matches full-parameter fine-tuning performance while preserving general capabilities. The method has two phases: an expert selection phase (no gradient computation, only forward passes to collect routing statistics) and a fine-tuning phase (standard supervised learning on the selected subset).

3.4.1 Preliminaries: MoE Layer Computation (Equations 1–5)

Before explaining ESFT's selection mechanism, we must understand what the MoE layer computes and what signals are available for making selection decisions. The paper uses the DeepSeekMoE variant (Dai et al., 2024), which extends the standard MoE formulation with two architectural innovations: fine-grained expert segmentation and shared expert isolation.

Standard MoE layer (Equations 1–3). In a transformer with MoE layers, the standard feed-forward network (FFN) sub-layer is replaced by a Mixture-of-Experts module. For a single token with input hidden state $\mathbf{u}_t^l$ at layer $l$ , the MoE layer computes:

$\mathbf{h}_t^l = \sum_{i=1}^{N} \left( g_{i,t} \cdot \text{FFN}_i(\mathbf{u}_t^l) \right) + \mathbf{u}_t^l$

where $N$ is the total number of experts, $\text{FFN}_i(\cdot)$ is the feed-forward transformation performed by expert $i$ , and $g_{i,t}$ is the gate value for expert $i$ on token $t$ .

The gate values are computed through a sparse Top-K selection:

$g_{i,t} = \begin{cases} s_{i,t}, & \text{if } s_{i,t} \in \text{TopK}(\{s_{j,t} \mid 1 \leqslant j \leqslant N\}, K) \\ 0, & \text{otherwise} \end{cases}$

where $K$ is the number of experts activated per token, and $s_{i,t}$ is the token-to-expert affinity computed as:

$s_{i,t} = \text{Softmax}_i\left( \mathbf{u}_t^{l\top} \mathbf{e}_i^l \right)$

where $\mathbf{e}_i^l$ is the learned centroid (embedding) vector for expert $i$ in layer $l$ .

What this computes, operationally. For each token at each MoE layer, the model computes the dot product between the token's hidden representation $\mathbf{u}_t^l$ and each expert's learned centroid $\mathbf{e}_i^l$ . These dot products pass through a softmax over all $N$ experts to produce normalized affinities $s_{i,t}$ (which sum to 1). The Top-K selection then retains only the $K$ highest affinities, zeroing out the rest, and these retained values become the gate values $g_{i,t}$ . The token is routed only to the $K$ winning experts, and the layer output is the sum of those experts' FFN outputs (weighted by their gate values) plus a residual connection from the input.

Why this form. The Top-K sparsification is the key efficiency mechanism of MoE: each token only activates $K \ll N$ experts, so the computational cost scales with $K$ rather than $N$ . The softmax ensures that gate values across experts form a probability distribution (before Top-K truncation), which gives the routing a probabilistic interpretation—the model learns which experts are appropriate for which input patterns. The dot-product affinity $s_{i,t}$ is the signal that ESFT exploits: it represents how strongly the model "prefers" expert $i$ for token $t$ , computed from the similarity between the token's current representation and the expert's learned centroid. A high $s_{i,t}$ means the token representation and expert centroid point in similar directions in the hidden space, which the model has learned (during pretraining) to associate with expert $i$ being useful for processing that token.

DeepSeekMoE extension (Equations 4–5). The DeepSeek-V2-Lite model used in this paper extends the standard MoE layer with two modifications. The full layer output becomes:

$\mathbf{h}_t^l = \sum_{i=1}^{K_s} \text{FFN}_i^s(\mathbf{u}_t^l) + \sum_{i=1}^{N} \left( g_{i,t} \cdot \text{FFN}_i^n(\mathbf{u}_t^l) \right) + \mathbf{u}_t^l$

where $K_s$ is the number of shared experts, $\text{FFN}_i^s$ denotes the shared expert FFNs (which process all tokens regardless of routing), and $\text{FFN}_i^n$ denotes the non-shared (routed) experts (which are activated by gate values).

The gate values for the non-shared experts are computed as:

$g_{i,t} = \begin{cases} s_{i,t}, & \text{if } s_{i,t} \in \text{TopK}(\{s_{j,t} \mid 1 \leqslant j \leqslant N\}, K - K_s) \\ 0, & \text{otherwise} \end{cases}$

What changed. Two things. First, some experts ( $K_s$ of them, specifically 2 in DeepSeek-V2-Lite) are designated as shared: they process every token unconditionally. Their outputs are added to the layer output without any gating. This reduces redundancy among the routed experts—if certain knowledge is universally useful (e.g., basic syntactic processing), it can be captured in shared experts rather than replicated across multiple routed experts. Second, the Top-K selection among routed experts now selects only $K - K_s$ experts (since $K_s$ slots are "used" by the shared experts). In DeepSeek-V2-Lite, $N = 66$ (non-shared experts), $K = 8$ (total activated experts per token), and $K_s = 2$ (shared experts), so each token activates $K - K_s = 6$ routed experts.

Fine-grained segmentation. The 66 non-shared experts in DeepSeek-V2-Lite are the result of fine-grained segmentation: instead of having, say, 8 large experts with top-2 routing (a coarse-grained design like Mixtral), the model has $m \times 8$ smaller experts with top- $m \times 2$ routing. With $m \approx 8.25$ , this yields 66 experts with top-6 routing, maintaining the same fraction of total parameters activated per token. This is critical because smaller experts have more focused specializations—each expert processes a narrower slice of the token distribution—making it possible to identify genuinely task-relevant experts from routing statistics.

3.4.2 Probing Expert Specialization (Motivating ESFT)

Before describing the selection mechanism, the paper establishes that MoE routing patterns exhibit the properties ESFT depends on. These probing experiments (Section 3.2) are not part of the ESFT algorithm itself—they are empirical motivation—but they define what "expert specialization" means concretely.

Data collection for probing. The authors run forward passes on six tasks (math, code, intent recognition, summarization, legal judgment prediction, translation) using the pre-trained DeepSeek-V2-Lite model (no fine-tuning). For each task, they collect the gate values $g_{i,t}$ for every token at every MoE layer. No gradient computation is performed.

Finding 1: Within-task routing concentration (Figure 2). For each task, the authors compute the sum of gate values assigned to each expert across all tokens in the task's dataset, then normalize by the total gate mass across all experts. Formally, for expert $i$ in layer $l$ :

$\text{normalized}_l(i) = \frac{\sum_{t \in \text{task}} g_{i,t}^l}{\sum_{j=1}^{N} \sum_{t \in \text{task}} g_{j,t}^l}$

When experts are sorted by this normalized gate value from high to low (Figure 2), the distribution is sharply decreasing: a small number of experts (roughly 5–15 out of 66) account for the majority of total gate mass. The shaded regions in Figure 2 show variance across layers, indicating that the degree of concentration varies by layer but the overall pattern holds. For most tasks, the top 8–16 experts capture 50–80% of the routing probability mass.

What this means for ESFT. If only a handful of experts handle most of the work for a task, fine-tuning only those experts should capture most of the task-relevant parameter updates, while freezing the majority of experts that are barely used. This is the efficiency argument: training 10 experts instead of 66 reduces trainable parameters by roughly 85% while affecting the experts that process most task tokens.

Finding 2: Cross-task routing divergence (Figure 3). For each task, the authors draw two independent samples of task data and compute the set of Top-6 routed experts (the 6 experts with highest total gate mass per layer). They then compute the overlap (size of intersection) between these expert sets for every pair of samples, averaging across layers. The result is visualized as a heatmap: the diagonal entries (same task, two different samples) have values near 6 (nearly perfect overlap), while off-diagonal entries (different tasks) have values near 0 (minimal overlap).

What this means for ESFT. The experts activated for one task are largely disjoint from those activated for another. This means the routing pattern is not random—it reflects genuine functional specialization. If the same experts fired for all tasks, selecting any subset would affect all tasks equally, and there would be no way to isolate task-specific parameters. The near-orthogonality of expert sets across tasks is what makes selective fine-tuning possible without degrading general performance: updating the experts used by task A should leave the experts used by task B untouched.

Limitation of this analysis. The probing is done on the pre-trained model before any fine-tuning. The routing distribution may shift during training (the model could start routing more tokens to different experts as their parameters change). ESFT's fixed selection means it does not adapt to such shifts—the selected experts are chosen based on pre-fine-tuning routing patterns and remain the only trainable ones throughout training. The paper's strong experimental results suggest this is not a problem in practice (the initial routing is a good enough signal), but it is a design choice worth noting.

3.4.3 Data Sampling for Expert Selection

The first step of ESFT is to collect routing statistics from which expert relevance will be computed. This step requires only forward passes, no gradient computation or parameter updates, making it computationally cheap relative to the fine-tuning phase.

Sampling procedure. From the downstream task's training data $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N}$ , the method randomly samples a subset $\mathcal{D}_s = \{(x_i, y_i)\}_{i=1}^{N_s}$ . The authors state that " $N_s$ samples of a fixed length $L = 4096$ is robust enough to select the most relevant experts." The specific value is $N_s = 32$ , determined empirically in Appendix C.

Why 32 samples? Appendix C (Figure 8) runs an overlap experiment: draw two independent samples of varying sizes from the same task, compute the Top-6 expert sets from each, and measure their overlap. As sample size increases, overlap increases, converging to approximately 5.5–6.0 shared experts (out of 6) at $2^{17}$ tokens—which equals 32 samples × 4096 tokens. This convergence indicates that 32 samples provide enough routing information to reliably identify the same expert set, making the selection robust to which particular 32 samples are drawn.

What is collected. For each token in each of the 32 sampled sequences, ESFT records the gate values $g_{i,t}^l$ for every expert $i$ at every MoE layer $l$ . These are the post-Top-K values: zero for non-selected experts, and the softmax-normalized affinity $s_{i,t}$ for selected experts. The key computational insight is that this is just a forward pass—no additional computation beyond what the model already does during inference.

Why not use gradients for selection. An alternative approach would be to run one step of training, compute gradients with respect to each expert's parameters, and select experts with the largest gradient norms—similar to how structured pruning methods select parameters. The paper argues implicitly against this by using routing statistics instead: gradients require backpropagation (roughly 2× the compute of a forward pass) and depend on the specific loss function and optimization state, while gate values are available "for free" during inference and reflect the model's learned functional organization independent of any training objective.

3.4.4 Expert Relevance Scoring Functions (Equations 6–7)

Once gate values are collected from the 32 forward passes, ESFT aggregates them into per-expert scalar scores that quantify "relevance to the task." The paper proposes two alternative scoring functions, both operating on the same raw gate data but emphasizing different aspects of expert usage.

Score 1: Average Gate Score (ESFT-Gate). This score computes the mean gate value assigned to expert $i$ across all tokens in the sampled data:

$g_i^l = \frac{1}{N_s} \sum_{j=1}^{N_s} \frac{1}{L_j} \sum_{k=1}^{L_j} g_{i,k}^l$

where $N_s = 32$ is the number of sampled sequences, $L_j$ is the length (in tokens) of the $j$ -th sampled sequence, and $g_{i,k}^l$ is the gate value for expert $i$ on the $k$ -th token of sequence $j$ at layer $l$ .

What it computes, operationally. For each MoE layer $l$ , the method: (1) runs 32 forward passes on the sampled task data, (2) at each token, records the gate value $g_{i,k}^l$ for expert $i$ (this is zero for most experts, since only $K - K_s = 6$ experts are activated per token), (3) averages these gate values across all tokens in all 32 sequences. The result is a single scalar per expert per layer between 0 and 1 (since gate values are softmax outputs and sum to at most 1 per token), representing the average routing probability assigned to that expert when processing task data.

Why this form. Averaging gate values measures the intensity of expert usage: an expert that is both frequently selected AND receives high softmax affinity scores will have a high average gate score. The double averaging (first over tokens within a sequence, then over sequences) ensures the score is robust to sequence length variation and sample noise. The key property is that this score accounts for both selection frequency AND selection magnitude—an expert selected 100% of the time with gate value 0.1 scores 0.1, while an expert selected 10% of the time with gate value 1.0 also scores 0.1. This treats routing probability mass as the conserved quantity and measures each expert's share of it.

A subtle point about the gate value interpretation. The gate value $g_{i,k}^l$ is the softmax-normalized dot product before Top-K truncation—it is NOT a probability that expert $i$ processes token $k$ (since truncated experts have zero gate value, not a small probability). It is better to think of $g_i^l$ as measuring "how much routing attention the model pays to expert $i$ on average," where routing attention is a finite resource (sums to $K - K_s = 6$ per token) allocated across experts.

Score 2: Token Selection Ratio (ESFT-Token). This score computes the fraction of tokens for which expert $i$ is selected (i.e., receives a non-zero gate value after Top-K truncation):

$r_i^l = \frac{1}{N_s} \sum_{j=1}^{N_s} \frac{1}{L_j} \sum_{k=1}^{L_j} \frac{\mathbb{1}\left(g_{i,k}^l > 0\right)}{K}$

where $\mathbb{1}(g_{i,k}^l > 0)$ is an indicator function that equals 1 if expert $i$ was selected (received a positive gate value) for token $k$ , and 0 otherwise, and $K = 8$ is the total number of activated experts per token (including shared experts, though the shared experts are excluded from this computation since they process all tokens).

What it computes, operationally. For each token in the 32 sampled sequences, the method checks whether expert $i$ was one of the $K - K_s = 6$ selected routed experts (i.e., whether $g_{i,k}^l > 0$ ). It counts these selections across all tokens and divides by the total number of tokens, then divides by $K$ (the total experts activated per token). The division by $K$ normalizes the ratio such that if all tokens always selected the same 6 experts, each of those experts would score approximately $1/K = 1/8 = 0.125$ (since the shared experts account for 2 of the $K=8$ slots).

Why this form. Unlike ESFT-Gate, this score ignores the magnitude of the gate value and counts only whether the expert was selected at all. This treats every selection equally—an expert selected with gate value 1.0 contributes the same as one selected with gate value 0.01. The division by $K$ normalizes to the total number of expert "slots" per token rather than to the total routing probability mass. The rationale (implicit in the paper) is that binary selection may be more robust: gate value magnitudes can be noisy and depend on the specific softmax temperature (here, temperature 1), while the binary selection decision is a coarser but more stable signal.

Which score to use? The paper does not prescribe one over the other—both are evaluated and the choice can be made based on task-specific validation performance. Table 1 shows ESFT-Token achieving 49.4 average specialized performance and ESFT-Gate achieving 50.2 (slightly better for Gate), while ESFT-Token achieves 61.5 average general performance versus ESFT-Gate's 60.6 (slightly better for Token). The authors note (Section 3.3) that both can be "chosen based on task-specific experimental performance." ESFT-Token tends to select fewer experts (Figure 4 shows ESFT-Token using 2–8 experts per layer across tasks, while ESFT-Gate uses 3–15), which may explain its better general-task retention (fewer parameters updated means less interference with other tasks' experts).

Why not use other aggregation methods? The paper does not discuss alternatives like taking the maximum gate value (which would select experts that fire strongly even if rarely) or using the gradient of the task loss with respect to routing decisions (which would capture causal importance, not just correlation). The choice of average-based aggregation implicitly assumes that frequently-used experts are more important to fine-tune than rarely-used ones, which is reasonable but unverified—an ablation comparing average gate to maximum gate or to gradient-based selection would strengthen the method's justification.

3.4.5 Expert Selection via Cumulative Threshold (Equation 8)

After computing per-expert relevance scores, ESFT must convert these continuous scores into a binary decision: train this expert or freeze it. The method uses a cumulative threshold that selects the smallest set of top-scoring experts whose combined relevance exceeds a hyperparameter $p$ .

The selection procedure. For each MoE layer $l$ , independently:

Sort all $N = 66$ non-shared experts in descending order of their relevance score $R_i^l$ (which is either $g_i^l$ from Equation 6 or $r_i^l$ from Equation 7).
Find the smallest set $\mathcal{E}_s^l$ such that:

$\sum_{i \in \mathcal{E}_s^l} R_i^l \geqslant p$

where $p \in (0, 1]$ is a hyperparameter controlling the fraction of total relevance mass to include.

Mark all experts in $\mathcal{E}_s^l$ as trainable; freeze all other experts in layer $l$ .

What this computes, operationally. The relevance scores across all 66 experts in a layer sum to some total $T^l = \sum_{i=1}^{66} R_i^l$ . For ESFT-Gate, $T^l$ is the average total gate mass per token in that layer (which is approximately $K - K_s = 6$ , since each token distributes 6 units of gate mass across experts). For ESFT-Token, $T^l$ is the average selection count per token divided by $K$ , which is approximately $(K - K_s)/K = 6/8 = 0.75$ . The threshold $p$ specifies what fraction of this total mass must be captured by the selected experts. For example, $p = 0.1$ means "select enough top-scoring experts such that their combined score is at least 10% of the total score in this layer." The process is repeated independently for each of the 26 MoE layers (DeepSeek-V2-Lite has 26 transformer layers, each with one MoE sub-layer).

Why cumulative threshold, not fixed top-m? A natural alternative would be to select the top- $m$ experts per layer, where $m$ is a fixed hyperparameter. The cumulative threshold adapts to the concentration of routing: in layers where routing is highly concentrated (one expert captures 50% of gate mass), $p = 0.1$ might select only 1–2 experts. In layers where routing is diffuse (mass spread across 20 experts), $p = 0.1$ might select 10–15 experts. This adaptivity is desirable because Figure 4 shows that the number of relevant experts varies substantially across layers (middle layers are more concentrated, requiring fewer experts), and a fixed $m$ would either over-select in concentrated layers (training unnecessary experts) or under-select in diffuse layers (missing relevant ones). The cumulative threshold provides a principled way to allocate trainable capacity proportional to routing concentration.

Hyperparameter values used. The paper sets $p = 0.1$ for ESFT-Gate and $p = 0.2$ for ESFT-Token. The difference reflects the different score distributions: ESFT-Token produces a flatter distribution (since it ignores gate magnitude), meaning a higher $p$ is needed to capture a comparable number of experts. These values were determined through the efficiency sweep in Section 6.2, which shows that performance saturates around $p = 0.1$ for ESFT-Gate and $p = 0.2$ for ESFT-Token (Figure 6).

What happens to non-selected experts during fine-tuning? Their parameters are frozen—they still participate in the forward pass and can be routed to, but their weights do not receive gradient updates. This is a crucial design choice: the model can still use all experts at inference time, it just cannot modify the parameters of non-selected ones. This means the fine-tuned model's routing distribution may shift (if the selected experts' outputs change, later layers' routing decisions may change), but the non-selected experts' behavior remains identical to the pre-trained model.

Layer-independent selection. Each layer's selection is made independently based only on that layer's routing statistics. The paper does not consider inter-layer dependencies—for instance, whether selecting more experts in early layers and fewer in later layers would be more effective. This simplicity enables the method to scale to any number of layers without combinatorial explosion in the selection space.

3.4.6 Fine-Tuning Procedure

After expert selection, ESFT proceeds with standard supervised fine-tuning, with the critical constraint that only the selected experts receive gradient updates.

What gets trained. For each MoE layer $l$ , the parameters of the experts in $\mathcal{E}_s^l$ are marked as trainable. Specifically, each expert is a standard Feed-Forward Network (two linear transformations with an activation function, typically SiLU in DeepSeek-V2), so "training expert $i$ " means updating its two weight matrices and bias vectors.

What stays frozen. Everything else:

Non-selected non-shared experts: the $N - |\mathcal{E}_s^l|$ experts not in the selected set remain at their pre-trained values.
Shared experts: the $K_s = 2$ shared experts per layer are always frozen in the default ESFT configuration. The ablation in Section 6.3 (Table 3) shows that training shared experts improves specialized performance slightly (49.8 → 50.8 when adding shared expert training to relevant non-shared training) but degrades general performance more significantly (60.7 → 60.3), so the default is to freeze them.
Router parameters: the expert centroids $\mathbf{e}_i^l$ (Equation 3) that determine routing decisions are frozen. This means the model cannot learn to route tokens differently during fine-tuning—it must work with the pre-trained routing distribution. This is intentional: allowing the router to change would undermine the entire selection rationale, since the experts selected based on pre-training routing patterns might no longer be the ones processing task tokens if the router shifts.
Attention parameters: the query, key, value, and output projections in the self-attention layers are frozen.
Layer normalization: the scale and shift parameters of the normalization layers are frozen.
Token embeddings and language modeling head: the input embedding matrix and output projection are frozen.

Training configuration. The paper reports these hyperparameters in Section 4.3, determined by a grid search over $\{10^{-5}, 3 \times 10^{-5}, 10^{-4}, 3 \times 10^{-4}\}$ :

Learning rate: $\text{lr} = 10^{-5}$ for ESFT (compared to $3 \times 10^{-5}$ for FFT and $10^{-4}$ for LoRA)
Batch size: 32 sequences
Sequence length: 4096 tokens
Maximum training steps: 500, with evaluation every 100 steps (best checkpoint selected based on validation performance)
Optimizer: not explicitly stated, but DeepSeek-V2 typically uses AdamW

The lower learning rate for ESFT compared to FFT is notable: FFT trains all 15.7B parameters and needs a higher learning rate to make meaningful updates to the full parameter space; ESFT trains only 1.4–1.85B parameters and can use a lower learning rate, reducing the risk of catastrophic forgetting in the selected experts.

Data mixing strategy. The paper uses a 1:1 ratio of task-specific data to general alignment data for FFT and LoRA (Section 4.3), which the authors find is "highly effective in preserving general abilities." However, for ESFT, they find (Appendix F, Tables 9–10) that mixing alignment data does not help and may slightly hurt performance. ESFT's default is therefore to train on task data only, without alignment data mixing. This is consistent with the specialization-preservation hypothesis: since ESFT only updates task-relevant experts, it does not need alignment data to "remind" the model of general capabilities—those are encoded in the frozen experts and attention parameters. FFT and LoRA, by contrast, modify all experts or all parameters, which can degrade general performance unless alignment data is mixed in to counteract forgetting.

Training time and storage (Figure 5). On the hardware described (2 nodes × 8 NVIDIA A100 PCIe GPUs on the HFAI cluster), ESFT-Token takes an average of 19.8 minutes per task, ESFT-Gate takes 20.9 minutes, compared to 28.5 minutes for FFT and 16.5 minutes for LoRA. The storage requirements for the trained parameters (i.e., what must be saved per task for later deployment) are 2.57 GB for ESFT-Token, 3.20 GB for ESFT-Gate, compared to 28.6 GB for FFT and an unstated smaller size for LoRA. The storage savings of up to 90% compared to FFT are the primary practical advantage, since each customized model variant can be stored as a compact set of expert weight updates rather than a full model copy.

3.4.7 The Optional Mix-in of Shared Expert and Non-Expert Training (Section 6.3)

Section 6.3 systematically explores variants of ESFT where additional parameter groups are trained beyond the task-relevant non-shared experts. This is not part of the core ESFT method but provides guidance for practitioners making tradeoffs between specialized performance, general performance, and training cost.

The configuration space (Table 3). The authors enumerate combinations of three binary choices:

Which non-shared experts to train: either only task-relevant ones (selected via ESFT-Token with $p = 0.2$ ) or all non-shared experts.
Whether to train shared experts: yes or no.
Whether to train non-expert parameters: this includes router centroids, attention weights, layer normalization parameters, and embeddings.

This yields $2^3 = 8$ possible configurations, but ESFT's default is the setting that trains only relevant non-shared experts and freezes everything else.

Key findings from the ablations. The average specialized ability score (across all 8 tasks, averaged) increases monotonically with the number of trainable parameters: from 47.4 (only shared experts trained, 450M parameters) to 51.0 (all parameters trained, 15.7B parameters). This is unsurprising—more trainable parameters provide more capacity to adapt.

However, general ability shows the opposite pattern: it decreases as more shared parameters are trained. Training only task-relevant non-shared experts achieves 61.5 general ability on average. Adding shared expert training drops this to 60.3–60.7 depending on configuration. Training all non-shared experts (as in FFT) drops it further to 58.8. The mechanism: shared experts and non-expert parameters (attention, embeddings) encode capabilities used across all tasks. Overwriting them with task-specific information causes forgetting on other tasks.

The recommended strategies. Based on these findings, the paper proposes two operational modes:

Maximize specialized performance: train task-relevant non-shared experts AND all shared experts AND non-expert parameters (2.7B total trainable parameters, achieving 50.8 average specialized score with 60.3 general score). This is still more efficient than FFT (15.7B parameters, 51.0 specialized, 58.8 general) while achieving near-identical specialized performance with better general retention.
Balance performance and efficiency (ESFT default): train only task-relevant non-shared experts (1.4B parameters, 49.4 specialized, 61.5 general). This maximizes general ability retention and parameter efficiency while sacrificing only 1.6 points of specialized performance compared to FFT.

Why is priority given to training non-shared over shared parameters? The shared experts process all tokens—they encode knowledge that is universally applicable. When fine-tuning on a specific task, updating shared experts causes them to specialize toward that task's patterns, degrading their universal applicability. The non-shared experts are already somewhat specialized (that is the point of routing), so updating only the subset already relevant to the task concentrates the adaptation where it is most needed without collateral damage to general capabilities. This interpretation is supported by the finding (Table 3) that even training only shared experts (450M parameters, a tiny subset) achieves a specialized score of 47.4—these experts do contain task-relevant knowledge—but the ESFT approach of training task-relevant non-shared experts achieves higher specialized performance (49.4) with no degradation to shared expert representations, preserving general ability.

3.4.8 Dependence on Fine-Grained Expert Segmentation (Section 6.4)

The paper argues and demonstrates that ESFT's effectiveness critically depends on the fine-grained expert segmentation of the backbone MoE model. This is not an implementation detail—it is a fundamental constraint on where the method can be applied.

The simulation experiment (Figure 7). Since no structurally-aligned MoE models with varying expert granularity are publicly available (a limitation the authors acknowledge), they simulate coarse-grained architectures by grouping experts in the DeepSeek-V2-Lite model. Appendix B describes the grouping procedure:

Sample 32 sequences of length 4096 from the alignment dataset.
For each MoE layer, construct a co-occurrence matrix of size $64 \times 64$ (64 non-shared experts excluding 2 shared), where entry $(i,j)$ counts how many tokens select both expert $i$ and expert $j$ in their top-6 choices.
Compute the cosine similarity between rows $i$ and $j$ of this matrix as the similarity between experts $i$ and $j$ .
Use greedy search to partition the 64 experts into groups of size $K$ (either 2 or 4) that maximize average intra-group similarity.

How grouping simulates coarse granularity. Once experts are grouped, each group is treated as a single "coarse expert": all members of the group share the same gate value (computed as the average of their individual gate values), and for training purposes, either all experts in a selected group are updated or none are. The total number of activated experts per token is kept at 1/8 of the total (the same fraction as in the fine-grained model, where 8 out of 64+2 experts are activated). With group size 2, this means 32 pseudo-experts with top-4 routing; with group size 4, 16 pseudo-experts with top-2 routing.

Results (Figure 7). As group size increases (experts become more coarse-grained), ESFT's performance on the MATH task degrades more severely than FFT. At group size 1 (the original fine-grained model with 66 experts), ESFT and FFT both achieve approximately 23 on MATH. At group size 2, ESFT drops to roughly 21 while FFT stays near 23. At group size 4, ESFT drops to roughly 18 while FFT drops only to roughly 21. Simultaneously, the number of experts ESFT trains increases (because each "coarse expert" contains multiple original experts, and selecting a coarse expert forces training all its constituents, even if only some are actually relevant).

Why coarse granularity hurts ESFT specifically. In a fine-grained model, an expert's specialization is narrow—it processes a specific type of linguistic or reasoning pattern. ESFT can select exactly the experts that handle the target task's patterns. In a coarse-grained model, each expert is a generalist that processes diverse input types. Selecting a coarse expert based on its relevance to the target task forces training all the knowledge encoded in that expert—including knowledge irrelevant or even detrimental to the task. This dilutes the specialization that makes selective training effective. Moreover, with fewer total experts, the routing distribution is less concentrated (each expert must handle a larger fraction of tokens), so the "top few experts" account for a smaller fraction of total routing mass, reducing the parameter savings.

Implication for practitioners. ESFT is not a universal PEFT method for all MoE models. It requires a backbone with fine-grained expert segmentation that produces genuine expert specialization. Models like Mixtral (8 experts, top-2 routing), DBRX (16 experts, top-4 routing), or Grok-V1 (8 experts, top-2 routing) are likely too coarse-grained for ESFT to be effective, though the paper does not test this directly due to model availability constraints. The method is specifically designed for and validated on the DeepSeek-V2 architecture family, where experts number in the dozens to hundreds with a correspondingly high degree of specialization.

3.4.9 Summary of Design Choices and Their Justifications

Routing-based selection over gradient-based selection. Gate values are available from forward passes with zero additional training cost and reflect the model's learned functional organization. Gradients require backpropagation and depend on the specific training loss and optimization state, making them a noisier and more expensive selection signal. The paper's probing experiments (Section 3.2) demonstrate that routing patterns are stable and task-specific, providing a reliable basis for selection.

Cumulative threshold over fixed top-m. The concentration of routing varies substantially across layers (Figure 4 shows some layers need 2 experts, others 15). A fixed $m$ would be inappropriate for all layers; the cumulative threshold adapts to each layer's routing concentration automatically.

Two scoring functions, not one. ESFT-Gate captures selection intensity (how strongly the model prefers each expert), while ESFT-Token captures selection frequency (how often each expert is activated). The paper provides both because different tasks may benefit from different selection criteria, and the choice can be made by validation performance. ESFT-Token tends to be more conservative (selects fewer experts), which benefits general ability retention.

Freezing the router. Allowing routing to change during fine-tuning would create a moving target: the experts selected based on initial routing might no longer be the ones processing task tokens after the router adapts. Freezing the router ensures the selection remains valid throughout training. This is a conservative choice that trades the potential benefit of improved routing for the reliability of the selection signal.

Fine-grained MoE as a prerequisite. The method fundamentally relies on experts being specialized enough that "task-relevant" is a meaningful distinction. In a coarse-grained MoE where every expert is a generalist, selecting a subset of experts based on routing would not isolate task-specific knowledge—it would arbitrarily limit the model's capacity. The paper validates this through simulation (Figure 7), showing that ESFT's advantage over FFT diminishes as experts become coarser.

4. Key Insights and Innovations

Innovation 1: Routing Statistics as a Pre-Training Parameter Selection Signal—Not a Post-Hoc Diagnostic

The dominant assumption across all prior PEFT work—whether adapter-based (Houlsby et al., 2019), prompt-based (Li and Liang, 2021; Liu et al., 2021), sparse-mask-based (Liao et al., 2023; Ansell et al., 2021), or low-rank (Hu et al., 2021)—is that the decision of which parameters to train must be made during or after gradient-based optimization begins. LoRA's low-rank matrices are initialized randomly and learned; unstructured pruning methods use gradient magnitudes or weight magnitudes computed during training; adapter placement is fixed by architecture regardless of task. In every case, the model's internal functional organization—what different parameters do—is treated as either unknowable before training or irrelevant to the selection decision.

ESFT makes a conceptually different move: it claims that in a fine-grained MoE model, the routing distribution provides a pre-training answer to the parameter selection problem. The gate values $g_{i,t}$ that the router computes during inference are not merely a computational efficiency mechanism—they are a direct, interpretable readout of which experts the model has learned to associate with which types of input. This readout exists before any fine-tuning gradient is computed. The expert selection phase of ESFT (Section 3.3) is therefore not an optimization step at all: it is a measurement step, requiring only forward passes with no backpropagation, no loss computation, and no parameter updates.

This reframes the PEFT problem from "learn which parameters matter" to "read off which parameters the model already uses." The intellectual shift is subtle but consequential:

Before ESFT: parameter selection is a learning problem (you must train to know what to train).
After ESFT: parameter selection is an inference problem (the model already knows what's relevant; you just need to ask it, via routing statistics, on a small sample of task data).

This is not merely a speedup (avoiding gradient computation during selection). It represents a fundamentally different relationship between the fine-tuning practitioner and the pre-trained model. Rather than treating the model as a tabula rasa whose internal organization is opaque and must be discovered through optimization, ESFT treats the model's expert specialization as legible—you can inspect which experts fire on your task data before deciding which ones to train. The probing experiments in Section 3.2 (Figures 2–3) serve as the empirical warrant for this legibility claim: if routing distributions were diffuse or task-agnostic, the readout would be uninformative. The sharply concentrated, task-divergent routing patterns demonstrate that the signal is real.

Comparison to prior parameter-selection methods. Structured pruning approaches (Guo et al., 2020; Gheini et al., 2021) select entire modules (layers, attention heads) based on magnitude or gradient criteria computed during training—they cannot pre-identify relevant modules from a forward pass alone. Unstructured sparse fine-tuning methods (Liao et al., 2023; Ansell et al., 2021) learn binary masks over individual weights during training, requiring gradient-based optimization to discover the mask. Even methods that "select existing parameters" (Section 2.1) use criteria internal to the optimization process (gradient norm, weight magnitude) rather than architectural signals that pre-exist training. ESFT is the first PEFT method, to the authors' knowledge, where the parameter selection decision is causally upstream of any gradient computation on the task data.

A limitation of this framing. The routing distribution is static—it reflects the pre-trained model's routing behavior, not what routing would be after fine-tuning. If fine-tuning causes the routing distribution to shift (e.g., because updated experts produce different hidden states that change later layers' routing decisions), the experts selected from pre-training routing may no longer be the most relevant mid-training. The paper does not address this dynamic, and ESFT provides no mechanism to update the selection during training. The strong experimental results suggest this is not a practical problem (the initial selection is good enough, and freezing the router parameters likely limits routing drift), but the conceptual claim—"read off what's relevant before training"—assumes routing stationarity that is empirically validated rather than theoretically justified.

Innovation 2: Fine-Grained Expert Specialization as an Enabling Condition for Selective Fine-Tuning—Not Just an Efficiency Optimization

Prior work on MoE architectures has treated fine-grained expert segmentation primarily as a mechanism for improving pretraining efficiency and model quality. DeepSeekMoE (Dai et al., 2024) introduced fine-grained segmentation with the explicit goal of enabling "higher expert specialization"—meaning each expert learns a more focused subset of knowledge, reducing redundancy and improving parameter utilization. The benefit was framed in terms of pretraining: given a fixed computational budget, a fine-grained MoE can achieve better perplexity than a coarse-grained one because experts don't waste capacity learning overlapping functions.

ESFT reveals a second-order consequence of fine-grained specialization that was not anticipated in the original DeepSeekMoE work: it enables parameter-efficient fine-tuning strategies that are impossible with coarse-grained architectures. This is not an incremental extension of DeepSeekMoE—it is a discovery about what expert specialization means for downstream adaptation. The causal chain is:

Fine-grained segmentation → each expert processes a narrower slice of the token distribution → experts become genuinely specialized in specific knowledge types or task patterns.
Genuine specialization → routing distributions are concentrated (few experts per task) and task-divergent (different tasks use different experts).
Concentrated, task-divergent routing → task-relevant experts can be identified from a small sample of task data without training.
Identifiable task-relevant experts → selective fine-tuning is possible without performance degradation.

The critical empirical evidence is Figure 7 (Section 6.4), which demonstrates the converse: when experts are artificially grouped to simulate coarse granularity, ESFT degrades more severely than FFT. This is not obvious a priori. One might expect that both ESFT and FFT would suffer equally from reduced granularity (since both have less specialized capacity to work with). The fact that ESFT suffers more—while FFT remains relatively robust—indicates that ESFT's mechanism depends on fine-grained specialization in a way that FFT's does not. FFT can compensate for coarse experts by jointly updating all parameters to re-specialize them during training; ESFT, which freezes most experts, cannot.

This reframes the design space for MoE architectures. Before this paper, the granularity of experts was primarily a pretraining decision: how fine-grained should experts be to maximize training efficiency and model quality? After this paper, expert granularity has a downstream adaptation dimension: how fine-grained must experts be to enable selective fine-tuning? This insight has implications for MoE architecture design that go beyond the models tested:

For model builders: if you want your MoE model to support efficient downstream customization, you should err on the side of finer granularity, even if the pretraining benefits saturate. The paper's finding that grouping experts from 64 to 16 (4× coarser) causes disproportionate ESFT degradation suggests a non-linear relationship where granularity benefits for adaptation outpace those for pretraining.
For practitioners: ESFT is not a universal MoE PEFT method. Applying it to Mixtral 8×7B (8 experts, top-2 routing) would likely fail, not because the method is flawed, but because the architectural prerequisite (fine-grained specialization) is absent. This is a precision diagnostic: the paper provides criteria (routing concentration, cross-task divergence) for determining whether ESFT will work on a given MoE model, not just how to apply it.

Comparison to prior work on MoE interpretability. Some prior work has analyzed expert specialization (e.g., Dai et al., 2024 show experts specialize in different knowledge types), but these analyses were primarily descriptive—they characterized what experts learn, not how that learning can be exploited for downstream tasks. ESFT is the first method that operationalizes expert specialization as a mechanism for parameter-efficient adaptation. It converts a diagnostic observation ("experts are specialized") into an algorithmic design choice ("select experts by routing affinity, then fine-tune only those").

Innovation 3: Catastrophic Forgetting as a Shared Parameter Phenomenon—Not a Universal Consequence of Fine-Tuning

A well-documented challenge in LLM fine-tuning is catastrophic forgetting: when a model is fine-tuned on a specialized task, its performance on general benchmarks degrades, sometimes severely. The standard mitigation is data mixing—interleaving general-domain data with task-specific data during fine-tuning to "remind" the model of its general capabilities. The paper uses exactly this strategy for FFT and LoRA (Section 4.3, 1:1 mixing ratio), and it is standard practice across the field.

ESFT's default configuration (train only task-relevant non-shared experts, freeze everything else, no alignment data mixing) reveals something unexpected: catastrophic forgetting is not a uniform property of fine-tuning—it is concentrated in specific parameter types, particularly shared experts and non-expert modules. Table 2 shows ESFT retaining 61.5 average general-task performance versus FFT's 58.8, despite ESFT achieving near-identical specialized performance (49.4 vs. 51.0). Section 6.3's ablation (Table 3) makes the mechanism explicit: training shared experts drops general performance from 61.5 to 60.3–60.7 depending on configuration, and training all non-shared experts (as in FFT) drops it further to 58.8.

The intellectual move here is reinterpreting forgetting from a quantity problem ("fine-tuning changes too many parameters") to a localization problem ("fine-tuning changes the wrong parameters—those that encode general capabilities"). The implication is that data mixing is a patch for a parameter-grouping problem: FFT and LoRA need alignment data because they modify shared parameters that are used across all tasks, causing interference. ESFT avoids the problem at its source by freezing those shared parameters, eliminating the need for data mixing entirely.

Evidence that this is a real mechanism, not a confound. If ESFT's better general-task retention were simply due to training fewer parameters, we would expect a monotonic relationship: fewer trainable parameters → less forgetting. But the ablation data in Table 3 contradicts this simple story. Training only shared experts (450M parameters—fewer than ESFT's 1.4B) achieves a general ability score of 61.2, which is lower than ESFT's 61.5 despite training fewer parameters. Conversely, training all parameters (15.7B) achieves 58.8, which is worse than training relevant non-shared + all shared + non-expert (2.7B, achieving 60.3). The relationship is not monotonic in parameter count—it depends on which parameters are trained, with shared experts and non-expert parameters being disproportionately responsible for forgetting.

Practical consequence: a new design principle for PEFT in MoE models. The finding suggests that PEFT methods for MoE should prioritize freezing shared parameters as a first-order design constraint, not an afterthought. LoRA, when applied to MoE, typically does not make this distinction—it adds low-rank adapters to all weight matrices indiscriminately, including those in shared experts and attention layers. A LoRA variant that exempts shared experts from adaptation (or uses lower rank for them) might improve general-task retention while maintaining the convenience of the LoRA framework. The paper does not explore this, but the ablation data strongly imply it.

Comparison to prior work on catastrophic forgetting. The standard narrative in the fine-tuning literature is that forgetting is caused by gradient updates shifting parameters away from a configuration that was optimal for the pre-training distribution, and that solutions include regularization (elastic weight consolidation), rehearsal (data mixing), or architectural isolation (separate task-specific modules). ESFT contributes a new mechanism to this taxonomy: structural isolation through routing-based parameter partitioning. Rather than adding new modules for each task (adapter-style) or regularizing updates, ESFT exploits the MoE architecture's built-in partitioning to achieve isolation without any additional parameters or loss terms. This is fundamentally enabled by the MoE design—it would be impossible in a dense architecture where every parameter participates in every forward pass.

Innovation 4: The `p`-Threshold as a Task-Agnostic Parameter Budget That Adapts to Routing Concentration

Most PEFT methods control capacity through a global, task-uniform hyperparameter: LoRA's rank $r$ , adapter's bottleneck dimension, or the number of fine-tuned layers. These hyperparameters are typically tuned per-task (or set once and applied uniformly), and they allocate a fixed amount of trainable capacity to each module regardless of how much adaptation that module needs.

ESFT's cumulative threshold $p$ (Equation 8) represents a different allocation philosophy: it specifies a fraction of total routing mass to capture, not an absolute number of experts. In layers where routing is highly concentrated (one expert captures 60% of gate mass), $p = 0.1$ may select only 1 expert. In layers where routing is diffuse (mass spread across 20 experts), the same $p = 0.1$ may select 10–15 experts. The hyperparameter is task-agnostic (the same $p$ is used across all tasks) while the resulting allocation is task-adaptive (the number of experts selected varies per layer and per task based on routing concentration).

This is a conceptual innovation in how we think about parameter budgets for fine-tuning. The standard approach—allocate $m$ experts per layer—implicitly assumes that every layer needs equal adaptation capacity. The $p$ -threshold approach assumes that adaptation capacity should be proportional to routing concentration: layers where the model already concentrates its routing onto a few experts should receive sparse updates (train only those few), while layers where routing is diffuse should receive broader updates (train more experts, because the task-relevant knowledge is distributed across them).

Figure 4 provides the empirical validation. The number of experts selected by ESFT varies from 2 to 15 out of 66 across layers and tasks, with middle layers consistently requiring fewer experts. This non-uniformity would be invisible to a fixed-top-m selection strategy, which would either over-train in concentrated layers (wasting parameters) or under-train in diffuse layers (missing relevant experts). The fact that ESFT achieves strong performance despite these large layer-wise variations suggests the $p$ -threshold adaptivity is doing meaningful work.

Why this matters beyond ESFT. The idea of allocating parameter budgets proportionally to some measure of "task concentration" could generalize beyond MoE architectures. In a dense model, one could compute sensitivity scores (gradient norms, Fisher information) for different layers or modules on a small sample of task data, then allocate a LoRA rank or adapter dimension proportionally to those scores. The $p$ -threshold provides a template: define a relevance measure, set a cumulative fraction threshold, and let the allocation emerge from the data rather than prescribing it uniformly.

A limitation. The $p$ values (0.1 for ESFT-Gate, 0.2 for ESFT-Token) are set once based on the efficiency sweep in Section 6.2 and applied uniformly across all tasks. The paper does not explore whether different tasks benefit from different $p$ values (e.g., highly specialized tasks might need smaller $p$ , while tasks requiring broad adaptation might need larger $p$ ). The task-agnostic choice works well enough empirically, but the principle of concentration-proportional allocation would suggest that the optimal $p$ might itself be task-dependent—a question the paper leaves open.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses six downstream tasks spanning two categories. Model enhancement tasks (Math and Code) use MetaMathQA (Yu et al., 2023) for training with GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021a) for evaluation, and the Python subset of evol-codealpaca (Luo et al., 2023) for training with HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for evaluation. Model adaptation tasks include four specialized tasks: Text-to-JSON Intent Recognition (BDCI-21 Smart HCI NLU Challenge), Text Summarization (BDCI-21 Summarization Challenge), Legal Judgment Prediction (BDCI-21 Law Event Prediction Challenge), and Low-resource Translation (ChrEn dataset; Zhang et al., 2020). General ability is evaluated on MMLU (Hendrycks et al., 2021b), TriviaQA (Joshi et al., 2017), HellaSwag (Zellers et al., 2019), ARC-Challenge (Clark et al., 2018), IFEval (Zhou et al., 2023), CEval (Huang et al., 2023), and CLUEWSC (Xu et al., 2020). Results on specialized tasks are reported individually; general tasks are averaged across all training experiments with standard deviation across tasks (Tables 1 and 2).
Base model(s). All experiments use DeepSeek-V2-Lite (DeepSeek, 2024), a fine-grained MoE model with 66 non-shared experts and 2 shared experts per transformer layer, with top-6 routing (plus 2 shared experts, yielding 8 total activated experts per token). The model is chosen because its fine-grained expert segmentation makes it "uniquely suitable at the time of this study for our method, which benefits from expert specialization" (Section 4.3). The authors first train the model on a carefully curated alignment dataset that excludes math and code data, obtaining a "vanilla model" checkpoint for subsequent experiments—this alignment phase activates model ability across various domains while keeping Math/Code ability elementary to better verify performance gains.
Metrics. For Math evaluation, the paper reports exact-match accuracy on GSM8K and MATH. For Code evaluation, it uses pass@1 on HumanEval and MBPP. For the Text-to-JSON Intent task, it calculates exact match between model output and reference answer. For the remaining specialized tasks (Summarization, Legal Judgment, Translation), it employs GPT-4 (gpt-4-1106-preview) to score model outputs between 0 and 10 given reference answers, with evaluation instructions provided in Appendix G (Table 11). All evaluations use few-shot examples. For general ability benchmarks, standard accuracy metrics are used per dataset.
Baselines. The paper compares against two baselines. Full-Parameter Fine-Tuning (FFT) trains all 15.7B parameters of the model. Low-Rank Adaptation (LoRA) (Hu et al., 2021) adds low-rank matrices (rank 8, scaling 2) to all parameters for training except token embeddings and the language modeling head, following the standard formulation where weight updates are decomposed as $\mathbf{W}' = \mathbf{W} + \mathbf{B}\mathbf{A}$ with $\mathbf{B} \in \mathbb{R}^{d \times r}$ and $\mathbf{A} \in \mathbb{R}^{r \times k}$ . The Vanilla Model (the pre-trained checkpoint before any fine-tuning) is also reported as a reference point.
Generation budget / compute accounting. Compute is measured primarily in terms of trainable parameters (billions), training wall-clock time (minutes), and storage space for the trained parameters (GB). These metrics are reported in Section 5.2 and Figure 5. The training budget for all methods is controlled by setting a maximum of 500 training steps with evaluation every 100 steps, selecting the best checkpoint. Batch size is fixed at 32 sequences with sequence length 4096 across all methods. All experiments run on 2 nodes of 8× NVIDIA A100 PCIe GPUs on the HFAI cluster. For the ESFT method, the hyperparameter $p$ (Equation 8) controls the cumulative relevance threshold and thereby the fraction of selected experts—this is the primary efficiency-performance tradeoff knob, varied in Section 6.2 (Figure 6) to compare ESFT against LoRA under varying compute budgets. For LoRA, rank controls the efficiency-performance tradeoff and is swept up to 512 (beyond which LoRA would have more trainable parameters than FFT).
Cross-validation / statistical protocol. No formal cross-validation is reported for strategy selection. Instead, hyperparameters are selected via grid search: learning rates are searched over {1e-5, 3e-5, 1e-4, 3e-4} with final values of 3e-5 (FFT), 1e-4 (LoRA), and 1e-5 (ESFT). The LoRA rank is set to 8 and scaling to 2, following Hu et al. (2021). The threshold $p$ is set to 0.1 for ESFT-Gate and 0.2 for ESFT-Token, with the efficiency sweep in Section 6.2 providing the empirical basis for these choices. For general ability evaluation, performance for each benchmark is averaged across all training experiments (i.e., across tasks), with standard deviation reported. This is not a standard held-out validation protocol; rather, the paper treats each downstream fine-tuning run as a separate experiment and aggregates results.

Main Quantitative Results

Specialized Task Performance (Table 1)

The headline result: ESFT-Gate achieves an average specialized task score of 50.2 across all 8 tasks, compared to 51.0 for FFT and 44.9 for LoRA—matching FFT within 0.8 points while training only ~11% of the parameters (1.4B–1.85B vs. 15.7B). The Vanilla Model baseline achieves 33.6 average, confirming that all fine-tuning methods provide substantial gains over the pre-trained checkpoint.

Model enhancement tasks (Math and Code). On MATH, ESFT-Gate achieves 23.2 and ESFT-Token achieves 22.6, compared to FFT's 23.4, LoRA's 20.6, and the Vanilla Model's 19.6. The gap between ESFT and FFT is 0.2–0.8 points, while LoRA lags FFT by 2.8 points. On GSM8K, ESFT-Token achieves 66.0, ESFT-Gate achieves 64.9, FFT achieves 66.4, and LoRA achieves 58.9—ESFT-Token essentially matches FFT (within 0.4 points) while LoRA trails substantially. On HumanEval, ESFT-Gate achieves 43.3 (best overall, exceeding FFT's 42.1 and the Vanilla Model's 42.1) while ESFT-Token achieves 41.5—notably, FFT does not improve over the Vanilla Model's 42.1 on this task, suggesting possible overfitting. On MBPP, ESFT-Token achieves 42.6, FFT achieves 42.2, and LoRA achieves 44.8 (best), but all methods are close to the Vanilla Model's 44.6, indicating limited headroom.

Model adaptation tasks. These tasks show the clearest gains from fine-tuning and the strongest evidence for ESFT's competitiveness. On Intent Recognition, ESFT-Gate achieves 78.6, FFT achieves 78.8, and LoRA achieves 67.8—both ESFT variants match FFT, while LoRA underperforms by ~11 points. On Summarization, FFT achieves 69.4, ESFT-Gate achieves 65.8, and ESFT-Token achieves 65.4—ESFT trails FFT by ~4 points but leads LoRA (64.7) by a modest margin. On Legal Judgment, ESFT-Gate achieves 49.1 (best overall, exceeding FFT's 47.0 and LoRA's 39.7). On Translation, FFT achieves 38.4, ESFT-Token achieves 36.2, ESFT-Gate achieves 35.2, and LoRA achieves 23.1—ESFT approaches FFT while LoRA lags substantially. The largest performance gaps between Vanilla and fine-tuned models occur on these adaptation tasks (e.g., Intent: 16.8 → 78.8; Law: 17.1 → 47.0–49.1), confirming they represent genuine customization challenges.

Key pattern in Table 1. ESFT consistently outperforms LoRA significantly (average: 49.4–50.2 vs. 44.9) and approaches FFT (51.0) closely. The largest ESFT advantages over LoRA occur on the most specialized tasks (Intent: +11.8 points for ESFT-Gate over LoRA; Translation: +13.1 points for ESFT-Token over LoRA), consistent with the hypothesis that specialized tasks benefit most from expert-selective training.

General Ability Retention (Table 2)

The headline result: ESFT-Token achieves an average general ability score of 61.5 across seven benchmarks, compared to 62.4 for the Vanilla Model, 58.8 for FFT, and 59.1 for LoRA. ESFT causes the least performance degradation from the pre-trained model (drop of 0.9 points vs. 3.6 for FFT and 3.3 for LoRA).

Per-benchmark patterns. On CLUEWSC, ESFT-Token (80.9 ± 0.9) and ESFT-Gate (81.4 ± 1.1) essentially match the Vanilla Model (81.5) and FFT (80.9 ± 1.1), while LoRA drops to 74.3 ± 7.7. On TriviaQA, ESFT-Token (66.7 ± 1.8) and ESFT-Gate (66.5 ± 2.3) are close to the Vanilla Model (67.7) and FFT (65.9 ± 0.7). On IFEval, ESFT-Token achieves 40.7 ± 1.3 and ESFT-Gate achieves 40.2 ± 1.5, compared to the Vanilla Model's 42.5—a modest drop, but better than FFT's 34.2 ± 4.1 (which is a substantial 8.3-point degradation). On HellaSwag, FFT shows a notable drop (67.9 ± 3.8 vs. Vanilla 74.0) and ESFT-Gate similarly drops (68.2 ± 9.9, with high variance), while ESFT-Token (72.3 ± 3.6) and LoRA (72.8 ± 1.9) retain more. On CEval, MMLU, and ARC, all methods perform near the Vanilla Model with modest variations—CEval shows ESFT-Token at 59.6 ± 0.8 vs. Vanilla 59.9, essentially unchanged.

Standard deviation analysis. The standard deviations across tasks (reported in Table 2 after ±) reveal that ESFT's general performance is more stable than FFT and LoRA across different fine-tuning tasks. For IFEval, FFT shows ±4.1 standard deviation, indicating that fine-tuning on different tasks causes highly variable degradation, while ESFT-Token shows ±1.3, suggesting much more consistent retention regardless of which task was fine-tuned.

Computational Efficiency Results (Section 5.2, Figure 5)

The headline result: ESFT-Token and ESFT-Gate take 19.8 and 20.9 minutes average training time, respectively, compared to 28.5 minutes for FFT (a ~30% reduction) and 16.5 minutes for LoRA. Storage requirements are 2.57 GB (ESFT-Token) and 3.20 GB (ESFT-Gate) for trained parameters, compared to 28.6 GB for FFT (~90% reduction). LoRA requires the least storage of all methods (exact value unstated), but its specialized task performance (44.9) is substantially worse than ESFT's (49.4–50.2).

The paper emphasizes that ESFT "performs significantly better than LoRA in downstream task performance" while achieving training times that are "relatively close" to LoRA's—the 3.3–4.4 minute gap between ESFT and LoRA training times is modest compared to the 4.5–5.3 point gap in specialized task performance.

Expert Selection Analysis: How Many Experts Does ESFT Train? (Section 6.1, Figure 4)

The headline finding: ESFT trains 2–15 experts per layer out of 66 (75–95% fewer trainable parameters than FFT), with substantial variation across layers and tasks. Figure 4 visualizes the number of trained experts per layer for each of the six tasks, revealing several patterns:

Task-dependent selection breadth. More specialized tasks (Math, Translation) use fewer experts on average—Math requires 2–8 experts per layer, Translation requires 2–6. This aligns with the hypothesis that specialized tasks benefit from narrow expert selection. Broader tasks (Intent, Law) use 5–15 experts per layer, reflecting a need for more diverse expert knowledge.

Layer-dependent concentration. For most tasks, middle layers consistently require fewer experts than early and late layers. The paper states: "few experts are chosen in the middle layers, indicating that expert distribution is more concentrated in these layers." This is visible in Figure 4 as a U-shaped pattern across layers. The interpretation: middle layers likely process more abstract, task-agnostic representations where routing is concentrated onto a small set of specialized experts, while early and late layers handle more varied processing (input token patterns, output token prediction) requiring broader expert coverage.

ESFT-Token vs. ESFT-Gate selection differences. ESFT-Token generally selects fewer experts than ESFT-Gate, consistent across tasks. For example, in Math, ESFT-Token uses 2–6 experts while ESFT-Gate uses 4–8. In Code, ESFT-Token uses 2–8 while ESFT-Gate uses 4–10. This explains why ESFT-Token achieves better general-task retention (61.5 vs. 60.6): fewer trained experts means less interference with other tasks' experts.

Efficiency Tradeoff Analysis: ESFT vs. LoRA Under Varying Budgets (Section 6.2, Figure 6)

The headline result: ESFT consistently outperforms LoRA at every point in the efficiency-performance tradeoff space on the Math task, for both specialized (MATH accuracy) and general (average of seven benchmarks) ability. Figure 6 plots specialized and general performance against the fraction of trained parameters, varying $p$ for ESFT and rank for LoRA.

Specialized performance. ESFT-Token peaks at $p = 0.5$ (approximately 26 on MATH) and ESFT-Gate peaks at $p = 0.3$ (approximately 25). Both plateau early—ESFT-Token performance saturates around $p = 0.2$ and ESFT-Gate around $p = 0.1$ . LoRA shows a monotonic increase with rank, reaching ~22 at rank ~512, but never matches ESFT's performance at any rank. The curves converge at the extreme (when ESFT trains nearly all experts and LoRA trains nearly full-rank), but ESFT maintains a clear advantage throughout the practically relevant regime.

General performance. ESFT-Token maintains stable general ability across all $p$ values (roughly flat curve near 62), while ESFT-Gate shows a slight decline at higher $p$ (dropping from ~62 at $p = 0.1$ to ~60 at $p = 0.5$ ). LoRA's general performance is more volatile—initially around 61 at low rank, dipping to ~59 at moderate ranks, then recovering to ~60 at rank ~512. The paper notes that "ESFT-Token peaks in both specialized and general ability at p=0.5, while ESFT-Gate peaks at p=0.3 for specialized and p=0.1 for general ability," highlighting a tension in ESFT-Gate between specialized and general performance at higher $p$ .

Key practical insight. ESFT-Gate achieves near-maximum specialized performance at $p = 0.1$ (only ~10% of routing mass needed), while further increasing $p$ to 0.3 brings diminishing specialized gains and degrades general ability. This suggests the threshold can be set conservatively (low $p$ ) with little performance penalty, maximizing efficiency and general-task retention simultaneously.

Training Shared vs. Non-Shared Parameters Ablation (Section 6.3, Table 3)

The headline result: Training only task-relevant non-shared experts achieves the best balance of specialized performance (49.4), general performance (61.5), and parameter efficiency (1.4B trainable parameters)—the ESFT default configuration. Table 3 systematically enumerates six configurations varying which of three parameter groups (non-shared experts, shared experts, non-expert parameters) are trained.

Key comparisons from Table 3:

All parameters (FFT, 15.7B): 51.0 specialized, 58.8 general. Best specialized performance, worst general retention.
Relevant non-shared + all shared + non-expert (2.7B): 50.8 specialized, 60.3 general. Near-FFT specialized performance (within 0.2) with ~6× fewer trained parameters and 1.5-point better general retention.
Relevant non-shared only, no shared, no non-expert (1.4B, ESFT default): 49.4 specialized, 61.5 general. Best general retention, only 1.6 points below FFT specialized.
Shared experts only (450M): 47.4 specialized, 61.2 general. Training only the tiny shared expert subset yields surprisingly strong specialized performance (only 3.6 below FFT) but slightly worse general retention than ESFT default.
All non-shared, no shared, no non-expert (1.85B): 49.8 specialized, 60.7 general. Training all non-shared experts instead of only relevant ones improves specialized by 0.4 points but degrades general by 0.8 points—a poor tradeoff.
Vanilla (no training, 0B): 33.8 specialized, 62.4 general.

The paper's proposed strategies emerge from this table:

Prioritize specialized ability: train all shared parameters and task-relevant non-shared experts (+ non-expert parameters, 2.7B total), achieving 50.8 specialized with reasonable general retention.
Balance specialized and general ability, and computational efficiency: train only task-relevant non-shared experts (1.4B, the ESFT default), achieving 49.4 specialized with 61.5 general.

The monotonic relationship. Specialized performance increases monotonically with trainable parameters (from 47.4 at 450M to 51.0 at 15.7B), while general performance decreases as shared parameters are trained. The paper interprets this as evidence that "training shared parameters is more likely to cause overfitting on downstream tasks and forgetting on general tasks compared to training non-shared parameters."

Math and Code Degradation from Specialized Fine-Tuning (Appendix H, Table 8)

The headline result: When fine-tuned on non-Math/Code specialized tasks, FFT causes average Math/Code performance to drop 9.0 points (from 40.5 to 31.5) and LoRA drops 12.4 points (to 28.1), while ESFT-Token drops only 0.8 points (to 39.7) and ESFT-Gate drops 0.7 points (to 39.8). This is the most dramatic evidence of ESFT's ability to preserve general-domain performance outside the fine-tuning task.

Breaking down by domain: on MATH, FFT drops to 15.1 (from 19.6), LoRA drops to 11.8, but ESFT-Token maintains 19.4 and ESFT-Gate maintains 19.5—essentially no degradation. On GSM8K, FFT drops to 40.3 (from 55.9), a 15.6-point decline, while ESFT-Token maintains 55.2. On HumanEval, FFT drops to 30.2 (from 42.1) while ESFT-Token maintains 39.5. On MBPP, all methods are within range of the Vanilla Model's 44.6.

This is a crucial negative result for FFT and LoRA: fine-tuning on a specialized task like legal judgment prediction substantially damages mathematical reasoning and code generation capabilities, even though these domains were not involved in training. ESFT's expert-selective approach isolates the parameter updates, preserving the experts that encode math and code capabilities (which are disjoint from those used by legal judgment, summarization, etc., as shown in Figure 3).

Ablation Studies and Robustness Checks

Expert relevance score importance (Table 4): Replacing ESFT's relevance-score-selected experts with randomly selected experts (keeping the same number of experts per layer) reduces average specialized performance by 2.8 points for ESFT-Token (from 49.4 to 46.6) and 4.4 points for ESFT-Gate (from 50.2 to 45.8). The largest degradation occurs on Translation (ESFT-Gate drops 20.4 points, ESFT-Token drops 13.5 points), indicating that correct expert selection is especially critical for low-resource, highly specialized tasks. The smallest degradation occurs on MBPP (-0.2 for ESFT-Token, +1.6 for ESFT-Gate, meaning random selection actually outperformed—a counterintuitive result suggesting MBPP may draw on broadly distributed expert knowledge). This ablation confirms that the relevance score functions (Equations 6–7) are not simply identifying experts that happen to be active—they identify experts whose specialization genuinely matters for task performance.
Fine-grained expert segmentation requirement (Figure 7): Simulating coarse-grained MoE by grouping experts (group size 1 = original 66 experts, group size 2 ≈ 32 pseudo-experts, group size 4 ≈ 16 pseudo-experts) reveals that ESFT's MATH performance degrades from ~23 (group size 1) to ~21 (group size 2) to ~18 (group size 4), while FFT degrades only from ~23 to ~21. Simultaneously, the number of experts ESFT must train increases (since each pseudo-expert contains multiple original experts that must be trained jointly). The faster degradation of ESFT relative to FFT demonstrates that fine-grained segmentation is not just beneficial for ESFT—it is a prerequisite. The paper describes this as showing "our method, and even effective LLM customization, highly rely on a fine-grained segmented LLM architecture with more specialized experts."
Data mixing for FFT/LoRA vs. ESFT (Appendix F, Tables 9–10): Mixing alignment data (1:1 ratio) with task data during fine-tuning improves general-task performance for FFT (from 58.5 to 58.8, +0.3) and LoRA (from 55.0 to 59.1, +4.1) but actually degrades ESFT-Token (from 62.3 to 61.5, −0.8) and ESFT-Gate (from 62.2 to 60.6, −1.6). For downstream specialized tasks, mixing data causes minor performance decreases across all methods (−0.6 to −2.2 for ESFT, −1.7 for FFT, −2.2 for LoRA). The paper explains this as evidence that "ESFT is inherently capable of adapting to downstream tasks without significant performance degradation in general tasks, even without added alignment data." Mechanistically, this makes sense: since ESFT only updates task-relevant experts, it doesn't interfere with shared-parameter-encoded general knowledge, so alignment data provides no benefit (and may dilute the task-specific training signal).
Expert affinity sample size (Appendix C, Figure 8): To determine how much data is needed for reliable expert selection, the paper measures the overlap of Top-6 expert sets between two independent samples of varying sizes. As sample size reaches $2^{17}$ tokens (equivalent to 32 sequences × 4096 tokens), the overlap converges to approximately 5.5–6 out of 6 shared experts for all tasks. At smaller sample sizes (e.g., $2^{13}$ = 4 sequences), overlap is lower and task-dependent, ranging from ~4–5 for Math and Code to ~3 for Intent and Law. This empirically justifies the choice of 32 samples as sufficient for stable expert selection while being computationally inexpensive (32 forward passes vs. potentially thousands of training steps).
ESFT with ReST $^{EM}$ (not applicable to this paper—no such experiment). The paper does not report any reinforcement learning or iterative self-improvement experiments. The revision model experiments from the example (Section 6 in the reference example) are from a different paper entirely and do not appear here.
Qualitative expert visualization (Appendix E, Figure 9): Visualizing which tokens are processed by trainable experts (deeper color = more trainable experts involved) shows that ESFT's selected experts cover key task-relevant words: "意图" (Intent) in intent recognition; "婚后" (Post-marriage), "要求" (request), "原告" (plaintiff), "被告" (defendant) in legal judgment; numerical tokens "3", "5", "6", "7" in Math; and keywords like "const" in Code. This provides intuitive confirmation that the expert selection is not arbitrary—the selected experts correspond to tokens that carry task-relevant semantic content.

Critical Assessment

Claim 1: "ESFT matches or even surpasses the performance of full-parameter fine-tuning" (Abstract, Section 5.1)

The data partially supports this claim, but with important qualifications. Table 1 shows ESFT-Gate's 50.2 average specialized score compared to FFT's 51.0—a 0.8-point gap that is small but consistent. ESFT never clearly surpasses FFT on average. The claim of "matching" is reasonable: the 0.8-point gap is modest compared to the 6.1-point gap between FFT and LoRA (51.0 vs. 44.9), and on individual tasks (HumanEval: 43.3 vs. 42.1; Legal Judgment: 49.1 vs. 47.0) ESFT does outperform FFT. However, the paper does not report confidence intervals or statistical significance for the task-level comparisons, making it impossible to determine whether these individual-task "wins" are reliable or noise. On a 500-step maximum training budget, FFT may be under-trained relative to its capacity—ESFT might match it simply because the training budget is insufficient for FFT to converge, not because ESFT's selective training is equivalently expressive.

What would strengthen this claim: Running FFT to convergence (more steps), reporting statistical significance for pairwise comparisons, and testing on a larger set of tasks and model scales.

Claim 2: "ESFT reduces the storage of up to 90% and training time up to 30% compared to full-parameter fine-tuning" (Abstract, Section 5.2)

Strongly supported with concrete numbers. Figure 5 and Section 5.2 report: FFT storage = 28.6 GB, ESFT storage = 2.57–3.20 GB → ~89–91% reduction. FFT training time = 28.5 minutes, ESFT training time = 19.8–20.9 minutes → ~27–31% reduction. These are direct measurements from the experimental hardware, not projections. The storage reduction is the more impactful figure: it means a practitioner can store 11 ESFT adapters in the space of 1 FFT checkpoint, which is practically significant for multi-task deployment.

A caveat: The training time reduction (30%) is modest—LoRA achieves 42% reduction while ESFT achieves 27–31%. The paper's claim that ESFT has "excellent performance in training time" while stating LoRA "achieves a shorter training time" is accurate but dampens the impact of the training time claim. The storage reduction is the primary efficiency contribution.

Claim 3: "ESFT prevents the decrement of specialization in full-parameter fine-tuning" and "better maintains performance in general tasks" (Section 1, Section 5.1)

Supported with strong evidence from Tables 2 and 8. Table 2 shows ESFT-Token retaining 61.5 general average vs. FFT's 58.8—a difference of 2.7 points across seven diverse benchmarks. Table 8 provides the most dramatic evidence: after fine-tuning on specialized tasks (Intent, Summary, Law, Translation), FFT's Math/Code average drops 9.0 points while ESFT's drops <1 point. This is a compelling demonstration that ESFT's parameter isolation genuinely prevents catastrophic forgetting in untrained domains.

However, the mechanism claim ("prevents the decrement of specialization") is less directly tested. The paper argues that FFT dilutes expert specialization by training all experts, while ESFT preserves it by training only task-relevant ones. But no experiment directly measures "expert specialization" before and after fine-tuning (e.g., by measuring routing concentration or within-task vs. across-task routing divergence post-training). The general-task retention numbers are consistent with the mechanism claim but do not directly verify it. An experiment measuring whether FFT's experts become less specialized (more uniform routing distributions, higher cross-task expert overlap) after fine-tuning would test the proposed mechanism directly.

What was not tested, and why it matters

Only one MoE model family (DeepSeek-V2-Lite) is evaluated. The paper acknowledges this in Limitations: "our method was only tested on the DeepSeek-V2-Lite MoE model. The conclusions drawn from this model require further validation when applied to other contexts." This is a significant constraint. DeepSeek-V2-Lite has specific architectural features (fine-grained segmentation, shared experts, specific K=8 routing) that may be necessary for ESFT's success. Without testing on other fine-grained MoE architectures (if and when they become available), the generality of ESFT beyond the DeepSeek ecosystem is unproven.

No comparison to MoE-specific LoRA variants (e.g., MoELoRA). The paper cites Liu et al. (2023)'s MoELoRA in Section 2.1 but does not implement or compare against it. MoELoRA applies LoRA selectively using MoE-style routing—it is a natural baseline for a paper claiming to be the first to exploit MoE structure for PEFT. The comparison to standard LoRA (which applies low-rank adapters universally) is weaker than a comparison to a method that, like ESFT, attempts to route adaptation to relevant components.

No experiment testing dynamic expert selection. ESFT selects experts once based on pre-fine-tuning routing and never updates the selection. An alternative would be to periodically re-evaluate routing statistics during training and adjust the selected set. The paper does not test this, making it unclear whether the static selection is a genuine feature (simplicity, stability) or a missed opportunity to improve performance (by adapting to routing shifts).

The difficulty estimation/selection cost is not fully accounted for. The expert selection phase requires 32 forward passes of sequence length 4096—this is modest (equivalent to processing 131K tokens) but is not included in the training time comparisons (Figure 5). For a fair comparison, ESFT's training time should include this selection overhead. In practice, 32 forward passes is negligible compared to 500 training steps, so this is a minor concern.

The paper does not test combination with LoRA-style adaptation within selected experts. ESFT trains the full weight matrices of selected experts. A hybrid approach could select experts by routing, then apply LoRA within those experts (training only low-rank updates rather than full weights). This could further reduce parameter count while maintaining the expert selection benefit. The paper does not explore this, focusing exclusively on full-weight fine-tuning of selected experts.

The 500-step training budget is a fixed, untuned hyperparameter. The paper sets a maximum of 500 steps for all methods and selects the best checkpoint every 100 steps. There is no analysis of whether different methods converge at different rates—ESFT, training fewer parameters, might converge faster and reach its best performance at 200 steps, while FFT might need 1000+ steps to converge. Using the same step budget for all methods could favor ESFT if FFT is under-converged. A more rigorous comparison would track validation performance across a wide range of step budgets and report the best performance each method can achieve given enough training.

The specialized task test sets are small or evaluator-dependent. The four adaptation tasks rely on GPT-4-based scoring (Summarization, Legal Judgment, Translation) or exact-match (Intent). GPT-4 scoring introduces evaluator variance that is not quantified—different prompts or model versions could produce different scores. The exact-match Intent task may be brittle (minor formatting differences count as errors). Task-level sample sizes are not reported, making it impossible to assess whether performance differences are statistically meaningful. The Code tasks (HumanEval: 164 problems; MBPP: ~500 problems) are reasonably sized; the Math tasks (GSM8K: 1,319 test; MATH: 5,000 test problems) are substantial. But the adaptation tasks' sizes are unstated.

Coarse-grained MoE simulation is imperfect. Appendix B's expert grouping method (greedy clustering by co-occurrence-based similarity) simulates coarse granularity but does not replicate the behavior of a genuinely trained coarse-grained MoE. A real coarse-grained model would have learned different routing patterns and expert specializations during pretraining. The simulation shows that ESFT degrades when forced to train grouped experts, but this might over- or under-estimate the degradation on a real coarse-grained model.

Summary of assessment

The paper demonstrates convincingly that on the DeepSeek-V2-Lite fine-grained MoE architecture, pre-selecting experts by routing affinity and fine-tuning only those experts achieves specialized task performance competitive with full fine-tuning while substantially preserving general capabilities and reducing storage by ~90%. This is a genuine contribution to the PEFT literature, particularly in establishing that MoE routing provides a usable signal for parameter selection.

The primary limitations are: (1) single model family evaluation, leaving generality to other MoE architectures unverified; (2) no comparison to MoE-aware PEFT baselines like MoELoRA; (3) absence of statistical testing for performance claims; and (4) the coarse-grained simulation experiment (Figure 7) is informative but not a substitute for testing on actual coarse-grained MoE models. The paper's strongest claims—the 90% storage reduction and the preservation of general ability—are well-supported by the reported data. The claim of "matching or surpassing" full fine-tuning is supported directionally but would benefit from convergence-equivalence checks and broader evaluation.

6. Limitations and Trade-offs

1. Single Model Family Evaluation — Generality to Other MoE Architectures Is Unverified

The assumption. ESFT's entire mechanism depends on two architectural properties: (a) fine-grained expert segmentation producing genuine expert specialization, and (b) routing distributions that are both within-task concentrated and cross-task divergent. The paper validates that these properties hold for DeepSeek-V2-Lite (66 non-shared experts, top-6 routing, with shared experts and fine-grained segmentation inherited from DeepSeekMoE). However, the authors explicitly acknowledge that this is the only model tested:

"Due to the limitation of the availability of other fine-grained MoE models, our method was only tested on the DeepSeek-V2-Lite MoE model. The conclusions drawn from this model require further validation when applied to other contexts." (Limitations, §7)

The consequence. A practitioner evaluating ESFT for a different MoE model cannot assume the method will transfer. The routing concentration patterns documented in Figures 2–3 may be specific to DeepSeek-V2-Lite's pretraining recipe, expert count, routing mechanism (top-6 with softmax gating), or shared expert design. For instance, MoE models trained with different load-balancing losses (which encourage more uniform expert utilization) may exhibit less concentrated routing, making the "few experts handle most tokens" property weaker and reducing ESFT's efficiency advantage. Models with different expert counts (e.g., 160 experts as in DeepSeek-V2 full) may exhibit different specialization granularity. Models without shared experts may route universal knowledge through non-shared experts, meaning freezing those experts could damage general capabilities in ways not captured by this paper's analysis. Most critically, coarse-grained MoE models like Mixtral (8 experts, top-2 routing) or DBRX (16 experts, top-4 routing) would almost certainly not support ESFT effectively — Figure 7 demonstrates that simulated coarse granularity causes disproportionate ESFT degradation. But the simulation relies on post-hoc expert grouping via co-occurrence similarity (Appendix B), which does not replicate the routing patterns a genuinely trained coarse-grained model would learn during pretraining.

Evidence in the paper. Figure 7 shows that when experts are grouped to simulate coarser granularity (group sizes 2 and 4), ESFT's MATH accuracy degrades from ~23 (original fine-grained) to ~21 (group size 2) to ~18 (group size 4), while FFT degrades only from ~23 to ~21. The number of experts ESFT must train also increases with group size (each "coarse pseudo-expert" bundles multiple original experts, forcing joint training). This demonstrates that architectural granularity matters causally for ESFT's effectiveness. However, this is a simulation on a single model, not a cross-model validation.

Mitigation status. The authors are transparent about the limitation (cited above). They identify validation on other fine-grained MoE models as necessary future work. No attempt is made to test ESFT on, e.g., Mixtral or DBRX — the paper restricts its claims to DeepSeek-V2-Lite. Practitioners using non-DeepSeek MoE architectures should validate ESFT on their specific model before adopting it, using the probing methodology in §3.2 (Figures 2–3) to check whether routing concentration and cross-task divergence exist at comparable levels.

2. Expert Selection Cost Is Not Included in Headline Efficiency Numbers

The assumption. The paper reports training time and storage reductions relative to FFT (§5.2, Figure 5): ESFT reduces training time by ~30% (19.8–20.9 minutes vs. 28.5 minutes for FFT) and storage by ~90% (2.57–3.20 GB vs. 28.6 GB). These numbers are computed from the fine-tuning phase only — the expert selection phase (32 forward passes of sequence length 4096 to collect routing statistics, as described in §3.3 and Appendix C) is excluded from the timing comparison. The paper does not explicitly claim to include this cost, but the headline framing ("ESFT exhibits several advantages in terms of training time") does not qualify the numbers.

The consequence. In absolute terms, the expert selection overhead is modest: 32 forward passes on sequences of length 4096 process approximately 131K tokens. On the experimental hardware (2 nodes × 8 A100 GPUs), this likely adds only 1–2 minutes of wall-clock time — negligible relative to 20–28 minutes of fine-tuning. However, the concern is methodological, not practical: if ESFT's training time advantage is reported as 30% without including the selection step, the comparison to FFT and LoRA (which require no pre-training selection phase) is incomplete. More importantly, the method's practical deployability depends on the availability of task-specific training data for the selection phase. ESFT needs a labeled sample from the target task to estimate routing statistics. If the practitioner's goal is to adapt to a task with very limited data (the exact scenario where PEFT is most valuable), the selection phase consumes a non-trivial fraction of the available data — and the paper does not characterize how selection quality degrades as data becomes scarcer. Figure 8 (Appendix C) shows that the overlap of Top-6 expert sets between two independent samples increases with sample size, converging around 32 sequences. Below this point (e.g., at 4–8 sequences), overlap is lower and task-dependent. A practitioner with only 8 labeled examples may get a less reliable expert selection, potentially reducing fine-tuning performance below the reported numbers.

Evidence in the paper. Figure 8 (Appendix C) quantifies the relationship between sample size and expert selection stability. At 32 sequences, overlap is 5.5–6.0 out of 6 for all tasks. At 16 sequences, overlap is ~5–5.5. At 8 sequences, overlap drops to ~4–5 for Math and Code but ~3–4 for Intent and Law. This implies that for adaptation-style tasks (Intent, Law, Summary, Translation), reliable expert selection requires at least 16–32 labeled examples — a modest but real data requirement for extremely low-resource scenarios. The paper does not report fine-tuning performance when experts are selected from smaller sample sizes, so the downstream consequence of unstable selection is unquantified.

Mitigation status. The paper acknowledges the cost in the efficiency tradeoff discussion but does not address it in the headline metrics. No experiment measures end-to-end time (selection + fine-tuning) vs. FFT. The paper's guidance on minimum sample size (Appendix C, Figure 8) is informative but doesn't map to performance outcomes. A complete accounting would either include selection cost in the timing numbers or demonstrate that selection quality remains adequate at sample sizes small enough to be negligible in all practical regimes.

3. Static Expert Selection — No Adaptation to Routing Shifts During Training

The assumption. ESFT selects experts once before any gradient-based training begins, using routing statistics collected from the frozen pre-trained model (§3.3). The selected set $\mathcal{E}_s^l$ remains fixed throughout the entire fine-tuning process. The router parameters (expert centroids $\mathbf{e}_i^l$ in Equation 3) are frozen, as are all non-expert parameters (attention weights, layer normalization, embeddings). This design choice assumes that the pre-training routing distribution is a valid proxy for which experts will be relevant after fine-tuning updates some experts' parameters.

The consequence. When ESFT updates the weights of selected experts, those experts' FFN outputs change. These changed outputs propagate through residual connections to subsequent layers, altering the hidden representations $\mathbf{u}_t^l$ that serve as input to later MoE layers' routers. Even though the router parameters $\mathbf{e}_i^l$ are frozen, the routing decisions can shift because the token representations $\mathbf{u}_t^l$ themselves have changed. A token that the router originally assigned to experts {3, 7, 12, 18, 25, 41} might now be assigned to a different set — including experts that ESFT froze. If routing shifts toward frozen experts, the model is forced to process task tokens through parameters that cannot adapt, potentially capping fine-tuning performance. If routing shifts away from selected experts, the trained parameters become underutilized, wasting the adaptation budget.

The paper provides no mechanism to detect or correct for routing shifts during training. An alternative design — periodic re-evaluation of routing statistics and dynamic adjustment of the selected expert set — is not explored. The static selection also means that if the initial routing statistics are noisy (due to small sample size, as discussed in Limitation 2), the error persists throughout training with no opportunity for correction.

Evidence in the paper. The paper does not directly measure whether routing distributions shift during ESFT fine-tuning. No experiment compares pre- and post-fine-tuning routing patterns for the same task data. The strong experimental results (ESFT matching FFT on specialized tasks, Table 1) suggest that routing shifts, if they occur, are not severe enough to undermine performance. However, this is an empirical observation on DeepSeek-V2-Lite, not a guarantee. The model's router may be relatively stable because the selected experts' output changes are localized (only 2–15 out of 66 experts per layer are updated, Figure 4) and the learning rate is low (10⁻⁵, §4.3). A different architecture with more interconnected routing or a task requiring larger parameter changes might exhibit more pronounced routing drift.

Mitigation status. The paper does not address this limitation explicitly. The design choice to freeze router parameters is intentional and motivated by consistency — the selection is based on these parameters, so changing them would undermine the selection rationale. But the possibility of de facto routing change through hidden state drift is not discussed. Future work could either measure routing stability during ESFT training (a simple diagnostic: compare Top-6 expert overlap for the same tokens before and after fine-tuning) or implement dynamic selection as an extension.

4. No Comparison to MoE-Aware PEFT Baselines — Only Standard LoRA Is Evaluated

The assumption. The paper positions ESFT as the first PEFT method that exploits MoE architecture structure for parameter selection (§2.1). The experimental baselines are Full-Parameter Fine-Tuning (FFT) and standard LoRA applied uniformly to all weight matrices. LoRA's limitations for MoE are discussed in §2 (Context and Motivation) — it modifies all experts identically, cannot allocate capacity differentially by layer, and does not preserve expert specialization. However, the paper does not compare against methods that, like ESFT, attempt to route adaptation to relevant parts of the model. The citation of MoELoRA (Liu et al., 2023) in §2.1 acknowledges its existence but does not implement or evaluate it.

The consequence. The demonstrated advantage of ESFT over LoRA (44.9 → 50.2 average specialized score, Table 1) conflates two distinct contributions: (a) selecting which parameters to train based on MoE routing, and (b) training full expert weights rather than low-rank adapters. A fairer baseline would be a method that uses MoE-aware routing for adaptation allocation but with a comparable parameter budget — for instance, MoELoRA, which applies LoRA adapters only to selected experts based on routing. If such a baseline achieved similar performance to ESFT with even fewer parameters (since LoRA updates are lower-rank than full-weight updates), the specific contribution of ESFT's full-weight fine-tuning would be called into question. Conversely, if MoELoRA underperformed ESFT, it would strengthen the claim that full-weight updates to task-relevant experts are necessary.

More broadly, the paper's claim to be the first PEFT method "selecting part of the experts based on their downstream task affinity, as a unique selection dimension exclusive to the sparse MoE architecture" (§2.1) would be undermined if prior work already proposed routing-based adaptation allocation, even if implemented differently. The paper's novelty relative to MoELoRA specifically needs to be established, not assumed.

Evidence in the paper. None. MoELoRA is cited in §2.1 as "an MoE-based parameter efficient fine-tuning method for multi-task medical applications" but is not used as a baseline, discussed in the experimental design, or mentioned in the limitations. The comparison space in Section 6.2 (Figure 6) varies rank for standard LoRA and $p$ for ESFT, showing ESFT outperforms LoRA at every point — but this is a comparison against an architecture-agnostic baseline, not against a method that also exploits routing.

Mitigation status. The paper does not address the omission. Adding MoELoRA (or a reimplementation) as a baseline would be a straightforward extension. The code release (https://github.com/deepseek-ai/ESFT) may enable such comparisons, but the paper's conclusions about ESFT's superiority should be understood as relative to architecture-agnostic PEFT methods, not to all possible MoE-aware approaches.

5. Hard Problems Receiving Minimal Absolute Gains — Test-Time Compute Analogy to Capability Ceilings

The assumption. ESFT selects experts based on the model's existing routing patterns. If the pre-trained model routes very few tokens to a particular set of experts for a given task — or routes tokens diffusely across many experts — the method has limited signal for which experts to train, and the trained experts have limited capacity to improve performance because they process only a small fraction of tokens.

The consequence. ESFT inherits a capability ceiling proportional to what the pre-trained model already knows. Tasks where the base model's performance is very low cannot be rescued by selective expert fine-tuning, because the routing distribution doesn't concentrate on a small set of task-relevant experts — the model hasn't learned to recognize the task patterns at all. This is visible in the paper's results: the Translation task (Cherokee-to-English, a low-resource language pair) shows the largest gap between ESFT (35.2–36.2) and FFT (38.4) in Table 1, and the random expert selection ablation causes the largest degradation on this task (ESFT-Gate drops 20.4 points with random experts, Table 4). The model's pre-trained routing provides a weaker signal for Cherokee translation, making expert selection less reliable and the ceiling lower.

More generally, ESFT can only amplify capabilities the model already possesses. If a task requires fundamentally new knowledge or reasoning patterns not present in the pre-trained expert specializations, ESFT has no mechanism to acquire them — because the experts that would be relevant aren't activated or don't exist as distinct specializations. FFT, by contrast, can repurpose any expert for the new task by retraining its weights from scratch. This is analogous to the finding in the test-time compute paper (Example) that hard problems (bin 5) show near-zero improvement regardless of inference budget — some capabilities can only be acquired through comprehensive retraining, not through selective amplification.

Evidence in the paper. Table 1 shows that ESFT's relative underperformance vs. FFT is largest on Translation (35.2–36.2 vs. 38.4) and Summarization (65.4–65.8 vs. 69.4) — tasks that are likely less well-represented in the pre-training data than Math or Code. Table 4 shows Translation suffers the largest absolute degradation from random expert selection (13.5–20.4 point drop), indicating that expert selection quality is poorest for this task. Figure 2 shows Translation's normalized gate distribution is less sharply concentrated than Math's or Code's (the curve declines more gradually), suggesting routing provides a weaker signal.

Mitigation status. The paper does not directly discuss this ceiling or characterize which tasks fall below it. The distinction between "model enhancement" tasks (Math, Code — where the base model already performs decently) and "model adaptation" tasks (Intent, Summary, Law, Translation — where the base model performs poorly) in §4.1 implicitly acknowledges that different task categories may behave differently, but ESFT's performance on adaptation tasks is presented as a success without analyzing which adaptation tasks push against the method's limits. A more complete analysis would correlate ESFT's relative performance vs. FFT with the base model's pre-training task affinity (e.g., routing concentration, pass@1 on the task before fine-tuning).

6. Training Budget Is Uniform Across Methods — FFT May Be Under-Converged

The assumption. All methods (FFT, LoRA, ESFT) are trained for a maximum of 500 steps with evaluation every 100 steps, selecting the best checkpoint (§4.3). Learning rates are tuned separately for each method (3 × 10⁻⁵ for FFT, 10⁻⁴ for LoRA, 10⁻⁵ for ESFT) via grid search. The implicit assumption is that 500 steps is sufficient for all methods to reach near-optimal performance on all tasks.

The consequence. FFT trains 15.7B parameters, ESFT trains 1.4–1.85B, and LoRA trains a low-rank decomposition applied to all weight matrices. These methods have fundamentally different optimization landscapes. FFT, with its much larger parameter space, may converge more slowly — it needs more steps to traverse the loss surface and may still be improving at step 500. ESFT, with fewer parameters and a lower learning rate, may converge faster and reach its performance ceiling well before step 500. If FFT is under-trained relative to its capacity, the paper's central claim — that ESFT "matches or even surpasses" FFT performance — may be an artifact of the fixed training budget rather than a genuine equivalence. Extending FFT training to 1000 or 2000 steps might widen the gap, revealing that ESFT's efficiency comes with a performance cost that is masked by insufficient FFT training.

The paper does not report training curves (validation performance vs. step) for any method, making it impossible to assess convergence. The 500-step budget is described as a maximum, with evaluation every 100 steps and best checkpoint selected — but whether any method had plateaued by step 500 is unknown. The lower learning rate for ESFT (10⁻⁵ vs. 3 × 10⁻⁵ for FFT) also means ESFT takes smaller steps per update, which could partially offset its faster per-step convergence.

Evidence in the paper. None. No training curves are shown. No experiment varies the training budget to assess convergence. The paper states the maximum steps and evaluation frequency as hyperparameters but provides no justification for the 500-step choice or evidence that it is adequate for all methods. The learning rate grid search (§4.3) is the only optimization tuning reported.

Mitigation status. The paper does not address this limitation. A standard remedy would be to train all methods until validation performance plateaus (early stopping on a held-out set) rather than imposing a fixed step budget, or to report performance at multiple budget levels to show that rankings are stable. The current evidence convincingly shows ESFT outperforms LoRA and approaches FFT under a shared 500-step budget, but the degree to which FFT would pull ahead given more training is unknown. This is a methodological weakness that weakens the strength of the "matches FFT" claim, though it does not undermine the practical efficiency advantages, since practitioners would also prefer methods that converge faster with fewer resources.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new axis of parameter selection for PEFT that is uniquely available in fine-grained MoE architectures, and in doing so, it reframes the relationship between a model's internal organization and the downstream adaptation process. The shift is not a paradigm overthrow — ESFT does not make LoRA or adapters obsolete — but it establishes a previously unrecognized design dimension: that routing statistics, collected from simple forward passes with no gradient computation, provide a task-specific parameter importance signal that can drive efficient fine-tuning without any learning-based selection.

The conceptual reframing. Prior to this work, the PEFT field operated on an implicit assumption that parameter importance for a downstream task must be discovered through optimization. LoRA learns which low-rank directions matter via gradient descent. Sparse fine-tuning methods learn binary masks through iterative pruning and retraining. Adapter-based methods sidestep the selection problem entirely by adding new parameters rather than selecting existing ones. ESFT challenges this assumption directly: in an MoE model with fine-grained expert specialization, the model already "knows" which experts are relevant to a task — it encodes this information in its routing distribution, and you can read it off before training begins. The paper's probing experiments (Figures 2–3) are not supplementary diagnostics; they are the empirical warrant for this claim. The sharply concentrated, task-divergent routing patterns demonstrate that the signal exists and is reliable across diverse task types.

This reframing matters because it changes what practitioners should measure before fine-tuning an MoE model. Rather than immediately launching into gradient-based training with a generic PEFT method, they should first ask: "Does this model's routing distribution concentrate on a sparse set of experts for my task?" If yes, ESFT provides a more efficient path than architecture-agnostic alternatives. If no, the model may lack the expert specialization that ESFT depends on, and a different method (LoRA, FFT) is more appropriate. The paper provides the diagnostic tools (routing concentration analysis, cross-task overlap heatmaps, sample size convergence plots in Appendix C) to answer this question empirically.

Reconciling contradictory perspectives on MoE interpretability. Prior work on MoE expert specialization (DeepSeekMoE, Dai et al., 2024) characterized what experts learn — showing that fine-grained segmentation produces experts specialized in different knowledge types. But this characterization was primarily descriptive; it did not answer the question of how this specialization could be exploited for downstream tasks. Other work on MoE routing (e.g., load-balancing analyses in Switch Transformer, Fedus et al., 2021) focused on training stability and computational efficiency, treating routing as a mechanism to be optimized rather than a signal to be read. ESFT bridges these perspectives: it shows that expert specialization is not merely an interesting property of pre-trained MoE models, but a usable resource for efficient adaptation. The routing distribution transitions from being an internal mechanism (how the model allocates compute) to being an external interface (how the practitioner selects what to train). This is a meaningful conceptual advance — it converts a descriptive finding ("experts specialize") into a prescriptive design rule ("select experts by routing affinity, then fine-tune only those").

Research directions that become more attractive. The paper's demonstration that routing statistics provide a valid pre-training parameter selection signal opens several new lines of inquiry that were not obvious before:

MoE architecture design for downstream adaptability. Before this work, expert granularity was primarily a pretraining efficiency consideration. The finding that coarse-grained simulation degrades ESFT more than FFT (Figure 7) implies that architectural decisions made during pretraining have downstream adaptation consequences that were previously invisible. Model builders designing MoE architectures may now consider how fine-grained the experts should be to support selective fine-tuning as an explicit design criterion alongside perplexity and training cost.
Routing analysis as a standard pre-fine-tuning diagnostic. The probing methodology in Section 3.2 (forward passes on a small task sample, measuring routing concentration and cross-task divergence) is lightweight and can be applied to any MoE model before deciding on a fine-tuning strategy. This could become a standard step in the adaptation workflow, analogous to how practitioners currently run data distribution analyses before choosing preprocessing pipelines.
Hybrid PEFT methods that combine routing-based selection with low-rank adaptation. ESFT trains full expert weights; LoRA trains low-rank updates but applies them uniformly. A natural synthesis is to use routing statistics to select which experts receive LoRA adapters, or to allocate variable LoRA ranks based on routing concentration. The paper does not explore this, but the conceptual foundation is now in place.

Research directions that become less attractive. The paper's negative results also redirect effort away from dead ends:

Architecture-agnostic PEFT for MoE models without routing analysis. The substantial gap between LoRA (44.9 average specialized) and ESFT (49.4–50.2) in Table 1, combined with ESFT's dramatically better general-task retention (61.5 vs. 59.1 in Table 2), suggests that applying standard LoRA to MoE models without considering routing structure leaves significant performance and efficiency on the table. Future PEFT research targeting MoE architectures should engage with routing structure rather than treating MoE as just another transformer variant.
Coarse-grained MoE as a target for selective expert fine-tuning. Figure 7 provides clear evidence that ESFT's effectiveness depends on fine-grained specialization. Attempting to apply ESFT or similar routing-based methods to coarse-grained models like Mixtral or DBRX is likely to yield disappointing results. The paper establishes a boundary condition: ESFT is for fine-grained MoE, not all MoE.

Follow-Up Research This Work Enables

Characterizing the routing stability boundary: How much can expert weights change before routing decisions shift enough to invalidate ESFT's static selection? The paper assumes — and empirical results suggest — that routing distributions remain sufficiently stable during ESFT fine-tuning that the pre-training expert selection remains valid. But this stability is not measured directly. A diagnostic experiment would: (1) run ESFT on a task, (2) at periodic checkpoints during training, re-compute the Top-6 expert overlap for the same task tokens using the updated model, (3) correlate any routing shifts with the magnitude of weight changes in the selected experts, (4) measure whether tasks where routing shifts more correspond to tasks where ESFT underperforms FFT by larger margins (e.g., Translation in Table 1). If routing stability is the mechanism behind ESFT's success, then the method's applicability can be predicted by a single pre-training measurement: measuring the Jacobian of routing decisions with respect to expert weight perturbations. Tasks where this Jacobian is large (routing is sensitive to expert weight changes) would be flagged as poor candidates for static ESFT and might benefit from dynamic re-selection during training.

ESFT combined with LoRA within selected experts: Does routing-based selection plus low-rank adaptation outperform either alone? ESFT trains the full weight matrices of selected experts — typically 2–15 experts per layer, each containing two large linear transformations. A hybrid method would: (1) use ESFT's relevance scoring (Equations 6–7) to select the same set of experts, (2) instead of training their full weights, apply LoRA (rank $r$ , scaling $\alpha$ ) to only those selected experts, (3) freeze all other experts, shared experts, and non-expert parameters as in standard ESFT. This could reduce the parameter count from 1.4–1.85B (full-weight ESFT) to perhaps 50–200M (low-rank ESFT) while preserving the expert selection benefit. The experiment would sweep LoRA rank for the hybrid method against full-weight ESFT and standard LoRA on the same six tasks, measuring both specialized and general performance. The key question is whether ESFT's performance advantage comes from which experts are trained (the selection) or from how much those experts are modified (full-weight vs. low-rank). If the hybrid method matches ESFT with substantially fewer parameters, the selection mechanism is the dominant contributor. If it underperforms, full-weight updates to selected experts are necessary for the observed gains.

Cross-model validation: Does ESFT transfer to other fine-grained MoE architectures, and what architectural features are necessary? The paper's single-model limitation (DeepSeek-V2-Lite) is the most urgent gap to close. As additional fine-grained MoE models become available (e.g., larger DeepSeek-V2 variants, or fine-grained MoE models from other labs), a replication study should: (1) run the probing experiments from Section 3.2 on each model to measure routing concentration and cross-task divergence, (2) apply ESFT with the same $p=0.1$ (Gate) and $p=0.2$ (Token) thresholds on a shared set of tasks, (3) correlate ESFT's relative performance vs. FFT with the routing concentration metrics. The hypothesis: ESFT's effectiveness is proportional to the degree of expert specialization, which can be quantified pre-training by the Gini coefficient of the normalized gate distribution (Figure 2) and the cross-task expert overlap (Figure 3). Models with higher routing concentration and lower cross-task overlap should see larger ESFT gains. This would transform ESFT from a method validated on one model to a general principle with a measurable applicability criterion.

The shared expert training tradeoff characterized across more tasks and model scales. Section 6.3 (Table 3) reveals a consistent pattern — training shared experts degrades general performance (61.5 → 60.3 to 60.7 depending on configuration) while providing only modest specialized performance gains — but this is based on averaging across the six evaluation tasks. A more detailed analysis would: (1) measure the gradient norm of the loss with respect to shared vs. non-shared expert parameters during the first few steps of FFT on each task, (2) test whether tasks where shared expert gradients are large relative to non-shared expert gradients are also tasks where ESFT's default (shared-experts-frozen) configuration underperforms FFT by larger margins, (3) for those tasks, test the "prioritize specialized ability" configuration (Table 3, row "Relevant + Shared + Non-expert") and measure whether training shared experts closes the gap. The practical output would be a decision rule: if the ratio of shared-to-non-shared gradient norm exceeds some threshold, unfreeze shared experts; otherwise, freeze them. This moves the shared expert training decision from a fixed policy to an empirically calibrated one.

ESFT for continual learning and multi-task adaptation without catastrophic interference. Table 8 (Appendix H) shows that after fine-tuning on a specialized task (e.g., legal judgment), ESFT preserves Math and Code performance almost perfectly (<1 point drop), while FFT causes 9–12 point drops. This suggests a natural extension: sequential fine-tuning on multiple tasks, where each task trains only its own relevant experts. If expert sets for different tasks are largely disjoint (as Figure 3 suggests), fine-tuning on task A (training experts for A) followed by fine-tuning on task B (training experts for B) should produce a model that performs well on both tasks, with minimal interference. A concrete experiment: (1) sequentially fine-tune on all four adaptation tasks (Intent, Summary, Law, Translation) using ESFT, each task training only its own selected experts, (2) evaluate performance on all tasks after the full sequence, (3) compare to FFT and LoRA sequential fine-tuning, which should exhibit catastrophic forgetting of earlier tasks. This directly tests the paper's claim that ESFT "maintains expert specialization" and could establish ESFT as a method for building multi-task MoE systems through modular expert updates.

Difficulty-aware expert selection: Does the optimal $p$ threshold depend on task complexity or base model capability? The paper uses a fixed $p$ (0.1 for ESFT-Gate, 0.2 for ESFT-Token) across all six tasks, and the efficiency sweep in Section 6.2 (Figure 6) shows that specialized performance saturates at different $p$ values for different scoring methods. But the sweep is only performed on the Math task. A systematic study would: (1) run the full $p$ -sweep (0.05 to 1.0) for all six tasks, (2) measure the $p$ at which specialized performance reaches 95% of its maximum for each task, (3) correlate this "saturation $p$ " with task characteristics — base model performance on the task (from the Vanilla Model row in Table 1), routing concentration (from Figure 2), and whether the task is enhancement or adaptation. The hypothesis: tasks where the base model already performs well (Math, Code) saturate at lower $p$ because only a few experts need refinement; tasks where the base model performs poorly (Translation) need higher $p$ because more experts must be substantially retrained. This would replace the fixed $p$ with a task-conditioned heuristic, further improving ESFT's efficiency by avoiding over-selection on easy tasks.

Practical Applications and Downstream Use Cases

Multi-tenant model serving with task-specific expert adapters. An organization deploying a single large MoE model (e.g., DeepSeek-V2) to serve diverse customer needs — one tenant requires legal document analysis, another needs code generation, a third needs customer support summarization — currently faces a difficult choice: deploy one general model that performs adequately on all tasks, deploy separate full fine-tuned copies for each tenant (28.6 GB each, as per Figure 5, quickly becoming prohibitive), or use LoRA adapters that modify all experts uniformly (limiting task-specific optimization). ESFT offers a third path: for each tenant, run the expert selection phase on a small sample of their task data (32 sequences of length 4096), fine-tune only the identified experts (2.57–3.20 GB per tenant), and load the appropriate expert adapter at inference time while sharing the frozen base model across all tenants. Storage scales with the number of tenants × ~3 GB rather than × ~28.6 GB, a ~90% reduction. Critically, because each tenant's ESFT adapter modifies only the experts relevant to their specific task (with near-disjoint expert sets across tasks, Figure 3), loading one adapter does not interfere with other adapters — they can coexist in storage and be swapped in as needed. This is not merely a storage optimization; it enables a serving architecture where task specialization and multi-tenancy are compatible, which FFT's storage cost would make impractical.

Domain-specific model customization for regulated industries. In healthcare, legal, and financial applications, practitioners often need to adapt a general-purpose LLM to domain-specific terminology, reasoning patterns, and regulatory constraints. Full fine-tuning is unattractive because it (a) requires substantial compute (28.5 minutes per task on 16 A100 GPUs, per Figure 5), (b) degrades general capabilities that are still needed (e.g., a medical model must still understand general English), and (c) produces large checkpoints that must be stored, versioned, and audited for compliance. ESFT addresses all three: training time is reduced by ~30% (19.8–20.9 minutes), general ability is substantially preserved (61.5 vs. 58.8 for FFT on general benchmarks, Table 2), and the compact adapter size (2.57–3.20 GB) simplifies storage and version control — a hospital system could maintain ESFT adapters for radiology, cardiology, and general practice as separate small files, each auditable independently. The paper's results on specialized adaptation tasks (Intent: 78.6, Legal Judgment: 49.1 for ESFT-Gate, Table 1) demonstrate that substantial domain specialization is achievable even when the base model performs poorly on the raw task (Intent Vanilla: 16.8; Legal Vanilla: 17.1), making ESFT viable for the common scenario where a general model needs significant adaptation to a specialized domain.

Efficient iterative model improvement in research and product cycles. When developing a downstream application powered by an LLM, teams typically iterate: collect task-specific training data, fine-tune, evaluate, collect more data or refine the data mixture, fine-tune again, and repeat. In an FFT regime, each iteration costs 28.5 minutes and produces a 28.6 GB checkpoint, making rapid iteration expensive and slow. In an ESFT regime, the expert selection phase (32 forward passes) needs to be run only once — when the task data distribution is first characterized. Subsequent iterations, even with expanded or refined training data, can reuse the same expert selection (provided the task domain hasn't fundamentally changed), reducing the per-iteration cost to just the fine-tuning phase (~20 minutes) with ~3 GB storage per iteration. Moreover, because ESFT freezes shared parameters and non-relevant experts, changes between iterations are localized to a known set of experts, making it easier to debug performance regressions — a drop in general benchmark scores can be traced to specific expert updates rather than diffused across all 15.7B parameters. This is not a speculative benefit; the paper's data directly supports it: the expert selection stability analysis (Appendix C, Figure 8) shows that expert sets converge to high overlap with 32 samples, implying that once a task is characterized, the selection remains valid across data collection iterations.