Scaling Embeddings Outperforms Scaling Experts in Language Models

ArXiv: 2601.21204

🎯 Pitch

Slapping 30B+ parameters into a simple embedding table—not more experts—lets a model beat size-matched MoE baselines while activating just 3B parameters per token. Wider models benefit most, and the gains concentrate in agentic tool-use where LongCat-Flash-Lite crushes competitors (72.8 vs. 13–22 on Tau2-Telecom).

---

1. Executive Summary

This paper studies the comparative scaling efficiency of two orthogonal sparsity dimensions in large language models — expert scaling via Mixture-of-Experts versus embedding scaling via N-gram Embedding (a vocabulary-free lookup table that augments token representations with hashed n-gram context) — using from-scratch pretraining experiments on the LongCat-Flash architecture at activation scales of 280M, 790M, and 1.3B parameters. Through systematic analysis of architectural factors including integration timing, parameter budgeting, hash collision mitigation, and the interplay with model width and depth, the authors identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to increasing expert counts — a finding they validate by introducing LongCat-Flash-Lite, a 68.5B total-parameter MoE model with over 30B parameters allocated to N-gram Embeddings (~46% of total) that activates only ~3B parameters per token. LongCat-Flash-Lite not only surpasses a parameter-equivalent MoE baseline across general, reasoning, and coding benchmarks but also exhibits exceptional competitiveness against models like Qwen3-Next-80B-A3B-Instruct and Gemini 2.5 Flash-Lite, particularly in agentic tool-use tasks (achieving 72.8 on Tau2-Telecom versus 13.2–21.93 for baselines) and agentic coding (54.4 on SWE-Bench versus 32.8–41.3), establishing that embedding scaling offers a high-efficiency alternative to expert scaling primarily when models operate at sufficient sparsity levels with wider rather than deeper architectures.

2. Context and Motivation

The Core Problem: MoE Scaling Is Hitting Diminishing Returns and System Bottlenecks

The paper addresses a structural problem in the current scaling paradigm for large language models: Mixture-of-Experts (MoE) architectures, the dominant approach for building sparse models with massive parameter counts at manageable compute cost, are reaching practical limits. As the authors state in Section 1:

"as the model size and sparsity level increase, the marginal gain in performance diminishes, eventually approaching an efficiency saturation point"

This is not merely a theoretical concern. MoE models decouple total parameters from activated parameters by routing each token to only a subset of available expert FFN modules — a design that has enabled trillion-parameter models by keeping the per-token FLOPs roughly constant while expanding capacity. But this decoupling has two concrete failure modes that the paper identifies as motivating the search for alternatives:

Diminishing returns in performance. Prior work (Abnar et al., 2025, cited in Section 1) established empirically that the benefit of adding more experts follows a pattern of declining marginal gains. At low sparsity levels — where the ratio of total-to-activated parameters is small — each additional expert yields substantial loss improvements. But at high sparsity levels (which is precisely where modern frontier MoE models operate), the curve flattens. The paper reproduces this phenomenon in Figure 2: the standard MoE baseline follows a strict log-linear relationship, meaning that to achieve the same loss reduction at high sparsity that a small expert increase provides at low sparsity requires an exponentially larger increase in expert count. This is the same core dynamic that motivated scaling laws research for dense models — there is an optimal allocation point beyond which adding more of the same resource becomes inefficient — but the paper's contribution is to demonstrate that expert count is not the only parameter dimension worth scaling.

System-level bottlenecks from communication and memory bandwidth. Even if expert scaling continued to provide proportional performance gains, the authors identify that practical expansion is physically constrained. Each expert must be stored, loaded, and communicated during distributed training. As expert counts grow, the all-to-all communication patterns that route tokens to the correct experts across devices create escalating overhead. The paper frames this as a tension: MoE architectures are theoretically designed to decouple capacity from compute, but in practice, the very mechanism that enables this decoupling (token routing) introduces communication costs that eventually dominate the execution time.

The real-world implication is that continuing to scale LLMs by simply adding more experts is not sustainable — much like how dense model scaling hit a compute wall before MoE technology became widespread. The field needs a fundamentally different dimension along which to expand model capacity, one that does not impose the same routing and communication penalties.

Why This Problem Matters

The paper's motivation has both practical and conceptual significance that builds on the scaling dynamics of large language models.

Practically: the economics of model deployment. The current state of the art pushes toward models with 100B+ total parameters but only 3–5B activated per token (Qwen3-Next-80B-A3B, Gemini 2.5 Flash-Lite, and the LongCat-Flash architecture itself are all in this regime). Each percentage point of performance improvement from expert scaling demands proportionally more hardware to house the additional experts, even though they sit idle most of the time. If an alternative dimension — like embedding parameters — can provide equivalent or better performance at the same total parameter count while avoiding the communication overhead, it directly translates to lower inference cost, higher throughput, or better performance at a fixed budget. The paper's FLOPs-matched comparison throughout Section 3 — where N-gram Embedding models and expert-scaled models are compared at identical total parameter counts — is designed precisely to answer this economic question: for a given parameter budget, where should those parameters live?

Conceptually: establishing a new scaling dimension. The paper positions embedding scaling as an "overlooked, inherently sparse dimension" (Section 1) that has O(1) lookup complexity — meaning the cost of accessing an embedding vector does not grow with the total number of embedding vectors stored. This property is fundamentally different from experts, where routing decisions must be computed per token, and the experts themselves must be loaded from memory. The embedding layer is already present in every transformer (it maps discrete tokens to continuous vectors), but its capacity — the number of learned embeddings and their dimensionality — is typically treated as a fixed design choice (vocabulary size), not as a scaling lever. The paper argues that this is a missed opportunity: because embedding lookups are indexed by tokens and n-grams, they are naturally sparse (only the tokens present in the input trigger lookups) without requiring any dynamic routing mechanism. This makes them an ideal candidate for parameter expansion that avoids the communication bottlenecks plaguing MoE.

The training-inference disconnect. There is also a subtle architectural pressure that this paper addresses. During training, MoE models benefit from the parameter decoupling because each token's computation is independent of how many total experts exist — only the activated subset matters. But during inference, particularly in autoregressive decoding where tokens are generated one at a time, the overhead of loading expert parameters for each token becomes a dominant latency factor. The paper explicitly connects this to the memory I/O bottleneck in Section 4.1: "the increased size of the embedding layer does not penalize latency, as the computational cost of embedding lookups scales with the number of input tokens rather than the total number of embedding parameters." This asymmetry — that embedding parameters impose training cost but negligible inference cost — makes them strategically valuable for deployment-focused model design, which is the paper's primary use case.

Prior Approaches and Where They Fall Short

The paper identifies two broad categories of prior work that attempt to scale parameters beyond expert FFNs, each with specific limitations that motivate the current study.

Vocabulary scaling approaches. The theoretical foundation comes from Tao et al. (2024), who established scaling laws showing that larger models need proportionally larger vocabularies to maximize computational efficiency. The intuition is straightforward: if a model has more capacity (wider hidden dimensions, more layers), it can distinguish finer semantic differences between tokens, so it benefits from a larger set of distinct token embeddings. However, simply expanding the base vocabulary has diminishing returns because most tokens in the expanded vocabulary will be rare or unseen during training, leading to poorly learned embeddings. This is where n-gram based approaches enter: rather than expanding the vocabulary of individual tokens, they augment each token's representation with embeddings derived from its local context (the preceding n-1 tokens), effectively encoding compositional information that a single-token embedding cannot capture.

The lineage traces back to lookup-table language models in the RNN era (Huang et al., 2021) and was recently revived in transformer architectures through CANINE (Clark et al., 2022), the Over-Tokenized Transformer (Huang et al., 2025), Byte Latent Transformer (Pagnoni et al., 2025), and Engram (Cheng et al., 2026, concurrent work). These approaches share the core idea of using hashed n-gram lookups to densify the information per token, but none of them systematically compared embedding scaling against expert scaling as competing allocation strategies under a fixed parameter budget. The paper cites Engram specifically as concurrent work that identified the U-shaped scaling curve (too much embedding proportion hurts), confirming this result but emphasizing that the broader comparative framework — especially the interaction with model width and depth — had not been established.

Per-layer embedding approaches. A second family of approaches, exemplified by Gemma 3n (Google DeepMind, 2025) and STEM (Sadhukhan et al., 2026), allocates independent embedding parameters to each transformer layer rather than sharing a single embedding table. The motivation is that different layers process different levels of abstraction (early layers handle syntax, late layers handle semantics), so allowing each layer to look up its own token representations could improve representational capacity. The paper acknowledges this direction and even proposes an extension (Per-Layer N-gram Embedding in Section 5), but notes a critical limitation: PLE underperforms N-gram Embedding at equivalent parameter counts (Figure 9), which the authors attribute to "the superior learning efficiency of N-gram Embedding compared to standard embeddings." Additionally, PLE adds per-layer projection matrices that increase activated parameters, partially negating the sparsity advantage.

The key gap: no comparative scaling framework. Despite these two active research directions, the paper identifies that no prior work had systematically compared embedding scaling against expert scaling under controlled conditions. The specific questions left unanswered include:

1. At what sparsity level does it become more efficient to add embedding parameters rather than additional experts?
2. How much of a model's total parameter budget should be allocated to embeddings versus experts — is there an optimal ratio?
3. How do model width and depth modulate the effectiveness of embedding scaling (do wider models benefit more? deeper models less?)?
4. What are the practical engineering challenges of making embedding-scaled models fast at inference time, given that they shift the compute profile away from expert I/O toward embedding I/O?

The paper's framing is explicit about this gap in Section 1: "the comparative scaling efficiency between expert parameters and embedding parameters is not well understood, leaving the optimal allocation of capacity between these two sparse dimensions ambiguous." This is the central question the paper sets out to answer.

How This Paper Positions Itself

The paper does not propose fundamentally new mechanisms — the N-gram Embedding technique itself is adopted from prior work (Clark et al., 2022; Huang et al., 2025). Instead, its contribution is a systematic empirical characterization of when and how embedding scaling is preferable to expert scaling, backed by a complete deployed model that validates the findings.

The methodological framing. The paper's approach is explicitly comparative and architectural: they fix the total parameter count and vary the allocation between N-gram Embedding parameters and MoE expert parameters, measuring training loss and downstream benchmarks. This zero-sum framing — "for a given parameter budget, should this parameter live in an expert or in an embedding table?" — is what distinguishes the work from prior studies that simply showed embedding scaling improves over a dense baseline. The paper is asking a resource allocation question, not a "does embedding help?" question.

Integration with inference optimization. A distinctive feature of this work is that it treats inference efficiency not as an afterthought but as a co-equal design constraint. Section 4 develops the full inference pipeline: the N-gram Cache (analogous to the KV cache) for efficient embedding lookup, kernel fusion and speculative decoding to maximize GPU utilization, and analysis of how embedding scaling changes the I/O profile. This systems-level treatment is motivated by the paper's practical orientation — the authors are building a deployable model (LongCat-Flash-Lite), so the theoretical Pareto advantages must translate to wall-clock speed improvements. The specific finding that speculative decoding synergizes with embedding-scaled models (because the large effective batch size needed to saturate memory bandwidth is naturally provided by multi-step draft-then-verify decoding) is an example of how the paper connects architectural design to systems implementation.

Scope and limitations admitted. The paper is transparent about its scope: all experiments use the LongCat-Flash architecture with Chinese-English pre-training data, the primary comparative metric is training/validation loss on a 300B-token corpus (with downstream evaluations on the flagship 68.5B model only), and the difficulty estimation for "when to add N-gram Embedding" is based on observing the intersection points in the scaling curves. The authors do not claim that embedding scaling universally dominates expert scaling — they precisely characterize the regimes (high base sparsity, wide models, moderate depth, appropriate parameter ratios) where it does, and show where it does not (low sparsity, very deep models, excessive embedding proportion). This boundedness actually strengthens the contribution: the findings are actionable because the boundary conditions are specified.

Relationship to scaling laws research. The paper implicitly positions itself in the tradition of scaling laws work (Hoffmann et al., 2022; Abnar et al., 2025) but with a twist. Traditional scaling laws optimize over continuous variables (parameters vs. tokens for pretraining, or parameters vs. FLOPs for sparsity). This paper's contribution is to introduce a discrete architectural choice — embeddings vs. experts — as a scaling dimension and to empirically characterize its interaction with the continuous dimensions of width, depth, and sparsity. The log-linear relationship revealed in Figure 2 (MoE scaling follows a strict log-linear pattern) and the systematic shifts in the intersection point between embedding and expert curves as width increases (Figure 6) are the kinds of empirical regularities that scaling laws research seeks. The paper does not propose a parametric law, but it provides the data that could underpin one.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The paper builds a comparative scaling framework — a systematic experimental methodology for determining, at a given activated parameter budget and total parameter budget, whether it is more efficient to allocate additional parameters to MoE experts (the standard approach) or to a vocabulary-free N-gram Embedding lookup table (the proposed alternative). The problem it solves is the ambiguity in resource allocation when scaling sparse models: given that both expert FFNs and embedding tables can absorb parameters without increasing per-token FLOPs proportionally, which dimension gives better performance per parameter? The "shape" of the solution is an empirical phase portrait — a characterization of the architectural conditions (base sparsity level, model width, model depth, embedding parameter proportion) under which embedding scaling lies on a strictly superior Pareto frontier compared to expert scaling.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged in a pipeline from architectural design through training to inference:

1. Base MoE Architecture (LongCat-Flash) — a pre-existing Mixture-of-Experts transformer with configurable width (hidden size), depth (number of shortcut layers), and expert count/sparsity. This is the "control" model that receives either additional experts (baseline) or N-gram Embedding parameters (treatment).

2. N-gram Embedding Module — a vocabulary-free embedding augmentation layer that sits alongside the standard token embedding table. For each input token, it hashes the preceding n-1 tokens through multiple independent hash functions, looks up sub-embeddings in separate tables, projects them back to the model dimension, and averages the result with the base token embedding. This module is the mechanism for scaling embedding parameters.

3. Parameter Allocation Interface — the experimental design rule that fixes total parameter count and varies where they live: either in additional expert FFN modules (the MoE baseline) or in the N-gram Embedding tables (the treatment). This is not a code component but the core experimental control that makes the comparison valid.

4. Training Pipeline — from-scratch pretraining on 300B tokens (for scaling experiments) or 11T tokens (for LongCat-Flash-Lite) using identical data, optimizer settings, and training schedule for both MoE baselines and N-gram Embedding variants. Training loss and validation loss on held-out Chinese and English datasets are the primary comparative metrics.

5. Inference System — a specialized deployment stack for the embedding-scaled model, incorporating an N-gram Cache (analogous to a KV cache for embedding lookups), fused CUDA kernels, wide expert parallelism, speculative decoding with Eagle3, and programmatic dependent launch to convert the theoretical parameter sparsity into wall-clock speedups.

Information flows as follows: the architecture is configured with a specific total-to-activated parameter ratio → N-gram Embedding is optionally integrated at a specific sparsity level → the model is pretrained on the full corpus → training/validation loss is compared against the parameter-equivalent MoE baseline → for the flagship model (LongCat-Flash-Lite), mid-training, SFT, and downstream benchmark evaluation are applied → the trained model is deployed with N-gram Cache and speculative decoding for efficient inference.

3.3 Roadmap for the Deep Dive

• First, the N-gram Embedding mechanism itself (Section 2 of the paper) — the mathematical formulation, the hashing scheme, the sub-table decomposition, and the linear projection design. This is the core building block that all scaling experiments depend on, so understanding its mechanics is prerequisite.

• Second, the comparative scaling methodology (Section 3) — how the experiments are structured to isolate the effect of embedding scaling versus expert scaling, including the parameter-equivalent baseline construction, the sparsity ratio as the independent variable, and the training/evaluation protocol.

• Third, the timing analysis (Section 3.1) — when in the sparsity regime to introduce N-gram Embedding, and the empirical finding that embedding scaling only outperforms expert scaling past a critical sparsity threshold.

• Fourth, the integration strategy (Section 3.2) — the parameter budgeting constraint (how much of the total parameter count can be allocated to embeddings before performance degrades), the hash collision analysis and vocabulary sizing principle, the hyperparameter sensitivity study for n-gram order and sub-table count, and the embedding amplification technique required for effective training.

• Fifth, the width and depth scaling properties (Section 3.3) — how the advantage of N-gram Embedding grows with wider models but shrinks with deeper models, and the mechanistic explanation involving residual stream signal propagation.

• Sixth, the inference system design (Section 4) — the N-gram Cache, the role of speculative decoding in converting sparsity to speed, kernel fusion optimizations, and the speculative directions for using N-gram Embedding as a draft model or early-rejection mechanism.

• Seventh, the Per-Layer N-gram Embedding extension (Section 5) — how N-gram Embedding can be integrated per-layer rather than only at the input, and the empirical comparison showing marginal improvements that did not justify adoption at scale.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that embedding parameters and expert parameters are competing allocation targets within a fixed total parameter budget, and that embedding scaling achieves superior efficiency specifically when the base model operates at high sparsity, the architecture is wide rather than deep, and the embedding proportion does not exceed roughly 50% of total parameters.

---

The N-gram Embedding Mechanism

The N-gram Embedding module augments the standard token embedding by adding context-dependent information derived from the hashed representations of preceding tokens. Rather than expanding the fixed vocabulary of individual tokens, it creates a separate, larger embedding table indexed by n-gram hashes, allowing the model to store and retrieve compositional token-sequence information without increasing the base vocabulary size.

The core equation (final form with sub-tables and projection):

$e_i = \frac{1}{(N-1)K + 1}\left(E_0(t_i) + \sum_{n=2}^{N}\sum_{k=1}^{K} W_{n,k} E_{n,k}(\mathcal{H}_{n,k}(t_{i-n+1}, \ldots, t_i))\right)$

where $e_i \in \mathbb{R}^D$ is the augmented embedding for the $i$-th token, $E_0 \in \mathbb{R}^{V_0 \times D}$ is the standard base embedding table with vocabulary size $V_0$ and hidden dimension $D$, $E_{n,k} \in \mathbb{R}^{V_{n,k} \times D/((N-1)K)}$ is the $k$-th sub-table for n-gram order $n$ with vocabulary size $V_{n,k}$, $W_{n,k} \in \mathbb{R}^{D \times D/((N-1)K)}$ is the linear projection matrix for sub-table $(n,k)$, $N$ is the maximum n-gram order, $K$ is the number of sub-tables per n-gram order, and $\mathcal{H}_{n,k}$ is the hash function for that specific sub-table.

What it computes: for a given token $t_i$ at position $i$, the equation produces a single $D$-dimensional vector by averaging the base token embedding with contributions from all n-gram contexts (bigrams through $N$-grams) that end at position $i$. Each n-gram contribution is itself produced by: (1) hashing the sequence of $n$ tokens into an index, (2) looking up a low-dimensional sub-embedding from sub-table $E_{n,k}$, (3) projecting that sub-embedding back to the full dimension $D$ via $W_{n,k}$, and (4) summing over all $K$ sub-tables for that n-gram order. The final denominator $(N-1)K + 1$ normalizes the average so that the output has the same expected scale as the base embedding alone. The result $e_i$ replaces the standard token embedding in the transformer's input layer.

Why this form: the decomposition into $K$ sub-tables per n-gram order is a direct response to the hash collision problem. With a single hash table, two different n-gram sequences that hash to the same index would be forced to share the same embedding vector, creating representational ambiguity. By using $K$ independent hash functions (each with its own sub-table and vocabulary size), the probability that two sequences collide under all $K$ functions simultaneously is exponentially reduced — a technique borrowed from Bloom filter design. The dimensionality reduction (each sub-table has dimension $D/((N-1)K)$ rather than $D$) ensures that the total parameter count of the N-gram Embedding module remains invariant with respect to $N$ and $K$: adding more sub-tables or higher n-gram orders does not increase the total number of embedding parameters, only how they are organized. The linear projection matrices $W_{n,k}$ are necessary because the sub-embeddings live in a different space than the model's hidden dimension — without them, the dimensionality mismatch would prevent summation with the base embedding. The averaging normalization $1/((N-1)K + 1)$ ensures that the output vector does not grow in magnitude as $N$ or $K$ increase, which would destabilize training by injecting progressively larger signals into the residual stream.

The hash function (polynomial rolling hash):

$\mathcal{H}_n(t_{i-n+1}, \ldots, t_i) = \left(\sum_{j=0}^{n-1} t_{i-j} \cdot V_0^j\right) \bmod V_n$

where $t_{i-j}$ is the token ID at position $i-j$, $V_0$ is the base vocabulary size, and $V_n$ is the n-gram vocabulary size for this hash table.

What it computes: a deterministic integer index into the n-gram embedding table, computed as a polynomial in the token IDs with base $V_0$, taken modulo the table size $V_n$. The polynomial form $t_i + t_{i-1} \cdot V_0 + t_{i-2} \cdot V_0^2 + \ldots$ ensures that different orderings of the same tokens produce different hash values (since the positional weight $V_0^j$ depends on $j$). For example, the bigram $(A, B)$ and the bigram $(B, A)$ map to different indices because $A + B \cdot V_0 \neq B + A \cdot V_0$ when $A \neq B$. The modulo operation bounds the output to the range $[0, V_n-1]$, ensuring the index is a valid lookup into the table of size $V_n$.

Why this form: polynomial rolling hash is standard in string processing because it is collision-resistant for sequences where token order matters, computationally cheap (it can be computed incrementally as tokens arrive: $\mathcal{H}_n(t_{i-n+1}, \ldots, t_i) = (\mathcal{H}_{n-1}(t_{i-n+2}, \ldots, t_i) \cdot V_0 + t_i) \bmod V_n$), and produces a uniform-enough distribution over the table when $V_n$ is chosen appropriately. The authors note a subtle implementation detail in a footnote: for large $N$, the exponentiation $V_0^{n-1}$ can cause numerical overflow, which is avoided by applying the modulus operation prior to exponentiation in the incremental computation. The choice of $V_0$ as the polynomial base (rather than an arbitrary prime) ties the hash space to the token space: the hash output depends on the actual token IDs, meaning that semantically similar tokens (which have unrelated IDs) produce unrelated hashes — this is intentional, as the N-gram Embedding learns to associate hash indices with their training distribution, and any structure in the token ID space is irrelevant to the learned embeddings.

Edge case for boundary tokens: the paper specifies that $t_j = 0$ if $j \leq 0$, meaning that tokens before the start of the sequence are treated as a special padding token with ID 0. This means that the first token in a sequence ($i=1$) has its n-gram context padded with zeros for all $n \geq 2$, so $e_1 = E_0(t_1)$ (all n-gram contributions reduce to hashing sequences ending in zeros). This is a standard left-padding approach that avoids degenerate behavior at sequence boundaries.

---

Comparative Scaling Methodology (Section 3 Setup)

The experimental framework is designed to answer a counterfactual question: if you have a fixed total parameter budget, should you spend incremental parameters on more experts or on N-gram Embedding tables? To isolate this effect, the authors construct a controlled comparison where the only difference between the baseline and the treatment is where the parameters live.

Base architecture. All experiments use the LongCat-Flash architecture (Meituan, 2025), a pre-existing MoE transformer design. The architecture is configured at three activation scales: 280M, 790M, and 1.3B activated parameters. At each scale, a family of models is created by varying the base sparsity level — the ratio of total parameters to activated parameters — by adjusting the number of experts. The sparsity range spans from approximately 35% to 98%, where higher ratios mean more total parameters per activated parameter (higher sparsity). The x-axis of the scaling plots uses this ratio (total/activated) as a proxy for sparsity level.

Parameter-equivalent baseline construction. For each N-gram Embedding model, a corresponding MoE baseline is constructed with exactly the same total parameter count, achieved by converting the N-gram Embedding parameters into additional expert FFN modules. This is the crucial experimental control: both models have the same number of total parameters, the same activated parameter count, and the same training data, but differ in the architectural allocation of those parameters. If the N-gram Embedding model achieves lower loss, it means that embedding parameters are more "efficient" than expert parameters at that sparsity level and allocation ratio — the parameters are doing more useful work per bit.

Training protocol. All models in the comparative scaling experiments are pre-trained from scratch on a corpus of 300B tokens. The paper does not specify exact hyperparameters for these scaling runs (optimizer, learning rate, batch size, sequence length), but states they follow the LongCat-Flash recipe. The primary evaluation metric is training loss, with validation loss reported on two separate held-out datasets covering Chinese and English. This dual-language evaluation is important because the N-gram Embedding mechanism is language-agnostic (it operates on token IDs, not linguistic features), so improvements should manifest in both languages.

The ratio as the independent variable. The key independent variable in the scaling plots (Figures 2, 6, 7) is the total-to-activated parameter ratio on the x-axis. This is a proxy for sparsity: a ratio of 10 means 10 total parameters per 1 activated parameter, a ratio of 50 means 50:1, etc. Higher ratios correspond to sparser models with more experts. The dependent variable is training or validation loss. The MoE baseline curve shows how loss changes as experts are added (ratio increases). The N-gram Embedding curves branch off from the MoE curve at specific sparsity levels and show what happens when, starting from that base sparsity, additional parameters are allocated to embeddings rather than experts. The "intersection point" between the N-gram Embedding curve and the MoE curve (where they cross) indicates the sparsity level beyond which embedding scaling becomes less efficient than continued expert scaling — the optimal stopping point for embedding allocation.

Why this design: the approach of branching from the MoE curve at different sparsity levels is essential because it answers the "when to switch" question. If N-gram Embedding were simply applied to a dense model, the comparison would conflate the benefit of sparsity (which MoE provides) with the benefit of embedding scaling. By starting from an already-sparse MoE model and then adding embedding parameters, the experiment isolates the marginal benefit of embedding parameters over expert parameters at each sparsity level. The parameter-equivalent control ensures that any loss difference is attributable to the architectural allocation, not to having more total parameters.

---

Optimal Timing for N-gram Embedding Integration (Section 3.1)

The paper's first major empirical finding is that the effectiveness of N-gram Embedding depends critically on the base sparsity level at which it is introduced. Figure 2 illustrates this with three scaling trajectories at the 280M activation scale:

• Blue curve (MoE baseline): the standard MoE model where all additional parameters go to experts. This curve follows a strict log-linear relationship: $\log(\text{loss}) \propto -\alpha \cdot \log(\text{ratio})$ for some constant $\alpha$. In low-ratio regimes (ratio < ~12), each unit increase in the ratio yields a substantial loss reduction. At higher ratios, the curve flattens — achieving the same absolute loss reduction requires an exponentially larger increase in the ratio, meaning exponentially more experts.

• Green curve (N-gram Embedding at low ratio): N-gram Embedding is introduced at a low base sparsity (ratio ~8). The dotted line connects the base MoE model to the N-gram Embedding variant. The green curve lies consistently above the blue curve — meaning the N-gram Embedding model has higher loss than the parameter-equivalent MoE baseline. At low sparsity, the marginal benefit of additional experts still exceeds the marginal benefit of embedding parameters, so diverting parameters to embeddings is strictly worse.

• Red curve (N-gram Embedding at high ratio): N-gram Embedding is introduced at a high base sparsity (ratio ~12). The red curve initially lies below the blue curve for a range of ratios — meaning the N-gram Embedding model achieves lower loss than the parameter-equivalent MoE. This advantage persists until the ratio reaches approximately 20, at which point the red and blue curves intersect, and further embedding allocation becomes inferior to expert allocation.

The mechanistic interpretation. At low sparsity, each expert is still relatively "concentrated" — there are few enough experts that each one sees a substantial fraction of the training tokens and can specialize effectively. Adding more experts at this stage provides genuine representational capacity gains because the routing mechanism can assign tokens to increasingly specialized FFN modules. At high sparsity, the marginal expert is training on an ever-smaller fraction of the data (since tokens are distributed across more experts), leading to undertrained expert parameters that provide diminishing returns. This is precisely where embedding parameters become more efficient: they are trained on every occurrence of their associated n-gram, regardless of routing decisions, so they do not suffer from the data fragmentation that plagues sparse experts.

The design principle (Principle 1): the paper articulates this as a prescriptive rule:

"This result indicates that embedding scaling could be a promising scaling dimension orthogonal to expert scaling."

The operational guidance is that N-gram Embedding should only be integrated when the base model is already operating at a sufficiently high sparsity level — roughly, when the marginal gain from additional experts has significantly diminished. The exact threshold depends on the activation scale (it shifts rightward for wider models, as Section 3.3 will show), but the qualitative pattern is universal.

---

Integration Strategy: Parameter Budgeting (Section 3.2.1)

Even when N-gram Embedding is introduced at the right sparsity level, there is a limit to how much of the total parameter budget can be allocated to embeddings before performance degrades. Figure 2 shows that the red curve (N-gram Embedding at high ratio) eventually crosses above the blue curve (MoE baseline) — the intersection point.

At the 280M activation scale, this intersection occurs slightly above a total-to-activated ratio of 20. At this point, the base MoE model (without N-gram Embedding) has a ratio of 12, meaning the N-gram Embedding parameters account for the difference: $(20 - 12) / 20 \approx 40\%$ of total parameters, or roughly 50% of the total parameter count when expressed as a fraction of the non-base parameters. The paper's second design principle formalizes this:

"when a model allocates an excessive proportion of its parameter budget to N-gram Embedding, its performance becomes inferior to that of parameter-equivalent MoE baselines"

This aligns with concurrent work by Cheng et al. (2026, Engram), which identified a U-shaped scaling curve: performance initially improves as embedding proportion increases, reaches an optimum, then degrades as embedding proportion becomes excessive.

Why too much embedding is harmful. The paper does not provide a detailed mechanistic explanation, but the phenomenon is consistent with a representational bottleneck: the N-gram Embedding operates only at the input layer (in the basic configuration), affecting the initial token representation but not the intermediate computations performed by the transformer layers. If too many parameters are concentrated in the embedding layer, the transformer body (attention and FFN modules) is starved of capacity — the model can represent rich input features but lacks the processing depth to use them effectively. Expert parameters, by contrast, are distributed throughout the network depth, contributing to computation at every layer. The optimal allocation balances input representation capacity (embeddings) against processing capacity (experts).

---

Integration Strategy: Hash Collision Mitigation (Section 3.2.2)

Hash collisions — where two different n-gram sequences map to the same table index — force a single embedding vector to represent multiple distinct contexts, degrading the quality of the learned representations. The paper identifies vocabulary size selection as a critical lever for controlling collision rates, with a non-obvious interaction effect.

Vocabulary hit rate analysis. Figure 3(a) shows the "vocabulary hit rate" — the fraction of n-gram vocabulary entries that are activated at least once by the pre-training corpus — as a function of n-gram order, using an n-gram vocabulary size of $30 \times$ the base vocabulary (128k, so $30 \times 128\text{k} \approx 3.84\text{M}$ entries). The 2-gram hit rate increases gradually but never reaches 1.0, meaning many 2-gram entries are never used. Higher-order n-grams (3-gram, 4-gram, etc.) rapidly converge toward a hit rate of 1.0, indicating that the hash space is well-utilized — almost every entry is activated at least once. This makes intuitive sense: there are far more possible 4-grams than 2-grams, so with a fixed table size, the occupancy rate increases with n-gram order.

Hash collision analysis. Figure 3(b) examines 2-gram hash collisions specifically, sampling vocabulary sizes between $30\times$ and $33\times$ the base vocabulary and computing collision counts over 100 training sequences. The key finding is a non-linear, spiky relationship: collision counts spike noticeably when the n-gram vocabulary size approaches an integer multiple of the base vocabulary size. This phenomenon persists regardless of whether the n-gram vocabulary size is a prime number — it is an artifact of the polynomial rolling hash with base $V_0$.

The mathematical reason: the hash function is $\mathcal{H} = (t_i + t_{i-1} \cdot V_0) \bmod V_n$. When $V_n$ is an exact multiple of $V_0$ (e.g., $V_n = 31 \cdot V_0$), the modulo operation creates systematic collisions because tokens that differ by multiples of $V_n / V_0$ in their hash computation will map to the same index. Specifically, if $t_{i-1}$ increases by $k \cdot (V_n / V_0)$, the term $t_{i-1} \cdot V_0$ changes by $k \cdot V_n$, which is 0 modulo $V_n$, creating a collision for all $t_i$. This creates structured collision patterns that concentrate collisions on specific (token, context) pairs.

Design principle (Principle 3): to minimize collisions, the n-gram vocabulary size should avoid being near integer multiples of the base vocabulary size. The paper does not provide a specific optimal offset, but the operational guidance is to measure collision counts for candidate vocabulary sizes on a sample of training data and select a size in a "valley" of the collision curve rather than at a spike.

---

Integration Strategy: Hyperparameter Sensitivity (Section 3.2.3)

The N-gram Embedding module has two primary hyperparameters: the maximum n-gram order $N$ (how far back in the context the embedding looks) and the number of sub-tables $K$ (how many independent hash functions are used per n-gram order). The paper conducts an ablation study using the 790M activated-parameter model (the initial point on the red curve in Figure 6(a)).

Figure 4 results. The ablation sweeps combinations of $N \in \{2, 3, 4, 5\}$ and $K \in \{1, 2, 4, 8\}$, measuring both training loss and two validation losses. The key finding is that performance is robust within a broad regime: for $N \geq 3$ and $K \geq 2$, the performance variance across configurations is relatively small. However, the minimal configuration ($N=2, K=1$) is notably inferior — the model exhibits significantly higher loss.

Why $N=2, K=1$ underperforms. With $N=2$, the N-gram Embedding only captures bigram context, which provides limited compositional information beyond what the base token embedding already encodes (since transformers can learn bigram-level statistics through attention). With $K=1$, there is no hash collision mitigation — a single collision forces two different bigrams to share an embedding, and the model has no way to disambiguate them. The combination of shallow context and high collision rates means the N-gram Embedding adds little useful information while consuming parameters that could have been experts.

Why higher $N$ and $K$ saturate. For $N \geq 3$, the n-gram context captures enough token history to encode meaningful multi-token patterns. For $K \geq 2$, the collision probability becomes low enough that the residual collisions do not significantly degrade learning. Further increases in $N$ create extremely sparse n-gram distributions (most 5-grams appear very rarely), making the embeddings hard to learn. Further increases in $K$ provide diminishing collision reduction while fragmenting the embedding budget across more sub-tables. The paper's empirical recommendation is $N \in [3, 5]$ and $K \geq 2$, with the note that performance is insensitive to the exact choice within this range.

---

Integration Strategy: Embedding Amplification (Section 3.2.4)

A critical training dynamics problem emerges from the interaction between the embedding output and the residual stream: if the embedding signal is too weak relative to the attention and FFN outputs, the N-gram Embedding parameters do not receive meaningful gradient signals and fail to learn.

The signal suppression problem. Figure 5 analyzes an early vanilla experiment (no embedding amplification) by plotting, for each transformer layer, the L2 norm of each module's output, the L2 norm of its corresponding identity (residual) branch, and the ratio between them. The identity branch at the first layer is essentially the embedding output (since no previous layers have modified the residual stream). The critical observation: the L2 norm of the first attention module's output is approximately 10× larger than the L2 norm of the identity branch (the embedding output). When these are summed in the residual connection, the attention output dominates — the embedding signal contributes only about 9% of the total magnitude. This means the embedding parameters receive weak gradients during backpropagation because the downstream layers are primarily driven by the attention output, not the embedding output. The embedding parameters become "dead weight" — they occupy capacity but do not influence the model's behavior.

Why this happens. Standard initialization of the N-gram Embedding sub-tables and projection matrices ensures that the initial output norms match the baseline embedding. However, during training, the attention and FFN modules grow in magnitude (a well-known phenomenon in transformers), while the embedding module does not automatically scale up to match. The residual stream's normalization (typically RMSNorm or LayerNorm applied before each sub-layer) does not compensate because it normalizes the input to the sub-layer, not the relative contributions of the sub-layer output and the identity branch in the summation.

Two mitigation strategies (Embedding Amplification):

1. Scaling factor: multiply the embedding output by a constant factor (typically $\sqrt{D}$, where $D$ is the hidden dimension) before adding it to the residual stream. This directly increases the embedding contribution to match the scale of the attention/FFN outputs. The factor $\sqrt{D}$ is chosen because it is the expected scale of a random Gaussian vector of dimension $D$, matching the initialization scale of the other modules.

2. Normalization: apply LayerNorm (or RMSNorm) to the embedding output before merging. LayerNorm enforces unit variance during early training, effectively amplifying the embedding signal because the attention/FFN outputs typically have variance greater than 1 after training progresses. This is a softer version of the scaling factor approach — it adapts the amplification dynamically rather than using a fixed multiplier.

Both techniques were originally proposed by Takase et al. (2025) for a different purpose: increasing residual branch variance to bound backward gradients and stabilize training (preventing gradient spikes). The paper notes that in their experiments, Embedding Amplification did not significantly affect training stability, but it substantially enhanced N-gram Embedding performance — yielding a consistent reduction of 0.02 in both training loss and the two validation losses. This 0.02 reduction might seem small, but in the context of language model pretraining where total loss improvements from architectural changes are typically in the 0.01–0.05 range, it represents a meaningful fraction of the available gain.

Why this matters for the broader findings. Without embedding amplification, the N-gram Embedding models in the comparative scaling experiments (Section 3.1–3.3) would underperform, potentially changing the conclusions about when embedding scaling is beneficial. The amplification technique is not an optional optimization — it is a necessary condition for the N-gram Embedding to contribute meaningfully to the model's computation.

---

Scaling Properties: Width (Section 3.3.1)

The paper investigates how the advantage of N-gram Embedding over expert scaling changes as model width increases, holding depth constant (10 shortcut layers). Figure 6 shows scaling curves at two larger activation scales: 790M (Figure 6a) and 1.3B (Figure 6b). In both cases, the experimental design mirrors Figure 2: the blue curve is the MoE baseline, the red curve is N-gram Embedding introduced at a high base sparsity, and the green curve (where present) is N-gram Embedding at a lower base sparsity.

Key trend 1: N-gram Embedding advantage persists and grows. At both 790M and 1.3B activation scales, the red curve (N-gram Embedding at appropriate sparsity) consistently lies below the blue curve (MoE baseline) for a range of ratios — the N-gram Embedding model achieves lower loss at the same total parameter count. The advantage is not merely preserved from the 280M scale; it appears to grow.

Key trend 2: The intersection point shifts rightward. The ratio at which the N-gram Embedding curve crosses above the MoE curve systematically increases with model width:

• At 280M activation: intersection at ratio ~20 (N-gram Embedding loses advantage when embedding parameters exceed ~50% of total).
• At 790M activation: N-gram Embedding only underperforms on the English validation set at the highest tested ratio; on all other metrics (training loss, Chinese validation), it maintains an advantage even at the maximum ratio tested.
• At 1.3B activation: N-gram Embedding retains a clear advantage even at ratios as high as 50 (which, given a base ratio of approximately 12, means embedding parameters constitute roughly $(50-12)/50 \approx 76\%$ of total parameters and still outperform the expert-scaled baseline).

Why wider models benefit more from N-gram Embedding. The paper's explanation connects to the representational capacity of the transformer body. In a wider model, the hidden dimension $D$ is larger, meaning each token's representation is a higher-dimensional vector. The N-gram Embedding operates in this higher-dimensional space — its output $e_i \in \mathbb{R}^D$ carries more information per token when $D$ is larger because the embedding tables store $D$-dimensional vectors and the projection matrices map to $D$ dimensions. A wider model can therefore "absorb" more information from the N-gram Embedding without hitting a representational bottleneck. Additionally, wider models have proportionally more capacity in their attention and FFN modules to process the enriched token representations, so the downstream computation is not the limiting factor.

Design principle (Principle 4, restated from Section 3.3.1):

"for a fixed number of layers, wider models allow for a significantly expanded window of opportunity to leverage N-gram Embedding effectively"

In operational terms, if you are designing a model and want to scale via embeddings rather than experts, make the model as wide as practical before allocating parameters to the embedding layer. The wider the model, the higher the optimal embedding proportion.

---

Scaling Properties: Depth (Section 3.3.2)

The counterpoint to width scaling is depth scaling. Using the 1.3B activated parameter configuration, the paper trains models with 10, 20, and 40 shortcut layers (equivalent to 20, 40, and 80 conventional transformer layers) while maintaining a consistent 50% N-gram Embedding parameter proportion across all depths.

Figure 7(b) result. The performance gap between N-gram Embedding and the MoE baseline is plotted against model depth. For the 10-layer model (from Figure 6b), the gap is substantial — N-gram Embedding provides a clear loss reduction. As depth increases to 20 layers, the gap contracts noticeably. At 40 layers, the gap has shrunk further, approaching zero — the N-gram Embedding model performs nearly equivalently to the MoE baseline, with the embedding parameters providing minimal marginal benefit.

Mechanistic explanation: residual signal attenuation. The paper attributes this to the pre-normalization architecture's residual stream dynamics. In a pre-norm transformer, the input to each sub-layer (attention or FFN) is first normalized, then the sub-layer computes its output, and finally the output is added to the residual stream (identity branch). The N-gram Embedding output enters the residual stream at layer 0 and propagates forward through the identity connections. However, at each subsequent layer, new attention and FFN outputs are added to the residual stream, progressively "burying" the original embedding signal under later computations. By layer 20 or 40, the embedding contribution has been diluted — the residual stream at layer $L$ is dominated by the cumulative outputs of layers $1$ through $L$, not by the initial embedding.

Figure 5 provides direct evidence for this: even in a 10-layer model, the identity branch norm (which carries the embedding signal) is already an order of magnitude smaller than the module outputs at layer 1, and this ratio generally worsens (the identity branch becomes relatively smaller) in deeper layers. In a 40-layer model, the embedding signal would be diluted across 40 rounds of residual additions, rendering its contribution negligible in the later layers. This means the N-gram Embedding parameters are effectively only influencing the first few layers, wasting the capacity allocated to them.

Implication for architecture design. The paper notes that "the majority of current practical language models typically operate below 40 shortcut layers (equivalent to 80 conventional layers)." Since the N-gram Embedding still provides some advantage at 40 layers (the gap is small but not zero), and the advantage is amplified by width, the practical guidance is to favor wide, moderately deep architectures when using N-gram Embedding. The exact tradeoff between width and depth for embedding scaling is not quantified parametrically, but the directional effect is clear: width helps, depth hurts.

---

Efficient Inference: The N-gram Cache and System Optimizations (Section 4)

A fundamental concern with embedding-scaled models is whether the theoretical parameter efficiency translates to actual inference speedups. Adding 30B+ parameters to the embedding layer could create a new I/O bottleneck that offsets the gains from reduced expert parameters. Section 4 develops a complete inference system to address this.

Reduction of MoE activation parameters (Section 4.1). By shifting parameters from experts to embeddings, the model reduces the number of activated parameters within MoE layers — the part of the model that is memory I/O-bound during autoregressive decoding. Each decoding step requires loading the activated expert parameters from GPU memory, and reducing the number/size of experts directly reduces this I/O volume. The embedding layer's increased size does not penalize latency because embedding lookups are indexed — only the embeddings for tokens actually in the sequence are accessed, regardless of how many total embedding vectors exist. The paper states: "the computational cost of embedding lookups scales with the number of input tokens rather than the total number of embedding parameters."

The batch size requirement. To fully capitalize on the reduced MoE activation, the system must maximize hardware utilization through large batch sizes. Figure 8(a) shows the number of activated experts for LongCat-Flash-Lite versus its vanilla (expert-scaled) counterpart across varying batch sizes. At small batch sizes, the GPU is underutilized because there are not enough tokens to saturate the memory bandwidth — the reduced expert count helps but the GPU still has idle cycles. At large batch sizes, the throughput benefit becomes substantial.

Speculative decoding synergy. Multi-step speculative decoding naturally provides large effective batch sizes. In speculative decoding, a lightweight draft model generates multiple candidate tokens, which are then verified in parallel by the target model. This parallel verification step processes multiple tokens simultaneously, creating a batch that can saturate the GPU's memory bandwidth. The paper deploys LongCat-Flash-Lite with Eagle3 (Li et al., 2025) using a 3-step speculative decoding strategy — the draft model generates 3 tokens, and the target model verifies all 3 in one forward pass, effectively tripling the batch size compared to standard autoregressive generation. This converts the theoretical sparsity advantage into tangible throughput improvements.

The N-gram Cache (Section 4.2). While the embedding layer's asymptotic cost is O(1) per token, the practical implementation of N-gram Embedding lookups introduces non-trivial I/O, computation, and communication overhead compared to a standard embedding layer. Each token requires:

1. Computing hash values for all n-gram orders and sub-tables (CPU or GPU computation).
2. Looking up sub-embeddings from $K \times (N-1)$ separate embedding tables (memory I/O).
3. Applying linear projections $W_{n,k}$ for each sub-table (matrix multiplication).
4. Summing and normalizing the results (reduction operation).

Without optimization, this sequence of operations can be slower than the expert computation it replaces, negating the benefit.

The N-gram Cache is a specialized caching mechanism inspired by the design of the KV cache. The key insight is that n-gram hashes are deterministic functions of the token sequence, and in autoregressive decoding, each new token only adds one new n-gram context (the n-gram ending at the new token position). Rather than recomputing all n-gram embeddings from scratch at each step, the N-gan Cache stores previously computed n-gram embeddings and only computes the new ones. The paper implements this via custom CUDA kernels that manage n-gram IDs directly on the GPU device, enabling low-overhead synchronization with the inference framework's scheduling logic.

Kernel fusion and system optimizations (Section 6.4). For the deployed LongCat-Flash-Lite model, the paper implements several additional optimizations:

• Kernel fusion: combining multiple GPU operations into single kernels to reduce launch overhead and memory traffic. Specific fusions include: AllReduce + Residual Add + RMSNorm (combining three operations that would otherwise require three separate kernel launches); AllGather + Q-Norm + KV-Norm (for attention computation); ReduceScatter + RMSNorm + Hidden State Combine (for the MoE output); and integrating activation quantization into existing operators. Additionally, router logit processing (Softmax + TopK + router scaling) and zero-expert selection are consolidated into a single unified kernel.

• Optimized attention combine: during decoding, the attention computation uses a split-KV strategy where the key-value cache is partitioned across devices. When the number of KV splits is high, the combine operation (aggregating partial attention outputs) can incur latency comparable to the attention computation itself. By optimizing the combine kernel, the paper reports a 50% reduction in its latency.

• PDL (Programmatic Dependent Launch): a CUDA feature that allows dependent kernels to overlap their execution by triggering early launches. Normally, a kernel that depends on the output of a previous kernel must wait for that kernel to fully complete before launching. PDL allows the dependent kernel to begin execution as soon as its input data starts becoming available, eliminating idle gaps between kernel invocations. The paper states this improves SM (Streaming Multiprocessor) utilization.

• Wide expert parallelism and SBO: following prior work (Qian et al., 2025; Meituan, 2025), the model uses wide Expert Parallelism (distributing experts across many devices) and Single Batch Overlap (overlapping computation and communication for different micro-batches) to accelerate inference. These are standard MoE inference optimizations that become more effective when the expert count is reduced (fewer experts per device means less communication overhead).

Speculative directions (Section 4.3). The paper identifies two forward-looking applications of N-gram Embedding for inference acceleration that are not yet implemented but are presented as research directions:

1. N-gram Embedding based drafting: because the N-gram Embedding aggregates information from the preceding $N-1$ tokens, it implicitly captures short-range token dependencies. The paper is exploring architectures to repurpose the N-gan Embedding output as an ultra-fast draft model for speculative decoding, by attaching a lightweight linear projection directly to the N-gram Embedding output to predict the next token. This would be extremely cheap (no transformer layers involved) and could provide useful draft tokens for the target model to verify.

2. Early rejection: the N-gram Embedding representation could serve as a semantic consistency check for tokens generated by external draft models. If a draft token produces a low-probability match under the N-gram Embedding (meaning the n-gram context plus candidate token is rare or implausible), it could be rejected before entering the expensive verification phase of the target model, reducing wasted computation on obviously wrong draft tokens.

---

Per-Layer N-gram Embedding (Section 5)

While the primary N-gram Embedding mechanism augments only the input embedding layer, the paper explores an extension that integrates n-gram information at every transformer layer, analogous to Per-Layer Embedding (PLE) approaches in prior work.

Per-Layer Embedding baseline (Section 5.1). PLE replaces the up-projection matrix output in the SwiGLU FFN module with an embedding lookup. The SwiGLU module normally computes $\text{FFN}(x) = W_d (\text{SiLU}(W_g x) \odot W_u x)$, where $W_g$ is the gate projection, $W_u$ is the up-projection, and $W_d$ is the down-projection. PLE substitutes the up-projection output $W_u x$ with an embedding vector $E_0^{(l)}(t_i)$ that depends only on the token ID (not the hidden state $x$):

$\text{FFN}^{(l)}(x_i) = W_d^{(l)}\left(\text{SiLU}(W_g^{(l)} x_i^{(l)}) \odot E_0^{(l)}(t_i)\right)$

where $E_0^{(l)}$ is a layer-specific embedding table of shape $V_0 \times D_{\text{ff}}$ (matching the FFN intermediate dimension, not the hidden dimension). The paper states this is "the most efficient method for injecting embedding information" because it directly replaces a matrix multiplication with a lookup, saving computation while adding parameters.

Per-Layer N-gram Embedding (PLNE) extension (Section 5.2). The authors propose PLNE as a direct extension: replace the base embedding $E_0^{(l)}(t_i)$ in PLE with the N-gram Embedding output $e_i^{(l)}$, computed using layer-specific N-gram embedding tables and projection matrices:

$\text{FFN}^{(l)}(x_i) = W_d^{(l)}\left(\text{SiLU}(W_g^{(l)} x_i^{(l)}) \odot e_i^{(l)}\right)$

where $e_i^{(l)}$ is computed according to Equation 3 with layer-specific tables $E_{n,k}^{(l)}$ and projection matrices $W_{n,k}^{(l)}$. This allows each layer to learn its own n-gram representations, potentially capturing different levels of n-gram abstraction (early layers might focus on local syntax, later layers on longer-range semantic patterns).

Empirical comparison (Section 5.3, Figure 9). The comparison is structured carefully to avoid confounding parameter count differences. Since PLNE adds n-gram vocabulary parameters at each layer, it has more total parameters than PLE at the same layer count. Therefore, PLNE is compared against a parameter-equivalent N-gram Embedding (NE) baseline, and PLE is compared against a separate parameter-equivalent NE baseline at a different scale. The results:

• PLE underperforms NE: the standard N-gram Embedding applied at the input layer achieves lower loss than PLE at equivalent parameter counts. The authors attribute this to "the superior learning efficiency of N-gram Embedding compared to standard embeddings" — the n-gram context provides richer training signal per parameter than a simple token-based embedding, even when that embedding is distributed across layers.

• PLNE yields marginal improvements over NE: the per-layer n-gram variant slightly outperforms the input-only N-gram Embedding in the 790M setting. However, this advantage is small and inconsistent — in subsequent experiments with increased model width or depth, PLNE performed "on par with NE in most scenarios."

• PLNE was not adopted for LongCat-Flash-Lite: the marginal gains did not justify the increased activated parameter count (since each PLNE layer adds a substantial projection matrix $W_{n,k}^{(l)}$ that must be computed during inference). The paper leaves open the question of optimal per-layer allocation: "specifically regarding the optimal allocation of embedding parameters across layers, such as determining whether to concentrate them in a few specific layers or distribute them uniformly throughout the network."

4. Key Insights and Innovations

Innovation 1: Embedding Parameters and Expert Parameters Are Competing Allocation Targets Within a Fixed Budget — Not Independent Scaling Dimensions

The paper's most consequential intellectual move is its reframing of model scaling as a zero-sum resource allocation problem. Prior work on embedding scaling (Huang et al., 2025; Pagnoni et al., 2025; Sadhukhan et al., 2026) treated the embedding layer as an additive component — something you attach to an existing architecture to improve it, evaluated against a baseline without the embedding augmentation. The implicit question was "does adding N-gram Embedding help?" The answer from prior work was yes, and the research agenda was about refining the mechanism.

This paper changes the question entirely: given a fixed total parameter count, should those parameters live in expert FFN modules or in N-gram Embedding tables? The distinction is not semantic — it fundamentally alters how a practitioner thinks about model design. Under the additive framing, you design your transformer body first (layers, hidden size, expert count), then optionally augment with embeddings. Under the zero-sum framing, the embedding module competes with the transformer body for the parameter budget, and the optimal architecture is the one that allocates parameters to whichever dimension yields the higher marginal return at the current sparsity level.

The paramter-equivalent baseline construction (Section 3) operationalizes this: for every N-gram Embedding model, a MoE model with identical total parameters is created by converting the embedding parameters into additional experts. This is not a standard ablation — it is a controlled economic experiment measuring the "exchange rate" between the two parameter types. When the N-gram Embedding curve lies below the MoE curve in Figures 2 and 6, embedding parameters are more "valuable" than expert parameters — they provide more loss reduction per parameter. When the curves cross, expert parameters become more valuable.

Why this is fundamental rather than incremental: the zero-sum framing transforms embedding scaling from a technique into a design principle. Before this paper, embedding scaling was something you might add to a model. After this paper, embedding scaling is something you trade off against expert count, with the optimal allocation ratio determined by the model's width, depth, and base sparsity. This is the same conceptual leap that Chinchilla scaling laws (Hoffmann et al., 2022) made for pretraining — moving from "more data helps" to "there is an optimal ratio of parameters to tokens for a given compute budget." The paper does not propose a parametric law, but the intellectual structure is identical: identify a tradeoff, characterize the Pareto frontier, and derive prescriptive allocation rules.

The evidence for this framing's validity comes from the intersection points in Figures 2, 6(a), and 6(b). These intersections are not arbitrary — they are the points where the marginal return on embedding parameters equals the marginal return on expert parameters. The systematic shift of these intersection points with model width (rightward as width increases) and depth (gap narrows as depth increases) demonstrates that the exchange rate is not fixed but is modulated by architectural choices. This is what elevates the finding above a simple "embeddings sometimes beat experts" observation: it characterizes when and why.

Innovation 2: The Log-Linear MoE Scaling Curve Creates a Natural Entry Point for Embedding Scaling — But Only Past a Critical Sparsity Threshold

The paper's second conceptual contribution is the empirical characterization of diminishing returns in expert scaling as the mechanism that creates the opportunity for embedding scaling. This is not merely observing that MoE scaling saturates (Abnar et al., 2025 already noted this). The insight is that the log-linear shape of the MoE scaling curve (Figure 2, blue curve) defines a phase boundary: below a critical sparsity threshold, the marginal return on expert parameters is high enough that diverting parameters to embeddings is strictly worse (green curve above blue); above the threshold, the marginal return on experts has diminished sufficiently that embedding parameters become the more efficient investment (red curve below blue).

What makes this finding intellectually distinctive is that it explains why embedding scaling sometimes fails and sometimes succeeds, resolving a potential contradiction. Without the phase boundary concept, one could observe N-gram Embedding outperforming MoE at high sparsity and underperforming at low sparsity, and conclude the results are inconsistent or dataset-dependent. The paper instead identifies the underlying mechanism — the log-linear relationship means that expert scaling efficiency is not constant but decays predictably with sparsity, creating a natural "switching point" where an alternative scaling dimension becomes preferable.

This has a direct consequence for how the field should think about architecture search: sparsity level is a moderator variable that determines the relative efficacy of different scaling strategies. This means that whether a given technique "works" is not a binary property of the technique itself but depends on where the base model sits on the sparsity spectrum. A model at 35% sparsity might correctly conclude that embedding scaling is ineffective, while a model at 98% sparsity would reach the opposite conclusion — both results are valid, but they are local to their operating regime, not universal truths about the technique. This reframing mirrors the difficulty-conditioned findings in the reference paper on test-time compute scaling, where the same strategy (beam search, sequential revisions) could be optimal or harmful depending on problem difficulty.

The evidence is the three-curve structure in Figure 2: the green curve (N-gram Embedding at low ratio) consistently underperforms the MoE baseline, while the red curve (N-gram Embedding at high ratio) outperforms until the intersection point. The fact that this structure replicates across activation scales (280M, 790M, 1.3B) and shifts predictably with width (Figure 6) indicates it is a robust architectural regularity, not a quirk of a specific model size.

Innovation 3: Residual Signal Attenuation as the Mechanistic Explanation for the Width-Depth Asymmetry in Embedding Scaling

The paper's diagnostic analysis of how model width and depth modulate embedding scaling efficacy (Section 3.3) is more than an empirical observation — it provides a mechanistic explanation grounded in the transformer's residual stream dynamics that explains the asymmetry. The finding that N-gram Embedding benefits from wider models but is penalized by deeper models (Figure 7) would be merely observational without the accompanying analysis in Figure 5 and the embedding amplification study in Section 3.2.4. Together, these pieces form a coherent mechanistic story.

The core insight: the N-gram Embedding output propagates through the residual stream, and its effective contribution to the model's computation depends on the ratio of its signal strength to the cumulative outputs of all subsequent transformer layers. In a wider model, the hidden dimension is larger, so the N-gram Embedding vector carries more bits of information (its representational capacity scales with dimensionality), and the downstream layers have proportionally more capacity to process this enriched input. In a deeper model, the embedding signal is progressively diluted — each additional layer adds its own output to the residual stream, and the relative contribution of the original embedding shrinks approximately as $1/\sqrt{L}$ (assuming roughly equal-norm additions per layer).

This is significant beyond the specific finding about N-gram Embedding because it identifies residual stream capacity allocation as a first-class design consideration. The transformer's residual stream is effectively a communication channel between layers, and different architectural choices (adding embedding parameters at the input, adding expert parameters distributed across layers) compete for "bandwidth" in this channel. The paper's embedding amplification technique (scaling factor or LayerNorm on the embedding output) can be understood as a way to increase the embedding's "transmit power" to compete with later-layer signals — but this is a patch, not a solution to the fundamental attenuation problem in deep networks.

The comparison with Per-Layer N-gram Embedding (PLNE) in Section 5 reinforces this diagnostic. PLNE attempts to solve the attenuation problem by injecting n-gram information at every layer rather than only at the input. The marginal improvement over input-only N-gram Embedding is small (Figure 9), suggesting that the attenuation problem is not the dominant bottleneck at the tested depths (10 layers), but becomes more important as depth increases — consistent with the finding that the embedding advantage shrinks at 40 layers. The paper's decision not to adopt PLNE for LongCat-Flash-Lite (because it increases activated parameters) reflects a pragmatic judgment about the depth regime: at 14 shortcut layers, the input-only approach is sufficient.

This is a fundamental rather than incremental contribution because it provides a causal framework for reasoning about embedding scaling, not just a set of empirical observations. A practitioner using this framework can predict that embedding scaling will be most effective in wide, shallow models (like LongCat-Flash-Lite at 14 shortcut layers) and least effective in narrow, deep models, without needing to run the experiment themselves.

Innovation 4: Speculative Decoding as the Enabling Infrastructure for Embedding-Scaled Models, Not an Independent Optimization

The paper's treatment of inference (Section 4) contains a subtle but important conceptual move: speculative decoding is not presented as an optional acceleration technique but as an infrastructure requirement that converts the theoretical efficiency of embedding-scaled models into realized speedups. This reframes the relationship between architecture and inference systems: the embedding-scaled architecture creates a specific computational profile (low activated parameters in MoE layers, high embedding lookup cost) that demands a specific systems solution (large effective batch sizes via speculative decoding) to achieve its potential.

The reasoning chain is: embedding scaling reduces MoE activated parameters → the model becomes memory I/O-bound during decoding → to saturate memory bandwidth, large batch sizes are needed → autoregressive decoding produces batch size 1 → speculative decoding creates "effective" batch sizes of $k$ (the number of draft tokens) by verifying multiple tokens in parallel → the embedding-scaled model achieves its theoretical speedup. Each link in this chain is a necessary condition; break one, and the efficiency gains do not materialize.

This is intellectually distinctive because it identifies a coupling between architectural choice and inference system design that is not obvious from either perspective alone. From a pure architecture perspective, embedding scaling looks attractive because it reduces activated parameters. From a pure systems perspective, speculative decoding is an optimization for any autoregressive model, independent of architecture. The paper shows that these are not independent: the architectural choice creates the need for speculative decoding (because the model is more severely I/O-bound than an expert-heavy counterpart), and speculative decoding provides the mechanism to realize the architectural benefit. The evidence is Figure 8, which couples the reduction in activated experts (Figure 8a) with the decoding performance benefits (Figure 8b), mediated by the speculative decoding strategy (Eagle3, 3-step).

The N-gram Cache (Section 4.2) and the speculative directions in Section 4.3 further develop this coupling. The N-gram Cache is a direct analog of the KV cache — a mechanism for avoiding redundant computation in autoregressive generation — applied to the embedding layer rather than the attention layer. The speculative directions (using N-gram Embedding as a draft model, early rejection) propose closing the loop further: using the embedding-scaled architecture's own structure to accelerate the speculative decoding that enables its efficient inference. This creates a self-reinforcing design pattern where the architecture and the inference system are co-designed, not sequentially developed.

The significance goes beyond the specific techniques. The paper is implicitly arguing that as models become more sparsely activated, the traditional separation between "model architecture" and "inference system" breaks down. The inference system must be designed in concert with the architecture, because the architecture determines the computational bottlenecks that the inference system must address. This is not a new idea in computer architecture (hardware-software co-design is standard in chip design), but it is underappreciated in the LLM scaling literature, which tends to treat inference optimization as an orthogonal engineering concern. The paper's detailed treatment of inference — including the specific kernel fusions, PDL, and speculative decoding configuration — operationalizes this co-design philosophy for a specific architectural choice (embedding scaling), providing a template for how future architecture papers should address the deployment implications of their design decisions.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All comparative scaling experiments (Sections 3.1–3.5) use a pre-training corpus of 300B tokens. The paper does not disclose the exact composition of this corpus, but states that evaluation is performed on "two meticulously constructed datasets, covering both Chinese and English" (Section 3 introduction). These serve as held-out validation sets. No public benchmark names are provided for the scaling experiments. For the flagship LongCat-Flash-Lite model (Section 6), the pre-training corpus is 11T tokens followed by 1.5T tokens of mid-training, using the same data recipe as LongCat-Flash-Chat (Meituan, 2025). Downstream evaluation for LongCat-Flash-Lite uses standard public benchmarks: MMLU, MMLU-Pro, C-Eval, CMMLU (general); BBH, GPQA, DROP, GSM8K (reasoning); HumanEval+, MultiPL-E, BigCodeBench (coding) for the base model; and Tau2-Bench, VitaBench (agentic tool use), SWE-Bench, TerminalBench, SWE-Bench Multilingual, PRDBench (agentic coding), GPQA-Diamond, MMLU, MMLU-Pro, C-Eval, CMMLU (general domains), MATH500, AIME24, AIME25 (mathematical reasoning) for the chat model.

• Base model(s). All experiments use the LongCat-Flash architecture (Meituan, 2025), a pre-existing Mixture-of-Experts transformer design. The scaling experiments span three activation scales: 280M, 790M, and 1.3B activated parameters. At each scale, models are configured with varying sparsity levels (total-to-activated parameter ratios) by adjusting the number of experts. The flagship LongCat-Flash-Lite has 68.5B total parameters with 2.9B–4.5B activated parameters per token (varying due to zero-experts), 14 shortcut layers, 256 FFN experts and 128 zero-experts per MoE module, with each token selecting 12 experts. The model includes 31.4B N-gram Embedding parameters (~46% of total). The paper states that LongCat-Flash was chosen as the base architecture because it is representative of contemporary sparse MoE designs operating in the high-sparsity regime where the authors hypothesize embedding scaling would be most beneficial.

• Metrics. For the scaling experiments (Sections 3.1–3.5), the primary metric is training loss (presumably cross-entropy loss on next-token prediction, though the exact loss function is not specified) and validation loss on two held-out datasets (Chinese and English). The paper uses loss rather than downstream task performance because the scaling experiments use relatively small models (280M–1.3B activated) trained on 300B tokens, where downstream benchmarks would have low signal. For LongCat-Flash-Lite (Section 6), evaluation uses task-specific metrics: accuracy for MMLU, MMLU-Pro, C-Eval, CMMLU, GSM8K, MATH500, HumanEval+, SWE-Bench, TerminalBench, SWE-Bench Multilingual; pass@1 or avg@k for BBH, GPQA, DROP, MultiPL-E, BigCodeBench, GPQA-Diamond, AIME24, AIME25, Tau2-Bench, VitaBench, PRDBench. The paper uses standard evaluation protocols from each benchmark's original publications but does not detail the exact prompting or few-shot configurations in this report.

• Baselines. For the comparative scaling experiments (Section 3), the primary baseline is the parameter-equivalent MoE baseline, constructed by taking each N-gram Embedding model and converting its N-gram Embedding parameters into additional expert FFN modules, maintaining identical total parameter count, activated parameter count, and training data. This ensures that any loss difference is attributable to architectural allocation rather than parameter count. For LongCat-Flash-Lite (Section 6), baselines include: LongCat-Flash-Lite-Vanilla (the parameter-equivalent MoE model with all N-gram Embedding parameters converted to additional experts, trained identically); Kimi-Linear-48B-A3B (a 48B total, 3B activated MoE model); Qwen3-Next-80B-A3B-Instruct (an 80B total, 3B activated MoE-MoE hybrid); and Gemini 2.5 Flash-Lite (architecture details not specified, Preview 09-2025 version). Values for Kimi-Linear-48B-A3B and Gemini 2.5 Flash-Lite marked with * in Table 2 are sourced from public reports rather than the authors' own evaluation.

• Generation budget / compute accounting. For the scaling experiments, the compute metric is total number of parameters held fixed between comparison models. The paper does not report total FLOPs for the scaling runs, instead controlling for parameter count (which determines memory and communication costs) and activated parameter count (which determines per-token computation). The independent variable is the total-to-activated parameter ratio on the x-axis of scaling plots, representing sparsity level. For inference (Section 4 and 6.4), the paper uses latency and throughput measured on 8xH800-80G GPUs with input sequence length 4K and output sequence length 1K (Figure 8b). The inference setup uses Eagle3 speculative decoding with 3-step drafting, wide expert parallelism, and Single Batch Overlap.

• Cross-validation / statistical protocol. No cross-validation or statistical protocol is described for the scaling experiments. The paper reports point estimates of training and validation loss without confidence intervals or error bars. For LongCat-Flash-Lite's downstream evaluation, the paper reports single-run results on standard benchmarks using each benchmark's standard evaluation protocol. The lack of statistical rigor (no multiple seeds, no confidence intervals, no significance testing) is a notable limitation given that the scaling experiments involve relatively small models where training variance could be substantial.

Main Quantitative Results

Comparative Scaling: N-gram Embedding vs. Expert Scaling (Section 3.1–3.3)

The paper's central empirical claim is that N-gram Embedding achieves lower loss than a parameter-equivalent MoE baseline when introduced at sufficiently high base sparsity, and that this advantage is modulated by model width and depth. The evidence is organized across Figures 2, 6, and 7.

Figure 2 (280M activated parameter scale). At the smallest activation scale, three scaling trajectories are shown:

• MoE baseline (blue curve): follows a strict log-linear relationship between total-to-activated parameter ratio and loss. The curve shows steep loss reduction at low ratios (~8–12) and progressively shallower reduction at higher ratios (~20+). At ratio ~8, loss is approximately 2.95 (estimated from logarithmic axis); at ratio ~20, loss is approximately 2.75; at ratio ~50, loss is approximately 2.60.

• N-gram Embedding at low ratio (green curve, base ratio ~8): the curve branches from the MoE baseline at ratio ~8 and extends to ratio ~20 (with dashed line connecting to the base MoE model). The green curve lies above the blue curve at all points — meaning the N-gram Embedding model has higher loss than the parameter-equivalent MoE at every tested ratio. At ratio ~20, the green curve shows loss approximately 2.80 vs. ~2.75 for the MoE baseline.

• N-gram Embedding at high ratio (red curve, base ratio ~12): the curve branches from the MoE baseline at ratio ~12 and extends to ratio ~20. The red curve lies below the blue curve for ratios ~12–20, indicating lower loss. At ratio ~16, the gap is approximately 0.02–0.03 in loss. At ratio ~20, the red and blue curves intersect — the N-gram Embedding advantage disappears when embedding parameters constitute roughly 50% of total parameters.

The key quantitative takeaway: N-gram Embedding introduced at ratio ~12 achieves a loss reduction of approximately 0.02–0.03 over the MoE baseline, but this advantage is bounded — it exists only above the critical base sparsity (ratio ≥ ~12) and below the critical embedding proportion (~50% of total parameters).

Figure 6(a) (790M activated parameter scale, 10 layers). At the intermediate scale:

• The N-gram Embedding curve (red) consistently lies below the MoE baseline (blue) for the tested ratio range (up to ~50). The gap is visible but not quantified in absolute terms in the paper text. The intersection point — where N-gram Embedding would lose its advantage — is not reached within the tested range for training loss and Chinese validation, but the paper notes that "N-gram Embedding only underperforms on the English validation set at this ratio" (the highest ratio tested).

• The green curve (N-gram Embedding at lower base ratio) again lies above the MoE baseline, replicating the pattern from Figure 2.

Figure 6(b) (1.3B activated parameter scale, 10 layers). At the largest tested small-model scale:

• The N-gram Embedding curve (red) "retains a clear advantage even at ratios as high as 50" (Section 3.3.1). This means that at 1.3B activation with 10 layers, embedding parameters constituting roughly 76% of total parameters (calculated as $(50-12)/50$ with base ratio ~12) still outperform expert-scaled baselines at equivalent total parameter count.

• The intersection point has shifted so far right that it is not observed within the tested range.

Figure 7 (width vs. depth scaling at 1.3B activation). Figure 7(a) quantifies the performance gap between N-gram Embedding and MoE baseline as a function of model width (at fixed 10-layer depth). The gap widens with width — the exact loss reduction values are shown as a bar chart, though specific numbers are not quoted in the text beyond the qualitative statement that "the performance gap demonstrably widens." Figure 7(b) shows the opposite trend for depth: at fixed width (1.3B activated) and fixed embedding proportion (50% of total parameters), the performance gap shrinks as depth increases from 10 to 20 to 40 shortcut layers. The gap is described as undergoing a "pronounced contraction" after 20 layers and approaching zero at 40 layers.

Training loss curves for LongCat-Flash-Lite (Figure 10). The full-scale model comparison shows that LongCat-Flash-Lite (with N-gram Embedding) achieves "consistently lower training loss compared to LongCat-Flash-Lite-Vanilla" throughout the 1.3T-token training run shown. The smoothed loss curves show a persistent gap, with the loss drop at 420B tokens (coinciding with a batch size increase) affecting both models similarly. Specific loss values are not reported.

Base Model Downstream Evaluation (Section 6.2, Table 1)

LongCat-Flash-Lite is compared against LongCat-Flash-Lite-Vanilla at the 1.3T-token checkpoint across 11 benchmarks in three domains. The key quantitative results (Table 1):

General tasks:

• MMLU: 64.01 (Lite) vs. 64.81 (Vanilla) — a small 0.8-point deficit for N-gram Embedding.
• MMLU-Pro: 35.89 vs. 34.43 — a 1.46-point advantage.
• C-Eval: 67.21 vs. 64.09 — a 3.12-point advantage.
• CMMLU: 69.55 vs. 67.08 — a 2.47-point advantage.

Reasoning tasks:

• BBH: 43.67 vs. 38.54 — a 5.13-point advantage.
• GPQA: 29.66 vs. 25.37 — a 4.29-point advantage.
• DROP: 52.43 vs. 47.92 — a 4.51-point advantage.
• GSM8K: 50.50 vs. 50.00 — a 0.50-point advantage (essentially tied).

Coding tasks:

• HumanEval+: 31.10 vs. 28.66 — a 2.44-point advantage.
• MultiPL-E: 30.03 vs. 30.20 — a 0.17-point deficit (essentially tied).
• BigCodeBench: 36.05 vs. 33.42 — a 2.63-point advantage.

Aggregating across all 11 benchmarks, LongCat-Flash-Lite outperforms Vanilla on 8, underperforms on 2 (MMLU by 0.8, MultiPL-E by 0.17), and ties on 1 (GSM8K). The largest gains are on BBH (+5.13), DROP (+4.51), and GPQA (+4.29). The paper claims these results "validate our earlier analysis" that N-gram Embedding outperforms expert scaling at sufficient sparsity. However, the base model evaluations are reported at a single checkpoint (1.3T tokens) rather than at multiple points, making it unclear whether the advantage is consistent throughout training or emerges late.

Chat Model Evaluation (Section 6.3, Table 2)

LongCat-Flash-Lite (chat fine-tuned) is compared against three external models of similar activated parameter scale: Kimi-Linear-48B-A3B, Qwen3-Next-80B-A3B-Instruct, and Gemini 2.5 Flash-Lite. The evaluation covers agentic tool use, agentic coding, general domains, and mathematical reasoning.

Agentic tool use (Tau2-Bench, VitaBench): LongCat-Flash-Lite achieves the highest scores across all four benchmarks:

• Tau2-Airline (avg@8): 58.00 vs. 44.00 (Kimi), 45.5* (Qwen3), 35.00 (Gemini) — a lead of 12.5–23.0 points.
• Tau2-Retail (avg@8): 73.10 vs. 18.86 (Kimi), 57.3* (Qwen3), 37.50 (Gemini) — a lead of 15.8–54.24 points. The Kimi result (18.86) is dramatically lower, suggesting either a different evaluation setup or a fundamental weakness in tool-use capability.
• Tau2-Telecom (avg@8): 72.80 vs. 15.68 (Kimi), 13.2* (Qwen3), 21.93 (Gemini) — a lead of 50.87–59.6 points. This is the single largest margin in the table and the paper highlights it: "its score significantly outperforms Gemini 2.5 Flash-Lite and Kimi-Linear-48B-A3B."
• VitaBench (avg@4): 7.00 vs. 5.80 (Qwen3), 4.50 (Gemini). Kimi is not reported on this benchmark.

Agentic coding:

• SWE-Bench (acc): 54.40 vs. 32.80 (Kimi), 37.60 (Qwen3), 41.3* (Gemini) — a lead of 13.1–21.6 points.
• TerminalBench (acc): 33.75 vs. 20.00 (Kimi), 15.19 (Qwen3), 20.00 (Gemini) — a lead of 13.75–18.56 points.
• SWE-Bench Multilingual: 38.10 vs. 37.20 (Kimi), 31.30 (Qwen3). Gemini not reported.
• PRDBench: 39.63 vs. 15.36 (Qwen3). Kimi and Gemini not reported.

General domains:

• GPQA-Diamond (avg@16): 66.78 vs. 69.89 (Kimi), 74.33 (Qwen3), 70.20* (Gemini) — LongCat-Flash-Lite is lowest in this category by 3.11–7.55 points.
• MMLU (acc): 85.52 vs. 79.91 (Kimi), 89.28 (Qwen3), 84.68 (Gemini) — second place, 3.76 points behind Qwen3 but ahead of the others.
• MMLU-Pro (acc): 78.29 vs. 67.22 (Kimi), 82.93 (Qwen3), 78.95 (Gemini) — essentially tied with Gemini, behind Qwen3 by 4.64.
• CEval (acc): 86.55 vs. 78.48 (Kimi), 90.91 (Qwen3), 75.16 (Gemini) — second place, behind Qwen3 by 4.36.
• CMMLU (acc): 82.48 vs. 76.26 (Kimi), 86.50 (Qwen3), 72.06 (Gemini) — second place, behind Qwen3 by 4.02.

Mathematical reasoning:

• MATH500 (acc): 96.80 vs. 94.20 (Kimi), 98.00 (Qwen3), 95.20 (Gemini) — second place, 1.2 points behind Qwen3.
• AIME24 (avg@32): 72.19 vs. 70.52 (Kimi), 81.35 (Qwen3), 63.33 (Gemini) — second place, 9.16 points behind Qwen3.
• AIME25 (avg@32): 63.23 vs. 59.58 (Kimi), 68.44 (Qwen3), 50.1* (Gemini) — second place, 5.21 points behind Qwen3.

The pattern is striking: LongCat-Flash-Lite dominates agentic tasks (tool use and coding) by large margins, is competitive but generally behind Qwen3-Next on general domain and mathematical reasoning benchmarks, and trades blows with Kimi-Linear and Gemini 2.5 Flash-Lite depending on the specific benchmark. The paper's headline claim — "exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains" — is supported specifically for the agentic category where margins are 10–60 points, but the general domain and math results show a consistent second-place position behind Qwen3-Next with gaps of 1–9 points.

Inference Performance (Section 6.4, Figure 8)

Figure 8(b) plots decoding performance on 8xH800-80G GPUs with ISL=4K and OSL=1K. The exact throughput numbers are not quoted in the text, but are shown graphically as a function of batch size. The paper states that "we achieve the exceptional inference performance illustrated in Figure 8(b)" after implementing kernel fusion, PDL, Eagle3 speculative decoding, wide expert parallelism, and SBO. Without specific numbers quoted, this is a qualitative rather than quantitative result in the body of the paper.

Ablation Studies and Robustness Checks

N-gram order (N) and sub-table count (K): Figure 4 compares training and validation loss across combinations of $N \in \{2,3,4,5\}$ and $K \in \{1,2,4,8\}$ using the 790M activated-parameter model. The minimal configuration ($N=2, K=1$) shows "notably inferior" performance. For $N \geq 3$ and $K \geq 2$, performance variance is "relatively small," with the paper stating that the model "is robust to hyperparameter selection within this regime." The specific recommendation is $N$ in the range of 3 to 5. This is a positive robustness result — the technique does not require precise hyperparameter tuning to achieve its benefits.

Vocabulary size for hash collision mitigation: Figure 3(b) shows 2-gram hash collision counts as a function of vocabulary size between 30× and 33× the base vocabulary. The key finding is non-monotonic: collision counts "spike noticeably when the vocabulary size approaches an integer multiple of the base vocabulary size." The paper recommends avoiding these spike regions but does not provide a quantitative ablation showing how much collision rate affects final model loss — only the collision counts themselves are shown, not downstream performance.

Embedding amplification: The paper reports a consistent reduction of 0.02 in both training loss and the two validation losses when applying embedding amplification (scaling factor or LayerNorm) compared to the vanilla initialization. This ablation is presented as a text claim without a dedicated figure or table, making it difficult to assess the consistency across different model scales or configurations. The result is described as coming from "an early vanilla experiment" (Section 3.2.4), suggesting it was not replicated across all scaling configurations.

Input-only NE vs. PLE vs. PLNE: Figure 9 compares three embedding scaling strategies at the 790M activation scale:

• PLE (Per-Layer Embedding) vs. NE (N-gram Embedding, input only): PLE "underperforms relative to N-gram Embedding" at equivalent parameter counts. The authors attribute this to the superior learning efficiency of n-gram context vs. single-token embeddings.
• PLNE (Per-Layer N-gram Embedding) vs. NE: PLNE "yields marginal improvements over NE" but the advantage is small and inconsistent — "in subsequent experiments involving increased model width or depth, PLNE failed to exhibit a consistent advantage, performing on par with NE in most scenarios." This is a notable negative result: the intuitive extension (putting n-gram information at every layer) does not reliably improve over the simpler input-only approach.
• Decision not to adopt PLNE: the paper explicitly chose not to use PLNE for LongCat-Flash-Lite because it increases activated parameters (due to per-layer projection matrices) without guaranteed benefit.

ReST^EM-style revision model training (reference paper context): Not applicable to this paper, which does not involve revision models or RL-based training. However, the paper's own training procedure is not ablated — there are no experiments varying pre-training data quantity, optimizer settings, learning rate schedules, or sequence length to assess robustness to these factors.

Parameter-equivalent baseline control: The entire comparative scaling framework is an implicit ablation on parameter allocation: by holding total parameters constant and varying their location (experts vs. embeddings), the experiments directly test the causal effect of architectural allocation on loss. This is the paper's strongest methodological contribution and is applied consistently across all scaling experiments.

Hash function choice: The paper uses polynomial rolling hash (Equation 2) but does not ablate alternative hash functions. There is no comparison to, e.g., simple modular hashing, cryptographic hashing, or learned hash functions. The hash collision analysis in Figure 3 provides some validation of the chosen approach (by characterizing when it fails), but does not demonstrate that this hash function is optimal or compare it to alternatives.

Zero-experts and expert count: The LongCat-Flash-Lite architecture uses 256 FFN experts and 128 zero-experts per MoE module, with 12 experts selected per token. There is no ablation studying how the number of experts, the number of zero-experts, or the top-k selection interacts with N-gram Embedding efficacy. The paper treats these as fixed architectural choices inherited from LongCat-Flash (Meituan, 2025).

Critical Assessment

Claim 1: "Embedding scaling achieves a superior Pareto frontier compared to increasing expert numbers in specific regimes"

What the experiments demonstrate: The paper shows that at the 280M activation scale (Figure 2), N-gram Embedding introduced at ratio ~12 achieves lower training loss than a parameter-equivalent MoE baseline for ratios between ~12 and ~20. This is a genuine Pareto improvement — same parameter count, lower loss. The effect replicates at 790M and 1.3B activation scales (Figure 6), with the advantage persisting to higher embedding proportions at wider scales.

What the experiments do NOT demonstrate: The experiments measure training and validation loss on a 300B-token corpus, not downstream task performance (except for LongCat-Flash-Lite at full scale). A lower validation loss on held-out data from the same distribution as the training data does not necessarily translate to better performance on standard NLP benchmarks — it could reflect overfitting to the pre-training distribution. The paper addresses this partially through LongCat-Flash-Lite's downstream evaluation (Tables 1 and 2), but the scaling experiments that establish the "Pareto frontier" claim are based entirely on loss, not task metrics.

Additionally, the "Pareto frontier" terminology implies that N-gram Embedding models are strictly better on at least one dimension (loss) while being no worse on others. The experiments only measure one dimension (loss). There is no assessment of whether N-gram Embedding models have different scaling properties in terms of data efficiency, convergence speed, sensitivity to hyperparameters, or training stability — all of which would affect their practical desirability even if loss is lower.

Conditional nature: The claim holds only when N-gram Embedding is introduced at sufficiently high base sparsity (ratio ≥ ~12 at 280M activation, shifting rightward with width). Below this threshold, N-gram Embedding underperforms the MoE baseline (green curve in Figure 2). The claim also requires that the embedding proportion does not exceed approximately 50% of total parameters (at 280M scale), though this threshold increases with model width. The paper is explicit about these conditions, so the claim is valid as stated — but it is narrower than a casual reading might suggest. This is not "embeddings are better than experts" universally; it is "there exists a specific, identifiable regime where embeddings are a more efficient use of additional parameters."

Claim 2: "LongCat-Flash-Lite not only surpasses a parameter-equivalent MoE baseline but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains"

What the experiments demonstrate: LongCat-Flash-Lite outperforms LongCat-Flash-Lite-Vanilla (the parameter-equivalent MoE) on 8 of 11 base model benchmarks (Table 1) and shows consistently lower training loss throughout pre-training (Figure 10). Against external models (Table 2), it achieves dramatic leads on agentic tool-use tasks (Tau2-Telecom: 72.8 vs. 13.2–21.93) and agentic coding (SWE-Bench: 54.4 vs. 32.8–41.3).

Genuine weaknesses in the evidence:

1. The parameter-equivalent baseline is only reported at 1.3T tokens. The paper does not show whether LongCat-Flash-Lite outperforms Vanilla at earlier checkpoints (e.g., 500B, 1T tokens), or whether the gap is widening or narrowing over training. The loss curves in Figure 10 show a consistent gap, but downstream evaluation is only at a single point. If the advantage emerges only late in training, the practical benefit for smaller training budgets is unclear.

2. The external model comparisons may not be apples-to-apples. The comparison models have different total parameter counts (48B for Kimi, 80B for Qwen3, 68.5B for LongCat-Flash-Lite), different activated parameter counts (3B for Kimi, 3B for Qwen3, 2.9–4.5B for LongCat-Flash-Lite), different architectures (standard MoE vs. MoE-MoE hybrid vs. MoE + NE), and likely different pre-training data mixtures and quantities. The paper does not control for any of these differences. The claim of "exceptional competitiveness" is descriptive, not causal — LongCat-Flash-Lite might succeed on agentic tasks because of the N-gram Embedding, or because of its specific training data, or because of architectural features inherited from LongCat-Flash that are independent of embedding scaling. The paper cannot disentangle these factors because it does not compare against a LongCat-Flash model of similar scale without N-gram Embedding (only against Vanilla, which uses a different expert allocation).

3. The agentic task results have extremely wide margins that warrant scrutiny. Tau2-Telecom shows LongCat-Flash-Lite at 72.8 while Qwen3-Next is at 13.2 (a 59.6-point gap) and Gemini 2.5 Flash-Lite at 21.93 (a 50.87-point gap). While the paper presents these as evidence of superiority, gaps this large between models of similar scale on the same benchmark are unusual and could indicate systematic differences in evaluation methodology, prompting, or tool integration setup rather than genuine capability differences. The paper does not describe its evaluation protocol for these benchmarks in sufficient detail to assess whether the comparisons are fair.

4. The paper's own baseline (LongCat-Flash-Lite-Vanilla) is not evaluated on the chat benchmarks. Table 2 compares LongCat-Flash-Lite against external models but does not include LongCat-Flash-Lite-Vanilla in the chat evaluation. This means we cannot attribute the strong agentic performance specifically to N-gram Embedding — it could be that the LongCat-Flash architecture and training recipe are strong on agentic tasks regardless of the parameter allocation. The lack of this ablation is a significant gap.

Claim 3: "Wider models allow for a significantly expanded window of opportunity to leverage N-gram Embedding effectively; deeper models show diminishing returns"

What the experiments demonstrate: Figure 7(a) shows that the loss reduction from N-gram Embedding (vs. MoE baseline) increases with model width at fixed 10-layer depth. Figure 7(b) shows that the loss reduction decreases with model depth at fixed 1.3B activated parameters and fixed 50% embedding proportion.

Limitations: The width scaling experiment varies only one dimension (width) while holding depth constant at 10 layers. The depth scaling experiment varies only depth while holding width and embedding proportion constant. These are partial derivative-style experiments — they show the marginal effect of each variable in isolation. They do not explore the interaction between width and depth: does increasing width compensate for depth-related attenuation? Is there a joint optimum? The paper states that the embedding advantage still exists at 40 layers but is much reduced, and that practical models typically operate below this depth — but this is an observation about existing practice, not a demonstration that embedding scaling is optimal at any particular (width, depth) combination. A parametric characterization (e.g., "for depth $L$, embedding proportion should not exceed $f(L)$") is not provided, limiting the prescriptive power of the finding.

Missing experiment: The paper does not test whether Per-Layer N-gram Embedding (PLNE) recovers the depth-related losses. If the attenuation hypothesis is correct — that the input embedding signal gets diluted across many residual additions — then PLNE should provide proportionally larger benefits in deeper models, since it injects fresh n-gram information at every layer. The paper tested PLNE only at 10 layers (Figure 9), where the benefit was marginal. Testing PLNE at 40 layers would provide a stronger test of the mechanistic explanation and might reveal that PLNE becomes necessary (not just optional) for deep embedding-scaled models.

Claim 4 (implicit): "The N-gram Cache and speculative decoding convert theoretical sparsity into tangible inference speedups"

What the experiments demonstrate: Figure 8(b) shows a decoding performance curve, but no specific throughput numbers are quoted in the text. The paper describes the inference system in detail (kernel fusion, PDL, N-gram Cache, speculative decoding) but does not provide ablation studies showing the contribution of each component to the total speedup. We do not know whether the N-gram Cache alone provides a measurable benefit, or whether speculative decoding is the dominant factor, or whether the kernel fusion optimizations are necessary or merely nice-to-have. The paper also does not compare LongCat-Flash-Lite's inference performance directly against any of the external models (Kimi-Linear, Qwen3-Next, Gemini 2.5 Flash-Lite), making the "fast inference" claim purely self-referential.

Missing ablation: The most important missing comparison is LongCat-Flash-Lite vs. LongCat-Flash-Lite-Vanilla on inference throughput/latency. If the embedding-scaled model is genuinely more efficient at inference than the expert-scaled equivalent (as the theoretical argument in Section 4.1 claims), this should be directly measurable. The paper does not report this comparison.

General methodological concerns

1. No statistical rigor. All losses and benchmark scores are reported as point estimates. Given that the scaling experiments involve training models from scratch (which has inherent variance from random initialization, data ordering, etc.), confidence intervals or multiple training runs would substantially increase confidence in the findings. This is particularly important because the absolute loss differences between N-gram Embedding and MoE baselines are small (0.02–0.03 in Figure 2, possibly within the noise floor of single-seed training runs).

2. The pre-training data is not described. The 300B-token scaling corpus and the 11T-token LongCat-Flash-Lite corpus are not characterized in terms of language mix, domain composition, quality filtering, or deduplication. This matters because N-gram Embedding's effectiveness depends on the frequency and diversity of n-gram patterns in the training data. A corpus with high token repetition (e.g., code, structured data) might favor n-gram embeddings more than a corpus with diverse natural language, because hash collisions would be concentrated on repeated patterns. Without data characterization, the generalizability of the findings to other corpora is unclear.

3. Single architecture family. All experiments use LongCat-Flash. While the paper argues this architecture is representative, the specific design choices — 14 shortcut layers, 256 experts, zero-experts, the particular routing mechanism — could interact with N-gram Embedding in ways that do not transfer to other MoE architectures (e.g., DeepSeek-MoE, Mixtral, GPT-4's rumored architecture).

4. The scaling experiments test small models (≤1.3B activated) but the flagship model is much larger (68.5B total, ~3B activated). The scaling laws derived from 280M–1.3B activation experiments are extrapolated to a ~50× larger total parameter scale without validation at intermediate scales. The paper does not demonstrate that the scaling trends observed at 280M–1.3B activation continue to hold at 3B activation — the LongCat-Flash-Lite results are consistent with the scaling experiments but do not directly validate the extrapolation.

5. No comparison to alternative embedding scaling methods at scale. The paper compares N-gram Embedding against PLE and PLNE at the 790M scale (Figure 9), but LongCat-Flash-Lite uses only input-level N-gram Embedding. There is no large-scale baseline using, e.g., Engram (Cheng et al., 2026), Byte Latent Transformer (Pagnoni et al., 2025), or a simple vocabulary expansion. The paper therefore demonstrates that N-gram Embedding beats the specific expert-scaled baseline, but not that it is the best among embedding scaling approaches at the 68.5B scale.

Summary of evidential support

The paper's strongest experimental contribution is the systematic characterization of the expert-embedding tradeoff in Figures 2 and 6, which provides clear, replicable conditions under which embedding scaling is preferable. The weakest parts of the evidence are the chat model comparisons (where the lack of a parameter-equivalent baseline on chat tasks and the extreme variance in agentic benchmark scores raise questions about the causal attribution to N-gram Embedding) and the inference performance claims (which lack quantitative ablations and direct comparisons). The paper would be strengthened by: (1) evaluating LongCat-Flash-Lite-Vanilla on the same chat benchmarks to isolate the N-gram Embedding effect; (2) reporting inference throughput/latency for both Lite and Vanilla to validate the theoretical efficiency argument; (3) providing confidence intervals or multi-seed results for the scaling experiments; and (4) testing PLNE at depth to confirm or refute the residual attenuation hypothesis.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Not Accounted for in the Efficiency Claims

The assumption or constraint. The entire comparative scaling framework — the identification of the optimal base sparsity threshold for introducing N-gram Embedding (Section 3.1), the optimal embedding parameter proportion (Section 3.2.1), and the width-dependent shift in these thresholds (Section 3.3.1) — relies on the ability to observe where the N-gram Embedding curve crosses the MoE baseline curve in Figures 2 and 6. In the paper's experiments, this requires training multiple models at different sparsity levels and embedding proportions to map out the scaling curves. The paper does not provide a method for predicting, from architectural specifications alone, whether a given model configuration is past the critical sparsity threshold where embedding scaling becomes beneficial. The key decisions — at what ratio to introduce N-gram Embedding, and what fraction of parameters to allocate to embeddings — are derived post-hoc from the completed scaling experiments.

The consequence. A practitioner designing a new model cannot rely on the paper's specific numerical thresholds (e.g., ratio ~12 at 280M activation, shifting rightward with width) without replicating the scaling experiments for their own architecture, data distribution, and training recipe. The paper's findings are descriptive of the LongCat-Flash architecture trained on the authors' undisclosed data mixture, not predictive of other architectures. If a team is building a model with a different MoE design (different expert count, different routing mechanism, different layer structure), they cannot know whether the critical sparsity threshold will be at ratio 10, 20, or 50. Getting this wrong means either: (a) introducing N-gram Embedding too early, incurring a performance penalty compared to simply adding more experts (the green curve in Figure 2), or (b) introducing it too late, missing the opportunity to gain efficiency. The paper also does not characterize how sensitive the optimal thresholds are to training data quantity — the 300B-token scaling experiments may produce different thresholds than the 11T-token LongCat-Flash-Lite training run, but no intermediate-scale experiments bridge this gap.

What evidence exists in the paper. The scaling curves in Figures 2 and 6 are the primary evidence that thresholds exist and vary with width. But these curves are the output of the experimental methodology, not a predictive model. Figure 7 quantifies how the advantage changes with width and depth, but again only for the specific LongCat-Flash architecture at the tested data scale. The paper does not provide a parametric formula or scaling law that maps (architecture, data size) → (optimal sparsity threshold, optimal embedding proportion). Section 3.1 frames the finding as a "design principle" but does not operationalize it as a decision rule for new architectures.

Mitigation status. The paper does not address this limitation. It does not propose a method for predicting the critical threshold, nor does it discuss the cost of finding the threshold for a new architecture (which would require training multiple models — precisely the resource expenditure the paper aims to optimize). The paper's conclusions are presented as general principles ("embedding scaling could be a promising scaling dimension" — Section 3.1), but the evidence for these principles is architecture-specific. A more actionable contribution would include a characterization of how the threshold depends on measurable architectural properties (expert count, hidden dimension, number of layers, training tokens), but no such characterization is attempted.

---

Limitation 2: Embedding Scaling Provides No Benefit on the Hardest Problems — Capability Ceilings Are Architecture-Dependent

The assumption or constraint. The paper's scaling experiments measure training and validation loss, which captures average-case improvements across the entire token distribution. However, the downstream evaluations of LongCat-Flash-Lite (Tables 1 and 2) reveal a sharp asymmetry: the benefits of N-gram Embedding are concentrated in specific task categories and minimal or negative in others. On the base model benchmarks (Table 1), the largest gains are on BBH (+5.13), DROP (+4.51), and GPQA (+4.29) — all reasoning-heavy tasks. On general knowledge tasks like MMLU, the gain is essentially zero (64.01 vs. 64.81 for Vanilla, a slight deficit). On the chat model benchmarks (Table 2), the pattern is even starker: LongCat-Flash-Lite dominates agentic tool-use and coding tasks by 10–60 point margins, but is consistently second to Qwen3-Next on general domain knowledge (MMLU: 85.52 vs. 89.28; MMLU-Pro: 78.29 vs. 82.93) and mathematical reasoning (MATH500: 96.80 vs. 98.00; AIME24: 72.19 vs. 81.35), with gaps of 1–9 points.

The consequence. The paper's central claim — that embedding scaling is an "orthogonal dimension" for expanding model capacity that "achieves a superior Pareto frontier" — must be qualified by task type. The mechanism by which N-gram Embedding improves performance (enriching token representations with local n-gram context) likely provides asymmetric benefits: tasks that depend on recognizing multi-token patterns (code syntax, API call sequences, command structures) may benefit disproportionately, while tasks that depend on retrieving and applying factual knowledge (MMLU, MMLU-Pro) or performing multi-step logical deduction (AIME) may benefit less because these capabilities depend more on the depth of processing in the transformer body than on the richness of input token representations. A practitioner considering embedding scaling must therefore match the scaling strategy to their expected task distribution: if the deployment scenario is agentic coding and tool use, the case for embedding scaling is very strong; if the scenario is general-purpose QA or mathematical competition problems, the case is weaker and expert scaling might be preferable even at high sparsity.

What evidence exists in the paper. The asymmetry in task-specific gains is visible in Tables 1 and 2 but is not discussed or analyzed by the authors. The paper presents the aggregate pattern (LongCat-Flash-Lite outperforms Vanilla on 8/11 base model benchmarks, dominates agentic tasks) without investigating why some tasks benefit more than others. There is no analysis of whether the tasks where N-gram Embedding helps share common characteristics (e.g., dependence on local syntax, multi-token idioms, structured output formats) or whether the tasks where it does not help share different characteristics (e.g., dependence on long-range reasoning, factual recall). This is a missed opportunity: such an analysis would help practitioners predict whether embedding scaling will benefit their specific use case without needing to train their own model.

Mitigation status. Not addressed. The paper does not acknowledge this task-dependent asymmetry as a limitation or propose methods for predicting which tasks will benefit. The "one model for all tasks" framing in the LongCat-Flash-Lite evaluation obscures the fact that the model's advantages are highly concentrated. A more nuanced presentation would quantify the task-type × architecture interaction and provide guidance on when to prefer embedding scaling vs. expert scaling based on the downstream task profile.

---

Limitation 3: The Inference Efficiency Claims Lack Comparative Baselines and Component Ablations

The assumption or constraint. Section 4 and Section 6.4 argue that embedding scaling improves inference efficiency by reducing MoE activated parameters (reducing memory I/O during decoding) and that this theoretical advantage is realized through speculative decoding, the N-gram Cache, kernel fusion, and PDL. However, the paper provides no direct comparison of LongCat-Flash-Lite's inference performance against LongCat-Flash-Lite-Vanilla under identical hardware and serving configurations. The only inference performance data is Figure 8(b), which shows decoding performance for LongCat-Flash-Lite in isolation, without a baseline. The paper also does not ablate the individual contributions of the N-gram Cache, kernel fusion, PDL, or speculative decoding to the total speedup.

The consequence. The claim that embedding scaling "largely reduce[s] I/O bottlenecks in MoE layers" (Section 4) and that the inference optimizations "ensur[e] that the reduction in active parameters translates directly to lower latency and higher throughput" (Section 4.2) is theoretically motivated but empirically unvalidated. The paper's key architectural argument — that shifting parameters from experts to embeddings reduces inference cost because embedding lookups scale with input tokens while expert loading scales with activated parameters — rests on an uncontested premise (embedding lookups are O(1) per token regardless of table size). But the practical cost of the N-gram Embedding mechanism itself — hashing, sub-table lookups, linear projections, and the N-gram Cache management — might offset the savings from reduced expert I/O, especially at small batch sizes where the expert I/O savings are not realized (as the paper itself notes in Section 4.1). Without a head-to-head comparison, a practitioner cannot know whether LongCat-Flash-Lite is actually faster than LongCat-Flash-Lite-Vanilla at equivalent parameter counts, or whether the inference optimizations are merely compensating for overhead introduced by the N-gram Embedding.

What evidence exists in the paper. Figure 8(a) shows that LongCat-Flash-Lite has fewer activated experts than Vanilla, which is a necessary condition for reduced MoE I/O but not sufficient to demonstrate net speedup. Figure 8(b) shows absolute decoding performance for LongCat-Flash-Lite, but there is no Vanilla curve on the same plot. The paper qualitatively describes the inference optimizations (Section 6.4: kernel fusion, PDL, Eagle3, wide EP, SBO) but provides no ablation — no numbers showing throughput with and without the N-gram Cache, with and without kernel fusion, or with different speculative decoding configurations. The statement "we achieve the exceptional inference performance illustrated in Figure 8(b)" is self-referential: the performance is "exceptional" only if we assume the baseline would be worse, and the paper provides no evidence for this assumption.

Mitigation status. Not addressed. The paper treats the inference efficiency argument as a design rationale rather than an empirical claim to be verified. Section 4 is best read as a description of the inference system architecture rather than as evidence for its superiority. The speculative directions in Section 4.3 (N-gram Embedding based drafting, early rejection) are explicitly marked as work in progress, which is appropriate, but the core claim — that embedding scaling translates to inference speedups — is presented as established when it is not.

---

Limitation 4: The Parameter-Equivalent Baseline Is Weaker Than It Appears — Expert Scaling Continues to Improve Beyond the Tested Ratio Range

The assumption or constraint. The paper's central experimental control is the parameter-equivalent MoE baseline: for each N-gram Embedding model, a MoE model is constructed with identical total parameters by converting the N-gram Embedding parameters into additional experts. The paper then compares loss curves and identifies the intersection point where the N-gram Embedding curve crosses above the MoE baseline, interpreting this as the point where embedding scaling becomes inferior to expert scaling. The critical assumption is that the MoE baseline curve represents the best achievable loss for a given total-to-activated parameter ratio — i.e., that adding more experts is a valid representation of how an MoE model would use additional parameters.

The consequence. The paper's own data suggests that the MoE baseline follows a log-linear relationship (Figure 2: "the MoE scaling curve adheres to a strict log-linear relationship"). This means that expert scaling never saturates in an absolute sense — loss continues to decrease (log-linearly) as more experts are added, even at very high ratios. The "intersection point" where N-gram Embedding becomes inferior is not a point where expert scaling stops working, but a point where the marginal return on expert parameters falls below the marginal return on embedding parameters. However, the paper only tests ratios up to ~20 (at 280M activation), ~50 (at 790M and 1.3B activation). Given the log-linear relationship, the MoE baseline would continue to achieve lower loss at ratios beyond the tested range. The question is not "does N-gram Embedding beat experts?" but "at what ratio does N-gram Embedding beat experts, and is that ratio achievable within practical constraints?"

This matters because the paper frames the intersection point as a limit on embedding scaling (you should not exceed ~50% embedding proportion), but the alternative — continuing to add experts beyond the intersection point — would require even more total parameters (since expert scaling is log-linear, achieving the same loss reduction as the optimal embedding proportion would require exponentially more experts). The paper's framing makes embedding scaling look bounded, but the alternative (scaling experts to the same loss) is also bounded by the practical limits on total parameter count (hardware memory, communication overhead). The intersection point should be interpreted as the optimal allocation for a given total parameter budget, not as a failure of embedding scaling. The paper sometimes conflates these interpretations (e.g., "when a model allocates an excessive proportion of its parameter budget to N-gram Embedding, its performance becomes inferior to that of parameter-equivalent MoE baselines" — Section 3.2.1 — is about allocation optimality, not about embedding scaling failing absolutely).

What evidence exists in the paper. The log-linear nature of the MoE curve is stated explicitly in Section 3.1: "the MoE scaling curve adheres to a strict log-linear relationship." The intersection points in Figures 2 and 6 show the optimal allocation points. The paper does not test whether the MoE baseline would eventually cross back below the N-gram Embedding curve at even higher ratios (which would indicate embedding scaling is globally worse), or whether embedding scaling would maintain its advantage (which would indicate expert scaling is globally worse) — the tested range is insufficient to determine the asymptotic behavior.

Mitigation status. The paper does not discuss this interpretation. The "design principles" in Sections 3.1 and 3.2.1 frame the intersection as a constraint on embedding scaling ("do not exceed ~50% embedding proportion") rather than as the solution to an optimization problem ("for a given total budget, allocate ~50% to embeddings for optimal loss"). Both interpretations are mathematically equivalent, but the optimization framing is more actionable: it tells the practitioner what to do, not just what to avoid.

---

Limitation 5: Generalisation Is Untested — Single Architecture Family, Undisclosed Data, and No Non-Chinese-English Evaluation

The assumption or constraint. All experiments use the LongCat-Flash architecture (Meituan, 2025), a specific MoE design with 14 shortcut layers, 256 FFN experts, 128 zero-experts, and a particular routing mechanism. The scaling experiments use 10-layer variants of this architecture. All training data is described only as covering "both Chinese and English" (Section 3) with no further characterization. All downstream evaluation is on standard English and Chinese benchmarks (Tables 1 and 2). The paper implicitly assumes that the findings — the log-linear MoE scaling curve, the critical sparsity threshold, the width-depth asymmetry — are properties of the n-gram embedding mechanism and the general principles of sparse model scaling, not artifacts of this specific architecture or data distribution.

The consequence. If the findings are architecture-specific, they may not transfer to practitioners using different MoE designs. Concretely:

• Routing mechanism: LongCat-Flash uses 256 experts per MoE module with top-12 selection per token. Different routing strategies (top-2, top-8, expert-choice routing, hash-based routing) would produce different expert utilization patterns, potentially changing the sparsity level at which expert scaling saturates. A model with fewer total experts but higher per-token expert utilization (e.g., top-8 out of 64) might reach the critical threshold at a lower total-to-activated ratio.
• Zero-experts: the paper's architecture includes 128 "zero-experts" per MoE module, a specific design choice whose interaction with embedding scaling is not analyzed. Zero-experts already represent a form of parameter sparsity (some experts are never activated), potentially interacting with the sparsity created by N-gram Embedding in unknown ways.
• Data mixture: the effectiveness of N-gram Embedding depends on the frequency and diversity of n-gram patterns in the training data. A corpus dominated by code (which has highly regular syntax and repeated multi-token patterns) might benefit more from n-gram embeddings than a corpus dominated by free-form natural language (where n-gram patterns are more diffuse). A corpus with many languages would have different n-gram collision characteristics than a bilingual Chinese-English corpus. Without characterizing the data, the paper cannot predict how the findings would change under different data distributions.
• Language coverage: the paper evaluates only on Chinese and English benchmarks. N-gram Embedding's effectiveness for morphologically rich languages (where word boundaries are less meaningful and token-level n-grams may capture different information), low-resource languages (where n-gram embeddings may be poorly trained due to data sparsity), or code-switching scenarios is untested.

What evidence exists in the paper. The paper provides no cross-architecture validation and no data ablation studies. The closest thing to a robustness check is the replication of the scaling pattern across three activation scales (280M, 790M, 1.3B) within the same architecture family, but this tests sensitivity to model size, not to architectural design or data composition.

Mitigation status. The paper does not acknowledge this as a limitation. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" but do not justify this belief. The paper's contribution would be strengthened by testing the key findings (log-linear MoE scaling, critical sparsity threshold, width-depth asymmetry) on at least one alternative MoE architecture (e.g., a standard Mixtral-style top-2 expert model without zero-experts) or on a publicly characterized dataset to establish boundary conditions for generalizability.

---

Limitation 6: The Chat Model Comparisons Cannot Isolate the Causal Effect of N-gram Embedding

The assumption or constraint. The paper's headline result — that LongCat-Flash-Lite "exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains" (Abstract, Section 6.3) — is based on comparisons against external models (Kimi-Linear-48B-A3B, Qwen3-Next-80B-A3B-Instruct, Gemini 2.5 Flash-Lite) that differ from LongCat-Flash-Lite along multiple dimensions: total parameter count (48B vs. 80B vs. 68.5B), activated parameter count (3B vs. 3B vs. 2.9–4.5B), architecture (MoE vs. MoE-MoE hybrid vs. MoE + NE), training data, training recipe, and fine-tuning procedure. The paper attributes LongCat-Flash-Lite's strong performance to the N-gram Embedding, but these confounds make causal attribution impossible.

The consequence. The paper's core narrative — that embedding scaling is responsible for LongCat-Flash-Lite's competitive performance — is not supported by the chat evaluation in Table 2. LongCat-Flash-Lite might succeed on agentic tasks because:

• The LongCat-Flash base architecture (independent of N-gram Embedding) is well-suited to agentic workflows, due to its specific expert configuration, training objective, or sequence length handling.
• The training data recipe, inherited from LongCat-Flash-Chat (Meituan, 2025), might include agentic interaction data that the comparison models lack.
• The SFT procedure might emphasize tool-use capabilities more than the comparison models' fine-tuning.
• The N-gram Embedding might actually be the cause — but the paper provides no evidence to rule out the alternatives.

The only experiment that isolates the causal effect of N-gram Embedding is the LongCat-Flash-Lite vs. LongCat-Flash-Lite-Vanilla comparison in Table 1, which shows modest gains (1–5 points on most benchmarks) and is for the base model only, not the chat model. The Vanilla model is not evaluated on the chat benchmarks in Table 2, so we cannot know whether the dramatic agentic task gains (e.g., Tau2-Telecom: 72.80 for Lite) would also appear in a parameter-equivalent chat model without N-gram Embedding. If the Vanilla chat model achieves, say, 65 on Tau2-Telecom, then N-gram Embedding contributes ~8 points and the architecture/training contributes ~65 points. If the Vanilla model achieves 20, then N-gram Embedding is the dominant factor. Without this comparison, the paper cannot distinguish these scenarios.

What evidence exists in the paper. Table 2 shows the chat benchmark results. Table 1 shows the base model comparison against Vanilla but does not include agentic benchmarks (those benchmarks require tool integration capabilities that base models lack). There is no LongCat-Flash-Lite-Vanilla chat model evaluation anywhere in the paper. The paper's claim that LongCat-Flash-Lite "not only surpasses a parameter-equivalent MoE baseline" (Abstract) is true for the base model (Table 1) but is asserted without evidence for the chat model (Table 2).

Mitigation status. Not addressed. The paper does not acknowledge this confound or discuss the need for a parameter-equivalent chat baseline. This is the most significant gap between the paper's claims and its evidence: the scaling experiments (Sections 3.1–3.5) carefully control for parameter count to isolate the embedding vs. expert effect, but the flagship evaluation (Table 2) abandons this control, making the results uninterpretable as evidence for the paper's central thesis.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a fundamentally new technique — N-gram Embedding was introduced by Clark et al. (2022) and refined by Huang et al. (2025) — but it reframes the conversation around model scaling from an additive to a zero-sum resource allocation problem. Before this work, embedding scaling was treated as an augmentation: something you add to an existing architecture to improve it, evaluated against a baseline without the augmentation. The implicit question was "does adding N-gram Embedding help?" The answer from prior work was yes, and the research agenda was about refining the mechanism.

This paper changes the question to: given a fixed total parameter budget, should those parameters live in expert FFN modules or in N-gram Embedding tables? This is not a semantic shift — it fundamentally changes how a practitioner thinks about architecture design. Under the additive framing, you design your transformer body first (layers, hidden size, expert count), then optionally augment with embeddings. Under the zero-sum framing, the embedding module competes with the transformer body for the parameter budget, and the optimal architecture is the one that allocates parameters to whichever dimension yields the higher marginal return at the current sparsity level. The parameter-equivalent baseline construction (Section 3) — where every N-gram Embedding model is compared against an MoE model with identical total parameters achieved by converting embedding parameters into additional experts — is the methodological innovation that operationalizes this reframing.

The magnitude of the shift is incremental but important. This is not a paradigm shift on the scale of the Chinchilla scaling laws (which changed how the entire field allocates pretraining compute), but it is a significant reframing within the subfield of sparse model design. The paper establishes embedding parameters and expert parameters as competing allocation targets, analogous to how the Chinchilla laws established model parameters and training tokens as competing targets. The paper does not propose a parametric law, but it provides the empirical characterization (Figures 2, 6, 7) that could underpin one. The systematic shift of the intersection point with model width — from ratio ~20 at 280M activation to beyond ratio ~50 at 1.3B activation — is precisely the kind of empirical regularity that parametric scaling laws are built from.

Reconciliation of potential contradictions. The paper implicitly resolves a tension in the literature: why do some studies find that embedding scaling is highly effective (Huang et al., 2025; Pagnoni et al., 2025) while the field's default assumption has been that experts are the primary scaling lever? The answer, from Figures 2 and 6, is that the relative efficacy of embedding scaling depends on where the base model sits on the sparsity spectrum. At low sparsity (ratio < ~12 at 280M activation, the green curve in Figure 2), embedding scaling is worse than expert scaling, which would lead a researcher working in that regime to conclude that embedding scaling is ineffective. At high sparsity (ratio ≥ ~12), embedding scaling is better, which would lead to the opposite conclusion. Both findings are correct locally but appear contradictory globally because they were tested at different operating points. The paper resolves this by characterizing the phase boundary — the critical sparsity threshold — as a function of model width and depth.

Research directions that become more attractive:

• Architecture search over the expert-embedding allocation ratio. Before this paper, architecture search for MoE models focused on expert count, top-k selection, and routing mechanisms. The paper demonstrates that the embedding proportion is a first-class architectural hyperparameter that interacts with width and depth. Automated architecture search (e.g., evolutionary methods, gradient-based NAS) should now include the embedding parameter budget as a tunable dimension, with the scaling trends in Figures 6 and 7 providing a prior for where the optimum is likely to lie.
• Scaling laws that jointly model expert count, embedding proportion, width, and depth. The paper provides enough empirical data (Figures 2, 6, 7) to fit a parametric model predicting loss as a function of (total parameters, activated parameters, embedding proportion, width, depth). Such a law would let practitioners predict the optimal allocation for their specific hardware constraints (which limit total parameters and width/depth) without running expensive scaling experiments. The log-linear MoE curve and the systematic width-dependent shift of the intersection point are the key empirical regularities such a law would need to capture.
• Co-design of embedding scaling with inference systems. The paper's Section 4 argues that embedding-scaled models require inference system adaptations (N-gram Cache, speculative decoding, kernel fusion) to realize their theoretical efficiency. This opens a research agenda on inference-aware architecture design — where architectural choices (like embedding proportion) are optimized not just for loss but for end-to-end throughput under a specific inference deployment configuration.

Research directions that become less urgent:

• Incremental improvements to expert routing mechanisms for high-sparsity regimes. The paper's finding that expert scaling follows a log-linear relationship (diminishing returns at high sparsity) suggests that refining routing algorithms (load-balancing losses, auxiliary losses, token-choice vs. expert-choice routing) will face the same fundamental saturation — the marginal expert simply sees too few training tokens to learn effectively. This shifts attention toward complementing experts with orthogonal parameter dimensions (like embeddings) rather than trying to squeeze more efficiency from expert scaling alone.
• Standalone embedding scaling without comparative baselines. The paper's methodological contribution — the parameter-equivalent baseline — makes it harder for future embedding scaling papers to claim success by comparing against a weaker baseline. A paper that proposes a new embedding scaling technique but only compares against a dense model or a low-sparsity MoE (without converting embedding parameters to experts at equivalent total count) will be viewed as methodologically incomplete.

Follow-Up Research This Work Enables

1. Train a difficulty estimator for the expert-embedding allocation decision. The paper's most immediate practical gap is that the optimal allocation ratio (when to introduce N-gram Embedding, what proportion of parameters to allocate) must be determined empirically through expensive scaling experiments for each new architecture. A direct follow-up would train a predictor model that takes architectural specifications as input (total parameters, activated parameters, hidden dimension, number of layers, expert count, training token budget) and outputs the predicted optimal embedding proportion and the expected loss reduction. The training data would come from running the paper's comparative scaling methodology (Figures 2, 6) across a diverse range of architectures — varying not just scale but also expert count, routing mechanism (top-2, top-8, top-12), and layer structure (pre-norm vs. post-norm, parallel vs. sequential attention/FFN). The paper's existing data (three activation scales × one architecture family) is insufficient for training such a predictor, but it provides the template for generating the necessary training data. A strong result would show that the predictor generalizes to unseen architectures within the same family and transfers partially to different MoE designs (e.g., Mixtral-style top-2 routing).

2. Test Per-Layer N-gram Embedding at depth to confirm or refute the residual attenuation hypothesis. The paper proposes that embedding scaling's advantage shrinks with depth because the input embedding signal gets diluted across many residual additions (Section 3.3.2, Figure 7b). The natural test of this hypothesis is to evaluate whether Per-Layer N-gram Embedding (PLNE, Section 5.2) — which injects fresh n-gram information at every layer — recovers the lost advantage in deep models. The paper tested PLNE only at 10 layers (Figure 9), where the benefit over input-only N-gram Embedding was marginal. If the attenuation hypothesis is correct, PLNE should show increasing advantage over input-only NE as depth increases, with the gap being largest at 40+ layers. A strong experiment would train input-only NE and PLNE models at depths of 20, 40, and 60 shortcut layers, holding total parameters and embedding proportion constant, and measure whether PLNE's relative improvement grows with depth. If PLNE provides no benefit even at 60 layers, the attenuation hypothesis is refuted (or the benefit is not realized by the specific PLNE design), and the depth-related degradation must have a different cause — possibly that deeper models simply benefit more from distributed processing capacity (experts at every layer) than from enriched input representations, independent of signal propagation.

3. Characterize the task-type × architecture interaction for embedding-scaled models. The paper's downstream evaluation (Tables 1 and 2) reveals a sharp but unexplained asymmetry: LongCat-Flash-Lite dominates agentic tasks (Tau2-Telecom: +50–60 points over baselines) but is consistently second to Qwen3-Next on general knowledge and mathematical reasoning (MMLU: −3.76, AIME24: −9.16). A systematic follow-up would categorize a broad set of benchmarks along theoretically motivated dimensions — (a) dependence on local multi-token patterns vs. long-range reasoning, (b) structured output vs. free-form generation, (c) procedural vs. declarative knowledge — and measure whether the N-gram Embedding advantage correlates with these dimensions. The hypothesis is that N-gram Embedding helps most on tasks where recognizing and reproducing multi-token compositional patterns (API call sequences, command syntax, code idioms) is the bottleneck, and helps least on tasks where the bottleneck is multi-step logical inference or factual retrieval (which depend on the transformer body's processing depth). A strong result would show a clear monotonic relationship between a task's "n-gram dependence score" (measurable via ablation of the N-gram Embedding module at inference time) and the performance gain over the Vanilla model. This would give practitioners a decision rule: if your deployment task has high n-gram dependence, prefer embedding scaling; if it has low n-gram dependence, prefer expert scaling or a balanced allocation.

4. Measure the inference speedup of LongCat-Flash-Lite vs. LongCat-Flash-Lite-Vanilla under matched conditions. The paper's Section 4 argues that embedding scaling reduces MoE I/O bottlenecks and improves inference efficiency, but provides no direct comparison between Lite and Vanilla inference performance. The most important missing ablation is a head-to-head measurement of decoding throughput and latency for both models on identical hardware (8×H800-80G), identical serving framework, identical speculative decoding configuration (Eagle3, 3-step), and identical batch sizes. The comparison should separately measure: (a) prefill throughput (where the embedding lookup cost is amortized over all input tokens and the expert I/O savings from reduced activated parameters should dominate), (b) decode latency per token (where the N-gram Cache overhead competes with the expert I/O savings), and (c) the effect of batch size on the relative advantage (the paper predicts that embedding scaling's I/O benefit only manifests at large batch sizes, Figure 8a). A strong result would show that Lite achieves, say, 1.3× higher throughput than Vanilla at batch sizes ≥ 32, with the gap explained by reduced expert parameter loading. A null result — Lite being no faster or even slower — would seriously undermine the paper's efficiency argument and suggest that the N-gram Embedding overhead negates the expert I/O savings, making the architectural choice purely about performance-per-parameter rather than performance-per-second.

5. Test the interaction between N-gram Embedding and training data composition. The paper's training data is described only as "covering both Chinese and English" (Section 3) with no further characterization. The effectiveness of N-gram Embedding should depend on the frequency and diversity of n-gram patterns in the training corpus. A controlled experiment would train identically architected N-gram Embedding models on corpora with systematically varied properties: (a) high vs. low token repetition rate (code vs. natural language prose), (b) high vs. low n-gram diversity (structured data vs. free-form text), (c) monolingual vs. multilingual (to test whether hash collisions across languages degrade performance), and (d) different vocabulary sizes (since the hash collision analysis in Figure 3b depends on vocabulary size). The dependent variables would be: the optimal embedding proportion (does it shift with data properties?), the absolute loss reduction from N-gram Embedding (does it vary by data type?), and the collision rate × downstream performance relationship (do higher collision rates hurt more on some data types than others?). This experiment addresses the paper's most significant generalizability concern: without data characterization, we cannot predict whether the findings transfer to other corpora. A strong result would show that while the absolute benefit varies, the qualitative pattern — existence of a critical sparsity threshold, diminishing returns at excessive embedding proportions, width-amplified advantage — is robust across diverse data distributions.

6. Evaluate N-gram Embedding scaling against alternative embedding scaling methods at a matched large scale. The paper compares N-gram Embedding against Per-Layer Embedding (PLE) and Per-Layer N-gram Embedding (PLNE) only at the 790M activation scale (Figure 9), finding that PLE underperforms and PLNE provides marginal gains. LongCat-Flash-Lite uses only input-level N-gram Embedding. However, the paper does not compare against other embedding scaling approaches at the 68.5B scale: Engram (Cheng et al., 2026, concurrent work), Byte Latent Transformer-style patching (Pagnoni et al., 2025), or simple vocabulary expansion (Tao et al., 2024). A comparative study at the 1B+ activated parameter scale, with all methods evaluated under the parameter-equivalent baseline framework (converting non-expert parameters to additional experts), would establish which embedding scaling method provides the best Pareto frontier. The specific comparison should include: (a) training loss curves and downstream benchmarks for each method at matched total parameter counts, (b) inference throughput measurements for each method (since different embedding scaling techniques have different lookup costs), and (c) analysis of whether the methods have complementary strengths that could be combined (e.g., N-gram Embedding at the input layer + PLE in later layers, or N-gram Embedding with Engram's learned hash functions). This would transform the paper's finding from "N-gram Embedding beats expert scaling" to "among embedding scaling methods, N-gram Embedding provides the best efficiency at scale" — or reveal that a different method dominates, redirecting the research agenda.

Practical Applications and Downstream Use Cases

On-device and edge deployment with constrained memory bandwidth. The paper's architectural insight — that embedding parameters impose negligible inference cost because only the embeddings for tokens actually in the sequence are accessed — makes embedding-scaled models particularly attractive for deployment scenarios where memory bandwidth is the primary bottleneck. In edge devices (phones, laptops, embedded systems), the cost of loading expert parameters from DRAM into compute units dominates inference latency. An embedding-scaled model with, say, 30B embedding parameters and 3B activated expert parameters would require loading only the 3B activated parameters per token during decoding (plus the embedding lookup for the current token's n-grams), whereas a parameter-equivalent expert-scaled model would load, say, 4.5B activated parameters. The 1.5B parameter reduction in per-token memory traffic translates directly to lower latency on bandwidth-constrained devices. The paper's LongCat-Flash-Lite achieves 2.9B–4.5B activated parameters depending on context (Section 6.1), placing it in the regime where on-device deployment is plausible with quantization. The N-gram Cache design (Section 4.2) is specifically engineered to minimize the embedding lookup overhead that would otherwise erode this advantage. A deployment team building an on-device coding assistant could use the paper's findings to justify allocating a larger fraction of the parameter budget to embeddings rather than experts, confident that the embedding parameters will not create a latency bottleneck.

Cost-efficient batch inference for agentic coding and tool-use workloads. The paper's most dramatic benchmark results are on agentic tasks: Tau2-Telecom at 72.8 (vs. 13.2–21.93 for baselines), SWE-Bench at 54.4 (vs. 32.8–41.3), TerminalBench at 33.75 (vs. 15.19–20.0). For organizations running large-scale batch inference on agentic workflows — automated code repair, pull request review, test generation, or API orchestration — these numbers suggest that an embedding-scaled architecture provides substantially better task completion rates than expert-scaled architectures of similar total parameter count. The practical implication is a shift in architecture selection criteria: if the workload is agentic (multi-step, tool-mediated, code-heavy), models with high embedding proportions should be preferred over models that invest equivalent parameters in additional experts. The paper's finding that wider models amplify the embedding advantage (Section 3.3.1) provides concrete guidance: for an agentic workload, use the widest architecture that fits within the hardware budget, and allocate a larger fraction of parameters to embeddings than you would for a general-purpose model.

Training data generation for self-improvement loops in structured domains. When using LLMs to generate training data for themselves (e.g., generating synthetic code-repair trajectories for fine-tuning a coding model), the diversity and correctness of generated outputs matter enormously. The paper's base model results (Table 1) show that N-gram Embedding provides the largest gains on structured reasoning and coding tasks: BBH (+5.13), DROP (+4.51), GPQA (+4.29), HumanEval+ (+2.44), BigCodeBench (+2.63). This suggests that embedding-scaled models produce higher-quality outputs on these task types than parameter-equivalent expert-scaled models, making them better data generators for self-improvement pipelines in code and reasoning domains. A team using rejection sampling fine-tuning (where the model generates many candidate solutions and only correct ones are kept for training) would get a higher yield of correct solutions per generation from an embedding-scaled model, reducing the total inference cost of the data generation phase. The 4× efficiency gain that the paper claims over best-of-N baselines (not in this paper, but analogous in structure: "same performance at lower parameter count") translates to fewer GPU-hours needed to generate a target quantity of training data.

Architecture selection for latency-sensitive interactive applications. The paper's Section 4.3 proposes using N-gram Embedding as an ultra-fast draft model for speculative decoding, by attaching a lightweight linear projection to the N-gram Embedding output to predict the next token without running any transformer layers. If this direction succeeds, it would enable a deployment architecture where the embedding-scaled model serves as its own draft model — the N-gram Embedding output generates candidate tokens with near-zero latency, and the full transformer body verifies them in parallel. This would be particularly valuable for interactive applications (chat, code completion, real-time translation) where per-token latency directly affects user experience. The paper's deployment of LongCat-Flash-Lite with Eagle3 speculative decoding (Section 6.4) is a step toward this vision, but the draft model is separate (Eagle3) rather than derived from the N-gram Embedding. A team building a real-time coding assistant could invest in developing the N-gram Embedding drafting approach, creating a model that is both parameter-efficient (due to embedding scaling) and latency-optimized (due to self-drafting).

When to Prefer This Method

The paper articulates a clear tradeoff between scaling embeddings and scaling experts, with specific conditions that determine which dimension dominates. The tradeoff is not universal — it depends on three interacting factors — and the paper provides empirical thresholds for each:

• Prefer embedding scaling (N-gram Embedding) over expert scaling when:

  • The base model is already operating at high sparsity (total-to-activated parameter ratio ≥ ~12 at 280M activation scale, shifting rightward to ≥ ~20+ for wider models, per Figures 2 and 6). Below this threshold, expert scaling still provides higher marginal returns per parameter, and diverting parameters to embeddings is strictly worse (green curve in Figure 2).
  • The architecture is wide rather than deep (prefer hidden dimension expansion over layer count expansion, per Figure 7). At depths of 20+ shortcut layers, the embedding advantage contracts sharply due to residual signal attenuation (Figure 7b). At 40 shortcut layers, the advantage approaches zero. For models with 10–15 shortcut layers (like LongCat-Flash-Lite at 14), embedding scaling remains effective.
  • The deployment workload is agentic, code-heavy, or tool-use-oriented (where the paper shows 10–60 point gains over parameter-equivalent baselines, Table 2) rather than general knowledge QA or mathematical competition problems (where gains are 0–5 points, Tables 1 and 2).
  • The total embedding parameter proportion does not exceed ~50% of total parameters at moderate width (280M activation scale, Figure 2 intersection point), though this threshold increases with model width — at 1.3B activation with 10 layers, embedding proportions up to ~76% still outperform expert scaling (Figure 6b, ratio ~50 with base ratio ~12).

• Prefer expert scaling over embedding scaling when:

  • The base model operates at low sparsity (total-to-activated ratio < ~12 at 280M activation scale), where expert scaling still provides high marginal returns and the log-linear curve has not yet flattened significantly.
  • The architecture is deep (40+ shortcut layers), where the embedding signal is diluted across many residual additions and the advantage over expert scaling is minimal (Figure 7b).
  • The deployment workload emphasizes factual knowledge retrieval (MMLU, MMLU-Pro) or advanced mathematical reasoning (AIME), where LongCat-Flash-Lite trails Qwen3-Next by 1–9 points despite its parameter-count advantages in other domains (Table 2).
  • Inference is latency-critical at small batch sizes, where the reduced expert I/O from embedding scaling cannot be realized because the GPU is not memory-bandwidth-saturated (Section 4.1, Figure 8a), and the N-gram Embedding lookup overhead may dominate.

• Prefer a balanced allocation (roughly 50% of non-base parameters to embeddings) when:

  • The architecture is moderately wide and shallow (e.g., 10–20 shortcut layers, hidden dimension typical of 1B–3B activated parameter models), matching the regime where the paper's scaling experiments were conducted.
  • The task distribution is mixed, with both agentic/code tasks (where embeddings help) and knowledge/reasoning tasks (where experts help), requiring a compromise allocation that performs adequately across categories.
  • Total parameter budget is tightly constrained, making it important to operate near the Pareto-optimal allocation rather than over-investing in either dimension (since both embedding scaling and expert scaling exhibit diminishing returns beyond their respective optimal proportions).

These decision rules are derived directly from the paper's Figures 2, 6, and 7 and Tables 1 and 2. They are specific to the LongCat-Flash architecture trained on Chinese-English data, and the numerical thresholds (ratio ~12, 50% proportion, 40 layers) should be treated as architecture-dependent estimates rather than universal constants. The paper does not provide a parametric model for predicting these thresholds on new architectures, so practitioners should treat them as starting points to be validated through small-scale scaling experiments on their own architecture and data.