Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models

🎯 Pitch

Deep–thin small language models, long assumed optimal for accuracy per parameter, are decisively not latency-optimal—a model with 2× the parameters can run 1.9× faster. Nemotron-Flash uses evolutionary search to blend attention, Mamba2, and DeltaNet into hybrid architectures that smash the accuracy–latency frontier, plus a weight normalization trick that squeezes out extra convergence gains.

1. Executive Summary

This paper introduces Nemotron-Flash, a new family of hybrid small language models (SLMs) that systematically targets latency-optimal design rather than the parameter-efficient design that dominates prior SLM work. Using a combination of controlled architectural exploration and an evolutionary search framework across emerging efficient attention alternatives (DeltaNet, Mamba2, and sliding window attention), the authors identify key determinants of real-device latency—depth–width ratios and operator choices—and develop a latency-optimal SLM design and training framework that includes augmented scaling laws (relating loss to both depth and width rather than just total parameters), automated evolutionary search over hybrid operator combinations (interleaving full attention, Mamba2, and DeltaNet to exploit complementary memory mechanisms), and a weight normalization technique (projecting weight matrices onto unit-norm spheres to increase effective learning rates in late training). The resulting Nemotron-Flash-3B achieves +5.5% higher average accuracy over 16 tasks (commonsense reasoning, math, coding, and recall), 1.3× lower decoding latency, and 18.7× higher throughput compared to Qwen3-1.7B on an NVIDIA H100, establishing that deep–thin architectures—despite their parameter efficiency—are not latency-optimal, and that real-device speed requires deliberate co-optimization of depth, width, and operator composition, with the optimal depth–width ratio generally increasing with the target latency budget.

2. Context and Motivation

The Core Problem: Parameter Efficiency ≠ Latency Efficiency

The fundamental gap this paper addresses is the disconnect between how small language models (SLMs) are designed and how they actually perform on real hardware. Most existing SLM research optimizes for parameter efficiency — maximizing accuracy for a given parameter count. The implicit assumption is that fewer parameters translate directly to faster inference. This paper demonstrates that this assumption is often wrong: a model optimized for parameter count can be slower on real devices than one with more parameters but better architectural choices.

The paper states this explicitly in Section 1:

"parameter-efficient models do not necessarily yield proportional latency reductions, especially on hardware AI accelerators like GPUs and TPUs."

This matters because SLMs are specifically intended for deployment scenarios where models run on-device or on resource-constrained hardware — environments where every millisecond of latency matters. If parameter-efficient design doesn't guarantee latency-efficient design, then the entire SLM development paradigm is optimizing the wrong metric. The consequence: models like MobileLLM [liu2024mobilellm] and SmolLM [allal2025smollm2smolgoesbig] achieve impressive accuracy-per-parameter numbers but leave substantial latency improvements on the table because their deep–thin architectures — many layers with narrow hidden dimensions — interact poorly with GPU hardware characteristics.

Why This Matters: The Hardware Reality

To understand why deep–thin models can be slow despite having few parameters, we need to examine what actually determines latency on modern AI accelerators. The paper identifies two central architectural factors (Section 3):

1. Depth–width ratio and small-batch latency. Modern GPUs excel at parallel computation but suffer when operations are too small to saturate compute units. Each layer introduces a fixed overhead — kernel launch time, memory transfers, synchronization barriers — that is independent of the layer's width. In a deep–thin model with 30 narrow layers, this overhead is paid 30 times, and each individual layer's computation may be too small to fully utilize the GPU's parallel cores. A wider–shallower model (fewer layers, each wider) amortizes the per-layer overhead better and keeps the GPU busier, even if it has more total parameters.

This is why Figure 2(b) shows that for a fixed latency budget of 3 seconds, a depth-12 model outperforms both depth-6 and depth-30 models. The depth-30 variant pays too much per-layer overhead; the depth-6 variant doesn't have enough representational capacity. The optimal point balances these forces — and that point varies with the latency budget.

2. Operator choice and throughput/latency. Standard full attention scales quadratically with sequence length in both computation and memory (KV cache), making it the primary bottleneck for long-context, large-batch throughput. But replacing attention with sub-quadratic alternatives (Mamba, DeltaNet, GLA, etc.) is not a simple win — these operators trade off recall capability and in-context learning performance for efficiency. The paper cites prior work showing that linear attention mechanisms "exhibit limited recall capabilities" [waleffe2024empirical] and "underperform standard attention mechanisms on in-context learning tasks" [park2024can]. The challenge is therefore not just finding a single efficient operator, but determining the right combination of operators — full attention for tasks requiring accurate retrieval, efficient linear attention elsewhere — such that the hybrid model achieves the best accuracy–latency trade-off.

The practical stakes are enormous. As the paper shows in Table 6, a well-designed SLM (Nemotron-Flash-3B) can achieve 18.7× higher throughput and 1.3× lower latency than a comparably capable model (Qwen3-1.7B) while also being more accurate. For a deployment serving millions of queries, this translates directly to cost reduction and user experience improvement. For edge devices with strict power and thermal constraints, it can mean the difference between a model that runs locally and one that requires cloud offloading.

Prior Approaches and Their Shortcomings

Single-operator models with handcrafted architectures. The dominant approach to SLM design has been to scale down LLM architectures (Llama, Qwen, etc.) by reducing layers and hidden dimensions, often with trial-and-error exploration. Models like MobileLLM [liu2024mobilellm], TinyLlama [zhang2024tinyllama], and PanGu-π Pro [tang2024rethinking] follow this paradigm. The key shortcoming is that these models target parameter counts, not latency. MobileLLM specifically recommends deep–thin configurations based on accuracy-per-parameter experiments — the paper acknowledges this prior finding (Section 3.1: "deep-thin models generally achieve better task accuracy than wide-shallow ones under the same parameter budget") but then demonstrates that this finding reverses when the metric shifts from parameters to latency on real hardware.

A secondary issue is the reliance on empirical trial-and-error rather than principled methodologies. The paper states (Section 1):

"current SLM development processes rely heavily on empirical trial-and-error rather than systematic and principled methodologies"

This makes the design process expensive, non-reproducible, and unlikely to discover non-obvious architectural configurations.

Hybrid models with manual operator mixing. Recognizing the limitations of pure attention or pure linear attention, several works have proposed hybrid models that combine both. Jamba [lieber2024jamba], Samba [ren2024samba], Zamba [glorioso2024zamba], and Griffin [de2024griffin] sequentially stack Mamba and attention layers. Hymba [dong2024hymba] proposes a hybrid-head structure with parallel attention and Mamba branches. These works demonstrate the potential of hybrid architectures but share a critical limitation: the operator combinations are designed manually. As new efficient attention operators proliferate (Mamba2, DeltaNet, Gated DeltaNet, GLA, RWKV7, JetBlock), the combinatorial space of possible hybrid configurations explodes. Manual exploration becomes infeasible, and there is no guarantee that existing handcrafted combinations are near-optimal.

The paper is explicit about this gap (Section 1):

"the potential synergies of combining these operators in hybrid models have not been thoroughly explored, leading to manual, heuristic-driven architecture decisions that are increasingly costly and less scalable."

Scaling laws treat parameters as a single dimension. Existing LM scaling laws (Hoffmann et al., 2022; Hu et al., 2024 from MiniCPM) parameterize loss as a function of model size $P$ and data size $N$ : $\mathcal{L}(P,N) = \mathcal{L}_0 + C_1 P^{-\alpha} + C_2 N^{-\gamma}$ . This treats all parameters as interchangeable — a 1B-parameter model is a 1B-parameter model regardless of whether it has 12 wide layers or 30 narrow ones. But as the depth–width experiments show, these two configurations have very different latency profiles. The existing scaling law framework cannot guide the choice of depth–width ratio, because it collapses them into a single $P$ variable. The paper's augmented scaling law in Equation 1 — $\mathcal{L}(D,W,N) = \mathcal{L}_0 + a D^{-\alpha} + b W^{-\beta} + c N^{-\gamma}$ — addresses this directly by decoupling depth and width as independent contributors to loss.

Training optimizations are architecture-agnostic. Prior work on improving SLM training — learning rate schedules, data mixing, distillation — typically treats the model architecture as fixed. The paper's weight normalization technique (Section 3.3) is motivated by an observation about weight matrix structure during training: weight matrices develop large magnitudes along certain dimensions, leading to small relative updates under standard gradient descent in late training. This structural insight is general across model families (Figure 7 shows consistent gains for Llama, DeltaNet, Mamba2, and hybrid architectures), suggesting that training methodology can and should be co-designed with architecture for latency-optimal SLMs.

How This Paper Positions Itself

The paper positions itself not as proposing a single new model or architecture, but as establishing a framework for latency-optimal SLM design and training that is generalizable beyond the specific models presented. This framework has four components, each addressing a gap in prior work:

Component 1: Depth–width co-optimization with augmented scaling laws (Section 3.1). Rather than assuming deep–thin is always better (the prior consensus), the paper systematically explores the accuracy–latency trade-off across the depth–width plane and provides a principled method — fitting a scaling law with separate depth and width terms, then profiling configurations against a latency constraint — to identify the sweet-spot ratio for any deployment setting. This directly addresses the observation that optimal depth increases with latency budget (Figure 2 and the associated finding: "the optimal depth-width ratio generally increases with the target latency budget"), providing a concrete guideline that prior work lacked.

Component 2: Automated evolutionary search over hybrid operator combinations (Section 3.2). Rather than manually designing hybrid architectures (the prior approach), the paper demonstrates that an aging evolution search algorithm — using short-training perplexity as a proxy with 88.8% Spearman correlation to final performance — can efficiently navigate the combinatorial space of operator types, ratios, and FFN configurations. The search discovers non-obvious architectures (e.g., DeltaNet-FFN-Mamba2-FFN and Attention-FFN-Mamba2-FFN interleaved blocks in Table 2) that outperform both pure models and naively scaled alternatives. Crucially, the search is metric-aware: architectures optimized for decoding latency differ structurally from those optimized for parameter count (more attention layers in the latter, deeper models in the latter, confirming that the optimization target matters).

Component 3: Weight normalization for effective late-stage training (Section 3.3). Recognizing that SLM potential is limited by training convergence, the paper identifies that large weight magnitudes in late training reduce the effective learning rate (since $|\Delta W| / |W|$ shrinks even if gradient magnitudes remain comparable). Weight normalization — projecting weight rows or columns onto a unit-norm sphere after each step — enforces angular updates, maintaining larger relative weight changes throughout training. The paper explicitly positions this as a simplified variant of nGPT [loshchilov2024ngpt], showing that weight normalization alone achieves comparable accuracy to full nGPT while avoiding the >20% training time overhead from nGPT's additional activation normalizations (Appendix D.1, Table 12).

Component 4: Benchmarking against the accuracy–latency–throughput frontier (Section 4). Rather than comparing only on accuracy (the typical SLM evaluation), the paper evaluates all models on a three-way trade-off: average accuracy across 16 diverse tasks, decoding latency (batch size 1, 8k tokens, CUDA Graph), and throughput (maximum batch size without OOM, 32k input length). This multi-dimensional evaluation reveals that existing SOTA SLMs like Qwen3-1.7B achieve strong accuracy but poor latency/throughput because their architectures were not designed for real-device efficiency. The paper's Nemotron-Flash models advance the Pareto frontier along all three dimensions simultaneously (Figure 1), demonstrating that the framework produces genuinely better models, not just models tuned to a single metric at the expense of others.

A critical nuance: the paper does not claim that parameter efficiency is irrelevant. Deep–thin models do achieve better accuracy per parameter (Figure 2a). The contribution is showing that this advantage does not carry over to the accuracy–latency frontier, and that a different optimization target — latency — leads to different architectural decisions. The paper thus provides a complementary design philosophy to the parameter-efficient SLM literature, not a replacement. For applications where total model size (memory footprint, download size) is the binding constraint, parameter-efficient design remains appropriate. For applications where inference speed is the binding constraint, latency-optimal design is the correct paradigm. The paper gives practitioners the tools and understanding to choose appropriately for their use case.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a framework for building small language models (SLMs) that are fast on real hardware, not just parameter-efficient on paper. The system takes a target latency budget and a library of efficient neural network operators, then produces a complete SLM architecture — specifying depth, width, and which operator goes in each layer — along with a training recipe that maximizes downstream accuracy under that latency constraint. The "shape" of the solution is a three-stage pipeline: first, determine the optimal depth–width ratio for the target latency using augmented scaling laws; second, search over hybrid combinations of efficient attention operators using an evolutionary algorithm with short-training proxies; third, apply a weight normalization technique during training that improves convergence by enforcing angular rather than radial weight updates.

3.2 Big-Picture Architecture (Diagram in Words)

The Nemotron-Flash framework has five major components, each addressing a specific design or training decision:

Depth–Width Scaling Law Fitter — takes profiled latency measurements from a range of (depth, width) configurations and fits a function $\mathcal{L}(D, W, N)$ that predicts language modeling loss from depth $D$ , width $W$ , and data size $N$ . Its responsibility is to identify, for a given latency budget, which $(D, W)$ pair minimizes predicted loss.
Operator Library — a collection of token-mixing operators (full attention, Mamba2, DeltaNet, Gated DeltaNet, GLA, RWKV7, sliding window attention) each with known accuracy and latency profiles. Its responsibility is to provide the building blocks that the search algorithm will combine.
Evolutionary Search Engine — an aging-evolution algorithm that explores the combinatorial space of hybrid architectures (which operators, in what ratios, with how many FFNs, stacked in what order across early/middle/late stages). It uses short-training perplexity (10B tokens) as a proxy for final performance, validated by an 88.8% Spearman rank correlation with full-training results.
Weight Normalization Module — a training-time operation that projects each weight matrix onto a unit-norm sphere after every optimizer step, with the projection dimension chosen based on whether the matrix operates on hidden features (row-wise normalization) or produces additive outputs (column-wise normalization). This effectively increases the relative learning rate in late training when weight magnitudes would otherwise grow large.
Final Model Trainer — takes the searched architecture and the weight normalization recipe, then trains the model at scale (4.5T tokens for the final Nemotron-Flash models) with a cosine learning rate schedule, a two-stage data curriculum (general corpus → high-quality math/code/reasoning data), and meta tokens for cache initialization.

Information flows as follows: a target latency budget enters the system → the depth–width scaling law produces a $(D, W)$ recommendation → the evolutionary search engine, constrained by that depth, searches for the optimal operator sequence → the resulting architecture is trained with weight normalization and meta tokens → the trained model is deployed with TensorRT-LLM's AutoDeploy kernels for attention, FlashLinearAttention for linear attention layers, and CUDA Graph wrapping the entire model.

3.3 Roadmap for the Deep Dive

First, the depth–width scaling law (Section 3.1), because it determines the fundamental structural trade-off — deep–thin versus wide–shallow — that constrains all subsequent design choices. Understanding why deeper isn't always faster is the prerequisite for everything else.
Second, the operator exploration and hybrid search (Section 3.2), because this is where the layer-level architecture is determined. We'll examine how individual operators compare on the accuracy–latency frontier, why certain operator pairs are complementary, and how the evolutionary search automates the combination process.
Third, the weight normalization technique (Section 3.3), because training methodology is co-designed with architecture. We'll examine the structural patterns in weight matrices that motivate normalization, the two cases for projection direction, and the connection to effective learning rates.
Fourth, the meta tokens technique (Section 3.4), as a lightweight add-on that improves both attention and linear attention mechanisms.
Fifth, the final model assembly and training protocol (Section 4), which integrates all components into the Nemotron-Flash-1B and 3B models and describes the data curriculum, optimizer settings, and deployment configuration.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical design paper whose core idea is that real-device latency for small language models is determined by depth–width ratio and operator choice — not just parameter count — and that these factors can be systematically optimized through augmented scaling laws, evolutionary architecture search, and training-time weight normalization.

The Augmented Depth–Width Scaling Law

Motivation. Prior scaling laws for language models (Hoffmann et al., 2022; Hu et al., 2024 from MiniCPM) express loss as a function of total parameter count $P$ and training data size $N$ :

$\mathcal{L}(P, N) = \mathcal{L}_0 + C_1 \cdot P^{-\alpha} + C_2 \cdot N^{-\gamma}$

This formulation treats all parameters as interchangeable — a 1B-parameter model with 6 wide layers is, for purposes of the scaling law, identical to a 1B-parameter model with 30 narrow layers. But as the profiling experiments in Figure 2 demonstrate, these two configurations have very different latency profiles. A scaling law that only depends on $P$ cannot guide depth–width trade-offs, which is precisely what's needed for latency-optimal design.

The augmented formulation. The paper decouples total parameters $P$ into two independent factors: model depth $D$ (number of transformer blocks) and model width $W$ (hidden dimension). The augmented scaling law is:

$\mathcal{L}(D, W, N) = \mathcal{L}_0 + a \cdot D^{-\alpha} + b \cdot W^{-\beta} + c \cdot N^{-\gamma}$

where $\mathcal{L}_0$ is the irreducible loss (the minimum achievable loss with infinite depth, width, and data), $D$ is the number of blocks or layers in the model, $W$ is the hidden size (width of the residual stream), $N$ is the number of training tokens, $a$ , $b$ , and $c$ are positive coefficients controlling the contribution of each dimension to loss reduction, and $\alpha$ , $\beta$ , and $\gamma$ are positive exponents that control the diminishing returns from scaling each dimension — larger exponents mean faster saturation (additional increases in that dimension yield smaller improvements).

What it computes: Given a specific depth $D$ , width $W$ , and data size $N$ , the function outputs a predicted language modeling loss (measured as perplexity). The three power-law terms $a D^{-\alpha}$ , $b W^{-\beta}$ , and $c N^{-\gamma}$ each represent the reducible loss that can be eliminated by scaling the corresponding dimension. As any dimension grows, its term shrinks toward zero, and the total loss approaches $\mathcal{L}_0$ .

Why this form: The additive, decoupled structure assumes that the benefits of increasing depth, increasing width, and increasing data are independent and additive in log-loss space. This is an approximation — in reality, depth and width interact (a wider model may need fewer layers to achieve the same expressivity, and vice versa) — but the paper validates that it extrapolates reasonably well to unseen configurations (within 5.3% of ground-truth perplexity, Figure 3). The power-law form is inherited from prior scaling law literature, where it has been empirically validated over many orders of magnitude. The key innovation is simply the decomposition of $P$ into $D$ and $W$ , which was not present in prior work.

Using the scaling law in practice. Since the data size term $c N^{-\gamma}$ is additive and decoupled from depth and width, the paper notes that "we can study the effect of depth and width on LM loss under a fixed data size, i.e., by neglecting the data size term" (Section 3.1). The practical procedure is:

Profile a range of $(D, W)$ configurations on the target hardware to measure their latency (e.g., decoding time for a fixed number of tokens at batch size 1).
Train models at a subset of these configurations on a fixed data budget to measure their perplexity.
Fit the parameters $a$ , $b$ , $\alpha$ , $\beta$ (and optionally $\mathcal{L}_0$ ) using these measured $(D, W, \text{PPL})$ triplets.
For a given target latency budget, find the $(D, W)$ pair that satisfies the latency constraint while achieving the lowest predicted loss under the fitted function.

Validation of extrapolation. Figure 3 shows the fitting and validation results. The scaling law is fit on a subset of configurations (smaller depth/width values) and then used to predict perplexity for larger depth/width settings that were not seen during fitting. The predicted values stay within 5.3% of the actual perplexity, demonstrating that the functional form generalizes beyond the interpolation regime. This is critical because the practical use case involves profiling smaller/faster models (cheap to measure) and extrapolating to larger configurations that meet the latency budget.

Key empirical finding from the depth–width exploration (Figure 2). Before introducing the scaling law, the paper conducts a systematic sweep over five depth settings (6, 12, 18, 24, 30 blocks) with varying widths, training Llama-style models on 100B tokens from Smollm-corpus. The results reveal three patterns:

Observation (1): Deeper models achieve better accuracy for the same parameter count (Figure 2a). This replicates the prior finding from MobileLLM and PanGu-π Pro — deep–thin is parameter-efficient. However, the benefit gradually saturates; going from depth 24 to 30 provides smaller gains than going from depth 6 to 12.
Observation (2): For latency-constrained regimes (Figure 2b), the advantage of deep–thin models may not hold. The paper gives a concrete example: "when the latency budget is 3 seconds, a depth of 12 achieves the best accuracy among the evaluated settings." This is the central counterintuitive finding — the model with the best accuracy per parameter (depth 30) is not the model with the best accuracy per millisecond. The reason is hardware overhead: each layer adds fixed costs (kernel launches, memory transfers) that are independent of layer width. Many narrow layers = many fixed costs = worse latency.
Observation (3): The optimal depth increases with the latency budget. When the latency budget is tight (e.g., 1 second), shallow models (depth 6) dominate because per-layer overhead is punishing. When the latency budget is generous (e.g., 5+ seconds), deeper models (depth 18–24) can amortize their per-layer overhead over more compute per layer and benefit from their greater representational capacity. This provides a general design principle: "the optimal depth-width ratio generally increases with the target latency budget."

What the scaling law adds beyond these qualitative observations. The three observations above are qualitative — they say "shallow is better for tight budgets, deeper is better for loose budgets," but don't specify exactly which ratio to pick for a specific budget. The augmented scaling law quantifies this: it provides a continuous function that can be queried for any $(D, W)$ pair, allowing precise selection of the configuration that minimizes loss subject to a hard latency constraint. This makes the framework actionable for practitioners who have specific latency targets (e.g., "must decode 8k tokens in under 30 seconds on an H100").

Operator Exploration: Pure and Hybrid Models

Motivation. Beyond depth and width, the specific token-mixing operator used in each layer (attention, Mamba, DeltaNet, etc.) fundamentally affects both accuracy and latency. Standard attention provides strong recall and in-context learning but introduces quadratic computation and a linearly growing KV cache. Linear attention variants (Mamba2, DeltaNet, GLA, RWKV7) offer sub-quadratic complexity but have been shown to underperform attention on tasks requiring precise token-level retrieval. The question is: which operators lie on the accuracy–latency Pareto frontier, and can combining them in hybrid models yield better trade-offs than any single operator alone?

Pure model comparison (Figure 4). The paper trains a series of 500M-parameter models, each using a single operator type throughout all layers, on 100B tokens from Smollm-corpus. The operators compared are: Mamba, Mamba2, GLA, DeltaNet, Gated DeltaNet, RWKV7, and sliding window attention (SWA) with a window size of 512. All models use the same number of blocks. Key implementation details: Mamba/Mamba2 use the official implementations without inserted FFNs (as specified in their original papers); all other operators have one FFN after each token-mixing operator; FlashAttention is used for SWA; FlashLinearAttention is used for all other linear attention variants.

The validation loss curves (Figure 4a) and the PPL–latency trade-off (Figure 4b, measured by decoding 8k tokens at batch size 1 on an A100 with CUDA Graph) reveal:

DeltaNet and Gated DeltaNet emerge as the most promising candidates on the PPL–latency Pareto frontier. They achieve lower perplexity than other linear attention variants (GLA, RWKV7) at comparable or better latency, and they are competitive with or better than Mamba2.
The architecture ranking stabilizes early in training. This is a critical observation that the paper leverages for the evolutionary search (discussed below). If ranking at 10B tokens correlates strongly with ranking at 100B tokens, then short training runs can serve as reliable proxies for final performance, dramatically reducing the cost of evaluating candidate architectures. The paper computes the Spearman rank correlation between short-training PPL and full-training PPL and finds it to be 88.8%, which is "sufficient to identify strong architectures within our search space."

Hybrid model exploration (Table 1). Recognizing that pure models have limitations (attention is accurate but slow, linear attention is fast but has recall weaknesses), the paper evaluates hybrid models that combine two operator types in a layer-wise interleaved manner. The building block structure is Operator1–Operator2–FFN, repeated across the model's depth. For fair comparison, the total number of operators (counting each token-mixing operator and each FFN as one operator) is kept constant at 24 across all pure and hybrid configurations. The models have 500M parameters.

Table 1 reports Wikitext perplexity (lower is better) and commonsense reasoning accuracy (higher is better) for four hybrid combinations, each tested with two different second operators (Attention or Mamba2):

DeltaNet–Mamba2–FFN hybrid: PPL 23.52, CR accuracy 48.07% with Mamba2 as second operator. With Attention as second operator: PPL 24.32, accuracy 47.48%. The Mamba2 pairing is clearly better.
Gated DeltaNet–Mamba2–FFN hybrid: PPL 23.37, CR accuracy 48.03% with Mamba2. With Attention: PPL 24.31, accuracy 47.80%. Again, Mamba2 pairing dominates.
GLA–Mamba2–FFN hybrid: PPL 24.04, CR accuracy 47.99% with Mamba2. With Attention: PPL 24.60, accuracy 47.63%.

Key findings from the hybrid exploration:

Pairing with Mamba2 consistently outperforms pairing with Attention. Across all three operator types tested (DeltaNet, Gated DeltaNet, GLA), the hybrid with Mamba2 achieves lower perplexity and higher accuracy than the hybrid with Attention. The paper does not fully explain this, but it likely relates to complementary memory mechanisms — Mamba2's state-space formulation may provide a different inductive bias that complements the linear attention mechanisms better than standard attention's token-level exact retrieval.
Hybrid models consistently outperform their corresponding pure models. For example, the DeltaNet–Mamba2 hybrid achieves PPL 23.52 and accuracy 48.07%, compared to pure DeltaNet's PPL 23.87 and accuracy 47.83%, and pure Mamba2's PPL 24.36 and accuracy 47.72%. The hybrid model is strictly better than either pure model individually. This is attributed to "the complementary and diverse memory mechanisms introduced by hybrid layers."
The performance gap between individual operators narrows in hybrid configurations. For example, Gated DeltaNet outperforms DeltaNet in pure-model language modeling (PPL 23.80 vs. 23.87), but when both are paired with Mamba2 in hybrid models, the performance becomes comparable (PPL 23.37 for Gated DeltaNet–Mamba2 vs. 23.52 for DeltaNet–Mamba2). Since DeltaNet is more computationally efficient than Gated DeltaNet, it becomes the preferable operator in hybrid models — the efficiency advantage is retained while the accuracy gap is closed.
GLA lags behind DeltaNet variants both in pure form (PPL 25.16, the worst among non-RWKV operators) and in hybrid form (the GLA-based hybrids trail the DeltaNet-based ones). This suggests that GLA's gating mechanism, while theoretically motivated, does not translate to better empirical performance in this training regime.

Design space for the evolutionary search. Based on these findings, the paper selects three operators as candidates for the search: DeltaNet, Attention, and Mamba2. The selection rationale is clear from the data: DeltaNet for its strong accuracy–latency position on the Pareto frontier, Mamba2 for its complementary strengths in hybrid configurations, and Attention for its essential recall capabilities despite its higher latency. Gated DeltaNet and GLA are excluded because their marginal accuracy benefits (or lack thereof) don't justify their higher computational cost relative to DeltaNet.

The Evolutionary Search Framework

Why search is necessary. The hybrid model exploration in Table 1 tested a handful of manually specified combinations (one operator pair, interleaved uniformly across all layers). But the true design space is much larger: how many different operator types should be used (2 or 3)? In what ratios should they be mixed? Should the mixture be uniform across all layers, or should different parts of the model (early, middle, late) use different operator combinations? How many FFNs should follow each token-mixing operator? These combinatorial choices create an enormous search space — far too large for manual exploration or grid search. The evolutionary search framework automates this exploration.

Search space definition (Appendix C.1). The architecture is partitioned into three stages — early, middle, and late — with each stage repeating one type of building block. This three-stage structure "balances operator heterogeneity with architectural regularity" — it allows different operator compositions in different parts of the model while maintaining enough structure that the search space is manageable. Within each stage, the search determines:

Operator types and ratios: At most two types of token-mixing operators (excluding FFNs) are used per stage. Their ratio can be 0:1 (only one operator type), 1:1, 1:2, or 1:3. For example, a stage with ratio 1:2 might have [DeltaNet, Mamba2, Mamba2] or [Attention, DeltaNet, DeltaNet].
FFN ratios: Each token-mixing operator can be followed by 0, 1, or 2 FFNs. This controls the ratio of feed-forward computation to attention/linear-attention computation within each block.
Number of building blocks per stage: Flexible, subject to a total operator count constraint (≤30 operators, where each token-mixing operator and each FFN counts as one). If the total exceeds 30, blocks are removed from the last stage to meet the constraint.

After the architecture structure is determined, a hidden size is selected from the set {1024, 1280, 1536, 1792, 2048, 2304, 2560}, choosing the largest size that satisfies the target latency constraint. This allows the short-training perplexity to be used purely for ranking architectures (since all compared architectures are fit to the same latency budget by adjusting hidden size).

Short-training perplexity as a proxy metric. Training a 500M+ parameter model to convergence (100B+ tokens) for every candidate architecture would make search infeasibly expensive. The critical enabler is the observation from Figure 4a that architecture rankings stabilize early in training. The paper validates this by computing the Spearman rank correlation between short-training (10B tokens) and full-training (100B tokens) perplexity: 88.8%. At this correlation level, the short-training proxy reliably distinguishes strong from weak architectures, even if the absolute perplexity values change with continued training. This reduces the evaluation cost per candidate by roughly 10× (10B tokens vs. 100B tokens), making evolutionary search practical — "training and evaluating a sampled architecture takes approximately 2 hours using 32 NVIDIA A100 GPUs" (Appendix C.2), versus ~20 hours for full training.

The aging evolution algorithm (Algorithm 1). The search uses aging evolution [real2019regularized], a population-based algorithm that balances exploration and exploitation through tournament selection and age-based replacement. The algorithm proceeds as follows (with specific hyperparameters from the paper):

Initialization: A population $\mathcal{P}$ of architectures is created from a mix of seeded architectures (known designs from the manual exploration) and randomly initialized ones. All architectures are short-trained (10B tokens) and evaluated for proxy perplexity. Latency is estimated from a precomputed lookup table (LUT) that maps architectural parameters to measured latency on the target hardware — no actual latency measurement is needed during search because all architectural factors (operator types, ratios, hidden size) have known latency costs that can be pre-characterized.
Cycle loop: For each of $C$ cycles (the number of cycles isn't explicitly stated, but Figure 5 shows the search trajectory over cycles), 10 new candidate architectures are evaluated in parallel (one per cycle iteration, with 10 parallel workers implied by the concurrent evaluation structure).
Tournament selection (parent selection): A random subset $\mathcal{S}$ of size $S$ is sampled from the current population $\mathcal{P}$ . The parent $p$ is selected as: $p = \arg\min_{x \in \mathcal{S}} \text{PPL}(x) \quad \text{subject to} \quad \text{latency}(x) \leq L_{\text{target}}$ That is, among the randomly sampled architectures that satisfy the latency constraint, the one with the lowest short-training perplexity is chosen as the parent. Tournament selection (rather than always picking the global best) preserves diversity by giving moderate-performing architectures a chance to reproduce, preventing premature convergence to a local optimum.
Mutation: The offspring $c$ is created by mutating exactly one factor of the parent: (a) changing one operator type in a building block, (b) varying the hybrid operator ratio in a building block type, (c) adjusting the FFN ratio in a building block type, or (d) modifying the number of building blocks across stages. Single-factor mutation (rather than multi-factor) ensures that offspring are meaningfully related to their parent, allowing the search to hill-climb in the architectural space. After mutation, adjustments are made to maintain the depth constraint (total operators ≤ 30).
Evaluation and replacement: The offspring is short-trained to obtain proxy perplexity, latency is queried from the LUT, and the oldest architecture in the population is removed and replaced by the offspring. Age-based replacement (rather than fitness-based replacement) ensures that all architectures eventually get replaced, preventing the population from stagnating. It also naturally balances exploration and exploitation: new architectures get time to be evaluated before they can be eliminated, while architectures that have been in the population for many cycles eventually age out regardless of their fitness.
Return: After $C$ cycles, the architecture in $\mathcal{P}$ with the lowest proxy PPL that satisfies the latency constraint is returned as the search output.

Search results: decoding-latency-optimized architecture (Table 2). The architecture discovered by the search when optimizing for decoding latency (8k tokens, batch size 1, A100) reveals a non-obvious structure:

Basic building blocks: The search converged on two block types, interleaved throughout the model: [DeltaNet, FFN, Mamba2, FFN] and [Attention, FFN, Mamba2, FFN]. These are not the naive combinations one might have guessed — they interleave three different operator families (linear attention, state-space model, full attention, FFN) in a specific pattern.
Operator distribution: The model contains 24 total operators. Mamba2 appears most frequently (as the second operator in all blocks), followed by DeltaNet and Attention. This aligns with the hybrid exploration finding that Mamba2 is the best second operator across pairings.
Comparison to baselines (Table 3): When benchmarked against pure models scaled to the same decoding latency (by increasing hidden size until latency matches), the searched architecture achieves: PPL 20.70 (vs. 20.90 for DeltaNet, the next best), CR accuracy 51.04% (vs. 50.38% for DeltaNet), with 837M parameters (vs. 852M for DeltaNet). The improvement is attributed to (1) efficient operator combinations that allow larger parameter counts at fixed latency, and (2) complementary memory mechanisms from hybrid operators.

Search with parameter count as the metric (Table 2, bottom row). To validate that the search framework adapts to different optimization targets, a second search was run using 500M parameter count as the constraint. The resulting architecture is qualitatively different:

More attention layers: The parameter-optimized architecture includes more full attention modules, which are parameter-efficient (attention has fewer parameters per layer than Mamba2 or DeltaNet at the same hidden size) but decoding-unfriendly (quadratic complexity, KV cache).
Greater depth: The model is deeper (more operators), another parameter-efficient design choice (as established in Section 3.1). This architecture would have worse latency than the latency-optimized one.
Performance: It achieves PPL 23.06 and CR accuracy 49.23%, which is better than all 500M baselines (Table 10 in Appendix C.3: +1.21% CR accuracy vs. best baseline, -0.74 PPL vs. best baseline).

Why this matters: The fact that different optimization targets produce structurally different architectures validates the core premise of the paper — that architecture design must be metric-aware. The parameter-optimal architecture is not the latency-optimal architecture, and vice versa. The evolutionary search provides a principled way to target whichever metric matters for a given deployment.

Lesson learned from the operator search (Section 3.2): Hybrid models show great promise, but the synergy between operators is complex and non-obvious, making automated search necessary. The relative stability of architecture rankings early in training (88.8% Spearman correlation) provides a reliable signal for iterating over designs, enabling search algorithms that would be infeasible if full training were required for each evaluation.

Weight Normalization for Effective Training

Motivation: structural patterns in trained weight matrices. The paper observes that when models are trained with standard optimization (AdamW, no normalization), weight matrices develop large magnitudes along certain structural dimensions. Specifically, as shown in the first row of Figure 6:

Case-1 matrices — weights applied to hidden features (e.g., the input projection in attention or FFN layers) — exhibit horizontal patterns: certain rows of the weight matrix grow to have much larger norms than others. This means some output dimensions are dominated by a small number of input dimensions with large weights.
Case-2 matrices — weights whose outputs are added back to hidden features (e.g., the output projection after attention or FFN, which feeds into the residual stream) — exhibit vertical patterns: certain columns grow to have much larger norms. This means some input dimensions have outsized influence on all output dimensions.

In both cases, the problem is the same: large weight magnitudes lead to small relative weight updates. If a weight has magnitude $|W| = 10$ and receives a gradient of magnitude $|\nabla| = 0.1$ , the relative update is $|\Delta W|/|W| \approx \eta \cdot 0.1 / 10 = 0.01\eta$ . If that same weight had magnitude $|W| = 1$ , the relative update would be $\approx 0.1\eta$ — ten times larger. In late training stages, when the learning rate $\eta$ is small (due to cosine decay), weights with large magnitudes receive trivially small updates, stalling convergence.

The normalization operation. To address this, after each optimizer step, weight matrices are projected onto a unit-norm sphere, eliminating the radial component and enforcing angular updates. The projection direction depends on the structural pattern:

For Case-1 (row-wise patterns, matrices applied to hidden features): Each row is normalized independently: $\mathbf{W}_{i,:} \leftarrow \frac{\mathbf{W}_{i,:}}{||\mathbf{W}_{i,:}||_2} \quad \text{for } i = 1, \dots, C_{\text{out}}$ where $\mathbf{W}_{i,:}$ is the $i$ -th row of the weight matrix $\mathbf{W} \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}}}$ , $||\cdot||_2$ is the L2 norm, $C_{\text{out}}$ is the output dimension, and $C_{\text{in}}$ is the input dimension.

What it computes: Each output neuron's incoming weight vector is rescaled to unit length. The direction of the weight vector (which determines which input patterns activate this output neuron) is preserved; only the magnitude is removed. After normalization, every output neuron's weight vector lies on the unit hypersphere, and gradient updates move it along the surface of that sphere (angular updates) rather than changing its length.
For Case-2 (column-wise patterns, additive output matrices): Each column is normalized independently: $\mathbf{W}_{:,j} \leftarrow \frac{\mathbf{W}_{:,j}}{||\mathbf{W}_{:,j}||_2} \quad \text{for } j = 1, \dots, C_{\text{in}}$ where $\mathbf{W}_{:,j}$ is the $j$ -th column.

What it computes: Each input dimension's outgoing weight vector is rescaled to unit length. This ensures that no single input dimension can dominate the residual stream through large-magnitude weights.

Why this form: The choice of normalization dimension (row-wise vs. column-wise) is driven by the observed structural patterns. Row-wise normalization for Case-1 directly targets the horizontal stripes in Figure 6; column-wise normalization for Case-2 directly targets the vertical stripes. A naive global normalization (dividing the entire matrix by its Frobenius norm) would not address these structured patterns — it would scale all entries equally, leaving the relative imbalances within rows or columns intact. The dimension-specific normalization explicitly equalizes the influence of each output neuron (Case-1) or each input feature (Case-2).

Additionally, the paper notes that activation normalization alone (without weight normalization) is ineffective — Table 12 shows that "nGPT without weight normalization" achieves worse accuracy than the vanilla baseline (PPL 19.38 vs. 18.67 for Llama 1B), suggesting that the benefit comes from constraining weight magnitudes and enabling larger effective updates, not from normalizing activations.

Effects observed during training (Figures 7 and 8):

Validation loss (Figure 7): In early training, the baseline (without weight normalization) converges faster because unconstrained weight updates can rapidly grow weight magnitudes in productive directions. However, as training progresses, the baseline's convergence slows (because large weights lead to small relative updates), while weight-normalized models maintain a more constant convergence rate. Eventually, the weight-normalized models surpass the baselines and achieve better final convergence. This pattern — slower start, faster finish, better final result — is consistent across four model families: Llama, DeltaNet, Mamba2, and hybrid architectures.
Gradient and weight norms (Figure 8): Weight normalization dramatically reduces the L2 norm of model weights (by construction, each row/column is normalized to unit norm). At the same time, the average element-wise gradient norm is slightly increased compared to the baseline. The combination — smaller weights, comparable or slightly larger gradients — means that relative weight updates $|\Delta W|/|W|$ are substantially larger throughout training. This effect is "particularly helpful in late training stages" when learning rates are small.
Quantitative improvements (Table 4): Across three model families trained on 100B tokens, weight normalization improves commonsense reasoning accuracy by an average of +1.20% and reduces Wikitext perplexity by an average of 0.66. The gains are consistent: Llama 1B (+1.04% accuracy, -0.64 PPL), DeltaNet 1B (+0.93%, -0.67 PPL), Mamba2 1B (+1.41%, -0.56 PPL).

Relationship to nGPT (Appendix D.1, Table 12). The paper's weight normalization is positioned as a simplified and more efficient variant of nGPT [loshchilov2024ngpt], which enforces spherical constraints on all computations (both weights and activations). The paper demonstrates:

Weight normalization alone achieves comparable accuracy to full nGPT. On Llama 1B, weight normalization reaches 54.85% CR accuracy vs. nGPT's 54.73%. On a hybrid 1B model, 55.74% vs. 55.33%. The differences are negligible.
Weight normalization is essential to nGPT. Removing weight normalization from nGPT (keeping only activation normalization) drops Llama 1B accuracy from 54.73% to 51.83% — substantially worse than the vanilla baseline (53.81%). This suggests that the core mechanism driving nGPT's improvements is the weight normalization, not the activation normalization.
Weight normalization is faster. nGPT's additional activation normalization layers increase training time by more than 30% (0.59 seconds per iteration for nGPT vs. 0.46 for weight normalization on Llama 1B). Given a fixed training duration, weight normalization enables processing more tokens, contributing to its practical advantage.

Implementation detail: no weight decay. Because weight normalization constrains weights to unit norm, weight decay (which penalizes large weight magnitudes) becomes redundant. The paper explicitly states that the optimizer is used "without weight decay, due to the use of weight normalization." This also simplifies hyperparameter tuning (one less parameter to sweep).

Meta Tokens for Cache Initialization

Motivation. Standard language models attend to all previous tokens when generating each new token. This creates an "attention sink" phenomenon [xiao2023efficient] where the model learns to allocate substantial attention to semantically unimportant tokens (e.g., initial tokens, punctuation) simply because they are always present — even though they carry no useful information for the current prediction. Meta tokens, introduced in Hymba [dong2024hymba], are a set of learnable embeddings prepended to the input sequence that serve as dedicated attention sinks, absorbing this forced attention and freeing the model to allocate attention more meaningfully to content tokens.

Extension to linear attention. The paper observes that meta tokens can also benefit non-softmax linear attention mechanisms. In linear attention, the recurrent formulation during decoding maintains a state matrix that accumulates key-value information from previous tokens. Prepending meta tokens provides a learned initialization for this recurrent state, analogous to how prepending learned prompts initializes the hidden state in recurrent neural networks. This learned initialization can encode general priors that improve the model's information processing.

Implementation and results (Table 5). The paper prepends 256 learnable meta tokens to the input sequence. These tokens are model parameters (trained during the normal training process) but do not correspond to any words in the vocabulary — they are purely internal learned embeddings. At inference time, they are prepended to every input.

Table 5 shows that meta tokens consistently improve both language modeling and reasoning accuracy across model types, with negligible overhead:

Mamba2-48L-1B: PPL improvement of 0.06, CR accuracy improvement of +0.62%
DeltaNet-48L-1B: PPL improvement of 0.13, CR accuracy improvement of +0.34%
Searched hybrid 24L-830M: PPL improvement of 0.12, CR accuracy improvement of +0.39%

The average accuracy gain across these three models is approximately +0.45%, which is modest but essentially free (256 additional tokens prepended to sequences that are typically thousands of tokens long adds negligible computation).

Final Model Assembly: Nemotron-Flash-1B and 3B

Model configuration (Table 11). The final Nemotron-Flash models adopt the decoding-latency-optimized architecture discovered by the evolutionary search, with depth and width scaled using the principles from Section 3.1:

Nemotron-Flash-1B: 0.96B parameters, 12 blocks (24 operators total: 12 token-mixing operators + 12 FFNs), hidden size 2048, FFN dimension 6144, 16 attention heads with 4 KV heads (group-query attention, group size 4). The operator sequence is: [D, F, M2, F, A, F, M2, F, D, F, M2, F, A, F, M2, F, D, F, M2, F, D, F, M2, F] where D = DeltaNet, M2 = Mamba2, A = Attention, F = FFN. This contains only 2 full attention layers (positions 5 and 13 in the sequence), yet achieves competitive recall accuracy, confirming that "maintaining full KV cache across all layers is unnecessary."
Nemotron-Flash-3B: 2.7B parameters, 36 operators, hidden size 3072, FFN dimension 9216, 24 attention heads with 6 KV heads. The operator sequence is an extended version of the 1B architecture with two additional Block-1 (D, F, M2, F) and one additional Block-2 (A, F, M2, F), still containing only 3 full attention layers.

Tokenizer choice (Appendix E). Unlike previous SLMs that use small-vocabulary tokenizers (e.g., Llama2 tokenizer) to reduce embedding layer parameters, the paper uses the Mistral-NeMo-Minitron tokenizer [blakeman2025nemotron] with a larger vocabulary. The rationale is that larger vocabularies produce more coarse-grained tokens — common phrases like "by the way" become single tokens — reducing the total token count for encoding the same text. Table 13 shows that this tokenizer reduces tokens per sample by 9.3% on Wikitext and 13.5% on AG News compared to the Llama2 tokenizer. Since decoding latency scales with the number of tokens generated, this token count reduction directly improves latency. The increased embedding parameters (from the larger vocabulary) add only ~5.8% latency overhead (Table 13 comparison at 8k tokens), which is smaller than the latency savings from generating fewer tokens. The paper notes: "the latency overhead of the enlarged embedding layer/LM head is small, while the more coarse-grained token representations reduce the token count when encoding the same sentence, resulting in more significant latency reduction."

Training protocol. Both models are trained for 4.5T tokens using 256 NVIDIA H100 GPUs with a batch size of 2M tokens and a context length of 4096 (extended to 29,000 for the final 25B tokens). Key training details:

Optimizer: Adam (not AdamW — weight decay is omitted due to weight normalization), with a cosine learning rate schedule and an initial learning rate of 1e-3. The learning rate of 1e-3 is notably high for language model training; the paper tunes this as "the tuned learning rate that achieves the best convergence for models without weight normalization" and then applies weight normalization on top, demonstrating that weight normalization can further improve performance even from this optimized baseline.
Data curriculum: Training proceeds in stages, starting with Zyda2 [tokpanov2024zyda], then switching to higher-quality datasets: commonsense reasoning datasets (Climb-Mix [diao2025climb] and Smollm-corpus [benallal2024smollmcorpus]), a proprietary high-quality dataset with high proportions of math and code, and MegaMath [zhou2025megamath]. This curriculum — general text first, specialized reasoning later — is a common practice but the paper doesn't ablate its importance.
Weight normalization is applied throughout training, with the dimension-specific projection described above.
Meta tokens (256 learnable embeddings) are prepended to all sequences.

Deployment configuration (Section 4, Appendix F). For latency benchmarking, Nemotron-Flash models are deployed using a custom integration of three backends: TensorRT-LLM's AutoDeploy kernels [nvidia2025autodeploy] with efficient KV cache management for full attention layers, the official Mamba2 implementation [dao2024transformers] for Mamba2 layers, and FlashLinearAttention [yang2024fla] for DeltaNet layers. The entire model is wrapped in a CUDA Graph to mitigate kernel launch overhead, which is "critical for edge applications with a batch size of 1." Appendix F (Table 14) validates that this deployment flow achieves latency within 15% of TensorRT-LLM for pure-attention models while additionally supporting linear attention variants that vLLM and TensorRT-LLM do not support.

Summary of design choices and their justifications:

DeltaNet–Mamba2–FFN and Attention–Mamba2–FFN as basic blocks: Discovered by evolutionary search, justified by complementary memory mechanisms and validated by superior PPL and accuracy vs. pure models at equal latency.
Only 2–3 full attention layers: Sufficient for recall tasks (Table 6 recall accuracy remains competitive) while dramatically improving throughput. The ablation in Section 4.3 confirms that even 2 FA layers + 1 SWA layer maintains NIAH performance, with throughput improving 1.3× over 3 FA layers.
Weight normalization over full nGPT: Comparable accuracy, >20% faster training, simpler implementation (no additional activation normalization layers).
Large-vocabulary tokenizer: Reduces token count (and thus decoding steps) more than it increases per-step latency from larger embedding lookups.
Short-training proxy for evolutionary search: Reduces evaluation cost by ~10× with 88.8% rank correlation to full-training results, making automated architecture search practical.
CUDA Graph wrapping: Eliminates per-layer kernel launch overhead, which is proportionally larger in SLMs (smaller per-layer compute) and at batch size 1 (no amortization across batch).

4. Key Insights and Innovations

Innovation 1: Latency, Not Parameters, Should Be the Primary Optimization Target for Small Language Models

The paper’s most fundamental conceptual move is to reject the dominant design paradigm for small language models and replace it with a different objective entirely. The implicit assumption driving prior SLM work — from MobileLLM [liu2024mobilellm] to SmolLM [allal2025smollm2smolgoesbig] to TinyLlama [zhang2024tinyllama] — is that parameter count is the correct efficiency metric to optimize, and that a model with fewer parameters will naturally be faster on real hardware. This paper demonstrates that this assumption is not just imprecise; it is wrong in a systematic and directional way. Deep–thin architectures, which are the consensus recommendation from parameter-efficient design, actually produce worse accuracy–latency trade-offs than wider–shallower alternatives at the same parameter budget (Figure 2b).

What makes this a genuine intellectual contribution rather than an obvious observation is that the paper provides a causal explanation for why the parameter-efficiency and latency-efficiency frontiers diverge. The explanation — per-layer fixed overhead costs on GPUs (kernel launches, memory transfers, synchronization barriers) that are independent of layer width — is a hardware–software interaction that the SLM design literature had largely ignored. Prior work treated SLM architecture design as a purely algorithmic problem (how to maximize accuracy for a given parameter budget), implicitly assuming that hardware efficiency would follow. This paper shows that the algorithmic optimum and the hardware optimum are structurally different, and that optimizing for the wrong one produces models that leave substantial performance on the table — in this case, 4× or more in throughput (Table 6).

A secondary but equally important conceptual contribution is the finding that the optimal depth–width ratio scales with the latency budget (Observation 3, Section 3.1). This is not a fixed recipe (“use 12 layers”) but a principle: tighter latency budgets favor shallower models because per-layer overhead is proportionally more costly; looser budgets allow deeper models to amortize overhead and benefit from greater representational capacity. This transforms depth selection from a one-time architectural choice into a deployment-contingent optimization, which has significant practical implications for model families that must serve diverse hardware targets.

Innovation 2: Decoupling Depth and Width in Scaling Laws Enables Principled Latency-Constrained Model Design

Prior scaling laws for language models (Hoffmann et al., 2022; Hu et al., 2024 from MiniCPM) collapse all architectural parameters into a single variable: total parameter count P. This formulation is sufficient when the goal is to predict loss as a function of compute during pretraining, because pretraining FLOPs scale primarily with total parameters and training tokens. But when the optimization target shifts to inference latency, the single-variable formulation becomes inadequate — two models with the same P can have very different latencies depending on whether those parameters are distributed across many narrow layers or few wide ones.

The paper’s augmented scaling law (Equation 1) — L(D, W, N) = L₀ + a·D^(-α) + b·W^(-β) + c·N^(-γ) — addresses this by treating depth D and width W as independent dimensions with separate scaling exponents. This is a conceptually clean extension of the existing scaling law framework that preserves its mathematical structure (additive power-law terms, irreducible loss floor) while making it applicable to a problem — latency-constrained architecture selection — that the original formulation could not address.

The intellectual significance of this move extends beyond the specific functional form. By establishing that depth and width have separable, power-law contributions to loss reduction, the paper provides a principled method for navigating the depth–width trade-off space: fit the scaling law on a subset of configurations, profile latency for candidate (D, W) pairs, and select the pair that minimizes predicted loss subject to a hard latency constraint. The validation that this extrapolates within 5.3% of ground-truth perplexity (Figure 3) demonstrates that the decoupling is not merely a notational convenience — it captures real, predictable structure in how depth and width independently contribute to model quality. This makes it possible to reason quantitatively about architecture decisions that were previously guided by trial-and-error or heuristics like “deeper is better for parameters.”

Innovation 3: Evolutionary Search Over Hybrid Operators Surfaces Architecture as a Multi-Dimensional, Metric-Aware Optimization Problem

Prior work on hybrid language models — Jamba [lieber2024jamba], Samba [ren2024samba], Zamba [glorioso2024zamba], Hymba [dong2024hymba] — demonstrates that combining full attention with sub-quadratic operators (typically Mamba) can improve the accuracy–efficiency trade-off compared to pure models. But the operator combinations in these works are manually specified, and each explores only a tiny fraction of the combinatorial design space. As the number of candidate efficient attention mechanisms proliferates (Mamba2, DeltaNet, Gated DeltaNet, GLA, RWKV7, etc.), manual exploration becomes both infeasible and increasingly likely to miss non-obvious synergies.

The paper’s evolutionary search framework addresses this by automating the exploration, but its deeper contribution is the demonstration that what you optimize for determines what you find. When the search is configured to minimize perplexity subject to a decoding latency constraint, it discovers architectures dominated by efficient operators (Mamba2, DeltaNet) with sparse attention layers (2–3 total). When the same search is configured to minimize perplexity subject to a parameter count constraint, it discovers deeper architectures with more attention layers (Table 2). These are structurally different solutions, confirming that the optimization metric is not a detail — it fundamentally shapes the architecture.

This finding has methodological implications beyond the specific models produced. It suggests that the proliferation of handcrafted hybrid architectures in the literature, each claiming to improve on some efficiency metric, may be partially an artifact of different implicit optimization targets. A Jamba-style architecture (sequential attention-Mamba stacking) might be near-optimal for a particular latency–accuracy trade-off point but suboptimal for a different one. The search framework provides a way to generate architectures tailored to specific deployment constraints rather than proposing a single “best” architecture, which is a more mature approach to the design problem.

The search methodology itself also introduces an insight with broad applicability: that short-training perplexity (10B tokens) achieves 88.8% Spearman rank correlation with full-training perplexity (100B tokens), making it a reliable proxy for architecture ranking. This is a diagnostic finding — it measures an empirical property of LM training dynamics — that enables the search to be practical. Without this correlation, evaluating each candidate would require full training, making evolutionary search infeasible. The paper validates this correlation across diverse operator types (attention, Mamba2, DeltaNet, GLA), suggesting it may be a general property of autoregressive LM training rather than specific to any architecture.

Innovation 4: Weight Normalization Isolated as the Critical Mechanism in nGPT, With a Dimension-Specific Projection Strategy

The paper’s weight normalization technique builds directly on nGPT [loshchilov2024ngpt], which proposed enforcing spherical constraints on all computations in a transformer — both weights and activations — to improve training convergence. This paper makes two intellectual contributions relative to that prior work.

First, it provides a mechanistic ablation that identifies weight normalization as the primary driver of nGPT’s benefits. Appendix D.1 (Table 12) shows that weight normalization alone achieves essentially identical final accuracy to full nGPT (54.85% vs. 54.73% CR accuracy on Llama 1B; 55.74% vs. 55.33% on a hybrid model), while nGPT without weight normalization (activation normalization only) performs substantially worse than the vanilla baseline. This is a clean diagnostic result: the activation normalizations that make nGPT theoretically elegant (enforcing all computations on a unit sphere) are not what drives the empirical gains; the weight constraint is. This matters because activation normalizations introduce >20% training overhead (Table 12: 0.59 vs. 0.46 seconds per iteration), meaning that the simplified technique preserves the benefits while eliminating a significant fraction of the cost.

Second, the paper introduces a dimension-specific projection strategy motivated by observed structural patterns in trained weight matrices (Figure 6). Weights applied to hidden features (Case-1) develop horizontal striations — certain output dimensions have outsized norms — so row-wise normalization equalizes their influence. Weights whose outputs are added back to the residual stream (Case-2) develop vertical striations — certain input dimensions dominate — so column-wise normalization prevents any single feature from saturating the residual. This is a more structured approach than naively normalizing entire weight matrices, and it is motivated by empirical observation rather than theoretical derivation. The paper does not claim that this dimension-specific approach is optimal — it follows the observed patterns — but it demonstrates that the choice of normalization dimension matters and should align with the matrix’s role in the architecture.

</output>

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary exploration experiments (Sections 3.1–3.3) use the Smollm-corpus [benallal2024smollmcorpus], a publicly available text corpus, training models on 100B tokens. For the final Nemotron-Flash models (Section 4), training data includes Zyda2 [tokpanov2024zyda], Climb-Mix [diao2025climb], Smollm-corpus, a proprietary high-quality dataset with high proportions of math and code, and MegaMath [zhou2025megamath], totaling 4.5T tokens. Evaluation is conducted across 16 tasks spanning MMLU (5-shot), six commonsense reasoning tasks (PIQA, ARCC, ARCE, Hellaswag, Winogrande, OBQA, all 0-shot), two math tasks (GSM8k at 5-shot, MathQA at 0-shot), four coding tasks (HumanEval, HumanEval-Plus, MBPP at 3-shot, MBPP-Plus at 3-shot), and three recall tasks (FDA, SWDE, Squad, all 0-shot). All evaluations use the lm-evaluation-harness framework [eval-harness]. For the scaling law experiments, accuracy is averaged over eight commonsense reasoning tasks: Lambda, PIQA, ARC-Easy, ARC-Challenge, Hellaswag, Winogrande, TruthfulQA, and SIQA (Appendix A).
Base model(s). The architectural exploration experiments (Sections 3.1–3.3) train Llama-style models with varying depth, width, and operator configurations from scratch on 100B tokens of Smollm-corpus. These are not pretrained checkpoints — they are purpose-trained for the ablation studies. The final Nemotron-Flash models (Section 4) are trained from scratch on 4.5T tokens. For benchmarking, the paper compares against a wide range of existing SLMs including Qwen3 [qwen3], Qwen2.5 [qwen2.5], Llama-3.2 [grattafiori2024llama], SmolLM2 [allal2025smollm2smolgoesbig], H2O-Danube3 [singer2024h2o], and AMD-OLMo [AMD-OLMo]. The choice to train all exploration models from scratch (rather than fine-tuning existing checkpoints) ensures controlled comparisons where only the factor under study varies, since different pretrained models have different data mixtures, tokenizers, and training curricula that would confound architectural comparisons.
Metrics. Three primary metrics are used throughout the paper:
- Accuracy: Reported as the percentage of questions answered correctly on each task, then averaged within domains (commonsense reasoning, math, coding, recall, MMLU) and across all 16 tasks for the overall score. Task-level accuracy is reported in Table 9.
- Decoding latency: Measured as the wall-clock time to generate 8,192 tokens with a batch size of 1 on an NVIDIA H100 GPU, using CUDA Graph wrapping. For the depth–width exploration in Section 3.1, latency is measured for 1,024-token generation on an A100. For the deployment flow ablation (Appendix F), latency is reported at multiple generation lengths (64 to 32,768 tokens) on an A100.
- Throughput: Measured as tokens per second when processing a 32,768-token input length using the maximum batch size that does not cause an out-of-memory (OOM) error for each model individually. This is a "saturating throughput" metric — it pushes each model to its hardware limit — rather than a fixed-batch-size comparison, which means models with smaller memory footprints (e.g., those with fewer full attention layers and thus smaller KV caches) can use larger batches and thus report higher throughput.
- Perplexity: Used as an intrinsic language modeling metric in the exploration experiments (Sections 3.1–3.3), measured on Wikitext. Perplexity is the exponentiated negative log-likelihood; lower is better.
Baselines. The paper compares against the following specific models in the final benchmark (Table 6): AMD-OLMo-1B (1B parameters, depth 16), Llama-3.2-1B (1.2B, depth 16), Qwen2.5-0.5B (0.5B, depth 24), Qwen3-0.6B (0.6B, depth 28), H2O-Danube3-4B (4B, depth 24), SmolLM2-1.7B (1.7B, depth 24), Llama-3.2-3B (3B, depth 28), Qwen2.5-1.5B (1.5B, depth 28), Qwen3-1.7B (1.7B, depth 28), and Qwen2.5-3B (3B, depth 36). For the pure operator comparison (Table 3, Figure 4), baselines include single-operator models using Attention (with sliding window of 512, hence SWA), GLA, DeltaNet, Gated DeltaNet, Mamba2, and Mamba2+FFN (an enhanced variant with FFNs inserted after each Mamba2 layer, unlike vanilla Mamba2). For the parameter-efficiency search (Appendix C.3, Table 10), additional baselines include pure RWKV and pure Mamba models.
Generation budget / compute accounting. The paper uses several different notions of "compute budget" depending on the experiment:
- Training tokens: The primary unit for exploration experiments is 100B training tokens from Smollm-corpus, with all compared models trained on exactly the same data to ensure fairness. For the final models, the budget is 4.5T tokens.
- Latency: Measured at fixed generation lengths (8k tokens for the main benchmark, 1k for the depth–width exploration) rather than fixed FLOPs. This means the comparison is "how much accuracy does each model achieve at a given latency," not "at a given computational cost." This is appropriate for the paper's deployment-centric framing but means models are not FLOPs-matched.
- Number of operators: In the hybrid model exploration (Section 3.2) and search (Section 3.2, Appendix C), all models are constrained to have the same number of total operators (24), where each token-mixing operator and each FFN counts as one. This ensures comparable depth and enables fair latency comparison.
- Short-training budget: For the evolutionary search, each candidate architecture is trained on 10B tokens (~2 hours on 32 A100s) as the evaluation budget. This is justified by the 88.8% Spearman correlation with 100B-token results.
Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests for any of its results. The architecture rankings from the evolutionary search are validated via Spearman rank correlation between short-training and full-training perplexity, but the final benchmark numbers in Table 6 are single-point estimates with no variance information. For the scaling law fitting (Figure 3), the model is fit on a subset of depth–width configurations and validated on held-out larger configurations, with extrapolation error reported as within 5.3% of ground-truth perplexity. This is a form of out-of-distribution validation for the functional form but not a statistical confidence measure.

Main Quantitative Results

The paper makes quantitative claims across four axes: depth–width ratio experiments, operator comparison and hybrid search, weight normalization ablation, and the final Nemotron-Flash benchmark against SOTA SLMs. I organize results accordingly.

Depth–Width Ratio Experiments (Section 3.1, Figures 2 and 3)

Headline finding: Deep–thin models achieve better accuracy per parameter, but wider–shallower models achieve better accuracy per millisecond of latency. The optimal depth–width ratio increases with the latency budget.

Figure 2 visualizes the trade-off. Figure 2a shows accuracy vs. parameter count for five depth settings (6, 12, 18, 24, 30 blocks) with varying widths. The deeper curves (depth 24, 30) lie above the shallower curves (depth 6, 12) — for a given parameter budget, deeper models are more accurate. This replicates the prior finding from MobileLLM and PanGu-π Pro. Figure 2b shows accuracy vs. decoding latency. The relationship reverses: the deepest models (depth 30) no longer dominate. The paper provides a concrete example: at a latency budget of 3 seconds, depth 12 achieves the best accuracy, outperforming both depth 6 (too shallow, insufficient representational capacity) and depth 30 (too deep, excessive per-layer overhead).

The scaling law fitting and extrapolation is shown in Figure 3. The paper fits Equation 1 on a subset of depth–width configurations and validates on held-out larger depth/width settings. The predicted perplexity values stay within 5.3% of ground-truth perplexity, demonstrating that the decoupled depth–width formulation extrapolates beyond the fitting regime. The paper states: "the model extrapolates reasonably well to unseen depth/width settings, staying within 5.3% of the ground-truth PPL, demonstrating that the fitted function generalizes beyond the observed training configurations."

The three key observations from this section are explicitly enumerated in the paper (Section 3.1):

Observation 1: "Deeper models generally achieve a better accuracy-parameter trade-off over a wide depth range, although the benefit gradually saturates"
Observation 2: "For the accuracy-latency trade-off, the advantage of deep-thin models may not hold, and there exists an optimal depth setting for a given latency budget"
Observation 3: "The optimal depth-width ratio generally increases with the latency budget"

Pure Operator Comparison and Hybrid Exploration (Section 3.2, Figure 4, Table 1)

Headline finding: DeltaNet and Gated DeltaNet lie on the PPL–latency Pareto frontier among pure operators. Hybrid models pairing operators with Mamba2 consistently outperform both their pure counterparts and hybrids pairing with Attention.

Figure 4a shows validation loss curves during training for 500M-parameter pure models. The relative ranking of architectures stabilizes early — by 10B tokens, the ordering of models by perplexity is largely predictive of their ordering at 100B tokens. This observation is quantified via the Spearman rank correlation between short-training (10B tokens) and full-training (100B tokens) perplexity, reported as 88.8%.

Figure 4b shows the PPL–latency trade-off after 100B tokens of training. The latency is measured as decoding 8k tokens at batch size 1 on an A100 with CUDA Graph. The results show:

DeltaNet: PPL 23.87, latency approximately 18.2 seconds (estimated from the figure)
Gated DeltaNet: PPL 23.80, latency approximately 17.9 seconds
Mamba2: PPL 24.36, latency approximately 17.8 seconds
Attention (SWA, window 512): PPL 24.44, latency approximately 18.0 seconds
GLA: PPL 25.16, latency approximately 18.2 seconds

DeltaNet and Gated DeltaNet are identified as "promising candidates, lying on the PPL-latency Pareto frontier," meaning no other pure operator simultaneously achieves lower PPL and lower latency.

Table 1 reports the hybrid model exploration. All models have 500M parameters and 24 total operators. Key comparisons:

DeltaNet–Mamba2–FFN hybrid achieves PPL 23.52 and CR accuracy 48.07% with Mamba2 as the second operator. The corresponding pure models (from Figure 4b) are DeltaNet (PPL 23.87, 47.83%) and Mamba2 (PPL 24.36, 47.72%). The hybrid is strictly better than both: PPL improved by 0.35 and 0.84 respectively, accuracy by 0.24% and 0.35%.
Gated DeltaNet–Mamba2–FFN hybrid achieves PPL 23.37 and CR accuracy 48.03%. Compared to pure Gated DeltaNet (PPL 23.80, 47.96%): PPL improved by 0.43, accuracy by 0.07%. Compared to pure Mamba2: PPL improved by 0.99, accuracy by 0.31%.
Pairing with Attention consistently underperforms pairing with Mamba2. For DeltaNet: with Mamba2 gets PPL 23.52, accuracy 48.07%; with Attention gets PPL 24.32, accuracy 47.48%. For Gated DeltaNet: with Mamba2 gets PPL 23.37, accuracy 48.03%; with Attention gets PPL 24.31, accuracy 47.80%. For GLA: with Mamba2 gets PPL 24.04, accuracy 47.99%; with Attention gets PPL 24.60, accuracy 47.63%.
The performance gap between DeltaNet and Gated DeltaNet narrows in hybrid configurations. As pure models, Gated DeltaNet (PPL 23.80) outperforms DeltaNet (PPL 23.87) by 0.07 PPL. When paired with Mamba2, the gap shrinks: Gated DeltaNet–Mamba2 (PPL 23.37) vs. DeltaNet–Mamba2 (PPL 23.52) — a difference of 0.15, but the accuracy gap essentially disappears (48.03% vs. 48.07%). Since DeltaNet is more computationally efficient than Gated DeltaNet, the paper concludes it is the preferable operator in hybrid models.

Evolutionary Search Results (Section 3.2, Tables 2 and 3, Figure 5)

Headline finding: The evolutionary search discovers non-obvious hybrid architectures that outperform scaled-up pure models at equivalent latency, and the discovered architecture depends on the optimization target (latency vs. parameter count).

Table 2 shows the architectures discovered under two different search metrics:

Decoding latency optimization: The searched architecture has 837M parameters, decoding latency of 17.71 seconds (8k tokens, batch size 1, A100), Wikitext PPL 20.70, and CR accuracy 51.04%. The operator sequence is: [D, F, M2, F, A, F, M2, F, D, F, M2, F, A, F, M2, F, D, F, M2, F, D, F, M2, F], with D = DeltaNet, M2 = Mamba2, A = Attention, F = FFN. This architecture uses a repeating pattern of two block types: DeltaNet-FFN-Mamba2-FFN and Attention-FFN-Mamba2-FFN, interleaved.
Parameter count optimization: The searched architecture has 497M parameters, decoding latency 16.94 seconds, PPL 23.06, and CR accuracy 49.23%. The operator sequence is different: it includes more attention modules and has greater depth (more operators). The paper notes: "the former [parameter-efficient architecture] generally includes more attention modules, which are parameter-efficient but decoding-unfriendly, and fewer Mamba2/DeltaNet modules, and has greater model depth, which is a parameter-efficient design choice."

Table 3 benchmarks the latency-optimized searched architecture against scaled pure models. All baselines are scaled to match the decoding latency of the searched architecture (by adjusting hidden size to increase parameter count until latency is equivalent). The searched architecture achieves:

PPL 20.70, compared to DeltaNet's 20.90 (the next best), Gated DeltaNet's 21.98, Mamba2+FFN's 21.43, and SWA's 23.33
CR accuracy 51.04%, compared to DeltaNet's 50.38%, Mamba2+FFN's 50.04%, and Gated DeltaNet's 49.99%
Parameter count 837M, less than DeltaNet's 852M and GLA's 862M

The paper attributes the improvement to two factors: "efficient operator combinations that allow for larger parameter counts under the same decoding latency, and the complementary roles played by the hybrid operators." The fact that the searched model achieves the best PPL and accuracy with fewer parameters than the DeltaNet and GLA baselines suggests that the operator combination itself (not just parameter scaling) is responsible for the gains.

Figure 5 visualizes the search trajectory. The x-axis is the number of search cycles; the y-axis is short-training perplexity. The trajectory shows a downward trend (improving PPL) as the search progresses, with diminishing returns in later cycles. The paper notes that "10 new architectures are sampled and evaluated in each cycle."

Appendix C.3 (Table 10) reports the parameter-efficiency search results benchmarked against 500M-parameter pure models. The searched architecture achieves PPL 23.06 and CR accuracy 49.23%, compared to the next best baseline (Gated DeltaNet): PPL 23.80 and accuracy 47.96%. The improvement is +1.21% CR accuracy and -0.74 PPL over the best pure baseline.

Weight Normalization Ablation (Section 3.3, Figures 7 and 8, Table 4)

Headline finding: Weight normalization consistently improves final convergence across model families, yielding an average +1.20% improvement in commonsense reasoning accuracy and -0.66 reduction in Wikitext perplexity.

Figure 7 shows validation loss trajectories for four model families (Llama, DeltaNet, Mamba2, and a hybrid architecture), each trained with and without weight normalization. The pattern is consistent: models without weight normalization converge faster in early training, but their convergence slows in later stages. Models with weight normalization converge more slowly initially but maintain a steadier convergence rate and surpass the baselines, achieving better final validation loss.

Figure 8 visualizes the mechanism for DeltaNet: the average element-wise gradient norm and the L2 norm of weight matrices during training. Weight normalization dramatically reduces the L2 norm of weights (to unit norm by construction) while slightly increasing the gradient norm. This combination — smaller weights, comparable or larger gradients — produces larger relative weight updates throughout training, particularly beneficial in late stages when the learning rate is small due to cosine decay.

Table 4 reports the quantitative improvements on 100B-token training:

Llama 1B: Without weight normalization: PPL 18.67, CR accuracy 53.81%. With weight normalization: PPL 18.03, CR accuracy 54.85%. Improvement: -0.64 PPL, +1.04% accuracy.
DeltaNet 1B: Without: PPL 18.86, 53.46%. With: PPL 18.19, 54.39%. Improvement: -0.67 PPL, +0.93% accuracy.
Mamba2 1B: Without: PPL 18.44, 53.30%. With: PPL 17.88, 54.71%. Improvement: -0.56 PPL, +1.41% accuracy.

The average improvement across the three model families is -0.66 PPL and +1.20% accuracy, with the paper noting "its general effectiveness as an add-on component."

Meta Tokens Ablation (Section 3.4, Table 5)

Headline finding: Prepending 256 learnable meta tokens consistently improves both language modeling and reasoning accuracy, with an average gain of +0.45% in CR accuracy and negligible overhead.

Table 5 reports:

Mamba2-48L-1B: Without meta tokens: PPL 19.04, CR accuracy 51.71%. With meta tokens: PPL 18.98, CR accuracy 52.33%. Improvement: -0.06 PPL, +0.62% accuracy.
DeltaNet-48L-1B: Without: PPL 19.60, 52.12%. With: PPL 19.47, 52.46%. Improvement: -0.13 PPL, +0.34% accuracy.
Searched hybrid 24L-830M: Without: PPL 20.61, 50.74%. With: PPL 20.49, 51.13%. Improvement: -0.12 PPL, +0.39% accuracy.

The gains are small but consistent across model types. The paper notes that meta tokens benefit linear attention by serving as "learned cache initialization when reformulating linear attention in a recurrent format during decoding," in addition to the attention sink mitigation effect established in prior work.

Final Nemotron-Flash Benchmark (Section 4, Tables 6 and 7, Figure 1)

Headline finding: Nemotron-Flash-3B achieves +5.5% higher average accuracy, 1.3× lower latency, and 18.7× higher throughput than Qwen3-1.7B.

Table 6 reports the main results across 16 tasks for Nemotron-Flash-1B and 3B against 11 baseline models. The domain-averaged accuracies and efficiency metrics are:

Nemotron-Flash-1B (0.96B parameters, depth 12, 24 operators, 2 full attention layers):

Decoding latency: 14.45 seconds (batch size 1, 8k tokens, H100 with CUDA Graph)
Throughput: 7289 tokens/second (maximum batch size without OOM, 32k input length)
Average accuracy: 49.63%
Domain breakdown: MMLU 44.63%, CR 54.46%, Math 34.86%, Coding 37.91%, Recall 67.11%

Comparing against Qwen3-0.6B (0.6B parameters, depth 28):

Accuracy: +5.52 percentage points (49.63% vs. 44.11%)
Latency: 1.91× lower (14.45s vs. 27.55s)
Throughput: 45.6× higher (7289 vs. 160 tok/s)

Comparing against Llama-3.2-1B (1.2B parameters, depth 16):

Accuracy: +8.18 percentage points (49.63% vs. 41.45%)
Latency: 1.48× lower (14.45s vs. 21.44s)
Throughput: 7.8× higher (7289 vs. 932 tok/s)

Nemotron-Flash-3B (2.7B parameters, depth 18, 36 operators, 3 full attention layers):

Decoding latency: 28.71 seconds
Throughput: 2939 tokens/second
Average accuracy: 60.98%
Domain breakdown: MMLU 61.19%, CR 61.02%, Math 57.62%, Coding 53.33%, Recall 73.25%

Comparing against Qwen3-1.7B (1.7B parameters, depth 28):

Accuracy: +5.51 percentage points (60.98% vs. 55.47%)
Latency: 1.26× lower (28.71s vs. 36.20s)
Throughput: 18.7× higher (2939 vs. 157 tok/s)

Comparing against Qwen2.5-3B (3B parameters, depth 36):

Accuracy: +2.02 percentage points (60.98% vs. 58.96%)
Latency: 1.72× lower (28.71s vs. 49.40s)
Throughput: 6.4× higher (2939 vs. 459 tok/s)

The paper also reports a throughput-optimized variant: Nemotron-Flash-3B-TP (Section 4.3) with a 1FA+2SWA attention configuration (one full attention layer, two sliding window attention layers with 8k window size):

Throughput: 4657 tokens/second, which is 1.58× higher than the standard 3FA variant (4657 vs. 2939)
Accuracy: 60.84%, essentially unchanged from the 3FA variant's 60.98%
Latency: 27.95s, comparable to 28.71s

The paper notes that this variant achieves "10.1× and 29.7× higher throughput compared to Qwen2.5-3B and Qwen3-1.7B, respectively."

Notable sub-domain results from Table 9:

Despite having only 3 full attention layers, Nemotron-Flash-3B achieves the highest recall accuracy (73.25%) among all models in the comparison, including attention-heavy models like Qwen2.5-3B (73.03%) and Llama-3.2-3B (70.01%). The paper highlights this as evidence that "maintaining full KV cache across all layers is unnecessary."
In coding tasks, Nemotron-Flash-3B achieves 53.33%, substantially outperforming all models except Qwen2.5-3B (49.46%) — it beats Qwen3-1.7B (43.76%) by 9.57 percentage points.
In math, Nemotron-Flash-3B achieves 57.62%, behind Qwen3-1.7B (53.71%) and Qwen2.5-3B (53.83%). It outperforms Qwen2.5-1.5B (48.14%) by 9.48 percentage points.

Instruct model benchmark (Table 7): Nemotron-Flash-3B-Instruct (instruction-tuned via two-stage SFT) is compared against SmolLM2-1.7B, Qwen2.5-1.5B, and Qwen3-1.7B instruct variants:

Nemotron-Flash-3B-Instruct achieves average accuracy 52.84% (averaged over MMLU, GPQA, GSM8K, IFEval), compared to Qwen2.5-1.5B's 48.17%, Qwen3-1.7B's 46.17%, and SmolLM2-1.7B's 45.27%.
Throughput is 4.3× higher than Qwen2.5-1.5B (2939 vs. 687 tok/s) and 18.7× higher than Qwen3-1.7B (2939 vs. 157 tok/s).
On GSM8K specifically, Nemotron-Flash-3B-Instruct scores 69.45%, compared to Qwen3-1.7B's 64.88% and Qwen2.5-1.5B's 56.03%.

Ablation Studies and Robustness Checks

Attention configuration ablation (Section 4.3, Table 8, Figure 9): The paper continuously pretrains the Nemotron-Flash-3B base model (originally with 3 full attention layers) for 25B tokens at 29k context length under three configurations: 3FA (three full attention), 2FA+1SWA (two full attention, one sliding window attention with 8k window), and 1FA+2SWA (one full attention, two sliding window attention). Table 8 reports:

Throughput increases substantially as FA layers are replaced by SWA: 3FA achieves 2939 tok/s, 2FA+1SWA achieves 3737 tok/s (1.27× higher), 1FA+2SWA achieves 4657 tok/s (1.58× higher than 3FA, 1.25× higher than 2FA+1SWA).
General benchmark accuracy is largely unaffected: 3FA: 60.98%, 2FA+1SWA: 60.69%, 1FA+2SWA: 60.84%. The differences are small (within 0.3 percentage points). Recall accuracy even slightly increases: 3FA: 73.25%, 2FA+1SWA: 73.37%, 1FA+2SWA: 73.77%.
Needle-in-a-haystack (NIAH) performance drops with fewer FA layers at longer contexts. Figure 9 shows Ruler benchmark NIAH scores: the 3FA and 2FA+1SWA configurations maintain performance at longer context lengths, but the 1FA+2SWA configuration drops "significantly at longer context lengths." The paper concludes: "we recommend maintaining at least two full attention layers even in SLMs."

Weight normalization vs. nGPT ablation (Appendix D.1, Table 12): The paper compares four training configurations on a Llama 1B and a hybrid 1B model: vanilla (no normalization), weight normalization only (the paper's method), nGPT without weight normalization (activation normalization only), and full nGPT (both weight and activation normalization). Key findings:

Weight normalization alone matches full nGPT in accuracy. Llama 1B: weight normalization 54.85% CR accuracy vs. full nGPT 54.73%. Hybrid 1B: 55.74% vs. 55.33%. The differences are negligible.
Weight normalization is essential to nGPT. Removing weight normalization from nGPT (keeping only activation norms) drops Llama 1B accuracy to 51.83% — substantially below the vanilla baseline of 53.81%. This is a negative result: activation normalization alone is harmful.
nGPT's activation normalizations add >30% training overhead. Llama 1B training time per iteration: vanilla 0.43s, weight normalization 0.46s (+7%), nGPT 0.59s (+37%). For the hybrid model: vanilla 0.78s, weight normalization 0.81s (+4%), nGPT 1.04s (+33%). The paper argues this overhead reduces the number of tokens that can be processed in a fixed training budget, giving weight normalization a practical advantage beyond the comparable accuracy.

Gradient norm evolution across models (Appendix D.2, Figure 10): The paper visualizes gradient norm trajectories for four model families (Llama, DeltaNet, Mamba2, and hybrid) with and without weight normalization. The consistent finding: weight normalization leads to "slightly increased gradient norm" throughout training, which — combined with the reduced weight norms (enforced by the projection) — means larger relative weight updates. This pattern holds across all four model types.

Token count reduction from larger vocabulary (Appendix E, Table 13): The paper compares the Llama2 tokenizer (small vocabulary) against the Mistral-NeMo-Minitron tokenizer (large vocabulary) on two datasets: AG News and Wikitext. The Mistral tokenizer reduces tokens per sample by 13.5% on AG News (55.08 vs. 63.64) and 9.3% on Wikitext (106.00 vs. 116.86). Table 13 also reports that the decoding latency gap between the two tokenizers at 8k tokens generation is only 5.8% (38.93s vs. 41.31s), meaning the token count reduction outweighs the per-step latency increase from the larger embedding layer. This supports the choice of the larger-vocabulary tokenizer as a net latency win.

Deployment flow validation (Appendix F, Table 14): The paper benchmarks a 550M Llama model across PyTorch, vLLM, TensorRT-LLM, and the paper's custom deployment flow (which integrates TensorRT-LLM's AutoDeploy for attention, Mamba2's official implementation, and FlashLinearAttention for linear attention variants, all wrapped in CUDA Graph). At 8,192-token generation with batch size 1 on an A100: PyTorch takes 113.93s, vLLM 17.82s, TensorRT-LLM 13.65s, and the paper's flow 17.54s. The paper's flow achieves a latency within ~28% of TensorRT-LLM for this pure-attention model while additionally supporting linear attention operators that vLLM and TensorRT-LLM do not support. At longer sequences (32,768 tokens), the paper's flow (71.65s) substantially outperforms vLLM (169.74s) and approaches TensorRT-LLM (60.41s).

Evolutionary search with parameter count as objective (Appendix C.3, Table 10): The parameter-efficiency search finds an architecture achieving PPL 23.06 and CR accuracy 49.23% at 500M parameters, compared to the best pure baseline (Gated DeltaNet): PPL 23.80 and 47.96%. This validates that the search can optimize for different metrics and produces structurally different architectures depending on the target (as noted in the main text discussion of Table 2).

Critical Assessment

The paper makes four central claims: (1) deep–thin architectures are not latency-optimal despite being parameter-efficient, (2) an augmented scaling law with decoupled depth and width terms enables principled latency-constrained architecture selection, (3) evolutionary search over hybrid operators discovers architectures that advance the accuracy–latency frontier beyond pure models and manually designed hybrids, and (4) weight normalization provides a training improvement that is both effective and more efficient than full nGPT. I examine each in light of the experiments.

Claim 1: Deep–thin models are not latency-optimal. This claim is supported with strong evidence from the controlled depth–width sweep in Figure 2. The experiment is well-designed: it trains models at five depth settings with multiple widths each, all on identical data (100B tokens of Smollm-corpus), and measures both parameter efficiency and latency on the same hardware. The finding that depth 12 outperforms depth 30 at a 3-second latency budget is concrete and replicable. The mechanism — per-layer overhead on GPUs — is plausible and well-motivated by hardware characteristics.

The limitation is the single hardware platform (A100 for the depth–width experiments, H100 for the final benchmark) and single batch size (1). The depth–width trade-off depends on hardware characteristics — a device with different kernel launch overhead or different parallelism characteristics might favor different depth–width ratios. The paper acknowledges this implicitly by framing the scaling law as a method that should be re-fitted for each deployment setting, but the general claim that "deep–thin is not latency-optimal" is based on experiments on NVIDIA GPUs specifically. The paper does not test on non-GPU hardware (mobile CPUs, edge TPUs, etc.), where the overhead characteristics might differ substantially.

Additionally, the comparison is between models trained from scratch on 100B tokens — a relatively small data budget. It is possible that with much larger training budgets (the 4.5T tokens used for the final models), the relative benefits of depth vs. width shift. The scaling law extrapolation (within 5.3% at larger depths/widths) provides some evidence that the trends hold, but this is extrapolation from 100B-token checkpoints and may not capture data-regime interactions.

Claim 2: Augmented scaling law enables principled depth–width selection. This claim is supported but the practical utility is not fully demonstrated. The extrapolation validation (Figure 3, within 5.3%) shows that the functional form generalizes, which is the critical requirement. However, the paper does not close the loop by showing that using the scaling law to select a depth–width ratio actually produces a better model than a reasonable heuristic (e.g., "use depth 12–18 for SLMs"). The scaling law is presented as a conceptual tool and a fitting exercise, but it is not used in an end-to-end experiment where: (a) a latency budget is specified, (b) the scaling law recommends a specific (D, W), (c) a model is built to that specification, and (d) it is compared against alternative (D, W) choices at the same latency. The final Nemotron-Flash models use architectures discovered by the evolutionary search, not by the scaling law directly. The paper states that "depth and width [are] configured based on the scaling laws in Sec. 3.1" for the final models, but does not provide the scaling law's specific recommendation or compare it against alternatives.

The fit uses perplexity on 100B tokens as the loss metric. The correlation between 100B-token perplexity and downstream task accuracy after 4.5T tokens of training (the final model scale) is not established. If perplexity rankings shift with continued training — and the paper's own Figure 4a shows that loss curves for different architectures can cross even between 10B and 100B tokens — then the scaling law fitted at 100B tokens may not reliably predict relative performance at 4.5T tokens.

Claim 3: Evolutionary search discovers superior hybrid architectures. This claim is well-supported for the specific search space and scale tested, but the generalizability is unclear. The searched architecture outperforms scaled pure baselines at matched latency (Table 3) and outperforms all 500M baselines at matched parameter count (Table 10). The search methodology is sound: the 88.8% Spearman correlation between short-training and full-training perplexity justifies using 10B-token proxies, and the aging evolution algorithm is a reasonable choice for this combinatorial space.

The primary limitation is the scale of the search. All searched models are trained on 100B tokens and evaluated at the 500M–830M parameter range. The final Nemotron-Flash models (1B and 3B) are trained on 4.5T tokens — 45× more data. The paper does not demonstrate that architectures that are optimal at 100B tokens remain optimal at 4.5T tokens. Architectural choices (operator ratios, depth, FFN counts) might interact with data scale — an architecture that outperforms at 100B tokens might saturate earlier or later than alternatives when scaled to trillions of tokens. The paper uses the searched 830M architecture as a template and scales it up to 1B and 3B by adding more blocks and increasing hidden size, but this assumes the architecture transfers across scales, which is not validated.

The search space also excludes some potentially important dimensions. It does not search over attention window sizes (fixed at 512 for the search, though the final models use full attention for a few layers and SWA is explored in the attention configuration ablation). It does not search over the ratio of full attention to linear attention within the hybrid blocks — the searched configuration always pairs operators as [Linear, FFN, Mamba2, FFN] or [Attention, FFN, Mamba2, FFN]. Alternative structures (e.g., parallel branches as in Hymba, different ordering, different ratios within blocks) might yield further improvements.

The comparison to baselines in Table 3 is against pure models scaled to match latency. This is a strong baseline, but it excludes manually designed hybrid models (Jamba-style, Hymba-style) that are also scaled to the same latency. Without this comparison, it is difficult to assess how much of the gain comes from "being hybrid" (which prior work already showed is beneficial) vs. from the specific operator combination discovered by the search.

Claim 4: Weight normalization is effective and more efficient than nGPT. This claim is strongly supported by the ablation in Table 12. The demonstration that weight normalization alone matches full nGPT accuracy while reducing training time by >20% is a clean, actionable finding. The mechanism — larger relative weight updates due to constrained norms — is supported by the gradient and weight norm trajectories in Figures 8 and 10, and the pattern of slower early convergence followed by better final convergence (Figure 7) is consistent with the proposed mechanism.

The limitation is that the ablation is performed on 1B-parameter models trained on 100B tokens. The relative benefits of weight normalization vs. vanilla training vs. nGPT might change with model scale and training duration. At very large scales (10B+ parameters) or very long training (trillions of tokens), the convergence dynamics might differ — the baseline might catch up given enough steps, or weight normalization might provide even larger benefits if weight magnitude growth becomes more severe. The paper does not explore this scaling dimension.

Additionally, the weight normalization requires omitting weight decay (since weights are projected to unit norm). The paper uses Adam without weight decay and tunes the learning rate for the no-normalization baseline, then applies weight normalization on top. This is a reasonable protocol for demonstrating the additive benefit, but it means the baseline is already well-tuned. In practice, weight normalization replaces one hyperparameter (weight decay) with another (the choice of normalization dimension: row-wise vs. column-wise for different matrix types). The paper's Case-1/Case-2 classification is intuitive but ad-hoc; a practitioner might misclassify a matrix and see different results.

The final benchmark (Table 6) supports the paper's overall thesis but has important caveats. The Nemotron-Flash models achieve better accuracy, lower latency, and higher throughput than all comparably-sized baselines. This is the paper's strongest result. However:

Throughput measurement methodology favors Nemotron-Flash. Throughput is measured at the maximum batch size without OOM for each model individually. Models with smaller memory footprints (and thus larger maximum batch sizes) will report higher throughput even if their per-token processing speed at matched batch sizes is comparable. Nemotron-Flash has only 2–3 full attention layers (and thus a small KV cache), dramatically increasing its maximum batch size relative to attention-heavy models like Qwen3-1.7B (28 full attention layers). This is a legitimate architectural advantage — the smaller KV cache is a direct benefit of the hybrid design — but the 18.7× throughput figure conflates this memory advantage with computational efficiency. A comparison at fixed batch sizes (or at the maximum batch size common to all models) would provide a complementary perspective.
Latency is measured with CUDA Graph, which disproportionately benefits shallow, operator-diverse models. CUDA Graph eliminates per-layer kernel launch overhead, which is proportionally larger for models with many small layers (like deep–thin attention models) and for batch size 1 (where there is no amortization across the batch). Nemotron-Flash's shallower depth (18 layers vs. 28–36 for competitors) and diverse operators might interact differently with CUDA Graph optimization than pure-attention deep models. The paper acknowledges that CUDA Graph is "critical for edge applications with a batch size of 1" but does not report latency without it, making it difficult to assess how much of the latency advantage comes from CUDA Graph optimization vs. inherent architectural efficiency.
The baselines use default configurations, not latency-optimized versions. The Qwen3, Llama-3.2, and SmolLM2 models are used as-is from their respective releases. These models were not designed for latency-optimal deployment — they are standard architectures (deep attention-only transformers) deployed with AutoDeploy and CUDA Graph. The paper demonstrates that Nemotron-Flash beats these models, which is valuable, but it does not demonstrate that the latency-optimal framework beats a strong latency-optimized baseline. What if one took a Qwen3-1.7B and simply reduced its depth (keeping total parameters roughly constant by increasing width) using the insights from Section 3.1? The paper does not perform this experiment, so the contribution of the hybrid operators relative to the depth–width optimization is not isolated.
The 16-task benchmark, while diverse, does not include long-context tasks except for the Ruler NIAH evaluation in the attention configuration ablation (Figure 9) and the recall tasks (FDA, SWDE, Squad). The Nemotron-Flash models with only 2–3 full attention layers might underperform on tasks requiring long-range retrieval or multi-document reasoning (beyond simple NIAH) compared to attention-heavy models. The Ruler results (Figure 9) showing that NIAH performance drops with fewer FA layers at long contexts is an early indicator of this trade-off. The paper acknowledges this limitation in Appendix G: "our architecture search... primarily uses language modeling as the search proxy, which may not fully capture other capabilities such as long-context understanding."
The instruct model benchmark (Table 7) is limited in scope. Only four tasks are evaluated (MMLU, GPQA, GSM8K, IFEval), and the baselines are SmolLM2-1.7B, Qwen2.5-1.5B, and Qwen3-1.7B. There is no comparison to Qwen2.5-3B Instruct or other 3B-class instruct models. The Nemotron-Flash-3B-Instruct beats the baselines handily (52.84% average vs. 46–48%), but it is a 2.7B model competing against 1.5B–1.7B models. The paper presents this as a fair comparison because Nemotron-Flash achieves better latency/throughput despite having more parameters, but the accuracy comparison is confounded by the parameter count difference.

Missing experiments that would strengthen the paper:

An end-to-end validation of the scaling law: Specify a latency budget, use the scaling law to recommend a (D, W) for a pure Llama model, build it, and show it outperforms alternative (D, W) choices at the same latency. This would close the loop on the paper's own framework.
A depth–width ablation of a hybrid model: The depth–width experiments use pure Llama models. The final Nemotron-Flash uses the hybrid architecture from the search. It is unclear whether the optimal depth–width ratio for hybrid models differs from that for pure attention models. The paper does not explore this interaction.
A latency-matched comparison against manually designed hybrids: Scale a Jamba-style or Hymba-style model to match Nemotron-Flash's latency and compare accuracy. This would isolate the contribution of the evolutionary search from the general benefit of hybrid architectures.
A fixed-batch-size throughput comparison: Report throughput at a batch size that all models can support (e.g., batch size 64 or 128) to separate memory-efficiency advantages from raw computational throughput.
Latency measurements without CUDA Graph: This would reveal how much of the latency advantage is due to CUDA Graph's overhead elimination vs. the architectural choices themselves. SLMs deployed on platforms without CUDA Graph support (mobile, web) would face different trade-offs.
Scaling the search to larger models and longer training: Validate that architectures discovered at 100B tokens and 500M parameters remain optimal when scaled to 4.5T tokens and 1B–3B parameters. This is a significant extrapolation that the paper does not verify.
An ablation where weight normalization is the only change in the final 4.5T-token training run. The weight normalization experiments are on 1B models at 100B tokens; the final models use it but the incremental benefit at 4.5T tokens is not isolated.

In summary, the paper demonstrates that latency-optimal design — considering depth–width ratios, hybrid operators, and training techniques — can produce SLMs that advance the accuracy–latency–throughput frontier compared to existing parameter-efficient models. The individual components (depth–width analysis, evolutionary search, weight normalization) are each validated in controlled experiments. The final integrated system (Nemotron-Flash) convincingly outperforms SOTA baselines. The primary gaps are: the scaling law is not validated in an end-to-end selection experiment, the search results at 100B tokens are assumed to transfer to 4.5T tokens without verification, the throughput measurement methodology favors the paper's models, and the contribution of each component to the final model's performance is not isolated.

6. Limitations and Trade-offs

Architecture Search Is Validated Only at Small Scale and Short Training, Not at the Final Model Scale

The assumption or constraint. The evolutionary search that discovers Nemotron-Flash's architecture operates entirely at the 500M–830M parameter scale, with candidate architectures evaluated using short-training perplexity on only 10B tokens (Appendix C.2). The paper states explicitly that "training and evaluating a sampled architecture takes approximately 2 hours using 32 NVIDIA A100 GPUs." The final Nemotron-Flash models are trained on 4.5T tokens — a 450× increase in data and a 2–6× increase in parameters over the searched configurations. The paper applies the searched architecture template (the specific interleaving pattern of DeltaNet, Mamba2, Attention, and FFN blocks) to the larger models by adding more blocks and increasing hidden size, but never validates that the architecture optimal at 100B tokens and ~800M parameters remains optimal at 4.5T tokens and 1B–3B parameters.

The consequence. Architecture rankings can shift with training duration. The paper's own evidence shows this: Figure 4a shows validation loss curves for different operators that are not perfectly parallel — some models converge faster early but plateau, while others converge slower but continue improving. If operator rankings shift between 10B and 100B tokens (within the regime the paper measures), they could plausibly shift further between 100B tokens and 4.5T tokens. An architecture that outperforms at 100B tokens might saturate earlier than alternatives at trillion-token scales, or a different operator ratio might become optimal when models are trained to full convergence. The paper assumes architectural transferability across scale without measuring it. A practitioner adopting the searched architecture for a model trained on trillions of tokens is relying on an extrapolation that the paper does not validate.

What evidence exists in the paper. The Spearman rank correlation of 88.8% between 10B-token and 100B-token perplexity (Section 3.2) validates the short-training proxy for the specific scale range tested. The 5.3% extrapolation error for the scaling law (Figure 3) validates that the depth–width functional form generalizes, but this is for pure Llama models — not for hybrid operator combinations. The paper reports no correlation between 100B-token and 4.5T-token performance for any architecture, nor any comparison of the searched architecture scaled to larger sizes against alternative architectures also scaled to larger sizes. The search results (Tables 2 and 3) are entirely at the 100B-token, 500M–830M scale.

Mitigation status. The paper does not address this limitation. Appendix G acknowledges a different scaling limitation (that the search "primarily uses language modeling as the search proxy, which may not fully capture other capabilities such as long-context understanding") but does not discuss the scale gap between search and deployment. The paper treats the searched architecture as a fixed template and scales it up, with no discussion of whether this is valid or what alternatives were considered.

Throughput Measurements Use Per-Model Maximum Batch Size, Conflating Memory Efficiency with Computational Throughput

The assumption or constraint. Throughput in Table 6 is measured using "the maximum batch size that does not cause out-of-memory (OOM) errors for each model individually" with a 32k-token input length. This means each model is evaluated at a different batch size — one determined by its memory footprint. Models with smaller KV caches (because they have fewer full attention layers) can fit larger batches in memory and thus report higher tokens-per-second, even if their per-token processing speed at a fixed batch size is identical to or slower than competing models. Nemotron-Flash has 2–3 full attention layers; Qwen3-1.7B has 28; Qwen2.5-3B has 36. The KV cache size difference between these models at 32k context length is enormous — roughly 10× or more — meaning Nemotron-Flash's maximum batch size is far larger than its competitors'.

The consequence. The headline throughput numbers (18.7× higher than Qwen3-1.7B, 45.6× higher than Qwen3-0.6B) conflate two separate advantages: (1) Nemotron-Flash genuinely processes tokens faster due to efficient operators and shallow depth, and (2) Nemotron-Flash fits more concurrent requests in memory due to its tiny KV cache. For a practitioner, these represent different deployment scenarios. If the deployment serves a single request at a time (batch size 1, typical for interactive edge applications), the memory advantage is irrelevant and the true throughput advantage is much smaller — likely closer to the latency advantage (1.3–1.9×). If the deployment batches many requests (server-side inference), the memory advantage is real and valuable, but the 18.7× figure cannot be decomposed into "how much faster per token" vs. "how many more tokens in parallel." A fixed-batch-size comparison (e.g., all models at batch size 64 or 128) would isolate computational throughput from memory capacity.

What evidence exists in the paper. Table 6 reports the maximum batch size and resulting throughput separately: Qwen3-1.7B runs at batch size 157 yielding 157 tok/s; Nemotron-Flash-3B runs at batch size 2939 yielding 2939 tok/s. This is an 18.7× throughput ratio, but the batch size ratio is 18.7× (2939/157). At batch size 1, the latency advantage is only 1.26× (28.71s vs. 36.20s). The near-identity between the batch size ratio and the throughput ratio strongly suggests that the throughput advantage is almost entirely driven by the larger maximum batch size, not by faster per-token processing. The paper never reports throughput at a fixed batch size common to all models. The attention configuration ablation (Table 8) reports throughput for the same model with different attention configurations, all using the same architecture, and the throughput improvements (1.3–1.6×) are much more modest — consistent with the interpretation that the 18.7× figure primarily reflects KV cache reduction rather than computational speedup.

Mitigation status. The paper does not address this conflation. It reports both maximum batch size and throughput in the same table, so an attentive reader can infer the relationship, but the abstract and introduction quote the throughput ratios without this caveat (e.g., "18.7×/45.6× higher throughput compared to Qwen3-1.7B/0.6B" in the abstract). The paper does not report or discuss fixed-batch-size throughput, which would provide the missing decomposition.

The Scaling Law Is Not Validated in an End-to-End Model Selection Experiment

The assumption or constraint. The augmented scaling law (Equation 1) is presented as a principled method for selecting the optimal depth–width ratio given a latency budget. The paper fits the law on a subset of Llama configurations trained on 100B tokens, validates extrapolation to held-out larger configurations (within 5.3% PPL error, Figure 3), and states that the final Nemotron-Flash models' "depth and width [are] configured based on the scaling laws in Sec. 3.1." However, the paper never closes the loop by: (a) specifying a concrete latency budget, (b) using the scaling law to select a (D, W) pair, (c) building and fully training a model to that specification, and (d) showing that it outperforms alternative (D, W) choices at the same latency. The scaling law is used as a conceptual guide, not as a decision tool with measured outcomes.

The consequence. A practitioner who wants to use the scaling law to design their own latency-optimal SLM cannot assess its reliability from the paper. The 5.3% extrapolation error in perplexity is on 100B-token checkpoints, but the relevant question is whether a model selected by the scaling law achieves better downstream task accuracy at a given latency than a model selected by a simpler heuristic (e.g., "use depth 12–18 for 1–3B models, as the paper does"). Without an end-to-end experiment, there is no evidence that the scaling law's recommendation is better than the heuristic the paper implicitly uses. The paper's own final models use the searched hybrid architecture — not a pure Llama architecture — so the depth–width choices are confounded with the operator choices. The contribution of the scaling law to the final model's performance cannot be isolated from the contribution of the hybrid operators, the evolutionary search, the weight normalization, or the training curriculum.

What evidence exists in the paper. The paper provides the fitting and extrapolation validation (Figure 3) and the qualitative observations from the depth–width sweep (Figure 2): that deeper is better for parameters, that optimal depth increases with latency budget, and that depth 12 outperforms depth 30 at a 3-second budget. These are informative but descriptive — they characterize the trade-off landscape, they do not demonstrate a selection method. The paper states that the final models are "configured based on the scaling laws" (Section 4) but does not report what (D, W) the scaling law recommended, what alternatives were considered, or what the predicted vs. actual accuracy was for the selected configuration. Appendix G does not mention this gap.

Mitigation status. The paper does not acknowledge this as a limitation. The scaling law is presented as a validated contribution ("fitting a scaling law with depth and width provides a more principled approach to identifying the sweet-spot ratios," Section 3.1 lesson), but the validation stops at extrapolated perplexity prediction, not at the downstream task of model selection. This is a missing experiment rather than an acknowledged limitation.

The Method Provides No Guidance for Long-Context Capabilities, and the Search Proxy May Not Capture Them

The assumption or constraint. The architecture search uses language modeling perplexity on relatively short sequences as the proxy metric for ranking architectures. The paper acknowledges in Appendix G: "In our architecture search for SLMs, we primarily use language modeling as the search proxy, which may not fully capture other capabilities such as long-context understanding." The search also uses a sliding window attention with window size 512 for attention layers during architecture evaluation, which is sufficient for the commonsense reasoning tasks used to validate the proxy but is explicitly restrictive for long-range dependencies. The final Nemotron-Flash models include full attention layers and are trained with a 29k context length for the final 25B tokens, but the architecture itself was selected without consideration of long-context performance.

The consequence. Nemotron-Flash's architecture — with only 2–3 full attention layers — makes an implicit trade-off: it prioritizes latency and throughput by minimizing KV cache size, at the potential cost of long-range retrieval and reasoning capabilities. The attention configuration ablation (Section 4.3, Figure 9) provides direct evidence of this trade-off: reducing from 3 full attention layers to 1 causes NIAH performance to "drop significantly at longer context lengths." The paper's recommendation to maintain "at least two full attention layers even in SLMs" is based on this NIAH evaluation, but NIAH is a narrow test of retrieval capability. Tasks requiring multi-hop reasoning across long contexts, long-document summarization, or maintaining coherence over very long generations are not evaluated. A practitioner deploying Nemotron-Flash for long-context applications (document QA, codebase understanding, long-form generation) cannot predict from the paper whether the 2–3 attention layers are sufficient or whether a different hybrid ratio (more attention, less Mamba2/DeltaNet) would be necessary.

What evidence exists in the paper. The recall tasks (FDA, SWDE, Squad) in Table 6 show that Nemotron-Flash-3B achieves competitive recall accuracy (73.25%, highest among all models) despite having only 3 full attention layers. However, these tasks operate on relatively short contexts and test factual extraction rather than long-range dependency tracking. The NIAH results (Figure 9) directly show the attention-dependence of long-context retrieval: the 2FA+1SWA configuration maintains performance at long contexts, but 1FA+2SWA degrades. The paper does not evaluate on standard long-context benchmarks (e.g., LongBench, L-Eval, ZeroSCROLLS) that test capabilities beyond needle retrieval. The search proxy limitation (Appendix G) is acknowledged but not measured — there is no comparison showing whether architectures ranked differently by short-context PPL would be ranked differently by long-context metrics.

Mitigation status. The paper partially addresses this through the attention configuration ablation (Section 4.3) and the resulting recommendation (keep at least 2 full attention layers). The NIAH results on Ruler (Figure 9) quantify the retrieval degradation when attention layers are removed. However, the search framework itself remains unmodified — it does not incorporate long-context metrics into the architecture selection process. The acknowledged limitation in Appendix G is stated as future work ("We will extend our search framework to optimize for these aspects in future work") but not addressed in the current paper. The final models use 2–3 full attention layers, which the paper's own evidence suggests is the minimum viable configuration for long-context retrieval, but the optimal number for tasks beyond NIAH is unknown.

Weight Normalization Is Evaluated Only at Small Scale, and Its Interaction with Large-Scale Training Is Unmeasured

The assumption or constraint. The weight normalization experiments (Section 3.3, Figures 7–8, Table 4) are conducted on 1B-parameter models trained on 100B tokens. The final Nemotron-Flash models (1B and 3B, trained on 4.5T tokens) use weight normalization, but the paper does not report an ablation isolating its contribution at that scale. The mechanism by which weight normalization improves training — larger relative weight updates in late training when learning rates are small — depends on the interaction between the learning rate schedule, the total training duration, and the growth of weight magnitudes. These dynamics could change substantially when training is extended from 100B tokens to 4.5T tokens.

The consequence. The paper cannot quantify how much of Nemotron-Flash's final performance is attributable to weight normalization vs. the architectural improvements, the data curriculum, the larger tokenizer, or the meta tokens. If weight normalization's benefit saturates or diminishes at trillion-token scales (e.g., because weight magnitudes stabilize naturally with longer training, or because the cosine learning rate schedule already reaches near-zero values where relative update size is less relevant), then the technique may be less important than the ablation experiments suggest. Conversely, if weight magnitudes continue to grow with longer training, weight normalization might be more important at 4.5T tokens than at 100B tokens. Neither case is measured. A practitioner deciding whether to adopt weight normalization for their own large-scale training cannot assess its importance from the paper's evidence.

The connection between weight normalization and the omission of weight decay is also untested at scale. The paper uses Adam without weight decay "due to the use of weight normalization" (Section 4). This is a consequential change — weight decay is standard in most LLM training recipes — and the paper does not compare weight normalization + no weight decay against weight decay alone at the 4.5T token scale. It is possible that simply tuning weight decay would achieve similar benefits to weight normalization with a simpler implementation.

What evidence exists in the paper. The 100B-token experiments (Table 4) show consistent improvements: +1.20% average CR accuracy and -0.66 average PPL across three model families. The nGPT comparison (Table 12, Appendix D.1) shows weight normalization matching full nGPT. These are convincing at the 100B-token scale. The gradient norm analysis (Figure 8) and weight distribution visualization (Figure 6) provide mechanistic evidence for why weight normalization helps. But none of this is reproduced or validated at 4.5T tokens. The final model training (Section 4) mentions weight normalization as part of the recipe but provides no ablation comparing with and without it at the final scale. Table 6 reports only the final Nemotron-Flash numbers, with no "without weight normalization" baseline.

Mitigation status. The paper does not address this scale gap. The weight normalization section (3.3) presents the technique as generally applicable and validated "across model families," but the validation is at a single training scale. The paper does not claim to have validated weight normalization at 4.5T tokens, nor does it acknowledge this as a limitation. The technique is presented as a drop-in component whose benefits at 100B tokens are assumed to transfer to much larger training runs.

All Experiments Use a Single Model Family and Training Paradigm, with No Evidence of Transfer to Other Base Architectures or Training Recipes

The assumption or constraint. Every experiment in the paper — the depth–width sweep, the operator comparison, the evolutionary search, the weight normalization ablation, and the final Nemotron-Flash training — uses the same fundamental training paradigm: models trained from scratch on the Smollm-corpus (or its extended version including Zyda2, Climb-Mix, etc.) using the Adam optimizer with a cosine learning rate schedule. The base architecture throughout is a decoder-only transformer (or its hybrid variants). The paper makes general claims about SLM design principles — "latency-optimal depth–width ratios," "hybrid operators advance the accuracy–latency frontier," "weight normalization improves convergence across model families" — but "across model families" in this context means Llama, DeltaNet, Mamba2, and hybrids built from the same basic decoder-only template with the same training data and optimizer. It does not mean across fundamentally different architecture families (encoder-decoder, mixture-of-experts, etc.) or training paradigms (distillation, multi-stage pretraining with different objectives, etc.).

The consequence. A practitioner using a different training recipe — for instance, a different optimizer (LION, Adafactor), a different learning rate schedule (constant with cooldown, WSD), or a multi-stage curriculum substantially different from the paper's — cannot assume that the paper's specific findings will hold. The optimal depth–width ratio might shift if a different optimizer changes convergence dynamics. The best hybrid operator combination might differ if training data distribution favors different inductive biases. Weight normalization's benefit might be larger or smaller depending on how weight magnitudes evolve under different optimizers. The paper's framework is validated within a specific training setup, and the generalizability of its specific recommendations (e.g., "DeltaNet-Mamba2-FFN blocks," "depth 12 for 1B models at tight latency budgets," "keep at least 2 full attention layers") to other training paradigms is unmeasured.

The paper also does not compare against distillation-based approaches to building SLMs — where a smaller model is trained to mimic a larger teacher — which is a common and often effective strategy. The Nemotron-Flash models are trained from scratch on data alone, with no distillation signal. It is unknown whether a distilled version of a standard architecture (e.g., a distilled Qwen2.5-3B) might achieve comparable or better accuracy–latency trade-offs than Nemotron-Flash without requiring hybrid operators. The paper benchmarks against publicly released models that may or may not use distillation, but does not control for this factor.

What evidence exists in the paper. All training uses Smollm-corpus or its derivatives. All architectures are decoder-only transformers or their hybrid variants. All optimization uses Adam (or AdamW for non-weight-normalized baselines) with cosine decay. The paper demonstrates consistency within this paradigm — depth–width effects hold across different layer counts, operator diversity is explored across many types, weight normalization helps across Llama, DeltaNet, Mamba2, and hybrid variants. But there is zero evidence testing any of these findings under a different training paradigm. The comparison to external baselines (Table 6) uses models trained with unknown or different recipes (Qwen, Llama, SmolLM), which provides some indirect evidence of transfer — Nemotron-Flash beats models trained differently — but this is a confounded comparison (different architectures, different data, different tokenizers, different everything). It does not isolate whether the paper's design principles would help within those alternative training paradigms.

Mitigation status. The paper does not discuss this limitation. It frames its contributions as generalizable principles ("actionable insights and guidelines," Section 5) and demonstrates them within a consistent experimental framework, but does not address whether the principles would transfer to substantially different training setups. The paper's claims are most accurately interpreted as "within the standard decoder-only SLM training paradigm we tested, these techniques work" rather than "these are universal principles of SLM design." The distinction matters because a practitioner whose training setup differs in important ways (optimizer, data distribution, architecture family) may find that the specific recommendations do not transfer.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around small language model design from a parameter-count-centric optimization to a latency-centric one. The shift is not merely a change of metric — it is a reframing of the design objective that produces structurally different models. The paper demonstrates, with controlled experiments, that designing for parameter efficiency (the dominant paradigm, exemplified by MobileLLM and SmolLM) and designing for latency efficiency lead to qualitatively different architectural choices: deep–thin models are parameter-efficient but latency-inefficient; wide–shallow models with carefully chosen hybrid operators achieve better accuracy per millisecond despite having more parameters. The two different optimization targets produce models that lie on different frontiers, and optimizing for the wrong one — as the field has largely done — leaves substantial performance on the table.

The magnitude of this reframing is substantial but bounded. It is not a paradigm shift that invalidates prior work; parameter-efficient design remains appropriate when total model size (memory footprint, storage, download bandwidth) is the binding constraint. Rather, it is a correction to an implicit assumption — that fewer parameters naturally means faster inference — that the SLM literature had largely accepted without systematic hardware-aware validation. The paper provides the systematic validation and shows the assumption is wrong in a directional, predictable way.

The paper resolves a latent tension in the SLM literature that the field may not have recognized as a tension. Prior work on deep–thin architectures (MobileLLM) and prior work on hybrid models (Jamba, Hymba, Samba) were optimizing different things without explicit acknowledgment. MobileLLM optimized parameters and found deep–thin to be best. Jamba and Hymba optimized for efficiency but through manual operator mixing without systematic depth–width consideration. This paper provides the unified framework: the correct optimization target is deployment latency (or throughput), and achieving it requires joint optimization of depth, width, and operator composition, because these factors interact in hardware-dependent ways. The consequence is that architecture design cannot be separated from deployment context. A model family intended for diverse hardware targets (edge GPU, server GPU, mobile CPU) may need different depth–width ratios and operator compositions for each, rather than a single parameter-efficient architecture deployed everywhere.

The paper also redirects research attention from inventing new efficient operators toward understanding how to combine existing ones. The operator comparison (Figure 4) shows that DeltaNet and Gated DeltaNet are on the Pareto frontier, and the hybrid experiments (Table 1) show that combinations consistently outperform pure models. But the evolutionary search demonstrates that the specific combination matters — different search targets produce different architectures — and that manual specification, the approach of prior hybrid model work, is unlikely to find optimal configurations in a growing combinatorial space. This suggests that architecture search, not operator invention, is the more leveraged research direction for improving SLM efficiency in the near term.

The identification of weight normalization as the active mechanism in nGPT (Appendix D.1, Table 12) is a smaller but practically significant contribution. It isolates a training technique that is both effective and cheap, by showing that nGPT's activation normalizations add >20% training overhead without improving accuracy beyond what weight normalization alone achieves. This should reduce the incentive to adopt full nGPT for SLM training, saving researcher and engineering effort. It also opens the question of whether other "all-computations-on-a-sphere" techniques can be similarly decomposed into simpler, more targeted interventions.

Follow-Up Research This Work Enables

1. Validating the architecture search at the deployment scale, with a controlled scale-transfer experiment. The paper's evolutionary search operates at 500M–830M parameters and 100B training tokens, but the final Nemotron-Flash models are trained on 4.5T tokens — a 45× data increase. The most important open question is whether architectures optimal at small scale remain optimal at large scale. A direct experiment: take the top-5 architectures discovered by the search at 100B tokens, scale them all to 1B parameters and train on 1T tokens (or more), and measure whether their relative ranking is preserved. If rankings invert — if an architecture that was 3rd-best at 100B tokens becomes 1st-best at 1T tokens — this would invalidate the paper's implicit scale-transfer assumption and force a redesign of the search protocol (e.g., using larger proxy models, or incorporating scaling trend extrapolation into the fitness function). If rankings are preserved, it would validate the methodology and make architecture search at small scale a reliable tool for large-scale model design.

2. Extending the search to include attention window sizes and parallel-branch configurations, with long-context metrics in the fitness function. The current search uses a fixed attention window of 512 and a sequential (not parallel) operator arrangement within blocks. The attention configuration ablation (Section 4.3) shows that replacing full attention with sliding window attention improves throughput substantially (1.6×) with minimal accuracy loss on general benchmarks, but that NIAH performance degrades with fewer full attention layers at long contexts. A richer search space would include attention window size as a searchable parameter per attention layer, would consider parallel operator branches (as in Hymba), and would incorporate a long-context metric (e.g., Ruler NIAH score at 32k+ context) into a multi-objective fitness function. The goal would be to discover architectures that maintain long-context retrieval capability while still achieving latency and throughput gains — potentially by allocating different attention window sizes to different layers rather than using a uniform window. The paper's recommendation of "at least 2 full attention layers" is a coarse guideline; a search could discover more nuanced configurations (e.g., 1 full attention + 2 SWA with different window sizes at different depths).

3. Stress-testing weight normalization at 1T+ training tokens across diverse model scales and optimizer configurations. The paper's weight normalization results are limited to 1B models at 100B tokens. The mechanism — larger relative weight updates in late training — should, in principle, help at any scale, but the magnitude of the benefit may change. A systematic study would apply weight normalization to models at 1B, 3B, and 7B scales, trained on 1T+ tokens, comparing against both vanilla AdamW and full nGPT. The key measurements: (a) whether the accuracy benefit scales with model size (does a 7B model gain more or less than a 1B model?), (b) whether the benefit persists or diminishes at very long training horizons (do vanilla-trained models eventually catch up?), and (c) interaction with learning rate schedule (does weight normalization matter more with cosine decay than with constant+cooldown schedules?). This would establish the technique's practical importance — if the benefit shrinks to a few tenths of a percent at 7B/1T-token scale, it may not be worth the implementation complexity; if it grows, it becomes a critical training ingredient for large SLMs.

4. A latency-matched comparison of the entire framework against strong latency-optimized baselines, decomposing the contribution of each component. The paper demonstrates that Nemotron-Flash outperforms standard SOTA SLMs, but it does not isolate how much of the gain comes from (a) the depth–width ratio shift (shallower and wider vs. deep–thin), (b) the hybrid operators (DeltaNet–Mamba2 combinations vs. pure attention), (c) the specific operator arrangement discovered by search (vs. a naive interleaving), (d) weight normalization, (e) the larger-vocabulary tokenizer, and (f) the training data curriculum. A decomposition experiment would start with a standard deep–thin pure-attention baseline (e.g., a Llama-style model at the same latency as Nemotron-Flash), then add each component incrementally, measuring the accuracy and efficiency at each step. This would reveal which components are load-bearing and which are marginal, guiding practitioners on where to invest effort when adopting the framework. The scaling law would also be validated end-to-end by using it to select depth–width for a pure attention model and comparing against the paper's heuristic choices.

5. Evaluating Nemotron-Flash and the search framework on long-context benchmarks beyond NIAH, and on generation tasks beyond perplexity. The paper's evaluation is strong on standard short-context benchmarks but thin on long-context capabilities (only NIAH in Figure 9 and recall tasks in Table 6) and entirely missing on generation quality metrics (no evaluation of long-form generation coherence, summarization quality, or instruction-following at long contexts). Extending the evaluation to LongBench, ZeroSCROLLS, or similar multi-task long-context benchmarks would reveal whether the 2–3 full attention layers in Nemotron-Flash impose a meaningful accuracy penalty on tasks requiring multi-hop reasoning, summarization, or temporal reasoning over long documents. A negative result — Nemotron-Flash substantially underperforms attention-heavy models on these tasks — would define the boundary of the architecture's applicability and motivate extending the search to include long-context metrics. A positive result — competitive performance despite few attention layers — would strengthen the claim that linear attention mechanisms can substitute for full attention even on retrieval-heavy tasks, provided the hybrid configuration is well-designed.

6. Porting the framework to non-NVIDIA hardware and non-GPU deployment targets, measuring whether the depth–width and operator recommendations transfer. The paper's latency measurements are exclusively on NVIDIA A100 and H100 GPUs. Per-layer overhead, kernel launch costs, and the relative efficiency of different operators depend on hardware characteristics that vary substantially across platforms — mobile GPUs, Apple Neural Engine, Qualcomm Hexagon, Intel integrated graphics, edge TPUs, and server CPUs all have different concurrency, memory bandwidth, and kernel launch overhead profiles. A replication of the depth–width sweep and the operator comparison on at least one non-NVIDIA platform (e.g., an Apple M-series chip for edge deployment, or an Intel Xeon for server CPU inference) would test the generalizability of the paper's core findings. It is plausible that on a CPU with no kernel launch overhead and different parallelism characteristics, deep–thin models might regain their advantage, or that different operators would emerge on the Pareto frontier. This would determine whether the paper's framework is GPU-specific or truly general, and would produce hardware-specific guidelines valuable for the many SLM deployments that do not use NVIDIA server GPUs.

Practical Applications and Downstream Use Cases

On-device language models for real-time interaction. For applications requiring sub-second latency on consumer devices — voice assistants, real-time translation, keyboard autocomplete, accessibility tools — the 1.9× lower decoding latency of Nemotron-Flash-1B vs. Qwen3-0.6B (14.45s vs. 27.55s for 8k tokens on an H100) translates directly to a faster user experience. More importantly, the depth–width principle means that for even tighter latency budgets (e.g., <1 second for short responses, typical of on-device voice interaction), models should be even shallower than Nemotron-Flash-1B's 12 layers — potentially 6–8 layers with proportionally larger width. The scaling law provides a recipe for determining the right ratio for any specific device and latency target, making the framework actionable for product teams building on-device SLMs.

High-throughput batch inference for data processing pipelines. The 18.7× throughput advantage of Nemotron-Flash-3B over Qwen3-1.7B (2939 vs. 157 tok/s at maximum batch size) matters most in settings where large volumes of text must be processed — document classification, entity extraction, content moderation, synthetic data generation. The throughput advantage comes primarily from the dramatically smaller KV cache (3 attention layers vs. 28), which allows fitting 18.7× more concurrent requests in GPU memory. For a cloud service processing billions of tokens per day, this directly reduces the number of GPU-hours required, cutting costs proportionally. The throughput-optimized variant (1FA+2SWA, achieving 4657 tok/s at essentially unchanged accuracy) pushes this further, suggesting that even the base Nemotron-Flash architecture is not throughput-optimal — further attention-to-SWA substitutions could yield additional gains for workloads where long-context retrieval is not required.

Latency-sensitive edge server deployment with strict power or cost constraints. Many enterprise deployments — customer support chatbots, internal knowledge base QA, code completion in IDEs — run on edge servers or small GPU clusters where hardware is provisioned for peak load. The combination of lower latency per request (serving more users per GPU) and higher throughput (processing more total volume per GPU) means that a fixed hardware budget can serve a larger user base or handle higher query volumes with Nemotron-Flash than with comparably accurate attention-heavy models. Concretely: a deployment that currently provisions 8 H100 GPUs to serve a Qwen3-1.7B-based application at target latency could potentially serve the same load with 1 GPU running Nemotron-Flash-3B (a ~8× reduction, conservatively, based on the throughput ratio adjusted for batch-size effects), or serve 8× more traffic with the same hardware. For teams evaluating TCO (total cost of ownership), the paper's latency and throughput numbers provide directly actionable estimates.

Self-improvement and synthetic data pipelines where inference cost dominates. When using SLMs to generate training data for larger models (distillation) or for iterative self-improvement (STaR-style bootstrapping), the inference cost of generating candidate solutions often dominates the total compute budget. Nemotron-Flash's throughput advantage means that for a fixed generation budget, more candidate solutions can be produced, potentially improving the quality of the resulting training data through better coverage or more aggressive filtering. The recall accuracy in Table 6 (73.25% for Nemotron-Flash-3B vs. 73.03% for Qwen2.5-3B) suggests that the model is not sacrificing factual extraction quality for speed, making it viable for data generation tasks that require accurate extraction from source documents. This use case benefits especially from the combination of high throughput and competitive accuracy — the paper's evidence that the accuracy–efficiency frontier has genuinely been advanced, not just that a fast-but-stupid model has been built.