Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

ArXiv: 2604.12374

🎯 Pitch

A 120B-parameter model with only 12B active parameters per token uses a new 'LatentMoE' architecture to slash inference cost while matching the accuracy of 10× larger dense rivals. It achieves up to 7.5× higher throughput than Qwen3.5-122B and introduces the first stable large-scale pretraining in NVFP4 precision.

---

1. Executive Summary

This paper introduces Nemotron 3 Super, a 120B total parameter (12B active) hybrid Mamba-Attention Mixture-of-Experts model optimized for agentic reasoning, and is the first model in the Nemotron 3 family to combine three architectural innovations: NVFP4 pretraining across 25 trillion tokens, LatentMoE — a new MoE architecture that projects tokens into a lower-dimensional latent space for routing and expert computation, using the resulting memory and communication savings to increase both the total number of experts and the top-K active experts per token (e.g., trading dimension reduction for more experts at constant inference cost) — and Multi-Token Prediction (MTP) layers that enable native speculative decoding through shared-weight autoregressive draft heads. The post-trained model achieves comparable accuracy to GPT-OSS-120B and Qwen3.5-122B across reasoning, agentic, and long-context benchmarks while delivering up to 2.2× and 7.5× higher inference throughput, respectively, on the 8K input / 64K output setting. The paper further demonstrates stable large-scale NVFP4 pretraining, quantized deployment checkpoints (FP8 and NVFP4) that preserve 99.8% median accuracy relative to the BF16 baseline, and a multi-stage RL post-training pipeline spanning 21 verifiable-reward environments and a dedicated SWE-RL stage for end-to-end software engineering tasks, establishing that sparse hybrid architectures can match dense model quality while substantially improving serving efficiency only when the architecture is co-designed with hardware-aware expert routing and low-precision training.

2. Context and Motivation

The Core Problem: How to Build Large Language Models That Are Both Accurate and Efficient to Serve

The fundamental challenge this paper addresses is the tension between model quality and inference efficiency in large language models (LLMs). Over the past several years, the field has observed a consistent pattern: scaling model parameters improves benchmark accuracy, but each additional parameter increases the computational cost of serving the model in production. This tension is particularly acute for models deployed in interactive or agentic settings — software engineering assistants, terminal-use agents, multi-turn conversational tools — where latency and throughput directly determine whether a model is practically usable or prohibitively expensive.

The paper frames this tension through a concrete metric: accuracy per unit of inference cost, where inference cost encompasses FLOPs, memory bandwidth, communication overhead in distributed serving, and memory footprint. A model that achieves high benchmark scores but requires minutes to generate a single tool call in a coding agent loop is not practically better than a slightly less accurate model that responds in seconds. This is not merely an engineering optimization problem; it reflects a structural inefficiency in how modern LLM architectures allocate parameters and compute.

Three specific trends in the LLM landscape motivate this work:

First, the shift toward Mixture-of-Experts (MoE) architectures has been driven by the observation that sparse activation patterns can decouple total parameter count from per-token FLOPs. Models like DeepSeek-V3 (DeepSeek-AI, 2025c), Qwen3 (Yang et al., 2025), and GLM-4.5 (GLM-4.5-Team, 2025) demonstrate that MoE models with, say, 120B total parameters but only 10–12B active parameters can match or exceed the quality of dense models with comparable active parameter counts, because the total parameter budget provides greater representational capacity even though only a fraction is used per token. However, the paper identifies a gap in how existing MoE architectures are designed: they optimize primarily for accuracy per FLOP (reducing active computation while expanding total capacity), but largely neglect accuracy per parameter — the memory bandwidth, sharding overhead, and all-to-all communication costs that dominate real-world inference latency. This is the gap that motivates LatentMoE, the first of the paper's architectural innovations.

Second, the emergence of state-space models (SSMs) as alternatives to self-attention has demonstrated substantial throughput gains, particularly for long-context workloads. The quadratic growth of the key-value (KV) cache in self-attention layers becomes a dominating bottleneck at sequence lengths beyond 8K–32K tokens. Models like Mamba-2 (Dao & Gu, 2024) replace self-attention with structured state-space computations that maintain a constant-size recurrent state during generation, eliminating the KV cache bottleneck entirely. Prior work in the Nemotron family — specifically Nemotron 3 Nano (NVIDIA, 2025c) — showed that a hybrid architecture interleaving Mamba-2 blocks with a small number of strategically placed self-attention layers could preserve global dependency modeling (via the attention "anchors") while offloading the majority of computation to the cheaper Mamba layers. Nemotron 3 Super scales this hybrid design to a much larger parameter count, combining it with MoE sparsity. The motivation is straightforward: if Mamba alone reduces memory overhead and attention alone provides global modeling, an architecture that thoughtfully combines both with sparse expert selection could push the throughput-quality frontier further than either approach alone.

Third, the growing importance of agentic workloads — where models must reason across multiple turns, issue tool calls, execute commands, and interact with environments over long contexts — makes inference efficiency a first-class design constraint rather than an afterthought. In a single-turn chat setting, generating 500 tokens with 200ms latency may be acceptable. In an agentic setting, a single task might require 20–50 tool-calling turns, each generating new context that must be incorporated into the model's state. A model that is 2× slower per token will produce 2× fewer agent turns in a given time budget, directly limiting the complexity of tasks it can solve before the user's patience (or the system's timeout) is exhausted. The paper explicitly positions Nemotron 3 Super for agentic reasoning — software engineering (SWE-Bench), terminal use (TerminalBench), multi-turn conversational tool use (TauBench V2), and search (BrowseComp) — which makes throughput not just a nice-to-have but a requirement. This is why the architecture co-designs for serving efficiency rather than training efficiency alone.

Where Prior Approaches Fall Short

The paper identifies four specific limitations in prior work that motivate its combined architectural, training, and quantization strategy.

Limitation 1: Standard MoE designs optimize the wrong bottleneck for online serving. Existing MoE architectures, including those in DeepSeek-V3 (DeepSeek-AI, 2025c) and GShard (Lepikhin et al., 2020), are largely motivated by high-level sparsity arguments: activate a fraction of parameters per token to scale total capacity while keeping FLOPs roughly constant. This framing implicitly assumes that FLOPs are the dominant cost in all deployment regimes. The paper argues — through the five design principles enumerated in Section 2.1.1 — that this assumption breaks down for real-world serving:

"Whereas accuracy per FLOP reflects computational efficiency, accuracy per parameter captures memory footprint, memory bandwidth, routing-induced communication, and sharding overhead. Neglecting these factors can yield architectures that appear efficient in aggregate compute yet incur substantial inefficiency in practice."

The paper's analysis distinguishes between two serving regimes that impose different bottlenecks. In throughput-oriented serving (processing many requests concurrently), all-to-all communication for expert routing scales as $d \times K$, where $d$ is the hidden dimension and $K$ is the number of active experts per token. In latency-oriented serving (processing individual requests quickly), the memory bandwidth cost of reading expert weights dominates — each expert matrix is $d \times m$ parameters, where $m$ is the expert intermediate dimension. Standard MoE designs treat $d$ and $m$ as architectural constants inherited from dense Transformer conventions, but the paper argues that $d$ is the most promising axis for reduction: making the hidden dimension smaller reduces both communication payload ($d \times K$) and weight-load bandwidth ($d \times m$), helping both regimes simultaneously. The catch is that $d$ cannot be reduced below a task-specific effective feature rank $r_{\text{eff}}$ without collapsing model quality — and this is precisely the constraint that LatentMoE navigates by performing expert computation in a compressed latent space while keeping non-routed operations at full dimension.

Limitation 2: Low-precision pretraining at the scale of 120B parameters with complex architectures has not been demonstrated stably. The paper pre-trains the entire 120B model in NVFP4 — a 4-bit floating-point format with E2M1 elements and 2D block scaling — for 25 trillion tokens. While 4-bit inference quantization (PTQ) is increasingly common, 4-bit pretraining — where weights, activations, and gradients are all quantized during forward and backward passes — is substantially more challenging because gradient quantization noise accumulates over thousands of training steps. Prior work from NVIDIA (NVIDIA, 2025d) established the NVFP4 pretraining recipe, but it had only been demonstrated on smaller models. Nemotron 3 Super is the first model to scale this to 120B total parameters with a hybrid Mamba-MoE architecture. The paper documents specific challenges that emerged at scale — notably the accumulation of zero-valued weight gradient elements reaching 7% of total parameters by the end of training, and the emergence of channel magnitude patterns in expert layers where low-norm output channels of FC1 align with low-norm input channels of FC2 (Figure 6) — and provides analysis linking these to NVFP4 underflow (Figures 7, 8). This is not just a scaling demonstration; it is an investigation of what goes wrong when 4-bit gradients interact with sparse expert routing over long training horizons, and it establishes that stable training is achievable despite these effects.

Limitation 3: Speculative decoding typically relies on external draft models, adding deployment complexity. Speculative decoding is a well-established technique for reducing decoding latency: a small "draft" model generates candidate tokens, and the main model verifies them in parallel, accepting those that match its own predictions. This works well when the draft model is closely aligned with the main model's distribution, but it introduces deployment friction: maintaining a separate draft model, managing its memory, and ensuring it generates compatible token IDs. The Multi-Token Prediction (MTP) approach introduced in DeepSeek-V3 (DeepSeek-AI, 2025c) and explored in Gloeckle et al. (2024) offers an alternative — train auxiliary prediction heads on the main model to predict future tokens, then use these heads as built-in draft models. However, the paper identifies a subtle limitation of standard MTP implementations: when heads are trained to predict fixed offsets (e.g., head 1 predicts token $n+2$, head 2 predicts token $n+3$), they cannot be reused autoregressively for longer drafts without encountering a training-inference distribution mismatch. The head trained to predict offset 3 was trained on ground-truth hidden states; at inference, when it conditions on its own (possibly erroneous) generated states from earlier draft steps, acceptance rates degrade sharply.

Nemotron 3 Super's solution — shared-weight MTP heads trained across multiple offsets — is a small architectural change with outsized practical consequences: the same head can be applied recursively at inference, enabling longer draft lengths with more stable acceptance behavior. Table 2 and Figure 4 show that this design achieves the highest average acceptance length (3.45 tokens per verification step) on SPEED-Bench, outperforming DeepSeek-R1 across all domains. This matters particularly for interactive agentic use cases where low latency is critical and the cost of an external draft model is hard to justify.

Limitation 4: RL post-training for agentic capabilities has not been scaled to the diversity and horizon of environments needed for robust real-world tool use. The paper's RL pipeline spans 21 environments in Stage 1 alone, covering math, code, STEM, safety, instruction following, long context, puzzles, conversational tool use, and terminal use — 37 distinct RL datasets in total. Prior models with strong agentic capabilities typically rely heavily on supervised fine-tuning (SFT) with distilled expert trajectories, which is simpler and cheaper but often leads to out-of-distribution (OOD) degradation: the model learns to mimic expert behavior patterns in scenarios that resemble training, but fails to generalize when the environment or task structure changes. End-to-end RL avoids this problem by optimizing the model's own policy against environment rewards, but has historically been limited to small sets of environments due to infrastructure complexity. The paper describes substantial infrastructure work — asynchronous GRPO with decoupled training and inference, in-flight weight updates, PivotRL for efficient agentic RL using offline SFT traces, and a SWE-specific RL environment with Apptainer container isolation, memory management, and multi-harness diversity — to scale agentic RL to thousands of GPUs across dozens of environments. This infrastructure allows the model to be trained simultaneously on math reasoning, code generation, safety alignment, and software engineering tasks without regressing on any individual task, which the paper identifies as a key challenge: single-environment RL training produced "severe regressions on other benchmarks."

How This Paper Positions Itself

The paper does not claim to invent any single technique from scratch. LatentMoE is introduced in a companion technical report (Elango et al., 2026); MTP with shared-weight heads builds on Gloeckle et al. (2024) and DeepSeek-AI (2025c); NVFP4 pretraining follows NVIDIA (2025d); the hybrid Mamba-Attention architecture was introduced in Nemotron 3 Nano (NVIDIA, 2025c); the RL infrastructure builds on NeMo Gym and NeMo RL. The paper's contribution is the integration and scaling of these techniques into a single model that demonstrates they can work together at the 120B parameter scale, with stable NVFP4 pretraining through 25T tokens, and with an RL pipeline that produces competitive agentic performance while maintaining throughput advantages.

This positioning is explicit in the paper's framing: it is the first model to simultaneously be (1) pre-trained in NVFP4, (2) leverage LatentMoE, and (3) include MTP layers. Each of these three components addresses a different axis of the accuracy-efficiency tradeoff. NVFP4 pretraining reduces the cost of training itself (memory, bandwidth, FLOPs) while producing a model that is already adapted to low-precision computation, making post-training quantization less lossy. LatentMoE addresses the serving bottleneck by reducing memory and communication costs per expert while increasing the number of experts and active experts per token, improving accuracy per inference dollar. MTP addresses inference latency through native speculative decoding without an external draft model, with the shared-weight design enabling longer effective draft lengths. The three components are complementary rather than redundant: even with optimal quantization and expert routing, decoding latency in autoregressive models remains serial; even with speculative decoding, the per-token cost of expert computation still matters for throughput; even with efficient expert computation, training the model in the first place remains expensive without low-precision pretraining.

The paper also positions Nemotron 3 Super relative to the rapidly evolving landscape of open-weight models. At the time of release, GPT-OSS-120B (OpenAI, 2025) and Qwen3.5-122B (Yang et al., 2025) represent the state of the art in the ~120B total parameter class. The paper's evaluation in Table 5 and Figure 1 shows that Nemotron 3 Super is competitive on benchmark accuracy while providing substantial throughput advantages — up to 2.2× over GPT-OSS-120B and 7.5× over Qwen3.5-122B on the 8K/64K input/output setting. The throughput comparison is nuanced: the paper uses the best serving framework (vLLM or TRT-LLM) for each model, uses MXFP4 quantization for GPT-OSS-120B and BF16 for Qwen3.5-122B, and measures on B200 GPUs. This is a realistic deployment comparison rather than a controlled FLOPs comparison — it answers the question "how fast can you serve this model on current hardware with current software?" rather than "how many FLOPs does the architecture theoretically require per token?" The 7.5× gap to Qwen3.5-122B is particularly notable and likely reflects both architectural efficiency (Mamba blocks avoiding KV cache overhead at long output lengths) and quantization advantages (NVFP4 on Blackwell vs. BF16).

Finally, the paper positions its release strategy as a contribution in itself: open-sourcing base checkpoints, post-trained checkpoints, quantized checkpoints, pretraining datasets (Nemotron-Pretraining-Specialized-v1.1), and post-training datasets (Nemotron-Super-Post-Training-Data) with detailed documentation. This includes not just the final artifacts but the data generation pipelines — synthetic code concepts, algorithmic problems, economic reasoning questions, formal logic, multiple-choice, conversational tool use trajectories, and SWE agent traces — which enables replication and extension by the broader community. The paper's emphasis on reproducibility (evaluation containers, Nemo Evaluator SDK, standardized harnesses) reflects a recognition that benchmarking in the agentic domain is particularly sensitive to evaluation protocol differences, and that transparent evaluation is necessary for meaningful comparisons.

3. Technical Approach

3.1 Reader Orientation

Nemotron 3 Super is a 120 billion-parameter language model that activates only 12 billion parameters per token, built from three interacting subsystems — a hardware-aware sparse expert architecture (LatentMoE), a hybrid sequence-modeling backbone (Mamba-2 with strategic attention anchors), and built-in speculative decoding heads (Multi-Token Prediction) — all pretrained from scratch in 4-bit floating-point precision. The system solves the problem of serving large language models efficiently for agentic workloads by co-designing the architecture so that the operations dominating inference cost (memory bandwidth for expert weight loads, all-to-all communication for expert routing, and serial autoregressive decoding) are each attacked by a different mechanism, with the three mechanisms designed to compose without interfering with each other. The "shape" of the solution is a sparse hybrid model whose efficiency gains come from structural changes to the computation graph, not from post-hoc compression of a conventionally designed dense model.

3.2 Big-Picture Architecture (Diagram in Words)

Conceptually, Nemotron 3 Super can be understood as a stack of 88 transformer-like layers, each containing either a Mamba-2 state-space block or a self-attention block followed by a LatentMoE feedforward block. At every layer, the sequence-modeling component (Mamba-2 or attention) produces a contextualized representation of each token; this representation is then processed by the MoE block, where only a fraction of the model's total feedforward parameters are activated. The architecture diverges from a standard Transformer in three critical places:

1. LatentMoE blocks (majority of layers): Before expert routing, each token's hidden representation is projected down from the full model dimension $d = 4096$ into a smaller latent dimension $\ell = 1024$. Routing, expert computation, and all-to-all communication happen in this compressed space, reducing their cost by a factor of $d/\ell = 4$. These savings are reinvested: the total number of experts increases from $N$ to $N' = N \times 4 = 512$, and the number of active experts per token increases from $K$ to $K' = K \times 4$, yielding higher model quality at roughly constant inference cost.

2. Mamba-2 sequence modeling (majority of layers): Instead of self-attention, most layers use Mamba-2 blocks that maintain a constant-size recurrent state during generation, eliminating the quadratically growing KV cache. A small minority of layers use standard self-attention as "global anchors" — their KV cache does grow with sequence length, but there are few enough of them that the total memory overhead remains manageable.

3. Multi-Token Prediction heads (final layers): Atop the last transformer layer, two auxiliary prediction heads share a single set of parameters and are trained to predict the next two tokens ($n+1$ and $n+2$) from the current hidden state. During inference, these heads function as built-in draft models: they propose candidate future tokens, and the main model verifies them in a single forward pass, accepting those that match its own predictions.

Information flows sequentially through the 88-layer stack: the input tokens pass first through an embedding layer (BF16), then through a repeating pattern of Mamba-2 + LatentMoE layers interspersed with attention + LatentMoE layers, and finally through the MTP heads and the output projection. At inference, the model supports context lengths up to 1M tokens, and the MTP heads can be applied recursively (the same head generating draft token 3 using its own output from draft token 2) to achieve longer speculative drafts.

3.3 Roadmap for the Deep Dive

• First, the LatentMoE architecture — how the latent projection, routing, and expert computation operate, why the dimension reduction is safe, and how the savings are quantified.
• Second, the inference efficiency analysis that motivated LatentMoE — the two serving regimes, the bottleneck identification, and the five design principles that constrain the architecture.
• Third, the Multi-Token Prediction mechanism — how the shared-weight heads are trained, how they support autoregressive drafting at inference, and why the training-inference mismatch is milder than with fixed-offset heads.
• Fourth, the hybrid Mamba-Attention interleaving pattern — the specific layer pattern, the role of global attention anchors, and the tradeoff between state size and modeling fidelity.
• Fifth, NVFP4 pretraining — the quantization scheme (formats, block sizes, stochastic rounding), the layer-type-specific precision assignments, the zero-gradient phenomenon, and the stability analysis.
• Sixth, the pretraining data mixture and two-phase curriculum, including the synthetic data generation pipelines for code, economics, logic, and multiple-choice questions.
• Seventh, the supervised fine-tuning recipe with its two-stage loss, reasoning mode control, and agentic dataset construction.
• Eighth, the reinforcement learning pipeline — the three-stage structure (multi-environment RLVR, SWE-RL, RLHF), the PivotRL algorithm for agentic tasks, and the async infrastructure.
• Ninth, the post-training quantization to FP8 and NVFP4, including the AutoQuantize mixed-precision search, the SSM cache quantization challenge, and the stochastic rounding solution.

This ordering follows the chronological pipeline of the model's creation — architecture design → pretraining → SFT → RL → quantization — which matches how practitioners would need to understand it to replicate or extend the work.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an integrated systems paper whose core technical idea is that inference efficiency for large language models cannot be achieved by any single technique applied in isolation, but instead requires co-designing the architecture, training precision, post-training recipe, and quantization strategy so that each component addresses a specific serving bottleneck without undermining the others. The architectural backbone is a hybrid Mamba-Attention model scaled sparsely with LatentMoE and equipped with native speculative decoding via MTP; the training backbone is NVFP4 pretraining across 25 trillion tokens with a two-phase data curriculum; the alignment backbone is a three-stage RL pipeline spanning 21+ environments with PivotRL for agentic tasks; and the deployment backbone is post-training quantization to FP8 and NVFP4 with AutoQuantize-guided mixed-precision assignment and stochastic rounding for the recurrent state cache. What follows is a sentence-level walkthrough of each of these components, starting from the lowest-level architectural primitives and building up to the full training and deployment pipeline.

---

LatentMoE: Projecting Expert Computation into a Compressed Latent Space

The LatentMoE architecture is the single most consequential architectural decision in Nemotron 3 Super, because it governs how the model's 120.6B total parameters are distributed across experts and how those experts are accessed during inference. A standard Mixture-of-Experts layer (Figure 3a) works as follows: given an input token representation $x \in \mathbb{R}^d$ (where $d$ is the model's hidden dimension), a routing network (gate) computes a score for each of $N$ available experts, selects the top $K$, and dispatches $x$ to those $K$ experts. Each expert is a feedforward network with its own weight matrices; it processes $x$ and returns an output. The outputs of the $K$ selected experts are then combined (typically via a weighted sum using the gate scores) to produce the final MoE layer output. Critically, in a standard MoE, the input $x$, the expert computation, and the all-to-all communication for dispatching and combining all operate in the full dimension $d$.

LatentMoE (Figure 3b) modifies this pipeline by inserting a compression step before routing. Specifically:

$x_{\ell} = W_{\downarrow} x$

where $W_{\downarrow} \in \mathbb{R}^{\ell \times d}$ is a learnable down-projection matrix that maps the token representation from the full hidden dimension $d = 4096$ into a lower-dimensional latent space of size $\ell = 1024$. The compressed representation $x_{\ell} \in \mathbb{R}^\ell$ is then what gets routed to the experts. Each expert in a LatentMoE layer is itself smaller than a standard MoE expert: its weight matrices operate in the latent dimension $\ell$ rather than $d$, so the per-expert parameter count is approximately $O(\ell \cdot m)$ rather than $O(d \cdot m)$, where $m$ is the expert's intermediate dimension. The routing gate also operates in the latent space (i.e., it takes $x_{\ell}$ as input), and the all-to-all communication between devices during distributed inference transmits vectors of size $\ell$ rather than $d$.

After the selected experts produce their outputs in the latent space, these outputs are aggregated (e.g., via the gate-score-weighted sum) to produce a combined latent representation $y_{\ell} \in \mathbb{R}^\ell$. Finally, a learnable up-projection matrix maps this back to the full model dimension:

$y = W_{\uparrow} y_{\ell}$

where $W_{\uparrow} \in \mathbb{R}^{d \times \ell}$. The output $y \in \mathbb{R}^d$ is then ready to be passed to the next layer's sequence-modeling component (Mamba-2 or attention block).

Why the dimension reduction is safe. The central risk of this approach is that compressing $x$ from dimension $d$ to $\ell$ might discard information needed for accurate routing or expert computation. The paper argues that this risk is bounded by a property of the data distribution: task-specific token representations in Transformer models, despite nominally living in $\mathbb{R}^d$, typically have an effective rank $r_{\text{eff}}$ that is substantially smaller than $d$. This is a well-documented empirical phenomenon in deep learning — the representations learned by large Transformers tend to lie near a low-dimensional manifold within the high-dimensional embedding space. As long as $\ell \geq r_{\text{eff}}$, the down-projection can preserve the essential information needed for routing and expert computation. The paper states this as Principle 4: "A task-specific effective feature rank $r_{\text{eff}}$ imposes a lower limit on how much $d$ can be reduced; reducing $d$ below this limit causes model quality to collapse." The choice of $\ell = 1024$ for a model with $d = 4096$ implies that $r_{\text{eff}}$ is believed to be no larger than 1024 for the target tasks (reasoning, coding, agentic tool use), and the empirical results — competitive accuracy against models with full-dimension MoE — validate this.

How the savings are quantified. The compression factor is $d/\ell = 4096/1024 = 4$. Every operation that scales with dimension in the MoE pathway — expert weight loads, expert computation, and all-to-all communication — is reduced by approximately this factor. The paper explicitly tracks three costs:

1. Memory bandwidth for expert weight loads: Each expert matrix has dimensions $\ell \times m$ rather than $d \times m$, reducing the number of bytes read from memory per token-expert pair by a factor of $d/\ell$.

2. All-to-all communication volume: The routing payload per token scales as the product of dimension and number of active experts. In LatentMoE, the payload is proportional to $\ell \times K'$ rather than $d \times K$, where $K'$ is the (increased) number of active experts.

3. Expert computation FLOPs: The matrix multiplications inside each expert operate on smaller matrices, reducing FLOPs per expert by approximately $d/\ell$ (though this is partially offset by the increased number of active experts, as described below).

How the savings are reinvested. The paper does not simply pocket the efficiency gains; it reinvests them to increase model quality at constant inference cost. Specifically:

• The total number of experts increases from $N$ to $N' = N \times d/\ell$. In Nemotron 3 Super, this means 512 total experts per LatentMoE layer.
• The number of active experts per token increases from $K$ to $K' = K \times d/\ell$. In Nemotron 3 Super, this means $K' = 22$ active experts per token (top-22 routing).
• The expert hidden dimension $m$ is held fixed to preserve the effective nonlinear capacity per expert (Principle 3).

The intuition is that increasing the number of experts and active experts per token expands the combinatorial space of expert combinations available to the model — two tokens might activate completely different subsets of the 22 active experts out of 512 total, giving the model exponentially more possible "paths" through the network. This increased routing flexibility comes at essentially zero additional inference cost because the per-token cost of each expert (weight loads, computation) has been reduced by the same factor $d/\ell$ that $K$ was increased by. The paper summarizes this with Principle 5: "Scaling both the total number of experts $N$ and the top-$K$ experts per token improves quality by exponentially expanding the space of expert combinations."

What stays in the full dimension. Critically, not all operations are compressed. The paper specifies that "all non-routed computations — including the routing gate (gating network), shared expert computation, and non-expert layers — remain in the full hidden dimension $d$, as they do not contribute significantly to the targeted bottlenecks." This is asymmetric: the routing gate itself is not the bottleneck (it's a relatively small matrix), so it stays at dimension $d$; the shared expert (a feedforward network applied to every token regardless of routing decision) also stays at dimension $d$, since it isn't part of the sparse communication pathway. The down-projection $W_{\downarrow}$ and up-projection $W_{\uparrow}$ themselves are additional parameters, but the paper states that their "step-time impact is negligible" — they represent a small fraction of total FLOPs and parameters compared to the expert computations they enable.

Concrete dimensions for Nemotron 3 Super. From Table 1:

• Model dimension: $d = 4096$
• Latent dimension (MoE): $\ell = 1024$
• Expert hidden dimension: $m = 2688$
• Shared expert intermediate size: 5376
• Total experts per LatentMoE layer: 512
• Activated experts (top-K): 22
• The routing function is a sigmoid router, not a softmax router: each expert gets an independent probability of selection rather than competing for a probability mass that sums to 1.

The load balancing strategy combines two mechanisms: an auxiliary-loss-free approach (following Wang et al., 2024 and DeepSeek-AI, 2025c) with an update rate of $10^{-3}$, plus a standard load balancing loss with coefficient $10^{-4}$ (following Lepikhin et al., 2020). The auxiliary-loss-free strategy adjusts expert biases dynamically to encourage uniform utilization without adding a term to the training loss, while the standard loss provides a small explicit regularization signal.

---

The Two Serving Regimes That Motivate LatentMoE

The LatentMoE design is not motivated by abstract sparsity principles but by a concrete analysis of where time and energy go during inference serving. The paper distinguishes two regimes that impose different constraints:

1. Low-latency (online) serving. When the system must respond to individual requests quickly, the dominant bottleneck is often memory bandwidth for reading expert weights. Each expert matrix of size $d \times m$ must be loaded from GPU memory (HBM) into compute units (SMEM/registers) whenever that expert is activated. The cost of this load scales with $d \times m$. Since the number of active experts $K$ determines how many such loads happen per token, the total bandwidth cost per token is proportional to $K \times d \times m$.

2. Throughput-oriented (batch) serving. When processing many requests concurrently, the bottleneck shifts to all-to-all communication between devices in a distributed serving setup. In a standard MoE, the model's experts are partitioned across multiple GPUs. For each token, the routing gate decides which experts (on which devices) need to process it. The token must be sent to those devices (dispatching), and the results must be sent back (combining). The volume of data transmitted scales as $d \times K$ — the token's embedding dimension times the number of active experts.

The paper's analysis identified that both bottlenecks can be addressed simultaneously by reducing $d$, but only if the model quality is preserved through other means (increased $K$ and $N$). This is synthesized in Principles 1–3:

• Principle 1: In low-latency serving, the memory bandwidth cost of reading expert weights dominates; reducing this cost requires decreasing $d$ or $m$.
• Principle 2: In throughput-oriented serving, all-to-all routing dominates; reducing this requires decreasing $d$ or $K$.
• Principle 3: Model quality depends on the effective nonlinear budget $K \cdot m$; to relieve bottlenecks without sacrificing quality, $K$ and $m$ should be held fixed.

Principles 1 and 2 together identify $d$ as the only variable that helps both regimes. Principle 3 clarifies that reducing $d$ by a factor $\alpha$ should be compensated by increasing $K$ by $\alpha$ (keeping $K \cdot m$ fixed). Principle 4 provides the safety bound: $d$ cannot be reduced below $r_{\text{eff}}$. Principle 5 justifies increasing both $N$ and $K$: the combinatorial expansion improves quality.

The concrete parameterization of Nemotron 3 Super — $d = 4096$, $\ell = 1024$, $K' = 22$ — reflects a specific tradeoff point on this design space. The paper does not report experiments varying $\ell$ or $K'$, but the LatentMoE companion paper (Elango et al., 2026) presumably provides the empirical validation for these choices.

---

Multi-Token Prediction for Native Speculative Decoding

The Multi-Token Prediction (MTP) mechanism serves two purposes simultaneously: it improves representation learning during training (by forcing the model to plan multiple steps ahead) and it enables native speculative decoding during inference (by providing built-in draft heads that propose future tokens).

Standard MTP training. In a standard next-token prediction setup, the model at position $n$ produces a hidden state $h_n$ and uses it to predict token $y_{n+1}$. MTP extends this by adding $N$ auxiliary prediction heads, each taking $h_n$ as input and predicting a different future token: head 1 predicts $y_{n+2}$, head 2 predicts $y_{n+3}$, and so on up to head $N$ predicting $y_{n+N+1}$. During training, the total loss is the standard next-token loss plus a weighted sum of the auxiliary head losses:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{next-token}} + \lambda \sum_{i=1}^{N} \mathcal{L}_{\text{aux}, i}$

where $\mathcal{L}_{\text{aux}, i}$ is the cross-entropy loss for predicting token $y_{n+i+1}$ from hidden state $h_n$, and $\lambda$ is a scaling factor (set to 0.3 in Nemotron 3 Super). This forces the main model's hidden states to encode information that is predictive of multiple future tokens, which the paper argues "encourages representations that capture multi-step dependencies and longer-range structure."

The training-inference mismatch in fixed-offset MTP. At inference time, speculative decoding works by using the auxiliary heads to generate draft tokens, then verifying them with the main model. With fixed-offset heads (the standard approach in DeepSeek-V3 and Gloeckle et al., 2024), head 1 is trained to predict offset +2, so it can only generate one draft token with high reliability. To get a second draft token, the system could try to reuse head 1 autoregressively: feed its own output as if it were the true token $y_{n+2}$, then use the resulting hidden state to predict $y_{n+3}$. But head 1 was trained on ground-truth hidden states — the hidden state produced after correctly predicting $y_{n+2}$. At inference, after generating a draft token that might be wrong, the hidden state used for the next prediction is from a potentially incorrect trajectory. This distribution shift causes acceptance rates to degrade sharply as draft length increases. The paper explicitly notes this as a "training–inference mismatch" where "the head is trained under ground-truth hidden states but, at inference, conditions on its own generated states."

Nemotron 3 Super's solution: shared-weight MTP heads. Instead of training $N$ independent heads each specialized to a fixed offset, Nemotron 3 Super trains one set of parameters that is shared across multiple offsets. During training, this shared head is exposed to prediction tasks at offsets 2, 3, ..., $N+1$. This means the head learns to predict future tokens from hidden states that may be at various distances from the target — including hidden states that are themselves the result of previous (possibly imperfect) predictions, if the training data construction introduces such sequences. The paper calls this a "unified prediction head exposed to multiple offsets" that "regularizes the head across prediction horizons."

At inference, this shared head can be applied recursively with much milder degradation: the same parameters that were trained to predict $y_{n+3}$ from $h_n$ also learned to handle the case where the intermediate $y_{n+2}$ might be imperfect, because the training data mixed different offset contexts. The paper reports that "while acceptance rates naturally decrease as draft length increases, the degradation is substantially milder than with independently trained offset heads."

Architecture details. From Table 1 and Section 2.1.2:

• Number of MTP layers: 2 (both sharing the same weights)
• These are positioned at the output of the final transformer layer.
• The MTP loss scaling factor is $\lambda = 0.3$.
• During post-training, an additional "MTP healing" stage is applied: after RLHF, the MTP heads are trained (with the rest of the model frozen) on prompts from the RLVR stage using standard negative log-likelihood, which "significantly improves MTP accuracy."

Speculative decoding performance. The paper evaluates MTP quality using SPEED-Bench (Abramovich et al., 2026), which reports average acceptance length — the mean number of draft tokens accepted by the main model per verification step, when drafting up to 7 tokens ahead. As reported in Table 2, Nemotron 3 Super achieves an average acceptance length of 3.45 across all categories, compared to 3.33 for Qwen3-Next and 2.70 for DeepSeek-R1. Figure 4 plots the acceptance rate as a function of draft index: all models decline monotonically, but Nemotron 3 Super maintains higher acceptance at every position, with the gap widening at positions 4–7. This confirms the shared-weight design's benefit for longer drafts.

Figure 5 shows the practical impact on Blackwell hardware: enabling MTP with draft length $D = 3$ shifts the throughput–latency Pareto frontier outward compared to MTP disabled, delivering higher output tokens per second (TPS) for any given latency target. The paper measures this on the SPEED-Bench Throughput-1k split with 1k output tokens per request, using TRT-LLM with tensor parallelism degree 1 on a B300 GPU.

Design choice: why shared weights rather than more independent heads? The alternative would be to increase $N$ — train 5 or 7 independent heads, one per offset, and use all of them for one-pass drafting without recursion. This avoids the training-inference mismatch but costs additional parameters (each head is a full output projection matrix). The shared-weight approach achieves comparable or better draft quality with fewer parameters, and the parameter savings are reinvested in the main model capacity.

---

Hybrid Mamba-Attention Architecture with Sparse MoE

The 88-layer stack of Nemotron 3 Super interleaves three component types: Mamba-2 blocks for efficient recurrent sequence modeling, self-attention blocks for global information routing, and LatentMoE blocks for sparse feedforward processing. The objective is to maximize inference throughput — particularly for long-context agentic workloads where KV cache memory is the dominant constraint — while preserving the modeling fidelity of attention-based Transformers.

Why Mamba-2 dominates the architecture. The primary system bottleneck in Transformer inference at long context lengths is the KV cache. In standard multi-head attention, the keys and values for every previous token must be stored for each attention layer. With $L$ layers, $H$ heads, head dimension $d_h$, and sequence length $S$, the KV cache memory is $2 \times L \times H \times d_h \times S$ floating-point numbers. For a model with 88 layers at 1M context length, this becomes prohibitively large. Mamba-2 (Dao & Gu, 2024) replaces this with a structured state-space model that maintains a constant-size recurrent state. During generation, the Mamba block updates its state via a linear recurrence; the state size does not grow with sequence length. The tradeoff is that Mamba-2's sequence modeling is fundamentally local — each token's state update depends only on its own state and input, without the direct all-to-all token interaction that attention provides.

The role of global attention anchors. To compensate for Mamba-2's lack of global interaction, the paper inserts a small number of self-attention layers at strategic positions in the stack. These "global anchors" enable tokens to attend to all other positions, allowing long-range information to propagate across the otherwise-local Mamba blocks. The paper states that these attention layers are "strategically inserted" and "enable full-token interaction and long-range information routing across the stack."

From Figure 2 and the layer pattern description, the interleaving follows a specific repeating motif:

• Blocks of 4 Mamba-2 + LatentMoE layers
• Followed by 1 attention + LatentMoE layer
• With variations (some blocks are 3 Mamba-2 + LatentMoE, some 1, some repetitive)

The total count is 88 layers, with the majority being Mamba-2 based and a minority attention-based. The attention layers use Grouped-Query Attention (GQA) with 32 query heads and 2 key-value heads (head dimension 128), which further reduces KV cache size compared to standard multi-head attention (since the 2 KV heads are shared across groups of query heads).

Consistent architectural conventions. The paper maintains several conventions from prior Nemotron models:

• No positional embeddings (the model learns positional information implicitly through the sequence modeling components)
• No dropout in any layer
• No bias terms in linear layers
• RMSNorm for normalization (not LayerNorm)
• Untied embedding and output weights (the input embedding matrix and the final output projection are separate parameter matrices)

Concrete configuration (Table 1).

• Total layers: 88
• Model dimension: 4096
• Query heads (attention layers): 32
• KV heads (attention layers): 2
• Head dimension: 128
• Mamba state dimension: 128
• Mamba groups: 8
• Mamba heads: 128
• Mamba head dimension: 64
• Expert hidden dimension (MoE): 2688
• Shared expert intermediate size: 5376
• Total experts per MoE layer: 512
• Top-K activated: 22
• MoE latent size: 1024
• MTP layers: 2 (shared weights)

The "Mamba state dimension" of 128 refers to the size of the recurrent state vector maintained per token per Mamba block. With 128 Mamba heads, this means each Mamba layer maintains a state of size $128 \times 128 = 16,384$ elements per token — constant regardless of sequence length, in contrast to the growing KV cache of attention.

---

NVFP4 Pretraining: Low-Precision Training at Scale

Nemotron 3 Super is pre-trained entirely in NVFP4, a 4-bit floating-point format, across 25 trillion tokens. This section details the quantization scheme, the precision assignments across layer types, the stability challenges, and the empirical analysis of what goes wrong with 4-bit gradients over long training horizons.

NVFP4 format specification. The NVFP4 format (NVIDIA, 2025d) represents each value as a 4-bit element with the following structure:

• Element format: E2M1 — 2 exponent bits and 1 mantissa bit, supporting a dynamic range suitable for neural network activations and gradients.
• Block scaling: Weights are quantized using two-dimensional (2D) block scaling, meaning a 16-element micro-block (e.g., a $4 \times 4$ tile) shares a single scale factor. The micro-block scaling factor itself is stored in E4M3 format (4 exponent bits, 3 mantissa bits — essentially FP8), and there is a second-level FP32 global scale per tensor.
• Weight quantization: During the forward pass, weights are quantized to NVFP4 using 2D block scaling. During the backward pass, the same 2D block scaling is applied to maintain consistency between the quantized weights used in forward and backward computations.
• Activation quantization: Activations are quantized to NVFP4 using one-dimensional (1D) block scaling along the reduction axis of the GEMM (matrix multiplication). This means blocks are 1D contiguous segments along the dimension being reduced.
• Gradient quantization: Gradients are also quantized to NVFP4 using 1D blocks along the GEMM reduction axis. Additionally, a Random Hadamard Transform (RHT) is applied to the inputs of the weight-gradient computation (wgrad) to decorrelate gradient elements before quantization, reducing structured quantization error. Stochastic rounding is applied to gradient tensors, meaning that when a value falls between two representable NVFP4 levels, it is rounded up or down with probability proportional to its distance to each level, making the rounding unbiased in expectation.

Layer-type-specific precision assignments (Table 3). Not all layers are quantized to NVFP4. The paper uses a hybrid precision scheme where sensitive operations are kept in higher precision:

The assignment reflects a pragmatic principle: quantize aggressively where the model is most compute-heavy (the bulk of linear layers, especially in MoE experts), but keep sensitive bottleneck operations in higher precision where the parameter/FLOP count is small enough that BF16 overhead is negligible.

Training hyperparameters (Section 2.4).

• Optimizer: AdamW (Loshchilov & Hutter, 2017) with weight decay 0.1, $\beta_1 = 0.9$, $\beta_2 = 0.95$
• Learning rate schedule: Warmup-Stable-Decay (WSD) over 25T tokens
• Warmup: first 200B tokens to peak LR of $4.5 \times 10^{-4}$
• Stable phase: constant LR for ~20T tokens
• Decay phase: final 5T tokens, minus-sqrt decay schedule to minimum LR of $4.5 \times 10^{-6}$
• Sequence length: 8,192 tokens
• Global batch size: 3,072 sequences, approximately 25.17M tokens per batch
• Long-context phase (end of pretraining): constant LR $4.5 \times 10^{-6}$, global batch size 16, 64-way context parallelism, 2-way tensor parallelism, 64-way expert parallelism on GB200 GPUs

MTP training objective during pretraining. The MTP auxiliary loss is computed per-token (cross-entropy on future tokens) and scaled by a factor of 0.3 relative to the main next-token prediction loss. The shared-weight MTP heads are trained from the beginning of pretraining.

The zero-gradient phenomenon. A notable finding documented in Sections 2.2 and Figures 6–8 is the accumulation of zero-valued weight gradient elements during NVFP4 pretraining. By the end of 25T tokens, 7% of total weight gradient elements were exactly zero. The paper investigates the root cause and concludes it is largely due to underflow — NVFP4's limited dynamic range causing small gradient values to be quantized to zero.

The investigation uses a combination of smaller-scale controlled experiments and tensor-level sampling. Key observations:

1. Channel magnitude patterns (Figure 6): In early expert layers, the norms of FC1 output channels and corresponding FC2 input channels converge toward zero as training progresses. By 23T tokens, low-norm channels are clearly visible as dark horizontal/vertical stripes in the weight matrix visualizations.

2. NVFP4 vs. BF16 comparison (Figure 7): An NVFP4-trained Nemotron 3 Nano model at 1T tokens reaches a similar count of zero-valued weight gradients as a BF16-trained model at 25T tokens. When a partially-trained NVFP4 model is switched to BF16 at 0.5T tokens, the zero-valued gradient count returns to baseline levels. The BF16 model still contains many small-magnitude gradients (< 1e-12), but NVFP4 quantization underflows these to exactly zero.

3. Underflow origin analysis (Figure 8): At 500B tokens, zero-valued weight gradients in FC1 are attributed almost entirely to underflows in the activation gradient (dgrad) of FC2. At 750B tokens, zero-valued weight gradients appear in both FC1 and FC2: FC1 zeros still come from dgrad FC2 underflows, while FC2 zeros now also come from underflows in the forward pass activation (fprop) of FC1.

The paper's interpretation is that NVFP4 quantization accelerates a process that would happen more slowly in higher precision — low-norm expert channels are less useful and their gradients naturally attenuate, but NVFP4's underflow turns "very small" into "exactly zero" much faster, which then propagates through the backward pass to create more zeros. Importantly, the paper does not claim these zeros cause training instability — the training completed successfully with competitive final accuracy — but documents them as a phenomenon that future NVFP4 training recipes should be aware of.

MXFP8 healing experiment. The paper tested whether switching all tensors to MXFP8 (higher precision) before learning rate annealing would improve final model quality. At 19T tokens (1T before annealing), the model was switched to MXFP8 and trained through 20.6T tokens. Figure 9 shows the result: while the loss trajectory improved, there were "no gains in downstream task accuracy" — none of the evaluated benchmarks showed sustained improvement over the NVFP4-only model. The final model was therefore trained entirely in NVFP4.

---

Pretraining Data: Two-Phase Curriculum and Synthetic Data Generation

The 25T token pretraining corpus follows a two-phase curriculum design proposed in Feng et al. (2024). Phase 1 (80% of tokens, 20T) emphasizes data diversity for broad coverage; Phase 2 (20%, 5T) shifts toward predominantly high-quality sources to refine performance.

Data mixture composition (Figure 10). The corpus spans 16 high-level categories. Web crawl data is the largest component, partitioned into five quality-based groups following the Nemotron-CC taxonomy (Su et al., 2025):

• crawl-medium, crawl-medium-high, crawl-high (increasing quality)
• syn-crawl-medium-high, syn-crawl-high (synthetic data generated from filtered web documents)

Beyond web crawl, the mixture includes math (from Nemotron-CC-Math by Mahabadi et al., 2025 and MIND by Akter et al., 2024), Wikipedia, code, Nemotron-CC-Code, academic text, Crawl++ (OpenWebText, BigScience ROOTS corpus, and Reddit datasets), multilingual data, finepdfs (Kydlíček et al., 2025), and synthetic SFT-style datasets divided into general-sft, stem-sft, and code-sft. The paper incorporates reasoning-focused datasets into pretraining, motivated by Akter et al. (2026) who demonstrated their effectiveness.

Phase 1 specific mixture (Figure 10a): The largest components are syn-crawl-high (22.4%), code (14.0%), syn-crawl-medium-high (11.3%), stem-sft (11.1%), math (6.4%), crawl-high (6.5%), finepdfs (6.1%), crawl-medium-high (5.7%), multilingual (5.0%), code-sft (3.3%), nemotron-cc-code (2.1%), crawl++ (1.8%), crawl-medium (1.8%), academic (1.7%), general-sft (0.2%), wiki (0.6%).

Phase 2 specific mixture (Figure 10b): Phase 2 shifts toward higher-quality sources. finepdfs increases to 14.3% (from 6.1%), syn-crawl-medium-high decreases to 6.2% (from 11.3%), code remains at 14.0%, stem-sft increases slightly to 11.8%, wiki, crawl-high, syn-crawl-high, math, multilingual, and nemotron-cc-code stay roughly constant. crawl-medium and crawl++ are removed entirely. The general-sft component drops to 0.1%. The overall effect is a concentration on curated, high-quality data sources.

Blending principle: "Sources with comparable estimated quality are assigned similar weights, while higher-quality datasets receive proportionally greater weight in the mixture."

Synthetic data generation pipelines. The paper introduces several new synthetic datasets (released as Nemotron-Pretraining-Specialized-v1.1) and describes their generation:

Synthetic Code Concepts: A taxonomy of 91 programming concepts (extracted from HumanEval via GPT-OSS-120B) is used to generate Python problem statements via GPT-OSS-20B. Up to 4 concepts are combined per generation, producing 14M problems. GPT-OSS-120B then generates 5 solutions per problem (restricted to 60 lines maximum), yielding 23M problem-solution pairs before cleaning. Cleaning involves: checking that solutions don't add imports not in the original problem, parsing the solution to extract only the code body, generating an AST to validate syntactically correct Python, and discarding invalid cases. The final dataset is 15M problem-solution pairs.

Synthetic Unconditional Algorithmic: Using minimalistic prompts ("Write a function," "Write a Python function," "Write a coding problem and solution for a student to solve"), the paper prompts Qwen3-235B-A22B (base) and GPT-OSS-120B to generate algorithmic problems. GPT-OSS-120B rewrites samples to handle edge cases, add unit tests, and reformat outputs. LeetCode-style problems are generated with randomly selected difficulty. A scoring-and-correction loop uses GPT-OSS-120B to verify solution correctness and correct errors. Decontamination uses exact-match filtering and semantic similarity filtering (Qwen3-Embedding-0.6 embeddings, >0.8 similarity threshold) against HumanEval, MBPP, CRUXEval, and LiveCodeBench. The dataset is small (0.2B tokens) but provides "1-2 points improvement to HumanEval, MBPP, and CRUXEval-O" when added to a redo of the last 100B tokens of pretraining.

Synthetic Economics: A curated list of microeconomics, macroeconomics, and econometrics topics (e.g., "Statistical Inference and Hypothesis Testing - Type I error") is used to generate multiple-choice questions via Qwen3-235B-A22B-Thinking-2507. Diverse formats include cloze, calculation, sentence completion, and multiple-response. The model is then prompted to create new questions using initial outputs as reference points. Each question-solution pair is verified by a model-based judge for clarity, ambiguity, solvability, and accuracy.

Synthetic Formal Logic: The paper generates logic problems covering translation between natural language and predicate/propositional logic, derivation of antecedents of conditional propositions, and truth-table-based solving. Variability is introduced via random personas, letters, and logic connectives ($\land, \lor, \supset, \equiv, \sim$) in the prompts. Qwen3-235B-A22B-Thinking-2507 generates and evaluates the problems.

Synthetic Multiple Choice: Starting from the MMLU auxiliary training set (which includes ARC, MC_TEST, OpenBookQA, and RACE), Qwen3-235B-A22B generates similar questions following the same format and difficulty. DeepSeek-V3 then solves each question with reasoning, and majority voting over multiple independent solutions identifies the most consistent answer. Only samples agreeing with the majority are retained. The pipeline produces 3.5M samples (1.6B tokens). An ablation on Nemotron Nano V3 (adding this data to the last 100B tokens) shows: MMLU improves from 77.22 to 77.51, MATH Level 5 from 78.55 to 79.05, AIME-2024 from 53.3 to 56.7, and MBPP from 74.8 to 75.2, with other benchmarks stable.

Long-context CPT (Section 2.6). After the main pretraining, a long-context continuous pretraining (CPT) phase equips the model for up to 1M token contexts:

• Phase LC-1: 34B tokens at 1M context length, using 20% long-context document QA data and 80% downscaled Phase 2 data.
• Phase LC-2: 17B tokens alternating between 1M and 4K sequence lengths, to mitigate minor impacts observed on math benchmarks.
• Constant learning rate $4.5 \times 10^{-6}$, global batch size 16, GB200 GPUs with 64-way context parallelism, 2-way tensor parallelism, 64-way expert parallelism.

---

Supervised Fine-Tuning: Two-Stage Loss and Agentic Data Construction

The SFT phase follows a two-stage design motivated by a specific degradation pattern: a single-stage SFT with token-level loss normalization led to "marked degradation on long-input-short-output scenarios" — the model became biased toward producing long outputs because long reasoning traces dominated the loss. The two-stage recipe addresses this.

Stage 1: Token-level (global) average (Equation 1).

$\mathcal{L}_{\text{tok}} = \frac{\sum_{c \in \mathcal{B}} \sum_{t \in \mathcal{O}_c} \ell_t}{\sum_{c \in \mathcal{B}} |\mathcal{O}_c|}$

where $\mathcal{B}$ is the packed global batch containing multiple conversations $c$, $\mathcal{O}_c$ is the set of output-token positions for conversation $c$, $|\mathcal{O}_c|$ is the number of output tokens in that conversation, and $\ell_t = -\log p_\theta(y_t \mid x, y_{<t})$ is the token-level negative log-likelihood.

What it computes: The total loss summed across all output tokens in all conversations, normalized by the total number of output tokens across the batch. This means each output token contributes equally to the loss, regardless of which conversation it came from.

Why this form: This stage "induces strong reasoning behavior" because reasoning traces tend to be long — by giving them proportional weight in the loss, the model learns to generate detailed, step-by-step reasoning.

Stage 2: Sample-level (per-conversation) average (Equation 2).

$\mathcal{L}_{\text{samp}} = \frac{1}{|\mathcal{B}|} \sum_{c \in \mathcal{B}} \left( \frac{1}{|\mathcal{O}_c|} \sum_{t \in \mathcal{O}_c} \ell_t \right)$

where each conversation's loss is first averaged across its own output tokens (the inner average), and then these per-conversation losses are averaged equally across all conversations in the batch (the outer average).

What it computes: The average loss per conversation, with each conversation weighted equally regardless of its output length. A conversation with 10,000 output tokens and a conversation with 100 output tokens contribute equally to the final loss.

Why this form: This "reduces the dominance of long outputs" — it prevents long reasoning traces from overwhelming the training signal for tasks that expect short responses (e.g., factual QA, simple instruction following). The paper states that this "restores long-input-short-output performance while retaining reasoning."

Training configuration. Stage 1: 256K sequence length packing, global batch size 64, constant LR $1 \times 10^{-5}$, 30K warmup samples. Stage 2: 512K sequence length packing with long-context data up to 512K tokens, global batch size 32, constant LR $1 \times 10^{-5}$.

MTP during SFT. The shared-weight MTP heads continue to be trained during SFT with the same auxiliary loss (0.3 scaling factor, per-token loss on two future positions), preserving both the quality and speculative decoding benefits.

Reasoning mode control. The paper trains three reasoning modes:

• Reasoning-off: Random 3% of SFT samples have their reasoning traces stripped, teaching the model to answer directly when reasoning is not requested.
• Regular: The default mode with full chain-of-thought reasoning.
• Low-effort: A new mode introduced during SFT using samples generated by GPT-OSS-120B in its low-effort mode (Du et al., 2025). These cover math reasoning, STEM QA, and instruction following, representing 2% of SFT samples by count. The mode is also optimized during RL stages. During RL, low-effort prompts have their reward adjusted as a function of both correctness and the number of generated tokens, incentivizing concise correct answers.

Inference-time budget control. After the main SFT, a short semi-on-policy SFT stage of 350 steps is added, where rollouts are collected from the current model checkpoint and 12% of reasoning traces are truncated to random reasoning budgets. This allows users to specify a reasoning budget at inference time.

SFT data blend (Figure 16). Over 7M total samples, with agentic tasks receiving a larger proportion compared to Nemotron 3 Nano. Key datasets include:

• Software Engineering: 96.5K samples from SWE-Gym, R2E-Gym, and SWE-rebench, with trajectories distilled from OpenHands using Qwen3-Coder-480B-A35B-Instruct as teacher.
• Agentic Programming: 15K CLI tasks from a taxonomy of 24 actions, 3K SWE tasks, and 10K web development tasks, with interaction traces recorded from Qwen-3-Coder-480B and Minimax M2.5 across Codex, OpenCode, Qwen Code CLI, and Stirrup harnesses (Figure 13).
• Long Context: Multi-document QA pairs requiring 4-7 distinct retrieval/reasoning steps, plus synthetic left-to-right reasoning tasks (7 types). Majority voting over 8 reasoning traces selects the shortest correct trace.
• Financial Reasoning: 366,243 Q&A pairs with reasoning traces, generated from 565 SecQue seed questions combinatorially expanded across S&P 500 companies and fiscal years, with GenSelect answer selection and answerability filtering.
• CUDA: 100K samples of (specification, kernel) pairs with validation in a CUDA evaluation environment, including kernel repair and optimization traces.
• Safety: Two-stage generation with explicit response policies and deliberative alignment, covering refusal, over-refusal, demographic bias, copyright, jailbreak attacks, and indirect prompt injection.
• Search: 4-8 hop Wikidata graph walks converted to obfuscated search riddles, with MiniMax-M2 generating search trajectories averaging 12 tool calls per sample.
• Terminal Use: 84,864 samples from the Terminal-Task-Gen pipeline (Pi et al., 2026), combining synthetic tasks from a terminal skill taxonomy with adapted Nemotron-Cascade data.
• Multilingual: Translations of English SFT into 6 languages using Qwen2.5-Instruct-14B, with Qwen3-4B-Thinking-2507 applied as a post-editor to restore format compliance.
• SQL: 96.5K samples spanning MySQL, PostgreSQL, SQLite across 60 industries and 90 SQL concepts, with injected distractor tables/columns and controlled prompt diversity (instruction style, register, politeness).
• Conversational Tool Use: 279,116 conversations across 838 domains, generated via a 6-stage pipeline (domain generation → policy/tool generation → scenario generation → trajectory collection → verification → difficulty filtering) using Qwen3-235B-A22B, DeepSeek-R1, DeepSeek-V3.2, and GPT-OSS-120B (Figure 14). This is a ~18× scale-up over Nemotron 3 Nano's 15,588 conversations in 5 domains.
• General-Purpose Tool Use: 1.5M tool-calling trajectories via three-LLM simulation (User-LLM, Assistant-LLM, Tool-LLM) with turn-level and trajectory-level judges, using DeepSeek-V3.2 and GLM-4.7 (Figure 15).

---

Reinforcement Learning: Three-Stage Pipeline and PivotRL

The RL phase consists of three sequential stages followed by MTP healing (Figure 12):

Stage 1: Multi-environment RL from Verifiable Rewards (RLVR). This is the primary training stage, optimizing the model jointly across 21 environments covering math, code, STEM, safety, chat, instruction following, long context, puzzles, and agentic tasks (37 distinct RL datasets total). The paper emphasizes that training on all environments simultaneously is critical: "single-environment training leads to severe regressions on other benchmarks," whereas the unified mixture "keeps each RL update informed by the complete environment distribution."

The RL algorithm is an asynchronous GRPO (Group Relative Policy Optimization) setup:

• Decoupled training and inference: Inference workers continuously generate trajectories (rollouts), which are stored in a buffer. When a batch of trajectories is ready, it is sent to the training engine for a gradient update.
• 256 prompts per step, 16 responses per prompt (4,096 trajectories per batch = one gradient update per rollout).
• Maximum generation length: 49K tokens initially, later increased to 64K.
• In-flight weight updates: Training can push updated weights to inference workers mid-rollout, so a single trajectory may contain tokens from different model versions. The KV cache is not recomputed after weight updates.
• Policy lag management: Inference workers are restricted to at most one step behind the latest model version to avoid excessive policy lag.
• Importance sampling masking: To stabilize training despite off-policy effects from the training-inference mismatch, the importance sampling ratio (computed from training and inference log-probabilities) is masked.

Difficulty-based curriculum. Prompts where the SFT model consistently provides correct answers are filtered out (since no learning signal). Remaining samples are sorted by difficulty for a curriculum.

Low-effort reasoning during RL. A subset of prompts is converted to low-effort mode, where the reward is adjusted as a function of correctness and generated token count. The low-effort mix starts at 2% of RL prompts (math, STEM QA, competitive coding) and is reduced to 1% (math and STEM QA only). For competitive coding, only problems withheld from SFT data are used for low-effort training, ensuring the model hasn't memorized solutions.

Stage 2: SWE-RL (end-to-end RL for software engineering). This stage addresses software engineering tasks specifically, separated from Stage 1 because "SWE rollouts are substantially slower to generate and typically require longer context lengths, creating a throughput bottleneck when co-trained with shorter-horizon environments."

The environment (Section 3.2.5):

• Each rollout launches an Apptainer container with the target repository (Apptainer is used because rootless Docker is unavailable on the cluster).
• An OpenHands agent loop produces a code patch by interacting with the repo via bash commands and file operations in a tmux-based session.
• Binary reward: The patch is evaluated against ground-truth tests.
• Multi-harness diversity: OpenCode and Codex agent classes are implemented within OpenHands, matching the tool formats of Claude Code and Codex CLI, so the model trains with multiple harnesses (varying tools and prompts) while reusing a single environment infrastructure.
• Memory management: A watchdog daemon monitors the tmux process tree RSS and kills runaway processes before they cause OOM on the shared host.
• Command blocklist: A regex-based filter blocks dangerous commands (killall, pkill) that could affect training processes or vLLM servers on the same node.
• Serialization: orjson replaces Python's json for HTTP payload serialization between the gym and model server, since trajectory payloads (token IDs, log probabilities) are large.

Stage 3: RLHF (Reinforcement Learning from Human Feedback). This stage uses a GenRM (generative reward model) trained from Qwen3-235B-A22B-Thinking-2507 as the reward signal. The GenRM is trained as a principle-following model (Wang et al., 2025b), meaning it can be guided by explicit principles covering domains like identity and safety. Training data: Helpsteer 3 (Wang et al., 2025c), commercially-friendly subsets of lmarena-140k (Chiang et al., 2024), and additional human preference data. Unlike Nemotron 3 Nano, the GenRM is used throughout Stage 1 as well as in a dedicated RLHF-only stage at the end.

PivotRL for agentic tasks (Section 3.2.4). A key efficiency challenge: agentic tasks involve long-horizon multi-turn interactions with environments (conversational tool use, code editing, terminal interaction, web search). SFT on expert trajectories is cheap but prone to out-of-distribution degradation. End-to-end RL avoids OOD issues but is extremely expensive because every update requires online interactive rollouts.

PivotRL (Yi et al., 2026) is an "assistant-turn-level RL method that addresses this tradeoff by reusing offline SFT expert trajectories during RL." The core idea:

1. Identify "pivots" — informative turns within SFT traces where the current policy has uncertainty over the next action.
2. Apply a domain-appropriate reward to match the policy's action to the expert action.
3. The model gets credit for actions similar to the expert action, not only for exact matches.

This allows efficient agentic RL without the OOD degradation of SFT. PivotRL is applied for all agentic domains: Agentic Programming, Search, Terminal Use, and Conversational Tool Use.

Stage 4: MTP Healing. The MTP heads are trained with the rest of the model frozen. Prompts from the RLVR stage are reused, and the MTP head is trained using standard negative log-likelihood on the generated responses. This "significantly improves MTP accuracy."

Infrastructure (Section 3.2.5).

• NeMo Gym + NeMo RL integration: NeMo RL controls the training loop using Megatron-Core for distributed training, routing all rollouts through NeMo Gym and vLLM.
• Ray orchestration: Both NeMo RL and NeMo Gym use Ray for resource management, deployed on SLURM. Megatron training workers, vLLM generation workers, Gym environments, and judge models are all scheduled on a single Ray cluster.
• Resiliency at 1K GPU scale: Hardware failures required full job restarts. Optimizations: parallel initialization, prefetching virtual environments and binaries, caching in upstream repos (vLLM, flashinfer). Port conflicts from TOCTOU (time-of-check to time-of-use) race conditions were a significant failure mode: Ray control plane, vLLM workers, TCP rendezvous, and NeMo Gym servers all needed ports, and parallel initialization exacerbated latent race conditions where a component would check port availability without exclusive claiming.

---

Post-Training Quantization: FP8 and NVFP4 with AutoQuantize

Two deployment checkpoints are produced: FP8 (W8A8 for Hopper GPUs) and NVFP4 (W4A4 for Blackwell GPUs, with mixed precision).

FP8 checkpoint (Section 4.1). Calibration uses 256 samples from the post-training SFT dataset at 65536 context length. Quantized operations: MoE GEMMs (routed and shared experts), Mamba linear layers. KV cache is stored in FP8; the Mamba state cache is stored in FP16 (not FP8) for speedup. Attention GEMMs (QKV projection and output projection), MoE latent projection GEMMs, and output layers remain in BF16.

NVFP4 checkpoint (Section 4.2). The NVFP4 PTQ recipe combines three techniques:

1. Weight per-block scale optimization: Instead of using the maximum absolute value per block (the default), weight per-block scales are swept to minimize per-block weight MSE. This is an offline calibration (done once, not at runtime), so the cost is acceptable.

2. Dynamic per-block max-based activation scaling: Activations are quantized with max-based per-block scaling at runtime, since scale search algorithms are impractical for online use.

3. AutoQuantize mixed-precision search (Appendix B.2): A neural architecture search (NAS)-inspired method selects per-operator precision from {NVFP4, FP8, BF16} under a sensitivity-cost optimization.

AutoQuantize algorithm. For each operator $i$ and candidate format $f$, the sensitivity is:

$S(\text{Op}_i, Q_{i,f}) \approx \sum_{k=1}^d (\Delta Y_{i,k})^2 (g_{i,k})^2$

where $\Delta Y_i = Y^{\text{bf16}}_i - Y^{Q_{i,f}}_i$ is the difference between BF16 output and quantized output at the measurement point, and $g_i = \nabla_{Y_i} \mathcal{L}$ is the gradient of the loss with respect to that output.

What this computes: A second-order Taylor approximation to the increase in loss caused by quantizing operator $i$. The diagonal Fisher approximation uses the squared gradient as a proxy for the diagonal Hessian — it measures how much the loss changes in each output dimension, weighted by how sensitive the loss is to changes in that dimension. Dimensions where the gradient is large AND the quantization error is large contribute most to the sensitivity.

Why this form: The full Hessian is too expensive to compute and store. The diagonal Fisher approximation is standard in quantization literature (GPTQ, LLM-MQ) because it is cheap (requiring only a forward and backward pass on calibration data) and captures the first-order interaction between quantization error and loss sensitivity.

The performance cost is defined as $\text{FLOPs}(\text{Op}_i, Q_{i,f})$. AutoQuantize then solves:

$\min_{\{f\}} \sum_i S(\text{Op}_i, Q_{i,f}) \quad \text{s.t.} \quad \sum_i C(\text{Op}_i, Q_{i,f}) \leq B$

where $B$ is the total deployment cost budget. The paper uses an "effective-precision budget of 4.75 bits," which constrains the average bit-width across searched operators.

Deployment-restriction-aware search. The optimization respects two hardware constraints:

• QKV fusion: The Q, K, V projection GEMMs must share one format (since they are typically fused into a single kernel). The sensitivity and cost are aggregated as $S(\text{Op}_{\text{qkv}}) = S(\text{Op}_{\text{q}}) + S(\text{Op}_{\text{k}}) + S(\text{Op}_{\text{v}})$ and similarly for cost.
• MoE expert coupling: All sparse experts within a layer must share one format (due to vLLM/TRT-LLM API constraints restricting all experts in a group to the same quantization). The sensitivity is measured at the MoE block output, and cost is summed over experts.

Final NVFP4 precision assignments (Table 7).

• Sparse expert (routed) GEMMs: NVFP4 throughout (AutoQuantize selected this for all experts, confirming the architecture is robust to 4-bit quantization)
• Shared expert GEMMs: Mix of NVFP4, FP8, BF16 (some layers more sensitive than others)
• MoE latent projection GEMMs: FP8 or BF16 (searched)
• Attention QKV and output projections: FP8 or BF16 (searched)
• Mamba projection GEMMs: FP8 or BF16 (searched)
• Mamba 1D Conv, attention BMM2, output layers: BF16 (not searched — these are small or sensitive)
• KV Cache + attention BMM1: FP8 (not searched)
• Router, embedding: BF16/FP32 (not searched)

The AutoQuantize search ran on a single B200 node (8 GPUs) using 512 samples at sequence length 4096, completing in under 2 hours. The resulting model achieves 99.8% median accuracy relative to BF16.

Mamba state cache quantization (Section 4.3). The Mamba SSM cache is stored in FP32 by default to preserve numerical precision during recurrent updates. Storing it in FP16 saves memory bandwidth (important in memory-bound decoding) but introduces quantization error that accumulates over time because the Mamba recurrence feeds previous-step quantized states into the next step's update.

The error accumulation can be seen by unrolling the recurrence (Equation 3):

$h_{q,t} = h_t + \sum_{i=0}^t \left( \prod_{j=i+1}^t A_j \right) e_i$

where $h_t$ is the true (FP32) state at step $t$, $h_{q,t}$ is the quantized state, $A_t$ is the state transition matrix at step $t$, and $e_t$ is the quantization error introduced at step $t$. The sum shows that error from earlier steps is propagated through subsequent state transitions, accumulating coherently.

What this equation computes: The quantized state at time $t$ equals the true state plus the sum of all previous quantization errors, each weighted by the product of state transition matrices from that error's time step to the current time. The earlier the error occurs, the more transition matrices it passes through, potentially amplifying its effect.

Why this matters: Standard round-to-nearest-even (RTNE) quantization introduces biased error — it always rounds the same value to the same representation, so the error has zero variance but non-zero bias. In a recurrent setting, this bias accumulates coherently. The error terms $e_i$ are systematically non-zero in the same direction, and their weighted sum grows over time.

The solution: stochastic rounding. Replacing RTNE with stochastic rounding (SR): a value falling between two representable FP16 levels is rounded up with probability proportional to its distance to the upper level, and down otherwise. This makes the error unbiased in expectation ($\mathbb{E}[e_t] = 0$), so the accumulated error becomes zero-mean noise rather than systematic drift.

Experimental validation (Table 9). Evaluated on LiveCodeBench and SciCode with multiple SSM cache recipes:

• FP32 (baseline): no verbosity increase, reference accuracy
• FP16: 37% verbosity increase (the model generates longer outputs to compensate for state degradation), accuracy maintained
• FP16 with stochastic rounding (Philox<5>): verbosity returns to baseline, accuracy maintained
• INT16 with per-block scaling over 128-element blocks along the state dimension: also eliminates verbosity (because per-block scaling increases effective dynamic range compared to global INT16 quantization)

Why Philox<5> was selected: (1) It avoids storing and loading per-block scale factors (unlike INT16 with block scaling), (2) Blackwell provides a dedicated PTX instruction for stochastic rounding during type conversion, (3) Blackwell supports Philox-based pseudorandom number generation through cuRAND. Philox<5> balances statistical quality of the pseudorandom numbers (more rounds = better randomness) against generation overhead (fewer rounds = faster).

4. Key Insights and Innovations

Innovation 1: The Expert Dimension Reduction → Expert Expansion Cycle as a Hardware–Software Co-Design Principle

The most intellectually distinctive contribution of this paper is not LatentMoE as a specific architecture, but the five design principles (Section 2.1.1) that argue for an entirely different way of thinking about MoE efficiency — one where the metric is accuracy per parameter and per byte, not accuracy per FLOP. This reframing matters because it exposes a structural inefficiency in how the field has been designing sparse models.

Prior MoE work — from GShard (Lepikhin et al., 2020) through DeepSeekMoE (Dai et al., 2024) and DeepSeek-V3 (DeepSeek-AI, 2025c) — optimized for a single regime: keep FLOPs constant by activating only a fraction of parameters per token, scale total parameters to increase representational capacity, and achieve better accuracy for the same compute. This framing assumes FLOPs are the bottleneck. The Nemotron 3 Super paper systematically identifies two deployment regimes where FLOPs are not the bottleneck — low-latency serving (dominated by memory bandwidth for expert weight loads) and throughput-oriented serving (dominated by all-to-all communication) — and shows that the hidden dimension d is the only architectural variable that simultaneously addresses both regimes (Principles 1 and 2).

What makes this a genuine conceptual advance rather than an engineering optimization is the cycle it identifies: reduce d by a factor of α to save memory bandwidth and communication, but compensate by increasing top-K by α to preserve the effective nonlinear budget K·m (Principle 3). This is not a one-time compression; it is a structural tradeoff that generates a new design axis — you can now dial up K and N together (Principle 5) at constant inference cost because the per-expert cost has been reduced. The paper essentially argues that standard MoE designs have been operating at a suboptimal point on a new Pareto frontier: they leave inference efficiency on the table because they treat d as fixed by convention rather than as a variable to be optimized jointly with K and N.

The empirical validation comes from the competitive benchmark accuracy of Nemotron 3 Super against models using full-dimension MoE (Table 5), but the conceptual contribution is the diagnostic framework itself — the identification that FLOPs-centric MoE design answers the wrong question for deployment, and that a dimension-reduction-and-expert-expansion cycle is the correct response. This is a fundamental reframing, not an incremental improvement, because it changes the objective function for MoE architecture design from "maximize total parameters at constant FLOPs" to "co-optimize the dimension, expert count, and active expert count against the actual serving bottleneck."

Innovation 2: Shared-Weight Multi-Offset Training as a Solution to the Speculative Decoding Training–Inference Mismatch

Multi-Token Prediction itself is not new — Gloeckle et al. (2024) introduced the training objective, and DeepSeek-V3 (DeepSeek-AI, 2025c) deployed fixed-offset MTP heads for native speculative decoding. What is new and conceptually elegant is the diagnosis of a specific failure mode in fixed-offset heads and the shared-weight solution that emerges from it.

The failure mode: a head trained to predict token n+3 from hidden state h_n was trained on ground-truth intermediate tokens. At inference, when that head is asked to predict token n+4 using its own (potentially incorrect) draft of token n+3 as context, it encounters hidden states from an incorrect trajectory — a distribution it never saw during training. This causes acceptance rates to collapse at longer draft lengths (the "training–inference mismatch").

The standard response to this problem would be engineering-oriented: train more independent heads (one per offset, no recursion), or use an external draft model, or simply accept shorter draft lengths. Nemotron 3 Super's insight is to treat the mismatch as a generalization problem: if the head is trained on multiple offsets, it learns to predict future tokens from hidden states at various distances — including hidden states that may themselves be the result of imperfect predictions, because the training data implicitly mixes "correct" and "potentially incorrect" hidden state contexts across different offsets. The shared parameters are regularized to be robust to the variance in hidden state quality.

This is a small architectural change — share weights between two MTP heads — but it represents a fundamentally different mental model: rather than viewing speculative decoding as "use auxiliary heads as independent drafters," it views it as "train a single generalizable future-token predictor that can be applied recursively." The evidence is in Table 2 and Figure 4: Nemotron 3 Super achieves the highest average acceptance length (3.45) on SPEED-Bench, with the advantage over DeepSeek-R1 widening at draft positions 4–7 where recursive drafting dominates. The throughput–latency improvement in Figure 5 shows this translates to real serving gains.

The significance beyond performance is that it suggests a more general principle: when a model component will be used autoregressively at inference, training it on diverse auto-regressive contexts rather than oracle contexts improves robustness to distribution shift. This has implications beyond MTP — it applies to any setting where a model conditions on its own outputs during deployment.

Innovation 3: NVFP4 Pretraining Stability Analysis at 120B Scale as Both Demonstration and Diagnostic Contribution

Training a 120B-parameter model in 4-bit precision is a significant engineering achievement, but the paper's deeper contribution is the forensic analysis of what goes wrong — the documentation of the zero-gradient phenomenon (Section 2.2) and its mechanistic attribution to NVFP4 underflow in expert layers. This transforms NVFP4 pretraining from a "recipe that works" into a "phenomenon that is understood."

The key diagnostic sequence:

1. Zero-valued weight gradient elements reach 7% of total parameters by the end of training.
2. These zeros correlate with emerging channel magnitude patterns in expert layers — low-norm FC1 output channels aligning with low-norm FC2 input channels (Figure 6).
3. Controlled ablation on Nemotron 3 Nano (Figure 7) shows that NVFP4 at 1T tokens produces similar zero-gradient counts as BF16 at 25T tokens, and switching from NVFP4 to BF16 mid-training restores normal gradient behavior.
4. Tensor-level sampling (Figure 8) traces the zeros to specific underflow locations: dgrad of FC2 at 500B tokens, then fprop of FC1 at 750B tokens.

What makes this a genuine contribution rather than a war story is the causal claim: NVFP4 does not cause fundamentally new training dynamics — it accelerates a process that would happen more slowly in BF16. Low-norm expert channels are already "dying" in the BF16 model (visible as small-but-nonzero gradient magnitudes), but NVFP4's limited dynamic range underflows these small values to exactly zero, which then propagates back through the computation graph. The paper is essentially arguing that NVFP4 acts as a gradient sparsification mechanism that reveals which parameters are marginal.

This is significant because it provides guidance for future low-precision training efforts: (1) monitor expert channel norms as an early warning signal, (2) expect zero-gradient accumulation but verify it does not cause instability, (3) the MXFP8 healing experiment (Figure 9) shows that recovering precision before annealing does not improve downstream accuracy, suggesting the zeroed-out parameters were genuinely uninformative rather than victims of quantization noise. This is a diagnostic contribution — it gives the field a vocabulary and measurement protocol for assessing whether low-precision training is degrading model quality or merely accelerating natural sparsification.

Innovation 4: The Two-Stage SFT Loss as a Targeted Intervention for a Specific Degradation Pattern, Not a Generic Recipe

The paper's SFT loss design — token-level global average in Stage 1, per-conversation average in Stage 2 — is easy to overlook as an implementation detail, but it represents a precise diagnostic-and-fix cycle that is rarely documented this clearly in LLM technical reports. The diagnostic move: "Single-stage SFT led to a marked degradation on long-input-short-output scenarios." The causal hypothesis: token-level normalization makes long reasoning traces dominate the loss, causing the model to produce verbose outputs even when short responses are appropriate.

The solution is not a hyperparameter sweep or a new loss function — it is a two-stage curriculum where the normalization scheme changes. Stage 1 uses token-level averaging (Equation 1) to induce strong reasoning behavior by giving long traces their proportional weight. Stage 2 switches to per-conversation averaging (Equation 2) to rebalance the loss so that a conversation with 100 output tokens and one with 10,000 output tokens contribute equally to the gradient. The paper explicitly states that this "restores long-input-short-output performance while retaining reasoning."

What elevates this from a practical trick to an insight is the non-obvious interaction it reveals between loss normalization and output length behavior. The dominant assumption in the field is that normalizing by the total number of tokens across the batch (global average) is the correct default — it treats every token as an equally weighted training example. This paper demonstrates that this default creates a systematic bias toward verbose outputs that is invisible in aggregate metrics (since reasoning benchmarks, which reward verbosity, will look fine) but causes real failures in deployment (when users ask for short factual answers). The per-conversation normalization is not obviously better — it underweights tokens from long conversations — but it is the right choice when the target behavior includes both long-form reasoning and short-form responses.

The significance is that it provides a diagnostic lens: if your instruction-tuned model is too verbose on short-answer tasks, the loss normalization might be the culprit, not the data mixture or the RL reward. This is a conceptually novel contribution because it identifies a mechanism (loss normalization scheme) that is typically treated as an engineering default but actually encodes a substantive assumption about what the model should optimize.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The base model is evaluated on a comprehensive suite spanning general knowledge (MMLU, MMLU-Pro, AGIEval-En, GPQA-Diamond), mathematical reasoning (GSM8K, MATH, MATH Level 5, AIME 2024), code generation (HumanEval, MBPP-Sanitized), commonsense understanding (ARC-Challenge, HellaSwag, OpenBookQA, PIQA, WinoGrande), reading comprehension (RACE), multilingual capability (MMLU Global Lite, MGSM), and long-context retrieval (RULER at 64K, 128K, 256K, 512K, and 1M). The post-trained model is evaluated on an extended suite adding reasoning benchmarks (AIME25, HMMT Feb25, GPQA, LiveCodeBench v5, SciCode, HLE), agentic benchmarks (Terminal Bench, SWE-Bench across multiple harnesses, TauBench V2, BrowseComp, BIRD Bench), chat/instruction following (IFBench, Scale AI Multi-Challenge, Arena-Hard-V2), and additional long-context (AA-LCR, RULER at extended lengths) and multilingual benchmarks (MMLU-ProX, WMT24++). All evaluations use the Nemo Evaluator SDK, with specific containers and harness details publicly documented for reproducibility (Section 3.3).

• Base model(s). The primary model is Nemotron 3 Super 120B-A12B Base (BF16), a 120.6B total parameter hybrid Mamba-Attention MoE model with 12.7B active parameters per forward pass, pretrained on 25 trillion tokens. For the base model comparison (Table 4), the baselines are Ling-flash-Base-2.0 and GLM-4.5-Air-Base, selected as "similarly sized state-of-the-art base models." For the post-trained comparison (Table 5, Figure 1), the baselines are GPT-OSS-120B (MXFP4 quantization) and Qwen3.5-122B-A10B (BF16). The ~14× larger model from the FLOPs-matched comparison referenced in the executive summary is GPT-OSS-120B at the post-trained level.

• Metrics. Accuracy is the primary metric, operationalized differently per task: exact match (EM) for math benchmarks (GSM8K, MATH, AIME24, HMMT, GPQA), pass@1 estimated from 32 generations for coding benchmarks (HumanEval, MBPP using the EvalPlus variants), accuracy_norm for ARC-Challenge and HellaSwag, and standard accuracy for MMLU and similar multiple-choice benchmarks. Agentic benchmarks use task-specific pass/fail criteria: SWE-Bench uses the standard resolved rate, Terminal Bench uses success rate on the hard subset, TauBench V2 uses per-domain accuracy, BrowseComp uses answer accuracy, and BIRD Bench uses execution accuracy. For throughput, the metric is output tokens per second per GPU measured on B200 GPUs using the best of vLLM or TRT-LLM per model (Figure 1). Quantization accuracy is reported as median accuracy relative to the BF16 baseline across the evaluation suite (Section 4.2: 99.8%).

• Baselines. For base model evaluation: Ling-flash-Base-2.0 and GLM-4.5-Air-Base. For post-trained evaluation: GPT-OSS-120B-A5B-MXFP4 (served in MXFP4 with MXFP8 activations and FP8 KV-Cache) and Qwen3.5-122B-A10B-BF16 (served in BF16). For speculative decoding: DeepSeek-R1 and Qwen3-Next on SPEED-Bench (Table 2). For quantization: the BF16 checkpoint serves as the accuracy baseline for both FP8 and NVFP4 quantized checkpoints (Table 8).

• Generation budget / compute accounting. For throughput comparisons, all models are measured on B200 GPUs on the 8K input sequence length / 64K output sequence length setting, using the best framework (vLLM or TRT-LLM) per model to represent realistic deployment conditions. Throughput is reported as relative throughput normalized to Nemotron 3 Super BF16 (Figure 1, Table 5). For speculative decoding, the draft length is fixed at 7 tokens and the metric is average acceptance length (tokens accepted per verification step). For coding benchmarks, pass@1 is estimated from 32 generations per prompt. For the zero-gradient analysis, the token horizon is used as the compute scale (e.g., comparisons at 500B, 750B, 1T, 25T tokens).

• Cross-validation / statistical protocol. The paper does not report formal cross-validation or confidence intervals for its main benchmark results. For the pretraining checkpoint merging analysis (Section 2.5), the best merge is selected among three sliding window sizes (125B, 250B, 500B tokens) evaluated offline. For the SFT and RL data curation, held-out sets are used for competitive coding low-effort prompts and for evaluating data contamination (e.g., decontamination against HumanEval, MBPP, CRUXEval, LiveCodeBench using exact match and embedding similarity). For the quantization AutoQuantize search, calibration uses 512 samples from the SFT dataset at sequence length 4096, but final evaluation is on the full benchmark suite. The reproducibility is addressed through public release of evaluation containers, configurations, and the Nemo Evaluator SDK.

Main Quantitative Results

Base Model Accuracy vs. Similarly Sized Base Models

Table 4 compares Nemotron 3 Super 120B-A12B Base against Ling-flash-Base-2.0 and GLM-4.5-Air-Base across 22 benchmarks. The headline result is that Nemotron 3 Super achieves the best score on 18 of 22 benchmarks.

In the general knowledge category, Nemotron 3 Super scores 86.01 on MMLU (5-shot) vs. 81.00 for both Ling-flash and GLM-4.5, and 75.65 on MMLU-Pro (5-shot CoT) vs. 62.10 and 58.20 — gaps of approximately 5 and 13–17 percentage points, respectively. On GPQA-Diamond (5-shot CoT), the score is 60.00 vs. 36.00 (Ling-flash) and 23.20 (GLM-4.5), a 24–37 point advantage.

In mathematical reasoning, Nemotron 3 Super scores 84.84 on MATH (4-shot) vs. 63.80 (Ling-flash) and 50.36 (GLM-4.5), a gap of 21–34 points. On the harder MATH Level 5 slice, the gap widens: 70.00 vs. 39.80 and 26.30. On AIME 2024 pass@32, Nemotron 3 Super achieves 53.33 vs. 30.00 and 20.00. GSM8K is the one math benchmark where the model does not lead: 90.67 vs. 90.75 for Ling-flash (essentially tied) and 82.60 for GLM-4.5.

In code generation, Nemotron 3 Super leads on HumanEval (0-shot pass@1: 79.40 vs. 70.10 for Ling-flash and 76.30 for GLM-4.5) and MBPP-Sanitized (78.38 vs. 77.30 and 77.50), though margins are narrower than in math.

In commonsense understanding, Nemotron 3 Super leads on ARC-Challenge (96.08 vs. 94.80, 93.90), HellaSwag (88.97 vs. 84.69, 87.70), OpenBookQA (50.20 vs. 47.00, 48.60), and PIQA (85.47 vs. 84.00, 84.22), but not on WinoGrande (78.93 vs. 78.37, 83.82).

In long-context capability, the advantage is substantial and grows with context length: RULER 64K scores 92.26 vs. 72.12 (Ling-flash) and 80.26 (GLM-4.5); at 128K, 88.26 vs. 52.03 and 61.70. Nemotron 3 Super is the only model with results reported above 128K, achieving 84.56 at 256K, 82.49 at 512K, and 71.00 at 1M.

Post-Trained Model Accuracy and Throughput vs. GPT-OSS-120B and Qwen3.5-122B

Table 5 and Figure 1 present the post-trained comparison. The central claim is that Nemotron 3 Super achieves "comparable accuracy" while providing 2.2× higher throughput than GPT-OSS-120B and 7.5× higher than Qwen3.5-122B on the 8K/64K input/output setting.

Throughput advantage (Figure 1, bar chart). On the 8K/64K ISL/OSL configuration on B200 GPUs, relative throughput normalized to Nemotron 3 Super BF16 (set to 1.0):

• Nemotron 3 Super NVFP4: 2.2 (2.2× the BF16 checkpoint)
• GPT-OSS-120B MXFP4: 0.6
• Nemotron 3 Super BF16: 1.0
• Qwen3.5-122B BF16: 0.3

The 7.5× advantage over Qwen3.5-122B comes from 2.2 / 0.3 ≈ 7.3, with rounding to 7.5 in the text. The advantage over GPT-OSS-120B is 2.2 / 0.6 ≈ 3.7× for the NVFP4 checkpoint, though the text reports "up to 2.2× higher throughput" (likely referencing the BF16-to-BF16 comparison: 1.0 / 0.6 ≈ 1.7×, or comparing NVFP4 to GPT-OSS at 2.2 / 0.6 ≈ 3.7×, with the 2.2× figure drawn more conservatively).

Reasoning capabilities (Table 5, upper section). Nemotron 3 Super is competitive on math reasoning benchmarks:

• AIME25 (no tools): 90.21 vs. 90.36 (Qwen3.5), 92.50 (GPT-OSS)
• HMMT Feb25 (no tools): 93.67 vs. 91.40, 90.00 — Nemotron 3 Super leads
• HMMT Feb25 (with tools): 94.73 vs. 89.55 for Qwen3.5 (GPT-OSS not reported with tools)
• GPQA (no tools): 79.23 vs. 86.60 (Qwen3.5, a 7-point gap), 80.10 (GPT-OSS)
• GPQA (with tools): 82.70 vs. 80.09 for GPT-OSS (Qwen3.5 not reported)
• LiveCodeBench v5: 81.19 vs. 78.93 (Qwen3.5), 88.00 (GPT-OSS, a 7-point gap)
• SciCode: 42.05 vs. 42.00, 39.00 — essentially tied with Qwen3.5
• HLE (no tools): 18.26 vs. 25.30 (Qwen3.5, a 7-point gap), 14.90 (GPT-OSS)
• HLE (with tools): 22.82 vs. 19.0 for GPT-OSS (Qwen3.5 not reported)

The pattern is that Nemotron 3 Super leads on tool-augmented math and HLE, trails Qwen3.5-122B on GPQA and HLE without tools by 7 points, and trails GPT-OSS-120B on LiveCodeBench by 7 points. On AIME25, HMMT, and SciCode, it is competitive (within 2 points).

Agentic capabilities (Table 5, middle section). Across SWE-Bench, TauBench, and terminal use benchmarks, Nemotron 3 Super generally outperforms GPT-OSS-120B and is competitive with Qwen3.5-122B on some harnesses:

• SWE-Bench (OpenHands): 60.47 vs. 66.40 (Qwen3.5), 41.9 (GPT-OSS) — trails Qwen3.5 by 6 points, leads GPT-OSS by 18.6 points
• SWE-Bench (OpenCode): 59.20 vs. 67.40 (Qwen3.5)
• SWE-Bench (Codex): 53.73 vs. 61.20 (Qwen3.5)
• SWE-Bench Multilingual (OpenHands): 45.78 vs. 30.80 (GPT-OSS) — leads by 15 points
• Terminal Bench (hard subset): 25.78 vs. 26.80 (Qwen3.5), 24.00 (GPT-OSS)
• Terminal Bench Core 2.0: 31.00 vs. 37.50 (Qwen3.5, a 6.5-point gap), 18.70 (GPT-OSS)
• TauBench V2 Average: 61.15 vs. 74.53 (Qwen3.5, a 13-point gap), 61.0 (GPT-OSS) — the gap to Qwen3.5 is driven by the Telecom domain (64.36 vs. 95.00, a 30.6-point gap)
• BrowseComp with Search: 31.28 vs. 33.89 (GPT-OSS)
• BIRD Bench: 41.80 vs. 38.25 (GPT-OSS) — leads by 3.6 points

The agentic results show a clear hierarchy: Qwen3.5-122B leads on most agentic benchmarks, often by significant margins (6–13 points on SWE-Bench, Terminal Bench Core, TauBench), Nemotron 3 Super consistently outperforms GPT-OSS-120B (by 15–18 points on SWE-Bench, 12 points on Terminal Bench Core), and the two extremes are the TauBench Telecom domain (where Qwen3.5 scores 95.00 vs. Nemotron 3 Super's 64.36) and SWE-Bench Multilingual (where Nemotron 3 Super leads GPT-OSS by 15 points).

Chat, instruction following, long context, multilingual (Table 5, lower sections).

• IFBench: 72.56 vs. 73.77 (Qwen3.5), 68.32 (GPT-OSS) — competitive
• Scale AI Multi-Challenge: 55.23 vs. 61.50 (Qwen3.5), 58.29 (GPT-OSS) — trails both by 3–6 points
• Arena-Hard-V2: 73.88 vs. 75.15 (Qwen3.5), 90.26 (GPT-OSS) — trails GPT-OSS by 16.4 points
• RULER at 256K: 96.83 vs. 96.74 (Qwen3.5), 52.30 (GPT-OSS)
• RULER at 512K: 95.22 vs. 95.95 (Qwen3.5), 46.70 (GPT-OSS)
• RULER at 1M: 91.64 vs. 91.33 (Qwen3.5), 22.30 (GPT-OSS)
• MMLU-ProX: 79.36 vs. 85.06 (Qwen3.5), 76.59 (GPT-OSS)

In long context, Nemotron 3 Super and Qwen3.5-122B are essentially tied across all RULER lengths, while GPT-OSS-120B degrades sharply above 256K. On Arena-Hard-V2, GPT-OSS-120B's 90.26 is 16.4 points above Nemotron 3 Super's 73.88 — the largest single-benchmark gap.

Quantization Accuracy Preservation

Table 8 compares the BF16, FP8, and NVFP4 post-trained checkpoints across the evaluation suite. The critical metric: the NVFP4 checkpoint preserves 99.8% median accuracy relative to BF16.

Key individual benchmark comparisons:

• MMLU-Pro: 83.73 (BF16) → 83.63 (FP8) → 83.33 (NVFP4) — loss of 0.40 points (0.48%)
• HMMT Feb25 (with tools): 94.73 → 94.38 → 95.36 — NVFP4 slightly outperforms BF16 by 0.63 points (within noise)
• GPQA (no tools): 79.23 → 79.36 → 79.42 — both quantized versions slightly outperform BF16 (within noise)
• LiveCodeBench v5: 81.19 → 80.99 → 80.56 — loss of 0.63 points (0.78%)
• SWE-Bench (OpenCode): 60.47 → not reported for FP8 → 59.90 — loss of 0.57 points (0.94%)
• TauBench V2 Average: 61.15 → 61.07 → 60.46 — loss of 0.69 points (1.13%)
• Arena-Hard-V2: 73.88 → 76.06 → 76.00 — both quantized versions outperform BF16 by ~2 points
• RULER 1M: 91.64 → 91.43 → 91.60 — essentially unchanged
• MMLU-ProX: 79.35 → 79.21 → 79.37 — essentially unchanged

The NVFP4 checkpoint occasionally outperforms BF16 (HMMT, GPQA, Arena-Hard-V2), which likely reflects evaluation noise or the regularization effect of quantization rather than genuine improvement. The maximum degradation on any single benchmark is approximately 1.6 points (Scale AI Multi-Challenge: 55.23 → 52.8, a 4.4% drop), but the median across all benchmarks is 99.8% of BF16 accuracy.

Speculative Decoding Performance (MTP)

Table 2 reports average acceptance lengths on SPEED-Bench with a fixed draft length of 7 tokens:

• Nemotron 3 Super: 3.45 average across all 11 categories
• Qwen3-Next: 3.33
• DeepSeek-R1: 2.70

Nemotron 3 Super leads on 8 of 11 categories, including the largest gaps on Multilingual (4.05 vs. 3.97, 2.83), RAG (3.78 vs. 3.53, 2.79), Coding (3.78 vs. 4.32 [Qwen3-Next leads], 2.99), and Roleplay (2.82 vs. 2.17, 2.19). Qwen3-Next leads on Coding (4.32 vs. 3.78) and Math (3.89 vs. 3.73).

Figure 4 breaks down acceptance rate by draft token index. At draft index 1, all three models have acceptance rates close to 1.0 (roughly 0.98–1.0). The decline is monotonic for all models. At index 7 (the longest draft), Nemotron 3 Super maintains approximately 0.20–0.25 acceptance vs. ~0.15 for DeepSeek-R1, with Qwen3-Next between them. The gap between Nemotron 3 Super and DeepSeek-R1 widens from negligible at index 1 to approximately 5–10 percentage points at indices 4–7.

Figure 5 demonstrates the throughput–latency benefit: on a B300 GPU with TRT-LLM (TP=1), increasing MTP draft depth from disabled to D=3 shifts the Pareto frontier outward. For example, at a user throughput of roughly 10 requests/second, the median latency drops from approximately 12 seconds (MTP off) to ~8 seconds (D=3), a 33% latency reduction at constant throughput.

Long-Context Scaling

Table 4 shows RULER performance for the base model from 64K to 1M context:

• 64K: 92.26
• 128K: 88.26
• 256K: 84.56
• 512K: 82.49
• 1M: 71.00

The post-trained model (Table 5) maintains or improves on these numbers: RULER 256K at 96.83, 512K at 95.22, and 1M at 91.64. The improvement from base to post-trained is particularly notable at 1M (71.00 → 91.64, a 20.6-point gain), suggesting the long-context SFT and RLHF stages substantially improve the model's ability to use extended context. GPT-OSS-120B degrades dramatically: 52.30 at 256K, 46.70 at 512K, and 22.30 at 1M. Qwen3.5-122B matches Nemotron 3 Super closely at all lengths (96.74, 95.95, 91.33).

Pretraining Checkpoint Merging

Figure 11 shows the average benchmark score (12 benchmarks) over the 25T token training run, comparing trained checkpoints to the best offline checkpoint merge. During the stable LR phase (tokens 0–20T), the best merge consistently outperforms the trained checkpoint by 2–4 points on the unweighted average. During the LR decay phase (20T–25T, shaded), the gap narrows substantially, and by the end of training, the two trajectories "largely coincide." The paper reports that the final base model checkpoint selected for downstream alignment was itself a 500B-token merge. Per-benchmark breakdowns (Appendix Figure 17) confirm this pattern across all 12 benchmarks individually, with the merge advantage being most pronounced on HumanEval, HumanEval+, MATH-500, MMLU-Pro, and MBPP+.

Ablation Studies and Robustness Checks

NVFP4 low-precision pretraining stability: The paper documents the zero-valued weight gradient accumulation phenomenon (7% of parameters by 25T tokens) and traces it to NVFP4 underflow via controlled experiments on Nemotron 3 Nano (Figures 6–8). The key ablation: an NVFP4-trained model switched to BF16 at 0.5T tokens shows zero-valued gradient counts returning to baseline levels (Figure 7), establishing that the effect is precision-dependent rather than architecture-dependent. The MXFP8 healing ablation (Figure 9) shows that switching to higher precision before annealing (at 19T tokens) improves the loss trajectory but yields no downstream accuracy gains on any of eight benchmarks tested (MMLU Pro, MMLU, Math 500, GSM8K, MBPP Sanitized, ARC Challenge, HumanEval, WinoGrande) — the improvements (%) relative to the NVFP4 model oscillate around zero with no sustained positive trend.

Synthetic multiple-choice data ablation: Adding 1B tokens of MMLU-aux-train-SDG data to the last 100B tokens of a Nemotron Nano V3 24.9T checkpoint produces: MMLU 77.22 → 77.51 (+0.29), MATH Level 5 78.55 → 79.05 (+0.50), AIME-2024 53.3 → 56.7 (+3.4), MBPP 74.8 → 75.2 (+0.4), with other benchmarks stable (Section 2.3.6). This demonstrates that synthetic MCQ data primarily strengthens mathematical and structured reasoning.

Synthetic code data ablation: Adding the code concepts and unconditional algorithmic datasets to the last 100B tokens of a 25T pretraining run yields "1-2 points improvement to HumanEval, MBPP, and CRUXEval-O" (Section 2.3.3), confirming the benefit of synthetic code training data even at small scale (0.2B tokens for algorithmic data).

SFT loss normalization (two-stage vs. single-stage): The paper reports that single-stage SFT with token-level loss normalization alone "led to a marked degradation on long-input-short-output scenarios" (Section 3.1). The two-stage recipe (Stage 1 token-level, Stage 2 per-conversation normalization) "restores long-input-short-output performance while retaining reasoning." No quantitative ablation is reported in the main text for this specific comparison — the claim is qualitative.

MTP shared-weight design: Table 2 and Figure 4 demonstrate the benefit of shared-weight MTP heads through comparisons against DeepSeek-R1 (which uses fixed-offset heads) and Qwen3-Next (which uses a different MTP design). Nemotron 3 Super achieves 3.45 average acceptance length vs. 2.70 for DeepSeek-R1, a 28% relative improvement, with the gap widening at draft indices 4–7 (Figure 4). This is an architectural comparison across different models, not a controlled within-model ablation where shared vs. independent heads are compared on the same architecture.

MTP healing: After RLHF, the MTP healing stage (training MTP heads on RLVR prompts with frozen model weights) "significantly improves MTP accuracy" (Section 3.2.4). No quantitative ablation is reported to quantify "significantly."

RL multi-environment vs. single-environment training: The paper states that "single-environment training leads to severe regressions on other benchmarks" (Section 3.2.1), motivating the unified 21-environment mixture. No quantitative ablation comparing multi-environment to single-environment RL is reported in the main text.

PivotRL for agentic RL: PivotRL is described as a method that "greatly improves the efficiency of our agentic RL, without facing the OOD degradation issues of SFT" (Section 3.2.4). No ablation comparing PivotRL to standard RL or to SFT-only on agentic benchmarks is reported.

FP4 PTQ algorithm ablations (Table 10, Appendix B.1): The paper evaluates five PTQ algorithms on MMLU-Pro, GPQA, LiveCodeBench, and AA-LCR. The BF16 baseline scores are 83.49, 79.92, 72.907, 53.00 respectively. Default NVFP4 (max-based per-block scaling) achieves 82.99, 79.29, 70.18, 55.50 — a notable drop on LiveCodeBench (72.907 → 70.18). Weight per-block scales minimizing per-block MSE (the chosen method) achieves 83.31, 79.92, 71.37, 56.75 — slightly better than BF16 on MMLU-Pro, tied on GPQA, still degraded on LiveCodeBench by 1.5 points. GPTQ achieves 83.11, 80.05, 69.79, 57.87 — best on GPQA and AA-LCR, worst on LiveCodeBench. The final AutoQuantize recipe (Table 8) addresses the LiveCodeBench gap: the NVFP4 checkpoint scores 78.69 on LiveCodeBench v6 (vs. 78.69 for BF16 on the same version) and 80.56 on v5 (vs. 81.19 for BF16, a 0.63-point gap). This is substantially smaller than the 2.7-point gap in the naive NVFP4 ablation, demonstrating the importance of AutoQuantize's selective precision promotion.

Mamba SSM cache quantization (Table 9): Multiple SSM cache recipes are evaluated on LiveCodeBench and SciCode, measuring both accuracy (pass@1 avg-of-8) and verbosity (completion tokens). Key findings:

• Direct FP16 casting increases verbosity by 36.95% on LiveCodeBench and 2.19% on SciCode (BF16 weights/activations) and by 40.27% on LiveCodeBench (W8A8 weights/activations), while maintaining accuracy.
• FP16 with stochastic rounding (Philox<5>) recovers verbosity to baseline levels (−1.73% on LiveCodeBench under BF16, +1.79% under W8A8) while maintaining accuracy within ±1 point.
• INT16 with per-block scaling (block size 128) also recovers verbosity (+2.90% under W8A8) with comparable accuracy.
• Reducing Philox rounds from 5 to 3 causes verbosity to increase by 10.70% under W8A8, indicating insufficient pseudorandom quality. Philox<4> is acceptable (−0.63% verbosity), and Philox<5> is selected for a balance of quality and generation overhead.

The negative result: naive FP16 casting causes up to 40% verbosity increase — the model produces longer outputs to compensate for SSM state degradation — which is invisible in accuracy-only evaluations but would severely impact serving throughput in deployment.

FP8 vs. NVFP4 at scale (Table 8): The FP8 and NVFP4 checkpoints are compared across 20+ benchmarks. Both quantized checkpoints show minimal degradation relative to BF16, with occasional apparent improvements (likely noise). The NVFP4 checkpoint's median relative accuracy of 99.8% reflects that performance is preserved across the board, with the largest single-benchmark drop being Scale AI Multi-Challenge (55.23 → 52.8, a 4.4% relative loss).

Critical Assessment

Does the paper demonstrate that Nemotron 3 Super achieves "comparable accuracy" to GPT-OSS-120B and Qwen3.5-122B?

The claim requires careful qualification. On the aggregate, Nemotron 3 Super is competitive with both models but not consistently at parity. Against GPT-OSS-120B, Nemotron 3 Super leads on the majority of benchmarks — particularly agentic tasks (SWE-Bench by 18.6 points, Terminal Bench Core by 12.3 points, HLE with tools by 3.8 points), long-context (40+ point gaps at 256K+), and math with tools — but trails significantly on Arena-Hard-V2 (73.88 vs. 90.26, a 16.4-point gap) and LiveCodeBench v5 (81.19 vs. 88.00, a 6.8-point gap). These are not small differences. The characterization "comparable" is reasonable for the aggregate profile but masks substantial variance.

Against Qwen3.5-122B, the pattern is more consistently negative: Nemotron 3 Super trails on GPQA (79.23 vs. 86.60, −7.4), HLE no tools (18.26 vs. 25.30, −7.0), SWE-Bench across all harnesses (−6 to −8 points), Terminal Bench Core 2.0 (31.00 vs. 37.50, −6.5), TauBench V2 Average (61.15 vs. 74.53, −13.4, driven by Telecom), and MMLU-ProX (79.36 vs. 85.06, −5.7). It leads convincingly only on HMMT with tools (94.73 vs. 89.55, +5.2) and essentially ties on AIME25, Long Context, and IFBench. The throughput advantage of 7.5× is a genuine strength, but the accuracy claim is better described as "trails Qwen3.5-122B on most reasoning and agentic benchmarks by 5-13 points, leads on tool-augmented math and long-context, and matches on chat/instruction following" — which is not synonymous with "comparable" in the usual sense.

Does the 2.2×/7.5× throughput advantage hold under realistic conditions?

The throughput measurement (Figure 1) uses the best framework per model (vLLM or TRT-LLM), best quantization per model (MXFP4 for GPT-OSS, BF16 for Qwen3.5), B200 GPUs, and the specific 8K input / 64K output setting. This is a realistic deployment comparison, but it is highly sensitive to the output length. At 64K output tokens, the KV cache overhead of attention-heavy models (GPT-OSS, Qwen3.5) becomes proportionally larger relative to the Mamba-dominant Nemotron 3 Super, inflating the throughput gap. At shorter output lengths (e.g., 1K output tokens typical of single-turn chat), the gap would likely be smaller because KV cache is not the dominant bottleneck. The paper does not report throughput at multiple output lengths, which would characterize where the advantage is largest.

Furthermore, the comparison is per-GPU. Since Nemotron 3 Super has 12B active parameters vs. ~10B for Qwen3.5-122B, a per-GPU comparison is reasonable, but total cost-of-ownership would also depend on the number of GPUs needed to hold the model's weights. Nemotron 3 Super has 120B total parameters vs. 122B for Qwen3.5, so memory footprint is comparable. However, the 512 experts per layer in Nemotron 3 Super, distributed across potentially many GPUs in an expert-parallel setup, introduces cross-GPU communication that the per-GPU throughput measurement may not fully capture in single-GPU (TP=1) setups.

Is the LatentMoE efficiency claim empirically validated within this paper?

The paper presents LatentMoE as a superior MoE architecture for accuracy per parameter and per FLOP (Section 2.1.1, Principles 1–5). However, the paper does not report an ablation comparing a LatentMoE Nemotron 3 Super against a standard-MoE version with the same total parameter count and active parameter count. The five design principles are motivated by analysis in the companion LatentMoE paper (Elango et al., 2026), but this paper relies on that external validation. Within this paper's own experiments, the evidence for LatentMoE's benefit is indirect: Nemotron 3 Super achieves competitive accuracy with strong throughput, but this is a property of the full architecture (LatentMoE + Mamba + MTP + quantization), not a controlled comparison isolating LatentMoE.

Does the MTP shared-weight design genuinely outperform fixed-offset heads, or are the comparisons confounded by model scale and training data?

The MTP comparison in Table 2 and Figure 4 is across different models (Nemotron 3 Super vs. DeepSeek-R1 vs. Qwen3-Next), which differ in architecture, training data, model scale, and optimization. Nemotron 3 Super's 3.45 average acceptance length vs. DeepSeek-R1's 2.70 could be attributable to the shared-weight design, but it could also be due to differences in base model quality, training data, MTP loss scaling, or training duration. A within-model ablation — training Nemotron 3 Super with fixed-offset heads vs. shared-weight heads, everything else held constant — would isolate the effect of the shared-weight design but is not reported. The paper's claim that "the degradation is substantially milder than with independently trained offset heads" (Section 2.1.2) is a cross-model observation, not a controlled finding.

How reliable are the base model comparisons (Table 4)?

The base model comparison against Ling-flash-Base-2.0 and GLM-4.5-Air-Base shows large advantages for Nemotron 3 Super, but several factors complicate interpretation. First, the models are not matched on active parameter count or total training tokens: GLM-4.5-Air-Base is a smaller model (likely in the ~10B active range, though specs are not reported in this paper), and Ling-flash-Base-2.0 is also not parameter-matched. Second, the evaluation protocols differ across model families — Nemotron 3 Super evaluations use the standardized Nemo Evaluator SDK with specific prompting conventions (e.g., ARC-Challenge with all options presented together "similar to MMLU"), while the baseline numbers are drawn from "officially reported numbers whenever available" or computed "using the official evaluation settings." If baseline evaluations use different prompting formats, shot counts, or grading functions, the comparison is not truly apples-to-apples. Third, the paper reports Nemotron 3 Super's RULER results up to 1M while baseline models lack results above 128K — the claim that Nemotron 3 Super is better at long context is supported at the lengths where comparisons exist (64K, 128K) but cannot be verified at longer contexts.

Is the quantization accuracy claim (99.8% median) meaningful?

The 99.8% median relative accuracy metric aggregates across all benchmarks in Table 8, including benchmarks where quantized models slightly outperform BF16 (likely due to noise). This inflates the median. A more informative metric would be the minimum relative accuracy or the accuracy on the most quantization-sensitive benchmarks. The Scale AI Multi-Challenge benchmark drops from 55.23 (BF16) to 52.8 (NVFP4), a 4.4% relative loss. The metric also does not capture distributional shifts in output quality (e.g., changes in reasoning coherence, instruction following fidelity) that might not be captured by benchmark accuracy but would affect user experience.

What experiments are missing that would strengthen the paper?

1. Within-model ablation of LatentMoE vs. standard MoE with matched total and active parameters, to isolate the architectural contribution.
2. Within-model ablation of shared-weight vs. fixed-offset MTP heads to confirm the training-inference mismatch hypothesis.
3. Throughput measurements at multiple output lengths (1K, 8K, 32K, 64K) to characterize where the Mamba advantage is largest.
4. Scaling curves for Nemotron 3 Super — benchmarks at intermediate pretraining checkpoints (10T, 15T, 20T tokens) to characterize whether the architecture scales differently from dense Transformers.
5. Single-environment RL vs. multi-environment RL ablation with quantitative results, given that multi-environment training is emphasized as critical.
6. PivotRL vs. standard RL vs. SFT-only ablation on agentic benchmarks, since PivotRL is presented as a key efficiency method.
7. FLOPs-matched comparison between Nemotron 3 Super and a dense Transformer of equivalent inference cost, to directly validate the "co-design" claim.
8. Confidence intervals or statistical significance tests for benchmark comparisons — many of the 1–3 point differences in Table 5 and Table 8 could be within evaluation noise.

Does the paper support the claim that NVFP4 pretraining is stable at the 120B scale?

Yes, with qualifications. The training completed successfully through 25T tokens and the final model is competitive. The zero-gradient phenomenon is well-documented (Figures 6–8) and the paper provides a mechanistic explanation involving NVFP4 underflow. However, the paper does not demonstrate that the zero-gradient accumulation is harmless — it documents it, shows it converges to a similar state as BF16 training over a longer horizon (Figure 7), and argues that low-norm channels are marginal anyway. This is plausible but not proven: an ablation comparing a fully BF16-trained Nemotron 3 Super against the NVFP4-trained version (which is computationally infeasible at 120B scale, to be fair) would be needed to quantify any quality loss from NVFP4 training. The MXFP8 healing experiment (Figure 9) partially addresses this — if NVFP4 training had caused systematic quality degradation, switching to higher precision before annealing should have produced sustained improvements, but it did not. This is suggestive but not definitive, since the 1T-token healing window may be too short to recover from accumulated quantization errors.

Does the paper demonstrate that the architecture is specifically suited for agentic workloads, or just that it's a generally good architecture that was trained on agentic data?

The evaluation (Table 5) shows strong agentic performance, but the benchmarks compare the full model (architecture + training data + RL pipeline) against baselines (architecture + their training data + their RL pipeline). Nemotron 3 Super's throughput advantage is architecture-dependent, but its agentic accuracy could be primarily driven by the scale and diversity of agentic training data (279K conversational tool use samples, 1.5M general tool calls, 84K terminal use samples, SWE-RL, PivotRL) rather than the architecture itself. The paper would need to show that a similarly-sized dense model trained on the same agentic data cannot achieve comparable agentic accuracy, or that the throughput advantage translates to better agentic outcomes within a fixed time/deployment budget. The throughput comparison (Figure 1) establishes the efficiency advantage, and the benchmark comparison (Table 5) establishes competitive accuracy, but the paper does not explicitly connect these two results by showing that Nemotron 3 Super's efficiency enables better agentic task completion within a realistic time or cost constraint — this connection is left implicit.

6. Limitations and Trade-offs

Limitation 1: No Controlled Ablation Isolating LatentMoE from the Rest of the Architecture

The assumption or constraint. The paper presents LatentMoE as its primary architectural innovation—a new MoE design that achieves "better accuracy per parameter and per FLOP than regular MoEs" (Section 1). The five design principles (Section 2.1.1) argue that LatentMoE improves efficiency by reducing the hidden dimension used for routing and expert computation, then reinvesting the savings into more experts and more active experts per token. However, the paper provides no within-model ablation comparing a LatentMoE Nemotron 3 Super against an otherwise-identical model using standard full-dimension MoE with matched total parameter count and active parameter count. The architectural contribution is validated indirectly—through competitive benchmark accuracy and strong throughput—but the causal claim that LatentMoE is responsible for these outcomes, rather than the Mamba backbone, the training data mixture, the RL pipeline, or the quantization recipe, is not isolated.

The paper acknowledges that the LatentMoE validation lives in a companion technical report: "We refer the reader to the LatentMoE technical report (Elango et al., 2026) for further details" (Section 2.1.1). But this paper's own evaluation cannot distinguish whether LatentMoE is essential or incidental to Nemotron 3 Super's performance.

The consequence. A practitioner considering whether to adopt LatentMoE for their own model cannot determine, from this paper alone, whether the architecture provides benefits over a standard MoE with the same total compute budget. It is possible that much of the throughput advantage comes from the Mamba-2 backbone (which eliminates KV cache overhead) and the NVFP4 quantization (which accelerates inference on Blackwell hardware), not from the latent-space expert computation specifically. If LatentMoE adds implementation complexity (latent projections, asymmetric precision assignments, tuning the latent dimension ℓ) without a demonstrable advantage over simply using more Mamba layers or more aggressive quantization, then the architecture is over-engineered relative to its contribution.

Furthermore, the design principles (1–5) are presented as general guidance, but without a controlled experiment showing that violating them—e.g., using a standard MoE with the same 512 experts but without latent projection—produces measurably worse accuracy or throughput at the same inference cost, the principles remain hypotheses rather than empirically validated laws.

What evidence exists in the paper. The paper provides no ablation of LatentMoE versus standard MoE. All comparisons in Table 5 and Figure 1 are against external models (GPT-OSS-120B, Qwen3.5-122B) that differ in architecture, training data, training recipe, and quantization strategy simultaneously. The throughput comparison (Figure 1) attributes Nemotron 3 Super's 7.5× advantage over Qwen3.5-122B to the full architecture, not to LatentMoE specifically. The paper does not even report what fraction of the throughput gain comes from Mamba-2 (vs. attention), what fraction comes from NVFP4 quantization (vs. BF16), and what fraction comes from LatentMoE (vs. standard MoE), making it impossible to assess each component's marginal contribution.

Mitigation status. The paper partially mitigates this by referencing the companion LatentMoE paper (Elango et al., 2026), which presumably contains the controlled comparisons. However, this paper does not summarize those results or provide enough detail for a reader to evaluate the claim without consulting an external source. The mitigation is therefore structural—the paper outsources its core architectural claim to a separate report—rather than empirical within this document.

---

Limitation 2: NVFP4 Pretraining Exhibits a Zero-Gradient Phenomenon Whose Impact on Model Quality Is Incompletely Characterized

The assumption or constraint. The paper trains the entire 120B-parameter model in NVFP4 for 25 trillion tokens, documenting that "zero-valued weight gradient elements accounted for 7% of total parameters" by the end of training (Section 2.2). The paper attributes this to NVFP4 underflow: small gradient values that would be representable in BF16 or MXFP8 are quantized to exactly zero in NVFP4. The investigation (Figures 6–8) links these zeros to specific underflow locations in the backward pass—dgrad of FC2 at 500B tokens, fprop of FC1 at 750B tokens—and argues that NVFP4 "accelerates" a natural attenuation process that would occur more slowly in higher precision.

The assumption is that these zeroed-out gradients correspond to genuinely uninformative parameters (low-norm expert channels that are already "dying"), and that training stability and final model quality are not meaningfully compromised. The MXFP8 healing experiment (Figure 9) is the primary evidence for this assumption: switching all tensors to MXFP8 at 19T tokens "improved the loss trajectory" but "yielded no gains in downstream task accuracy," suggesting that the parameters zeroed out by NVFP4 were not contributing to benchmark performance.

The consequence. This characterization is plausible but incomplete. The fact that 1T tokens of MXFP8 healing does not recover accuracy does not prove that NVFP4 caused no harm—it proves that 1T tokens of healing is insufficient to recover whatever was lost over the preceding 19T tokens of NVFP4 training. Parameters that were zeroed out early in training may have permanently altered the optimization trajectory: if an expert channel's gradient goes to zero at 500B tokens, that channel stops receiving updates for the remaining 24.5T tokens, and subsequent training cannot recover whatever function that channel might have developed. The recovery experiment tests whether higher precision after the damage helps, not whether the damage occurred in the first place.

A more direct test—training a fully BF16 Nemotron 3 Super for 25T tokens—is computationally infeasible, but the paper does not attempt smaller-scale controlled experiments that would quantify the quality loss: e.g., training a Nemotron 3 Nano-scale model in both BF16 and NVFP4 and measuring the accuracy gap as a function of training tokens, to establish whether NVFP4-induced gradient sparsification imposes a permanent ceiling on model quality that would be visible even at 120B scale.

There is also a subtler concern: the paper documents the existence of the zero-gradient phenomenon but does not characterize whether it affects which parameters are zeroed. If NVFP4 underflow disproportionately affects certain expert types, certain layers, or certain domains of the training data (e.g., rare tokens, long-tail knowledge), then the model may have systematic weaknesses that are invisible in aggregate benchmark averages but manifest in deployment on specific task distributions.

What evidence exists in the paper. Section 2.2 provides detailed tensor-level analysis of the zero-gradient phenomenon (Figures 6–8), the controlled BF16-vs-NVFP4 comparison on Nemotron 3 Nano (Figure 7), and the MXFP8 healing experiment (Figure 9). The evidence for existence and mechanism is strong. The evidence for harmlessness is weak: it consists of (1) the observation that a BF16-trained Nemotron 3 Nano eventually develops a similar pattern of zero-valued gradients (Figure 7), and (2) the null result from the healing experiment. Neither of these establishes that NVFP4 training produces a model of equivalent quality to what BF16 training would have produced at the same scale.

Mitigation status. The paper is transparent about the phenomenon and its investigation, which is a strength. The authors do not claim to have proven harmlessness—they present the evidence and their interpretation. However, the paper does not acknowledge the inferential gap between "the healing experiment showed no improvement" and "NVFP4 caused no permanent quality loss," which is a limitation that practitioners should weigh when deciding whether to adopt NVFP4 pretraining for their own models.

---

Limitation 3: The Accuracy Advantage Over Qwen3.5-122B Is Inconsistently Demonstrated, Undermining the "Comparable Accuracy" Claim

The assumption or constraint. The paper's headline claim is that Nemotron 3 Super "achieves comparable accuracy on common benchmarks, while also achieving up to 2.2× and 7.5× higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively" (Abstract). The throughput advantage is well-supported (Figure 1). The accuracy claim relies on Table 5, which compares Nemotron 3 Super against both baselines across ~25 benchmarks.

"Comparable accuracy" against GPT-OSS-120B is defensible: Nemotron 3 Super leads on most benchmarks (agentic tasks by 12–18 points, long-context by 40+ points, math with tools) and trails notably on only two (Arena-Hard-V2 by 16.4 points, LiveCodeBench v5 by 6.8 points). The wins outweigh the losses.

Against Qwen3.5-122B, the claim is substantially weaker. Nemotron 3 Super trails Qwen3.5-122B on 11 of the 16 directly comparable benchmarks in Table 5 (where both models have reported numbers). The gaps are not uniformly small: GPQA (79.23 vs. 86.60, −7.4), HLE no tools (18.26 vs. 25.30, −7.0), SWE-Bench across all three harnesses (−6 to −8 points), Terminal Bench Core 2.0 (31.00 vs. 37.50, −6.5), TauBench V2 Average (61.15 vs. 74.53, −13.4, driven by a 30.6-point gap on Telecom), MMLU-ProX (79.36 vs. 85.06, −5.7). Nemotron 3 Super leads convincingly only on HMMT with tools (94.73 vs. 89.55, +5.2) and ties on AIME25, RULER, and IFBench.

The consequence. A practitioner choosing between Nemotron 3 Super and Qwen3.5-122B faces a tradeoff that the paper only partially surfaces: the 7.5× throughput advantage is real, but it comes at the cost of 5–13 point deficits on most reasoning and agentic benchmarks. For a deployment where throughput is the bottleneck (e.g., high-volume batch inference, cost-sensitive applications), this tradeoff may be favorable. For a deployment where accuracy on software engineering or tool-use tasks is the primary requirement, Qwen3.5-122B would be the stronger choice despite its lower throughput. The paper's framing of "comparable accuracy" obscures this tradeoff by aggregating across benchmarks where the model does and does not compete, leaving the reader to manually assess whether the accuracy gaps matter for their use case.

The Telecom domain of TauBench V2 exposes a more severe concern: Nemotron 3 Super scores 64.36 vs. Qwen3.5-122B's 95.00, a 30.6-point gap. This is not a minor variance—it suggests that Nemotron 3 Super has a specific weakness in a particular agentic subdomain that is several standard deviations below the baseline. The paper does not investigate or explain this outlier, leaving open the possibility that other unmeasured subdomains may have similarly large gaps.

What evidence exists in the paper. Table 5 provides the raw numbers. The paper does not report aggregate metrics (e.g., win rate, average gap, statistical significance) that would characterize the overall accuracy relationship. The text in Section 3.3 describes Nemotron 3 Super as "competitive with GPT-OSS-120B, while lagging behind Qwen-3.5-122B slightly" for reasoning and "competitive to Qwen 3.5 122B on some harnesses" for agentic tasks—language that is more measured than the Abstract's "comparable accuracy" but still understates the systematic nature of the gaps.

Mitigation status. The paper does not attempt to mitigate this limitation—it does not contextualize the Qwen3.5-122B accuracy gaps, propose explanations, or suggest that future work could close them. The strength of the paper is its transparency (all numbers are reported in Table 5 for independent analysis), but the framing in the Abstract and Figure 1 overstates the accuracy parity.

---

Limitation 4: The Difficulty of Estimating When the 7.5× Throughput Advantage Materializes in Practice

The assumption or constraint. The throughput advantage of 7.5× over Qwen3.5-122B (and 2.2× over GPT-OSS-120B) is measured at a single configuration: 8K input tokens / 64K output tokens on B200 GPUs, using the best framework (vLLM or TRT-LLM) per model (Figure 1). The paper does not report throughput at other input/output length combinations, batch sizes, or hardware configurations.

Nemotron 3 Super's throughput advantage derives substantially from its Mamba-2 backbone, which eliminates the quadratically-growing KV cache that dominates attention-based models at long sequence lengths. At 64K output tokens, the KV cache for an attention-heavy model like Qwen3.5-122B is enormous—each of its attention layers must store 64K keys and values per head, and this memory must be read and written for every generated token. Nemotron 3 Super's Mamba-2 layers, by contrast, maintain a constant-size recurrent state (128 × 128 = 16,384 elements per token per layer), so their per-token cost does not grow with sequence length. The throughput advantage is therefore a function of output length: longer outputs → larger KV cache → larger gap.

The consequence. The throughput advantage of 7.5× is not a universal property of the architecture—it is specific to the long-output regime. For short-output tasks (e.g., single-turn chat with 100–500 output tokens, factual QA, simple instruction following), the KV cache overhead is negligible, and the throughput gap would shrink substantially. The paper does not characterize how much it shrinks, leaving practitioners unable to assess whether the architecture provides meaningful throughput benefits for their specific workload distribution.

Conversely, for workloads with extremely long contexts (e.g., 1M-token inputs for document analysis), the Mamba advantage should grow even larger because attention's KV cache at 1M tokens becomes proportionally more expensive. But the paper does not report throughput at 128K, 256K, 512K, or 1M input lengths—the RULER benchmarks (Table 5) measure accuracy at these lengths, not throughput. So the upper bound of the throughput advantage is also uncharacterized.

A related concern: the throughput measurement uses TP=1 (tensor parallelism degree 1, i.e., a single GPU per model replica) on B200 GPUs. At larger tensor parallelism degrees (e.g., TP=8, where the model is split across 8 GPUs), the communication patterns change—all-to-all routing for MoE experts becomes a larger fraction of total time, and LatentMoE's reduction in communication volume (by a factor of d/ℓ = 4) would provide a larger relative advantage. The paper does not explore this scaling dimension.

What evidence exists in the paper. Figure 1 reports throughput at exactly one configuration (8K/64K, B200, best framework per model). Section 2.1.1 discusses the two serving regimes (latency-oriented and throughput-oriented) and how LatentMoE addresses both, but does not provide measured throughput data in either regime beyond the single Figure 1 data point. None of the other figures or tables provide throughput measurements at different configurations.

Mitigation status. The paper does not address this limitation. It does not report throughput scaling curves, does not discuss how the advantage varies with sequence length or batch size, and does not provide guidance on what workload characteristics favor Nemotron 3 Super. The single-configuration measurement is sufficient for establishing the existence of a throughput advantage but insufficient for practitioners to predict whether that advantage applies to their deployment.

---

Limitation 5: The Agentic RL Pipeline's Contributions Are Not Ablated, So the Source of Agentic Performance Is Unclear

The assumption or constraint. The post-training pipeline (Section 3) introduces several components that are presented as important for agentic performance: multi-environment RLVR across 21 environments (Section 3.2.1), a dedicated SWE-RL stage for end-to-end software engineering tasks (Section 3.2.2), PivotRL for efficient agentic RL using offline SFT traces (Section 3.2.4), and a large volume of agentic SFT data (279K conversational tool use samples, 1.5M general tool calls, 84K terminal use samples, Section 3.1.1). The paper states that "single-environment training leads to severe regressions on other benchmarks" (Section 3.2.1) and that PivotRL "greatly improves the efficiency of our agentic RL, without facing the OOD degradation issues of SFT" (Section 3.2.4).

However, the paper reports no quantitative ablations for any of these claims. There is no comparison of multi-environment vs. single-environment RLVR accuracy. There is no comparison of PivotRL vs. standard end-to-end RL vs. SFT-only on agentic benchmarks. There is no comparison of the full RL pipeline against a simpler pipeline (e.g., SFT only, or RLVR only, or RLVR + RLHF without SWE-RL). The MTP healing stage is said to "significantly improve MTP accuracy" (Section 3.2.4) but no numbers are reported.

The consequence. A practitioner cannot determine which components of the post-training pipeline are necessary and which are incidental. If SWE-RL provides minimal marginal benefit over SFT alone, the infrastructure complexity of the SWE-RL environment (Apptainer containers, memory watchdog daemons, command blocklists, multi-harness diversity) is unjustified. If PivotRL is essential for agentic performance, teams without access to high-quality SFT expert trajectories cannot replicate the approach. If multi-environment training is critical to prevent regression, smaller teams with limited environment diversity may see worse results.

More broadly, the paper cannot attribute Nemotron 3 Super's agentic performance to any specific post-training innovation. It is plausible that the agentic benchmarks in Table 5 are primarily driven by the scale and coverage of agentic SFT data—the 279K conversational tool use samples across 838 domains (an ~18× scale-up over Nemotron 3 Nano's 15K samples in 5 domains) and the 1.5M general tool-calling trajectories—rather than by the RL pipeline. If so, the expensive RL infrastructure (thousands of GPUs, async GRPO, in-flight weight updates, PivotRL) adds complexity without commensurate benefit.

What evidence exists in the paper. None. The paper describes each component and asserts its importance, but does not provide ablations, scaling curves, or controlled comparisons. The only quantitative ablation in the post-training section is the MTP healing claim, which itself lacks numbers.

Mitigation status. The paper provides detailed descriptions of the post-training methodology (data generation pipelines, RL infrastructure, PivotRL algorithm) that would enable replication, but replication does not establish necessity. The lack of ablations means the paper's claims about which components matter are unvalidated hypotheses. This is a significant gap for a paper that positions agentic capability as a primary contribution.

---

Limitation 6: The Mamba State Cache Quantization Verbosity Problem Reveals a Fragility in the Inference Stack

The assumption or constraint. The paper documents that quantizing the Mamba SSM cache from FP32 to FP16—a natural optimization for memory-bandwidth-bound decoding—causes up to a 40% increase in output verbosity (Table 9). The model generates substantially longer responses when its recurrent state is degraded by quantization noise, even though benchmark accuracy remains largely unchanged. The paper identifies stochastic rounding as the solution, with Philox<5> pseudorandom number generation recovering verbosity to baseline levels.

This finding implies that Mamba-based architectures are sensitive to recurrent state precision in ways that standard accuracy benchmarks do not capture. The verbosity increase is invisible in pass@1 or exact-match metrics (Table 9 shows that LiveCodeBench and SciCode accuracy are essentially unchanged across all SSM cache recipes, including the naive FP16 casting that causes 40% verbosity inflation), meaning that standard evaluation protocols would miss this degradation entirely.

The consequence. The verbosity problem has two practical implications. First, it means that deploying a Mamba-based model with aggressive quantization requires evaluation protocols that go beyond accuracy—output length, latency per request, and tokens-per-task must be monitored to detect unintended behavioral changes. A deployment that naively quantizes the SSM cache to FP16 would see 40% higher per-request cost (more tokens generated = more FLOPs, more latency, higher serving cost) while accuracy metrics show no regression, masking the degradation.

Second, it suggests that Mamba-based models may have other precision-sensitive failure modes that are not yet characterized. The paper identified verbosity through manual inspection (noticing that outputs were longer), not through automated monitoring. There may be other behavioral changes (e.g., reduced reasoning coherence, increased hallucination, changed refusal patterns) that are caused by recurrent state degradation but are not captured by existing benchmarks. The verbosity problem is a "known unknown"—it demonstrates that precision matters in ways the standard evaluation suite does not test, but the paper does not systematically explore what other behaviors might be affected.

What evidence exists in the paper. Table 9 provides the quantitative evidence: completion tokens increase by 36.95% on LiveCodeBench under BF16 weights/activations with FP16 SSM cache, and by 40.27% under W8A8 quantization. FP16 with stochastic rounding (Philox<5>) recovers verbosity to within ±2% of baseline. The paper also provides Equation (3) unrolling the recurrent error accumulation, which mathematically demonstrates why quantization error compounds over time in the Mamba state update.

Mitigation status. The paper addresses the specific verbosity problem with stochastic rounding (Philox<5>), selected based on Blackwell hardware support (dedicated PTX instruction for stochastic rounding, cuRAND for Philox PRNG generation). This is an effective mitigation for the observed symptom, but it does not address the underlying fragility: the fact that a seemingly innocuous optimization (FP16 state caching) caused a major behavioral change that standard benchmarks would not detect. The paper does not propose systematic methods for detecting or preventing similar precision-induced behavioral changes, nor does it provide guidance on what precision assignments in other components of the Mamba recurrence (the A matrix, B matrix, input projections) might cause analogous problems. The mitigation is point-solution rather than principled—it fixes the FP16 SSM cache specifically, but leaves open whether other quantization choices in the Mamba pathway could introduce undetected behavioral degradation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a single breakthrough technique; it demonstrates that a set of previously-separate efficiency innovations can be composed into a single model at the 120B-parameter scale without mutual interference, and that the resulting system is competitive with state-of-the-art dense and sparse models while providing substantial throughput advantages. The contribution is integration and scaling rather than invention, but the integration itself constitutes a methodological shift: it establishes that hardware-aware expert routing (LatentMoE), state-space sequence modeling (Mamba-2), native speculative decoding (shared-weight MTP), 4-bit pretraining (NVFP4), and mixed-precision deployment quantization can coexist in one architecture without any single component undermining the others. This is not obvious a priori — each technique introduces its own failure modes (gradient underflow for NVFP4, recurrent state degradation for Mamba, training-inference mismatch for MTP, all-to-all communication bottlenecks for MoE), and the paper documents and mitigates several of these interactions directly (the zero-gradient phenomenon in Section 2.2, the Mamba cache verbosity problem in Section 4.3).

The reframing that matters most is the shift from FLOPs-centric efficiency to deployment-bottleneck-centric efficiency encoded in the LatentMoE design principles (Section 2.1.1). Prior MoE work optimized accuracy per FLOP — keep total compute constant, scale total parameters. This paper argues that FLOPs are not the bottleneck in real serving: memory bandwidth for expert weight loads dominates low-latency serving, and all-to-all communication dominates throughput-oriented serving. By identifying the hidden dimension $d$ as the only variable that addresses both bottlenecks simultaneously, and showing that $d$ can be reduced (via latent projection) while preserving quality through increased expert count and top-K, the paper provides a diagnostic framework for MoE architecture design that subsequent models can adopt independent of the specific LatentMoE implementation. The five design principles are the portable contribution — any future MoE architecture can evaluate whether it is optimizing the right bottleneck by asking whether it reduces $d$, $m$, or $K$, and what the effective feature rank $r_{\text{eff}}$ of its target tasks permits.

The paper also resolves a tension that has been implicit in the speculative decoding literature. Standard MTP with fixed-offset heads (DeepSeek-V3, Gloeckle et al.) works well for short drafts but degrades at longer draft lengths due to training-inference distribution mismatch. The shared-weight multi-offset training approach demonstrated here — training one head on multiple offsets so it learns to predict from hidden states at varying quality levels — shows that the mismatch is not fundamental to MTP but is a consequence of a specific training design choice. This reframes the problem from "MTP degrades with draft length" to "train MTP heads to be robust to the quality of their own previous outputs," which is an actionable principle for future work: any component that will be used autoregressively at inference should be trained on diverse auto-regressive contexts, not only oracle contexts.

Finally, the NVFP4 pretraining analysis (Section 2.2) introduces a diagnostic vocabulary for low-precision training at scale: the zero-gradient phenomenon, channel magnitude patterns aligning across FC1 and FC2 in expert layers, underflow cascades (dgrad of FC2 → wgrad of FC1 zeros at 500B tokens, fprop of FC1 → wgrad of FC2 zeros at 750B tokens). This transforms NVFP4 pretraining from "a recipe that was shown to work" into "a phenomenon whose mechanisms are understood," which lowers the barrier for other teams to adopt 4-bit pretraining by providing specific signals to monitor (expert channel norms, zero-valued gradient counts, underflow rates per tensor) and an expectation for how training dynamics differ from BF16 (accelerated sparsification of marginal parameters, no recovery benefit from late-stage precision promotion).

What becomes more attractive as a research direction: Co-designing architectures, training recipes, and quantization strategies as a unified optimization problem rather than treating each as a sequential pipeline. The paper's success at composing LatentMoE + Mamba + MTP + NVFP4 + AutoQuantize suggests that the gains from joint optimization exceed what would be expected from applying each technique independently, because the techniques address complementary bottlenecks and their failure modes can be diagnosed and mitigated when they are developed together.

What becomes less attractive: Pursuing incremental accuracy improvements on standard benchmarks through pure scale (more parameters, more data) without corresponding attention to inference efficiency. The paper demonstrates that a 12B-active model with architectural efficiency can match or approach models with comparable active parameters and substantially higher inference cost (GPT-OSS-120B, Qwen3.5-122B). As the field moves toward agentic workloads where inference cost dominates total cost of ownership, architectures that trade off some accuracy for large throughput gains become increasingly compelling. The 7.5× throughput gap to Qwen3.5-122B (at 8K/64K on B200 GPUs) is large enough that even a model 5–10 points lower on several benchmarks could be the economically rational choice for many deployments, particularly if accuracy gaps can be narrowed through better post-training (the agentic SFT and RL pipeline described here provides one template).

---

Follow-Up Research This Work Enables

Ablation of LatentMoE vs. standard MoE at matched inference cost. The paper's core architectural claim — that projecting expert computation into a latent space improves accuracy per parameter and per FLOP — is validated only in the companion LatentMoE report (Elango et al., 2026), not in this paper. A direct follow-up would train two versions of Nemotron 3 Super at a smaller scale (e.g., the Nemotron 3 Nano 30B-A3B architecture) with identical data, identical training tokens, and matched inference cost: one using LatentMoE with latent dimension ℓ = d/4, expanded expert count, and expanded top-K, and one using standard full-dimension MoE with the same total and active parameter counts. The comparison would measure benchmark accuracy, per-token inference latency at multiple batch sizes, and all-to-all communication volume during distributed serving. This would isolate whether the LatentMoE design principles (1–5) translate to measurable improvements over standard MoE, or whether the gains come from the Mamba backbone, data mixture, and training recipe.

Latent dimension ℓ as a tunable hyperparameter with per-layer variation. The paper fixes ℓ = 1024 uniformly across all LatentMoE layers. Principle 4 (effective feature rank $r_{\text{eff}}$) suggests that the minimum safe $d$ varies by layer depth and task type — early layers may require higher dimensionality for input encoding, while later layers may operate on more compressed representations. A follow-up study could measure $r_{\text{eff}}$ per layer using PCA or intrinsic dimension estimation on the BF16 hidden states of a trained Nemotron 3 Super, then train a variant with per-layer latent dimensions (e.g., ℓ = 512 in late layers where $r_{\text{eff}}$ is low, ℓ = 2048 in early layers where it is high). The prediction is that this would improve accuracy per parameter further without increasing inference cost, since savings from aggressive compression in low-$r_{\text{eff}}$ layers could be reinvested into more experts or higher ℓ in bottleneck layers.

Characterizing the throughput advantage as a function of sequence length. The paper reports throughput at exactly one configuration (8K input / 64K output, B200 GPUs). The throughput advantage derives substantially from Mamba-2 eliminating the KV cache, which grows quadratically with attention layers and linearly with sequence length. A comprehensive throughput scaling study would measure tokens-per-second for Nemotron 3 Super, GPT-OSS-120B, and Qwen3.5-122B across input lengths {1K, 8K, 32K, 128K, 256K} and output lengths {128, 1K, 8K, 32K, 64K}, at batch sizes {1, 8, 32}, on both B200 and H100 hardware. This would produce a throughput surface that practitioners could use to estimate the advantage for their specific workload distribution, and would identify the crossover points (output lengths and batch sizes) where the Mamba advantage becomes negligible. This study would also test whether the LatentMoE all-to-all communication reduction matters more at high batch sizes (throughput regime) or low batch sizes (latency regime).

Within-model shared-weight vs. fixed-offset MTP ablation with acceptance rate scaling curves. The paper claims that shared-weight MTP heads reduce the training-inference mismatch compared to fixed-offset heads, but the evidence is cross-model (Nemotron 3 Super vs. DeepSeek-R1 on SPEED-Bench, Table 2). A controlled experiment would train Nemotron 3 Super variants from an intermediate pretraining checkpoint (e.g., 20T tokens) with three MTP configurations: (a) two shared-weight heads as in the paper, (b) two independent fixed-offset heads, and (c) four independent fixed-offset heads. All variants would be trained on the same data for the same number of tokens, then evaluated on SPEED-Bench with draft lengths from 1 to 16, measuring acceptance rate at each draft index. The prediction is that shared-weight heads maintain higher acceptance at indices beyond their training horizon (e.g., draft index 5–16) because they generalize better to autoregressive contexts, while fixed-offset heads collapse after their training offset. This would establish whether the shared-weight design is causally responsible for the acceptance rate advantage, or whether it is an artifact of model scale or training data differences.

Verification that NVFP4 pretraining causes no permanent quality loss via small-scale controlled training. The paper's MXFP8 healing experiment (Figure 9) shows no recovery benefit from switching to higher precision at 19T tokens, which the authors interpret as evidence that NVFP4-zeroed parameters were marginal. This inference is indirect. A more direct test would train a Nemotron 3 Nano-scale model (30B-A3B) in both BF16 and NVFP4 from scratch for an equivalent compute budget, then compare not just final benchmark accuracy but also: (a) per-expert channel norm distributions across training, (b) expert specialization patterns (do experts in the NVFP4 model specialize in different token distributions?), (c) performance on long-tail knowledge probes (entities, facts, tasks that appear rarely in training), and (d) robustness to distribution shift (out-of-distribution benchmarks). If NVFP4 systematically degrades performance on rare patterns — because low-norm expert channels that handle rare tokens get zeroed out early — this would be invisible in aggregate benchmark averages (dominated by common patterns) but would matter for deployment. The experiment would establish whether NVFP4 pretraining imposes a permanent quality ceiling that narrows as model scale increases (suggesting it is safe at 120B) or is scale-independent (suggesting caution).

Systematic probing of Mamba recurrent state sensitivity to quantization across all recurrence components. The paper diagnoses a specific verbosity problem when the SSM cache is quantized to FP16 (Table 9), but only tests the state cache itself. The Mamba-2 recurrence involves multiple matrices — the state transition matrix A, the input projection B, the output projection C, and the state update — all of which could be quantized in deployment. A systematic study would independently quantize each component of the recurrence (A to FP16, B to FP8, the state to FP16, the input projection to NVFP4, etc.) and measure not just benchmark accuracy but also output length, repetition rate, reasoning coherence (via LLM-as-judge on generated reasoning traces), and token distribution shift (KL divergence from BF16 outputs). This would produce a precision sensitivity map for Mamba-based architectures, identifying which components are safe to quantize aggressively and which require higher precision or stochastic rounding. The verbosity problem with the SSM cache suggests that other components may have similarly hidden failure modes that benchmark accuracy alone would miss.

---

Practical Applications and Downstream Use Cases

Cost-efficient agentic coding assistants. A deployment of Nemotron 3 Super NVFP4 on Blackwell GPUs for a coding assistant that handles multi-turn software engineering tasks (SWE-Bench-style issue resolution, code exploration, debugging) would benefit directly from the 2.2× throughput advantage over GPT-OSS-120B and the 7.5× advantage over Qwen3.5-122B at long output lengths. In a typical agentic coding loop, the model generates many intermediate tool calls, reads large files, and produces substantial code patches — a single task might involve 50–100K output tokens across 20–30 turns. At 64K output tokens per request, the throughput difference translates to serving 2.2–7.5× more concurrent users per GPU, or equivalently, reducing per-task latency by a corresponding factor. For a service processing 10,000 coding tasks per day, this could mean 4–14× fewer GPUs required, or the ability to serve the same load with 2–4 B200 GPUs instead of 14. The SWE-Bench scores (60.47 on OpenHands) indicate that the model's coding capability is sufficient to justify the deployment for real-world use, and the tool-augmented reasoning benchmarks (HMMT with tools at 94.73, HLE with tools at 22.82 vs. GPT-OSS's 19.0) suggest that the model benefits from tool integration in the agentic loop.

Long-context document analysis and retrieval at scale. Nemotron 3 Super's RULER performance at 1M tokens (91.64 post-trained, 71.00 base) combined with its Mamba-based throughput advantage makes it well-suited for applications that require processing extremely long documents — legal contract review, financial report analysis, academic literature synthesis, or multi-document question answering. The key advantage is that the model maintains high retrieval accuracy at context lengths where attention-based models either cannot run (GPT-OSS-120B scores 22.30 on RULER at 1M) or have not been evaluated (Qwen3.5-122B scores 91.33 at 1M but with unknown throughput at that length). A deployment analyzing 1M-token financial filings (10-K reports, earnings call transcripts, industry analysis) could load entire documents into context and ask cross-document questions without chunking, retrieving, or summarizing — the model's AA-LCR score of 58.31 (vs. 51.00 for GPT-OSS, 66.90 for Qwen3.5) suggests it is competitive on this task class. The throughput advantage would be particularly pronounced here because the attention-based alternatives would need to manage enormous KV caches at 1M tokens, while Nemotron 3 Super's Mamba layers maintain constant state size.

On-device or edge-deployed agentic tools with quantized checkpoints. The NVFP4 checkpoint's 99.8% median accuracy relative to BF16, combined with the model's 12B active parameter count, makes it feasible to deploy Nemotron 3 Super on hardware configurations that would struggle with dense 120B-parameter models. While 12B active parameters is not "on-device" by mobile standards, it is within range for workstation-class GPUs or multi-GPU edge servers. A terminal-use assistant (Terminal Bench scores: 25.78 hard subset, 31.00 Core 2.0) or a general-purpose tool-calling agent (TauBench V2 Average 61.15) could run on a small cluster of B200 or even H100 GPUs, serving multiple concurrent users with the NVFP4 checkpoint providing 2.2× higher throughput than the BF16 baseline. The AutoQuantize recipe (completing in under 2 hours on a single 8-GPU node) means that the quantization process is lightweight enough to be rerun if the model is fine-tuned on domain-specific data, enabling custom deployments without requiring quantization expertise. The stochastic rounding fix for the Mamba SSM cache (Philox<5>, supported by Blackwell PTX instructions) ensures that the quantized model does not silently inflate output length, which would negate the throughput gains in deployment.

---

When to Prefer This Method

The paper does not explicitly frame Nemotron 3 Super as one option in a tradeoff against named alternatives with clear decision criteria. It presents the model as demonstrating that a specific set of architectural and training choices can compose successfully, but does not provide direct ablations against standard-MoE, dense, or attention-only architectures with controlled budgets. A forced "prefer A when... prefer B when..." matrix would impose structure the paper does not provide.

The closest the paper comes to articulating a tradeoff is the throughput-vs-accuracy comparison in Figure 1 and Table 5, which implicitly suggests: if your deployment bottleneck is inference throughput (particularly at long output lengths) and you can tolerate 5–13 point accuracy gaps relative to Qwen3.5-122B on specific agentic and reasoning benchmarks, Nemotron 3 Super offers substantial cost savings. Conversely, if accuracy on software engineering (SWE-Bench, Terminal Bench Core) or graduate-level reasoning (GPQA, HLE) is the primary requirement and throughput is secondary, Qwen3.5-122B's higher scores would be preferable. The paper does not develop this into an explicit decision framework, so none is provided here.