Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

🎯 Pitch

A 30B-parameter MoE model with native audio support delivers 3× the throughput of Qwen3-Omni while topping leaderboards in document and long video understanding. By fusing every two video frames into a single token, the model cuts temporal tokens in half with less than a 0.5-point accuracy loss—proving aggressive compression need not hurt omni-modal reasoning.

1. Executive Summary

This paper introduces Nemotron 3 Nano Omni, an omni-modal model that natively supports audio alongside text, images, and video, built on the Nemotron 3 Nano 30B-A3B Mixture-of-Experts hybrid backbone with the C-RADIOv4-H vision encoder and Parakeet-TDT-0.6B-v2 audio encoder. The model advances over its predecessor (Nemotron Nano V2 VL) through a staged training recipe that progressively introduces modalities and scales context length to 256K tokens, dynamic image resolution that preserves native aspect ratios, Conv3D-based temporal video compression (fusing every two frames into one tubelet to halve temporal tokens), and a compute-optimal multi-stage curriculum spanning seven SFT stages followed by reinforcement learning with MPO and GSPO. Nemotron 3 Nano Omni achieves leading results on document understanding benchmarks (OCRBench-V2 EN 65.8, MMLongBench-Doc 46.1 without reasoning), delivers 3× higher single-stream output token throughput than Qwen3-Omni and 9× higher output token throughput per GPU at fixed interactivity on NVIDIA B200, while retaining the text reasoning performance of its LLM backbone (MMLU-Pro 77.3, GPQA 72.2), establishing that efficient omni-modal modeling can match or exceed larger models on vision-language-audio benchmarks only when token-reduction techniques—Conv3D coupled with Efficient Video Sampling at q=0.7—preserve accuracy to within half a point of the uncompressed baseline.

2. Context and Motivation

The Core Problem: Scaling LLMs to Multiple Modalities Without Sacrificing Efficiency

The fundamental challenge this paper addresses is deceptively simple: how do you build a single model that simultaneously understands text, images, audio, and video, while remaining efficient enough to deploy at scale? Prior to this work, the landscape of multimodal large language models (MLLMs) was split between two unsatisfying extremes: models that handled multiple modalities reasonably well but were prohibitively expensive to run, and models that were efficient but limited to vision-language understanding, completely omitting audio—a modality that matters enormously for real-world applications like meeting transcripts, video comprehension with speech, voice assistants, and accessibility tools.

The gap is concrete. Before Nemotron 3 Nano Omni, the Nemotron multimodal family (represented by Nemotron Nano V2 VL) was a capable vision-language model but had no native audio support whatsoever. If you wanted to understand a video with sound—say, a lecture recording or a product demo—you would need a separate automatic speech recognition (ASR) system feeding text into the vision-language model, an architecture that breaks the tight temporal coupling between what is seen and what is heard. The Qwen3-Omni model (Xu et al., 2025) did offer omni-modal capabilities, but as the paper demonstrates through direct measurements (Section 4.8), it delivered substantially lower throughput—Nemotron 3 Nano Omni achieves 3× higher single-stream output token throughput and 9× higher output token throughput per GPU at a fixed interactivity target of 50 tokens per second per user. For production deployments where cost per query determines whether a model is economically viable, these are not marginal differences; they are ecosystem-defining.

Practical and Theoretical Stakes

This problem matters for several reasons that the paper implies rather than states explicitly, but which are worth unpacking:

The modality gap in frontier models is closing unevenly. Proprietary systems like GPT-4o, Gemini, and Claude have demonstrated that unified multimodal understanding is possible, but they are closed-source, expensive, and opaque. The open-source community has been racing to catch up, but the models that achieve strong omni-modal performance (such as Qwen3-Omni-30B-A3B) are often dense transformer architectures that hit throughput walls on long multimodal sequences. A 30-minute video with audio might produce hundreds of thousands of input tokens—processing this through a dense transformer is a quadratic scaling problem in both compute and memory. This paper's core bet is that a hybrid Mixture-of-Experts architecture (the Nemotron 3 Nano 30B-A3B backbone) can break this scaling bottleneck: by activating only a fraction of parameters per token, it dramatically reduces the per-token inference cost while maintaining enough total capacity for multimodal understanding.

Long-context multimodal reasoning is the next frontier. The paper extends the maximum context length from 128K to 256K tokens. This is not an arbitrary number—it is the difference between being able to process a 15-minute video and a 30-minute video, or between analyzing a 20-page document and a 50-page document with embedded charts and tables. Real-world document understanding (the paper cites MMLongBench-Doc, OCRBench-V2) requires reasoning across pages of visually rich content: contracts, financial reports, academic papers, technical manuals. These applications are not niche; they are core enterprise use cases. Similarly, agentic computer use (ScreenSpot, OSWorld) requires the model to interpret GUI screenshots that are fundamentally visual-textual hybrids and reason about what to click next. A model that can do this efficiently enables desktop automation, accessibility tools for visually impaired users, and automated testing pipelines—applications with massive economic value.

Audio is not a luxury, it is the missing piece for true omni-modal reasoning. Consider the DailyOmni benchmark (Zhou et al., 2026), which tests temporal alignment, event understanding, and causal reasoning across synchronized audio-visual inputs. A model that only sees video frames but cannot hear the accompanying narration, music, or environmental sounds is fundamentally limited in what it can understand about that video—it is like watching a movie on mute. The WorldSense benchmark (Hong et al., 2026) pushes this further with 1,662 long-context videos and 3,172 multiple-choice questions testing sound grounding and temporal reasoning. Without native audio support, performance on these benchmarks is not just lower; it is structurally impossible for certain question types that explicitly require audio information. The paper's decision to add a dedicated audio encoder and interleave audio tokens temporally with visual tokens (Section 2) represents a genuine architectural capability unlock rather than an incremental improvement.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in prior work, though it does so indirectly through its design choices and comparison baselines:

Tiling-based image processing distorts aspect ratios. Nemotron Nano V2 VL used a tiling strategy for images—essentially chopping images into fixed-size tiles regardless of their native proportions. This is computationally convenient (all tiles are the same size, so batch processing is straightforward) but harms performance on tasks where aspect ratio carries semantic information. A bar chart that is 3× wider than it is tall communicates something fundamentally different from one that is 3× taller than wide; squashing both into square tiles loses this signal. The paper replaces this with dynamic resolution processing (Section 2) that preserves native aspect ratios, allowing images to be decomposed into a variable number of 16×16 patches with total visual tokens constrained between 1,024 and 13,312—roughly corresponding to images from 512×512 to 1840×1840 for square inputs. This is more complex to implement but directly improves document and chart understanding, where spatial layout is the primary information carrier.

Video token budgets were unsustainable for long sequences. Processing video by naively embedding every frame as an independent image produces a token count that scales linearly with video duration. For a 30-minute video at 1 frame per second, this is 1,800 frames, each potentially producing thousands of visual tokens—easily exceeding 100K tokens before the language model even sees any text. The paper introduces two complementary mechanisms to combat this: Conv3D, which fuses every two consecutive frames into a single "tubelet" before the vision transformer, halving temporal tokens at the architectural level during both training and inference; and Efficient Video Sampling (EVS), a runtime-only pruning mechanism that drops tokens after the vision encoder based on cosine dissimilarity between consecutive tubelets, keeping only the most informative spatial positions up to a budget set by a pruning rate $q$ . The paper shows that with Conv3D enabled, a 512-frame video drops from ~141K input tokens to ~75K (a 47% reduction), and stacking EVS at $q = 0.5$ takes it to ~42K (70% reduction vs. baseline). These are not marginal optimizations—they are the difference between a model that can process only short video clips and one that can handle feature-length content.

Staged training for omni-modal models was underexplored. Training a model with three separate encoders (vision, audio, text) and a shared decoder (the LLM backbone) using a naive "mix everything together" approach risks catastrophic forgetting and modality interference. If you start training all parameters on all modalities simultaneously, the gradients from abundant text data can wash out the signal from scarce audio-visual training examples, or the vision encoder can drift in ways that break its carefully pretrained representations. The paper's seven-stage SFT pipeline (Sections 3.1.1–3.1.7) is a deliberate curriculum: first warm up the vision projector while keeping the LLM and vision encoder frozen (Stage 0), then jointly fine-tune vision and language (Stage 1), then warm up the audio projector (Stage 2), then jointly train the audio encoder and projector while keeping the LLM frozen (Stage 3), and only then—after both modality-specific alignments are stable—unfreeze everything for joint omni-modal training (Stages 4–6). This is a significant engineering contribution that the paper understates: getting all three modalities to coexist without mutually degrading each other is genuinely hard, and the staged approach is a generalizable recipe that other teams can follow.

Reinforcement learning for multimodal reasoning was nascent. Applying RL to improve multimodal LLMs is substantially harder than text-only RL because (1) reward signals are harder to construct (how do you verify whether a model correctly described the temporal relationship between a speaker's gestures and their words?), (2) the action space includes visual grounding (predicting click coordinates, not just tokens), and (3) balancing improvement across modalities risks overfitting to one at the expense of others. The paper's Omni-RL stage (Section 3.2.4) curates a 120K-prompt corpus spanning 113 sub-datasets across image, video, audio, and text reasoning, with task-specific verifiers that range from string matching for free-form answers to coordinate-distance-based rewards for GUI grounding (where the reward smoothly decays with distance from the correct click target). The pass-rate filtering—retaining only prompts where the initial policy achieves 0.1–0.9 pass rate—ensures that RL focuses on prompts that are neither trivially solvable (wasting compute) nor completely intractable (providing no learning signal). This is a careful practical solution to a problem the field is still learning to formalize.

How This Paper Positions Itself

The paper positions itself at the intersection of three trends in AI research, each of which it claims to advance simultaneously:

First, it is an efficiency play in an efficiency-conscious moment. The abstract and introduction foreground throughput numbers—3× faster than Qwen3-Omni, 9× higher throughput per GPU, 2× faster than Nemotron Nano V2 VL—alongside accuracy benchmarks. This dual emphasis on capability and cost is a direct response to the growing recognition that scaling laws alone (bigger models, more data) are economically unsustainable for broad deployment. The MoE hybrid backbone (activating 3B of 30B parameters per token) is the primary driver of this efficiency, but the token-reduction techniques (Conv3D, EVS, pixel shuffle with 4× downsampling) are equally important because they reduce the token count before the LLM ever processes them, which is a multiplicative win: fewer tokens × fewer active parameters per token = dramatically lower total FLOPs.

Second, it is an open-source contribution in a field where proprietary models dominate omni-modal capabilities. The paper releases model checkpoints in BF16, FP8, and NVFP4 formats alongside training data (~6.9M samples), data generation pipelines, and training code (Megatron-Bridge). The FP4 checkpoint is particularly notable: at 4.98 effective bits per weight (20.9 GB total from a 61.5 GB BF16 reference), the paper demonstrates that aggressive quantization to FP4 preserves median accuracy to within less than 1% of BF16 across 25 benchmarks while enabling 7.5× higher output token throughput at iso-interactivity on B200 GPUs. This dramatically lowers the barrier to deployment—a 20.9 GB model can run on a single consumer GPU, while a 61.5 GB model requires expensive datacenter hardware.

Third, it demonstrates that native audio support can be added to an existing vision-language architecture without degrading vision or text performance. This is a more subtle positioning claim but one that the evaluation suite (Sections 4.1–4.4) is designed to support. The paper benchmarks Nemotron 3 Nano Omni against its LLM backbone (Nemotron 3 Nano 30B-A3B) on text-only tasks and shows that multimodal training preserves text performance: MMLU-Pro drops only from 78.3 to 77.3, GPQA from 73.0 to 72.2. Against the prior vision-language model (Nemotron Nano V2 VL), the paper shows consistent gains across vision benchmarks: MathVista from 69.0 to 71.9 without reasoning, MMLongBench-Doc from 32.1 to 46.1. The implicit claim is that the staged training recipe successfully avoids the negative transfer that plagues many multimodal training efforts—adding audio did not break vision, and adding both did not break text.

The Unstated Motivation: Enabling Agentic Multimodal Applications

Reading between the lines, there is a deeper motivation that the paper gestures toward but does not fully articulate. The benchmarks it chooses to emphasize—ScreenSpot, ScreenSpot-Pro, OSWorld for GUI understanding; MMLongBench-DOC for document analysis; WorldSense and DailyOmni for audio-visual reasoning—are not random selections from the multimodal evaluation landscape. They are precisely the capabilities needed for autonomous agents that interact with the world through screens, documents, and conversations. An agent that can look at a desktop screenshot, understand where to click, listen to a user's spoken instructions, and reason about a long document is not a narrow tool; it is a general-purpose digital assistant. The architecture decisions—256K context, dynamic resolution for high-res screenshots, temporal audio-visual alignment—make far more sense when viewed through this lens. The model is not just designed to answer questions about images and audio; it is designed to act on multimodal inputs in agentic loops, though the paper only explores this direction through the GUI grounding and OSWorld evaluations rather than through end-to-end agent benchmarks.

The Data Challenge: Where Multimodal Training Data Comes From

One of the underappreciated challenges in building omni-modal models is data scarcity. While text data is abundant (the internet), and image-text pairs are increasingly plentiful (thanks to alt-text, captions, and synthetic generation), audio-visual data with high-quality annotations is rare. The paper addresses this through a synthetic data pipeline that is described across multiple sections: for vision-language data (Stage 1), the team uses frontier open-source models like Qwen3-VL, Qwen 3.5, GPT-OSS-120B, and Kimi-K2.5 to generate question-answer pairs and reasoning traces from images, videos, and OCR-extracted text; for audio data (Stage 3), they use Qwen3-Omni-30B-A3B to produce captions and specialized music tools to produce metadata, then GPT-OSS-120B to generate QA pairs from those captions; for omni-modal data (Stage 5), videos are segmented into 20-second clips, audio-visual metadata is extracted, and QA pairs plus reasoning traces are generated with GPT-OSS-120B. The scale is substantial: Stage 1 alone uses 86.3M samples (214.8B tokens), and the full SFT pipeline spans 434.1M samples across 466.9B tokens (Table 1). This reliance on synthetic data is not just a practical convenience—it is a statement about the maturity of the field. Open-source models are now good enough to serve as data generators for training the next generation, creating a flywheel effect where each generation's outputs enable the next.

Summary: What Gap Is Being Filled?

Nemotron 3 Nano Omni fills a specific, well-defined gap: there was no open-source, efficiently deployable model that could simultaneously understand text, images, audio, and video at production-grade throughput while preserving or exceeding the accuracy of its single-modality counterparts. Prior open-source omni models (Qwen3-Omni) achieved the capability but not the efficiency. Prior efficient models (Nemotron Nano V2 VL) achieved vision-language efficiency but lacked audio entirely. The paper's contribution is not a single breakthrough but a carefully orchestrated combination of architecture (MoE hybrid backbone, token reduction), training recipe (seven-stage SFT, modality-progressive RL), data pipeline (synthetic generation at scale), and deployment engineering (FP4 quantization, EVS runtime pruning) that collectively closes this gap. The question the experiments must answer is whether this combination actually delivers on the promise of "efficient and open multimodal intelligence" without hidden costs—loss of text capability, brittleness to quantization, or overfitting to synthetic data distributions—and we will examine those results in subsequent sections.

3. Technical Approach

3.1 Reader Orientation

Nemotron 3 Nano Omni is a multimodal language model that takes in text, images, audio, and video as input and produces text as output, built by connecting three specialized encoders (one for vision, one for audio, one for text) to a shared language model backbone through learned adapter layers. The system solves the problem of efficiently processing long, mixed-modality inputs—such as hour-long videos with synchronized audio, multi-page documents with charts, and spoken conversations—by using token-reduction techniques (Conv3D for temporal compression, Efficient Video Sampling for spatial pruning, pixel shuffle for visual downsampling) that dramatically reduce the number of tokens flowing into the language model before it ever processes them, combined with a Mixture-of-Experts architecture that activates only 3 billion of its 30 billion parameters per token, collectively delivering throughput that is 3× to 9× higher than comparable omni-modal models at equivalent accuracy.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a feed-forward pipeline:

Vision Encoder (C-RADIOv4-H) — a vision transformer that converts images and video frames into visual feature vectors. For images, it applies dynamic resolution processing that preserves native aspect ratios and uses pixel shuffle with 4× downsampling to reduce token count. For video, it applies Conv3D temporal compression that fuses every two consecutive frames into a single tubelet, halving the temporal token count. The output is a sequence of visual embeddings.
Audio Encoder (Parakeet-TDT-0.6B-v2) — a FastConformer-based speech encoder that converts raw audio waveforms (resampled to 16 kHz mono) into audio feature vectors. It computes log-mel spectrogram features with 10 ms hop size, applies three stride-2 convolutional subsampling layers for ~8× temporal downsampling, and produces approximately 12.5 tokens per second of audio. Audio streams are segmented into 30-second clips (~375 tokens each).
MLP Projectors (Vision Adaptor and Audio Adaptor) — learned multi-layer perceptron layers that map the heterogeneous output spaces of the vision and audio encoders into the unified embedding space of the language model. These are trained to align modalities.
Nemotron 3 Nano 30B-A3B LLM Backbone — a hybrid Mixture-of-Experts language model that serves as the shared decoder. It processes the concatenated sequence of visual, audio, and text embeddings autoregressively, producing the output text. The MoE architecture means only a subset of parameters are active per token, providing high throughput on long sequences.
Efficient Video Sampling (EVS) — a runtime-only pruning module that sits between the vision encoder/projector and the LLM. It computes cosine dissimilarity between consecutive frame tubelets at each spatial position and keeps only the globally most-dissimilar tokens up to a budget set by the pruning rate $q$ , with the first tubelet always preserved as an anchor. This is a post-hoc sparsification mechanism that is not trained.

Information flow: Raw multimodal input → modality encoders produce modality-specific embeddings → MLP projectors map to shared embedding space → EVS optionally prunes video tokens → interleaved token sequence fed to LLM backbone → autoregressive text generation.

3.3 Roadmap for the Deep Dive

First, the overall encoder-projector-decoder architecture and the specific encoder configurations, since these define the input processing pipeline and the token budget per modality—understanding the token counts is essential for understanding why token-reduction matters.
Second, the token-reduction mechanisms (dynamic resolution, pixel shuffle, Conv3D, Efficient Video Sampling), since these are the architectural innovations that enable efficient long-context multimodal processing and directly differentiate the model from prior work.
Third, the LLM backbone and how it handles interleaved multimodal sequences, including how modality tokens are temporally ordered for joint audio-visual reasoning.
Fourth, the seven-stage supervised fine-tuning (SFT) pipeline, since this is the staged curriculum that progressively aligns modalities and scales context length—the recipe that makes omni-modal training stable.
Fifth, the reinforcement learning pipeline (MPO, text RL, image RL, omni RL), since this is where reasoning, instruction-following, and safety are refined after SFT.
Sixth, the quantization strategy (FP8 and NVFP4 formats), since this is how the model achieves deployment efficiency and is a key part of the "efficient" claim.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core contribution is the careful integration of heterogeneous modality encoders with an efficient MoE language model backbone, enabled by a staged training curriculum and complementary token-reduction techniques at both the architectural and runtime levels.

The Encoder-Projector-Decoder Architecture

The model follows a standard encoder-projector-decoder design common to multimodal LLMs, but with modality-specific encoders chosen for their efficiency-accuracy tradeoffs. The vision encoder is C-RADIOv4-H (Ranzinger et al., 2026; Heinrich et al., 2025), a vision transformer that processes images and video frames into visual embeddings. The audio encoder is Parakeet-TDT-0.6B-v2 (Xu et al., 2023; Rekesh et al., 2023; Sekoyan et al., 2025), a FastConformer-based speech recognition model whose encoder is repurposed to produce general audio embeddings. The text encoder is the standard tokenizer of the Nemotron 3 Nano 30B-A3B language model. Each non-text modality passes through two MLP projector layers (one for vision, one for audio) that map the modality-specific embedding dimensions into the unified embedding space expected by the language model.

The design choice to use separate, specialized encoders rather than a single unified encoder for all modalities is deliberate. Vision and audio have fundamentally different temporal and spectral characteristics—images are 2D spatial grids with high spatial resolution, audio is a 1D temporal signal with high temporal resolution but low per-sample information density. A unified encoder would need to handle both simultaneously, which typically leads to compromises in either resolution or compute efficiency. By keeping the encoders separate, each can be optimized for its modality: the vision encoder uses 16×16 patchification with variable patch counts per image, while the audio encoder uses log-mel spectrogram features with convolutional subsampling that reduces the temporal dimension by ~8× before the transformer layers.

The MLP projectors serve a critical alignment function. During early training stages (Stage 0 for vision, Stage 2 for audio), only these projectors are trained while all other components remain frozen. This is a projector warmup phase: the goal is to find a linear-plus-nonlinear mapping from the encoder output space to the LLM embedding space without disturbing the carefully pretrained representations in either the encoders or the LLM. Without this warmup, joint training from scratch risks gradient interference where the LLM's loss signal distorts the encoder representations before they have a chance to align, leading to training instability or poor final performance. The paper uses this two-stage alignment (first projector-only, then encoder-plus-projector) as a general recipe for integrating new modalities.

Dynamic Image Resolution and Visual Token Processing

The paper replaces the tiling-based image processing from Nemotron Nano V2 VL with a dynamic resolution strategy that preserves native aspect ratios. Under the tiling approach, images were divided into fixed-size square tiles regardless of their actual proportions, which is computationally convenient (all tiles share the same dimensions, enabling efficient batching) but destroys the spatial layout information that is critical for document understanding, chart interpretation, and GUI grounding—a wide bar chart and a tall bar chart encode different information partly through their aspect ratios.

Under the dynamic approach, each image is decomposed into a variable number of 16×16 pixel patches, with the total number of visual tokens per image constrained to a range:

$\text{min\_tokens} = 1,024, \quad \text{max\_tokens} = 13,312$

where the minimum of 1,024 tokens corresponds to an image of 512×512 pixels (for square images: $(512/16)^2 = 32^2 = 1,024$ ), and the maximum of 13,312 tokens corresponds to an image of approximately 1,840×1,840 pixels (for square images: $(1840/16)^2 \approx 115^2 = 13,225$ ). The actual number of patches depends on the image's native resolution and aspect ratio—a wide image might produce a 3×1 grid of patch rows and columns, while a tall image produces a 1×3 grid, preserving the geometric relationships that carry semantic information.

After patchification but before the patches are fed to the LLM, the system applies a pixel shuffle operation with 4× downsampling. Pixel shuffle is a spatial-to-depth rearrangement: it takes a 2×2 block of neighboring patches and stacks them along the channel dimension, reducing the spatial resolution by a factor of 2 in each dimension, then does this a second time for a total 4× spatial reduction. This means that the number of tokens presented to the language model is 4× fewer than the number of patches produced by the vision encoder, but each token carries information from a larger spatial region. The tradeoff is between spatial granularity and token count—for tasks like document reading where fine-grained text recognition matters, this downsampling could theoretically lose information, but in practice the vision encoder's patch representations are sufficiently rich that the 4× compression is a net win for efficiency without measurable accuracy loss (the paper does not ablate the pixel shuffle specifically, but the overall vision pipeline achieves strong OCR and document understanding results).

Video Processing: Conv3D Temporal Compression and Efficient Video Sampling

Video processing is where the token-count problem becomes acute. A naive approach that treats each video frame as an independent image would produce token counts that scale linearly with frame count, quickly overwhelming the LLM's context window even for relatively short videos. Nemotron 3 Nano Omni addresses this through two complementary mechanisms that operate at different stages of the pipeline.

Conv3D Patch Embedding. This is an architectural change applied during both training and inference. Instead of embedding each frame independently through the vision encoder's standard 2D patch embedding layer, consecutive frames are fused into "tubelets" before entering the first vision transformer block:

Every $T = 2$ consecutive frames are fused into a single tubelet via a 3D convolutional patch embedder.

Operationally, a Conv3D kernel with temporal stride $T$ and spatial stride matching the vision encoder's patchification stride processes a stack of $T$ frames, producing a single output embedding per spatial position per tubelet. The effect is to halve the number of temporal tokens: if a video has $F$ frames, it produces $F/T = F/2$ tubelets. This reduces the number of vision tokens flowing through both the vision transformer (reducing ViT prefill cost) and the language model (reducing LLM prefill, attention compute, and KV-cache footprint). The paper reports that for a 512-frame video, Conv3D alone reduces the number of input tokens for the LLM from approximately 141K to approximately 75K, a 47% reduction (Section 4.6, Table 12).

The Conv3D approach is superior to simple frame subsampling (e.g., keeping every other frame) because it preserves information from all frames in the compressed representation. The 3D convolution can learn to combine information from consecutive frames—detecting motion, capturing temporal transitions, or averaging out noise—rather than simply discarding half the temporal signal. This is an architectural inductive bias: the model "knows" during training that it will receive temporally fused representations, so it can learn representations that exploit this compression.

Efficient Video Sampling (EVS). This is a runtime-only mechanism (Bagrov et al., 2025) that operates after the vision transformer blocks and the vision adapter, immediately before tokens enter the LLM. It is a post-hoc pruning step that drops tokens based on their information content. For each spatial position $(h, w)$ in the video frame grid, EVS computes the cosine dissimilarity between the embeddings of consecutive tubelets at that position:

$\text{dissimilarity}(h, w, t) = 1 - \cos(\text{emb}(h, w, t), \text{emb}(h, w, t-1))$

where emb $(h, w, t)$ is the vision adapter output at spatial position $(h, w)$ of tubelet $t$ . The cosine dissimilarity ranges from 0 (identical embeddings) to 2 (opposite embeddings), with higher values indicating that the scene content at that spatial position changed significantly between consecutive tubelets.

EVS then globally ranks all spatial positions across all tubelets by their dissimilarity score and keeps only the top $k$ most-dissimilar tokens, where $k$ is determined by the pruning rate $q$ :

$k = \text{floor}((1 - q) \times \text{total\_tokens})$

The first tubelet ( $t = 1$ ) is always assigned the maximum dissimilarity score, ensuring it is always retained as an anchor reference point regardless of the pruning rate. This prevents the model from losing the initial frame context, which is often critical for understanding the overall scene.

The key insight behind EVS is that adjacent video frames are highly redundant in the spatial dimension—most pixels change very little from frame to frame. By keeping only the positions where significant changes occur (high cosine dissimilarity), EVS eliminates tokens that carry near-duplicate information, reducing the LLM input size without losing the temporal dynamics that matter for understanding. The mechanism composes multiplicatively with Conv3D: Conv3D halves the temporal token count through architectural fusion, then EVS prunes the remaining tokens spatially. The paper reports that with Conv3D enabled and EVS at $q = 0.5$ , the 512-frame video drops from ~141K tokens to ~42K—a 70% reduction versus the baseline with neither mechanism (Table 12).

Table 13 in the paper demonstrates that EVS is remarkably robust to aggressive pruning rates: with Conv3D enabled, per-benchmark accuracy on DailyOmni, Video-MME, and WorldSense is essentially flat through $q = 0.7$ , with LongVideoBench being the most sensitive benchmark (showing a slight decline at $q = 0.7$ and a more noticeable drop beyond $q = 0.8$ ). Time-to-first-token (TTFT) improves monotonically with higher pruning rates, for approximately a 14% reduction at $q = 0.7$ versus no EVS. This means that for deployment, $q = 0.7$ is a sweet spot: it provides meaningful latency reduction at essentially zero accuracy cost on most benchmarks.

An important detail: the EVS pruning rate $q$ controls what fraction of tokens are dropped, not kept. So $q = 0.5$ means 50% of tokens are pruned (kept: 50%), $q = 0.7$ means 70% are pruned (kept: 30%), and $q = 0.95$ means 95% are pruned (kept: 5%).

Audio Encoding and Temporal Alignment

Audio inputs are processed through a dedicated pipeline designed for efficient temporal compression. The audio signal is first resampled to 16 kHz mono, standardizing all inputs to a single channel and sample rate regardless of the original recording quality. This is a practical choice that balances audio fidelity against computational cost—16 kHz preserves frequencies up to 8 kHz (covering the critical range for speech intelligibility and most environmental sounds) while producing fewer samples per second than higher rates like 44.1 kHz.

The Parakeet-TDT-0.6B-v2 FastConformer encoder processes the audio in two stages. First, it computes log-mel spectrogram features with a 10 ms hop size, converting the raw waveform into a time-frequency representation with 10 ms temporal resolution. The log-mel transformation compresses the dynamic range (making quiet and loud sounds more comparable) and emphasizes perceptually relevant frequency bands. The 10 ms hop size means that for every second of audio, there are 100 spectrogram frames, each representing a 25-30 ms window of audio (the exact window length depends on the FFT parameters, but the hop size defines the stride between consecutive windows).

Second, three stride-2 convolutional subsampling layers are applied sequentially to the spectrogram frames. Each stride-2 convolution reduces the temporal dimension by a factor of 2, so three such layers provide an overall ~8× temporal downsampling:

$\text{frames\_after\_subsampling} \approx \frac{\text{spectrogram\_frames}}{8}$

This reduces the 100 frames per second of audio to approximately 12.5 tokens per second (~80 ms per token). The subsampled features are then processed by the FastConformer's transformer layers to produce the final audio embeddings.

Audio streams are segmented into 30-second clips, with each clip corresponding to approximately 375 tokens (30 seconds × 12.5 tokens/second). The last clip accounts for any remainder—if a stream is 45 seconds, it produces one 30-second clip (375 tokens) and one 15-second clip (~188 tokens). Streams shorter than 30 seconds are not padded, so a 5-second clip produces ~63 tokens with no waste. The paper states that the model is trained to handle inputs ranging from 0.5 seconds to 20 minutes, but the model context length can theoretically accommodate audio input of over 5 hours. This wide dynamic range is achieved by the combination of aggressive temporal subsampling (12.5 tokens/second) and the 256K context window: at 12.5 tokens/second, 256K tokens can hold approximately 20,480 seconds = 341 minutes ≈ 5.7 hours of audio, though practical performance on such long inputs is limited by training data coverage.

Multimodal Sequence Construction and Temporal Interleaving

For inputs containing both visual and audio streams—such as videos with synchronized audio—the modality tokens are interleaved in temporal order during sequence construction. The paper states this explicitly in Section 2:

"modality tokens are interleaved in temporal order during sequence construction to enable joint temporal reasoning across modalities"

This is a non-trivial design choice with significant implications for how the model learns cross-modal relationships. Consider a 30-second video clip: the system extracts video frames (producing visual tokens at certain timestamps) and audio segments (producing audio tokens at a much finer temporal granularity—12.5 tokens per second). If these tokens were simply concatenated as separate blocks (all visual tokens first, then all audio tokens), the model would need to learn long-range correspondences across potentially tens of thousands of tokens to associate a visual event at frame 150 with a sound at time 15 seconds. By interleaving in temporal order, tokens that are temporally proximate in the original media are also spatially proximate in the token sequence. A visual token from frame at time $t$ sits near audio tokens from the same time $t$ , making it easier for the transformer's attention mechanism to learn cross-modal associations.

The actual implementation details of this interleaving—how exactly visual and audio tokens are interleaved when they have different temporal granularities—are not specified in the paper. A reasonable inference is that visual tokens from a frame are inserted at the position corresponding to that frame's timestamp in the audio token stream, or that temporal segments of audio are placed between visual tokens from bracketing frames. This is a low-level sequence construction detail that affects cross-modal attention patterns and is an area where the paper leaves room for further specification.

The LLM Backbone: Nemotron 3 Nano 30B-A3B

The language model backbone is a hybrid Mixture-of-Experts architecture with approximately 30 billion total parameters but only 3 billion active parameters per token (hence "30B-A3B"). The "hybrid" designation refers to the fact that it combines Mamba state-space model layers with standard transformer attention layers, a design choice optimized for efficient processing of long sequences. The Mamba layers excel at modeling long-range dependencies with linear complexity in sequence length (as opposed to the quadratic complexity of self-attention), while the transformer attention layers provide the precise token-level interaction that benefits reasoning tasks. This hybrid design is particularly well-suited to multimodal inputs because multimodal sequences are inherently long—a video with audio can easily produce 50K+ tokens—and pure attention mechanisms become computationally prohibitive at these lengths.

The MoE component means that the model's feed-forward network (FFN) layers are divided into multiple "experts" (specialized sub-networks), and a router mechanism selects a subset of these experts for each input token. With 32-way expert parallelism (as specified in the training hyperparameters in Section 3.1.8), each token activates only a fraction of the available experts, dramatically reducing the per-token computation compared to a dense model of equivalent total parameter count. The paper reports using expert parallelism (EP) with 32-way parallelization during training, indicating that experts are distributed across multiple GPUs to manage memory and compute.

The "A3B" suffix in the model name indicates that the architecture activates approximately 3 billion parameters per forward pass per token, despite having a total parameter count of 30 billion. This is the core efficiency mechanism: at inference time, the model only computes through the active experts, so the computational cost scales with active parameters (3B) rather than total parameters (30B), while maintaining the representational capacity of the full 30B model. This is why the paper can claim 3× higher throughput than a model like Qwen3-Omni (which uses a different architecture) at comparable accuracy—the per-token cost is fundamentally lower.

The LLM backbone is trained with standard language modeling objectives during pretraining (not described in this paper, as it uses a previously trained checkpoint from NVIDIA et al., 2025a), and then fine-tuned during the multimodal SFT stages with standard next-token prediction loss on the combined multimodal sequences.

The Seven-Stage Supervised Fine-Tuning Pipeline

The SFT training is the centerpiece of the paper's training methodology. Rather than mixing all modalities together from the start, the pipeline is divided into seven stages that progressively introduce new modalities, unfreeze additional components, and scale context length. This curriculum is explicitly designed to "promote stable cross-modal alignment and mitigate catastrophic forgetting while improving multi-modal understanding" (Section 3.1). Table 1 summarizes the scale: 434.1 million total samples across 466.9 billion tokens.

The progression logic is: vision alignment → vision-language capabilities → audio alignment → audio capabilities → joint omni-modal training → extended context → ultra-long context. At each transition, the risk of catastrophic forgetting or training instability increases, so the earlier stages establish stable foundations before the later stages introduce complexity.

Stage 0: Vision Projector Warmup (Context: 16K tokens)

Trainable: Only the vision MLP projector. Frozen: LLM backbone, vision encoder, audio components. Data: 9.35M vision-text samples (~15.5B tokens) covering captioning, OCR, document understanding, VQA, visual grounding, and GUI understanding. Learning rate: $1 \times 10^{-3}$ .

The purpose of this stage is to find a mapping from the frozen vision encoder's output space to the frozen LLM's embedding space. Since both the encoder and the LLM have rich, pretrained representations, training only the projector prevents the LLM's loss signal from distorting the vision encoder's features or the LLM's internal representations. The high learning rate ( $1 \times 10^{-3}$ ) reflects that the projector is randomly initialized and needs to learn quickly.

The data mixture includes a portion of the Stage 1 dataset, chosen to cover diverse vision-language tasks so that the projector learns to map visual features for a wide range of downstream uses rather than overfitting to a narrow task.

Stage 1: Vision SFT 16K (Context: 16K tokens)

Trainable: LLM backbone, vision encoder, vision projector. Frozen: Audio components. Data: 86.3M vision-language samples (~214.8B tokens). Learning rate: $5 \times 10^{-5}$ .

This is the largest single stage in terms of token count and is where the model develops its core vision-language capabilities. All vision-related parameters (encoder, projector) and the LLM backbone are jointly trained. The data builds on the Nemotron Nano V2 VL Stage 1 dataset with several enhancements: text-only subsets replaced with higher-quality reasoning samples from Nemotron 3 Nano 30B-A3B; noisy labels re-annotated using Qwen3-VL series models; reasoning traces incorporated from Qwen3-VL, Qwen3.5, and Kimi-K2.5; and coverage expanded across GUI understanding, visual grounding, charts, tables, document understanding, video understanding, and multiple languages.

The synthetic data pipeline for this stage uses frontier open-source models as data generators. For each domain (images, videos, OCR-extracted text), the team generates question-answer pairs with domain-specific instructions, distills reasoning traces, and applies strict filtering for correctness and quality. The scale—86.3 million samples—is feasible only because of this synthetic pipeline; manual annotation at this scale would be prohibitive.

A notable detail: the vision encoder's CPE (Conditional Positional Encoding) layers are kept in evaluation mode during this stage (as noted in Section 3.1.8, Table 6). The CPE layers implement position-dependent transformations, and freezing them during training is a stabilization technique—these layers can introduce training instability if they drift during fine-tuning, and their positional information is likely already well-calibrated from pretraining.

Stage 2: Audio Projector Warmup (Context: 16K tokens)

Trainable: Only the audio MLP projector. Frozen: LLM backbone, vision encoder, vision projector, audio encoder. Data: 59.2M ASR samples (~11.4B tokens) from the Granary v1.1 dataset. Learning rate: $1 \times 10^{-3}$ .

This stage is the audio analog of Stage 0: it warms up the audio projector while keeping everything else frozen. The choice of ASR data for projector warmup is interesting—automatic speech recognition data has a clear, well-defined mapping from audio to text (transcription), making it an ideal task for learning the basic audio-to-language alignment before introducing more complex audio understanding tasks.

The Granary v1.1 dataset (Koluguri et al., 2025) provides diverse ASR data across varied acoustic conditions, speaking styles, and languages, ensuring the projector learns a robust mapping rather than overfitting to clean speech.

Stage 3: Audio Encoder & Projector Training (Context: 16K tokens)

Trainable: Audio encoder and audio projector. Frozen: LLM backbone, vision encoder, vision projector. Data: 242.0M audio samples (~100.5B tokens). Learning rate: $2.5 \times 10^{-5}$ .

This stage unfreezes the Parakeet-TDT audio encoder (which was previously frozen) and jointly trains it with the audio projector. The LLM and vision components remain frozen to prevent cross-modal interference. The data mixture, shown in Table 2, comprises four categories: ASR (113.8M samples, 22.7% of tokens), sound understanding (61.0M samples, 24.4% of tokens), music understanding (19.8M samples, 43.3% of tokens), and speech understanding (47.5M samples, 9.6% of tokens).

The audio training data is paired with captions, multiple-choice questions, and open-ended questions, with a subset augmented with reasoning traces. The synthetic data generation leverages Qwen3-Omni-30B-A3B for captions and specialized music tools for metadata, with GPT-OSS-120B generating QA pairs from these intermediate representations. This two-step pipeline (extract metadata → generate QA) is a common pattern in synthetic data generation: the metadata extraction stage provides structured information about the audio content (what sounds are present, their temporal relationships, speaker characteristics), and the QA generation stage converts this structured information into diverse natural language questions.

Stage 4: Omni SFT 16K (Context: 16K tokens)

Trainable: All model parameters (LLM, vision encoder, vision projector, audio encoder, audio projector). Data: 30.5M omni-modal samples (~57.3B tokens). Learning rate: $1 \times 10^{-5}$ .

This is the first stage where all modalities are trained jointly. The data mixture, detailed in Table 3, spans vision data (14.6M samples, 53.4% of tokens), audio data (9.3M, 16.9%), short video data (1.3M, 11.0%), ASR (2.9M, 1.1%), text data (948K, 6.1%), and smaller amounts of safety, reasoning, and omni (audio+video) data. The vision data dominates the token budget (30.6B tokens), reflecting the relative abundance of vision-language training data compared to audio-visual data.

The lower learning rate ( $1 \times 10^{-5}$ ) compared to earlier stages reflects that all components already have well-tuned representations from prior stages—the goal now is fine-grained joint optimization rather than rapid adaptation.

For this stage, the omni-modal data (audio-visual captions, open-ended QA, multiple-choice QA) is generated from videos less than 2 minutes in length. The pipeline extracts audio-visual metadata from videos using multimodal models, then uses that metadata to generate QA pairs and summaries with Qwen3-Omni-30B-A3B and GPT-OSS-120B. Note the use of a different model for metadata extraction (Qwen3-Omni) versus QA generation (GPT-OSS)—this separation likely ensures that the generated questions are grounded in actual video content rather than hallucinated by the QA generator.

Stage 5: Omni SFT 48K (Context: 49,152 tokens)

Trainable: All model parameters. Data: 6.08M samples (~33.5B tokens). Learning rate: $1 \times 10^{-6}$ .

This stage extends the context length from 16K to 48K tokens and rebalances the data mixture toward longer sequences. Table 4 shows the per-category breakdown: medium and long video, omni data with joint audio-video understanding, and reasoning traces receive the bulk of the training budget, while short video and omni data are downsampled. The largest single category is medium+long omni data (710K samples, 39.1% of tokens), reflecting the focus on extended audio-visual sequences.

The context length extension requires architectural changes: the paper introduces 2-way context parallelism (CP) at this stage (Table 6), which distributes the long sequence across GPUs along the sequence dimension, with each GPU processing a contiguous segment and communicating at attention boundaries.

The omni-modal data for this stage is generated by segmenting videos into 20-second clips, extracting audio-visual metadata, and generating QA pairs and reasoning traces with GPT-OSS-120B. The segmentation into 20-second clips is likely a practical constraint—the metadata extraction models may have limited context windows—but it also provides a natural unit for question generation: questions about specific temporal segments are easier to verify than questions about hour-long videos.

The learning rate drops significantly to $1 \times 10^{-6}$ , reflecting that the model is now in a fine-tuning regime where large parameter updates would risk degrading the representations learned in earlier stages.

Stage 6: Omni SFT 256K (Context: 262,144 tokens)

Trainable: All model parameters except the audio encoder and audio projector. Data: 623K samples (~34.0B tokens). Learning rate: not explicitly stated (assumed similar to Stage 5). Context parallelism: 16-way.

This stage pushes the context length to the maximum 256K tokens and focuses exclusively on long-context text and document understanding. The audio components are frozen to "focus model capacity on long-context text and document understanding" (Section 3.1.7)—a pragmatic choice that acknowledges the tension between long-context capability and audio performance, and prioritizes the former for this stage.

Table 5 shows the data composition: long-context vision data dominates (508K samples, 90.9% of tokens, 30.9B tokens), with smaller amounts of text (63K, 7.3%), long-context text (2.2K, 1.5%), and standard-context vision (50K, 0.3%). The long-context vision data consists of documents spanning 10 to 100+ pages—academic papers, financial reports, presentations—with synthetic QA pairs and reasoning traces at the page, multi-page, and full-document levels. This is the data that enables the model's strong performance on MMLongBench-DOC and related document understanding benchmarks.

The paper notes that the team releases "runnable data pipeline recipes using NeMo Data Designer" for this stage, suggesting that the long-document QA generation pipeline is a reusable contribution that other teams can adapt.

Training Infrastructure and Hyperparameters

All SFT stages are trained using the Megatron framework (Shoeybi et al., 2019) with Transformer Engine and the Megatron Energon dataloader. Training is conducted on 32–128 nodes of NVIDIA H100 GPUs, depending on the stage (Table 6). The parallelization strategy uses 2-way tensor parallelism (splitting individual transformer layers across GPUs), 32-way expert parallelism (distributing MoE experts across GPUs), and sequence parallelism (distributing the sequence dimension). Context parallelism (CP) is introduced in later stages: 2-way CP in Stage 5, 16-way CP in Stage 6.

The paper uses several memory-saving techniques to fit long sequences in GPU memory:

Selective activation recomputation for the LLM backbone: recomputing core attention, MLP, LayerNorm, and MoE activations during the backward pass rather than storing them.
Full block-level recomputation for all 32 vision encoder layers.
Sound model activation recomputation starting from Stage 4.
Vision and sound projection recomputation from Stage 5 onward.

These recomputation strategies trade additional computation (recomputing activations during backpropagation) for reduced memory usage, enabling longer sequence training at the cost of increased training time.

All stages use the AdamW optimizer with $\beta_1 = 0.9$ and $\beta_2 = 0.999$ , a cosine learning rate decay schedule with linear warmup, and BF16 mixed precision. Online sequence packing with a balanced greedy knapsack algorithm maximizes GPU utilization by combining multiple short sequences into a single training example.

Reinforcement Learning Pipeline

After SFT, the model undergoes multiple rounds of reinforcement learning to improve instruction following, reasoning, and safety alignment. The RL pipeline follows a curriculum: Preference Optimization → Text-RL-stage-1 → Image-RL → Omni-RL → Text-RL-stage-2.

Mixed Preference Optimization (MPO)

The first RL stage uses Mixed Preference Optimization (Wang et al., 2024a), which combines a preference loss (Direct Preference Optimization, DPO) and a quality loss (Binary Classifier Optimization, BCO) during offline RL. The paper explains this combination as using "both preference-level and quality-level supervision."

DPO (Rafailov et al., 2023) optimizes the model to prefer chosen responses over rejected ones, using pairs of responses where one is preferred and the other is not. The DPO objective is:

$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)} \left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$

where $\pi_\theta$ is the policy being optimized, $\pi_{\text{ref}}$ is the reference policy (the SFT checkpoint), $y_w$ is the preferred response, $y_l$ is the rejected response, and $\beta$ is a temperature parameter controlling how far the policy can deviate from the reference. The intuition is that DPO increases the relative probability of preferred responses while keeping the policy close to the reference, preventing reward hacking.

BCO adds an additional quality loss that rewards responses labeled as correct (positive samples) and penalizes those labeled as incorrect (negative samples), providing a stronger signal than relative preference alone.

The training data is constructed via rejection sampling in the vision domain: candidate responses are generated from the initial policy, evaluated for outcome correctness, and assigned binary labels (positive for correct, negative for incorrect).

Text-RL Stages

During text-only RL, only the language model parameters are trained. The paper reuses the RL data and infrastructure from the post-training of Nemotron 3 Nano and Super (NVIDIA et al., 2025b, 2026). A key detail: during text-only RL stages, the LLM input token embedding parameters are frozen to "mitigate representational drift between multi-modal stages." This is a practical stability measure—the input embeddings are the interface between the modality projectors (which produce embeddings in the LLM's space) and the LLM's transformer layers. If the input embeddings drift during text RL, the vision and audio projectors would need to re-learn their mapping to the shifted embedding space, potentially degrading multimodal performance.

The RL algorithm used is Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025; Shao et al., 2024), an adapted version of the standard policy optimization framework. The global batch size is 4,096 with 16 rollouts per prompt and a micro-batch size of 1.

Image RL

Image RL is the first multimodal RL stage, focusing on visual reasoning tasks. The training corpus is divided into five categories:

Chart, document, and text-rich image reasoning (~28K prompts): numerical, comparative, and trend reasoning over plots, tables, diagrams, infographics, and text-containing natural images.
STEM and mathematical problems (~19K prompts): geometry, algebra, functions, and counting, in both English and Chinese.
Game and puzzle reasoning (~12K prompts): rule-based reasoning over rendered game-board states.
Visual question answering (~8K prompts): open-ended and multiple-choice questions covering spatial relations, attribute recognition, and yes/no judgments.
Visual grounding (~7K prompts): click-coordinate prediction on desktop, mobile, and web screenshots.

The reward for each prompt is a scalar in $[0, 1]$ that linearly combines an outcome score and a format score. The outcome verification uses four rule-based verifiers chosen per prompt:

String-match for free-form text answers: exact or fuzzy string matching against ground truth.
Mathruler for symbolic equivalence on numeric and algebraic answers: checking mathematical equivalence rather than string equality.
Multiple-choice for selected-letter answers: checking if the chosen option (e.g., "A", "B") matches the correct answer.
GUI-coordinate for click-target predictions: a reward that smoothly decays with Euclidean distance from the correct target coordinates, using a hand-designed decay function.

The format score rewards a specific output structure: a single "thinking" reasoning block (enclosed in ... tags) followed by a single \boxed{answer} entry. Partial credit is awarded when the policy emits extra reasoning or boxed entries, rather than zeroing out the reward entirely. This is a practical concession: VLM checkpoints after SFT often produce outputs that are slightly malformed (extra reasoning, multiple boxed answers), and a strict format requirement would zero out rewards for otherwise correct answers. The partial credit approach keeps these correct answers in the training signal while still encouraging proper formatting.

To ensure informative learning signals, the paper applies pass-rate filtering using 8 rollouts per prompt from the initial policy checkpoint. Only prompts with an empirical pass rate below 0.8 are retained; prompts that are trivially solvable at initialization are discarded because they provide no room for improvement. The filtering uses the same verifiers that are used during training, maintaining consistency.

The corpus and verifier suite from Image RL are inherited by the subsequent Omni-RL stage as the image component of its mixed-modality training mixture.

Omni-RL

Omni-RL is the most complex RL stage, designed to improve coherent reasoning across all modalities simultaneously. The training corpus comprises approximately 120K prompts spanning 113 sub-datasets across four modality groups:

Omni RL data (~17.6K samples): synthetic data generated from video content with accompanying audio, covering diverse visual understanding and temporal reasoning tasks. This is the only data type that explicitly combines audio and visual modalities.
Video RL data (~8.5K samples): video-only question-answer pairs targeting spatial, temporal, and causal reasoning.
Image RL data (~32K samples): the image reasoning set inherited from the Image RL stage, redistributed across OCR (~10.5K), chart analysis (~8.9K), game-related visual QA (~11.9K), GUI grounding (~7.1K), and additional curated domains.
Audio RL data (~4.2K samples) and ASR (~3.8K samples): audio question-answering and automatic speech recognition tasks at various utterance lengths.

The verification pipeline supports five task types: multiple-choice (34% of prompts), string matching (31%), mathematical rule-based verification (26%), GUI coordinate grounding (6%), and ASR evaluation (3%).

The ASR verifier is particularly interesting: the reward is $1 - \text{WER}$ , where WER (Word Error Rate) is computed after text normalization. This provides a continuous reward signal between 0 (completely wrong transcription) and 1 (perfect transcription), with partial credit for partially correct transcriptions. The use of a WER-based reward is a natural choice because WER is the standard ASR evaluation metric and captures fine-grained differences in transcription quality. Text normalization (e.g., lowercasing, removing punctuation, normalizing numbers) is applied before computing WER to avoid penalizing superficial formatting differences.

Pass-rate filtering for Omni-RL uses different thresholds depending on the task: prompts are retained if the base model achieves a pass rate between 0.1 and 0.9 overall, with stricter 0.3–0.7 bands for AudioQA tasks. The tighter bands for AudioQA reflect that audio question-answering is particularly sensitive to difficulty calibration—prompts that are too easy provide no learning signal, while prompts that are too hard may be impossible for the current policy to solve, leading to random exploration rather than structured improvement.

An additional design choice: the Omni-RL training mixture includes a small set of unanswerable or modality-mismatched samples (~4K). These are prompts where the visual evidence is insufficient to answer the question, or where the image and text do not correspond. Training the model to abstain (output a refusal or an "unanswerable" token) on these samples prevents it from hallucinating answers when evidence is missing, which is critical for safe deployment in agentic applications where acting on incorrect information could be harmful.

RL Training Infrastructure

RL training is conducted on NVIDIA B200 and H100 GPU clusters using a Ray-based distributed training framework built on NeMo-RL. The algorithm is an adapted version of Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025; Shao et al., 2024). The paper does not provide detailed equations for GSPO, but references indicate it is a variant of policy gradient methods that optimizes over groups of sequences simultaneously, leveraging the relative quality of different rollouts to provide a stronger training signal than single-sequence policy gradients.

A multimodal deduplication strategy is used during the generation phase to leverage a unique multimodal tensor with rollouts associated with each prompt. This is a systems-level optimization that avoids redundant computation when the same prompt is used across multiple RL iterations.

Quantization Strategy

The paper releases model checkpoints in three formats: BF16 (baseline, 61.5 GB, 16.0 effective bits per weight), FP8 (~8.5 bpw, 32.8 GB), and NVFP4 (~4.98 bpw, 20.9 GB). The quantization strategy is asymmetric—different layers are quantized to different precisions based on their sensitivity to quantization error.

NVFP4 format: This uses FP4 E2M1 values (4-bit floating point with 2 exponent bits and 1 mantissa bit) with per-block FP8 E4M3 scales over groups of 16 elements, plus an additional per-tensor FP32 global scale. The precision assignment is:

Routed MoE experts: quantized to NVFP4. These are the bulk of the model's parameters and the primary target for aggressive quantization.
Mamba in_proj / out_proj, shared experts, attention o_proj: quantized to FP8 (per-tensor E4M3 values with per-tensor FP32 scale). These are kept at higher precision because they are used for every token and errors in these linear projections would propagate through the entire forward pass.
Remaining language-model layers: kept in BF16. These are the most sensitive parameters, likely including layer norms, embeddings, and the MoE router.
Vision and audio encoders and their MLP projectors: kept in BF16. These modality-specific components are smaller than the LLM backbone and keeping them in BF16 preserves their carefully calibrated representations with minimal memory overhead.

The KV cache uses FP8 precision, while the Mamba SSM state cache is kept at FP32 at serving time. The Mamba state cache stores the hidden state of the state-space model layers and is critical for long-sequence generation—FP32 precision prevents accumulation of quantization errors over long generation sequences.

FP8 format: Every linear layer in the language model is quantized to per-tensor E4M3 (FP8 with 4 exponent bits and 3 mantissa bits, with a per-tensor FP32 scale), with the exception of the MoE router and lm_head. The vision and audio encoders and their MLP projectors are excluded entirely, remaining in BF16. The KV cache uses FP8. This yields ~8.5 bits per weight (32.8 GB), representing a moderate compression that preserves most of the BF16 accuracy while roughly halving the model size.

The paper reports that both FP8 and NVFP4 achieve a median accuracy drop of less than 1% versus BF16 across 25 text, image, video, and audio benchmarks (Table 14). This is a strong result for such aggressive quantization (4.98 bpw is less than one-third of the original 16 bpw), and it validates the mixed-precision strategy of keeping sensitive components at higher precision while aggressively quantizing the MoE experts.

Reasoning Budget Control

The paper includes a mechanism for controlling the inference-time reasoning budget, evaluated in Section 4.5. The model operates in two modes: a base configuration with a maximum sequence length of 16,384 tokens (no explicit reasoning budget), and a reasoning-enabled configuration with a 13K reasoning budget, a 1,024-token "grace period," and the same 16,384-token maximum sequence length.

The reasoning budget operates by truncating the model's chain-of-thought reasoning trace if it exceeds the budgeted length. The 1,024-token grace period allows the reasoning trace to slightly exceed the 13K budget before truncation, preventing harsh cutoffs mid-sentence. The paper reports that this yields accuracy gains on select benchmarks (MathVista from 80.3 to 82.8, MMLongBench-Doc from 54.5 to 56.8) with no degradation on others. The hypothesized mechanism is that budget control early-terminates malformed reasoning traces with repetition loops on out-of-distribution tasks and truncates overly verbose reasoning chains for problems requiring minimal reasoning—essentially, it prevents the model from "overthinking" simple problems or getting stuck in loops on hard ones.

4. Key Insights and Innovations

The dominant framing in multimodal LLM development has been capability-first: design the most powerful vision, audio, and language components you can, integrate them, and accept whatever computational cost results. Efficiency—if considered at all—is treated as a compression or distillation step applied post-hoc, often trading away accuracy gains that the integration achieved. Nemotron 3 Nano Omni inverts this relationship. The paper's central reframing is that efficiency must be co-designed with multimodal capability from the start, and that this co-design enables capabilities (256K context, 5+ hours of audio, feature-length video) that are practically inaccessible to a capability-first architecture at the same parameter scale.

This is not merely a "smaller, faster" story. The hybrid MoE backbone (30B total, 3B active per token) is not a retrofit onto an existing dense architecture—it is the starting point around which all modality integration decisions are made. The paper demonstrates that this architecture enables processing decisions that would be unsustainable on a dense model: token-reduction techniques like Conv3D and EVS are valuable precisely because the per-token cost is already low (3B active parameters), making the total FLOPs budget manageable for sequences that push toward 256K tokens. On a dense 30B model, reducing tokens from 141K to 42K would still leave a prohibitive per-token cost; on a 3B-active model, it drops the sequence into a tractable regime. The innovation is not the individual techniques but the architectural commitment to low per-token cost as the precondition for aggressive context scaling across modalities.

The throughput comparisons in Section 4.8 make this concrete: 3× higher single-stream output token throughput than Qwen3-Omni, 9× higher throughput per GPU at fixed interactivity. These numbers are not the result of low-precision inference alone (though NVFP4 helps)—they reflect the fundamental advantage of a model designed for efficiency from the ground up, where the MoE routing, Mamba-Transformer hybrid, and token-reduction pipeline are mutually reinforcing rather than independently optimized.

Comparison to prior work: Most open-source multimodal models (LLaVA family, InternVL, prior Nemotron releases) use dense vision encoders feeding into dense or modestly-MoE language models. Qwen3-Omni—the paper's primary comparison—achieves omni-modal capability but at substantially lower throughput. The paper's contribution is not the first MoE multimodal model, but the first to make the architectural case that MoE + token reduction + staged training is the path to production-viable omni-modal understanding, and to validate that case with throughput measurements at deployment-relevant concurrency and precision levels.

Significance: This reframes the conversation from "can we build an omni-modal model?" to "can we build one that someone can actually deploy at scale?" The paper argues implicitly that the former question is already answered (yes, with sufficient resources), and that the field's bottleneck has shifted to the latter. By releasing FP4 checkpoints that preserve median accuracy to within <1% of BF16 (Table 14) at 20.9 GB—a size that fits on a single consumer GPU—the paper demonstrates that the efficiency claim is not theoretical but immediately actionable for practitioners.

Is this fundamental or incremental? Fundamentally a reframing. Individual components (MoE, Conv3D, staged training) exist in prior work. The integration and the demonstration that co-designed efficiency unlocks qualitatively different context scales (256K tokens, 5+ hours of audio) without sacrificing accuracy represents a new design paradigm, not an incremental optimization.

Innovation 2: Temporal Interleaving of Modality Tokens as the Mechanism for Joint Audio-Visual Reasoning

The paper makes a subtle but consequential design choice that the field has not systematically explored: rather than concatenating all visual tokens followed by all audio tokens (block-concatenation), Nemotron 3 Nano Omni interleaves modality tokens in temporal order during sequence construction. Section 2 states this concisely: "modality tokens are interleaved in temporal order during sequence construction to enable joint temporal reasoning across modalities."

This matters for a reason that block-concatenation obscures. When visual and audio tokens are blocked separately, a vision token from frame at time $t$ and an audio token from time $t$ may be separated by tens of thousands of intervening tokens in the transformer's input sequence. The attention mechanism must learn long-range correspondences to associate them, which becomes increasingly difficult as the temporal separation grows. Temporal interleaving places tokens from the same time window adjacently or proximally in the sequence, making cross-modal attention a local operation rather than a long-range one. The transformer's inductive bias toward local attention (stronger attention weights between nearby tokens, learned through positional encodings and training dynamics) now naturally captures cross-modal temporal alignment.

This is the architectural analog of a common insight from video understanding: fusing modalities early (at the representation level) rather than late (at the decision level) improves performance because it allows joint reasoning about temporally coincident events. The paper's interleaving achieves this early fusion at the token-sequence level rather than through a dedicated fusion module, making it architecturally simpler but potentially more effective because the transformer's self-attention can learn arbitrary cross-modal interaction patterns rather than being constrained by a hand-designed fusion architecture.

Comparison to prior work: Most multimodal LLMs concatenate modalities as separate blocks (image tokens → text tokens, or video tokens → audio tokens). This is straightforward to implement and conceptually clean (each modality's tokens form a contiguous chunk), but it forces the model to learn long-range cross-modal correspondences. Some prior work fuses modalities through cross-attention mechanisms or separate fusion modules, but these add architectural complexity and parameters. The paper's temporal interleaving is a zero-parameter solution: it changes only the sequence ordering, not the model architecture. This is elegant because it leverages the existing self-attention mechanism's capacity for modeling local dependencies without adding new components.

The paper demonstrates the effectiveness of this approach through benchmark results on DailyOmni (reasoning off: 74.5 vs. Qwen3-Omni's 71.9) and WorldSense (55.2 vs. 54.0), but the deeper significance is that it provides empirical validation of a design principle rather than just a competitive score. The principle—that sequence ordering matters for cross-modal reasoning and that temporal locality should be preserved across modalities—generalizes beyond this specific architecture to any multimodal LLM that processes time-aligned streams.

Significance: This is more a diagnostic insight than a theoretical advance. It identifies a specific mechanism (temporal interleaving) that is easy to implement, costs nothing in parameters or compute, and improves cross-modal reasoning. The paper does not ablate this choice—it does not compare temporally-interleaved sequences against block-concatenated sequences with the same data—so the causal contribution of interleaving to benchmark performance is unmeasured. However, the design choice is well-motivated and represents a concrete hypothesis about what enables joint audio-visual reasoning in transformer architectures. Future work that ablates this choice would help quantify its importance relative to other factors like data scale and encoder quality.

Is this fundamental or incremental? Incremental as an architectural mechanism—it is a sequence ordering choice, not a new component—but potentially fundamental as a design principle if future work validates that temporal interleaving consistently outperforms block-concatenation across architectures and tasks. The paper makes the case implicitly through benchmark comparisons; a direct ablation would strengthen the claim considerably.

Innovation 3: Staged Modality-Introduction Curriculum as a General Recipe for Preventing Catastrophic Forgetting

Training a model with three heterogeneous encoders (vision, audio, text) sharing a single decoder is hard for a specific, well-documented reason: gradients from different modalities can interfere destructively. If you train all parameters on all modalities simultaneously from the start, the abundant text data can dominate the gradient signal, causing the vision and audio encoders to drift toward representations that are suboptimal for their modality-specific pretraining objectives. The result is catastrophic forgetting—the model loses its pretrained capabilities in one or more modalities as it adapts to the multimodal mixture.

The paper's seven-stage SFT pipeline is a deliberate solution to this problem, but its contribution is not merely the sequence of stages. The deeper insight is that the order of modality introduction and parameter unfreezing should follow a dependency hierarchy: align each new modality to the frozen LLM backbone before integrating it with other modalities, and only unfreeze all components simultaneously after individual alignments are stable. This transforms a high-dimensional joint optimization problem into a sequence of lower-dimensional alignment problems, each of which is better conditioned because fewer parameters are trainable and the frozen components provide a stable target.

The dependency hierarchy is:

Align vision → LLM (Stage 0: projector only; Stage 1: encoder + LLM)
Align audio → LLM (Stage 2: projector only; Stage 3: encoder + LLM)
Joint omni-modal training with all components (Stage 4: 16K context)
Scale context length (Stage 5: 48K; Stage 6: 256K)

The crucial design choice is that audio alignment (Stages 2–3) happens after vision-language capabilities are established (Stage 1) but before joint training (Stage 4). This means that when audio is introduced, the LLM already has a stable understanding of how to process and reason about visual information. The audio encoder can learn to map its representations into a multimodal LLM that "speaks vision" fluently, rather than trying to simultaneously learn both the vision-to-language and audio-to-language mappings.

Comparison to prior work: Prior multimodal training recipes often train vision and language jointly from the start, or use single-stage alignment followed by joint training. The paper's staging—particularly the separation of vision alignment (Stage 0–1) and audio alignment (Stage 2–3) into distinct phases—is a concrete instantiation of a curriculum learning principle that the field has discussed but rarely implemented at this scale for three modalities. The paper demonstrates through its text-only evaluations (Table 10) that this staging works: MMLU-Pro drops only from 78.3 to 77.3, GPQA from 73.0 to 72.2—minimal degradation despite adding two entirely new modalities and 256K context. This is the strongest evidence that the curriculum prevents catastrophic forgetting.

Significance: This is a methodological contribution: a training recipe that other teams can adopt when integrating new modalities into existing multimodal LLMs. The recipe generalizes beyond this specific architecture—any team adding audio, video, point clouds, or other modalities to a vision-language model can follow a similar pattern of projector warmup → encoder unfreezing → joint training with progressive context scaling. The paper's release of training code (Megatron-Bridge) and data pipelines makes this recipe actionable.

However, the paper does not ablate the staging itself. It does not compare the seven-stage curriculum against a simpler alternative (e.g., training all modalities jointly from the start, or using fewer stages). The evidence for the curriculum's effectiveness is therefore indirect: the model achieves strong results across all modalities without catastrophic forgetting of text capabilities, which is consistent with the curriculum working as intended but does not isolate its causal contribution. A controlled experiment comparing staged vs. joint training would be valuable but is likely prohibitively expensive at this scale.

Is this fundamental or incremental? Incremental as a specific recipe (many teams use staged training), but validated at a scale and modality count that makes it a trustworthy reference point for future work. The paper's contribution is the detailed documentation of the staging, the data compositions per stage, and the hyperparameter choices—making it more of an engineering guide than a theoretical advance, but one backed by substantial empirical evidence across modalities.

Innovation 4: Task-Specific Reward Engineering for Multimodal RL as a Bridge Between Structured and Unstructured Outputs

Applying reinforcement learning to multimodal LLMs faces a verification problem that text-only RL does not. In text reasoning, a verifier can check whether the final answer matches a ground-truth string, or whether code passes unit tests. In multimodal settings, outputs can be heterogeneous: free-form text descriptions, multiple-choice answers, coordinate predictions for GUI grounding, speech transcriptions. A single reward function cannot handle this diversity; using a generic reward (e.g., a learned reward model) risks rewarding surface-level patterns rather than task success.

The paper's Image RL and Omni-RL stages address this through a suite of task-specific, rule-based verifiers that provide precise, domain-appropriate reward signals for each output type. The four verifier types—string-match, mathruler for symbolic equivalence, multiple-choice, and GUI-coordinate with distance-based decay—cover the output space without requiring a learned reward model. The ASR verifier in Omni-RL extends this further: the reward is $1 - \text{WER}$ (after text normalization), providing a continuous, fine-grained signal that captures transcription quality at the word level.

This matters because it solves a practical problem that has hindered multimodal RL: how to train a single policy across tasks with structurally different output formats without a reward model that collapses all signals into a single scalar of questionable fidelity. The solution is to let each task define its own verifier and treat the RL objective as optimizing the expected reward across a heterogeneous distribution of verifiers. This is a form of multi-task RL where the tasks differ not only in content but in the structure of successful completion.

The pass-rate filtering mechanism (retaining prompts with empirical pass rate between 0.1 and 0.9) further refines this: it ensures that the policy receives a meaningful learning signal on each prompt. Prompts with pass rates near 0 or 1 provide little gradient information—either everything is wrong (uninformative negative signal) or everything is right (no room for improvement). Filtering these out focuses RL compute on the "learnable" region of difficulty, analogous to curriculum learning but applied at the batch construction level.

Comparison to prior work: Text-only RL for LLMs (e.g., DeepSeekMath R1-style reasoning RL, Nemotron 3 Nano RL stages) typically uses a single verifier type (string match, unit tests) because the output space is more homogeneous. Multimodal RL in prior work often uses learned reward models (trained on human preferences) because rule-based verification is harder to define for open-ended visual descriptions. The paper's approach is pragmatic: it restricts the RL training to tasks where rule-based verification is possible (structured outputs, multiple choice, grounding coordinates, ASR) and designs per-task verifiers rather than attempting a one-size-fits-all reward model. This sacrifices the generality of a learned reward model but gains reliability—rule-based verifiers do not hallucinate, do not drift during training, and do not require preference data.

Significance: This is less a theoretical advance than a demonstration of feasibility. The paper shows that multimodal RL can work at scale using task-specific verifiers and pass-rate filtering, achieving improvements on reasoning benchmarks (Section 4.5 shows accuracy gains with reasoning budget control) without reward hacking or mode collapse. The significance for practitioners is that this approach is reproducible: the verifier types are standard, the pass-rate filtering threshold is tunable, and the infrastructure (NeMo-RL) is open-source. For researchers, it establishes a baseline for multimodal RL that future work can build on or compare against.

Is this fundamental or incremental? Incremental as a contribution—the individual techniques are established—but the systematic combination of heterogeneous verifiers with pass-rate filtering represents a validated engineering template for multimodal RL. The paper would be stronger with ablations showing that the per-task verifiers outperform a single learned reward model, or that pass-rate filtering improves final performance compared to training on the full prompt distribution. These comparisons are not provided, leaving the causal contribution of each design choice unquantified.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans a broad suite of benchmarks rather than a single dataset. Visual evaluations use MMMU, MathVista-Mini, MMLongBench-Doc, OCRBench, OCRBench-V2, ChartQA, AI2D, TextVQA, DocVQA, InfoVQA, OCR-Reasoning, CharXiv, TreeBench, CV-Bench, RefCOCO, ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, OSWorld, and Video-MME (Section 4.1). Audio evaluations use the OpenASR leaderboard (English subset: AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, TED-LIUM, VoxPopuli), TED-LIUM Longform, MMAU, and VoiceBench (Section 4.2). Audio-visual evaluations use DailyOmni and WorldSense (Section 4.3). Text-only evaluations use MMLU-Pro, GPQA-Diamond, LiveCodeBench v5, AIME-2025, IFBench, TauBench V2, SciCode, and AA-LCR (Section 4.4). No single dataset split is shared across the entire evaluation; each benchmark follows its own standard test or validation protocol as specified by the benchmark authors.
Base model(s). The primary model is Nemotron 3 Nano Omni (30B-A3B), built on the Nemotron 3 Nano 30B-A3B LLM backbone. Comparisons are made against Nemotron Nano V2 VL (12B, the previous Nemotron multimodal release), Qwen3-Omni (30B-A3B), and Qwen3.5-Omni (Flash, non-open-source). The text-only evaluation additionally compares against the standalone Nemotron 3 Nano 30B-A3B LLM backbone. The choice of baselines reflects the paper's positioning: Nemotron Nano V2 VL demonstrates improvement over the prior generation, Qwen3-Omni provides the closest open-source omni-modal comparison at comparable parameter scale, and Qwen3.5-Omni Flash serves as a frontier reference (though not open-source).
Metrics. The primary metric is accuracy (percentage of questions answered correctly, higher is better) for most benchmarks. Exceptions: ASR benchmarks use Word Error Rate (WER, lower is better); GUI grounding uses click accuracy or coordinate-based scoring depending on the benchmark; TextVQA uses the standard VQA accuracy metric; RefCOCO uses pointing accuracy; OSWorld uses task success rate. For text-only benchmarks, the paper reports Pass@1 average of 8 runs for AIME-2025, an average of 4 runs for GPQA-Diamond, and a single-run score for SciCode, LiveCodeBench, IFBench, and TauBench V2 (Section 4.4). MMLU-Pro is reported as a single accuracy score. For quantization and efficiency experiments, the paper additionally reports Time-To-First-Token (TTFT) in milliseconds and output tokens per second as throughput metrics.
Baselines. The paper compares against three categories of baselines. (1) Nemotron Nano V2 VL (NVIDIA et al., 2025c): the direct predecessor, a dense 12B vision-language model without native audio support, evaluated in both reasoning-off and reasoning-on modes. (2) Qwen3-Omni-30B-A3B (Xu et al., 2025): the primary open-source omni-modal competitor, evaluated in Instruct (reasoning-off) and Thinking (reasoning-on) modes. (3) Qwen3.5-Omni (Flash, non-open-source): a frontier reference model included where evaluation results are available. For efficiency comparisons (Section 4.8), Qwen3-Omni is evaluated with dynamic FP8 quantization, Nemotron Nano V2 VL in NVFP4, and Nemotron 3 Nano Omni in NVFP4. For the text-only evaluation (Table 10), the Nemotron 3 Nano 30B-A3B LLM backbone (NVIDIA et al., 2025a) serves as a baseline to measure text capability retention.
Generation budget / compute accounting. For standard accuracy evaluations (Sections 4.1–4.4), generation is performed with a maximum output length of 131,072 tokens (text-only), temperature 1.0, and top-p 1.0. The visual and audio evaluations use the VLMEvalKit framework with a vLLM backend. The reasoning budget experiments (Section 4.5) use a base configuration with maximum sequence length of 16,384 tokens, and a reasoning-enabled configuration with a 13K reasoning budget, 1,024-token grace period, and the same 16,384-token maximum. The efficiency measurements (Section 4.8, Tables 12–13) use a controlled setup: single NVIDIA B200 GPU, vLLM nightly as of 2026-04-19, EVS at 50%, text input sequence length 50 and output sequence length 8,000. Multi-document workload contains 32 images at 1024×1536 resolution; long-video workload contains 512 frames at 512×512 resolution. For the Conv3D and EVS ablation (Tables 12–13), each accuracy column reports per-benchmark scores at 128 and 256 sampled frames with reasoning off, averaged across three runs with identical settings. TTFT is averaged across five concurrency-1 aiperf runs against a synthetic 512-frame, 512×512 video at 30 fps.
Cross-validation / statistical protocol. The paper does not employ cross-validation for the main evaluation results. Each benchmark is evaluated once per configuration (model, reasoning mode, precision format). For the reasoning budget experiments (Section 4.5), results are reported as single evaluations with and without the budget constraint. For the quantization evaluation (Table 14), each model format (BF16, FP8, NVFP4) is evaluated once across the benchmark suite, with the median accuracy drop reported as the summary statistic. For the Conv3D and EVS experiments, accuracy is averaged across three runs with identical settings, but no confidence intervals or standard deviations are reported. The absence of statistical significance testing or confidence intervals is a limitation—the paper reports point estimates throughout without quantifying uncertainty, which makes small differences (e.g., 0.5–1.0 percentage point gaps) difficult to interpret as genuine improvements versus sampling noise.

Main Quantitative Results

Vision-Language Evaluation Results

Table 7 presents the comprehensive vision evaluation comparing Nemotron 3 Nano Omni against Nemotron Nano V2 VL, Qwen3-Omni, and Qwen3.5-Omni. The headline finding is that Nemotron 3 Nano Omni with reasoning off outperforms Nemotron Nano V2 VL across all vision benchmarks, often by substantial margins, and matches or exceeds Qwen3-Omni on document understanding, OCR, and GUI grounding tasks.

On document understanding and OCR benchmarks—the paper's stated areas of strength—the improvements over Nemotron Nano V2 VL are dramatic. MMLongBench-Doc rises from 32.1 to 46.1 (reasoning off), a 14-point absolute gain (~44% relative improvement). OCRBench-V2 EN increases from 62.0 to 65.8, and ZH from 44.2 to 52.0. OCR-Reasoning shows a striking jump with reasoning enabled: 54.14 versus Nemotron Nano V2 VL's 21.0 (reasoning on) and versus Qwen3-Omni's 33.9. CharXiv-Reasoning questions improve from 41.7 to 49.1 (reasoning off) and to 63.6 with reasoning on, substantially exceeding Qwen3-Omni's 41.3 (Instruct) and 61.1 (Thinking).

On GUI understanding benchmarks, the gap is even more pronounced. ScreenSpot rises from 39.4 to 90.3 (reasoning off)—a more than 2× improvement over the predecessor, and versus Qwen3-Omni's 42.5. ScreenSpot-v2 shows the same pattern: 93.4 versus 41.7 for Nemotron Nano V2 VL and 42.8 for Qwen3-Omni. ScreenSpot-Pro reaches 59.3 versus 4.8 for the predecessor and 5.5 for Qwen3-Omni, though Qwen3.5-Omni Flash achieves a comparable 59.7. The OSWorld benchmark (available only with reasoning on) shows 47.4 for Nemotron 3 Nano Omni versus 11.1 for Qwen3-Omni and 29.0 for Qwen3.5-Omni. These GUI results are particularly notable because they represent the model's potential for agentic computer use—a key application highlighted in the paper's introduction.

On STEM reasoning, the improvements are more modest. MMMU (val) increases from 55.3 to 55.2 (reasoning off, essentially flat) and to 70.8 with reasoning on, compared to Qwen3-Omni's 67.8 (Thinking) and Qwen3.5-Omni's 76.9. MathVista-Mini rises from 69.0 to 71.9 (reasoning off) and to 82.8 (reasoning on), exceeding Qwen3-Omni's 75.5 (Thinking) and roughly matching Qwen3.5-Omni's 82.9.

On standard VQA and chart benchmarks, results are strong but the gap over prior work narrows. ChartQA (Test) reaches 89.9 (reasoning off) versus 89.8 for Nemotron Nano V2 VL (essentially tied) and 84.9 for Qwen3-Omni. DocVQA (Test) scores 93.3 versus 94.7 for the predecessor—a slight regression—but rises to 95.6 with reasoning on. TextVQA (Val) at 85.1 (reasoning off) is marginally below Nemotron Nano V2 VL's 85.4 but substantially above Qwen3-Omni's 76.1. InfoVQA (Test) at 83.6 (reasoning off) improves from 79.4 on the predecessor and 80.4 on Qwen3-Omni.

On visual grounding and spatial reasoning, the model shows mixed results. TreeBench reaches 43.7 (reasoning off) versus 38.5 for the predecessor. CV-Bench scores 84.2 versus 81.0. RefCOCO achieves 80.6 (reasoning off) and a notably strong 90.5 with reasoning on, though Qwen3.5-Omni reaches 92.6.

On video understanding, VideoMME (without subtitles) reaches 70.8 (reasoning off), improving from 66.0 on Nemotron Nano V2 VL, and reaches 72.2 with reasoning on versus Qwen3-Omni's 70.5 (Thinking) and Qwen3.5-Omni's 77.0.

Audio Evaluation Results

Table 8 presents the audio evaluation results. Nemotron 3 Nano Omni achieves leading ASR performance: on the OpenASR English subset, the model averages 5.95 WER (lower is better) versus Qwen3-Omni's 6.55, with particularly strong performance on SPGISpeech (1.98 vs. 3.69) and VoxPopuli (5.6 vs. 8.26). The model's WER is higher on AMI (11.09 vs. 12.52) and Earnings22 (11.27 vs. 12.30) compared to Qwen3-Omni, but lower on GigaSpeech (9.66 vs. 8.49) and TED-LIUM (3.44 vs. 2.38). On long-form ASR (TED-LIUM Longform), Nemotron 3 Nano Omni achieves 3.11 WER versus Qwen3-Omni's 2.4—a modest regression.

On audio understanding (MMAU), the results are competitive but not dominant. MMAU average is 74.6 (reasoning off) versus Qwen3-Omni's 77.5 and Qwen3.5-Omni's 80.4. Broken down by subcategory: Music 74.2, Audio 76.9, Speech 72.8. The paper does not provide subcategory breakdowns for Qwen3-Omni on MMAU, making it unclear whether the gap is uniform across subcategories.

On voice interaction and reasoning (VoiceBench), Nemotron 3 Nano Omni achieves 89.4 average with reasoning on, marginally exceeding Qwen3-Omni's 88.8 and Qwen3.5-Omni's 87.8. The strongest subcategory results are IFEval (88.7 vs. 80.6), BBH (91.1 vs. 88.9), AdvBench (100 vs. 97.2), and CommonEval (91.3 vs. 90.5). The model scores lower than Qwen3-Omni on AlpacaEval (95.0 vs. 96.4), OpenBookQA (93.0 vs. 94.3), MMSU (82.3 vs. 83.0), and SD-QA (71.4 vs. 78.1).

Audio-Visual Evaluation Results

Table 9 presents the results on the two audio-visual benchmarks that are central to the paper's omni-modal claims. On DailyOmni (audio-visual QA with temporal reasoning across six tasks, 1,197 multiple-choice questions), Nemotron 3 Nano Omni achieves 74.5 (reasoning off) and 74.1 (reasoning on), versus Qwen3-Omni's 71.9 (Instruct) and 73.6 (Thinking). The reasoning-off advantage over Qwen3-Omni Instruct is 2.6 percentage points. Qwen3.5-Omni Flash reaches 81.8, a substantial 7.3-point gap over Nemotron 3 Nano Omni.

On WorldSense (large-scale omni-modal benchmark with 1,662 long-context videos and 3,172 multiple-choice questions across 26 tasks), Nemotron 3 Nano Omni achieves 55.2 (reasoning off) and 55.4 (reasoning on), versus Qwen3-Omni's 54 (Instruct)—with no Thinking result available for Qwen3-Omni on this benchmark. Qwen3.5-Omni Flash reaches 57.8. The margin over Qwen3-Omni is narrow (1.2–1.4 points) and the reasoning-on configuration provides essentially no improvement over reasoning-off (55.4 vs. 55.2), which is notable given that other benchmarks show meaningful gains from enabling reasoning.

Text-Only Evaluation Results

Table 10 compares Nemotron 3 Nano Omni against its LLM backbone (Nemotron 3 Nano 30B-A3B) and Qwen3-Omni on selected text-only benchmarks. The key claim—that multimodal training preserves text reasoning capability—is supported by the comparison against the LLM backbone. MMLU-Pro drops from 78.3 to 77.3 (~1 point, or ~1.3% relative). GPQA (no tools) drops from 73.0 to 72.2 (~0.8 point). However, larger gaps appear on other benchmarks: LiveCodeBench drops from 68.3 to 63.2 (~5.1 points), AIME-2025 drops from 89.1 to 82.1 (~7 points), and SciCode drops marginally from 33.3 to 32.0. The IFBench (prompt) score actually increases from 71.5 to 74.2, and AA-LCR increases from 35.9 to 41.0, suggesting that multimodal training may improve some instruction-following capabilities.

Compared to Qwen3-Omni, Nemotron 3 Nano Omni substantially outperforms on MMLU-Pro (77.3 vs. 61.6), AIME-2025 (82.1 vs. 73.7), and is roughly tied on GPQA (72.2 vs. 73.1).

Reasoning Budget Control Results

Table 11 examines the effect of applying an inference-time reasoning budget. Under the base configuration (no explicit reasoning budget, maximum sequence length 16,384), and the reasoning-enabled configuration (13K reasoning budget, 1,024-token grace period, 16,384 maximum sequence length), the paper reports gains on several benchmarks: MathVista-Mini improves from 80.3 to 82.8, MMLongBench-Doc from 54.5 to 56.8, CharXiv-RQ from 61.8 to 64.0, RefCOCO from 90.4 to 90.6, and VideoMME from 67.5 to 70.3. DocVQA (Val) is essentially flat (95.3 vs. 95.2). The paper hypothesizes that these gains arise from "early termination of malformed reasoning traces with repetition loops on out-of-distribution tasks, as well as truncation of overly verbose reasoning chains for problems requiring minimal or straightforward reasoning" (Section 4.5). However, no analysis of the reasoning traces themselves is provided to support this mechanism claim.

Efficiency Results: Conv3D and EVS

Tables 12 and 13 present the core efficiency-accuracy tradeoff results for video processing. Table 12 compares four configurations—baseline (no Conv3D, no EVS), EVS alone, Conv3D alone, and Conv3D + EVS combined—for both BF16 and NVFP4 checkpoints, with EVS fixed at pruning rate $q = 0.5$ .

For BF16 at 256 frames, the baseline achieves 66.72 average accuracy across the four benchmarks (DailyOmni, LongVideoBench, Video-MME, WorldSense) with TTFT of 7,969 ms. Adding EVS alone reduces TTFT to 6,452 ms (−19%) while accuracy actually increases slightly from 66.72 to 66.92 (+0.20 points). Adding Conv3D alone reduces TTFT to 5,984 ms (−25%) with a very small accuracy cost: 66.72 to 66.64 (−0.08 points). Combining Conv3D + EVS yields TTFT of 5,313 ms (−33% vs. baseline) with accuracy of 66.32 (−0.40 points). The paper reports that a 512-frame video produces ~141K input tokens at baseline, drops to ~75K with Conv3D enabled (−47%), and drops further to ~42K with Conv3D + EVS at $q = 0.5$ (−70%).

For NVFP4, the baseline accuracy is lower (65.26 vs. 66.72 in BF16) due to quantization, but the efficiency gains follow the same pattern. Conv3D + EVS achieves 65.00 accuracy (at 256 frames) with TTFT of 5,083 ms, compared to 6,885 ms at baseline—a 26% reduction.

Table 13 sweeps the EVS pruning rate $q$ from 0.5 to 0.95 on BF16 with Conv3D enabled. At $q = 0.5$ , average accuracy is 66.32 (256 frames). At $q = 0.6$ , accuracy rises slightly to 66.56, but then declines: 65.82 at $q = 0.7$ , 65.47 at $q = 0.8$ , 63.47 at $q = 0.9$ , and 61.36 at $q = 0.95$ . The key finding is that accuracy is essentially flat through $q = 0.7$ (−0.82 points or ~1.2% relative to baseline without EVS at 66.64), with LongVideoBench being the most sensitive to aggressive pruning—dropping from 67.20 at no-EVS to 65.40 at $q = 0.7$ and to 57.90 at $q = 0.95$ . TTFT improves monotonically: 5,984 ms at no-EVS, 5,124 ms at $q = 0.7$ (−14%), and 4,804 ms at $q = 0.95$ (−20%).

Quantization Accuracy Results

Table 14 reports the accuracy of BF16, FP8, and NVFP4 checkpoints across 25 text, image, video, and audio benchmarks. The key summary statistic: the median accuracy drop across 11 non-ASR benchmarks is less than 1% for both FP8 and NVFP4 versus BF16. Specifically, the mean drop is −0.37 points for FP8 and −0.40 points for NVFP4. Individual benchmarks show mixed patterns: some benchmarks show slight regressions (MathVista-Mini drops from 71.90 to 71.05 in FP8 and 71.30 in NVFP4; CV-Bench actually improves from 84.20 to 85.62 in FP8 and 85.27 in NVFP4, which may reflect run-to-run variance or a calibration effect). The ASR benchmarks are essentially unchanged: TED-LIUM Longform WER fluctuates from 3.11 (BF16) to 3.12 (FP8) to 3.04 (NVFP4); the OpenASR average across 8 short-form datasets is 5.95 across all three precisions—a remarkable stability for 5× compression (NVFP4 at 4.98 bpw vs. BF16 at 16.0 bpw).

Inference Efficiency Results

Section 4.8 provides throughput and latency measurements on NVIDIA B200. With NVFP4 precision, Nemotron 3 Nano Omni achieves up to 7.5× the output token throughput at iso-interactivity compared to BF16: 18,200 tokens/second versus 2,400 tokens/second at 150 tokens/second/user on a single-image reasoning use case.

For low-latency single-stream inference (concurrency = 1), the model reaches more than 500 output tokens/second, approximately 2.4–2.9× faster than Qwen3-Omni (175–210 tokens/second depending on input size) and 2× faster than Nemotron Nano V2 VL (250 tokens/second). For multi-document workload (32 images at 1024×1536), time-to-first-token is approximately 1.3 seconds versus more than 2.5 seconds for Qwen3-Omni.

At maximum concurrency on a single B200, the model reaches 5,000 output tokens/second on the multi-document workload. At an iso-interactivity target of 50 output tokens/second per user, the deployment provides 9× higher output throughput than Qwen3-Omni on long-video workloads and 7.5× higher on multi-document workloads. Compared to Nemotron Nano V2 VL, throughput is 3× higher at the same interactivity target.

Ablation Studies and Robustness Checks

Conv3D vs. EVS vs. Combined (Table 12): The comparison of four configurations—baseline, EVS-only, Conv3D-only, and Conv3D+EVS—reveals that the two mechanisms are complementary but not purely additive. On BF16 at 256 frames, baseline accuracy is 66.72 with TTFT 7,969 ms. Conv3D alone provides 66.64 accuracy (−0.08 points) with TTFT 5,984 ms (−25%). EVS alone provides 66.92 accuracy (+0.20 points, no accuracy cost within measurement noise) with TTFT 6,452 ms (−19%). Combined Conv3D+EVS yields 66.32 accuracy (−0.40 points) with TTFT 5,313 ms (−33%). The superadditive accuracy cost (0.40 > 0.08 + 0) suggests mild interaction between the two mechanisms: Conv3D's temporal fusion may make EVS's dissimilarity computation noisier, leading to slightly more aggressive pruning of informative tokens.

EVS Pruning Rate Sensitivity (Table 13): Sweeping $q$ from 0.5 to 0.95 reveals a threshold effect. Accuracy is essentially flat from $q = 0$ (no EVS) through $q = 0.7$ , with LongVideoBench showing the first signs of degradation at $q = 0.7$ (65.40 vs. 67.20 at no-EVS). Beyond $q = 0.7$ , degradation accelerates: average accuracy at $q = 0.8$ drops to 65.47 (−1.17 points from no-EVS), at $q = 0.9$ to 63.47 (−3.17 points), and at $q = 0.95$ to 61.36 (−5.28 points). This non-linear response suggests a "graceful degradation" region up to $q \approx 0.7$ followed by a collapse beyond, likely because at high pruning rates, structurally important tokens (background context, scene-establishing frames) begin to be dropped, breaking the model's ability to construct a coherent temporal narrative. The finding that $q = 0.7$ is a practical sweet spot—14% TTFT reduction at minimal accuracy cost—is a concrete deployment recommendation.

Quantization Robustness (Table 14): The comparison of BF16, FP8, and NVFP4 across 25 benchmarks tests whether the mixed-precision quantization strategy preserves accuracy. The finding that NVFP4 at 4.98 effective bits per weight achieves median accuracy within <1% of BF16 is a strong result, but the table reveals benchmark-specific sensitivity. MathVista-Mini drops from 71.90 (BF16) to 71.05 (FP8) and 71.30 (NVFP4)—~0.6–0.9 point drops. CV-Bench actually improves from 84.20 (BF16) to 85.62 (FP8) and 85.27 (NVFP4)—~1.1–1.4 point increases that exceed plausible run-to-run noise and may indicate a calibration improvement from reduced-precision activations. The ASR benchmarks are remarkably stable across precisions (TED-LIUM Longform WER: 3.11, 3.12, 3.04), suggesting that the audio pipeline—kept in BF16 for encoders and projectors—is well-isolated from the LLM quantization. The mixed-precision strategy (MoE experts to NVFP4, Mamba/attention projections to FP8, sensitive layers in BF16) appears validated by these results.

Reasoning Budget Effect (Table 11): The comparison of with and without an explicit reasoning budget on six benchmarks shows mixed effects. MathVista-Mini improves from 80.3 to 82.8 (+2.5 points), MMLongBench-Doc from 54.5 to 56.8 (+2.3 points), CharXiv-RQ from 61.8 to 64.0 (+2.2 points), and VideoMME from 67.5 to 70.3 (+2.8 points). However, the mechanism claimed—early termination of malformed reasoning traces and truncation of overly verbose chains—is not demonstrated through any analysis of the reasoning traces themselves. The improvement could alternatively arise from the budget forcing the model to be more concise (producing shorter but equally correct answers) or from preventing the model from confusing itself with excessively long chains of thought. Without trace-level analysis, the causal mechanism remains speculative.

Text Capability Retention (Table 10, implicit ablation): While not a controlled ablation, the comparison of Nemotron 3 Nano Omni against its LLM backbone implicitly tests whether seven-stage multimodal SFT + RL causes catastrophic forgetting. MMLU-Pro drops only 1.0 point (78.3 → 77.3), but AIME-2025 drops 7.0 points (89.1 → 82.1), LiveCodeBench drops 5.1 points (68.3 → 63.2), and SciCode drops 1.3 points (33.3 → 32.0). The larger drops on math and code reasoning benchmarks suggest that multimodal training may partially overwrite specialized reasoning capabilities, despite the general knowledge retention on MMLU-Pro and GPQA. This is not a "catastrophic" forgetting (the model remains functional), but it is a measurable regression that the paper does not directly comment on.

Vision Encoder CPE Layers Kept in Eval Mode (Section 3.1.8, implicit ablation): The paper mentions that the vision encoder's CPE layers are kept in evaluation mode during Stages 1, 4, and 5. No ablation is provided to demonstrate that this stabilization is necessary or beneficial, but the design choice reflects a hypothesis that CPE layers—which implement position-dependent transformations—are particularly prone to training instability during fine-tuning. The strong vision benchmark results are consistent with this hypothesis, but a controlled comparison against a configuration where CPE layers are trainable would be needed to establish causality.

Critical Assessment

Claim 1: "Nemotron 3 Nano Omni delivers consistent accuracy improvements over Nemotron Nano V2 VL across all modalities."

Largely supported, but with qualifications. Table 7 shows universal improvements on vision benchmarks when comparing reasoning-off modes: 43 of the ~50 reported metric comparisons show improvement, with the largest gains on document understanding (MMLongBench-Doc: +14 points), OCR (OCR-Reasoning: +33 points with reasoning on), and GUI grounding (ScreenSpot: +51 points). However, a few metrics are essentially flat or show slight regression: MMMU reasoning-off is essentially unchanged (55.2 vs. 55.3), ChartQA is tied (89.9 vs. 89.8), TextVQA drops marginally (85.1 vs. 85.4), and DocVQA reasoning-off drops marginally (93.3 vs. 94.7). The audio evaluation (Table 8) shows generally superior ASR performance (OpenASR average 5.95 vs. Qwen3-Omni's 6.55) but slightly lower on long-form ASR (3.11 vs. 2.4) and MMAU audio understanding (74.6 vs. 77.5). The improvements are real and substantial on the metrics the paper emphasizes (document understanding, GUI, OCR), but the claim of "consistent" improvement is slightly overstated—there are pockets of regression or stasis, particularly on standard VQA benchmarks and certain audio subcategories. The more precise characterization is "substantial improvements on document, GUI, and multimodal reasoning benchmarks, with modest improvements or maintenance on standard VQA and STEM benchmarks, and mixed results on audio understanding."

Claim 2: "It delivers leading results in real-world document understanding, long audio-video comprehension, and agentic computer use."

Supported for document understanding and GUI/agentic tasks, less so for long audio-video comprehension. Document understanding: Nemotron 3 Nano Omni reasoning-off achieves MMLongBench-Doc 46.1 versus Qwen3-Omni's 38.0—a clear lead. OCRBench-V2 EN 65.8 versus 54.8 for Qwen3-Omni. CharXiv-RQ 49.1 versus 41.3. These are unambiguous wins. Agentic computer use: ScreenSpot 90.3 versus 42.5, ScreenSpot-v2 93.4 versus 42.8, OSWorld 47.4 versus 11.1—these are enormous gaps that likely reflect architectural choices (dynamic resolution for GUI screenshots) rather than mere data or training differences. However, long audio-video comprehension is a weaker story. DailyOmni: 74.5 versus Qwen3-Omni's 71.9 (Instruct)—a 2.6-point lead, but versus Qwen3.5-Omni Flash at 81.8, a 7.3-point gap. WorldSense: 55.2 versus 54.0—only 1.2 points. These are competitive results but not "leading" in the sense of clearly outperforming all alternatives; Qwen3.5-Omni Flash substantially exceeds Nemotron 3 Nano Omni on both metrics. The claim should be qualified as "leading on document and GUI benchmarks among open-source models, with competitive but not dominant audio-visual reasoning."

Claim 3: "Innovative multimodal token-reduction techniques deliver substantially lower inference latency and higher throughput than other models of similar size."

Supported with quantified evidence. The 3× single-stream throughput advantage over Qwen3-Omni, 9× throughput per GPU at iso-interactivity, and the TTFT comparisons (1.3s vs. >2.5s for Qwen3-Omni on multi-document workload) are concrete and well-measured. The Conv3D+EVS ablation (Tables 12–13) demonstrates the contribution of each token-reduction technique to latency reduction, with EVS providing ~14% TTFT reduction at $q = 0.7$ with essentially flat accuracy. However, the throughput comparisons are made against Qwen3-Omni evaluated with dynamic FP8 quantization, which is appropriate but means the comparison is against a model that may not be maximally optimized for throughput (Qwen3-Omni's serving configuration might not be latency-tuned). Additionally, the 9× throughput claim is at a specific interactivity target (50 tokens/s/user) and workload (long-video)—these conditions should be carried forward when citing the number. The core efficiency claim holds, but the specific multipliers are workload- and configuration-dependent.

Claim 4: "The model retains the text reasoning performance of the Nemotron 3 Nano 30B-A3B backbone."

Partially supported. Table 10 shows that MMLU-Pro and GPQA are largely preserved (1.0 and 0.8 point drops, respectively), and IFBench and AA-LCR actually improve. However, AIME-2025 drops 7.0 points (from 89.1 to 82.1) and LiveCodeBench drops 5.1 points (from 68.3 to 63.2). Qualitatively, the model retains general academic knowledge (MMLU-Pro) and graduate-level scientific reasoning (GPQA), but competitive math and coding capabilities are notably degraded. The paper's text (Section 4.4) frames the goal as maintaining text benchmarks "while adding vision and audio understanding capabilities," but characterizing the outcome as "retaining" text performance is only accurate for knowledge-intensive benchmarks—for reasoning-intensive benchmarks, there is measurable regression. A more precise claim would acknowledge that some specialized reasoning capabilities degrade during multimodal training, consistent with a capacity tradeoff.

Missing experiments and analyses that would strengthen the paper:

Temporal interleaving ablation. The paper claims temporal interleaving of modality tokens enables joint audio-visual reasoning, but never compares this against block-concatenation. A controlled experiment on DailyOmni or WorldSense comparing interleaved vs. blocked sequences would quantify whether this design choice matters, and by how much.
Staged curriculum ablation. The seven-stage SFT pipeline is presented as a recipe for stable training, but no comparison against a simpler curriculum (e.g., fewer stages, or joint training from the start) is provided. While the computational cost of such an ablation at this scale is prohibitive, the absence means the necessity of the seven-stage structure is asserted rather than demonstrated empirically.
Trace-level analysis for reasoning budget. The paper claims that reasoning budget gains come from terminating malformed traces and truncating verbose chains, but provides no analysis of the actual reasoning traces to support this. A comparison of trace lengths, repetition patterns, and error types with and without the budget would transform this from a speculative mechanism to an empirically grounded one.
Statistical significance on small benchmarks. Several reported improvements are in the 0.5–2.0 point range on benchmarks with 500–2,000 test examples. Without confidence intervals or multiple runs, these differences could be within sampling noise. The paper reports averaging across 3 runs for the EVS experiments but does not report standard deviations; for the main benchmark results (Table 7), no run-to-run variance is reported at all.
Breakdown of which audio-visual tasks benefit from temporal interleaving. DailyOmni reports an aggregate score across six tasks; WorldSense across 26 tasks. The paper does not provide per-task breakdowns, making it impossible to identify which specific types of cross-modal reasoning (temporal alignment? event understanding? causal reasoning?) drive the improvements. This granularity would strengthen the claim that temporal interleaving specifically enables temporally-aligned cross-modal reasoning.
Omni-RL ablation. The paper describes a 120K-prompt Omni-RL training stage with task-specific verifiers and pass-rate filtering, but never compares the final checkpoint against a version without Omni-RL (only SFT + earlier RL stages). The causal contribution of Omni-RL to benchmark performance is therefore unmeasured—the model might achieve similar results without this stage, or the stage might be critical for specific benchmarks (e.g., DailyOmni, WorldSense) that explicitly require cross-modal reasoning.
Comparison of FP4 against other quantization methods. The paper demonstrates that NVFP4 preserves accuracy to within <1% of BF16, but does not compare against alternative quantization schemes (e.g., standard INT4, AWQ, GPTQ). While the mixed-precision strategy is described in detail, the absence of comparisons makes it unclear whether the NVFP4 format itself provides benefits beyond what a well-tuned INT4 quantization would achieve on the same hardware.

Where claims hold and where they don't:

The claim of leading document understanding holds clearly against open-source alternatives (Qwen3-Omni) but is not tested against proprietary systems (GPT-4o, Claude 3.5, Gemini 2.0 Flash). The model likely leads among open-source, efficiently-deployable models, which is a meaningful but narrower category.
The claim of 3× to 9× throughput advantage holds under the specific measurement conditions (NVIDIA B200, vLLM nightly, NVFP4 precision, specific workloads) but may not generalize to different hardware (H100, A100), different serving frameworks, or different workload characteristics (shorter sequences, fewer images, different image resolutions).
The claim of retaining text reasoning performance holds for knowledge-intensive benchmarks (MMLU-Pro, GPQA) but not for reasoning-intensive benchmarks (AIME, LiveCodeBench), and the paper should distinguish between these categories rather than treating text performance as a monolith.
The claim of native audio support without sacrificing vision or text holds for vision (consistent gains over Nemotron Nano V2 VL) and general knowledge (MMLU-Pro preserved), but shows some regression on math and code reasoning, and on audio-only understanding (MMAU behind Qwen3-Omni by 2.9 points). This is a net tradeoff rather than a pure Pareto improvement.

Overall assessment of the experimental evidence: The experiments are comprehensive in breadth (covering ~30 benchmarks across four modalities) and provide convincing evidence for the core efficiency claims and the document/GUI performance advantages. The benchmarking is transparent (all evaluation frameworks and configurations are named) and the comparisons against prior work are on equal footing (same evaluation harness, vLLM backend). The efficiency measurements are particularly strong—the Conv3D+EVS sweep across pruning rates with simultaneous accuracy and TTFT reporting is exactly the kind of analysis that practitioners need for deployment decisions. However, the evaluation is weaker on statistical rigor (no confidence intervals, single runs for most benchmarks), mechanism validation (no ablations of key architectural claims like temporal interleaving or staged curriculum), and RL contribution measurement (no SFT-only baseline for comparison against the RL-enhanced model). The paper demonstrates that the model works well; it does not fully demonstrate why each design choice matters.

6. Limitations and Trade-offs

No Ablation of the Staged Training Curriculum

The assumption or constraint. The seven-stage SFT pipeline (Sections 3.1.1–3.1.7) is presented as a carefully designed curriculum where modalities are progressively introduced and context length is gradually extended, with the explicit goal of promoting "stable cross-modal alignment and mitigate catastrophic forgetting while improving multi-modal understanding." The paper describes the rationale for each stage transition—projector warmup before encoder unfreezing, audio alignment after vision capabilities are established, context scaling only after joint omni-modal training is stable—but never tests whether this specific staging is necessary for the achieved results. The paper does not compare the seven-stage curriculum against a simpler alternative (e.g., three stages: vision alignment → audio alignment → joint training with all context lengths, or even single-stage joint training on all modalities at 256K context). The claim that the curriculum prevents catastrophic forgetting rests entirely on the observation that the final model retains most text capabilities (Table 10), without isolating the curriculum's causal contribution versus other factors like data quality, model scale, or the MoE architecture's inherent resistance to interference.

The consequence. For practitioners who want to train their own omni-modal models by integrating new modalities into existing vision-language systems, the paper does not provide guidance on which stages are essential versus which are nice-to-have. If the seven-stage curriculum is over-engineered—if, for example, three stages would achieve comparable results—teams following the recipe would waste substantial compute on unnecessary intermediate stages. Conversely, if the curriculum is genuinely necessary, the paper provides no evidence of why it works (e.g., gradient interference measurements, loss landscape analysis, intermediate checkpoint evaluations) that would help practitioners debug training failures or adapt the recipe to new modalities beyond audio. The absence of stage-by-stage intermediate evaluations is particularly limiting: we do not know whether the model after Stage 1 (vision SFT) already achieves most of its final vision performance, whether Stage 4 (joint omni SFT) is the critical turning point for audio-visual reasoning, or whether Stage 6 (256K context) provides benefits proportional to its training cost (~34B tokens on only 623K samples).

What evidence exists in the paper. The only indirect evidence for the curriculum's effectiveness is the comparison of the final model against its LLM backbone (Table 10), showing that MMLU-Pro drops only 1.0 point (78.3 → 77.3) and GPQA drops 0.8 points (73.0 → 72.2). However, this comparison is confounded by all other training choices—data, RL, architecture—and does not isolate the staging effect. The paper also notes that the vision encoder's CPE layers are kept in eval mode during Stages 1, 4, and 5 to "stabilize training" (Section 3.1.8), implying that training instability is a real concern, but provides no ablation showing that this stabilization is needed or that the staging prevents it. No intermediate checkpoint evaluations are reported for any SFT stage, making it impossible to trace how capabilities develop across the curriculum. The paper does not acknowledge this as a limitation.

Mitigation status. Not addressed. The paper presents the seven-stage curriculum as a methodological contribution without empirical validation of its necessity or efficiency relative to simpler alternatives. This is an honest description of what was done rather than an empirically validated recipe that others should follow. Future work that ablates the number of stages, the order of modality introduction, or the context-length scaling schedule would clarify which elements of the curriculum are load-bearing and which are incidental.

Difficulty Estimation Cost for Omni-RL Pass-Rate Filtering Is Unaccounted For

The assumption or constraint. The Omni-RL stage (Section 3.2.4) applies pass-rate filtering to construct its 120K-prompt training corpus: prompts are retained only if the initial policy checkpoint achieves an empirical pass rate between 0.1 and 0.9 (with stricter 0.3–0.7 bands for AudioQA). Computing this pass rate requires generating 8 rollouts per prompt from the initial policy checkpoint (the same filtering protocol used in Image RL, Section 3.2.3), then evaluating each rollout against the task-specific verifier. For a corpus of 120K retained prompts, this implies generating and evaluating on the order of 960K rollouts (8 × 120K)—plus additional rollouts for the prompts that were filtered out, since the total candidate pool was larger than the retained set. The paper does not account for the cost of this filtering anywhere in its compute budget or training efficiency analysis.

The consequence. The pass-rate filtering cost represents a hidden compute overhead that precedes the actual RL training. For long-context omni-modal prompts (videos with audio, multi-page documents), generating 8 rollouts per prompt is extremely expensive: a single rollout on a video prompt might produce thousands of output tokens, and the verifier evaluation for GUI grounding or ASR requires additional computation beyond token generation. The Omni-RL stage's reported effectiveness therefore implicitly depends on a compute investment (the filtering rollouts) that is not amortized or reported. In a production setting where the initial policy checkpoint changes frequently (e.g., after each round of RL or SFT), this filtering cost would need to be paid repeatedly—potentially making it a dominant term in the total RL compute budget. The paper's efficiency narrative—3× to 9× throughput advantages over competitors—applies to inference serving, not to the training pipeline where this overhead lives, creating an asymmetry between the deployment efficiency claims and the training efficiency reality.

What evidence exists in the paper. The paper explicitly describes the pass-rate filtering protocol in Section 3.2.3: "we apply pass-rate filtering using 8 rollouts per prompt from the initial policy checkpoint, retaining only prompts whose empirical pass rate is below 0.8." Section 3.2.4 states that Omni-RL applies the same approach (with different thresholds) to its 120K-prompt corpus. However, the compute cost of generating these 8 rollouts per prompt is never quantified. The paper reports the SFT training costs in terms of total tokens (Table 1: 466.9B tokens across all stages) and GPU node counts (Table 6), but provides no analogous accounting for the RL data generation and filtering overhead. The global RL batch size is 4,096 with 16 rollouts per prompt and micro-batch size 1 (Section 3.2.5)—these are the rollouts used during RL training itself, distinct from the 8 filtering rollouts generated beforehand.

Mitigation status. Not addressed. The paper does not acknowledge the filtering cost, does not report the number of candidate prompts that were considered and filtered out versus retained, and does not discuss whether this cost is amortized across RL rounds or paid anew each time. This is a significant omission in a paper whose central thesis is efficiency. A practical mitigation—using fewer rollouts for filtering, or reusing rollouts from the RL training phase for difficulty estimation—is not explored.

Hard Audio-Visual Reasoning Problems Remain Essentially Unsolved

The assumption or constraint. The paper demonstrates strong performance on document understanding and GUI grounding benchmarks (Table 7: MMLongBench-Doc 46.1, ScreenSpot 90.3, OSWorld 47.4), but the audio-visual reasoning benchmarks—which are the most direct test of the model's claimed omni-modal capability—tell a different story. On WorldSense, Nemotron 3 Nano Omni achieves 55.2 with reasoning off and 55.4 with reasoning on (Table 9), versus Qwen3-Omni's 54.0. This is a 1.2–1.4 point gap on a benchmark with 3,172 questions across 26 tasks—well within the range of sampling noise given the test set size. On DailyOmni, the model achieves 74.5 (reasoning off) and 74.1 (reasoning on), versus Qwen3-Omni's 71.9 (Instruct) and 73.6 (Thinking)—a 2.6-point advantage over Qwen3-Omni Instruct, but a 7.3-point gap behind Qwen3.5-Omni Flash at 81.8. More tellingly, enabling reasoning does not improve performance on these benchmarks: DailyOmni drops from 74.5 to 74.1 with reasoning on; WorldSense improves only from 55.2 to 55.4. This is in stark contrast to vision benchmarks like MMMU (55.2 → 70.8 with reasoning), MathVista (71.9 → 82.8), or OCR-Reasoning (22.2 → 54.14), where reasoning mode provides dramatic gains.

The consequence. The model's omni-modal reasoning capability—its ability to jointly reason about temporally aligned audio and visual streams—appears to be near the ceiling of what the current training recipe can achieve, with minimal benefit from additional inference-time reasoning. The WorldSense score of ~55% on a 26-task benchmark suggests that roughly half of cross-modal reasoning questions are answered correctly, and this proportion does not meaningfully improve with chain-of-thought reasoning. This implies a fundamental capability bound: the model can handle simple cross-modal associations (matching a sound to its visual source, recognizing that a speaker's voice corresponds to their on-screen presence) but struggles with complex cross-modal inference (causal reasoning about audio-visual events, long-range temporal dependencies spanning minutes of video). For practitioners building applications that require reliable audio-visual understanding—video summarization with dialogue, meeting transcription with speaker diarization, content moderation for video platforms—the ~55% accuracy ceiling on WorldSense and the ~74% on DailyOmni represent significant failure rates that would require human-in-the-loop verification for production use.

What evidence exists in the paper. Table 9 directly shows the WorldSense and DailyOmni results, with the reasoning-on vs. reasoning-off comparison revealing the absence of a reasoning benefit. Table 7 provides the contrast: vision benchmarks like MMMU gain 15.6 points from reasoning, while the omni benchmarks gain ~0–0.2 points. The paper does not comment on this discrepancy or analyze why reasoning helps vision tasks but not audio-visual tasks. No per-task breakdown of WorldSense or DailyOmni is provided, making it impossible to identify whether specific subcategories of cross-modal reasoning (temporal alignment, event understanding, causal reasoning, sound grounding) drive the low scores or whether the model uniformly struggles.

Mitigation status. Not addressed. The paper does not acknowledge the audio-visual reasoning ceiling, does not analyze the failure modes, and does not discuss whether improving Omni-RL data quality, scaling the audio-visual training data, or architectural changes (e.g., dedicated cross-modal fusion layers) would be needed to push past the ~55% WorldSense barrier. The contrast with the strong document and GUI results—where the model clearly excels—makes the audio-visual weakness more conspicuous, and the paper's silence on this gap is a notable omission.

Text Reasoning Regression on Math and Code Benchmarks, Not Acknowledged as a Tradeoff

The assumption or constraint. The paper frames the text-only evaluation (Section 4.4) as demonstrating that the omni-modal model "retains the text reasoning performance of the Nemotron 3 Nano 30B-A3B backbone." Table 10 shows that this claim holds for knowledge-intensive benchmarks—MMLU-Pro drops from 78.3 to 77.3 (−1.0 point, −1.3% relative), GPQA from 73.0 to 72.2 (−0.8 points, −1.1% relative)—but breaks down for reasoning-intensive benchmarks. AIME-2025 (competition math) drops from 89.1 to 82.1 (−7.0 points, −7.9% relative). LiveCodeBench v5 drops from 68.3 to 63.2 (−5.1 points, −7.5% relative). These are substantial regressions that, if observed in a model marketed primarily for its reasoning capabilities, would be cause for concern. The paper's summary statement—"the goal of the Omni model is to maintain text benchmarks of the LLM while adding vision and audio understanding capabilities"—is true for factual knowledge but misleading for mathematical and algorithmic reasoning, where the model demonstrably degrades.

The consequence. Practitioners who would consider deploying Nemotron 3 Nano Omni as a unified model for both multimodal tasks and text-only reasoning tasks—a natural use case for a model that can process documents, answer questions about videos, and write code—must contend with the fact that the model is measurably worse at math and coding than the LLM backbone it was built on. If an application requires strong mathematical reasoning (e.g., analyzing quantitative data in documents, solving geometry problems from diagrams, or generating code from API documentation with screenshots), the multimodal model may underperform a pipeline that routes math/code queries to the original LLM backbone and routes multimodal queries to the omni model. The tradeoff—better vision/audio/GUI at the cost of some math/code capability—may be acceptable for many applications, but the paper does not frame it as a tradeoff, leaving practitioners to discover it through their own evaluation.

What evidence exists in the paper. Table 10 directly shows the regressions on AIME-2025 and LiveCodeBench. However, the paper's text in Section 4.4 does not comment on the magnitude of these drops, instead summarizing the results with a general statement about maintaining text benchmarks. The AIME-2025 result is reported as "Pass@1 average of 8 runs" and LiveCodeBench as "score of 1 run," which provides adequate precision but the averaging for AIME may mask run-to-run variance. The paper does not evaluate the omni model on the full text benchmark suite used for the LLM backbone (the backbone paper, NVIDIA et al., 2025b, presumably reports a more comprehensive evaluation), making it impossible to determine whether the regression is limited to math and code or extends to other reasoning domains.

Mitigation status. Not addressed as a limitation. The paper notes that text-only RL stages freeze the LM input token embedding parameters "to mitigate representational drift between multi-modal stages" (Section 3.2.2), which acknowledges the risk of drift but does not prevent the observed regression on math and code. No analysis is provided of whether the regression is caused by catastrophic forgetting during multimodal SFT, interference from the RL stages, or capacity saturation in the 30B-A3B architecture. The paper does not discuss whether training with a higher proportion of math and code data in the multimodal SFT stages, or applying additional text-only RL after omni-modal training, could recover the lost reasoning capability.

The assumption or constraint. The paper reports aggregate accuracy for the two key audio-visual benchmarks: DailyOmni (74.5, single score across 1,197 questions spanning 6 tasks) and WorldSense (55.2, single score across 3,172 questions spanning 26 tasks). No per-task breakdowns, per-category analyses, or error-type distributions are provided for either benchmark. This is in contrast to the audio evaluation (Table 8), which reports per-subtask breakdowns for MMAU (Music 74.2, Audio 76.9, Speech 72.8) and per-benchmark breakdowns for VoiceBench (IFEval 88.7, BBH 91.1, AdvBench 100, etc.). The paper's central architectural claim—that temporal interleaving of modality tokens "enables joint temporal reasoning across modalities" (Section 2)—makes specific predictions about which tasks should benefit: temporally-aligned audio-visual associations (e.g., identifying which object made a sound, matching speech to a speaker's lip movements) and temporally-extended reasoning (e.g., understanding causal sequences across modalities) should show the largest gains. Without per-task breakdowns, these predictions are untestable.

The consequence. Practitioners evaluating whether Nemotron 3 Nano Omni is suitable for a specific audio-visual application cannot determine whether the model's aggregate DailyOmni score of 74.5 reflects uniformly competent performance across all six tasks or strong performance on easy tasks masking near-random performance on harder ones. For example, if the model achieves 95% on simple audio-visual matching but 55% on causal reasoning questions, an application requiring causal audio-visual understanding (e.g., a video monitoring system that must determine whether a sound preceded or followed a visual event) would face much higher error rates than the aggregate score suggests. Similarly, the WorldSense aggregate of 55.2% across 26 tasks could hide strong performance on short-clip tasks and near-chance performance on long-range dependency tasks—information that is critical for deployment decisions involving long-form video analysis. The absence of per-task analysis also prevents the research community from identifying specific failure modes (e.g., does the model fail when audio and visual streams are intentionally misaligned? when there are multiple simultaneous sound sources? when the relevant audio event is quiet or brief?) that could guide future architectural improvements.

What evidence exists in the paper. The only DailyOmni and WorldSense results are the aggregate accuracy scores in Table 9. The paper describes DailyOmni as covering "six tasks, testing temporal alignment, event understanding, causal reasoning, and cross-modal consistency" (Section 4.3) and WorldSense as covering "26 tasks, evaluating long-range dependencies, sound grounding, temporal reasoning, and complex cross-modal inference" (Section 4.3), but provides no scores for these individual task categories. The benchmark descriptions come from the original benchmark papers (Zhou et al., 2026; Hong et al., 2026), not from the paper's own analysis. The Conv3D+EVS ablation (Tables 12–13) does report per-benchmark scores for DailyOmni, LongVideoBench, Video-MME, and WorldSense, but these are benchmarks as wholes, not task categories within them.

Mitigation status. Not addressed. The paper does not acknowledge the absence of per-task breakdowns as a limitation, nor does it indicate whether such breakdowns will be made available in a future release or in supplementary materials. This is a missed opportunity given that the paper positions itself as contributing to open research—per-task scores for the omni benchmarks would be more valuable to the community than aggregate scores, because they would enable targeted comparisons and failure analysis.

EVS and Conv3D Efficiency Gains Validated Only on Synthetic Workloads with a Single GPU

The assumption or constraint. The efficiency measurements in Section 4.8 and the Conv3D/EVS ablations in Tables 12–13 are conducted on a single NVIDIA B200 GPU using vLLM nightly as of 2026-04-19, with EVS at 50%. The TTFT measurements (Tables 12–13) are averaged across five concurrency-1 aiperf runs against a synthetic 512-frame, 512×512 video at 30 fps—not against real videos from the evaluation benchmarks. The throughput comparisons in Section 4.8 use two artificial workloads: a multi-document workload (32 images at 1024×1536) and a long-video workload (512 frames at 512×512). These workloads are fixed and uniform rather than drawn from a distribution of realistic inputs. The paper does not measure efficiency on the actual DailyOmni, WorldSense, or Video-MME test sets, where video lengths, resolutions, and audio content vary substantially and EVS's pruning effectiveness might differ (e.g., static videos with little visual change would be pruned more aggressively than action-heavy videos, altering the accuracy-efficiency tradeoff in content-dependent ways).

The consequence. The claimed efficiency gains—Conv3D reducing TTFT by 25%, EVS reducing it by 19%, combined reducing it by 33% at $q = 0.5$ , with up to 14% additional reduction at $q = 0.7$ (Tables 12–13)—are measured under idealized, homogeneous conditions that may not reflect real deployment. In production, video content varies dramatically: a talking-head lecture video has minimal frame-to-frame change (EVS would prune aggressively, potentially losing critical information from slides or gestures that change slowly), while a sports broadcast has rapid motion across the entire frame (EVS would prune less, reducing the latency benefit). The accuracy-efficiency tradeoff is therefore content-dependent, and the single synthetic workload cannot characterize the distribution of tradeoffs that practitioners would encounter across diverse video types. Additionally, the single-GPU measurements do not address whether EVS and Conv3D provide proportional benefits in multi-GPU deployments (where tensor parallelism or pipeline parallelism changes the bottleneck from per-GPU compute to inter-GPU communication) or at high concurrency (where the GPU is compute-bound rather than memory-bandwidth-bound, potentially reducing the relative benefit of token reduction).

The paper also does not evaluate EVS on the audio-visual benchmarks (DailyOmni, WorldSense) in Tables 12–13—the accuracy columns report DailyOmni and WorldSense scores, but these are from standard evaluation runs, not from the synthetic 512-frame video used for TTFT measurement. The connection between the efficiency measurements (synthetic video, concurrency 1) and the accuracy measurements (benchmark videos of varying lengths and content) is therefore indirect: we know EVS at $q = 0.7$ preserves accuracy on the benchmarks, and we know EVS reduces TTFT on a synthetic video, but we do not know the TTFT reduction on the actual benchmark videos or how it varies with content.

What evidence exists in the paper. Tables 12–13 provide per-benchmark accuracy (DailyOmni, LongVideoBench, Video-MME, WorldSense) alongside TTFT measured on a synthetic workload, with the note that "TTFT is averaged across five concurrency-1 aiperf runs against a synthetic 512-frame, 512×512 video at 30 fps." The inference efficiency section (4.8) states: "All measurements use a single NVIDIA B200 GPU and vLLM nightly as of 2026-04-19 with EVS 50%. Nemotron 3 Nano Omni is evaluated in NVFP4, Qwen3-Omni with dynamic FP8 quantization, and Nemotron Nano V2 VL in NVFP4. Text ISL=50 and OSL=8000. The multi-document workload contains 32 images at 1024×1536 resolution. The long-video workload contains 512 frames at 512×512 resolution." The paper is transparent about the measurement setup, but does not discuss how the synthetic workload relates to the benchmark videos or how content-dependent pruning would affect production deployments.

Mitigation status. Partially addressed through transparency about the measurement setup, but not through content-diverse efficiency evaluation. The paper does not report efficiency measurements on the actual benchmark videos, does not characterize the distribution of EVS pruning rates across different video types, and does not discuss how practitioners should select $q$ for a given content distribution. This is a practical gap: a deployment engineer needs to know what $q$ value to use for their specific video content mix, and the paper provides only a single-point recommendation ( $q = 0.7$ based on the synthetic workload) without evidence that this generalizes. Future work evaluating EVS across a diverse video corpus with content-annotated efficiency metrics would close this gap.

7. Implications and Future Directions

How This Work Changes the Landscape

Nemotron 3 Nano Omni does not introduce a single breakthrough technique that will be cited as the origin of a new subfield. What it does—and this is arguably more impactful for practitioners—is demonstrate that omni-modal capability and production-grade efficiency are not in tension, and that the architecture, training recipe, and deployment engineering choices that reconcile them are sufficiently mature to be documented, released, and replicated. This is a reframing rather than a paradigm shift: the paper converts "can we build an omni-modal model?" (answered affirmatively by GPT-4o, Gemini, and Qwen3-Omni) into "can we build one that runs at 500+ tokens/second on a single GPU while preserving median accuracy within <1% of BF16 at 5 bits per weight?" The answer, validated across ~30 benchmarks and four modalities, is yes.

The landscape change is in what the field should optimize for going forward. Prior to this work, open-source multimodal models largely pursued capability parity with proprietary systems, treating efficiency as a compression step applied after the fact. Nemotron 3 Nano Omni inverts this: efficiency is the architectural starting point (the 30B-A3B MoE hybrid backbone), and capability is layered on through a staged curriculum that demonstrably avoids catastrophic forgetting of text reasoning. This flips the burden of proof. A new multimodal model that matches Nemotron 3 Nano Omni's accuracy but requires 3× more compute per query is now a harder sell than it would have been before this paper, because the paper establishes that the efficiency ceiling is much higher than previously demonstrated for open omni-modal models.

The paper also reconciles a latent contradiction in multimodal training: the widespread belief that adding modalities inevitably degrades text performance. The evidence has been mixed—some multimodal models show significant text regression, others maintain it, and the training recipe differences that explain the variation have been opaque. This paper provides a concrete, documented recipe (seven SFT stages with progressive modality introduction and carefully managed parameter freezing/unfreezing) that preserves MMLU-Pro and GPQA while adding native audio, video, and 256K context. The recipe's release alongside code (Megatron-Bridge) and data pipelines makes it a reference point that future models will be compared against, not just on final accuracy, but on text-capability retention. If a future omni-modal model drops 5 points on MMLU-Pro, the community will have a concrete counterexample showing that such regression is avoidable.

The research directions that become more attractive after this work:

Token-reduction as a first-class architectural concern. Conv3D and EVS demonstrate that processing video through a vision transformer at native temporal resolution is unnecessarily expensive, and that learned temporal compression (Conv3D) plus information-theoretic pruning (EVS) can reduce LLM input tokens by 70% with negligible accuracy cost. This opens the door to more aggressive token-reduction techniques: learned token merging within the ViT, cross-modal token compression that exploits audio-visual redundancy, or adaptive frame-rate sampling based on scene complexity. The paper's EVS sweep (Table 13) provides a calibration curve—accuracy flat through q=0.7, then gradual degradation—that future methods can use as a baseline to beat.
Staged modality integration as a general recipe. The paper's curriculum (vision alignment → vision+LLM → audio alignment → joint omni) is a template for integrating any new modality—point clouds, depth maps, tactile sensors, structured data tables—into an existing multimodal LLM. The template is: projector warmup with frozen encoder and frozen LLM → encoder unfreezing with frozen LLM → joint training with all components → context scaling. The paper's success with this recipe for audio makes it natural to ask whether the same recipe works for other modalities with minimal modification.

The research directions that become less attractive:

Building omni-modal models on dense backbones at this parameter scale. The throughput comparisons (3× single-stream over Qwen3-Omni, 9× throughput per GPU at iso-interactivity) are sufficiently large that a dense 30B model would need a qualitatively different efficiency innovation to compete. The MoE hybrid architecture is not the only path to efficiency, but it is now the path with the strongest open-source evidence. Dense architectures for omni-modal models at >10B parameters face a steeper burden of proof after this paper.
Treating quantization as a post-hoc afterthought. The paper's NVFP4 results—median accuracy drop <1% across 25 benchmarks at 5 effective bits per weight—are achieved through a carefully designed mixed-precision strategy where MoE experts are aggressively quantized (NVFP4), sensitive projections are kept at FP8, and modality encoders remain in BF16. This is not a one-size-fits-all INT4 quantization; it is a precision allocation problem solved through architecture-specific sensitivity analysis. The implication is that future models should co-design their quantization strategy with their architecture, not apply it as a uniform compression after training.

Follow-Up Research This Work Enables

Ablating the seven-stage curriculum to identify which stages are load-bearing. The paper's most significant unvalidated claim is that the seven-stage SFT pipeline is necessary for stable omni-modal training. A controlled experiment—training the same architecture and data with a 3-stage curriculum (vision alignment, audio alignment, joint training) and comparing final accuracy on MMAU, DailyOmni, and WorldSense—would establish which intermediate stages (Stage 0 projector warmup? Stage 3 audio encoder training separate from LLM? The 48K intermediate context stage?) actually matter. The experiment is expensive (re-training even a subset of the pipeline costs thousands of GPU-hours) but would transform the training recipe from a description of "what we did" to a validated guide with identified essential versus optional components. The paper's release of training code makes this experiment directly replicable by a team with sufficient compute.

Measuring whether temporal interleaving of modality tokens improves audio-visual reasoning over block-concatenation. The paper claims that temporally interleaving audio and visual tokens "enables joint temporal reasoning across modalities" (Section 2), but provides no ablation comparing interleaved versus block-concatenated sequences. A direct experiment would train two versions of the model—one with interleaved token ordering, one with all visual tokens followed by all audio tokens (or vice versa)—on identical data, then evaluate on DailyOmni's six tasks and WorldSense's 26 tasks with per-task breakdowns. The hypothesis is that interleaving should improve performance specifically on tasks requiring fine-grained temporal alignment (matching a sound onset to a visual event, tracking which speaker is talking) while having minimal effect on tasks requiring global video understanding (overall scene classification, summarization). If the hypothesis is confirmed, temporal interleaving becomes a recommended design pattern with quantified benefits; if not, it suggests that the transformer's long-range attention is sufficient for cross-modal temporal reasoning and interleaving is an unnecessary implementation complexity.

Characterizing how EVS pruning effectiveness varies with video content type. Tables 12–13 demonstrate EVS's accuracy-efficiency tradeoff on four benchmarks, but the TTFT measurements use a synthetic 512-frame, 512×512 video—not the actual benchmark videos. A content-diverse efficiency evaluation would measure EVS's token reduction ratio (what fraction of tokens are pruned at each q), TTFT, and per-benchmark accuracy across a taxonomy of video types: talking-head lectures (low motion, static background), sports broadcasts (high motion, full-frame changes), screen recordings (sparse changes, text-heavy), nature documentaries (slow panning, gradual scene changes), and conversational videos (moderate motion, audio-visual synchronization critical). The key measurement is whether the optimal pruning rate q varies systematically with content type—if q=0.7 is universally optimal, EVS is a set-and-forget optimization; if different content types require different q values, adaptive EVS (varying the pruning rate based on online estimates of frame-to-frame dissimilarity) becomes a necessary extension. The paper's release of EVS as part of the inference stack makes this evaluation straightforward for a team with access to a diverse video corpus.

Scaling Omni-RL data volume and measuring its causal contribution to audio-visual reasoning. The Omni-RL stage uses 120K prompts (Section 3.2.4), but the paper never compares the final checkpoint against a version without Omni-RL—only SFT + earlier RL stages. A scaling experiment that varies Omni-RL data volume (e.g., 30K, 60K, 120K, 240K prompts) and measures accuracy on WorldSense and DailyOmni would reveal whether audio-visual reasoning benefits from RL at all, and whether the benefit saturates at the current 120K scale or would continue to improve with more data. This experiment connects to a broader question the paper does not address: for omni-modal tasks where ground-truth verification is difficult (unlike math or code, where answers can be checked automatically), can outcome-based RL with rule-based verifiers actually improve cross-modal reasoning, or does the verifier signal become too noisy? If Omni-RL provides no measurable benefit on WorldSense beyond the SFT baseline, that would be a important negative result indicating that better verifiers—or entirely different training paradigms—are needed for audio-visual reasoning.

Training a lightweight difficulty predictor to replace expensive pass-rate filtering in multimodal RL. The Omni-RL pass-rate filtering generates 8 rollouts per prompt from the initial policy checkpoint to estimate difficulty—an unaccounted compute cost that likely dominates the RL training cost itself for long-context omni prompts. A follow-up could train a small classifier (a few million parameters, operating on the prompt embedding produced by the frozen LLM backbone) to predict pass rate from the prompt alone, trained on the filtering rollouts from the first RL round and then used to filter prompts for subsequent rounds without generating additional rollouts. The metric is the correlation between predicted and actual pass rates on a held-out set of prompts, and the downstream impact on RL training efficiency. If the predictor achieves >0.8 correlation, it would reduce the filtering cost from 8 rollouts per prompt to a single forward pass through a lightweight model—making iterative RL training on multimodal prompts practically feasible at scale, which it currently is not given the unaccounted filtering overhead.

Systematic evaluation of text reasoning regression across the full Nemotron 3 Nano benchmark suite. Table 10 shows that AIME-2025 drops 7 points and LiveCodeBench drops 5 points after multimodal training, but the paper evaluates only a subset of the text benchmarks used for the LLM backbone. A comprehensive evaluation—running the omni model on every text benchmark reported in the Nemotron 3 Nano 30B-A3B paper (NVIDIA et al., 2025b)—would characterize the full pattern of regression: is it limited to math and code, or does it extend to other reasoning domains? Do multiple-choice benchmarks (which test recognition) degrade less than generative benchmarks (which test recall and synthesis)? Is there a systematic relationship between a benchmark's reliance on structured reasoning and its susceptibility to multimodal interference? This evaluation requires no additional training, only inference runs, and would provide practitioners with a complete capability profile rather than the cherry-picked subset in Table 10. If the regression is concentrated in specific reasoning types (e.g., symbolic manipulation, algorithmic thinking), that would suggest targeted mitigation strategies: include more math and code data in multimodal SFT, or apply additional text-only RL after multimodal training specifically on the degraded capabilities.

Practical Applications and Downstream Use Cases

Document processing pipelines for enterprise. Nemotron 3 Nano Omni's strong document understanding results—MMLongBench-Doc 46.1 (reasoning off) and 57.5 (reasoning on), OCRBench-V2 EN 65.8, CharXiv-RQ 63.6 with reasoning—combined with 256K context and the FP4 checkpoint at 20.9 GB, make it immediately deployable for enterprise document workflows. A financial services firm processing 50-page quarterly reports with embedded tables and charts could run the model on a single consumer GPU (RTX 4090 or similar, which has 24 GB VRAM just above the FP4 model size), extracting structured data, answering analyst questions, and generating summaries at ~500 tokens/second (the single-stream throughput reported in Section 4.8). The Conv3D and EVS mechanisms extend this to video documents—recorded earnings calls, product demos with narration—where the 70% token reduction (141K to 42K tokens for a 512-frame video) keeps latency manageable. The key practical advantage over cloud-based alternatives (GPT-4o, Claude) is data sovereignty: the model runs entirely on-premises, eliminating the need to send sensitive financial documents to external APIs.

GUI automation agents for accessibility and testing. The ScreenSpot results—90.3 (reasoning off) and 89.3 (reasoning on) on ScreenSpot, 93.4 (reasoning off) on ScreenSpot-v2, 47.4 (reasoning on) on OSWorld—represent a step-change in GUI grounding accuracy compared to prior open-source models (Qwen3-Omni scores 42.5, 42.8, and 11.1 respectively). For accessibility applications, a model that can accurately locate UI elements on a screenshot (the ScreenSpot task) can power screen readers that describe not just text content but spatial layout and interactive affordances—"the Submit button is in the bottom-right corner of the dialog" rather than just "there is a Submit button." For automated testing, the OSWorld result (47.4% task success) is not yet production-ready for fully autonomous operation, but it is high enough to serve as a filter: the model can attempt a GUI task and flag its own low-confidence predictions for human review, reducing the human workload by roughly half compared to manual testing alone. The dynamic resolution processing—preserving native aspect ratios up to 1840×1840 pixels—is critical here because GUI screenshots are typically wide (laptop) or tall (mobile), and tiling-based approaches that squash aspect ratios would degrade grounding accuracy.

Voice assistants with visual context. The combination of strong ASR (OpenASR average 5.95 WER, VoiceBench 89.4 average) and vision-language capability makes Nemotron 3 Nano Omni suitable for voice-driven multimodal assistants. A user could speak a query while sharing their screen or camera feed—"what does this error message mean?" while pointing their phone at a laptop screen, or "summarize the last 5 minutes of this meeting" during a video call—and the model processes the audio and visual streams jointly with temporal alignment. The audio efficiency (12.5 tokens per second of audio, ~375 tokens per 30-second clip) means that continuous listening for minutes at a time produces manageable token counts even before EVS reduction. For a 5-minute interaction, audio tokens (~3,750) plus periodic video frames with EVS pruning might total ~10K–15K tokens—well within the context window and fast enough for real-time response given the ~500 tokens/second output throughput. The on-device deployment potential (FP4 at 20.9 GB) is particularly relevant here because voice assistants are latency-sensitive and privacy-sensitive—processing audio locally rather than streaming to the cloud addresses both concerns, and the model's efficiency makes local deployment on a workstation-class GPU feasible.

When to Prefer This Method

The paper does not explicitly frame itself as "prefer X over Y under conditions A, B, C," and its comparisons are empirical benchmarks rather than a structured tradeoff analysis against named alternatives. The primary alternatives are Qwen3-Omni (open-source omni-modal, 30B-A3B) and Nemotron Nano V2 VL (prior Nemotron vision-language model, 12B), with Qwen3.5-Omni Flash as a frontier reference. The paper's implicit positioning is that Nemotron 3 Nano Omni should be preferred when:

Throughput and latency are primary constraints. The 3× single-stream throughput advantage over Qwen3-Omni and 9× throughput per GPU at iso-interactivity (Section 4.8) are the paper's strongest differentiating claims. If a deployment serves many concurrent users or requires sub-2-second time-to-first-token on long multimodal inputs, Nemotron 3 Nano Omni's MoE hybrid backbone and token-reduction pipeline provide advantages that no amount of accuracy tuning on a denser architecture would recover. This includes on-device or single-GPU deployments where the NVFP4 checkpoint fits in consumer GPU memory.
Document understanding, OCR, and GUI grounding are the dominant use cases. Tables 7 and 8 show that Nemotron 3 Nano Omni's advantages are largest on precisely these tasks—MMLongBench-Doc (+8.1 points over Qwen3-Omni Instruct), OCRBench-V2 EN (+11 points), ScreenSpot (+47.8 points), ScreenSpot-v2 (+50.6 points)—while its advantages are narrower or non-existent on general VQA (DocVQA, TextVQA), STEM reasoning (MMMU comparable), and audio understanding (MMAU 2.9 points behind Qwen3-Omni). An application whose workload is dominated by document processing, chart interpretation, and GUI interaction would benefit disproportionately from this model's specific strengths.
Text reasoning capability retention matters and math/code regression is acceptable. The model preserves MMLU-Pro and GPQA within ~1 point of its LLM backbone (Table 10), but loses 5–7 points on AIME and LiveCodeBench. A deployment that primarily uses the model for multimodal tasks (document QA, video understanding) with occasional factual text queries would find the text regression acceptable; a deployment that expected the model to serve as a unified code-generation-and-document-analysis engine would need to measure the code regression on their specific workload and potentially route code queries to a separate model.
Open-source release and on-premises deployment are requirements. The paper releases BF16, FP8, and NVFP4 checkpoints alongside training data, code, and data pipeline recipes. For organizations that cannot use proprietary APIs due to data governance, privacy regulations, or cost at scale, the availability of a fully open omni-modal model that can run on owned hardware is a categorical advantage over Qwen3.5-Omni Flash (non-open-source) and GPT-4o/Gemini (proprietary), regardless of relative benchmark scores.

Conversely, the paper's results suggest that Qwen3-Omni or Qwen3.5-Omni Flash would be preferred when audio-only understanding (MMAU) is the primary workload (Qwen3-Omni leads by 2.9 points, Qwen3.5-Omni by 5.8 points), or when the absolute highest audio-visual reasoning accuracy is required regardless of throughput (Qwen3.5-Omni Flash leads on DailyOmni by 7.3 points and WorldSense by 2.4 points). The paper does not explicitly make these recommendations, but they follow directly from the benchmark comparisons in Tables 7–9 and the efficiency measurements in Section 4.8.