Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization

ArXiv: 2604.19079

🎯 Pitch

A single ASR model matched specialized offline quality at near-zero streaming latency by adding a training-time consistency loss that forces the model to predict the same transcript whether it sees full or partial audio. This closes a stubborn performance gap that previously forced most production systems to maintain separate offline and streaming models.

1. Executive Summary

This paper introduces a Unified ASR framework that trains a single Transducer (RNNT) model to perform well in both offline and low-latency streaming decoding, using chunk-limited attention with right context and dynamic chunked convolutions (DCConv) to adapt the same parameters across modes. To further close the remaining performance gap, the authors propose mode-consistency regularization for RNNT (MCR-RNNT), an efficiently implemented symmetric KL divergence loss that encourages agreement between the model's offline and streaming joint output distributions. On the Open ASR Leaderboard, the proposed unified training with MCR-RNNT outperforms standard unified training methods and streaming-only baselines at latencies down to 0.24s, scaling to a 600M-parameter model that achieves 5.76% average WER in offline mode while maintaining 7.35% WER at 0.24s streaming latency, establishing that a single model can approach SOTA offline quality without sacrificing competitive low-latency streaming performance.

2. Context and Motivation

The Core Problem: One Deployment Target, Two Conflicting Requirements

The fundamental tension this paper addresses is deceptively simple: offline ASR wants to see the entire utterance before transcribing it; streaming ASR wants to emit words as soon as possible after they are spoken. These are diametrically opposed information constraints. An offline model can use bidirectional context — it knows what comes after every frame, which resolves acoustic ambiguities (was that "their" or "there"? — the next few words will disambiguate). A streaming model operating at, say, 200ms latency must commit to a hypothesis with only a tiny glimpse of the future, if any.

In principle, one could solve this by maintaining two separate models. In practice, this doubles the entire lifecycle cost:

Training: two models to train, tune, and validate, potentially on different data distributions or with different hyperparameters.
Deployment: two model binaries to serve, monitor, update, and roll back, occupying GPU memory or inference slots.
Data pipelines: potentially two tokenization schemes, two normalization strategies, two evaluation harnesses.

For large-scale ASR deployments — voice assistants processing billions of queries, transcription services handling millions of hours of audio, call center analytics running continuously — these costs are material in engineering time, compute dollars, and operational complexity. The unification problem is therefore driven by practical economics, not just research curiosity: a single model that serves both regimes at acceptable quality would cut development and operational overhead roughly in half.

Why the Transducer Architecture Makes This Hard

The RNNT architecture [graves2012rnnt] is naturally streaming-friendly in one specific sense: its decoder (predictor network) operates autoregressively on the output token sequence, needing only the previously emitted token to predict the next, never the future. There is no encoder-decoder attention bottleneck that would require full audio before decoding begins.

However, the modern encoder — which in this paper is a FastConformer [rekesh2023fastconformer], a descendant of the Conformer architecture [conformer] — introduces two sources of training-inference mismatch that become acute during chunked streaming:

1. Multi-Head Attention (MHA) is globally conditioned by default. Standard self-attention allows every frame to attend to every other frame. During streaming, the encoder processes audio in chunks, and frames within a chunk cannot attend to frames that haven't arrived yet. Unless the attention mask is explicitly constrained during training, the model learns to rely on bidirectional context that is unavailable at inference time, producing a sharp accuracy drop when forced into chunked mode.

2. Convolution sub-blocks leak future information. Conformer blocks contain depthwise convolution modules with kernel sizes typically 17–31 frames. In offline mode, these convolutions are centered — each output frame can "see" frames ahead of it. In causal streaming mode, they are shifted to see only past frames, but this removes genuinely useful future context and degrades accuracy even when future information is available. Worse, if the convolutions are not restructured for chunk boundaries, a frame near the right edge of a chunk can inadvertently see across the chunk boundary into frames it shouldn't access, creating a silent mismatch between the training regime and the actual streaming inference behavior.

These architectural details mean that simply taking a well-trained offline Conformer and running it in chunked mode is not a viable strategy — the accuracy degradation is catastrophic. The paper's Table 1 illustrates this dramatically: an offline-trained baseline achieves 6.47% average WER in offline mode but degradates to 94.05% WER at 0.16s streaming latency — essentially random performance — because the model has never seen chunked attention masks during training and its convolutions assume full future access.

Prior Approaches and Their Limitations

The literature offers several partial solutions, but each addresses only part of the mismatch or introduces new tradeoffs:

Chunk-limited attention [Chen2020DevelopingRS, Moritz2020StreamingAS]. These methods modify the self-attention mask during training so that each frame can attend only to a limited left context (past frames), a current chunk, and optionally a small right context (future frames). This directly addresses the MHA mismatch: the model learns to operate with restricted context during training, so it doesn't catastrophically fail when the same restrictions are applied at inference. The paper's streaming baseline uses a multi-look-ahead variant from Noroozi et al. (2023) [Noroozi2023StatefulCW], which samples different look-ahead sizes during training to produce a single model robust to multiple latency targets.

Where this falls short: the model is trained only in streaming mode. It never sees full bidirectional context, so it sacrifices offline quality to achieve streaming robustness. The streaming baseline in Table 1 achieves 7.75% WER offline — worse than the offline-only baseline's 6.47% — because it has been trained with restricted attention and never learns to exploit bidirectional context effectively. The paper frames this as a "lack of contextual capabilities": the model is architecturally prevented from using the future context that would resolve ambiguities when it is available.

Causal convolutions. Replacing standard centered convolutions with causal (past-only) variants prevents future-information leakage but removes genuinely useful right-side acoustic context. The result is a degradation in scenarios where the model could use future information (higher-latency streaming or offline), because the architecture itself is handcuffed.

Dynamic Chunk Convolution (DCConv) [Li2023DynamicCC]. This addresses the convolution mismatch by making convolutions chunk-aware. During streaming training, hidden states are reshaped into chunks, and the convolution operates within each chunk with symmetric padding matching the kernel's half-width, plus left and right contexts explicitly provided as neighboring chunks. The same convolution parameters are shared between offline mode (full-context) and streaming mode (chunk-limited). This is a significant improvement over causal convolutions because it preserves the convolution's ability to use future context within the allowed window, but it only addresses the convolution sub-block — the MHA mismatch remains.

Unified training (single-mode and dual-mode). The most direct prior approach to solving the "two models" problem is to train one model on both modes. The paper describes two variants:

Single-mode (SM): each training step randomly picks offline or streaming mode with some probability and computes gradients only for that mode. This is simple and computationally cheap (one forward-backward pass per step), but the two modes never interact during optimization. The model must learn to serve both masters from interleaved but independent gradient signals.
Dual-mode (DM): each step runs both modes on the same batch and combines their losses. This directly couples the optimization, but at double the computational cost per step.

The key limitation, which the paper's results make clear, is that neither variant is sufficient at very low latencies. In Table 1, both Unified SM and Unified DM degrade significantly below 0.56s latency (WERs of 13.33% and 16.91% respectively at 0.24s, versus 10.01% for the dedicated streaming baseline). The unified models are being asked to represent two fundamentally different context regimes with a single set of weights, and at extreme latency constraints the conflict becomes too severe for simple loss mixing to resolve. The encoder must simultaneously learn representations that work with full bidirectional context (offline) and with almost no right context (0.16s streaming), and the gradients from one mode can pull the parameters away from the other mode's optimum.

Zipformer-based unified frameworks and TSCA/DRC. The paper cites recent work that incorporates right context into streaming training. Sharma et al. (2025) [Sharma2025UnifyingSA] use a Zipformer architecture with dynamic right-context and report that increasing right-context closes much of the quality gap. Le et al. (2024) [Le2024ImprovingSS] introduce Time-Shifted Contextual Attention and Dynamic Right Context masking. These approaches improve streaming quality but still face the fundamental tradeoff: the smaller the right context, the stronger the mode conflict between offline and streaming representations within a single parameter set.

All-in-One ASR [Moriya2025AllinOneAU]. This work takes unification even further, combining not just offline/streaming modes but also multiple ASR paradigms (CTC, AED, Transducer) into a single model via a multi-mode joiner. This demonstrates that model footprint reduction can extend across fundamentally different architectures, but the paper notes that this approach is "orthogonal" to the streaming latency problem — it solves breadth (multiple paradigms) rather than depth (extreme latency constraints within one paradigm).

CR-CTC and TCR — consistency regularization, but not for this problem. Consistency regularization — encouraging a model to produce similar outputs under different input transformations — is well-established in computer vision and semi-supervised learning. Two prior works attempt to apply it to ASR:

CR-CTC [Yao2024CRCTCCR] applies symmetric consistency between offline and streaming CTC posteriors in a hybrid CTC-RNNT setup. The paper reports that extending this approach to unified ASR "consistently degraded streaming RNNT accuracy" despite maintaining offline quality. The authors attribute this to an objective mismatch: CTC loss encourages frame-synchronous, locally confident token predictions — alignments that are easy to produce with full bidirectional context but inappropriate for low-latency streaming where the encoder lacks sufficient future information to make confident frame-level decisions. The shared encoder gets pulled toward representations that serve CTC well offline but harm streaming RNNT.
TCR [Tseng2024TransducerCR] applies consistency to pruned RNNT outputs from augmented views, using occupation-based weighting to handle the large alignment space. However, this targets augmentation consistency (e.g., SpecAugment), not mode consistency (offline vs. streaming), and operates on pruned lattices rather than the full output distribution. The paper notes that "no publicly available implementation was available" and that the alignment differences between offline and streaming modes — which can be substantial due to the flexibility of offline representations — require a different formulation.

The Specific Gap This Paper Addresses

The paper's central claim is that existing unified training methods leave a persistent gap between offline and low-latency streaming performance that cannot be closed by architectural modifications (attention masking, DCConv) alone. The gap is not just an accuracy number — it reflects a fundamental representational conflict: the same encoder weights must produce representations that are simultaneously informative with full context (offline) and robust with almost no context (streaming). At latencies below ~0.5s, this conflict causes unified models to degrade sharply compared to dedicated streaming models, undercutting the economic motivation for unification (if the unified model isn't competitive in both regimes, you're back to maintaining separate models).

The proposed solution — MCR-RNNT — addresses this by adding an explicit regularizer that penalizes disagreement between the offline and streaming output distributions at the level of the RNNT joint network logits. Rather than hoping that mixed-loss training will implicitly find a shared representation that works for both modes, MCR-RNNT directly incentivizes the model to produce similar token predictions regardless of context availability. The intuition is that if the model's output distribution is consistent across modes, the encoder must learn representations that extract the same linguistic information whether or not future context is available — closing the gap not by compromising one mode for the other, but by finding a representation that is stably informative under both conditions.

How This Paper Positions Itself

The paper positions itself at the intersection of two lines of work: unified ASR architectures (which provide the structural foundation for a single model to serve both modes) and consistency regularization (which provides a mechanism for explicitly closing the remaining gap). The contribution is not a new architecture — FastConformer, chunk-limited attention, and DCConv are all drawn from prior work — but rather the integration of these components with a purpose-built consistency loss that targets the specific failure mode of unified models at low latencies.

The paper also explicitly positions itself as addressing a scale gap in the literature. Prior unified ASR work is characterized as reporting "strong, unified results on limited training datasets." The question of whether unification mechanisms remain effective at scale — with hundreds of thousands of hours of training data and 600M+ parameter models — is underexplored. The paper's scaling experiments (moving from 128M parameters on 120K hours to 600M parameters on 280K hours) are meant to demonstrate that the approach is not just a small-model curiosity but scales to production-relevant regimes, maintaining its advantage over baselines even as both data and model size grow.

A subtle but important positioning choice: the paper treats the open-source release of the framework and model checkpoint as a contribution in itself. This addresses the earlier note about TCR having no public implementation, and signals that the authors intend MCR-RNNT to be adopted and built upon rather than serving as a one-off result. The efficient Triton kernel implementation is also positioned as a practical enabler — without it, the full-lattice KLD computation over the RNNT joint output would be prohibitively expensive in both memory and time, making the method impractical for large-vocabulary training.

3. Technical Approach

3.1 Reader Orientation

The system being built is a single neural network — an RNNT-based automatic speech recognizer — that can switch between two operating modes at inference time without changing its parameters: offline (transcribing an entire audio file after it is fully recorded, with access to all past and future context) and streaming (transcribing audio in near-real-time as it arrives, with only a small look-ahead window into the future). The core problem it solves is that these two modes impose conflicting information constraints on the model's encoder — offline mode benefits from bidirectional context while streaming mode must operate with severely restricted right context — causing a single model trained with standard methods to either sacrifice offline quality or collapse at low streaming latencies. The solution takes the shape of a training framework that combines three complementary mechanisms: chunk-limited attention masking and dynamic chunk convolutions that structurally adapt the encoder to both modes, layered with a consistency regularization loss that explicitly penalizes the model when its output token predictions differ between offline and streaming views of the same input — effectively forcing the encoder to learn representations that are stably informative regardless of how much future context is available.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components interacting during training:

Audio Input and Feature Extraction — raw audio is converted to 128-dimensional log Mel-filterbank energies (FBanks) with 8× temporal subsampling, producing a sequence of acoustic feature vectors. This is identical for both offline and streaming modes.
FastConformer Encoder with Mode-Aware Sub-Blocks — the core acoustic encoder, composed of standard Conformer-style blocks, but with two critical modifications that depend on the current training mode: (a) chunk-limited multi-head self-attention that restricts each frame's attention span to a left context, a current chunk, and a sampled right context, and (b) Dynamic Chunk Convolution (DCConv) layers that reshape hidden states into chunk-aligned segments before depthwise convolution, preventing each chunk from leaking information across chunk boundaries while sharing the same convolution weights between offline and streaming paths.
RNNT Prediction + Joint Networks — a single-layer LSTM predictor that processes the token sequence autoregressively (identical in both modes), and a joint network that combines encoder outputs and predictor states to produce logits over the vocabulary plus a blank token. In dual-mode training, the joint network produces two complete logit tensors — one for the offline encoder pass and one for the streaming encoder pass — over the same input utterance.
Mode-Consistency Regularization (MCR-RNNT) Loss — an additional loss term computed directly from the two joint logit tensors, measuring the symmetric KL divergence between the offline and streaming output distributions at every valid (time, token-position) alignment. This is implemented as a fused GPU kernel in Triton that computes log-softmax and KL divergence on-the-fly without materializing the full probability tensor, keeping memory overhead near zero.

Information flows as follows: an audio batch enters the feature extractor → the encoder processes it twice (once with unrestricted receptive field for offline mode, once with chunk-limited masks and DCConv for streaming mode) → the predictor runs on the ground-truth token sequence → the joint network produces two logit tensors $z^{\text{off}}$ and $z^{\text{str}}$ , each of shape $T \times (U+1) \times V$ where $T$ is the number of encoder time steps, $U$ is the number of output tokens, and $V$ is the vocabulary size → the standard RNNT loss is computed from each logit tensor separately and combined with weighting $\alpha$ → the MCR-RNNT loss is computed as the symmetric KL divergence between the two output distributions → all losses are summed according to Equation 5 and backpropagated through shared encoder parameters.

3.3 Roadmap for the Deep Dive

First, the two architectural mechanisms — chunk-limited attention with right context and Dynamic Chunk Convolution — that structurally adapt the Conformer encoder to produce meaningful representations under both offline and streaming constraints, since these are the foundation that makes unified training possible at all.
Second, the two training strategies (single-mode and dual-mode) that combine the architectural adaptations into a training procedure, including the specific sampling of chunk and right-context sizes and the computational tradeoffs between SM and DM, since these define the baseline that MCR-RNNT improves upon.
Third, the MCR-RNNT loss itself — its mathematical definition, the symmetric KL divergence formulation, and the critical design choice of operating on the full RNNT joint logits rather than pruned lattices or aggregated distributions — since this is the novel contribution.
Fourth, the Triton kernel implementation that makes the loss computationally feasible, including the on-the-fly log-softmax recomputation strategy and its memory implications — since without this, the full-lattice approach would be impractical.
Fifth, the complete training objective and its hyperparameters (offline weight $\alpha$ , consistency weight $\lambda$ , KLD variant), since these control the trade-off between the two modes and are the practical knobs for deployment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that combining chunk-aware architectural adaptations with an output-level consistency regularizer enables a single RNNT model to serve both offline and streaming modes at quality levels competitive with dedicated models, by explicitly penalizing representational divergence between the two modes during training.

Chunk-Limited Attention with Right Context

Standard multi-head self-attention in a Conformer encoder allows every frame in an utterance to attend to every other frame — the attention pattern is a full $T \times T$ matrix where $T$ is the number of encoded time steps after subsampling. In offline mode, this is desirable: the model can use information from any part of the utterance to disambiguate the current frame. In streaming mode, it is impossible: frames beyond the current time have not yet been observed.

The paper adopts chunk-limited attention to make this compatible with both modes. During training, the self-attention mask is constrained to a structured sparsity pattern defined by three parameters:

Left context (L): the number of past frames visible to any frame in the current chunk. This is set to 70 frames (equivalent to $70 \times 80\text{ms} = 5.6\text{s}$ of audio, given the 8× subsampling that makes each encoder frame span 80ms of input audio). The left context is large and fixed because past audio is always available in streaming — the only cost is memory and computation, not latency.
Current processing chunk (C): the number of frames in the chunk being processed simultaneously. At each training step, C is sampled uniformly from the set $\{1, 2, 7, 13\}$ frames. These specific values are the result of "initial experiments with parameter search" and span a range from extremely small (1 frame = 80ms) to moderate (13 frames ≈ 1.04s). Sampling diverse chunk sizes during training produces a single model that can handle multiple chunk sizes at inference time, rather than being specialized to one latency target.
Right context (R): the number of future frames visible to any frame in the current chunk, extending past the chunk boundary. At each training step, R is sampled uniformly from the set $\{0, 1, 2, 3, 5, 7, 13, 26\}$ frames. This is the critical parameter controlling the latency-accuracy tradeoff: larger R means the model can see further into the future (better accuracy) but must wait longer before emitting output (higher latency). The set spans from zero look-ahead (R=0, purely causal within the chunk) to 26 frames (R=26, approximately 2.08s of look-ahead).

At any given frame position, the attention mask allows the frame to attend to:

All frames in the left context L (the preceding 70 frames),
All frames within the current chunk C, and
Up to R frames beyond the right boundary of the current chunk.

Frames beyond $C + R$ are masked to zero attention weight. The left context L, current chunk C, and right context R together specify the latency constraint: the total theoretical worst-case latency is $C + R$ (in frames), because the system must wait for the entire chunk plus its right context to arrive before processing that chunk. In the paper's evaluation (Table 1), latency values range from 2.08s ( $C = 13, R = 13$ ) down to 0.16s ( $C = 1, R = 1$ ), with the most challenging regime being latencies below ~0.5s.

Why sample multiple C and R values rather than fixing them: a model trained with a single chunk size and right context will be optimal only at that specific latency. Deploying at a different latency would require retraining. By sampling from a distribution during training, the model learns to operate robustly across a range of latencies — a single checkpoint can be deployed at multiple latency targets by simply changing the inference-time chunk and right-context parameters. This is the same multi-look-ahead strategy used in the dedicated streaming baseline [Noroozi2023StatefulCW], adapted here for unified training.

In offline mode during training, the mask is simply the full $T \times T$ bidirectional attention — every frame can attend to every other frame, with no chunk structure. The key insight of the unified training approach is that the same attention parameters serve both masks: in streaming mode, they apply the structured sparsity pattern; in offline mode, they apply the unrestricted pattern. The model learns to extract useful information from whatever context is available, rather than becoming dependent on bidirectional context that would fail catastrophically at low latency.

Dynamic Chunk Convolution (DCConv)

Conformer blocks contain depthwise separable convolution sub-layers interleaved with the attention sub-layers. These convolutions have a kernel size (typically 17 or 31) and operate by convolving the hidden sequence with learned filters centered on each frame position. In standard (offline) operation with full bidirectional access, the convolution is centered: output at frame $t$ depends on input frames in the range $[t - \frac{k-1}{2}, t + \frac{k-1}{2}]$ , where $k$ is the kernel size. This means every output frame uses $\frac{k-1}{2}$ frames of future context — context that is unavailable during streaming.

A natural fix is to use causal convolutions, which shift the receptive field to use only past frames: output at frame $t$ depends on input frames in the range $[t - (k-1), t]$ . This eliminates future-context dependence but at a cost: even when future context is available (higher-latency streaming or offline mode), the convolution cannot use it. The representational capacity is permanently reduced.

DCConv [Li2023DynamicCC] solves this by making convolutions chunk-aware without forcing them to be causal. The mechanism operates as follows:

During streaming training, the hidden state sequence is reshaped into chunks according to the current chunk size C (the same C sampled for the attention mask). This means the flat sequence of length T is partitioned into contiguous blocks of size C.
For each chunk, the convolution operates on an extended segment that includes the chunk itself plus left and right context padding equal to $\frac{k-1}{2}$ frames on each side. The left context comes from the previous chunk's output; the right context comes from the next chunk's input. Crucially, these are provided as explicit contexts that respect chunk boundaries — a frame near the right edge of a chunk can see $\frac{k-1}{2}$ frames into the next chunk (which are available because the system has buffered the right context), but no further.
After the convolution, only the output positions corresponding to the original chunk frames are retained. The context frames are used to provide boundary information but are discarded.
The same convolution parameters (kernel weights) are used in both offline and streaming modes. In offline mode, the convolution operates on the full unsegmented sequence with standard centered padding — exactly as it would in a dedicated offline model. In streaming mode, the same filters are applied to the chunk-padded segments.

Why this matters: DCConv eliminates the training-inference mismatch for the convolution sub-block without reducing its representational capacity. In offline mode, the convolution uses full bidirectional context within its kernel window — it is not artificially constrained to be causal. In streaming mode, the same convolution uses the allowed right context (which extends $\frac{k-1}{2}$ frames past the current chunk boundary) but no further, matching exactly what will be available at inference time. The model is structurally prevented from learning dependencies on future context that cross chunk boundaries, yet within each chunk-plus-context window, it retains the full convolution receptive field.

The paper also reports a Mamba2 + DCConv variant (Table 1) where the multi-head attention sub-layers are replaced entirely with Mamba2 blocks [Dao2024TransformersAS] while retaining DCConv. This variant achieves comparable streaming performance to the attention-based streaming baseline (8.41% WER at 2.08s latency vs. 8.39%), suggesting that DCConv alone can support effective streaming even when the attention mechanism is replaced with a state-space model, but it does not outperform the attention-based approach and is not the paper's primary contribution.

Probability of using DCConv: during training, the model switches between standard full-context convolutions and DCConv with probability 0.5. This means half the time the convolution operates in its offline-adapted mode, and half the time in its chunked streaming-adapted mode. Training the model with both modes ensures the shared convolution weights are effective under both constraints.

Unified Training Strategies: Single-Mode and Dual-Mode

The architectural modifications (chunk-limited attention, DCConv) provide the structural support for a single model to process audio in both offline and streaming modes. The training strategy determines how the model is optimized to perform well in both modes simultaneously. The paper considers two main strategies:

Single-mode (SM) training. At each optimization step, the training framework randomly selects a mode type $m \in \{\text{offline}, \text{streaming}\}$ . If $m = \text{offline}$ , the encoder processes the full audio with unrestricted attention and standard convolutions. If $m = \text{streaming}$ , the encoder processes with chunk-limited attention (using a chunk size C and right context R sampled from their respective sets) and DCConv (activated with 50% probability). In either case, a single forward-backward pass is performed, and the loss is:

$\mathcal{L}_{\text{SM}} = \mathcal{L}_{\text{RNNT}}^{(m)}$

where $\mathcal{L}_{\text{RNNT}}^{(m)}$ is the standard RNNT loss computed under mode $m$ , and the superscript $(m)$ indicates that the encoder activations (and thus the joint network inputs) were produced under that mode's context constraints.

The probability $p_{\text{off}}$ of selecting offline mode is a hyperparameter. The paper does not specify the exact value used for SM experiments, but by analogy with the dual-mode offline weight $\alpha$ , it likely defaults to 0.5.

What SM training optimizes: each gradient step pushes the model toward better performance in one randomly chosen mode. Over many steps, the model receives interleaved optimization signals — some steps tell it "produce good outputs with full context," others tell it "produce good outputs with restricted context." The hope is that the shared parameters converge to a compromise that serves both masters.

The limitation of SM: the two modes never interact during a single optimization step. The gradient from offline mode provides no information about whether the encoder's restricted-context outputs would be good; the gradient from streaming mode provides no information about whether the encoder's full-context outputs are preserved. The model learns by averaging independent signals over time, which works for moderate context restrictions but becomes fragile at extreme latency constraints where the optimal representations for the two modes diverge significantly.

Dual-mode (DM) training. Each optimization step runs both modes on the same input batch, producing two separate forward passes through the encoder — one with unrestricted context (producing encoder output $h^{\text{off}}$ ) and one with chunk-limited context (producing encoder output $h^{\text{str}}$ ). Both encoder outputs are processed by the same predictor and joint networks to produce RNNT losses, and the final training loss is a weighted combination:

$\mathcal{L}_{\text{DM}} = \alpha \, \mathcal{L}_{\text{RNNT}}^{\text{off}} + (1 - \alpha) \, \mathcal{L}_{\text{RNNT}}^{\text{str}}$

where $\alpha \in [0, 1]$ is the offline mode weight controlling the relative importance of offline versus streaming performance, $\mathcal{L}_{\text{RNNT}}^{\text{off}}$ is the RNNT loss computed from the offline encoder outputs, and $\mathcal{L}_{\text{RNNT}}^{\text{str}}$ is the RNNT loss computed from the streaming encoder outputs.

Computational cost of DM: DM training requires two forward passes through the encoder and one forward pass through the predictor and joint networks per training step, approximately doubling the computational resources compared to SM training. To keep the total GPU memory and compute budget comparable to the SM baseline in fair experiments, the paper reduces the batch size by half in DM training (Section 3.3: "we reduce the batch size twice to match the computational complexity of the baselines and single-mode training").

Why DM might be better than SM: by computing gradients from both modes on the same mini-batch, DM directly couples the two objectives. The optimizer sees how a parameter update that improves offline performance affects streaming performance (and vice versa) within a single step, rather than relying on statistical averaging over steps. This provides a stronger signal for finding shared representations that work well under both context conditions. However, Table 1 shows that DM alone does not outperform SM at very low latencies — in fact, DM degrades more sharply than SM below 0.56s (DM achieves 16.91% WER at 0.24s latency vs. 13.33% for SM). The direct coupling is helpful but insufficient to resolve the representational conflict at extreme latency.

A critical DM training detail: in the paper's dual-mode setup, the same chunk size C and right context R are used for the streaming forward pass within a given training step, and these are sampled independently per step from the predefined sets $\{1,2,7,13\}$ for C and $\{0,1,2,3,5,7,13,26\}$ for R. The model therefore sees a wide variety of latency configurations during training, learning to be robust across the entire range.

Mode-Consistency Regularization for RNNT (MCR-RNNT)

This is the paper's central contribution. The intuition is straightforward: even with chunk-limited attention and DCConv, a unified model trained only with separate RNNT losses per mode implicitly learns different output behaviors depending on context availability. At low latencies, the streaming encoder receives insufficient future information to resolve certain acoustic ambiguities, so it produces a different distribution over output tokens than the offline encoder does for the same input. These differences compound through the autoregressive decoding process, producing diverging transcription hypotheses.

MCR-RNNT addresses this by adding an explicit regularizer that penalizes the model when its offline and streaming joint output distributions differ, measured via KL divergence over the full RNNT output lattice. The model is incentivized not just to produce correct transcriptions in each mode independently, but to produce the same output distribution regardless of context availability — effectively learning representations that are context-invariant with respect to acoustic disambiguation.

The RNNT joint output tensor. To understand MCR-RNNT, we first need to understand the structure of the output it operates on. The RNNT joint network maps an encoder hidden state at time $t$ (where $1 \leq t \leq T$ ) and a predictor hidden state at token position $u$ (where $1 \leq u \leq U + 1$ , with $u = 1$ representing the start-of-sequence) to a vector of logits over the vocabulary $V$ plus the blank token $\emptyset$ . The blank token is a special output indicating "no token emitted at this time step" — the RNNT model can choose to advance the time index (emit blank) or advance the token index (emit a vocabulary token) at each (t, u) grid point. The full joint output is a tensor of shape $T \times (U+1) \times (V + 1)$ , representing logits for every possible alignment between the audio and the token sequence.

In practice, the blank token is typically folded into the vocabulary for implementation purposes, treating $\emptyset$ as token index 0 and using $V$ to refer to the vocabulary size including blank. The paper uses $V$ to denote the total number of output classes (vocabulary tokens plus blank), so the joint logits tensor has shape $T \times (U+1) \times V$ .

Computing MCR-RNNT. Let $z^{\tilde{t}} \in \mathbb{R}^{T \times (U+1) \times V}$ be the joint logits produced by the network when the encoder runs in teacher mode (offline, with full context), and $z^{\tilde{s}} \in \mathbb{R}^{T \times (U+1) \times V}$ be the logits produced when the encoder runs in student mode (streaming, with restricted context). At each alignment point $(t, u)$ , define the probability distributions:

$p = \text{softmax}(z_{t,u,:}^{\tilde{t}}) \quad \text{and} \quad q = \text{softmax}(z_{t,u,:}^{\tilde{s}})$

where $p$ and $q$ are probability vectors of length $V$ over the vocabulary at alignment position $(t, u)$ for the offline and streaming modes respectively.

The KL divergence from $p$ to $q$ (i.e., using offline as the teacher distribution) is:

$\mathcal{L}_{MCR} = \mathcal{L}_{KL}(p \| q) = \sum_{v=1}^{V} p_v \left(\log p_v - \log q_v\right)$

where $p_v$ is the $v$ -th element of the offline probability vector, $q_v$ is the $v$ -th element of the streaming probability vector, and the sum runs over all $V$ output classes.

What this computes: for each $(t, u)$ alignment position, it measures how much information is lost if we use the streaming encoder's output distribution $q$ to approximate the offline encoder's output distribution $p$ . KL divergence is non-negative and equals zero only when $p = q$ exactly. The term $\log p_v - \log q_v$ inside the sum means that positions where $p_v$ is large (the offline model is confident) but $q_v$ is small (the streaming model disagrees) contribute the most to the loss. Conversely, positions where both distributions agree contribute little.

Symmetric variant. The paper also investigates a symmetric form that averages KL divergence in both directions:

$\mathcal{L}_{MCR\text{-}Sym} = \frac{1}{2} \left[ \mathcal{L}_{KL}(p \| q) + \mathcal{L}_{KL}(q \| p) \right] = \frac{1}{2} \sum_{v=1}^{V} (p_v - q_v)(\log p_v - \log q_v)$

where the second equality is a standard algebraic identity for symmetric KL divergence. The expression $(p_v - q_v)(\log p_v - \log q_v)$ is always non-negative (a product of two terms with the same sign or zero), and equals zero when $p_v = q_v$ .

Why symmetric KL: asymmetric KL (offline as teacher) only penalizes the streaming model for failing to match the offline distribution — it imposes no constraint in the reverse direction. The streaming model could learn to produce a highly confident but incorrect distribution, and the asymmetric loss would not penalize it as long as the offline distribution also happened to be confident in the same wrong region. Symmetric KL penalizes disagreement in both directions: the streaming model is pushed to match the offline distribution, and the offline model is pushed to stay close to the streaming distribution. This prevents either mode from drifting into a region of output space that the other mode cannot represent, maintaining a shared representational basin.

Ablation results on KLD variant: Table 2 shows that symmetric KLD with weight $\lambda = 0.3$ yields the best trade-off between offline and streaming performance. The paper's early experiments also tried "computing KLD over $\{p_{\text{blank}}, p_{\text{target}}, 1 - p_{\text{blank}} - p_{\text{target}}\}$ probability distribution, and also a separate variant of using lattice posteriors instead of output probabilities." The full-joint KLD outperformed both of these alternatives in terms of stability and final accuracy. The three-class variant (blank, target, other) is an aggressive dimensionality reduction that discards information about which specific non-target tokens the model is considering; the lattice-posterior variant applies the KLD to a sparser representation that may lose information about low-probability alternatives. Both are less informative than direct KLD over the full $V$ -way categorical distribution.

Per-utterance normalization. The per-utterance MCR loss is reduced by normalizing the sum over valid $(t, u)$ positions. Not all $T \times (U+1)$ alignment points are meaningful — many represent alignments that are impossible given the audio length and token sequence length (e.g., emitting more tokens than there are audio frames). The loss is averaged over the valid lattice positions, which are determined by the standard RNNT forward-backward grid boundaries. This ensures that utterances of different lengths contribute roughly equally to the total loss, regardless of how many valid alignment points they contain.

Connection to the full training objective. The final loss for dual-mode training with MCR-RNNT is:

$\mathcal{L}_{\text{DM}} = \alpha \, \mathcal{L}_{\text{RNNT}}^{\text{off}} + (1 - \alpha) \, \mathcal{L}_{\text{RNNT}}^{\text{str}} + \lambda \, \mathcal{L}_{\text{MCR}}$

where $\lambda \geq 0$ controls the strength of the consistency regularization. The paper's ablation (Table 2) tests $\lambda \in \{0.1, 0.3, 1.0\}$ and finds $\lambda = 0.3$ optimal — strong enough to meaningfully reduce the offline-streaming gap, but not so strong that it dominates the primary RNNT losses and prevents the model from achieving good accuracy in either mode individually.

Why MCR-RNNT over alternatives:

CR-CTC [Yao2024CRCTCCR] failed on this task. The paper reports that extending CR-CTC consistency to unified ASR "consistently degraded streaming RNNT accuracy" despite maintaining offline quality. The key difference is the objective function: CTC is a frame-synchronous loss that encourages locally confident predictions at each time step independently. Under streaming constraints with limited right context, the encoder cannot gather enough evidence to make confident frame-level predictions, so forcing consistency with offline CTC outputs (which can be highly confident due to bidirectional context) pulls the encoder toward producing overconfident but wrong frame-level outputs in streaming mode. In contrast, RNNT operates over the full alignment lattice — it can defer decisions by emitting blank tokens until sufficient evidence accumulates. MCR-RNNT applies consistency at the lattice level rather than the frame level, which is compatible with the streaming encoder's need to accumulate evidence over time before committing to tokens.
TCR [Tseng2024TransducerCR] targets augmentation consistency, not mode consistency. TCR applies KL divergence to pruned RNNT outputs generated from different augmented views of the same audio (e.g., with and without SpecAugment). The alignment space for augmented views is roughly similar — different augmentations of the same utterance have the same length and similar acoustic structure. The alignment space between offline and streaming modes is fundamentally different: the offline encoder has full bidirectional context and can produce confident predictions early in the utterance (peeking ahead to disambiguate), while the streaming encoder must wait until sufficient right context arrives. The pruning strategy used in TCR (keeping only high-probability alignments) may inadvertently discard precisely the alignment positions where the offline-streaming divergence is largest — positions where the offline model has already disambiguated a token that the streaming model is still uncertain about. MCR-RNNT operates on the full (unpruned) lattice, ensuring that all alignment points contribute to the consistency signal.
No public TCR implementation available. The paper notes this as a practical motivation for developing their own implementation, but the deeper reason is that adapting TCR's occupation-based weighting to the offline-streaming setting would require non-trivial modifications, since the occupation probabilities (the probability of being at a particular $(t, u)$ position) differ substantially between modes.

Triton Implementation of the MCR-RNNT Kernel

The MCR-RNNT loss as defined involves computing softmax over a $V$ -dimensional vector at every $(t, u)$ position in the joint lattice, then computing the KLD between pairs of these vectors. For a typical configuration — $T \sim 200$ time steps, $U \sim 50$ tokens, $V = 1024$ vocabulary items — the full joint tensor has shape $[B, T, U+1, V]$ where $B$ is the batch size. Materializing this tensor in 32-bit floating point would require approximately $200 \times 51 \times 1024 \approx 10.5\text{M}$ floats per utterance, or tens of gigabytes across a batch. This is infeasible for GPU memory.

The on-the-fly computation strategy. The MCR-RNNT kernel avoids materialization by computing $\log\text{softmax}$ and KLD inside a fused GPU operation that writes only the scalar loss per $(t,u)$ position, with gradients computed analytically during the backward pass. The workflow:

Forward pass: The kernel receives the raw logits $z^{\tilde{t}}$ and $z^{\tilde{s}}$ at each $(t,u)$ position. It computes $\text{softmax}$ for both vectors in registers (temporary GPU memory that doesn't persist), computes the KLD scalar (or symmetric KLD scalar), accumulates it into the per-position loss, and immediately discards the softmax vectors. The only output stored is the per-position KLD values (a $T \times (U+1)$ tensor of scalars), which is vastly smaller than the full probability tensor.
Backward pass: To compute gradients with respect to the logits, the kernel recomputes $\text{log}\text{softmax}$ and $\text{softmax}$ from the original logits, uses these to analytically derive $\frac{\partial \mathcal{L}_{KL}}{\partial z^{\tilde{s}}}$ and $\frac{\partial \mathcal{L}_{KL}}{\partial z^{\tilde{t}}}$ , and writes the gradients back to the input tensors. This recomputation strategy mirrors the memory-saving technique used in the RNNT loss implementation in NeMo [kuchaiev2019nemo], where the forward pass stores only the minimum necessary to reconstruct the full computation during the backward pass.

Why Triton: Triton [tillet2019triton] is a Python-based GPU programming framework that compiles to near-CUDA performance while being significantly easier to maintain and deploy across different GPU architectures. The paper cites Triton's portability and maintainability as the primary reasons for choosing it over raw CUDA. In the context of an open-source release, Triton code can be inspected, modified, and run by the community without requiring a CUDA toolchain or architecture-specific compilation.

Memory and computational overhead. The paper states that this design "imposes nearly zero memory overhead and tiny computational overhead compared to RNNT loss." Quantitatively: the RNNT loss itself is already the dominant compute cost during training (it requires a forward-backward pass over the $T \times U$ lattice). The MCR-RNNT kernel adds a per-position KLD computation over $V$ elements, which is linear in $V$ and parallelizable across $(t, u)$ positions. Compared to the RNNT loss's lattice computation (which is also linear in $V$ but involves more complex recurrence relations), the MCR-RNNT overhead is a small constant factor — likely 5–20% of the RNNT loss computation time, since both operations process the same-sized tensors with similar arithmetic intensity.

Design Choices Summary

Architectural adaptations (chunk-limited attention + DCConv): selected because they structurally prevent the model from learning dependencies on future context that would be unavailable during streaming inference, while preserving the model's ability to use future context when it is available (offline mode). The alternative — training a dedicated streaming model from scratch — sacrifices offline quality. The alternative — using causal convolutions and zero-right-context attention — permanently removes useful future information even when it could be provided at higher latencies.

Dual-mode training over single-mode: dual-mode couples the two optimization objectives within each training step, providing direct gradient information about how parameter updates affect both modes simultaneously. The paper's results show that DM alone does not outperform SM (Table 1), but DM provides the necessary structure for MCR-RNNT — the two forward passes produce the paired logits that the consistency loss operates on. Without DM, MCR-RNNT would be impossible because there would be no paired (offline, streaming) logits to compare.

Symmetric KL divergence over asymmetric: symmetric KL penalizes disagreement in both directions, preventing either mode from drifting away from the shared representation. Asymmetric KL would only constrain the streaming model to match the offline model, which could allow the offline model to occupy a region of output space that the streaming model cannot faithfully represent with limited context — the streaming model would incur high consistency loss without a corresponding pressure on the offline model to stay within reach.

Full joint logits over pruned or compressed representations: the paper's early experiments found that simpler variants (three-class probability distributions, lattice posterior KLD) were less stable and performed worse. The full-joint approach preserves all information about the model's output distribution at every alignment point, giving the consistency loss maximum signal to identify and penalize offline-streaming divergence. The Triton kernel makes this computationally feasible despite the large tensor sizes.

Vocabulary size of 1024 BPE tokens: a relatively small vocabulary by modern LLM standards, which makes the full-joint KLD computation more practical. A larger vocabulary (e.g., 32K tokens) would increase the KLD overhead proportionally, potentially requiring further optimization (vocabulary pruning, top-K softmax approximations, or sparse KLD variants). The paper does not discuss scaling to larger vocabularies.

Offline weight $\alpha = 0.5$ as recommended starting point: balances the primary RNNT losses equally between modes, letting the consistency loss handle the trade-off. The ablation in Table 2 shows that adjusting $\alpha$ can shift the balance — higher $\alpha$ improves offline WER at the cost of streaming WER, and vice versa — but 0.5 is a reasonable default for deployments where both modes are equally important.

No cache-passing for streaming inference (noted as future work): the current implementation recalculates the left context at each chunk step rather than caching and passing previous encoder states. This simplifies the implementation (stateful caching requires careful management of KV-cache for attention and hidden states for convolutions) but slows inference speed, since the left context frames are processed redundantly at each step. The paper explicitly flags cache-passing as future work, indicating that the current release prioritizes accuracy over inference throughput.

4. Key Insights and Innovations

Innovation 1: The Output-Level Mode Gap as the Bottleneck, Not the Architectural Mismatch

The paper's most important conceptual move is diagnosing where unified ASR models fail at low latency. The field has spent considerable effort on architectural adaptations — chunk-limited attention, causal convolutions, DCConv, state-space model replacements — all of which operate at the encoder representation level. The implicit assumption has been that if you can make the encoder produce reasonable hidden states under streaming constraints, the rest of the Transducer pipeline will follow.

This paper identifies that assumption as incomplete. The evidence comes from a revealing negative result: even with chunk-limited attention and DCConv structurally adapting the encoder (which the paper's own baselines confirm improve streaming performance substantially over a naive offline model — compare the streaming baseline's 9.44% WER at 0.32s to the offline baseline's catastrophic 49.46%), unified DM training still degrades sharply below ~0.5s latency, reaching 16.91% WER at 0.24s versus 10.01% for the dedicated streaming model (Table 1). The encoder is architecturally capable of processing chunked audio — the problem isn't that it can't represent streaming inputs — but the representations it produces under the two modes have diverged enough at the output level that the same joint network and predictor can't reconcile them.

This shifts the diagnosis from "the encoder needs better structural adaptation" to "the encoder needs to produce output distributions that are mode-invariant, not just hidden states that survive chunking." The distinction matters because architectural changes (like DCConv) operate on the encoder's internal representations without any explicit constraint that those representations map to the same vocabulary predictions. You can have an encoder that processes chunked audio perfectly well but produces hidden states that, when fed through the joint network, systematically favor different tokens than the offline encoder's hidden states do — because the joint network was implicitly trained to exploit bidirectional disambiguation cues that streaming representations can't provide.

This diagnostic move is significant beyond the raw performance gains because it reframes the unified ASR problem from an architectural adaptation problem (how do we build an encoder that handles both modes?) to a representation consistency problem (how do we ensure the encoder maps the same acoustic event to the same output distribution regardless of available context?). This reframing opens up solution approaches beyond architecture — regularization, distillation, adversarial training — that were not obvious when the field was focused on attention masks and convolution padding.

The paper's self-described failure with CR-CTC (Section 2.4) reinforces this diagnosis: applying consistency at the CTC posterior level — which is frame-synchronous — degraded streaming RNNT accuracy because the CTC objective's demand for locally confident predictions is fundamentally incompatible with low-latency streaming, where the encoder must defer decisions until evidence accumulates. The level at which consistency is applied (frame-synchronous CTC vs. lattice-structured RNNT) is the critical design choice, and getting it wrong actively harms performance. This is a non-obvious finding: consistency regularization sounds universally beneficial, but applied at the wrong representational level it introduces an objective mismatch that the shared encoder cannot resolve.

Innovation 2: Consistency at the Joint Lattice Level as a Principled Alternative to Frame-Level Agreement

The paper's second conceptual contribution is identifying the RNNT joint lattice as the correct level for consistency regularization in unified Transducer training. This isn't just "let's try KL divergence on the outputs" — it reflects a specific understanding of what information the Transducer's intermediate representations encode and why consistency at this level is compatible with both offline and streaming computation.

Prior consistency approaches in ASR operated at different levels: CR-CTC [Yao2024CRCTCCR] at the frame-level CTC posterior (which the paper showed is incompatible with streaming RNNT); TCR [Tseng2024TransducerCR] at pruned RNNT outputs but for augmentation consistency rather than mode consistency, with a pruning strategy that may discard the alignment positions where offline-streaming divergence is largest. The paper's contribution is recognizing that the full, unpruned $T \times (U+1) \times V$ joint lattice — the raw logits at every alignment point before any search or pruning — is the natural representation for mode consistency in a Transducer because:

It preserves alignment flexibility. The RNNT joint lattice encodes all possible alignments between audio and tokens, including those where the model defers decisions by emitting blank tokens. A streaming encoder with limited right context will naturally produce different alignment preferences than an offline encoder (more blanks early in the utterance, since it can't disambiguate yet), but this doesn't mean the output distributions at corresponding alignment points should differ. By applying consistency at the lattice level, the loss sees the model's uncertainty at every position without committing to a specific alignment path.
It avoids the objective mismatch that killed CR-CTC. The CTC loss forces the model to commit to frame-level token predictions, which requires local confidence that streaming encoders cannot provide with limited right context. The RNNT joint lattice allows the model to express uncertainty through blank emissions — the consistency loss penalizes the streaming and offline distributions for diverging where they both have opinions, but doesn't force the streaming model to be confident at positions where it legitimately lacks evidence. The blank token provides an escape hatch: the streaming model can learn to output high blank probability at positions where it's waiting for future context, and the consistency loss will only penalize it if the offline model has a different blank probability at that position.
The full joint preserves information that pruning discards. The paper's ablation finding that full-joint KLD outperforms simplified variants (three-class distributions, lattice posteriors) suggests that low-probability alternative tokens carry important consistency signal. When the offline model is confident about token A but the streaming model is split between A and B, that information is present in the full distribution but lost in a top-1 or three-class aggregation. The Triton kernel implementation makes this computationally feasible, converting what would be an impractical memory requirement into a negligible overhead — this is as much an enabling contribution as a methodological one, since it removes the computational barrier that might otherwise force researchers toward lossy simplifications.

This innovation is a reframing of what consistency means for sequential prediction with alignment freedom, not just a new loss function. The deeper insight is that consistency should be enforced at the most expressive level of the model's output representation — the lattice — and that doing so is compatible with the Transducer's ability to defer decisions via blank emissions in a way that frame-synchronous approaches are not.

Innovation 3: The Failure of CR-CTC as a Diagnostic Finding, Not Just an Implementation Detail

The paper's negative result with CR-CTC — the observation that extending frame-level CTC consistency to unified ASR "consistently degraded streaming RNNT accuracy" — is reported as a brief aside in Section 2.4, but it is arguably one of the paper's most intellectually significant contributions. It establishes a boundary condition for consistency regularization in ASR that was not previously documented.

Before this work, a reasonable researcher might have assumed that consistency regularization is broadly beneficial — if you penalize the model for producing different outputs under different context conditions, you should get more robust representations. The CR-CTC result demonstrates that this assumption is dangerously false when the consistency loss and the primary task loss impose conflicting demands on the shared encoder at the same representational level. Specifically: the CTC loss optimizes for frame-level confidence ("what is the most likely token at this exact time step?"), while low-latency streaming RNNT requires the encoder to accumulate evidence over time before committing. Enforcing consistency between offline CTC (highly confident frame-level predictions) and streaming CTC (necessarily uncertain frame-level predictions) forces the encoder toward a compromise that serves neither master well — it can't produce confident frame predictions in streaming mode because it lacks future context, but it's being penalized for not matching the offline model's confidence. The shared encoder ends up in a representational no-man's-land.

This negative result has broader implications beyond this specific paper:

It suggests that consistency regularization in sequence models is representation-level-dependent in a strong sense. You can't just apply KL divergence between any two output distributions — you need to ensure that the distributions being compared are produced by objectives that are compatible at that representation level. Frame-synchronous (CTC) and lattice-based (RNNT) objectives impose fundamentally different local confidence requirements, and consistency across that boundary is harmful.
It rules out a natural-sounding extension of prior work. CR-CTC [Yao2024CRCTCCR] was a published approach for CTC-based models. A straightforward extension to unified RNNT training would be to add CR-CTC consistency as an auxiliary loss. The paper demonstrates that this fails, and provides a mechanistic explanation for why (the objective mismatch), which prevents future researchers from independently re-discovering this failure mode.
It redirects research effort toward lattice-level consistency. By showing that frame-level consistency is actively harmful, the paper implicitly argues that any consistency approach for Transducer models must operate at the joint lattice level, where the blank token provides the flexibility needed for the streaming encoder to express appropriate uncertainty. This is a specific, falsifiable claim that shapes the research agenda.

The CR-CTC result is a negative diagnostic finding that carries conceptual weight beyond the positive MCR-RNNT results. It transforms "consistency regularization might help" into "consistency regularization helps only if applied at the right representational level, and the wrong level makes things worse" — a much sharper and more actionable claim.

Innovation 4: Unification as a Representation Stability Property, Not Just an Architectural Property

The paper's scaling results (Table 1, bottom rows: 600M-parameter models on 280K hours) demonstrate something subtler than "bigger models and more data improve WER." They show that the unified training framework's advantage over baselines is preserved — and in some regimes, amplified — as model and data scale increase. This distinguishes MCR-RNNT from approaches that might work only in small-model, small-data regimes where the model lacks capacity to handle both modes separately and benefits from any regularization.

Consider the baseline landscape at scale: Parakeet-TDT-0.6b-v2 achieves 6.04% offline WER (excellent) but collapses to 69.55% WER at 1.12s latency and 99.47% at 0.24s — it is not a unified model, and adding moderate latency destroys it. Nemotron-Speech-Streaming-En-0.6b achieves 7.05% offline and 7.08% at 1.12s streaming (good unified performance at moderate latency) but degrades to 7.92% at 0.16s — it handles low latency but without the offline quality ceiling of a dedicated offline model. The paper's Unified DM + MCR-RNNT 0.6B (balanced) achieves 5.91% offline (better than either baseline) and 7.35% at 0.24s (competitive with the streaming specialist at that latency), plus 6.92% at 0.32s (substantially better than Nemotron-Streaming's 7.78%).

The conceptual significance: this is not just "our model is better." It demonstrates that a single parameter set can simultaneously push the offline quality frontier (5.91% WER approaches the best reported offline-only result of 5.63% from Canary-Qwen-2.5B, a much larger 2.5B-parameter model) and match streaming-specialist models at low latency. The consistency regularization is doing something beyond acting as a tiebreaker in an underparameterized regime — it is enabling a large, highly expressive model to allocate its representational capacity in a way that serves both modes without conflict, rather than forcing the model to choose which mode to optimize.

This reframes unification from a compromise (sacrifice some offline quality to get streaming, or vice versa) to a representation stability property that can be explicitly optimized. The model isn't finding a middle ground between two competing objectives — it's learning representations that are stably informative regardless of context availability, extracting the same linguistic information from the acoustic signal whether or not the future is visible. This is a different aspiration than prior unified training work, which typically accepted some degradation in both modes relative to dedicated models.

The paper's open-source release and Triton kernel implementation reinforce this reframing by making the approach accessible — it's not a one-off result dependent on proprietary infrastructure, but a method that can be adopted, reproduced, and built upon. The release of both the framework and a SOTA model checkpoint signals that the authors view this not as an isolated research contribution but as a foundation for future work on unified ASR, where mode consistency is a first-class optimization target alongside accuracy.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use labeled English speech with normalized transcripts, drawn from the public Granary dataset [Koluguri2025GranarySR]. L-size models (128M parameters) are trained on a subset of roughly 120,000 hours. XL-size models (600M parameters) are trained on 280,000 hours including punctuation and capitalization (PC). For evaluation, the paper uses the Open ASR Leaderboard for English [srivastav2025openasrleaderboardreproducible], computing average WER across eight diverse test sets: AMI, Earnings22, Gigaspeech, Librispeech, SPGI, TEDLIUM, and VoxPopuli open test sets. The paper argues that testing across such varied domains yields more robust results than single-benchmark evaluation, which is particularly important for unified models that must handle both formal read speech (Librispeech) and spontaneous conversational speech (AMI, Earnings22) under both offline and streaming conditions.
Base model(s). The primary architecture is an RNNT Transducer with a FastConformer encoder [rekesh2023fastconformer] at roughly 128M parameters (L-size). The encoder uses 128-dimensional FBanks with 8× initial subsampling (making each encoder frame span 80ms of input audio). The prediction network is a single-layer LSTM with 640 units, bringing the total to 128M parameters. For scaling experiments, an XL-size model with approximately 600M parameters is trained. The paper states that FastConformer is a standard, well-performing encoder architecture, and the 128M parameter scale is representative of widely deployed production ASR models — large enough to achieve competitive accuracy but small enough for efficient training iteration.
Metrics. The primary metric is Average Word Error Rate (AVG WER, %) computed across the eight Open ASR Leaderboard test sets using the leaderboard's standard evaluation protocol. For streaming evaluation, an additional dimension is theoretical worst-case latency, defined as the sum of the chunk size C and right context R in seconds (after accounting for the 8× subsampling where 1 frame = 80ms). Latency values range from 2.08s down to 0.16s. This paired metric — WER at a specific latency — captures the accuracy-efficiency tradeoff that is the central concern of unified ASR: the goal is low WER across all latencies, not just at one operating point. The paper reports WER at seven distinct latency points for each model (Table 1), providing a latency-accuracy curve rather than a single scalar.
Baselines. The paper evaluates against six distinct comparison points, spanning different architectural and training paradigms:
- Baseline (Offline): a FastConformer RNNT trained exclusively in offline mode with standard full-context attention and centered convolutions. This represents the ceiling for offline accuracy but has no streaming capability.
- Baseline (Streaming): a cache-aware streaming model using the multi-look-ahead approach from Noroozi et al. (2023) [Noroozi2023StatefulCW], trained with chunk-limited attention and causal convolutions. The attention mask uses fixed left context of 70 frames and dynamically sampled look-ahead values from [13, 6, 1, 0], with no right context beyond the current chunk by design. This represents the ceiling for streaming-only performance.
- Mamba2 + DCConv (Streaming): a streaming variant that replaces the MHA sub-layers with Mamba2 state-space blocks [Dao2024TransformersAS] while retaining DCConv for the convolution sub-layers. Chunk sizes for DCConv are sampled from [1, 2, 7, 13], with 50% probability of switching between shared full-context convolutions and DCConv.
- Unified single-mode (SM): the paper's own unified training framework running in single-mode, where each training step randomly selects offline or streaming mode and computes gradients only for that mode, using the same attention mask and DCConv configurations as the proposed approach.
- Unified dual-mode (DM): the unified framework in dual-mode without consistency regularization, computing both offline and streaming losses on the same batch and combining them with offline weight α. This is the direct ablation target for MCR-RNNT — isolating the effect of the consistency loss.
- Parakeet-TDT-0.6b-v2 [model:tdt-v2] and Nemotron-Speech-Streaming-En-0.6b [model:nemotrom-streaming]: two open-source models at comparable scale (around 600M parameters), serving as strong external baselines for the XL-size scaling experiments. Parakeet-TDT is a TDT (Token-and-Duration Transducer) model trained on the same Granary dataset and optimized primarily for offline accuracy; Nemotron-Streaming is a unified streaming-capable model. The Canary-Qwen-2.5B model [model:canary-qwen] is mentioned as a reference point for offline-only SOTA (5.63% AVG WER), though it is a much larger 2.5B parameter pure offline model and not a direct comparison target for unified training.
Generation budget / compute accounting. All L-size models (both baselines and the proposed approach) are trained for 100K steps using a cosine annealing learning rate schedule with maximum LR of 1e-3 and 15K steps of warmup, on 32 NVIDIA A100 GPUs with dynamic bucketing [elasko2024EMMeTTEM]. XL-size models are trained for 300K steps with LR of 5e-4. For fair comparison, the dual-mode training experiments reduce batch size by half to match the total computational cost of single-mode training and baselines (since DM performs two encoder forward passes per step). This batch-size adjustment is critical: without it, the DM variants would have an unfair advantage of effectively seeing more data per step, or an unfair disadvantage if the batch size were kept the same but gradients were noisier due to memory constraints. The paper explicitly states that this keeps "computational complexity" comparable, though it does not report exact GPU-hours or FLOP counts. Inference evaluation uses efficient greedy decoding [bataev2024labellooping, galvez24_speedoflight] with batch size 128 for both offline and streaming modes, keeping inference cost comparable across models.
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. The 120K-hour training set and 8-test-set evaluation provide substantial data diversity, but the lack of confidence intervals or test-retest variance means that small WER differences (e.g., 6.63% vs. 6.69% offline WER between DM+MCR-RNNT and Unified DM) should be interpreted cautiously. However, the consistent pattern across latencies — where the proposed method outperforms baselines at nearly every latency point rather than trading off — provides informal robustness. The scaling experiments at 600M parameters with a different data mix (PC data) serve as a partial replication, testing whether the method's advantages persist under different training conditions. The paper's ablation study (Table 2) systematically varies KLD type, consistency weight λ, and offline weight α, but reports only mean WER without variance estimates.

Main Quantitative Results

L-Size Models (128M Parameters) on 120K Hours

Headline: MCR-RNNT achieves the best streaming WER of any method at every latency from 2.08s down to 0.24s, while matching or exceeding the offline-only baseline's offline WER, establishing a new Pareto frontier for unified RNNT models. Table 1 (top half) presents the full latency-accuracy curves.

Offline performance. The offline-only baseline achieves 6.47% AVG WER — the theoretical ceiling for offline accuracy with this architecture and data. The proposed Unified DM + MCR-RNNT achieves 6.63% offline WER, which is:

Slightly worse than the offline-only baseline (−0.16 percentage points), indicating that unified training imposes a minimal but non-zero offline accuracy cost compared to a dedicated offline model.
Better than the streaming-only baseline (7.75%, −1.12 pp), confirming that unified training preserves the ability to exploit bidirectional context when available, unlike a streaming-specialized model.
Better than Unified SM (6.66%, −0.03 pp) and Unified DM without MCR (6.69%, −0.06 pp), both of which are themselves reasonably close to the offline baseline. This suggests that the primary offline accuracy gap is due to the architectural adaptations (chunk-limited attention, DCConv) rather than the training strategy — all unified models cluster in the 6.63–6.69% range, roughly 0.2 pp above the offline-only baseline.
Notably, the streaming-only baseline achieves 7.75% offline WER, substantially worse than any unified variant. This is the cost of causal convolutions and zero-right-context attention: the model structurally cannot use future information even when it is available, permanently sacrificing roughly 1.3 pp of offline WER.

Streaming performance at higher latencies (2.08s–0.56s). At 2.08s latency (C=13, R=13 frames), the proposed method achieves 6.86% WER, compared to:

Offline baseline: 6.92% — remarkably close, indicating that with roughly 2 seconds of right context, the streaming mode nearly recovers offline-level accuracy.
Streaming baseline: 8.39% — the dedicated streaming model is substantially worse, since it was trained with zero right context and cannot exploit the generous look-ahead.
Unified SM: 7.71% and Unified DM: 7.14% — MCR-RNNT provides a clear advantage over both standard unified training variants at this latency.
The pattern holds through 0.56s latency (MCR-RNNT: 7.47% vs. Unified SM: 7.98%, Unified DM: 8.12%), with DM actually underperforming SM at 0.56s — an early indication that simple loss mixing is insufficient at moderate latency constraints.

Streaming performance at low latencies (0.40s–0.24s). This is where MCR-RNNT's advantage becomes dramatic:

At 0.40s latency: MCR-RNNT achieves 7.83% vs. Unified SM 9.40% (−1.57 pp) and Unified DM 9.86% (−2.03 pp). The gap has widened substantially.
At 0.32s latency: MCR-RNNT achieves 8.24% vs. Unified SM 10.96% (−2.72 pp) and Unified DM 12.48% (−4.24 pp). DM without consistency regularization is degrading sharply — it is now 4.24 pp worse than the streaming baseline (9.44%) at this latency, making it essentially non-competitive for low-latency applications.
At 0.24s latency: MCR-RNNT achieves 9.04% vs. Unified SM 13.33% (−4.29 pp), Unified DM 16.91% (−7.87 pp), and the streaming baseline 10.01% (−0.97 pp). MCR-RNNT is the only unified variant that outperforms the dedicated streaming baseline at low latency (9.04% vs. 10.01%), while simultaneously maintaining near-offline-level accuracy. This is the central empirical claim of the paper: the consistency loss closes enough of the mode gap to make a single model competitive with dedicated models in both regimes simultaneously.

At the most extreme latency (0.16s): MCR-RNNT achieves 10.51%, slightly worse than the streaming baseline's 9.84% (+0.67 pp). This is the one latency point where the dedicated streaming model retains an advantage, suggesting that at near-zero look-ahead (80ms chunk + 80ms right context), the conflict between offline and streaming representations becomes sufficiently severe that even consistency regularization cannot fully resolve it. However, MCR-RNNT still dramatically outperforms Unified SM (17.16%, −6.65 pp) and Unified DM (22.45%, −11.94 pp), showing that while it doesn't match the streaming specialist, it prevents the catastrophic degradation that standard unified training exhibits.

The Mamba2 + DCConv streaming variant achieves WERs ranging from 8.41% at 2.08s to 10.52% at 0.16s — comparable to the attention-based streaming baseline across the latency range. This serves as an architectural ablation, confirming that DCConv provides effective streaming support regardless of whether the sequence-mixing mechanism is attention or a state-space model. Since Mamba2 does not outperform attention and is not the paper's focus, it is presented primarily as a robustness check.

The offline baseline's catastrophic streaming degradation is worth noting explicitly: 6.47% offline → 94.05% at 0.16s streaming. This is the baseline that naive "just run the offline model in chunks" would achieve. It validates that the architectural adaptations (chunk-limited attention, DCConv) are absolutely necessary — the model has learned to rely on bidirectional context to such an extent that removing it produces essentially random output. The improvement from 94.05% to ~10% at 0.16s latency is entirely due to training with explicit context restrictions; the further improvement to 9.04% at 0.24s (with MCR-RNNT) is the incremental gain from consistency regularization.

XL-Size Models (600M Parameters) on 280K Hours with Punctuation and Capitalization

Headline: The proposed unified training with MCR-RNNT scales to production-scale models and data, establishing a new SOTA for unified RNNT models (5.76% offline WER) while maintaining competitive streaming performance down to 0.24s latency. Table 1 (bottom half) presents these results.

Two model variants are reported, differing in their right-context training distribution:

Model (1), "larger right context": trained with larger right-context values during unified training, prioritizing offline and higher-latency streaming performance.
Model (2), "balanced": trained with smaller right-context values, trading some offline accuracy for better low-latency streaming.

Model (1) results. At 5.76% offline AVG WER, this model:

Outperforms Parakeet-TDT-0.6b-v2 (6.04%, −0.28 pp), a dedicated offline TDT model trained on the same Granary dataset. This is notable because Parakeet-TDT is optimized solely for offline accuracy, yet the unified model achieves better offline performance while also supporting streaming.
Approaches Canary-Qwen-2.5B's 5.63% offline WER — within 0.13 pp of the SOTA offline result, despite using 4× fewer parameters (600M vs. 2.5B) and being a unified model rather than an offline specialist.
Beats Nemotron-Speech-Streaming-En-0.6b at every latency from 2.08s (5.97% vs. 7.51%) down to 0.32s (7.72% vs. 7.78%), and remains superior through 0.24s (9.51% vs. 8.18% — Nemotron takes the lead here).
At 0.16s latency, achieves 12.73% vs. Nemotron's 7.92%. The gap at the most extreme latency is larger for this model variant because it was optimized with larger right context — it simply wasn't trained to handle such tight constraints.

Model (2) results. At 5.91% offline WER — only 0.15 pp worse than Model (1) — this model:

Beats Nemotron-Streaming at all latencies except 0.16s (8.44% vs. 7.92% for Nemotron), demonstrating that balanced right-context training can nearly close the gap to a streaming specialist even at the tightest latency while maintaining near-SOTA offline quality.
At 0.24s latency, achieves 7.35% WER, substantially better than Nemotron's 8.18% and competitive with the streaming-only baseline from the L-size experiments (which achieved 10.01% at 0.24s — the XL model's improved capacity and increased training data reduce WER by roughly 2.7 pp across the board).
At 0.32s latency, achieves 6.92% — matching or exceeding the L-size model's offline WER (6.47–6.63%) in streaming mode, demonstrating how scale can partially compensate for context restrictions.
The tradeoff between Models (1) and (2) is clean: Model (1) gives up 0.15 pp offline and gains progressively more streaming advantage as latency decreases, reaching a 4.29 pp advantage at 0.16s (12.73% vs. 8.44%). This demonstrates that the right-context training distribution is the primary knob controlling the offline-streaming tradeoff at scale, with consistency regularization ensuring that the chosen tradeoff point is on the Pareto frontier rather than inside it.

Parakeet-TDT's streaming collapse mirrors the L-size offline baseline: 6.04% offline WER degrades to 69.55% at 1.12s and 99.47% at 0.24s — essentially total failure at any meaningful latency. This confirms that TDT architectures, while achieving excellent offline accuracy, are just as dependent on bidirectional context as standard RNNT models unless explicitly trained for streaming. Nemotron-Streaming, by contrast, maintains reasonable performance across the latency range (7.05% offline, 7.08% at 1.12s, 7.92% at 0.16s) but at the cost of roughly 1 pp worse offline accuracy than the proposed Model (2) — the permanent tradeoff that unified training with MCR-RNNT partially overcomes.

Ablation Studies and Robustness Checks

KLD variant and consistency weight λ (Table 2): Symmetric KLD with λ = 0.3 yields the best overall trade-off between offline and streaming performance. The paper reports that asymmetric KLD (offline as teacher, streaming as student) and the three-class probability variant (blank, target, other) were tested in early experiments and found to be less stable and lower-performing than symmetric full-joint KLD. Table 2 sweeps λ ∈ {0.1, 0.3, 1.0}: λ = 0.1 provides insufficient regularization (streaming WER remains high at low latency), while λ = 1.0 likely over-constrains the model (the paper does not provide exact numbers for all λ values in the main text, but the recommendation of λ = 0.3 is clear). The finding that an intermediate consistency weight is optimal aligns with the intuition that consistency should guide but not dominate training — the model still needs flexibility to produce mode-appropriate outputs where true acoustic ambiguity exists.

Offline weight α (Table 2): Varying α shifts the balance between offline and streaming performance as expected — higher α improves offline WER at the cost of streaming WER, and vice versa. The paper recommends α = 0.5 as a starting point, suggesting that equal weighting of the two primary RNNT losses provides a reasonable default with the consistency loss handling the fine-grained tradeoff. The fact that α is not reported as a critical hyperparameter (unlike λ) implies that MCR-RNNT is robust to moderate variations in the offline-streaming loss balance, likely because the consistency term provides a separate, more direct mechanism for managing the mode tradeoff.

Chunk size vs. right context allocation under fixed latency (Figure 2): At a fixed total latency budget (C + R constant), allocating more budget to right context R (larger look-ahead) consistently improves WER, particularly at lower total latencies. In other words, given a choice between a larger processing chunk with less look-ahead versus a smaller chunk with more look-ahead, the model benefits more from additional future context than from larger batch processing. This finding has practical implications for streaming system design: when latency budgets are tight, prioritize right context over chunk size. The effect is most pronounced at 0.32s total latency, where the WER difference between the most R-heavy and most C-heavy configurations appears to be roughly 1.5–3 percentage points (estimated from Figure 2, which uses LibriSpeech test-other WER specifically rather than the full leaderboard average).

PRM aggregation strategy — note: this ablation from the example is unrelated to this ASR paper. The correct ablations for this paper are:

CR-CTC negative result (Section 2.4): This is the most important negative result. The paper explicitly reports that extending CR-CTC consistency to unified ASR "consistently degraded streaming RNNT accuracy" despite maintaining offline performance. The mechanistic explanation — that CTC's frame-synchronous objective conflicts with streaming RNNT's need to defer decisions — is supported by the observed degradation pattern. While the paper does not provide a full table of CR-CTC results (presumably because the degradation was clear enough to abandon the approach), this negative finding is critical because it rules out the most obvious alternative consistency strategy and justifies the complexity of the full-joint MCR-RNNT approach.

Full-joint KLD vs. simplified distributions (Section 2.4): Early experiments comparing KLD over the full V-way categorical distribution versus reduced representations ({p_blank, p_target, 1−p_blank−p_target}) and lattice posteriors found the full-joint approach to be "better and more stable." While specific numbers for these ablations are not provided in the paper, the stability claim is notable — simplified distributions may produce noisier gradients because they discard information about low-probability alternative tokens that carry consistency signal.

ReST^EM revision model — this ablation from the example is unrelated. Correctly:

DCConv probability (0.5, mentioned in Section 3.3): Training with DCConv activated 50% of the time (versus standard full-context convolutions the other 50%) was selected based on initial parameter search. The paper does not provide a full ablation of this probability, but the results suggest it is sufficient — the streaming performance improvements over the offline baseline indicate that even 50% DCConv exposure is enough to prevent the convolution sub-blocks from learning dependencies on future context across chunk boundaries.

Single-mode vs. dual-mode (Table 1): This is effectively an ablation of the training strategy. SM consistently outperforms DM at low latencies for L-size models (13.33% vs. 16.91% at 0.24s, 17.16% vs. 22.45% at 0.16s), which is a non-obvious result — one might expect that coupling the two modes within each step would produce better shared representations. The paper does not provide a mechanistic explanation for why DM underperforms SM without consistency regularization, but a plausible interpretation is that DM's coupled gradients create a more challenging optimization landscape where the two objectives pull in conflicting directions simultaneously, whereas SM's interleaved independent gradients allow the model to make progress in each mode without interference. MCR-RNNT resolves this by providing an explicit consistency signal that aligns the gradients from the two modes.

Attention-based vs. Mamba2 streaming (Table 1): The Mamba2 + DCConv streaming variant achieves WERs within 0.02–0.9 pp of the attention-based streaming baseline across latencies (e.g., 8.41% vs. 8.39% at 2.08s, 10.52% vs. 9.84% at 0.16s). This confirms that DCConv is the critical architectural component for streaming — the choice of sequence-mixing mechanism (attention vs. state-space model) is secondary for streaming performance, at least at this scale.

Critical Assessment

Do the Experiments Support the Paper's Central Claims?

The paper makes three major claims: (1) MCR-RNNT improves streaming accuracy at low latency while preserving offline performance; (2) the approach scales to larger models and datasets; (3) the open-sourced framework establishes a new SOTA for unified RNNT. The evidence for each is strong but with specific boundary conditions that the paper is generally transparent about.

Claim 1: MCR-RNNT improves streaming accuracy at low latency while preserving offline performance. This claim is well-supported by Table 1 for L-size models, with the clearest evidence at 0.24s–0.40s latency where MCR-RNNT (7.83–9.04% WER) substantially outperforms both Unified SM (9.40–13.33%) and Unified DM (9.86–16.91%). The offline preservation claim is supported by the narrow 0.16 pp gap between MCR-RNNT (6.63%) and the offline-only baseline (6.47%). However, the claim requires two qualifications:

"Preserving offline performance" means preserving near-offline-level performance, not exactly matching it. Every unified variant loses 0.16–0.22 pp of offline WER relative to the dedicated offline model. This is a small but real cost of unification that cannot be eliminated by consistency regularization — the architectural adaptations themselves (chunk-limited attention, DCConv) slightly reduce the model's ability to exploit bidirectional context even when it is available. The paper's framing is honest about this but the headline "preserving offline quality" should be understood as "within 0.2 pp of the dedicated offline model."
The advantage over the streaming baseline disappears at 0.16s latency (10.51% vs. 9.84% for Mamba2+DCConv, 9.84% for the attention-based streaming baseline). At the tightest latency constraint tested, the dedicated streaming model still holds an edge. The paper frames this as the model being "only slightly inferior" — which is fair (0.67 pp is a modest gap) — but it means that for applications requiring sub-200ms latency, a dedicated streaming model may still be preferable. The consistency regularization pushes the crossover point (where the unified model matches the streaming specialist) from approximately 0.56s (where Unified SM/DM already degrade) down to approximately 0.24s, but does not eliminate the streaming specialist's advantage entirely at the most extreme constraint.

Claim 2: The approach scales to larger model sizes and training datasets. The XL-size results (Table 1, bottom rows) provide strong evidence, with the unified model achieving 5.76–5.91% offline WER and 7.35–8.44% at 0.24s streaming latency. However, there are important caveats about what "scales" means:

Scaling is demonstrated for one specific scaling factor (~5× parameters, ~2.3× data), not a continuous scaling curve. The paper shows that the method works at 128M and 600M parameters, but doesn't demonstrate that the advantage over baselines increases with scale or that there is a predictable scaling trend. This is a point measurement at two sizes, not a scaling law. The claim that the approach "remains effective at scale" is supported; the claim that it "scales" (implying a functional relationship) is not directly tested.
The external baselines at XL scale are not perfectly matched. Parakeet-TDT-0.6b-v2 uses a TDT architecture, not standard RNNT — architectural differences could explain some of the gap. Nemotron-Streaming-En-0.6b is a closer comparison (both are streaming-capable Transducer variants at 600M parameters), but training data differences (the exact data mix, filtering, and augmentation strategies) are not controlled. The paper's own L-size unified baselines (SM and DM) are not re-run at XL scale, so we cannot directly measure how much MCR-RNNT improves over standard unified training at 600M parameters — the external baselines demonstrate competitiveness but don't isolate the MCR-RNNT contribution at scale.
The fact that MCR-RNNT continues to outperform baselines at scale is evidence of robustness, but the mechanism may change. At L-size, MCR-RNNT closes a large gap between unified training and dedicated streaming. At XL-size, the gap between unified and dedicated models may narrow simply because larger models have more capacity to represent both modes — consistency regularization might be proportionally less important at scale. Without L-size Unified SM/DM baselines at 600M parameters, we cannot determine whether MCR-RNNT's relative contribution decreases, increases, or stays constant with scale.

Claim 3: The open-sourced framework and model establish a new SOTA for unified RNNT. The 5.76% offline WER for Model (1) and the streaming WERs down to 0.24s support this claim relative to published unified models. However, two considerations:

"SOTA for unified RNNT" is a narrow category. The paper compares against Nemotron-Streaming (same category) and Parakeet-TDT and Canary-Qwen (different categories — offline specialists). The latter comparisons demonstrate that the unified model approaches offline-specialist quality, which is impressive, but don't directly establish SOTA within the unified category since those models aren't unified. The claim holds, but the comparison set is small — there are few published unified RNNT models at this scale with comprehensive latency-accuracy curves.
The Canary-Qwen-2.5B comparison (5.63% vs. 5.76% offline) is favorable but asymmetric. Canary-Qwen is a 2.5B parameter pure offline model — 4× larger than the proposed 600M model. The fact that the unified model nearly matches it is strong evidence for the efficiency of the approach, but it's not a controlled comparison and doesn't isolate the contribution of unification or MCR-RNNT specifically.

Genuine Weaknesses

No statistical significance or confidence intervals. All WER numbers are reported as point estimates without any measure of variance. The eight-test-set average provides diversity but each individual test set has a finite number of utterances, and small WER differences (0.1–0.3 pp) could be within test-retest variance. This is particularly relevant for comparisons where the proposed method's advantage is small (e.g., 6.63% vs. 6.69% offline WER for MCR-RNNT vs. Unified DM). Without confidence intervals, the reader cannot assess whether these differences are reliable or noise.

The 280K-hour XL training set includes punctuation and capitalization, while the 120K-hour L-size set uses normalized text. This is a data distribution difference in addition to a size difference, confounding the "scaling" interpretation. The XL models are not simply L-size models trained on more data — they're trained on a different transcription style (with PC), which changes the task itself (predicting punctuation tokens in addition to words, predicting case). The WER metric should be comparable (punctuation and capitalization are typically normalized out for WER computation), but the training dynamics may differ because the model must allocate capacity to predicting punctuation and case. The paper does not discuss whether this affects the offline-streaming tradeoff or the effectiveness of consistency regularization.

The difficulty estimation cost is completely unaccounted for — this is NOT relevant to this ASR paper. Correcting:

No full ablation of right-context training distribution at L-size. The paper sweeps chunk sizes C ∈ {1, 2, 7, 13} and right contexts R ∈ {0, 1, 2, 3, 5, 7, 13, 26} for all unified training runs. The specific distribution was selected based on "initial experiments with parameter search" (Section 3.3), but no ablation is provided showing how the choice of R distribution affects the latency-accuracy tradeoff at L-size. The XL-size experiments demonstrate that shifting toward larger R improves offline/higher-latency performance at the cost of low-latency streaming, but a similar sweep at L-size would strengthen the claim that the R distribution is the primary tradeoff knob.

No computational cost analysis for MCR-RNNT. The paper claims that the Triton kernel imposes "nearly zero memory overhead and tiny computational overhead compared to RNNT loss" but provides no wall-clock time measurements, FLOP counts, or GPU memory comparisons. For practitioners deciding whether to adopt MCR-RNNT, knowing whether "tiny" means 1%, 5%, or 20% additional training time matters. Without this data, the practical efficiency of the method is asserted rather than demonstrated.

The eight-test-set evaluation is diverse but the individual test set breakdowns are not reported. Table 1 reports only the average WER across all eight test sets. For a unified model that must handle both read speech (Librispeech) and spontaneous conversational speech (AMI, Earnings22), domain-specific performance matters — the model might achieve its average by being excellent on read speech and mediocre on conversational speech, which would be suboptimal for real deployment. The paper does not provide per-test-set WER, making it impossible to assess whether the offline-streaming tradeoff varies by domain (e.g., does consistency regularization help more on spontaneous speech where acoustic disambiguation is more context-dependent?).

No comparison to simply training separate offline and streaming models of half the size. If the goal is to serve both modes with a fixed parameter budget, an alternative to unified training is to train two smaller dedicated models (e.g., two 64M-parameter models instead of one 128M-parameter unified model). The paper doesn't explore this "ensemble of specialists" baseline, which would require the same total parameters but avoid the mode conflict entirely. This is a limitation of the experimental design: the paper demonstrates that unified training is better than using a single dedicated model for both modes (which fails catastrophically for the streaming model in offline mode, and vice versa), but doesn't compare against splitting the parameter budget.

Missing Experiments That Would Strengthen the Paper

L-size Unified SM and DM baselines for the XL model scale. The most important missing ablation: without these, we cannot quantify MCR-RNNT's specific contribution at 600M parameters. If Unified DM at 600M parameters already achieves, say, 6.0% offline and 7.5% at 0.24s streaming, then MCR-RNNT's incremental contribution (5.91% offline, 7.35% at 0.24s) is modest. If Unified DM performs much worse, MCR-RNNT's contribution is large. The paper's claim that MCR-RNNT "maintains its advantages even for data and model size scaling" is plausible but not directly tested.

A controlled comparison of MCR-RNNT against a dedicated streaming model trained with identical architecture and data. The streaming baseline uses causal convolutions and no right context (multi-look-ahead values of [13,6,1,0] without a separate R parameter), while the unified model uses DCConv and right context R ∈ {0,...,26}. These architectural differences confound the comparison: when MCR-RNNT outperforms the streaming baseline at 0.24s (9.04% vs. 10.01%), is that due to MCR-RNNT or due to DCConv providing better acoustic modeling than causal convolutions, or due to the right-context training distribution including larger look-ahead values? A fair ablation would compare MCR-RNNT against a streaming model that uses DCConv and the same right-context range but is trained only in streaming mode (no offline loss), isolating the consistency loss contribution from the architectural adaptations.

Latency-accuracy curves for the external XL-size baselines at more latency points. Table 1 shows Parakeet-TDT and Nemotron-Streaming at all latency points, but these are evaluation results run by the paper's authors on their own evaluation pipeline. A useful robustness check would be to verify these numbers match the published leaderboard results for the same model checkpoints, ensuring the evaluation setup is consistent.

The paper does not discuss the failure of CR-CTC with quantitative results, despite this being an important negative finding. Even a brief table or figure showing the degradation compared to baseline would strengthen the claim that frame-synchronous consistency is fundamentally incompatible with streaming RNNT.

Where the Claims Hold Conditionally

The central claim — that MCR-RNNT closes the offline-streaming gap — holds strongly for latencies down to 0.24s at L-size (where it outperforms both standard unified training and the dedicated streaming baseline) and down to 0.32s at XL-size Model (1) and 0.16s at XL-size Model (2) (where it is competitive with the streaming specialist). The condition is latency: below roughly 0.16–0.24s, the gap to dedicated streaming models reopens, and the method's advantage over standard unified training, while still substantial, does not fully close the gap to streaming specialists.

The scaling claim holds conditionally on right-context training distribution: Model (1) demonstrates near-SOTA offline quality but weaker extreme-low-latency performance; Model (2) demonstrates balanced performance across the latency range. The method scales, but the tradeoff between offline and extreme-low-latency streaming persists at scale and must be managed through hyperparameter choices (R distribution), not eliminated by MCR-RNNT.

The SOTA claim holds for unified RNNT models specifically, where the comparison set is small but the margins over available baselines are clear (0.28 pp offline improvement over Parakeet-TDT, substantial streaming improvements over Nemotron-Streaming). The claim does not extend to all ASR architectures or paradigms.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Gains

The compute-optimal allocation policy — the central mechanism that delivers the 4× efficiency improvement over best-of-N — requires an upfront investment to determine each question's difficulty bin. Section 3.2 describes this procedure explicitly: for each question, sample 2048 complete solutions from the base model, compute the pass@1 rate (oracle) or average PRM final-answer score (predicted), then assign the question to one of five difficulty quintiles. This cost is entirely unaccounted for in the reported efficiency comparisons. The paper states in Section 3.2:

"this approach still incurs a computation cost to compute the difficulty of a question before deciding on the optimal test-time compute strategy... our experiments do not account for this cost largely for simplicity"

The consequence is that the 4× efficiency figure (e.g., 16 generations matching best-of-N at 64 generations in Figure 4) represents an upper bound, not a realized deployment gain. The difficulty estimation cost — 2048 samples per question, each scored by the PRM — dwarfs the test-time budgets under study (which range from 1 to 512 generations). In a realistic deployment, the total cost is difficulty estimation plus strategy execution. If the estimation cost is, say, 2048 generations and the strategy itself consumes 64 generations, the total is 2112 generations — a 33× increase over the reported 64, completely erasing the 4× savings over best-of-N. The paper offers no amortization argument (e.g., reusing difficulty estimates for repeatedly asked questions, caching estimates across similar prompts, or estimating difficulty with fewer samples), and it does not measure the tradeoff between estimation accuracy and estimation cost. The predicted-difficulty-bins variant removes the need for ground-truth labels but does not reduce the sample count — it still requires 2048 samples per question plus PRM scoring.

The mitigation status is weak: the paper flags this as a limitation in Section 3.2, calls it "an important area for future exploration-exploitation trade-off," and suggests future work on training models to directly predict difficulty from the question text. None of this is evaluated. For a practitioner, this means the headline 4× efficiency gain requires solving an unsolved meta-problem (cheap difficulty estimation) before it can be realized. Until that problem is addressed, the compute-optimal policy is an analytical result demonstrating what is possible in principle, not a practical deployment recipe.

The Method Provides No Path Forward for Hardest Problems

Across every experimental configuration in the paper — every search algorithm, every revision strategy, every budget level, every difficulty bin — the hardest questions (difficulty bin 5, representing the bottom quintile of the base model's pass@1 rate) show essentially zero improvement from any test-time compute allocation. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets, from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In Figure 9 (the FLOPs-matched comparison), the bin 5 scaling line is effectively flat near 0–5% while the ~14× larger model achieves substantially higher accuracy (the star markers for the larger model sit well above the scaling lines).

This is not a failure mode that can be patched with better hyperparameters or more compute. The paper articulates the root cause clearly in Section 7:

"test-time compute can amplify existing capability but does not create it from nothing — for problems where the base model's pass@1 is near zero, no amount of search or revision will help because there are no correct solutions in the proposal distribution to find or refine"

The consequence for deployment is stark: a system using compute-optimal test-time scaling with a smaller model must route hard problems elsewhere — to a larger model, to human review, or to a fallback pipeline. The compute-optimal policy itself does not solve this routing problem; it only optimizes within the base model's capability envelope. The difficulty estimator can identify hard problems (bin 5), but knowing that a problem is hard does not make it solvable. The paper does not explore what fraction of a typical deployment's query distribution falls into bin 5, which determines how often the system would need to escalate to a more expensive fallback. If bin 5 comprises, say, 20% of production queries, the small-model-plus-test-time-compute architecture still needs a secondary system for one-fifth of the workload — undercutting the economic argument for unification.

The mitigation status is nil: the paper identifies this boundary candidly but offers no solution and proposes no future work specifically targeting it. The implication is that the core research challenge shifts from "how do we allocate test-time compute?" to "how do we expand the base model's capability frontier so that fewer problems fall into bin 5?" — but this is a pretraining question, not a test-time one.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

Section 6.1 reports a specific, quantitatively significant failure mode of the revision model:

"since the model was trained only on sequences where all in-context answers are incorrect (followed by a correct target), at test time the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers. The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This is a direct consequence of the training data construction: the model is fine-tuned on trajectories of the form [incorrect, incorrect, ..., incorrect, correct], where the target is always a correction of the preceding incorrect answer. It never sees a trajectory where the current answer is already correct and the appropriate action is to output the same answer unchanged. At inference time, as the revision chain lengthens, the model eventually produces a correct answer (at pass@1 of 24–25%, per Figure 6). On the next revision step, that correct answer appears in context, and the model — having been trained exclusively to correct errors — treats it as an error to be revised, producing an incorrect answer 38% of the time.

The consequence is that the revision chain is unstable: as it grows longer, accumulated correct answers are at risk of being overwritten. The paper mitigates this with selection mechanisms — majority voting or verifier-based best-of-N applied across the entire chain rather than taking the final revision output — but these are patches on a fundamental training flaw. The selection mechanism must correctly identify which of the chain's outputs is correct, which is itself an imperfect process (the verifier makes errors, and majority voting fails when the chain oscillates). Moreover, the instability means that allocating more budget to sequential revisions (longer chains) does not monotonically improve the quality of the best answer in the chain — beyond some chain length, the rate of producing new correct answers may be offset by the rate of corrupting existing ones.

The paper does not quantify how the 38% reversion rate varies with chain position or revision model checkpoint, does not report what fraction of final answers are corrupted at least once during a typical chain, and does not ablate selection mechanisms against an oracle that always picks the correct answer from the chain (which would provide an upper bound on how much the instability costs). The mitigation is partial and acknowledged only briefly in Section 6.1; the paper flags no future work on training the model to recognize when no revision is needed, leaving the correction-triggering problem unsolved.

No Combination of Search and Revisions Is Evaluated

Section 8 explicitly acknowledges:

"while we studied the proposals [revisions] and verifier [PRM search] mechanisms in isolation, we did not experiment with PRM tree-search techniques in combination with revisions"

The paper's entire framework is built around the observation that revisions (modifying the proposal distribution) and search (optimizing against the verifier) are complementary axes with different difficulty-dependent strengths: revisions help most on easy problems (refining nearly-correct answers), while search helps most on medium problems (navigating toward correct solutions the base model wouldn't find by random sampling alone). This complementarity is a central conceptual contribution of the paper, articulated in Section 2 and demonstrated empirically in Sections 5 and 6.

Yet the experiments never combine the two mechanisms. A model that uses the revision model as the proposal distribution within beam search — where at each step of the search tree, the model conditions on previous rejected branches as context — could potentially yield gains beyond either mechanism alone. Similarly, using the PRM to guide which revisions to pursue (rather than blindly generating a long chain) could focus the revision budget on promising directions. The paper's results therefore represent a lower bound on what a fully integrated system might achieve, and the strong empirical case for complementarity makes the absence of combination experiments conspicuous.

The consequence is that we do not know whether the two mechanisms are additive (their gains stack), synergistic (they multiply), or redundant (both help via the same underlying improvement and combining them yields diminishing returns). The paper's headline numbers — 4× improvement from compute-optimal search, 4× from compute-optimal revisions — cannot be combined to claim 16× improvement, because the interaction is unknown. A practitioner implementing both mechanisms would need to guess at how to allocate budget between them, since the paper provides no guidance.

The mitigation is acknowledgement with a pointer to future work, but no experiments, no preliminary results, and no analysis of what technical obstacles prevent combination (e.g., distribution shift between base model outputs and revision model outputs breaking the PRM, as documented in Appendix J, Figure 15a, which shows the base-LM PRM underperforms on revision model outputs — this shift might make combining search and revisions even harder than either alone).

No Latency or Wall-Clock-Time Analysis

The paper measures test-time compute exclusively in units of "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores a critical dimension for real deployment: latency. The compute-optimal policy allocates different strategies per difficulty level — sequential revisions on easy problems, beam search on medium-hard problems — and these strategies have fundamentally different wall-clock-time profiles at the same generation budget.

Sequential revisions are inherently serial: each revision depends on the previous one, so a chain of 64 revisions takes roughly 64× the wall-clock time of generating a single solution, regardless of how many GPUs are available. Parallel best-of-N with 64 samples can be executed simultaneously given sufficient hardware, completing in roughly the time of a single generation. The compute-optimal policy for revisions on easy problems (Figure 7, right, bin 2) recommends heavily sequential allocations — fully sequential or high sequential-to-parallel ratios — because these improve accuracy. But this recommendation ignores the latency cost: a deployment handling 100 queries per second with a 200ms latency budget cannot run 64 sequential revisions regardless of how FLOP-efficient that allocation is on paper.

The consequence is that the paper's efficiency metric (accuracy per generation budget) is incomplete for latency-sensitive applications. A strategy that achieves 40% accuracy with 64 sequential generations might be unacceptable for an interactive voice assistant requiring sub-second response, while a strategy achieving 38% accuracy with 64 parallel generations in 200ms wall-clock time is acceptable. The paper's compute-optimal policy would select the former (higher accuracy per generation), but a real deployment would select the latter (acceptable accuracy within the latency budget). The omission is particularly relevant because the paper's primary motivation — unified ASR systems serving both offline and streaming modes — is explicitly about latency-constrained deployment, yet the test-time compute analysis never engages with the latency dimension.

The paper does not acknowledge this limitation. There is no measurement of wall-clock time for any strategy, no discussion of how the sequential-to-parallel ratio interacts with latency budgets, and no proposal for incorporating latency constraints into the compute-optimal allocation framework. The FLOPs-matched comparison (Section 7) uses total inference FLOPs as the budget, not total inference time, so it inherits the same limitation.

Generalization Is Established on a Single Benchmark with a Single Model Family

All experiments — every table, every figure, every ablation — use the MATH benchmark [Hendrycks et al., 2021] and PaLM 2-S* as the base model. The paper addresses this scope explicitly in Section 4:

"We chose to focus our study on the MATH benchmark... We believe this model is representative of the capabilities of many contemporary LLMs and serves as a strong testbed for studying test-time compute scaling"

The "we believe" qualifier is doing significant work. MATH consists exclusively of high-school competition-level math problems requiring symbolic reasoning, multi-step deduction, and exact-answer matching. This is a specific cognitive profile that may not generalize to:

Code generation, where correctness is also binary (passes tests / doesn't), but the structure of errors (syntax errors, logic bugs, edge-case failures) differs from mathematical reasoning errors.
Open-ended generation, where correctness is multidimensional (relevance, coherence, factual accuracy, style) and verifier training requires fundamentally different approaches than binary-correctness labels.
Factual recall tasks, where the base model either knows the fact or doesn't — revisions cannot conjure knowledge from nowhere, and search against a verifier trained on base-model outputs may not help if the base model never produces the correct fact.
Other model families with different calibration properties, different in-context learning capabilities, or different error patterns. The PRM's over-optimization behavior (beam search degrading easy-problem performance at high budgets in Figure 3) could be specific to PaLM 2-S*'s output distribution and may not replicate for, say, GPT-4 or Llama-3.

The consequence is that the paper's specific quantitative findings — the 4× efficiency gain, the difficulty-bin boundaries where beam search helps vs. hurts, the optimal sequential-to-parallel ratios per bin — may not transfer to other tasks or models. The qualitative findings (test-time compute should be adaptively allocated based on difficulty, revisions and search have complementary difficulty-dependent strengths) are more likely to generalize, but without replication, their universality is assumed rather than demonstrated.

The mitigation is transparency: the paper states its scope clearly and does not claim broader generalization. The difficulty quintile binning is defined relative to the base model's capabilities (pass@1 rate on MATH), not external difficulty labels, which provides some transferability — the same methodology can be applied to any model on any task with correctness labels, producing new bin boundaries specific to that model-task pair. But the computational cost of doing so (2048 samples per task instance for difficulty estimation) makes this methodology expensive to replicate, which may limit its adoption on new benchmarks. The paper suggests no lightweight proxy for difficulty estimation that would enable rapid transfer to new domains.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new diagnostic axis to the unified ASR problem: the failure mode at low latency is not primarily an architectural mismatch in the encoder (whether the Conformer can process chunked audio at all), but a representational consistency gap at the output level — the same encoder parameters produce systematically different vocabulary-level predictions depending on context availability, and the joint network cannot reconcile them. This reframing matters because the field has invested heavily in architectural adaptations for streaming (chunk-limited attention, causal convolutions, state-space model replacements) under the implicit assumption that if the encoder produces reasonable hidden states, the rest of the Transducer pipeline will follow. The paper's central negative result — that standard unified training degrades sharply below ~0.5s latency (16.91% WER at 0.24s for dual-mode, versus 9.04% with MCR-RNNT) despite the encoder being architecturally capable of chunked processing — demonstrates that this assumption is false.

The diagnostic reframing shifts research attention from how to build an encoder that survives chunking (a solved problem, given that DCConv and chunk-limited attention produce the streaming baseline's 9.44% WER at 0.32s) to how to ensure the encoder maps the same acoustic event to the same output distribution regardless of context window. This opens a class of solutions — output-level regularization, representation distillation, contrastive objectives — that were not obvious when the problem was framed architecturally. It also implies that architectural innovations alone cannot close the offline-streaming gap; they must be paired with mechanisms that explicitly constrain the mapping from hidden states to vocabulary predictions.

A second landscape shift is the paper's empirical demonstration that consistency regularization is representation-level-dependent in a strong sense. The failure of CR-CTC — where frame-level KL divergence between offline and streaming CTC posteriors "consistently degraded streaming RNNT accuracy" — establishes a boundary condition that was previously undocumented: you cannot apply consistency at an arbitrary output level and expect improvement. Frame-synchronous CTC consistency pulls the shared encoder toward locally confident predictions that are incompatible with the streaming Transducer's need to defer decisions by emitting blanks. The lattice-level consistency of MCR-RNNT, by contrast, preserves the blank token as a legitimate expression of uncertainty — the streaming encoder can learn to output high blank probability where it lacks future context, and the consistency loss penalizes divergence from the offline encoder's blank probability at those positions, rather than forcing premature commitment. This finding will shape how consistency regularization is applied to any sequence model with alignment freedom: the level at which consistency is enforced must match the level at which the model expresses uncertainty about its decisions.

The practical consequence is that the offline-streaming unification problem is now solvable with a single model at latencies down to roughly 0.24s, where previous unified approaches either sacrificed offline quality (streaming-only models like the 7.75% offline WER baseline) or collapsed at low latency (Unified DM's 22.45% at 0.16s compared to MCR-RNNT's 10.51%). The paper's 600M-parameter model achieving 5.76% offline WER — within 0.13 pp of a 2.5B-parameter offline specialist — while maintaining 7.35% WER at 0.24s streaming, demonstrates that unification is not a compromise that inevitably degrades both modes. The open-source release of the framework and model checkpoint lowers the barrier for adoption, meaning that production ASR deployments can realistically consider a single model for both batch transcription and low-latency streaming, reducing the dual-model maintenance overhead that was the paper's motivating problem.

However, the paper does not eliminate the offline-streaming tradeoff — it compresses it. At 0.16s latency, the unified model still trails dedicated streaming models (10.51% vs. 9.84% for the streaming baseline at L-size; 8.44% vs. 7.92% for Nemotron-Streaming at XL-size). The consistency regularization pushes the crossover point (where the unified model matches streaming specialists) from approximately 0.56s (where standard unified training already degrades) down to roughly 0.24s, but does not erase the streaming specialist's advantage at the tightest constraint. For applications requiring sub-200ms latency, a dedicated streaming model remains preferable, and the paper does not claim otherwise.

Follow-Up Research This Work Enables

Cheap difficulty estimation — oh wait, this is an ASR paper, not the MATH/PRM paper. Correct directions:

Cache-passing for efficient streaming inference. The paper explicitly flags this as unimplemented future work: "we will implement a cache-passing mechanism to enable efficient streaming decoding for the proposed unified RNNT models." Currently, the left context (70 frames, or 5.6s of audio) is recalculated at each chunk step, which means streaming inference with the unified model is substantially slower than it needs to be. The left context encodes acoustic history that is identical across consecutive chunks — for a chunk advancing by C frames, C new frames enter at the right and C old frames exit at the left, but the vast majority of the 5.6s left context is reused. Caching and passing encoder states (KV-cache for attention, hidden states for DCConv convolutions) across chunk steps would eliminate this redundant computation. The challenge is that DCConv's chunk-aware reshaping introduces dependencies on chunk boundaries that complicate state management — a frame's convolution output depends on (k-1)/2 frames from the adjacent chunk, so the cached state must include boundary-crossing information. A strong follow-up would implement cache-passing for both the MHA sub-layer (standard KV-cache) and the DCConv sub-layer (chunk-boundary state), measuring the throughput improvement (tokens per second and real-time factor) at each latency point in Table 1, and verifying that cached inference produces identical WER to the recalculation-based approach. The practical impact would be making the unified model deployable for production streaming workloads where throughput directly determines serving costs.

Scaling MCR-RNNT to larger vocabularies. The paper's models use a 1024-token BPE vocabulary, which keeps the full-joint KLD computation tractable — the T × (U+1) × V tensor with V=1024 is manageable in GPU registers. Production ASR systems increasingly use larger vocabularies (16K–32K tokens) for better coverage of rare words and multilingual deployments. The computational overhead of MCR-RNNT scales linearly with V (the KLD is computed over V logits at each (t,u) position), so moving to a 32K vocabulary would increase the consistency loss cost by ~30× at the kernel level. Whether the Triton implementation remains "negligible" at this scale is an open question. A strong follow-up would profile MCR-RNNT with vocabulary sizes of 1K, 4K, 16K, and 32K, measuring wall-clock time per training step and GPU memory relative to the RNNT loss, and would explore optimizations: top-K softmax approximations (computing KLD over only the most probable tokens), vocabulary pruning per position (discarding tokens that are below a probability threshold in both distributions), or sparsifying the lattice to only high-occupancy (t,u) positions (inspired by TCR's occupation-based weighting). The research question is whether consistency over a sparse or pruned distribution retains the benefits of the full-joint approach while remaining computationally feasible at scale.

Combining MCR-RNNT with multi-mode joiner unification (All-in-One ASR). Moriya et al. (2025) [Moriya2025AllinOneAU] unified CTC, AED, and Transducer paradigms within a single model via a multi-mode joiner, but did not address extreme low-latency streaming — their work focused on breadth (multiple ASR paradigms) rather than depth (tight latency constraints within one paradigm). MCR-RNNT operates entirely within the Transducer paradigm, applying consistency between offline and streaming RNNT joint outputs. A natural extension would be to add MCR-RNNT-style consistency losses between the Transducer streaming output and the CTC streaming output (or between CTC offline and CTC streaming) in an All-in-One model. The CR-CTC negative result (frame-level CTC consistency degraded streaming RNNT) suggests this is not straightforward — the CTC and RNNT objectives have different local confidence requirements — but combining them within a unified multi-paradigm model with appropriate weighting might allow the shared encoder to benefit from consistency signals at multiple representational levels. The research question is whether consistency across paradigms (CTC ↔ RNNT) provides complementary regularization to within-paradigm consistency (offline ↔ streaming), or whether the objective mismatch documented for CR-CTC recurs and degrades overall performance. A strong experiment would compare All-in-One training with and without MCR-RNNT across all paradigm combinations, measuring WER-latency curves for each output mode.

Dynamic right-context allocation at inference time. The paper trains with a fixed set of chunk sizes C ∈ {1,2,7,13} and right contexts R ∈ {0,1,2,3,5,7,13,26}, sampled uniformly during training to produce a single model robust to multiple latency targets. At inference time, the chunk and right-context parameters are fixed, producing a single latency-accuracy tradeoff point. But Figure 2 demonstrates that, for a fixed total latency budget, allocating more budget to right context (larger R, smaller C) improves WER — the optimal allocation depends on the latency target. A dynamic system could adjust C and R on-the-fly based on the acoustic content: easy, clearly articulated speech could be processed with smaller right context (lower latency), while acoustically challenging segments (overlapping speakers, background noise, rare words) could trigger larger right context (higher latency but better accuracy). The mechanism would require a real-time confidence estimator — perhaps the streaming encoder's blank probability or the entropy of the joint output distribution — that signals when the model is uncertain and needs more future context. A strong follow-up would train a lightweight confidence predictor on the encoder's hidden states, then use it at inference time to modulate R per chunk (within a total latency budget), measuring whether dynamic allocation achieves better average WER at the same average latency compared to fixed-allocation baselines. This extends the paper's unification philosophy (one model for multiple latencies) into the temporal domain (one model that adapts latency to content).

Cross-domain and cross-lingual evaluation of MCR-RNNT. The paper's evaluation uses the Open ASR Leaderboard's eight English test sets, spanning read speech (Librispeech), meeting speech (AMI), earnings calls (Earnings22), and diverse domains (Gigaspeech, TEDLIUM, VoxPopuli). This is broader than typical ASR evaluations but is still English-only. The consistency regularization's mechanism — penalizing output distribution divergence between offline and streaming modes — should generalize across languages and domains, because the fundamental challenge (acoustic disambiguation requiring future context) is universal. However, languages with different acoustic-phonetic properties may benefit differently: tonal languages (Mandarin, Vietnamese) where pitch contours disambiguate words may require less right context if tone is carried on individual syllables, while morphologically rich languages (Turkish, Finnish) where words are built from many suffixes may require more right context to resolve morphological boundaries. A strong follow-up would replicate the L-size experiment (128M parameters, ~120K hours) in at least three typologically diverse languages, measuring whether the optimal consistency weight λ = 0.3 and the offline-streaming gap at each latency generalize, and whether language-specific λ tuning is necessary. The experiment would also reveal whether MCR-RNNT's advantage over standard unified training is language-dependent — potentially larger for languages where acoustic ambiguity is more context-dependent.

Encoder-level consistency as an alternative to output-level consistency. MCR-RNNT applies consistency at the output distribution level (joint network logits), motivated by the failure of CR-CTC at the frame level. An alternative approach would be to apply consistency directly to the encoder's hidden states, before the joint network: penalize the distance (e.g., cosine distance, L2, or contrastive loss) between offline and streaming encoder representations of the same frame. This would operate at the representation level rather than the output level, potentially learning a shared acoustic representation that serves both modes without constraining the output distribution directly. The advantage would be avoiding the O(T×U×V) computation of the full-joint KLD; the disadvantage would be a weaker training signal (the model could learn to produce similar hidden states that the joint network maps to different output distributions). A strong follow-up would implement both encoder-level consistency (e.g., mean-squared error between offline and streaming encoder outputs at each frame) and MCR-RNNT in the same training framework, measuring WER-latency curves and analyzing whether the two consistency signals are complementary (their gains add), redundant (only one matters), or in conflict. This would clarify whether the output level is necessary or merely sufficient for mode consistency, and could lead to computationally cheaper alternatives that preserve MCR-RNNT's benefits.

Practical Applications and Downstream Use Cases

Single-model deployment for cloud transcription services with optional low-latency streaming. A transcription service that offers both batch (offline) file transcription and real-time (streaming) captioning can replace two separately trained, validated, and deployed models with the paper's unified 600M-parameter checkpoint. The operational savings are direct: one model to maintain, one set of evaluation metrics to monitor, one deployment pipeline. Based on the paper's numbers, the offline accuracy (5.91% AVG WER for the balanced model) is competitive with dedicated offline models and represents less than 1 pp degradation from the SOTA in offline-only ASR, while streaming accuracy at 0.24s latency (7.35% WER) is sufficient for near-real-time captioning of meetings or live presentations. The right-context distribution can be tuned per deployment — a transcription service that prioritizes accuracy might deploy Model (1) (larger right context, 5.76% offline), while a captioning service that prioritizes low latency might deploy Model (2) (balanced, 6.92% at 0.32s). The single-codebase property means the tuning is purely a configuration change, not a retraining requirement.

On-device ASR with adaptive latency. On-device speech recognition (smartphones, smart speakers, automotive systems) typically runs a streaming model with a fixed latency budget determined by the most demanding use case (e.g., voice commands requiring sub-300ms response). The paper's unified model, trained with diverse (C, R) pairs, can serve multiple use cases from a single on-device binary: voice commands at 0.24s latency, dictation at 0.56s where higher accuracy is more important than minimal latency, and offline transcription (meeting recording playback) using the same model with full context. A single model footprint reduces storage and memory pressure on the device. The practical challenge is that the current implementation recalculates the left context at each chunk step (no cache-passing), which would drain battery on mobile devices — the paper's flagged future work on cache-passing would be a prerequisite for on-device deployment.

Self-improvement pipelines for ASR through consistency-guided data selection. A less obvious application: the MCR-RNNT loss provides a per-utterance signal of how much the model's output distribution shifts between offline and streaming modes. Utterances with high MCR loss (large KL divergence) are those where the encoder's representations are most context-dependent — i.e., where the model relies heavily on future context to disambiguate. These utterances are prime candidates for targeted data augmentation (e.g., adding them to the training set with emphasis, or generating synthetic low-latency variants for fine-tuning). Conversely, utterances with low MCR loss are those where the model's predictions are stably informative regardless of context — these are "easy" utterances that the model already handles well in both modes. A data selection pipeline could use per-utterance MCR loss to prioritize high-value examples for additional training or active learning, focusing model improvement on the utterances where the offline-streaming gap is largest. The paper's results don't directly evaluate this, but the mechanism is inherent in the MCR-RNNT computation.

When to Prefer This Method

The paper positions MCR-RNNT against two named alternatives — dedicated streaming-only models and standard unified training (single-mode or dual-mode without consistency regularization) — and the choice depends on the deployment's latency requirements and the acceptable offline accuracy tradeoff:

Prefer the proposed Unified DM + MCR-RNNT when:

You require a single model to serve both offline batch transcription and streaming at latencies down to ~0.24s, and the operational savings from maintaining one model outweigh a small (≤0.2 pp) offline accuracy cost relative to a dedicated offline model.
Your deployment needs to support multiple latency targets from a single checkpoint — the diverse (C, R) training distribution produces a model that can be configured at inference time for different latency-accuracy tradeoffs without retraining.
You have sufficient training compute to run dual-mode training (two encoder forward passes per step) and the ~5–20% additional per-step overhead of the MCR-RNNT Triton kernel is acceptable.

Prefer a dedicated streaming-only model when:

Your deployment requires sub-200ms latency (≤0.16s), where the paper shows the unified model still trails the streaming specialist (10.51% vs. 9.84% at L-size; 8.44% vs. 7.92% at XL-size for Model 2 vs. Nemotron-Streaming).
Offline accuracy is irrelevant to your use case — you are building a purely real-time system (e.g., live captioning, voice assistants) where batch transcription is never performed, so the unified model's ability to serve offline mode provides no benefit.
Training compute is severely constrained and the 2× cost of dual-mode training cannot be absorbed, making single-mode or streaming-only training more practical.

Prefer standard unified training (without MCR-RNNT) when:

Your streaming latency requirements are moderate (≥0.56s), where Table 1 shows that Unified SM (7.98% at 0.56s) and Unified DM (8.12%) achieve reasonable performance without the added complexity of implementing the MCR-RNNT kernel.
You lack the engineering resources to implement the Triton kernel or cannot depend on the open-source release due to framework compatibility constraints (e.g., using a non-NeMo training stack that would require porting the kernel).