Efficient Sequence Transduction by Jointly Predicting Tokens and Durations

ArXiv: 2304.06795

🎯 Pitch

Transducer models are slow because they crawl through input frame by frame. TDT fixes this by predicting how long each token lasts, letting the model jump ahead during decoding — cutting ASR inference time by up to 2.8× while improving accuracy.

1. Executive Summary

This paper introduces a Token-and-Duration Transducer (TDT) architecture that extends conventional RNN-Transducer models by jointly predicting both an output token and its duration — the number of input frames covered by that token emission — using a joint network with two independently normalized output heads. Evaluated across three speech tasks — automatic speech recognition on LibriSpeech (Conformer-Large, ~120M parameters), speech translation on MUST-C, and spoken language understanding on SLURP — TDT models achieve both better accuracy and significantly faster inference than conventional Transducers, with inference speed-ups reaching up to 2.82× on German ASR and 2.27× on speech translation while also improving BLEU by over 1 point on MUST-C. The architecture establishes that frame-skipping guided by learned duration predictions enables faster decoding without accuracy degradation — but only when the audio-to-text length ratio is sufficiently large, as demonstrated by the smaller speed-ups (1.28×) on SLURP where audio sequences are shorter than their corresponding text.

2. Context and Motivation

The Core Problem: RNN-Transducer Decoding Is Inherently Slow

The fundamental problem this paper addresses is that RNN-Transducers are computationally expensive at inference time despite their many architectural advantages. To understand why, we need to examine how a conventional Transducer performs decoding.

An RNN-Transducer processes its input — in speech applications, an acoustic feature sequence — frame by frame. At each time step t, the model computes a probability distribution over the vocabulary, which includes all possible output tokens plus a special blank symbol ∅. This blank symbol serves as a "do-nothing" action: when the model emits blank, it advances to the next input frame without producing an output token. When the model emits a non-blank token, it produces that token but stays on the same input frame. This means the model can only advance through the input one frame at a time, and a blank emission is the sole mechanism for moving forward temporally.

This design creates a direct tension: the model must emit a blank for every input frame that does not correspond to a token emission. Since speech feature extraction typically operates at 10ms frame intervals with a 25ms window — and the encoder applies additional subsampling (e.g., 4×, yielding 40ms frames) — a typical utterance contains hundreds of frames. For a 10-second utterance with 4× subsampling, there are roughly 250 frames. The Transducer must process each one sequentially during autoregressive decoding, regardless of how sparse the actual linguistic content is. This includes long stretches of silence, background noise, or steady-state vowel sounds where no new token output is needed.

This is not merely a theoretical concern. In production ASR systems, inference latency directly impacts user experience. For batch processing pipelines — transcribing thousands of hours of audio — inference speed determines infrastructure cost. For streaming applications, per-frame processing creates a hard lower bound on latency that cannot be circumvented through parallelism because the decoder is autoregressive: each output depends on previous tokens.

Why Existing Solutions Fall Short

The paper implicitly identifies a gap in how prior work has attempted to address Transducer efficiency. Let's examine the landscape:

Architectural improvements to encoder/decoder efficiency — replacing LSTM encoders with Transformers, ContextNets, or Conformers as cited by the authors — primarily reduce the cost of computing representations at each frame. They do not reduce the number of frames that must be processed. An attention-based encoder can compute all frame representations in parallel, but the decoder must still step through each frame sequentially during inference. This is a fundamental algorithmic bottleneck, not merely a per-frame computational cost issue.

Decoder design innovations — such as the stateless decoder from Ghodsi et al. (2020), which concatenates embeddings of the last two history words instead of maintaining an LSTM hidden state, or the Echo State Network decoder from Shrivastava et al. (2021) — reduce the computational cost per decoding step. But they do nothing to reduce the number of steps, which is dictated by the number of input frames plus the number of output tokens. If an utterance has 250 frames and 50 output tokens, the Transducer must execute approximately 300 decoding steps regardless of how efficient each step is.

Regularization approaches like FastEmit (Yu et al., 2021) introduce gradient bias to encourage earlier token emissions, improving streaming latency. But this only shifts when tokens are emitted within the frame sequence — it does not reduce the total number of decoding steps, because the model still must process every frame by emitting blanks.

Multi-blank Transducers (Xu et al., 2022) represent the closest prior work and are directly acknowledged as intellectual predecessor to TDT. The MBT introduces "big blank" symbols — special blank tokens that each cover multiple input frames. When a big-blank-duration-4 is emitted, the model skips 4 frames instead of 1. This directly reduces the number of decoding steps. However, the MBT has an important limitation that the paper highlights: frame-skipping is restricted to blank emissions only. When the model emits a non-blank token in an MBT, no frame advancement occurs — exactly as in a conventional Transducer. This means that even with big blanks enabled, the model must still emit a blank symbol to advance through the input. The speed-up is consequently bounded by how aggressively the model can be trained to prefer long-duration blanks over short ones, and non-blank emissions provide no opportunity for temporal advancement.

The paper explicitly compares TDT with MBT (Table 9 in Section 5.5) and demonstrates that TDT achieves substantially larger speed-ups at equivalent accuracy levels. For example, with maximum duration 8, MBT achieves 1.76× speed-up on LibriSpeech test-other while TDT achieves 2.12× — a difference of roughly 20% relative improvement. This gap exists precisely because TDT allows both blank and non-blank tokens to have associated durations, creating more opportunities for frame-skipping.

Why a New Architecture Is Needed: The Duration Prediction Gap

Stepping back, there is a deeper architectural insight that motivates TDT. In conventional Transducers, the model learns a soft alignment between input frames and output tokens through the blank mechanism — the model decides frame-by-frame whether to emit a token or advance. But this alignment is implicit and distributed: to understand when a token was emitted, you must trace the path of blank vs. non-blank decisions through the probability lattice. The model never explicitly represents "this token spans frames 40 through 45."

This is peculiar when you consider what the model is actually doing. During training, the Transducer loss sums over all possible alignments weighted by their probability. The model sees many valid paths where a given token could be emitted at slightly different frames. But fundamentally, each token does correspond to some temporal extent in the audio — a word, a syllable, a subword unit — and that extent has a natural duration. The Transducer architecture provides no mechanism to explicitly model this duration.

TDT addresses this gap by making duration an explicit, predicted output. The model simultaneously answers two questions at each step: "What token should I emit?" and "How many frames does this token cover?" This transforms the inference process: instead of emitting blanks one by one to advance through frames, the model can advance by d frames in a single step, where d is the predicted duration — and crucially, this applies whether the emitted token is blank or non-blank.

The Real-World Significance

The practical motivation for this work is made concrete by the paper's evaluation across three distinct tasks. Let's examine what each tells us about the problem's importance:

Speech recognition is the canonical transducer application. The paper demonstrates that TDT achieves up to 2.82× faster inference on German MLS without accuracy degradation, and in some cases (Spanish CallHome) with accuracy improvements of nearly 2 absolute WER points. For a production ASR system, a 2× speed-up with no accuracy penalty effectively halves the infrastructure cost or doubles the throughput — a substantial operational impact.

Speech translation pushes the boundary further. Here, the input is English audio and the output is German text. This task requires both acoustic understanding and cross-lingual generation, making the decoder's role more complex than in monolingual ASR. The paper reports that TDT improves BLEU by over 1 point and runs 2.27× faster than a conventional Transducer. The accuracy improvement is noteworthy because it suggests that the duration modeling doesn't just maintain performance — it may actually be a better inductive bias for the sequence-to-sequence mapping, perhaps because explicitly modeling temporal alignment helps the model handle the mismatch between source-language timing and target-language word order.

Spoken language understanding on SLURP tests the model on a task where the output is structured text (a Python dictionary of intents and slots) and the audio-to-text length ratio is dramatically different. In ASR, this ratio is approximately 5.5:1 to 7:1 (many more audio frames than output tokens). In SLURP, the average ratio is 0.89:1 — the text output is actually longer than the audio input when measured in frames vs. subword tokens. The paper still achieves 1.28× speed-up with better intent accuracy, demonstrating that the approach generalizes beyond the "long audio, short text" regime where it's most naturally suited. However, the smaller speed-up also reveals a fundamental boundary condition that the paper explicitly acknowledges: duration-based skipping provides diminishing returns when the average token already covers few frames. This honest assessment of when the method works — and by how much — strengthens the contribution.

How the Paper Positions Itself

The paper positions TDT not as a replacement for existing efficiency techniques but as a complementary mechanism that addresses a distinct bottleneck. The authors maintain compatibility with standard Transducer infrastructure: the encoder and decoder architectures (Conformer-Large with stateless decoder) are unchanged from their baseline. The innovation is entirely in the joint network output structure and the training/inference algorithms that exploit it.

This is an important aspect of the positioning. Improvements like FastEmit, stateless decoders, and big blanks operate at different levels: regularization, decoder efficiency, and blank semantics. TDT operates at the alignment semantics level — changing what information the model represents about the input-output relationship. The paper implies that all these techniques could be combined, though the experiments focus on comparing TDT against baselines that already incorporate FastEmit and stateless decoders. This layered compatibility makes the approach practically adoptable: an existing Transducer system can be upgraded to TDT by modifying the joiner output dimension, the loss function, and the inference algorithm, while preserving the encoder and decoder investments.

The paper also positions itself within the open-source toolkit ecosystem, with explicit commitment to releasing the implementation in NVIDIA's NeMo toolkit. This is significant because TDT requires custom forward-backward algorithm implementations and custom CUDA kernels for efficient training — the paper notes that automatic differentiation is "highly inefficient" for Transducer loss in general (Section 3.1), motivating the detailed analytical gradient derivations. By providing production-ready implementations, the paper aims to lower the barrier to adoption.

The Training Challenge: Efficient Gradients for a New Loss Surface

A final piece of context that the paper addresses in depth concerns how to train such a model efficiently. The TDT architecture introduces a new loss function (Equation 9) that requires summing over both token and duration choices at each state, making the probability lattice denser than a conventional Transducer's. This is not merely an implementation detail — it's a fundamental challenge that distinguishes TDT from a simple augmentation.

The paper dedicates substantial space to deriving the analytical gradients (Section 3.1, Section 3.2, and Appendices A, B, C), including the extension of the "transducer function-merging" technique from Li et al. (2019) to directly compute gradients with respect to pre-softmax logits. This is necessary because automatic differentiation through the forward-backward algorithm — which contains nested summations over durations — would be memory-intensive and slow. The authors also extend the "logit under-normalization" technique from their prior MBT work to encourage the model to prefer longer durations during training, introducing a hyperparameter σ (set to 0.05 for ASR/ST, reduced to 0.02 for SLU) that scales down token probabilities to make duration-based advancement more attractive relative to emitting many short-duration tokens.

These training innovations are part of what makes TDT practically viable. Without the analytical gradients, training would be prohibitively slow. Without under-normalization, the model might learn to emit many short-duration tokens rather than exploiting the efficiency of long-duration predictions. The paper's engineering contributions in making TDT trainable are as important to its practical impact as the architectural innovation itself.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

The system is an end-to-end neural network for converting speech to text that learns, during training, to predict how many audio frames each output word or subword covers, so that at inference time it can skip ahead through the audio rather than examining every single frame. The problem it solves is that conventional Transducer models must process the input frame-by-frame — one decoding step per audio frame even for silence or steady sounds — making inference slow; TDT solves this by adding a second output to the model that explicitly predicts "how far to jump" after emitting each token, regardless of whether that token is a real word or a blank placeholder.

3.2 Big-picture architecture (diagram in words)

The TDT architecture has three core components, identical in structure to a conventional RNN-Transducer except for the output layer:

1. Encoder (Conformer-Large): Consumes the acoustic input (a sequence of audio frames, each 10ms with 25ms windows, subsampled by a factor of 4 to produce 40ms frames) and produces a sequence of hidden representations enc[t] for each time step t.
2. Decoder / Prediction Network (Stateless): Consumes the history of previously emitted tokens (the last 2 tokens, embedded and concatenated) and produces a hidden representation dec that encodes linguistic context.
3. Joint Network (Joiner): Takes enc[t] and dec as input, combines them through a feed-forward network, and produces two separate output vectors: one of size vocab_size (for token probabilities), and one of size |D| (for duration probabilities), where D is the set of supported durations, e.g., {0, 1, 2, 3, 4, 5, 6, 7, 8}. These two vectors are independently normalised — the first with softmax over the token vocabulary, the second with softmax over the duration set.

Information flow at inference: The encoder processes all audio frames in parallel (as typical for Conformer-based Transducers). Decoding proceeds autoregressively: at each step, the decoder encodes the current hypothesis history, the joiner produces token and duration distributions from enc[t] and the decoder state, the model selects the most likely token AND the most likely duration, emits the token if non-blank, and advances t by the predicted number of frames (duration) regardless of whether the emission was blank or non-blank. This is the critical departure from conventional Transducers — t can increment by more than 1 in a single step.

3.3 Roadmap for the deep dive

• First, the probability model — the joint token-duration distribution and the conditional independence assumption (Equation 5) — because this defines what the model is computing at each (t,u) lattice node.
• Second, the forward-backward algorithm extension (Equations 6 and 8) — how the probability lattice works when transitions can skip multiple frames, and how the total sequence probability P_TDT(y|x) is computed, because this is the core computational primitive that distinguishes TDT from conventional Transducers.
• Third, the training loss and gradient derivations — the closed-form solutions for gradients with respect to token probabilities (Equation 10), duration probabilities (Equation 12), and pre-softmax logits (Equation 14), including the transducer function-merging technique and logit under-normalisation (Equation 17), because these are what make training computationally tractable.
• Fourth, the inference algorithm (Algorithm 2) — how the predicted durations are used at test time to skip frames, including the batched inference challenge and the loss-sampling solution (Equation 18), because this is where the speed-up materialises.
• Fifth, the full set of training and architectural hyperparameters — model dimensions, training data, regularisation settings, duration configurations, and the subtle choice of σ (under-normalisation strength) across different tasks, because these details are essential for reproducibility and understanding the method's operating envelope.

3.4 Detailed, sentence-based technical breakdown

This is primarily a methods paper with substantial mathematical engineering — the core idea is to augment the Transducer probability model so that each emission carries an explicit duration, then to derive the efficient forward-backward recursions and analytical gradients needed to train this model, and finally to exploit the predicted durations at inference time to skip frames.

Joint Token-Duration Probability Model

At every node (t, u) in the decoding lattice — where t indexes the encoder output time step (1-indexed, ranging from 1 to T, the total number of encoder output frames) and u indexes the number of output tokens emitted so far (0-indexed, ranging from 0 to U, the target sequence length) — the TDT joiner produces two distributions: one over the vocabulary (including blank ∅) and one over a discrete set of supported durations D. The paper makes a conditional independence assumption: the probability of emitting token v with duration d jointly factors as the product of the token probability and the duration probability:

$P(v, d \mid t, u) = P_T(v \mid t, u) \cdot P_D(d \mid t, u)$

where $P_T(v \mid t, u)$ is the probability assigned to token $v$ at lattice position $(t, u)$ (computed via softmax over the first vocab_size joiner outputs), and $P_D(d \mid t, u)$ is the probability assigned to duration $d$ at that same lattice position (computed via softmax over the remaining $|\mathcal{D}|$ joiner outputs, starting at index vocab_size). The set $\mathcal{D}$ is a configurable set of consecutive integers starting from 0, written in shorthand notation — e.g., "0-4" means $\mathcal{D} = \{0, 1, 2, 3, 4\}$, and "0-8" means $\mathcal{D} = \{0, 1, 2, ..., 8\}$.

What it computes: For a given acoustic context at time t and linguistic context after u emitted tokens, the model produces a factored joint distribution over all possible emissions. Each emission is a (token, duration) pair. If the token is non-blank (a real vocabulary item), the model advances u by 1 (a new output token is produced) and advances t by d frames. If the token is blank (the special symbol ∅), only t advances by d frames — u stays the same. A special restriction applies: blank emissions are not allowed to have duration 0, because a blank with duration 0 would mean "advance nothing and emit nothing," creating a self-loop that contributes zero information. Duration 0 is permitted for non-blank emissions (meaning "emit a token without advancing in time at all"), which corresponds to the behaviour of a conventional Transducer's non-blank emission.

Why this form: The conditional independence assumption decomposes the otherwise combinatorial choice — the model would need to produce a flat distribution of size vocab_size × |D| over all (token, duration) pairs, which would explode the output dimension and make training harder. By factorising the problem, the model learns token identity and temporal extent as separate but co-occurring decisions, each supervised by the same alignment signal. The restriction that blank-with-duration-0 is disallowed is a deliberate design choice: "this makes the model not strictly probabilistic unless we renormalize the duration probabilities excluding duration = 0 for blank emissions computation. Although in practice we find that this does not matter, since duration=0 is in general rarely predicted according to Figure 4, and this design makes the derivation of gradients much easier."

Implementation detail (from Section 3 footnote): In the actual code, the joiner's final layer maps a hidden activation to a tensor joiner_out of size vocab_size + |D|. For example, with vocabulary size 1024 (including ∅) and durations {0, 1, 2, 3, 4} (5 durations), the joiner outputs a 1029-dimensional vector. Then joiner_out[:1024] is passed through a softmax to get P_T, and joiner_out[1024:] is independently passed through a separate softmax to get P_D.

The Extended Forward-Backward Algorithm

The core computational challenge of TDT is computing the total probability P_TDT(y|x) of the target sequence y given the audio x, which requires summing over all valid paths through the probability lattice. In a conventional Transducer, each path consists of a sequence of blank and non-blank emissions where each blank advances t by 1 and each non-blank advances u by 1. In TDT, each emission advances t by the predicted duration d (which can be any value in D), making the lattice significantly denser: from any state (t, u), there are |D| possible blank transitions (to (t+d, u) for each d ∈ D \ {0}) and |D| possible non-blank transitions (to (t+d, u+1) for each d ∈ D).

The paper extends the standard forward-backward recursion to handle this denser connectivity. The forward variable α(t, u) represents the total probability of all paths that start at the initial state and end at state (t, u) — i.e., the total probability of reaching time step t having emitted exactly u tokens:

$\alpha(t, u) = \sum_{d \in \mathcal{D} \setminus \{0\}} \alpha(t - d, u) \cdot P(\emptyset, d \mid t - d, u) + \sum_{d \in \mathcal{D}} \alpha(t - d, u - 1) \cdot P(y_u, d \mid t - d, u - 1)$

where $P(v, d \mid t, u) = P_T(v \mid t, u) \cdot P_D(d \mid t, u)$ from Equation 5. The base condition is $\alpha(1, 0) = 1$ (at time step 1 with 0 tokens emitted, the probability is 1 — we haven't produced anything yet).

What it computes: The first sum accounts for all paths where the last transition was a blank emission of duration d (excluding d=0): we look back to state (t-d, u), where we had reached time step t-d with u tokens emitted, and then emitted a blank with duration d, which jumped us to time t without producing a new token. The second sum accounts for all paths where the last transition was a non-blank emission of token y_u (the u-th target token, 1-indexed) with duration d: we look back to state (t-d, u-1), where we had reached time t-d with u-1 tokens emitted, and then emitted y_u with duration d, jumping to time t and incrementing u by 1.

Why this form: This recursion mirrors the conventional Transducer forward recursion (Equation 2) but replaces the single-step transitions (t-1 for blanks, t for non-blanks) with summations over all possible durations. The exclusion of d=0 from the blank sum is the mathematical expression of the design decision that "blank with duration 0" is disallowed. The inclusion of d=0 in the non-blank sum allows the model to emit tokens without advancing time (the conventional Transducer behaviour for non-blank emissions), which is important for tasks where multiple tokens might be emitted at effectively the same acoustic time.

The backward variable β(t, u) is the mirror image: the total probability of all paths that start at state (t, u) and reach the terminal state:

$\beta(t, u) = \sum_{d \in \mathcal{D} \setminus \{0\}} \beta(t + d, u) \cdot P(\emptyset, d \mid t, u) + \sum_{d \in \mathcal{D}} \beta(t + d, u + 1) \cdot P(y_{u+1}, d \mid t, u)$

with base condition $\beta(T+1, U) = 1$ (at the terminal state just past the final time step, having emitted all U tokens, the probability is 1).

What it computes: From state (t, u), we consider all possible next transitions. The first sum: for each possible duration d > 0, if we emit a blank with duration d, we jump to (t+d, u) and continue from there — β(t+d, u) is the remaining path probability. The second sum: for each possible duration d (including 0), if we emit the next target token y_{u+1} with duration d, we jump to (t+d, u+1) and continue from β(t+d, u+1).

The total sequence probability can be read off from either the forward or backward variable at the boundary:

$P_{\text{TDT}}(y \mid x) = \alpha(T + 1, U) = \beta(1, 0)$

Important notational nuance: The paper uses a slightly non-standard boundary convention compared to the original RNN-T literature. For TDT, the forward recursion goes up to α(T+1, U) (one past the final time step), and the backward recursion starts from β(T+1, U) = 1. This choice is made because, with multi-frame skips, transitions can land exactly at or beyond T. The boundary α(T+1, U) = P_TDT(y|x) captures the total probability mass that reaches the end of the sequence having emitted all U tokens, possibly with a final blank emission that skips past the last frame. The conventional Transducer boundary was α(T, U) · P(∅ | T, U), which is equivalent but less convenient when durations complicate the final-step bookkeeping.

Lattice structure (Figure 3): The paper visualises this with a lattice diagram where each node (t, u) has incoming and outgoing arcs labelled with durations. A complete path through the lattice from (1, 0) to (T+1, U) corresponds to one valid alignment between the audio and the text. Unlike the conventional Transducer lattice where arcs only go right (blank, t→t+1) or down (non-blank, u→u+1), the TDT lattice has diagonally-skipping arcs that jump multiple columns rightward, corresponding to multi-frame durations.

The TDT Training Loss

The TDT loss is the negative log-probability of the target sequence:

$\mathcal{L}_{\text{TDT}} = -\log P_{\text{TDT}}(y \mid x)$

What it computes: For a given (audio, text) training pair, the loss is the negative log of the total probability that the model assigns to the correct text sequence, summed over all valid alignments (all possible ways to assign durations to each token emission and blank emission). Minimising this loss encourages the model to raise the probability of all paths that produce the correct text, weighted by how well the durations fit the actual temporal structure of the audio.

Why this form: This is the standard maximum-likelihood objective for sequence-to-sequence models with latent alignments. By summing over all alignments rather than committing to a single forced alignment, the model learns a soft, probabilistic mapping between acoustic frames and output tokens, which is more robust to timing variation than hard alignment approaches like CTC (which requires monotonic alignment and cannot model token durations explicitly).

Analytical Gradients with Respect to Probabilities

The paper does not rely on automatic differentiation through the forward-backward recursion, because "automatic differentiation for transducer loss is highly inefficient." Instead, it derives closed-form analytical gradients. Following the same pattern as the original RNN-T gradient derivation by Graves (2012), the authors extend the derivation to account for the duration-summation terms.

For the token probability gradient at a specific lattice node (t, u) and token v:

$\frac{\partial \mathcal{L}_{\text{TDT}}}{\partial P_T(v \mid t, u)} = -\frac{\alpha(t, u) \cdot b(v, t, u)}{P_{\text{TDT}}(y \mid x)}$

where $b(v, t, u)$ is an auxiliary quantity that aggregates backward variables reachable from $(t, u)$ via emissions of token $v$, weighted by duration probabilities:

$b(v, t, u) = \begin{cases} \sum_{d \in \mathcal{D}} \beta(t + d, u + 1) \cdot P_D(d \mid t, u), & v = y_{u+1} \\ \sum_{d \in \mathcal{D} \setminus \{0\}} \beta(t + d, u) \cdot P_D(d \mid t, u), & v = \emptyset \\ 0, & \text{otherwise} \end{cases}$

What it computes: The gradient of the loss with respect to the probability of emitting token v at state (t, u). It is proportional to α(t, u) (the probability of reaching state (t, u)) times b(v, t, u) (a weighted sum of probabilities of the future paths from states that would be reached if token v were emitted). When v = y_{u+1} (the correct next token), b sums over all possible durations for emitting that token, each weighted by: (a) the duration probability P_D(d|t, u) and (b) the backward probability from the resulting state β(t+d, u+1). When v = ∅, it sums over all non-zero durations for the blank. For all other tokens, the gradient is zero — those tokens are not on the correct alignment path at this position.

Why this form: The gradient has the intuitive interpretation: the loss decreases (probability increases) for emissions that connect high-probability past paths (α) to high-probability future paths (β), weighted by duration likelihood. The derivation (Appendix A.2) starts from a generalised "diagonal sum" identity for TDT — Equation 23 in the appendix — which accounts for the fact that, unlike in conventional Transducers where all paths must cross any (t+u = n) diagonal, TDT paths can "jump over" diagonals via multi-frame skips. This generalisation introduces extra terms in the decomposition of P_TDT(y|x) that the conventional Transducer derivation doesn't need.

For the duration probability gradient:

$\frac{\partial \mathcal{L}_{\text{TDT}}}{\partial P_D(d \mid t, u)} = -\frac{\alpha(t, u) \cdot c(d, t, u)}{P_{\text{TDT}}(y \mid x)}$

where:

$c(d, t, u) = \begin{cases} \beta(t, u + 1) \cdot P_T(y_{u+1} \mid t, u), & d = 0 \\ \beta(t + d, u + 1) \cdot P_T(y_{u+1} \mid t, u) + \beta(t + d, u) \cdot P_T(\emptyset \mid t, u), & d > 0 \end{cases}$

What it computes: The gradient for the duration probability at state (t, u). For d = 0, the only valid contribution is from emitting the correct next token y_{u+1} without advancing time — this connects α(t, u) to β(t, u+1), weighted by the probability of that token. For d > 0, there are two contributions: one from emitting y_{u+1} with duration d (connecting to β(t+d, u+1)), and one from emitting blank with duration d (connecting to β(t+d, u)). Both are weighted by their respective token probabilities.

Why this form: The duration gradient captures the temporal consistency signal: a duration d gets positive gradient (loss decreasing) when transitioning by d frames connects a high-probability prefix to a high-probability suffix, whether via blank or via the correct token. This explicitly trains the model's duration predictions to be consistent with the token emission timing — essentially learning "how many frames typically elapse before the next token should appear."

Gradients with Respect to Pre-Softmax Logits (Function Merging)

Computing ∂L/∂P is only the first step — the actual neural network parameters need ∂L/∂h, where h are the pre-softmax logits. The paper extends the "transducer function-merging" technique from Li et al. (2019), which avoids explicitly materialising the intermediate ∂P_T/∂h Jacobian by algebraically combining the softmax derivative with the loss gradient into a single closed-form expression.

For the token logits $h^v(t, u)$ (the pre-softmax value for token v at lattice position (t, u)):

$\frac{\partial \mathcal{L}_{\text{TDT}}(y \mid x)}{\partial h^v(t, u)} = \frac{P_T(v \mid t, u) \cdot \alpha(t, u) \cdot [\beta(t, u) - b(v, t, u)]}{P_{\text{TDT}}(y \mid x)}$

where $b(v, t, u)$ is defined identically to Equation 11 above.

What it computes: The direct gradient of the loss with respect to the raw (pre-softmax) token logit for token v at state (t, u). It is proportional to: the token's own probability P_T(v|t, u) (the softmax output), times the forward probability α(t, u) (how likely we are to be at this state), times the difference β(t, u) - b(v, t, u).

Why this form: The expression β(t, u) - b(v, t, u) has an elegant interpretation. β(t, u) is the total future probability from state (t, u). b(v, t, u) is the future probability specifically through emitting token v (summed over durations). The difference β(t, u) - b(v, t, u) is therefore the future probability through all emissions other than v. When this difference is large (v's future-path contribution is small relative to total future probability), the gradient pushes the logit for v downward — the model was assigning too much probability to v at this state given that most future paths go through other tokens. When b(v, t, u) dominates β(t, u) (v is the primary token used in good paths from this state), the difference becomes negative, pushing the logit upward.

Why not separate softmax backprop: Computing ∂L/∂h via the chain rule would require two steps: first compute ∂L/∂P for all tokens (a vector of size vocab_size), then multiply by the vocab_size × vocab_size softmax Jacobian ∂P/∂h. The merged form collapses this into a single expression that only needs P_T, α, β, and b — all of which are already computed during the forward-backward pass. This eliminates the materialisation and multiplication of the large Jacobian, reducing both memory and compute. The paper notes that function merging is applied only to token logits, not duration logits, "since the latter usually has very small dimensions, and the negligible efficiency improvements do not outweigh the added complexity in implementation" (Section 3.2).

The appendix derivation (B.2): The function merging derivation for TDT is substantially longer than for the conventional Transducer because the b(v, t, u) terms contain duration-summed β values rather than single-step β(t, u+1) or β(t+1, u). The derivation considers three cases: v = ∅, v = y_{u+1}, and v being any other token. Each case involves algebraic manipulation to express ∂L/∂h in terms of P_T, α, β, and the duration-weighted b terms, collapsing the softmax Jacobian terms ∂P/∂h using the identity ∂y_i/∂x_j = y_i(δ_{ij} - y_j) throughout.

Logit Under-Normalisation

The paper adopts the "logit under-normalisation" method from the Multi-blank Transducer work (Xu et al., 2022) to encourage the model to emit longer durations during training. Otherwise, the model might learn to emit many short-duration tokens to minimise loss, which is easier to optimise initially but defeats the inference-time speed-up purpose.

The mechanism works by training the model with pseudo-probabilities P'_T instead of true probabilities P_T. In the log domain:

$\log P'_T(v \mid t, u) = \log_{\text{softmax}_v}(h^{v'}(t, u)) - \sigma$

where $\log_{\text{softmax}_v}$ is the standard log-softmax over the token dimension, and $\sigma > 0$ is a small constant (set to 0.05 for ASR and ST experiments, 0.02 for SLU).

What it computes: The pseudo-probability $P'_T$ is the true softmax probability divided by $\exp(\sigma)$. Since $\exp(0.05) \approx 1.051$, each token probability is scaled down by about 5%. This means that everywhere in the forward-backward computation where token probabilities are used — the $P_T(v|t,u)$ factors in the recursion — the model "sees" slightly lower probabilities than it actually produces.

Why this form: The under-normalisation reduces the apparent contribution of each token emission to the total path probability. Recall that every token emission (whether blank or non-blank) has an associated duration d and contributes P_T(v|t,u) · P_D(d|t,u) to the path probability. If we artifically scale down P_T, the model can compensate by emitting tokens less frequently — i.e., with longer durations between them — because longer-duration blanks and tokens advance t by more frames, reducing the total number of emissions needed to cover the T frames.

The gradient expression with under-normalisation (Equation 17, derived in Appendix C) is:

$\frac{\partial \mathcal{L}_{\text{TDT}}(y \mid x)}{\partial h^v(t, u)} = \frac{P_T(v \mid t, u) \cdot \alpha(t, u) \cdot \left[\beta(t, u) - \frac{b(v, t, u)}{\exp(\sigma)}\right]}{\exp[\mathcal{L}_{\text{TDT}}(y \mid x)]}$

The key change from Equation 14 is that b(v, t, u) is scaled by 1/exp(σ). This means the gradient for emitting token v at (t, u) is reduced by a factor that depends on σ — the contribution of b (the future-path-through-v) to the negative term inside the bracket is diminished. Since b appears with a negative sign, reducing b makes the whole bracket [β - b/exp(σ)] larger for tokens that are on good paths (where b is large relative to β). This effectively penalises emission decisions, pushing the model to prefer fewer emissions, which manifests as longer durations.

Task-dependent σ: The paper uses σ = 0.05 for ASR and speech translation, but σ = 0.02 for SLU. The justification (Section 4.3 footnote): "This is caused by a much smaller ratio between the audio and text length of SLU datasets: the average ratio of audio to text is 0.89:1 for SLURP, compared to around 5.5:1 for ASR for example. Since on average audio is shorter than text, setting σ too high, which encourages large duration outputs, will hurt training. A smaller σ alleviates the issue." In other words, when text is longer than audio, the model fundamentally cannot emit many long-duration tokens because there aren't enough frames per token. A large σ would create a conflict between the under-normalisation pressure (emit few tokens) and the data (requires many tokens), destabilising training.

Inference Algorithm: Greedy Decoding with Frame Skipping

Algorithm 2 in the paper specifies the greedy TDT inference procedure. The critical difference from conventional Transducer greedy inference (Algorithm 1) is on lines 9 and 13:

8:  idx = argmax(joined[:vocab_size])
9:  duration_idx = argmax(joined[vocab_size:])
10: if token is not blank then
11:    hyp.append(idx2token[idx])
12: end if
13: t += duration_idx2duration[duration_idx]

What it computes: At each decoding step, the model selects the maximum-probability token (line 8) and the maximum-probability duration (line 9) from the joiner's two output heads. The token is added to the hypothesis if it's non-blank (lines 10-12). Crucially, t is incremented by the predicted duration (line 13) regardless of whether the token was blank or non-blank. This is the key departure from conventional Transducer inference, where t is only incremented (by exactly 1) when the emission is blank, and non-blank emissions leave t unchanged. In TDT, a non-blank token with predicted duration 4 advances t by 4 frames, and a blank with predicted duration 3 advances t by 3 frames.

Why this form: This inference procedure directly exploits the learned duration predictions to reduce the number of decoding steps. If the model predicts long durations for blank and non-blank emissions, t advances rapidly through the encoder output, and the while t < len(enc) loop terminates sooner. This is why speed-up correlates with maximum supported duration (Tables 1-4): the 0-8 configuration can skip up to 8 frames per emission, whereas 0-2 can only skip up to 2.

Beam search limitation: The paper explicitly states that "Beam search for TDT models is highly complex since the search space spans both token and duration dimensions" (Section 4, footnote). All reported results use greedy search. The authors acknowledge this as a limitation and flag beam search as future work, which is significant because beam search typically improves WER/BLEU over greedy decoding in conventional Transducers. The TDT accuracy advantages reported in the paper are thus measured against a greedy baseline — TDT greedy vs. RNNT greedy — and it's an open question whether TDT beam search would close or widen the gap relative to RNNT beam search.

Batched Inference and the Loss Sampling Solution

Batched inference — processing multiple utterances simultaneously — is critical for production throughput, but TDT faces a fundamental challenge: different utterances in the same batch may predict different durations at the same decoding step. If utterance A predicts duration 3 and utterance B predicts duration 6, how many frames should the batch as a whole advance?

The naive approach and its failure: The paper initially tried "selecting the minimum of predicted durations" — in the example, advancing the whole batch by 3 frames. However, this "resulted in significantly increased insertion errors with the same tokens repeated multiple times." The reason is subtle: when utterance B is forced to advance by only 3 frames instead of its predicted 6, it finds itself at a time step where the acoustic context is essentially unchanged (the model expected to skip 6 frames of silence or steady sound). The model is "not ready to emit the next token, but can only emit previously emitted tokens instead" — it gets stuck emitting the same token repeatedly because the decoder state and the (barely-changed) acoustic input continue to favour the same emission decision.

The loss sampling solution (Equation 18): To fix this, the paper introduces a stochastic training objective that mixes TDT loss with conventional Transducer loss:

$\mathcal{L}_{\text{sampled}} = \begin{cases} \mathcal{L}_{\text{Transducer}}, & \text{with probability } \omega \\ \mathcal{L}_{\text{TDT}}, & \text{with probability } 1 - \omega \end{cases}$

where $\omega = 0.1$ in the experiments (Table 7). For 10% of training steps, the model is trained with the standard Transducer loss, which forces it to also learn the "advance by 1 frame per blank, stay on same frame per non-blank" behaviour of a conventional Transducer. For 90% of steps, it trains with the TDT loss.

What it computes: When the sampled loss selects L_Transducer, the duration logits are not used and not updated — the loss is computed only from the token logits using the conventional Transducer probability model. The duration head is effectively frozen for that step. This interleaving of objectives produces a model that can both predict multi-frame durations (from TDT training) and handle being advanced by fewer frames than expected (from the Transducer training, which teaches the model what to do when t doesn't jump as far as desired).

Why this form: The mixture objective acts as a regulariser that prevents the model from becoming "brittle" — overly dependent on its predicted durations being exactly followed. Without this, the model never experiences the mismatch between predicted and actual frame advancement during training, so at batched inference time, when the minimum-duration constraint forces sub-optimal advancement, it has no learned behaviour for that situation. The 10% Transducer loss provides exactly this exposure. Table 7 shows that ω = 0.1 not only resolves the insertion error problem but sometimes slightly improves accuracy (e.g., TDT 0-2 achieves 4.94% WER on test-other vs. 5.50% in the non-batched setting of Table 2), and maintains speed-ups of 1.51× to 1.88× for batch size 4.

Speed-up in batched mode: The paper notes that "The speed-up for batched inference is slightly smaller than for non-batched case because 1. the overhead related to padding for batched computation and 2. all utterances in the batch advance by the minimum of predicted durations which increases the number of decoding steps." For example, TDT 0-8 achieves 2.12× speed-up non-batched (Table 2, test-other) but only 1.79× batched (Table 7).

Full Training and Model Configuration

The paper uses a consistent architecture across all tasks:

Encoder: Conformer-Large with 17 layers, 8 attention heads, hidden dimension 512, feed-forward expansion factor 4, convolution kernel size 31, relative position embeddings. The encoder begins with convolution-based subsampling at rate 4, meaning the original 10ms audio frames (with 25ms windows) are downsampled to 40ms frames before entering the conformer stack. Total parameters are approximately 120M, varying slightly with vocabulary size and duration set size.

Decoder: Stateless decoder as proposed by Ghodsi et al. (2020), which concatenates the embeddings of the last 2 history words. This avoids the sequential computation of an LSTM decoder while still providing linguistic context.

Acoustic features: 10ms frame interval, 25ms window size, producing mel-spectrogram features (standard for ASR).

Training regime: All models trained for "no more than 200 epochs" with the Adam optimiser. Checkpoint averaging is performed on the 5 best checkpoints (by validation performance) to produce the final evaluation model. FastEmit regularisation is used with λ = 0.01 for ASR models.

Text representation: Byte-Pair Encoding (BPE) with vocabulary size 1024 for ASR tasks, and YouTokenToMe with 16k vocabulary for speech translation. The SLU task uses a structured text representation where intents and slots are formatted as a Python dictionary string.

Duration configurations tested: All use consecutive integers starting from 0, denoted by shorthand: "0-2" means D = {0, 1, 2}, "0-4" means D = {0, 1, 2, 3, 4}, "0-6" means D = {0, 1, ..., 6}, and "0-8" means D = {0, 1, ..., 8}.

Regularisation: FastEmit with λ = 0.01 for ASR. Logit under-normalisation with σ = 0.05 for ASR and ST, σ = 0.02 for SLU. The sampled loss (for batched inference experiments) uses ω = 0.1.

Design Decisions and Their Justifications

Why a separate duration head rather than a single joint distribution? A single softmax over vocab_size × |D| outputs would require the model to learn a flat distribution over all (token, duration) pairs. This has two disadvantages: (1) the output dimension grows multiplicatively, increasing parameter count and making optimisation harder, and (2) the model cannot independently specialise — the token prediction and duration prediction would be entangled in a way that prevents learning, for example, that certain tokens tend to have long durations regardless of the specific token identity. The factorised form P_T · P_D with independent normalisation allows each head to specialise while the joint network's hidden representation captures their interaction.

Why disallow blank-with-duration-0? A blank that advances t by 0 frames and u by 0 tokens is a self-loop that adds no information — it's a pure computational waste that could cause the model to learn degenerate paths where it emits many zero-duration blanks at the same time step without making progress. Disallowing it simplifies both the mathematics and the training dynamics. The paper notes duration-0 is empirically rarely predicted anyway (Figure 4), so the restriction has minimal practical impact on model expressiveness.

Why allow non-blank-with-duration-0? This preserves the conventional Transducer behaviour where non-blank emissions don't advance t — useful when multiple tokens correspond to the same acoustic frame (e.g., very fast speech, character-level output). Removing it would force every non-blank token to advance time, which could hurt accuracy on rapid or character-dense outputs.

Why use function merging only for token logits? The duration head has very few outputs (e.g., 5 to 9 values, compared to 1024+ for the token head). The computational overhead of the separate softmax-backprop chain for the duration head is negligible, while the complexity of deriving and implementing the merged form for the duration logits is non-trivial. This is a pragmatic engineering tradeoff.

Why choose σ = 0.05 for ASR and 0.02 for SLU? The under-normalisation strength controls the pressure toward longer durations. For ASR, where audio is roughly 5.5× longer than text (in frames vs. tokens), the model has ample room to learn long durations — the ratio naturally supports average durations of ~5 frames per token. σ = 0.05 provides enough pressure to encourage this without destabilising. For SLU, where audio is often shorter than text, the model cannot afford long durations on average (each token covers less than 1 frame on average), so strong under-normalisation pressure would conflict with the data, potentially causing the training loss to diverge. σ = 0.02 provides gentler encouragement toward duration usage without forcing impossible alignments.

Why not combine PRM search with revisions? (Not applicable — this is from the reference example. Skipping.)

Why the T+1 boundary convention for α and β? With multi-frame skips, transitions can land exactly at or beyond T. The α(T+1, U) convention cleanly handles the case where a final blank emission with duration d jumps from some t ≤ T to t+d > T. In the conventional Transducer, the final blank always goes from T to T+1, so α(T, U) P(∅|T, U) suffices. For TDT, the general boundary condition captures all paths that cross the end of the sequence, regardless of where they originated.

4. Key Insights and Innovations

Innovation 1: Duration as an Explicit First-Class Output Rather Than an Implicit Latent Variable

The fundamental conceptual move in this paper is elevating token duration from an implicit, distributed property of alignment paths to an explicit, predicted model output. This is not an incremental extension of the Transducer — it changes what the model represents about the input-output mapping.

Before TDT, the dominant paradigm for Transducer models treated the temporal relationship between audio frames and output tokens as a soft alignment learned through the blank mechanism. A token's temporal extent — how many frames it "covers" — was never directly represented anywhere in the computation graph. You could infer approximate durations post-hoc by examining the alignment lattice (e.g., counting how many blank emissions precede each non-blank), but the model itself had no explicit notion of "this subword spans frames 40 through 45." The Multi-blank Transducer (Xu et al., 2022) took a first step by allowing blank symbols to skip multiple frames, but even there, the skip was a property of a special blank token rather than a general mechanism applicable to all emissions, and non-blank tokens remained duration-less — they always covered zero frames in the temporal axis.

TDT's innovation is reframing the problem: every emission decision — blank or non-blank — is simultaneously a decision about "what" and "for how long." The conditional independence assumption (P(v,d) = P_T(v) × P_D(d)) encodes the insight that token identity and temporal extent are related but separable aspects of the speech-to-text mapping. A vowel tends to be longer than a stop consonant regardless of which specific vowel it is; a blank during silence should cover many frames regardless of what token comes next. By factorising the prediction into two independently normalised heads, TDT enables the model to learn duration-general patterns (e.g., "pauses between sentences are long") that transfer across token identities, while still allowing token-specific duration patterns (e.g., certain words are spoken faster) through the shared hidden representation of the joint network.

This conceptual reframing matters because it converts a latent alignment problem into a supervised prediction problem. In a conventional Transducer, the model's temporal behaviour emerges from the optimisation of the marginal likelihood over all alignments — there's no direct signal telling the model how many frames a token should cover. In TDT, the forward-backward computation propagates gradients through both the token and duration probabilities, providing direct supervision for the duration head: a duration d gets positive gradient when advancing by d frames connects a high-probability prefix to a high-probability suffix (Equation 12). The model learns durations by being rewarded for temporal jumps that produce globally consistent alignments.

The evidence that this is a fundamental shift rather than a trick is the paper's demonstration that TDT provides benefits beyond inference speed-up: it improves accuracy on speech translation (over +1 BLEU on MUST-C, Table 5) and spoken language understanding (up to +1.5% absolute intent accuracy on SLURP, Table 6), and dramatically improves robustness to noise (Figure 6) and repeated tokens (Table 8). These are not speed-related improvements — they reflect better modelling of the underlying sequence-to-sequence mapping, consistent with the idea that explicit duration modelling provides a useful inductive bias for temporal structure in speech.

Innovation 2: The Difficulty-Dependent Nature of Duration-Based Speed-Up as a Diagnostic Principle

The paper's second conceptual contribution is more subtle: it establishes — through empirical evidence across three tasks with different audio-to-text length ratios — that the effectiveness of duration-based frame skipping is fundamentally governed by the ratio of input frames to output tokens, not by any absolute property of the model or task. This provides a diagnostic framework for predicting when TDT-like approaches will help, and by how much.

Consider the speed-up factors the paper reports. On German ASR (MLS dataset), TDT 0-8 achieves 2.82× speed-up. On English ASR (LibriSpeech), the same configuration achieves 2.19× on test-clean and 2.12× on test-other. On Spanish ASR (CallHome), it achieves 1.96×. On speech translation, 2.27×. But on SLURP (spoken language understanding), the speed-up drops to only 1.28×. The paper explicitly explains this pattern: German MLS has the longest average text sequences (68 subword tokens per utterance, per the footnote in Table 4), giving an audio-to-text length ratio around 7:1 or higher. SLURP has an average ratio of 0.89:1 — the text is longer than the audio. In ASR generally, the ratio is roughly 5.5:1 to 7:1.

What makes this a diagnostic insight rather than merely an observation is that it provides a predictive boundary condition. You can estimate the expected maximum speed-up for a new task by computing the audio-to-text length ratio: if the ratio is R:1, the maximum achievable speed-up is bounded by roughly R (since each token can cover at most R frames on average if every token, including blanks, perfectly covers its temporal span). The paper's data validates this: LibriSpeech test-other has R ≈ 5.5, and TDT 0-8 achieves 2.12× speed-up — not reaching the theoretical maximum because the model doesn't always predict the longest possible duration, but getting close. SLURP has R ≈ 0.89, and TDT achieves only 1.28× — there's simply not enough temporal extent per token to build large skips.

This also explains why the paper's logit under-normalisation strength σ must be tuned per-task (0.05 for ASR/ST, 0.02 for SLU). When R < 1, encouraging long durations through σ creates a conflict with the data — the model needs to emit more tokens than there are frames, which is impossible if each token covers multiple frames. The training instability the paper reports for σ = 0.05 on SLURP is a direct consequence of this conflict.

This diagnostic framing is significant beyond the paper's immediate results because it provides a reusable heuristic for practitioners: before implementing TDT, estimate the audio-to-text ratio of your task. If it's substantially above 1, expect meaningful speed-ups. If it's near or below 1, duration-based skipping will provide marginal benefits. This transforms TDT from a "try it and see" architecture to one whose applicability can be assessed analytically.

Innovation 3: The Loss Sampling Strategy as a General Solution to Training-Inference Mismatch in Duration Models

While Sections 5.2 appears modest — a half-page subsection on batched inference — the loss sampling solution it introduces (Equation 18) represents a conceptual advance with implications beyond TDT: interleaving the training objective with a simpler, non-duration-based loss resolves the mismatch between independently predicted per-utterance durations and the synchronised advancement required by batched inference, without sacrificing the benefits of duration modelling.

The problem this solves is fundamental to any architecture that makes state-dependent skip predictions processed in batch. When different utterances predict different skip amounts, the batch-as-a-whole must choose a single advancement (the paper uses the minimum), which means some utterances advance less than their model predicted. This creates an out-of-distribution input at the next step: the model encounters acoustic features from a time step it didn't expect to be at, and its decoder state reflects a hypothesis built under the assumption of a larger skip. The naive consequence — insertion errors from stuck tokens — is a failure mode not just for TDT but for any batched sequence model with adaptive frame skipping.

The prior approach to such mismatches would typically be architectural: design the model to accept any advancement amount, or use dynamic batching that groups utterances by predicted skip. Both add complexity and have their own failure modes. The paper's insight is instead to treat the problem as a training objective design issue: expose the model during training to the possibility that its duration predictions will not be perfectly followed, so that at inference time the mismatch is within distribution.

The mechanism — stochastically switching 10% of training steps to conventional Transducer loss, where the duration head receives no gradient — is elegantly minimal. It doesn't require architectural changes, auxiliary losses, or modified inference procedures. It doesn't even require changing the TDT loss itself. It simply acknowledges that the duration head is predicting an ideal advancement that may not be honoured at batch inference time, and ensures the token prediction head (which is shared across both losses) learns to be robust to this.

The evidence (Table 7) shows this works remarkably well: batched TDT 0-8 achieves 2.13%/5.03% WER on test-clean/test-other — comparable to or better than the non-batched results (2.11%/5.16% from Tables 1-2) — while preserving 1.79× speed-up. The fact that accuracy sometimes improves with the sampling strategy (e.g., TDT 0-2: 2.10%/4.94% batched vs. 2.35%/5.50% non-batched) suggests the Transducer loss interleaving may act as a regulariser that prevents the model from overfitting to its own duration predictions.

What makes this an innovation rather than an engineering trick is that it identifies a general principle for training models with adaptive computation: when the inference-time execution of your model's predictions may deviate from the training-time assumption (due to batching constraints, hardware limitations, or routing decisions), stochastic interleaving with a simpler objective that doesn't rely on those predictions can provide robustness without sacrificing the benefits of adaptive computation. This principle potentially extends to other architectures with learned skipping, early exiting, or dynamic routing — anywhere the model makes decisions about how much computation to perform that may be overridden by system-level constraints.

Innovation 4: Verifier-Free Duration Learning Through Global Alignment Consistency

The paper's approach to training the duration head represents a methodological insight that distinguishes TDT from alternative duration modelling approaches: durations are learned entirely from the global consistency of the alignment lattice, without requiring external duration labels, forced alignments, or auxiliary duration-specific loss terms. This makes the method self-supervised with respect to duration and broadly applicable to any sequence transduction task where the input-output alignment is monotonic.

Prior work on duration modelling in speech typically required explicit duration supervision: either from forced alignments produced by a separate model (e.g., a hybrid ASR system that aligns phonemes to frames), or from an external duration predictor trained on such alignments. Even approaches that learned durations implicitly, like the variational attention mechanisms in some text-to-speech models, often required auxiliary duration losses or architectural components specifically designed to capture temporal extent. The Multi-blank Transducer used a similar forward-backward training approach for its big blank durations, but limited it to blank symbols only — the duration signal came entirely from blank placement in the lattice.

TDT extends this self-supervised principle to all emissions. The gradient derivation (Equation 12 and the c(d,t,u) terms in Equation 13) shows that the duration head receives signal from two sources: (1) when an emission of token v with duration d connects a high-probability prefix α(t,u) to a high-probability suffix β(t+d, u(+1)), weighted by the token probability P_T(v|t,u), and (2) the complementary case for blank emissions. This is a pure consistency gradient: the model is rewarded for duration predictions that make the overall alignment more probable, without ever being told what the "correct" duration for any specific token should be.

The significance of this is that it eliminates the need for any duration-labelled training data. You don't need to know how long each phoneme, word, or subword actually lasted in the audio. The model discovers the temporal structure of speech from the requirement that its token predictions and duration predictions jointly produce high-probability alignments with the target text. This is what makes TDT applicable to speech translation (where word order differs between source and target languages, making traditional forced alignment impossible) and spoken language understanding (where the output is structured text rather than a transcription, making temporal alignment of output tokens to audio frames even less well-defined).

The evidence for the effectiveness of this self-supervised duration learning is in the duration distributions (Figure 4): the model learns to use longer durations predominantly for blank emissions (which correspond to pauses and silence), and to use moderate durations for non-blank emissions that roughly match the typical speaking rate of the training data. The model wasn't told to do this — it emerges from the global alignment consistency pressure. The alignment visualisations in Appendix D (Figures 7-10) further confirm that the durations learned on synthetic data produce intuitively correct diagonal alignments with slopes matching the input-output length ratio, and respond appropriately to FastEmit regularisation by shifting token emissions earlier.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three distinct tasks, each with its own dataset(s): (1) English ASR: LibriSpeech 960-hour training set (Panayotov et al., 2015), evaluated on test-clean and test-other; (2) Spanish ASR: Combined training from Mozilla Common Voice, Multilingual LibriSpeech (MLS), Voxpopuli, and Fisher (total ~1340 hours), evaluated on Spanish CallHome (LDC96S35); (3) German ASR: Combined training from MCV, MLS, and Voxpopuli (~2000 hours), evaluated on MLS test set; (4) English-to-German Speech Translation: Combined MUST-C V2, CoVoST V2, ST-TED, Europarl-ST, plus English audio from CommonVoice v6 and VoxPopuli v2 with German text from an NMT model trained on WMT21 data, evaluated on MUST-C V2 Test; and (5) Spoken Language Understanding (SICSF): SLURP dataset (Bastianelli et al., 2020), evaluated with intent accuracy and SLURP-F1. All ASR training data is publicly available.

Base model(s). All experiments use Conformer-Large (~120M parameters) with 17 layers, 8 attention heads, hidden dimension 512, feed-forward expansion factor 4, convolution kernel size 31, and relative position embeddings. The encoder applies convolution-based subsampling at rate 4, converting 10ms audio frames (with 25ms windows) to 40ms encoder frames. The decoder is a stateless prediction network (Ghodsi et al., 2020) that concatenates embeddings of the last 2 history words. The exact parameter count varies slightly with vocabulary size and duration set size. The authors argue this model is "representative of the capabilities of many contemporary LLMs" in the speech domain.

Metrics. (1) Word Error Rate (WER) for ASR, computed using the standard edit-distance metric. (2) BLEU score for speech translation. (3) Intent accuracy and SLURP-F1 for spoken language understanding. (4) Inference time (in seconds) measured as total decoding time for the test set, with relative speed-up factor computed as the ratio of baseline inference time to TDT inference time. All evaluations use non-batched greedy search inference (beam search is noted as future work due to complexity in the joint token-duration search space).

Baselines. (1) RNNT: A conventional Conformer Transducer with stateless decoder, trained with FastEmit (λ = 0.01), serving as the primary baseline across all tasks. (2) Multi-blank Transducer (MBT) (Xu et al., 2022): Compared directly in Section 5.5 (Table 9) with matched maximum duration configurations. (3) For speech translation, the best publicly available model at writing time from Indurthi et al. (2021) is included for reference (28.88 BLEU), though trained on different data and not directly comparable. (4) For SLU, two state-of-the-art baselines from ESPNet-SLU (Arora et al., 2022) and SpeechBrain (Wang et al., 2021c) are included, both using HuBERT encoders pretrained on LibriLight-60k. Unless noted, no external language model is used.

Generation budget / compute accounting. The universal unit of test-time compute is inference time (wall-clock seconds for decoding the full test set), measured for non-batched greedy search (batch size = 1) in the main results, and for batched inference (batch size = 4) in Section 5.2. For the MBT comparison in Table 9, inference time is measured under identical conditions. The paper does not report FLOPs directly, relying instead on measured wall-clock time, which captures the practical impact of reduced decoding steps. All speed-up factors are relative to the RNNT baseline under the same inference configuration.

Cross-validation / statistical protocol. No cross-validation is reported. Checkpoint averaging is performed on the 5 best checkpoints (by validation set performance) to produce the final evaluation model. All models are trained for "no more than 200 epochs." For the SLU task, the RNNT baseline is initialised from a pretrained ASR model. For TDT configurations, the paper tests consecutive duration sets denoted by shorthand (e.g., "0-4" means D = {0, 1, 2, 3, 4}) and compares five settings: 0-2, 0-4, 0-6, 0-8, plus the RNNT baseline.

Main Quantitative Results

English ASR on LibriSpeech (Tables 1 and 2)

On test-clean (Table 1), the RNNT baseline achieves 2.14% WER with 256 seconds decoding time. TDT 0-8 matches this accuracy (2.11% WER) while running 2.19× faster (117 seconds). TDT configurations show a consistent pattern: accuracy remains flat or slightly improves as maximum duration increases — 0-2: 2.35%, 0-4: 2.17%, 0-6: 2.14%, 0-8: 2.11% — while speed-up grows from 1.46× to 2.19×. TDT 0-2 is the only configuration that degrades WER relative to RNNT (2.35% vs. 2.14%).

On test-other (Table 2), the RNNT baseline achieves 5.11% WER with 244 seconds. The best TDT accuracy is 5.05% WER (0-6 configuration) with 2.07× speed-up. TDT 0-8 achieves 5.16% WER (slightly worse than baseline) at 2.12× speed-up. The accuracy range across TDT configurations is narrow: 5.05% to 5.50%, with 0-4 (5.06%) and 0-6 (5.05%) matching or beating the baseline. As with test-clean, TDT 0-2 shows the largest accuracy degradation (5.50%).

The key pattern: TDT with sufficiently large maximum duration (≥4) matches or improves upon RNNT accuracy while providing substantial speed-ups, and the accuracy is not monotonic with duration — 0-6 edges out 0-8 on test-other.

Spanish ASR on CallHome (Table 3)

This is where TDT shows the largest accuracy improvements. The RNNT baseline achieves 19.84% WER with 47 seconds decoding time. Every TDT configuration outperforms the baseline by a substantial margin: TDT 0-2 achieves 17.95% WER (nearly 2 absolute points better) at 1.42× speed-up; TDT 0-6 achieves 18.06% WER at 1.96× speed-up; TDT 0-8 achieves 18.73% WER at 1.96× speed-up. The accuracy gain is largest for the smallest maximum duration (0-2: −1.89 absolute WER), and diminishes slightly as maximum duration grows (0-8: −1.11 absolute WER). Speed-up plateaus at 1.96× for both 0-6 and 0-8.

This result is significant because it demonstrates TDT can provide both better accuracy and faster inference simultaneously — there is no accuracy-speed tradeoff on this dataset. The CallHome dataset has conversational Spanish with natural pauses and disfluencies, which may benefit from explicit duration modelling more than read speech.

German ASR on MLS (Table 4)

The RNNT baseline achieves 3.99% WER with 558 seconds. TDT 0-4 achieves the best WER at 3.93% (marginally better than baseline) with 2.41× speed-up. TDT 0-8 achieves 3.95% WER with the maximum observed speed-up across all ASR experiments: 2.82× (198 seconds vs. 558). The speed-up is notably larger than for English and Spanish because MLS has longer average text sequences: the footnote to Table 4 explains that MLS utterances average 68 subword tokens, compared to ~20 for LibriSpeech and ~40 for Spanish CallHome. Since the decoder is autoregressive and must be run sequentially, longer text sequences means more decoding steps dominate total inference time, so reducing the number of steps via frame-skipping yields proportionally larger wall-clock gains.

Speech Translation (Table 5)

On MUST-C V2 Test (English-to-German), the RNNT baseline achieves 23.21 BLEU with 218 seconds. TDT configurations consistently improve BLEU: 0-2: 24.03 (+0.82), 0-4: 24.15 (+0.94), 0-8: 24.47 (+1.26). Speed-up grows from 1.52× to 2.27× as maximum duration increases. This is the strongest evidence that duration modelling provides an inductive bias that improves modelling capacity, not just inference efficiency — speech translation involves cross-lingual reordering where the alignment between source acoustic frames and target-language tokens is more complex than in monolingual ASR. The fact that TDT's BLEU gains are consistent and monotonic with maximum duration suggests the duration head helps the model learn better input-output alignments.

The reference model from Indurthi et al. (2021) achieves 28.88 BLEU but is trained on different data and uses task-aware multi-task learning, so it serves only as a point of reference, not a direct comparison.

Spoken Language Understanding on SLURP (Table 6)

The RNNT baseline achieves 88.53% intent accuracy and 79.41 SLURP-F1. Several TDT configurations improve upon this: TDT 0-4 achieves 89.85% intent accuracy (+1.32 absolute) and 80.03 F1; TDT 0-6 achieves 89.28% intent accuracy and the best F1 at 80.61; TDT 0-8 achieves 90.07% intent accuracy (the highest, +1.54 absolute) and 79.90 F1. The speed-up is modest compared to ASR/ST: 1.17× for 0-2 and 0-4, and 1.28× for 0-6 and 0-8.

This setting is uniquely valuable because it tests TDT in a regime where the audio-to-text length ratio is very different from ASR. The paper notes (Section 4.3) that SLURP has an average audio-to-token ratio of 0.89:1 — the text is actually longer than the audio on average. The smaller speed-up is therefore expected and predicted by the paper's own diagnostic framework: when there are few frames per token, long-duration skips are rarely applicable. The fact that TDT still achieves 1.28× speed-up with accuracy improvements demonstrates that the method is not brittle to this ratio, even if the efficiency gains are naturally bounded by the data statistics.

The TDT models also establish new state-of-the-art on SLURP: both ESPNet-SLU (86.52% intent accuracy, 76.91 F1) and SpeechBrain (87.7% intent accuracy, 76.19 F1) are outperformed by the RNNT baseline (88.53%, 79.41), and TDT pushes this further.

Batched Inference Results (Table 7)

With batch size 4 and loss sampling (ω = 0.1), TDT models trained with the mixed objective achieve: TDT 0-2: 2.10%/4.94% WER (test-clean/test-other), outperforming the RNNT baseline (2.13%/5.11%) at 1.51× speed-up; TDT 0-6: 2.10%/4.91% WER (the best test-other WER in this table) at 1.88× speed-up; TDT 0-8: 2.13%/5.03% WER at 1.79×. The speed-ups are smaller than non-batched inference (compare 1.79× vs. 2.12× for 0-8 on test-other from Table 2) due to (1) padding overhead for batched computation and (2) all utterances in the batch advancing by the minimum predicted duration, which increases the number of decoding steps.

Notably, the loss sampling training sometimes improves accuracy compared to pure TDT training: batched TDT 0-2 achieves 4.94% WER on test-other vs. 5.50% for non-batched TDT 0-2 in Table 2. This suggests the interleaved Transducer loss acts as a regulariser that prevents the model from overfitting to its own duration predictions.

TDT vs. Multi-Blank Transducers (Table 9)

On LibriSpeech test-other, with matched maximum durations, TDT consistently achieves larger speed-ups at comparable WER: at max-duration 2, MBT achieves 5.15% WER at 1.17× speed-up vs. TDT at 5.50% WER at 1.43× (+22% relative speed-up for TDT); at max-duration 4, MBT: 5.05% WER at 1.52× vs. TDT: 5.06% WER at 1.91× (+26% relative); at max-duration 8, MBT: 5.18% WER at 1.76× vs. TDT: 5.16% WER at 2.12× (+20% relative). The speed-up advantage of TDT over MBT grows as max-duration increases, which is expected since TDT can skip frames on both blank and non-blank emissions while MBT can only skip on big blanks. The WER numbers are essentially tied at all configurations, confirming that TDT's speed advantage does not come at an accuracy cost.

Noise Robustness (Figure 6)

On LibriSpeech test-clean augmented with noise at SNRs of 0, 5, 10, 15, 20, and +∞ (clean), TDT 0-8 and RNNT perform similarly at high SNRs (clean: both ~2.1% WER; SNR 20: both ~3% WER). As noise increases, TDT pulls ahead: at SNR 10, TDT is roughly 2 absolute WER points better; at SNR 5, the gap widens to roughly 5-6 points; at SNR 0, TDT achieves roughly 50% WER vs. roughly 63% for RNNT. The inference time for TDT remains essentially constant across all SNR levels. The paper does not provide exact numbers in the text, only the Figure 6 plot and the statement that "TDT models gradually outperform RNNT as more noise is added."

Repeated Tokens Robustness (Table 8)

On a TTS-generated dataset of 100 utterances with digits repeated 3-5 times each, RNNT with LSTM decoder achieves 59.95% WER, and RNNT with stateless decoder achieves 64.62% WER — both essentially unusable. TDT models are dramatically better: TDT 0-2: 12.59%, 0-4: 9.35%, 0-6: 6.12%, 0-8: 5.78%. The WER improves monotonically with maximum duration, showing more than a 10× error reduction from the best RNNT (5.78% vs. 59.95%). The paper attributes this to TDT's duration predictions preventing the model from staying on the same frame and emitting the same token repeatedly (the "infinite loop" failure mode of RNNTs with repeated tokens).

Ablation Studies and Robustness Checks

Duration configuration sweep (Tables 1-4, 6): Across all tasks, increasing maximum supported duration from 2 to 4 provides the largest jump in speed-up (e.g., LibriSpeech test-other: 1.43× → 1.91×), with diminishing returns from 4 to 8 (1.91× → 2.12×). Accuracy is largely flat across configurations, with 0-2 occasionally worse (LibriSpeech test-clean: 2.35% vs. baseline 2.14%) and no configuration being uniformly best across all datasets. This suggests that maximum duration is best chosen based on the dataset's audio-to-text ratio rather than optimised per-task — the paper settles on 0-8 for most experiments but notes that 0-6 can edge out 0-8 on some metrics.

Loss sampling probability ω for batched inference (Table 7 vs. Tables 1-2): Training with ω = 0.1 (10% conventional Transducer loss) enables batched inference without insertion errors. The paper does not sweep ω, but the results in Table 7 demonstrate that the approach works. Accuracy with ω = 0.1 sometimes exceeds non-batched TDT accuracy (e.g., TDT 0-6 test-other: 4.91% batched vs. 5.05% non-batched), suggesting the mixed objective provides beneficial regularisation. The speed-up in batched mode is 10-20% lower than non-batched due to minimum-duration batching and padding overhead.

Under-normalisation strength σ across tasks: The paper uses σ = 0.05 for ASR and ST, and σ = 0.02 for SLU. The SLU reduction is justified by the low audio-to-text ratio (0.89:1) — the footnote in Section 4.3 states that σ = 0.05 "may destabilize training for SLURP." This is an important sensitivity: the under-normalisation strength must be tuned to the dataset's temporal characteristics, and the failure mode (training instability) is severe enough to require the tuning.

MBT vs. TDT comparison (Table 9): This serves as the primary architectural ablation, testing whether duration prediction on all emissions (TDT) provides benefit over duration only on blanks (MBT). The result is clear: at matched max-duration, TDT provides 20-26% larger relative speed-up with equivalent WER, confirming the value of extending frame-skipping to non-blank emissions.

RNNT with different decoders for repeated tokens (Table 8): RNNT-LSTM (59.95% WER) and RNNT-stateless (64.62% WER) both fail catastrophically on repeated tokens, showing the problem is not specific to the decoder architecture — it's a fundamental issue with the Transducer's frame-by-frame blank/non-blank dynamics. TDT's duration mechanism inherently avoids this failure mode by advancing the time index on every emission.

Duration distribution analysis (Figures 4 and 5): Figure 4 shows that TDT 0-2 and 0-4 models use the maximum duration for nearly all emissions, while 0-6 and 0-8 models show a more uniform distribution of durations (fewer maximal-duration emissions). This is consistent with the LibriSpeech audio-to-text ratio of ~5.5:1 — on average, the model needs to cover ~5.5 frames per token, so when the maximum supported duration is 8, the model cannot use 8 every time without exceeding the total frame budget. Figure 5 shows that blank emissions decrease dramatically as maximum duration increases, nearly vanishing for 0-6 and 0-8, confirming that TDT approaches the theoretical minimum number of decoding steps.

FastEmit interaction (alignment simulations in Appendix D, Figures 8-10): While not a formal ablation, the synthetic alignment experiments show that FastEmit (λ) reduces token emission delay without preventing long-duration usage, and that the effect of duration support and FastEmit are largely orthogonal — duration controls the temporal extent of emissions, FastEmit controls when emissions begin. This supports the paper's implicit claim that these regularisation techniques compose.

TDT without logit under-normalisation: The paper does not report results for TDT trained without under-normalisation (σ = 0). This is a notable missing ablation, since the duration distributions in Figure 4 might look substantially different without the explicit pressure toward longer durations, and some of the accuracy-vs-speed characteristics could change. The paper inherits this technique from MBT (Xu et al., 2022) and treats its necessity as established.

Critical Assessment

Claim: TDT achieves better accuracy and significantly faster inference than conventional Transducers

What the experiments actually demonstrate: On English ASR (Tables 1-2), the accuracy claim is supported only in the sense that TDT matches RNNT accuracy — the differences are within 0.03% WER on test-clean and 0.06% on test-other, which are well within typical variance for LibriSpeech at this performance level. On Spanish ASR (Table 3), the accuracy claim is strongly supported: TDT configurations improve WER by 1.1-1.9 absolute points, a substantial and likely statistically meaningful gain. On German ASR (Table 4), TDT marginally improves WER (3.93% vs. 3.99% for 0-4) or matches it (3.95% for 0-8). On speech translation (Table 5), the accuracy claim is strongly supported: +0.8 to +1.3 BLEU is a meaningful improvement. On SLU (Table 6), the claim is also supported: +1.3 to +1.5 absolute intent accuracy.

Qualification: The "better accuracy" claim is dataset-dependent. On the largest and most standardised ASR benchmark (LibriSpeech), TDT does not meaningfully improve accuracy — it preserves it while providing speed-up. On the other four datasets (Spanish CallHome, German MLS, MUST-C, SLURP), TDT does improve accuracy, with the largest gains on Spanish ASR and speech translation. The paper's abstract statement that "TDT models achieve both better accuracy and significantly faster inference" is therefore true across the tested tasks in aggregate but overstates the ASR case if read as a universal claim. A more precise characterisation would be: TDT matches or improves accuracy while providing 1.3-2.8× speed-up, with accuracy improvements being larger on tasks with more complex alignment structure (translation, conversational speech, SLU).

Weaknesses: The paper does not report confidence intervals or statistical significance tests for any WER, BLEU, or F1 differences. On LibriSpeech test-clean with WER around 2.1%, a 0.03% difference could easily be noise. The test sets range from 500 utterances (LibriSpeech test splits) to ~2,600 (LibriSpeech test-other) to unknown sizes for the other datasets. Without statistical testing, whether the small accuracy improvements on LibriSpeech are real or noise is unknowable.

Claim: TDT achieves inference speed-up of up to 2.82×

What the experiments actually demonstrate: The 2.82× figure comes from German ASR on MLS (Table 4, TDT 0-8 vs. RNNT, batch size 1). Other maxima: 2.27× for speech translation (Table 5), 2.19× for English ASR test-clean (Table 1), 2.12× for English ASR test-other (Table 2), 1.96× for Spanish ASR (Table 3), 1.28× for SLU (Table 6). Batched inference (Table 7) reduces these to 1.51-1.88×.

Qualifications: (1) All speed-ups are measured with non-batched greedy search (batch size 1). Batched inference reduces speed-ups by 10-20% (compare Table 7's 1.79× for 0-8 on test-other vs. Table 2's 2.12×). (2) No beam search comparisons are made — the paper explicitly states beam search is future work. If beam search provides larger accuracy gains for RNNT than for TDT (which is unknown), the accuracy-vs-speed tradeoff might shift. (3) The German MLS speed-up of 2.82× benefits from unusually long text sequences (68 tokens/utterance on average), which amplifies the decoder's contribution to total inference time. This is the best case for TDT; typical ASR datasets with shorter utterances will see speed-ups closer to the 1.9-2.2× range. (4) All measurements are wall-clock time using the paper's PyTorch/NeMo implementation. Absolute times depend on hardware (unspecified) and implementation quality, though relative speed-ups should be less sensitive.

What's missing: No measurement of encoder computation time vs. decoder computation time breakdown. TDT's speed-up only affects the decoder portion (by reducing the number of decoding steps); the encoder cost is unchanged since it processes all frames in parallel. If the encoder dominates total time for short utterances, TDT's speed-up on total wall-clock time will be smaller than reported on datasets with longer utterances where the decoder dominates.

Claim: TDT is more robust to noise than conventional Transducers

What the experiments actually demonstrate: Figure 6 shows TDT 0-8 achieving lower WER than RNNT at SNR ≤ 10 on noise-augmented LibriSpeech test-clean, with the gap widening as SNR decreases. The experiment uses MUSAN and Freesound noise samples at 5 SNR levels (0, 5, 10, 15, 20) plus clean, with one random noise sample per utterance.

Qualifications: (1) Only one TDT configuration (0-8) is tested — it's unknown whether shorter-duration TDT models also show the robustness benefit. (2) Only one type of noise augmentation is tested (additive background noise at fixed SNRs). Other types of acoustic degradation (reverberation, channel distortion, codec compression) are not evaluated. (3) The paper provides only a figure (Figure 6), not a numerical table, making it difficult to quote exact WER differences. (4) The mechanism for noise robustness is speculated but not investigated — the paper doesn't explain why TDT is more noise-robust, though one plausible hypothesis is that long-duration blank predictions during noisy segments allow the model to effectively "wait" for clean signal rather than making uncertain token predictions frame by frame.

Claim: TDT is significantly more robust to repeated tokens than RNNT

What the experiments actually demonstrate: Table 8 shows catastrophic RNNT failure (59.95% and 64.62% WER) vs. TDT success (5.78% WER for 0-8) on a TTS-generated dataset of 100 utterances with repeated digits.

Qualifications: (1) The test set is synthetic (TTS-generated) and small (100 utterances). Whether the failure mode occurs on natural speech with repeated tokens (e.g., "seven seven seven" spoken naturally) is not tested. (2) The experiment uses digits (a very limited vocabulary), which may not generalise to repeated subword tokens in general vocabulary. (3) The comparison is one model per architecture — we don't know if the RNNT failure is reproducible across training runs or specific to this checkpoint.

Strength: Despite these limitations, the 10× error rate reduction (59.95% → 5.78%) is large enough that it's unlikely to be explained by test set size or training variance alone. The Appendix F analysis provides a plausible mechanistic explanation (RNNT's decoder state becomes essentially unchanged when the same token repeats, causing emission loops on the same frame), and TDT's duration mechanism directly breaks this loop by advancing the time index on every emission.

Missing experiments and baselines that would strengthen the paper

Missing: TDT with beam search. The paper acknowledges this as future work, but it's a significant gap because beam search typically improves Transducer accuracy by non-trivial margins. Without knowing whether TDT and RNNT benefit equally from beam search, the accuracy comparisons are incomplete. If RNNT benefits more from beam search, TDT's accuracy advantages might shrink or reverse in a beam-search comparison.

Missing: Statistical significance testing. With a 500-utterance test set (LibriSpeech test-clean/test-other) and WER differences of 0.01-0.10%, standard bootstrap confidence intervals would clarify whether the small improvements are real. This is particularly important for the "TDT achieves better accuracy" claim on English ASR.

Missing: Sweep of σ (under-normalisation strength). The paper uses σ = 0.05 for ASR and 0.02 for SLU, but never shows what happens at σ = 0 (no under-normalisation), σ = 0.1, or intermediate values. This ablation would reveal how sensitive duration usage and the speed-accuracy tradeoff are to this hyperparameter.

Missing: Sweep of ω (loss sampling probability). Only ω = 0.1 is tested for batched inference. What happens at ω = 0.05, 0.2, or 0.5? Is there a tradeoff between speed-up and accuracy as ω increases?

Missing: Comparison against a speed-matched RNNT. An alternative way to speed up RNNT is to use a smaller encoder or fewer encoder layers. How does TDT (full encoder, fewer decoder steps) compare to a smaller RNNT encoder (fewer FLOPs per step, same number of steps) at matched inference time? This would test whether the speed-up from frame-skipping is more parameter-efficient than simply reducing model size.

Missing: Latency analysis for streaming. The paper focuses on total inference time (throughput) but not on streaming latency (time to first token, or partial recognition delay). TDT with long durations might increase emission delay since tokens are emitted less frequently. FastEmit partially mitigates this (Appendix D, Figure 8), but no end-to-end latency measurements are reported with streaming decoders. For streaming ASR applications, this is a critical metric.

Missing: Encoder cost profiling. All speed-ups are reported as total wall-clock time. A breakdown showing that the decoder time is reduced while encoder time is unchanged would clarify the ceiling on TDT speed-ups (no matter how few decoding steps, encoder time remains constant) and explain dataset-dependent speed-up variation.

Summary assessment

The experiments support the paper's central contributions with appropriate breadth (three diverse tasks, multiple languages, multiple duration configurations) and sufficient depth for a methods paper. The speed-up results are robust and consistent across datasets, with the key qualifiers (audio-to-text ratio dependence, batched vs. non-batched, MBT comparison) explicitly acknowledged and investigated. The accuracy results are more mixed: on some datasets TDT meaningfully improves accuracy (Spanish ASR, speech translation, SLU), on others it's at parity (English ASR, German ASR). The noise robustness and repeated-token results are intriguing but preliminary — each is tested on a single synthetic/augmented setting and would benefit from broader validation. The most significant unaddressed question is beam search: since all results are greedy only, the practical accuracy comparison for production systems (which typically use beam search for Transducers) remains open.

6. Limitations and Trade-offs

1. All Evaluations Use Greedy Search Only — Beam Search Behaviour Is Completely Unknown

The assumption or constraint. The paper explicitly states that "Beam search for TDT models is highly complex since the search space spans both token and duration dimensions. That being said, it is possible to come up with different pruning methods to speed up the TDT beam search, which will be our future work" (Section 4, footnote 9). Every result in the paper — every WER, BLEU, F1, and speed-up measurement — uses greedy decoding only.

The consequence. This is not merely an implementation gap; it fundamentally affects how the accuracy claims should be interpreted. Conventional RNN-Transducers typically achieve 0.3–0.5% absolute WER improvement from beam search over greedy decoding on LibriSpeech-scale tasks. If RNNT benefits more from beam search than TDT does (which is entirely plausible — the RNNT search space is simpler and beam search may more effectively correct its errors), then the "TDT matches or improves accuracy" headline could reverse in the beam-search comparison that practitioners actually deploy. Conversely, if TDT's duration predictions make its greedy outputs already near-optimal, beam search may provide diminishing returns for TDT while helping RNNT substantially, narrowing or reversing TDT's accuracy advantages on speech translation and SLU.

The search space complexity is genuinely difficult: at each step, TDT chooses both a token and a duration, creating vocab_size × |D| possible transitions per beam. With beam width K, the number of active hypotheses at each step explodes combinatorially compared to RNNT's vocab_size × 1 transitions. Pruning strategies that work for RNNT beam search (e.g., pruning by cumulative log-probability, or state-level beam pruning) do not trivially extend because a hypothesis with lower current score but a well-chosen duration might skip to a more informative future frame — the value of a duration choice only becomes apparent later.

What evidence exists in the paper. None. The paper provides no beam search results, no preliminary experiments with beam search, and no analysis of how the TDT search space differs from RNNT in practice. The greedy-only limitation is stated as a footnote but never explored quantitatively. This is the single largest evidentiary gap in the paper for production-oriented readers.

Mitigation status. The paper explicitly defers beam search to future work and does not attempt any mitigation. A practitioner adopting TDT today would need to either (a) accept greedy-only decoding (which may disadvantage TDT relative to beam-search-equipped RNNT baselines in their production system), or (b) develop their own TDT beam search implementation, which the paper's own description suggests is a non-trivial research engineering problem.

2. The Difficulty Estimation Cost for Duration Policy Selection Is Not Accounted For

The assumption or constraint. TDT requires selecting a duration configuration (e.g., 0-2, 0-4, 0-6, 0-8) before training. The paper implicitly assumes this is done once based on dataset-level statistics (the audio-to-text length ratio) rather than per-utterance. There is no mechanism for the model to dynamically adapt its maximum duration based on the input — an utterance with long silence gets the same duration support as one with rapid speech. Furthermore, the paper notes that logit under-normalisation strength σ must be tuned per-task (0.05 for ASR/ST, 0.02 for SLU), as the wrong value "may destabilise training" (Section 4.3, footnote 14), but provides no principled method for selecting σ beyond trial and error.

The consequence. The efficiency gains are configuration-dependent in ways that require dataset-specific tuning. If maximum duration is set too low (e.g., 0-2 on German MLS), the speed-up is suboptimal (1.59× vs. the achievable 2.82× from Table 4). If set too high, accuracy may degrade (0-8 achieves 5.16% WER vs. 0-6's 5.05% on test-other, Table 2) — though the accuracy cost is small in the reported experiments. More critically, for a dataset with variable utterance lengths (typical in production), a fixed global maximum duration may be simultaneously too conservative for long utterances (leaving speed-up on the table) and too aggressive for short ones (risking accuracy degradation).

The σ sensitivity is potentially more serious: training instability from incorrect σ choice is a hard failure, not a gradual degradation. The paper reports that σ = 0.05 destabilises SLURP training but does not characterise how narrowly σ must be tuned — would σ = 0.03 work? 0.04? A practitioner facing a new task with unknown audio-to-text ratio must guess σ, and guessing wrong means the model fails to train at all, not just suboptimal performance.

What evidence exists in the paper. The duration configuration sweep (Tables 1-6) shows that accuracy is relatively flat across 0-4 to 0-8 on most datasets, suggesting the maximum duration choice is not highly sensitive once above a threshold. However, the σ sensitivity is only demonstrated by a single failure case (SLURP at σ = 0.05) and a single successful adjustment (σ = 0.02), with no intermediate values tested. The alignment simulations in Appendix D (Figures 7-10) show that duration behaviour responds to input-output length ratio and FastEmit strength, but these are synthetic experiments on a 70×10 joint tensor — not a real model — and do not provide guidance for hyperparameter selection.

Mitigation status. The paper provides heuristics (audio-to-text ratio governs both maximum useful duration and appropriate σ) but no automated method, learned policy, or validation protocol for configuration selection. This means deploying TDT on a new task requires hyperparameter search across both the duration set and σ, each of which requires a full training run (up to 200 epochs). For a team with limited compute, this overhead may be prohibitive.

3. Batched Inference Speed-Ups Are Significantly Smaller Than the Headline Numbers, and the Loss Sampling Fix Is Empirically Validated but Not Theoretically Justified

The assumption or constraint. The headline speed-up figures (2.82× on German MLS, 2.19× on LibriSpeech test-clean, 2.27× on MUST-C) are all measured with batch size 1. This is explicitly noted (Section 4: "We run non-batched greedy search inference for all evaluations reported in this Section"). Batched inference — the default in production systems — is addressed separately in Section 5.2, which acknowledges both the algorithmic challenge (different utterances predict different durations) and the performance impact.

The consequence. For any production deployment using batching (which is virtually all of them — batching is essential for GPU utilisation), the effective speed-up is 15-25% lower than the headline numbers. On LibriSpeech test-other, TDT 0-8 drops from 2.12× (batch 1, Table 2) to 1.79× (batch 4, Table 7). On German MLS — the source of the paper's headline 2.82× figure — no batched results are reported at all, so the batched speed-up is unknown. A practitioner reading "2.82× faster inference" in the abstract and expecting that in their batched serving system will be disappointed.

The loss sampling fix (Equation 18, ω = 0.1) introduces its own tradeoffs that are poorly characterised. First, it requires training with both TDT and conventional Transducer losses, which means the model must be architecturally compatible with both — the duration head exists but is unused on 10% of training steps. This increases training complexity (two loss functions, two forward-backward implementations) and may slow convergence. Second, only one value of ω is tested (0.1). It's unknown whether larger ω improves batched accuracy at the cost of speed-up (more Transducer training → better batched behaviour but less duration usage), or whether smaller ω preserves more speed-up but reintroduces insertion errors.

What evidence exists in the paper. Table 7 provides the only batched results: LibriSpeech test-clean and test-other, batch size 4, ω = 0.1, for all five duration configurations. The batched speed-up ranges from 1.51× (0-2) to 1.88× (0-6), with TDT 0-8 achieving 1.79×. Accuracy in batched mode is sometimes better than non-batched (TDT 0-2 test-other: 4.94% batched vs. 5.50% non-batched), which the paper attributes to regularisation but does not investigate. No batched results exist for Spanish, German, speech translation, or SLU. No sweep of ω or different batch sizes is performed.

Mitigation status. The paper provides a working solution (loss sampling) but characterises it only at a single operating point. The gap between headline and batched speed-ups is acknowledged but not emphasised — the abstract and conclusion quote the non-batched figures without qualification. The batched inference section (5.2) is only four paragraphs and leaves the crucial question unanswered: what is the batched speed-up on the dataset that produced the 2.82× headline? Without that number, the abstract's claim cannot be evaluated in a deployment context.

4. The Approach Fundamentally Cannot Help When Audio Is Shorter Than Text (Low Audio-to-Text Ratio Regime)

The assumption or constraint. TDT's frame-skipping mechanism provides speed-up by reducing the number of decoding steps. Each decoding step produces (at most) one output token and advances by a predicted number of frames. In the best case (every non-blank emission advances by d_max frames, blanks are rarely needed), the number of decoding steps approaches U + (T/d_max), where U is the target sequence length and T is the encoder output length. When the audio-to-text ratio R = T/U is small, the second term is negligible and the total steps are dominated by U — the model must emit each token as a separate non-blank, each advancing t by (at best) a small number of frames. The speed-up over RNNT (which also requires roughly U non-blank emissions plus T blank emissions) is therefore bounded by approximately (U + T) / U = 1 + R.

The consequence. When R ≤ 1 (audio frames ≤ output tokens), the theoretical maximum speed-up is at most 2×, and in practice — since not every token will use the maximum duration, and some blanks are still needed — it is substantially less. The paper's SLURP results (Table 6) confirm this: with R ≈ 0.89, the observed speed-up is only 1.28×, despite using the 0-8 duration configuration that achieves 2.19× on LibriSpeech. Furthermore, the logit under-normalisation that encourages long durations must be weakened (σ = 0.02 vs. 0.05) to avoid training instability, and even with this adjustment, the speed-up is marginal.

This limitation is structural, not a matter of better tuning. TDT cannot overcome the fact that when there are more output tokens than input frames, the average token simply cannot cover many frames. The model is forced to emit tokens with short durations (often duration 0, meaning no frame advancement), which is essentially RNNT behaviour. The duration head adds parameters and training complexity but provides diminishing returns as R decreases.

What evidence exists in the paper. The SLURP results (Table 6) are the direct evidence: speed-ups of 1.17–1.28×, far below the 1.96–2.82× achieved on ASR/ST tasks with higher R. The paper provides the audio-to-text ratio analysis explicitly (Section 4.3: "the typical ratio between audio length to text length is around 7:1 [for ASR], and the ratio is around 0.89:1 for the SLURP testset"), and notes that "larger durations occur much less for SLURP, resulting in smaller speed-ups compared to ASR and ST." Figure 4's duration distribution analysis shows that on LibriSpeech (R ≈ 5.5), the 0-8 model uses a range of durations but rarely uses 8 — implying that even at R ≈ 5.5, the practical speed-up is below the theoretical maximum. The alignment simulations in Appendix D (Figure 9) directly demonstrate that when T is small relative to U, the model prefers short durations.

Mitigation status. The paper is admirably transparent about this limitation — it provides the R ratios, explains the mechanism, and does not overclaim about SLURP. However, the transparency does not mitigate the limitation itself. A practitioner with a low-R task (e.g., character-level ASR where R ≈ 2, or any task with verbose output like detailed captioning) should expect proportionally smaller speed-ups and must weigh the implementation complexity of TDT against modest efficiency gains. The paper provides no method to extend TDT's benefits to the low-R regime (which would require fundamentally changing the relationship between tokens and frames, perhaps by allowing multiple tokens per frame, which would violate the monotonic alignment assumption of Transducers).

5. Single Model Architecture and Acoustic Domain — No Evidence for Generalisation Beyond Conformer Transducers on Close-Talk Speech

The assumption or constraint. All experiments use the identical base architecture: Conformer-Large (~120M parameters) with a stateless decoder, trained on close-talk microphone speech datasets (read speech, conversational telephone speech, parliamentary speech). The paper states the model is "representative of the capabilities of many contemporary LLMs" but provides no evidence with other encoder architectures (QuartzNet, ContextNet, vanilla Transformers), other decoder architectures (LSTM-based), other model sizes, or other acoustic domains (far-field, noisy industrial, telephony with codec compression beyond the synthetic noise augmentation in Section 5.3).

The consequence. Several aspects of TDT's behaviour could be architecture-dependent in ways the paper cannot characterise:

• Encoder representational quality: The Conformer encoder produces frame-level representations that are locally smooth (due to convolution) and globally contextualised (due to self-attention). A pure Transformer encoder without convolution might produce less temporally coherent frame representations, making duration prediction harder — the model might struggle to learn when a single duration value is appropriate across a span of frames.
• Decoder context length: The stateless decoder uses only the last 2 tokens. An LSTM decoder with longer memory might alter the token-duration relationship — for example, if the decoder state already encodes some temporal information, the duration head might have less to learn, or conversely might benefit from richer linguistic context for predicting token-specific speaking rates.
• Model scale: All experiments use ~120M parameters. It's unknown whether TDT's duration learning works as well with smaller models (say 30M parameters, where the joint network has less capacity to simultaneously represent token and duration distributions) or larger models (say 500M+, where the baseline RNNT might already learn implicit duration cues well enough that making them explicit provides diminishing returns).
• Acoustic domain: The noise robustness experiment (Section 5.3, Figure 6) is the only departure from clean speech, and it tests only additive noise at controlled SNRs — not real-world far-field scenarios with reverberation, overlapping speakers, or channel distortion. In highly degraded acoustic conditions, the model might learn to predict very short durations (because frame-level features are unreliable, making large skips risky), eliminating the speed-up benefit.

What evidence exists in the paper. None across architectures, model sizes, or real acoustic domains. The paper is a methods paper proposing a new architecture, and the single-architecture evaluation is standard for such contributions. However, the claim that the model is "representative" (Section 4) is unverified. The noise robustness result (Figure 6) is suggestive that TDT generalises to one type of acoustic degradation, but it's a single synthetic experiment on LibriSpeech test-clean augmented with MUSAN/Freesound noise — it does not constitute evidence of domain generalisation.

Mitigation status. The paper does not address this limitation and does not claim generalisation beyond the tested settings. This is a standard scope limitation for a methods paper rather than a flaw — but a practitioner deploying TDT in a different architecture or domain should treat the speed-up and accuracy figures as unvalidated until reproduced in their setting. The open-source release in NeMo partially mitigates this by enabling others to test on their own architectures and data, but the burden of validation falls entirely on the adopter.

6. Duration Under-Normalisation Is Essential but Poorly Understood — No Ablation of the Core Training Mechanism

The assumption or constraint. Logit under-normalisation (Equation 16, σ > 0) is used in all TDT experiments. The paper inherits this technique from the Multi-blank Transducer work (Xu et al., 2022) and treats it as a necessary component — there is no experiment with σ = 0 (standard softmax, no under-normalisation). The only evidence about σ's effect comes from the SLURP training stability observation (σ = 0.05 destabilises, σ = 0.02 works) and the alignment simulations in Appendix D, which use synthetic data and demonstrate that σ affects the learned alignment but do not prove that under-normalisation is necessary for real-data training.

The consequence. It is unknown whether TDT can work at all without under-normalisation. If σ = 0 causes the model to emit very short durations (making TDT functionally identical to RNNT but with extra parameters), then the entire speed-up benefit depends on this one hyperparameter. If σ = 0 works but produces slightly lower speed-ups, practitioners could choose between a simpler training setup (no under-normalisation) and a more complex one (with σ tuning) depending on their need for speed.

More importantly, the mechanism by which under-normalisation encourages long durations is understood at the gradient level (Equation 17 shows that b(v,t,u) is scaled by 1/exp(σ), reducing the penalty for emitting tokens) but not at the learning dynamics level. Does the model initially learn short durations and then shift to longer ones as σ takes effect? Does under-normalisation cause the model to ignore certain alignment paths entirely? Could the same effect be achieved with a simpler regulariser (e.g., directly penalising short durations, or adding a duration-length bonus to the loss)? Without a σ = 0 baseline, these questions are unanswerable.

What evidence exists in the paper. Only indirect evidence. The alignment simulations (Appendix D, Figures 7-9) are all run with σ = 0.05 — no σ = 0 condition is shown. Figure 4 shows that TDT models do learn to use long durations, but this is under σ = 0.05 training. The SLURP footnote reports that σ = 0.05 causes instability, but this is a different regime (low audio-to-text ratio) and doesn't illuminate how σ = 0 would behave in the ASR/ST regime where TDT works best. The MBT comparison (Table 9) uses under-normalisation for TDT but the paper doesn't specify whether MBT also used it, making the comparison confounded if the under-normalisation treatment differs.

Mitigation status. The paper does not acknowledge this as a limitation. The under-normalisation technique is presented as a standard component of the TDT training recipe rather than as a potential confounding factor. The missing σ = 0 ablation is one of the most significant experimental gaps in the paper, because it leaves open the possibility that under-normalisation — not the joint token-duration architecture itself — is responsible for some or all of the speed-up benefit. A practitioner seeking to understand why TDT works must take on faith that both the architecture and the under-normalisation are necessary, since the paper provides no evidence separating their contributions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that duration prediction can be elevated from an implicit, distributed property of Transducer alignment paths to an explicit, jointly trained model output — and that doing so yields both inference speed-ups and accuracy improvements across multiple speech tasks. This is not a paradigm shift in the sense of replacing the Transducer framework; rather, it is a methodological reframing of what information the Transducer's probability lattice ought to represent. The paper demonstrates that the conventional Transducer's blank mechanism — where temporal advancement is a side effect of emitting a special "do-nothing" token — is unnecessarily indirect. By making duration a first-class prediction, TDT separates what to emit from how long it spans, and the model learns both from the same global alignment consistency signal.

The most significant landscape change is that TDT resolves the tension between accuracy and inference speed that has characterised Transducer research. Prior approaches forced a tradeoff: FastEmit (Yu et al., 2021) improved streaming latency at potential accuracy cost; Multi-blank Transducers (Xu et al., 2022) provided speed-up but only through blank-skipping; architectural improvements (stateless decoders, Conformer encoders) reduced per-step cost but not step count. TDT is the first method to demonstrate that both accuracy and speed can improve simultaneously on some tasks — Spanish ASR improves by nearly 2 absolute WER points while running 1.96× faster (Table 3), and speech translation improves BLEU by over 1 point at 2.27× speed-up (Table 5). Even when accuracy is merely preserved (English ASR, Tables 1-2), the speed-up is "free" in accuracy terms.

This shifts the research conversation around Transducer efficiency. Before TDT, the implicit assumption was that faster inference required sacrificing something — model size, accuracy, or architectural simplicity. TDT demonstrates that the efficiency bottleneck is not fundamental to the Transducer's autoregressive nature, but rather an artifact of the blank-as-frame-advancement design. By making every emission capable of advancing time, TDT approaches the theoretical minimum number of decoding steps (Figure 5 shows blank emissions nearly vanishing for 0-6 and 0-8 configurations), suggesting that further architectural innovations in the Transducer family may find additional efficiency by questioning other implicit assumptions rather than optimising within the existing paradigm.

Perhaps most importantly, TDT provides a reusable diagnostic for when duration-based methods will help. The paper's analysis of audio-to-text length ratio R across tasks (5.5:1 for ASR vs. 0.89:1 for SLURP) and its correlation with observed speed-ups (2.19× vs. 1.28×) gives practitioners a simple, pre-training heuristic: estimate R for your task, and if it's substantially above 1, TDT is likely to provide meaningful speed-up. This transforms duration modelling from an empirical "try it and see" proposition to a decision that can be partially justified analytically. The fact that the paper explicitly documents both the success case (ASR/ST, high R) and the diminished-returns case (SLU, low R) makes this diagnostic more credible than if only favourable results were reported.

The paper also contributes a training methodology that extends beyond TDT itself: the loss sampling strategy (Equation 18) for resolving training-inference mismatch in batched adaptive-computation models. The insight — that stochastically interleaving the primary loss with a simpler objective that doesn't depend on the adaptive predictions can make the model robust to having those predictions overridden at inference time — applies to any architecture where per-sample decisions about computation depth, routing, or skipping must be synchronised across a batch. This general principle, validated on TDT's batched inference (Table 7), may influence how future work on dynamic computation (early exiting, adaptive width, mixture-of-experts routing) handles batching constraints.

Finally, the paper rehabilitates the idea of explicit duration modelling in neural sequence transduction, which had largely been abandoned in end-to-end ASR in favour of attention-based or CTC/Transducer alignments that treat duration as latent. Previous duration models (e.g., in HMM-based hybrid systems or attention-based TTS) required explicit duration labels or auxiliary losses. TDT shows that durations can be learned self-supervisedly from the same maximum-likelihood objective that trains the rest of the model, making duration modelling practical for tasks where ground-truth temporal alignments are unavailable — crucially including speech translation, where cross-lingual reordering makes traditional forced alignment impossible. This opens the door to incorporating explicit temporal structure into other sequence-to-sequence architectures (attention-based encoder-decoders, CTC variants) without requiring additional annotation.

Follow-Up Research This Work Enables

TDT beam search: characterising the accuracy ceiling and search space. The paper's most significant open question is how TDT performs with beam search, since all reported results use greedy decoding. A strong follow-up would implement a TDT beam decoder — likely using a two-stage pruning approach: first pruning hypotheses by token-score alone (standard Transducer beam pruning), then expanding each surviving hypothesis along the duration dimension and pruning again by joint token-duration score. The key measurement would be WER/ BLEU vs. beam width for both TDT and RNNT on LibriSpeech test-clean/test-other, producing curves that answer: (a) Does TDT benefit from beam search to the same degree as RNNT? (b) At what beam width do the returns saturate for each? (c) Does TDT beam search close, maintain, or widen the greedy accuracy gap between TDT and RNNT? A negative result — RNNT benefiting substantially more from beam search than TDT — would significantly qualify the paper's accuracy claims. A positive result — TDT maintaining or growing its advantage — would strengthen the case for production adoption. The paper's acknowledgment that beam search is "highly complex" (Section 4, footnote 9) suggests this is a genuine research engineering challenge, not merely implementation work.

Joint optimisation of σ (under-normalisation) and D (duration set) across diverse audio-to-text ratios. The paper establishes that σ must be tuned per-task (0.05 for ASR, 0.02 for SLU) and that maximum useful duration depends on R. A systematic study would train TDT models on a constructed dataset with controlled R (by varying either audio length via speed perturbation or text length via tokenisation granularity), sweeping σ in {0, 0.01, 0.02, 0.05, 0.1} and D in {0-2, 0-4, 0-6, 0-8, 0-12} at each R. This would produce a phase diagram of TDT hyperparameters that practitioners could consult before training: given an estimated R for their task, which (σ, max_duration) combinations are likely to work, and which will cause training instability? The σ = 0 condition is especially important — the paper never demonstrates that under-normalisation is necessary for TDT to learn long durations, and a finding that σ = 0 works for high-R tasks would simplify the training recipe substantially. A clean negative result showing that σ > 0 is essential for any speed-up would be equally valuable, confirming that the duration-learning dynamics depend fundamentally on this regularisation.

TDT with streaming / chunked attention for latency-sensitive applications. The paper measures total decoding time (throughput) but TDT's duration mechanism has an unexplored latency implication: long-duration predictions delay token emissions, which could increase the time-to-first-token in streaming scenarios. A streaming TDT study would implement chunked attention in the Conformer encoder (so the encoder produces partial outputs as audio arrives) and measure: (a) partial recognition latency (time from audio input to each token output) for TDT vs. RNNT baseline, (b) the interaction between FastEmit strength λ and maximum duration — since Appendix D, Figure 8 shows FastEmit can reduce emission delay even with long durations enabled, and (c) whether TDT's noise robustness (Figure 6) translates to fewer corrections in streaming partial results. The hypothesis to test is that TDT's duration predictions during non-speech segments (silence, noise) serve as implicit endpointing — the model predicts a long blank duration rather than emitting uncertain tokens — which could make streaming TDT more stable in noisy conditions than streaming RNNT, where the model must emit blank after blank, each time risking a premature or delayed token emission.

Cross-architecture validation: TDT with QuartzNet, ContextNet, and LSTM decoders. All experiments use Conformer-Large with a stateless decoder. To determine whether TDT's benefits are architecture-specific, a replication study would test TDT with at least two other encoder architectures (a fully convolutional encoder like QuartzNet, which lacks self-attention and may produce less globally coherent frame representations, and a pure Transformer encoder without convolution, which lacks local smoothness) and at least one other decoder (LSTM, which maintains longer linguistic history than the stateless decoder's 2-token context). The key measurements: (a) Does TDT training converge with these architectures, or does the duration head require the Conformer's specific combination of local and global context? (b) Do the speed-up factors transfer, or do different encoders produce frame representations that make duration prediction harder/easier? (c) Does an LSTM decoder's longer memory alter the token-duration relationship — perhaps by encoding temporal information in its hidden state, reducing the need for explicit duration prediction? This would establish whether TDT is a general Transducer improvement or a Conformer-specific one, which matters for the substantial fraction of production systems using non-Conformer architectures.

TDT for open-vocabulary or character-level ASR. The paper's experiments all use subword tokenisation (BPE with vocab size 1024), where the audio-to-text ratio R is around 5.5:1. Character-level ASR would have R ≈ 2:1 (each character covers fewer frames on average), which is an intermediate regime between ASR and SLU where TDT's benefits are uncertain. A character-level TDT experiment on LibriSpeech (using the same Conformer-Large encoder, replacing BPE targets with character targets) would test: (a) what speed-up is achievable when R is moderate but still above 1, (b) whether the character-level duration modelling is reliable given that individual characters have highly variable durations (the "t" in "stop" is short but the "o" is longer), and (c) whether the explicit duration mechanism helps with the character-level Transducer's known tendency to produce insertions and deletions, by providing temporal grounding for each character emission. A positive result (meaningful speed-up with maintained or improved CER) would expand TDT's applicability to tasks where subword tokenisation is undesirable (e.g., name recognition, code-switching, or languages without good subword segmentations). A negative result would refine the R threshold below which duration modelling provides diminishing returns.

Understanding and improving TDT noise robustness: mechanism and limits. The paper shows (Figure 6) that TDT is more robust to additive noise than RNNT, with the accuracy gap widening at lower SNRs, but provides no mechanistic explanation. A follow-up would investigate why: hypothesis 1: TDT's long-duration blank predictions during noisy segments allow the model to "wait" for cleaner signal rather than making uncertain token predictions frame-by-frame; hypothesis 2: the duration head acts as a regulariser that prevents the token head from overfitting to frame-level noise patterns; hypothesis 3: the joint training of token and duration objectives produces more noise-robust encoder representations. These could be tested by: (a) analysing the alignment paths (via α(t,u)·β(t,u) as in Appendix D, Equation 52) of TDT vs. RNNT on noisy utterances, measuring whether TDT concentrates its token emissions on high-energy, cleaner frames; (b) comparing the L2 norm of encoder representations for clean vs. noisy speech in TDT vs. RNNT to see if TDT's encoder is less disrupted by noise; (c) an ablation where TDT is trained without the duration head but with the same architecture, testing whether the robustness is architectural (more parameters in the joiner) or algorithmic (duration predictions guiding attention away from noisy frames). Beyond the mechanism, testing TDT on real far-field data (CHiME, AMI, or LibriCSS) — rather than synthetically noise-augmented close-talk speech — would determine whether the Figure 6 result translates to deployment-relevant acoustic conditions.

Practical Applications and Downstream Use Cases

Production ASR serving with batched inference for cost reduction. The most direct application is replacing a conventional Transducer with TDT in a batched ASR serving system. At the batched speed-ups reported in Table 7 (1.51–1.88× for English ASR with batch size 4), a deployment transcribing 10,000 hours of audio per day could reduce its GPU-hours by roughly 35–47%. The loss sampling training strategy (ω = 0.1) enables this without the insertion errors that naive batched TDT would produce. The practical implementation path is: (1) estimate the audio-to-text ratio of the target domain (likely 4:1 to 7:1 for typical ASR); (2) select a maximum duration (0-6 or 0-8, based on the ratio); (3) train with σ = 0.05 and ω = 0.1; (4) deploy with batch size tuned to the available GPU memory, using minimum-duration advancement across the batch. The paper's open-source release in NeMo provides a starting implementation. The main deployment risk is the lack of beam search support — a system currently using RNNT with beam search would need to either (a) accept greedy TDT accuracy (which may be comparable to RNNT beam search accuracy, but this is unverified), or (b) invest in TDT beam search development.

Speech translation systems where speed and accuracy are both critical. TDT on English-to-German speech translation (Table 5) achieves +1.26 BLEU and 2.27× speed-up simultaneously. For a speech translation service (e.g., live lecture translation, video conferencing), this double benefit is unusually attractive — most efficiency techniques sacrifice translation quality. The BLEU improvement likely comes from TDT's explicit duration modelling providing a better inductive bias for the cross-lingual alignment problem, where the temporal relationship between source speech and target text is more complex than in monolingual ASR. A deployment could use TDT 0-8 with σ = 0.05 (the same settings as the paper) on the combined MUST-C/CoVoST/Europarl training recipe, achieving both lower latency (from the speed-up) and higher translation quality. The open question for practical deployment is streaming: speech translation is often used in live settings, and TDT's latency behaviour with chunked encoding (not evaluated in the paper) would determine whether the speed-up translates to reduced end-to-end delay or only to reduced compute cost.

On-device ASR for smartphones and wearables targeting the low-resource regime. TDT's speed-up comes from reducing decoder steps, which directly reduces the autoregressive computation on device. For on-device ASR with a smaller Conformer (e.g., 30-50M parameters running on a phone DSP or smart speaker), the decoder can be the latency bottleneck because it cannot be parallelised. TDT's approach — fewer steps, each slightly more expensive (due to the additional duration head) — trades off per-step cost for step count. If the duration head adds negligible compute (it's a small softmax over 5-9 values, compared to the 1024-way token softmax), the net effect is a substantial reduction in total decoder work. The SLURP results (Table 6) are particularly relevant: even when audio is short relative to text, TDT achieves 1.28× speed-up with improved accuracy, suggesting that on-device voice assistants (which use SLU-like intent/slot prediction) could benefit. The main practical barrier is the training complexity — on-device models are often trained via distillation from larger models, and distilling a TDT teacher into a TDT student (or into a smaller architecture) is untested.

Self-improvement and data generation pipelines for ASR fine-tuning. When using ASR models to generate pseudo-labels for unlabelled audio (semi-supervised learning) or to transcribe training data for downstream models, inference speed directly determines how much data can be processed within a compute budget. TDT's 1.5–2.8× speed-up translates to 1.5–2.8× more pseudo-labelled data for the same GPU allocation, which can improve downstream model quality. More interestingly, TDT's noise robustness (Figure 6) might make it a better pseudo-labeler for noisy or naturally-occurring audio than a conventional Transducer — the WER gap at SNR 10 (~2 absolute points) means TDT pseudo-labels would contain fewer errors, which is critical because errors in pseudo-labels propagate to the student model. A practical setup would: (1) train TDT on the available labelled data; (2) use TDT to transcribe a large unlabelled corpus; (3) filter transcriptions by confidence score (using either the token probability or a combination of token and duration confidence); (4) fine-tune a (possibly smaller) deployment model on the combined labelled + pseudo-labelled data. The paper's MBT comparison (Table 9, TDT 2.12× vs. MBT 1.76× at max-duration 8) suggests TDT would process substantially more unlabelled audio than an MBT-based alternative in the same time — a direct throughput advantage for semi-supervised learning pipelines.