ArXiv: 2509.14128

🎯 Pitch

A 1B-parameter speech model matches or beats giants like Whisper-large-v3 and Seamless-M4T-v2-large across 25 European languages while running 10Γ— faster. The secret isn't scale but a dynamic data balancing schedule that deliberately shifts training away from dominant parliamentary recordings toward cleaner, general-domain datasets during fine-tuningβ€”recovering a 25% relative drop in multilingual error rates.

1. Executive Summary

This report introduces Canary-1B-v2 and Parakeet-TDT-0.6B-v3, two multilingual encoder-decoder models for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST) across 25 European languages, trained on a curated 1.7M-hour dataset combining the Granary speech corpus with high-quality human-annotated data and non-speech audio to suppress hallucinations. The paper develops a two-stage pre-training and fine-tuning strategy with dynamic weight scheduling — a data balancing procedure that gradually adjusts language and corpus sampling probabilities from an initial imbalance toward a uniform target distribution according to a cosine schedule during fine-tuning (e.g., phasing out dominant narrow-domain sources like VoxPopuli parliamentary recordings while upweighting clean datasets like FLEURS) — which recovers a 25% relative Word Error Rate reduction on multilingual ASR (from 11.03% to 8.40% on FLEURS) and a 6-point absolute COMET gain on X→En translation, while a companion nGPT encoder study reveals that ALiBi positional encodings outperform RoPE for long-form ASR but underperform for translation. Canary-1B-v2 surpasses Whisper-large-v3 on English ASR benchmark accuracy with ~10× faster inference and remains competitive with Seamless-M4T-v2-large (2.3B parameters) across multilingual AST despite having less than half the parameters, establishing that specialized speech encoders with multi-stage fine-tuning can match or exceed much larger general-purpose transformer architectures on multilingual speech tasks only when the training data balancing strategy actively counteracts narrow-domain corpus dominance during the adaptation phase.

2. Context and Motivation

The Core Problem: Bridging Efficiency, Multilingual Coverage, and Multi-Task Capability in Speech Models

The paper addresses a fundamental tension in modern speech processing systems: how to simultaneously achieve strong performance across Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST) in many languages while maintaining high inference throughput. This is not merely an academic exercise in architecture design β€” it reflects concrete deployment constraints where serving speech models at scale means choosing between accuracy, language coverage, and computational cost.

The dominant paradigm for state-of-the-art speech models has been scale: train massive Transformer encoder-decoder architectures on web-scale weakly supervised data and let the model's size absorb the complexity of multilingual and multi-task learning. Whisper (Radford et al., 2023) and SeamlessM4T (Barrault & Others, 2023) exemplify this approach, with SeamlessM4T-v2-large reaching 2.3 billion parameters. These models achieve impressive results — SeamlessM4T-v2-large reports strong COMET scores on X→En translation and competitive WER on multilingual ASR — but their computational footprint creates a practical bottleneck. The paper's evaluation (Table 5) shows Whisper-large-v3 achieving an RTFx of approximately 74 on the Hugging Face Open ASR Leaderboard benchmarks, while Canary-1B-v2 reaches RTFx of 749 — roughly 10× faster inference. For production deployments handling millions of utterances daily, this difference translates directly to cloud compute costs, latency budgets, and whether models can run on-device at all.

The multi-task dimension amplifies this tension. Training separate models for ASR and AST in each language pair would multiply deployment complexity (25 languages × 3 tasks = up to 75 specialized models). A unified model that handles all language-task combinations eliminates this fragmentation, but the training data for different tasks within a given language is rarely balanced: ASR data may be plentiful while X→En translation pairs are scarce, or vice versa. The paper's Figure 4 reveals exactly this problem — ASR and En→X translation dominate the training corpus (658K and 675K hours respectively), while X→En translation lags at 368K hours. Within individual languages, the imbalance is even starker: Figure 5 shows Ukrainian with only 790 hours of ASR data versus 20K+ hours for En→X translation. A model trained naively on this mixture will specialize in the high-resource directions and underperform on the low-resource ones — precisely the opposite of what a unified model is supposed to achieve.

Why This Matters: The Narrow-Domain Problem in Multilingual Speech Data

The paper identifies a subtle but consequential issue that goes beyond raw data quantity: corpus dominance by narrow-domain sources. The VoxPopuli dataset (Wang et al., 2021), which forms the backbone of the MOSEL corpus within Granary, consists of European Parliament recordings. As the authors note in Section 3.3, for most low-to-medium resource languages, VoxPopuli "dominates, comprising over 95% of the data" (Figure 6). This creates two interrelated problems:

Domain shift. Parliamentary speech is characterized by formal register, prepared remarks, deliberate pacing, and low background noise — all of which differ substantially from the conversational, spontaneous, or noisy speech encountered in deployment (e.g., YouTube videos, customer service calls, meeting transcripts). A model trained predominantly on VoxPopuli learns to transcribe parliamentary speech well but may generalize poorly to other acoustic domains. The paper provides direct evidence for this in Section 6.1.2: Canary-1B-v2's X→En translation performance on CoVoST2 — which contains more spontaneous speech — benefits disproportionately from fine-tuning that rebalances away from VoxPopuli dominance.

Length homogeneity. VoxPopuli segments are predominantly 30 seconds long (Section 3.3), which limits the model's exposure to both very short utterances (commands, questions) and very long ones (lectures, meetings). This length bias has consequences for long-form inference: a model that rarely sees 60+ second utterances during training may exhibit degraded attention patterns or positional encoding failures when confronted with such inputs at test time, which the paper directly investigates in Section 6.4.

This domain concentration is not unique to this paper's dataset β€” it reflects a structural property of multilingual speech corpora. High-resource languages (English, Spanish, French, German) benefit from multiple diverse data sources: Common Voice (crowd-sourced, varied recording conditions), MLS (audiobooks), YouTube-derived data (informal, noisy), and parliamentary recordings. Low-resource languages often have access to only the parliamentary recordings. When training a single model across all languages, the data distribution for low-resource languages is not just smaller but qualitatively narrower. The model learns a distorted representation of what speech in those languages looks like.

Where Prior Approaches Fall Short

The paper positions itself against three classes of prior work, each with specific limitations:

1. Massive general-purpose Transformers (Whisper, SeamlessM4T). These models treat speech processing as a sequence-to-sequence problem using standard Transformer architectures trained on enormous datasets. Whisper, for instance, was trained on 680K hours of web-collected speech. The approach works β€” Whisper-large-v3 achieves strong ASR performance β€” but the paper identifies two drawbacks. First, the inference speed is limited by full self-attention over long audio sequences, which becomes the primary bottleneck (Section 2.1). The FastConformer encoder's aggressive 8Γ— subsampling and depthwise separable convolutions directly target this bottleneck, achieving a 2–3Γ— speedup over the original Conformer. Second, general-purpose Transformers lack speech-specific inductive biases β€” the convolutional modules in Conformer/FastConformer are explicitly designed to capture fine-grained local acoustic patterns (formant transitions, phoneme boundaries) that multi-head self-attention alone may struggle to represent efficiently.

2. Streaming ASR architectures (RNN-T, TDT, CTC). These models prioritize low-latency, online decoding β€” a requirement for live transcription and voice assistants. However, the paper notes (Section 2) that for speech translation specifically, "autoregressive Transformer-based decoders remain dominant, as they excel at modeling cross-lingual dependencies and generating fluent text." The RNN-T or CTC decoders optimized for streaming ASR lack the full-sequence context needed for high-quality translation, where reordering of words across languages (e.g., English "I love you" β†’ French "je t'aime") requires looking ahead in the source audio. Canary-1B-v2 uses a Transformer decoder specifically to support both ASR and AST in a single model, accepting the higher latency of autoregressive decoding in exchange for translation quality. The companion Parakeet-TDT-0.6B-v3 uses a TDT decoder (Xu et al., 2023) optimized for ASR-only, achieving even higher throughput (RTFx = 3332.74, per Table 5) but without translation capability β€” the two models represent different points on the accuracy-latency coverage tradeoff.

3. LLM-augmented speech models (SALM, BESTOW, Phi-4-Multimodal). A recent trend integrates pre-trained Large Language Models as decoders, adapting speech encoders to interface with them via lightweight projection layers or LoRA adapters. This leverages the LLM's linguistic knowledge and scales across modalities. The paper evaluates Voxtral-Mini-3B-2507 (3B parameters, LLM-based) and Phi-4-multimodal-instruct (5.6B parameters) as competitive baselines (Section 6). While these models achieve strong ASR and AST results β€” Phi-4-multimodal-instruct reaches 78.90 COMET on Xβ†’En CoVoST2 translation in the common-language evaluation β€” they remain substantially slower. Table 5 shows Phi-4-multimodal-instruct running at RTFx β‰ˆ 61, roughly 12Γ— slower than Canary-1B-v2. The paper's argument is not that LLM-based approaches are ineffective, but rather that specialized speech encoders with well-designed training curricula can match or approach their accuracy with dramatically lower computational cost, making them more suitable for throughput-sensitive deployments.

4. The canary model family v1 (Dhawan et al., 2023; Hu et al., 2025). The paper builds directly on earlier Canary iterations. The v1 model used concatenated tokenizers β€” separate SentencePiece vocabularies for different language groups β€” which the paper found to underperform a unified tokenizer in preliminary experiments (Section 4.1). The timestamp generation in v1 relied on predicting special frame-number tokens interleaved with text tokens (Section 5.1.2), which required careful data curation and introduced an additional training task that could interfere with transcription quality. Canary-1B-v2 replaces both approaches: a unified BPE tokenizer across all 25 languages, and an external NeMo Forced Aligner (NFA) pipeline for timestamp generation that decouples alignment from the primary ASR/AST model. These changes directly address identified weaknesses in the predecessor system.

Data Balancing as the Central Challenge

Section 3.3.1 frames the core technical challenge more precisely than most model-focused papers do: the training data is imbalanced along at least three orthogonal axes — task (ASR vs. X→En vs. En→X), language (English dominance, low-resource sparsity), and corpus domain (VoxPopuli parliamentary vs. diverse sources). A naive training run on the full 1.7M-hour dataset would over-represent high-resource tasks (English ASR, En→X translation) and narrow-domain sources (VoxPopuli for low-resource languages), producing a model that performs well on benchmarks resembling the training distribution but poorly under distribution shift.

The paper frames its multi-stage training strategy as a direct response to this challenge. The two-stage pre-training — first on X→En + English ASR + non-speech, then on the full task mixture — is designed to establish a strong English-centric foundation before introducing the rarer En→X translation task (which has 480K supplementary hours but was not in the original Granary corpus). The fine-tuning stage with dynamic weight scheduling (Section 4.3) then actively counteracts corpus dominance: sampling probabilities for each dataset shift over the 10K fine-tuning steps from reflecting the natural data volume toward a uniform distribution across languages and corpora within each task group. The cosine scheduling of these weights, inspired by Parmar et al. (2024)'s observation that data distribution shifts should be introduced when the learning rate can accommodate them, prevents "shocking" the model — Figure 7 illustrates this visually, with MOSEL (VoxPopuli-heavy) weight decreasing and FLEURS weight increasing as the learning rate decays.

This framing changes the narrative from "we trained a model on a big dataset" to "we designed an active intervention to make the model learn a balanced representation despite massively unbalanced training data." The 25% relative WER reduction and 6-point COMET gain from fine-tuning (Table 4) are not attributable to seeing more data β€” the fine-tuning set is a subset (15K hours) of the pre-training data β€” but to seeing the right data with the right sampling probabilities at the right point in training.

Non-Speech Data and Hallucination

A secondary motivation concerns hallucination — the tendency of ASR models to transcribe speech when no speech is present (e.g., producing text output for silence, music, or noise). The paper includes non-speech audio data (music, noise, silence) paired with appropriate null outputs during training (Section 3.2), with coverage across all task combinations (X→X, En→X, X→En). This is a practical production concern: in real-world applications, audio inputs may contain long stretches of non-speech content, and an ASR model that hallucinates transcription during these stretches degrades user trust and downstream processing. The paper does not quantify hallucination reduction as a separate metric, but frames it as an aspect of "robustness" alongside the noise-robustness evaluation in Table 6.

Positional Encoding for Long-Form Speech

The paper also addresses a more specific technical gap: positional encoding strategies for encoder-based speech models are under-explored compared to their use in language models. Rotary Positional Embeddings (RoPE) have become standard in LLMs due to their length generalization properties, but Section 6.4.2 reveals that they perform poorly for long-form ASR compared to Attention with Linear Biases (ALiBi) β€” the opposite of what the language modeling literature would predict. The paper's contribution here is adapting ALiBi for the bidirectional encoder setting (using a symmetric bias matrix, Figure 3, instead of the original causal formulation) and demonstrating its superiority for ASR while confirming RoPE's advantage for translation. This finding has implications beyond this specific model: it suggests that positional encoding choices should be task-dependent within speech processing, not inherited uncritically from the NLP literature.

3. Technical Approach

3.1 Reader Orientation

Canary-1B-v2 is a multilingual encoder-decoder model that takes raw audio as input and produces either a transcription in the source language (Automatic Speech Recognition, ASR) or a translation into a target language (Speech-to-Text Translation, AST) for 25 European languages. The system solves two intertwined problems simultaneously: how to achieve state-of-the-art accuracy across many languages and tasks while running substantially faster than larger general-purpose models, and how to train a single model effectively when the underlying training data is massively imbalanced along three axes β€” task frequency, language representation, and corpus domain. The solution's shape is a specialized speech encoder (FastConformer) paired with an autoregressive Transformer decoder, trained through a carefully orchestrated two-stage pre-training and fine-tuning pipeline where data sampling probabilities are actively manipulated β€” not passively observed β€” using dynamic weight scheduling to force balanced learning from an unbalanced dataset.

3.2 Big-Picture Architecture

The system has six major components, connected in a sequential training pipeline and a parallel inference pipeline:

  1. Training Data Mixer β€” a sampling system that selects training examples according to computed probability distributions over corpora, languages, and tasks, implementing the two-tier balancing policy described in Section 3.3.1. It produces batches with guaranteed diversity (at least 14 distinct language pairs per batch).
  2. FastConformer Encoder β€” an 8Γ— subsampled, convolution-augmented encoder that compresses the raw audio spectrogram (80-dimensional log-mel filterbank features at 10 ms frame rate) into a sequence of high-level acoustic representations at 80 ms temporal resolution. This is the primary encoder for Canary-1B-v2.
  3. nGPT Encoder (Alternative) β€” a hyperspherically normalized Transformer encoder explored as an alternative architecture, with experiments comparing RoPE versus ALiBi positional encodings. Not used in the final released model.
  4. Transformer Decoder — a standard autoregressive Transformer that generates output text tokens one at a time, conditioned on the encoder representations via cross-attention. It handles both ASR (X→X) and AST (X→En, En→X) through task-defining special tokens in the input prompt.
  5. NeMo Forced Aligner (NFA) β€” an external, standalone timestamp generation module that takes the ASR model's transcribed text and the original audio, feeding both into an auxiliary CTC-based Parakeet model to produce segment-level start/end times via Viterbi alignment. This runs post-hoc, not during ASR/AST decoding.
  6. Unified BPE Tokenizer — a single SentencePiece tokenizer with 16,384 tokens (including 1,162 task-defining special tokens) shared across all 25 languages, trained on a balanced sample of En→X translation text plus English YouTube and NeMo ASR Set data.

Information flows differently during training versus inference:

  • Training: Raw audio chunks (30–40 second segments, batched using Lhotse dynamic bucketing and duration bins) β†’ FastConformer encoder β†’ Transformer decoder autoregressively predicts target text tokens β†’ cross-entropy loss against reference transcription/translation. The data mixer controls which audio-text pairs enter the pipeline and in what proportions, with the mixing weights dynamically adjusted during fine-tuning.
  • Inference (ASR): Audio input β†’ FastConformer encoder β†’ Transformer decoder generates text β†’ NFA pipeline (Canary ASR output + original audio β†’ auxiliary Parakeet CTC model β†’ Viterbi alignment) β†’ text with segment-level timestamps.
  • Inference (AST): Same as ASR, but the decoder generates translated text. The NFA pipeline still runs Viterbi alignment between the source audio and the translated output text, producing segment-level timestamps despite the non-monotonic cross-lingual relationship.
  • Inference (long-form): Audio is split into overlapping 30–40 second chunks, processed in parallel as a batch through the encoder-decoder, then merged using Longest Common Subsequence (LCS) at the token level to resolve overlaps. For recordings longer than one hour, a hierarchical scheme processes hour-long blocks independently.

3.3 Roadmap for the Deep Dive

  • First, the training data balancing policy (Section 3.3.1), because every subsequent training stage depends on which data the model sees and in what proportions. Understanding the two-tier sampling formula is prerequisite to understanding why the two-stage pretraining and fine-tuning stages produce different behaviors.
  • Second, the unified tokenizer design (Section 4.1), since it defines the output space the decoder operates over and its compression properties directly affect both training efficiency and downstream performance. The tokenizer also interacts with the balancing policy because the training text for vocabulary construction must be balanced to avoid under-representing low-resource languages.
  • Third, the two-stage pretraining procedure (Section 4.2), covering what data each stage uses, the learning rate schedules, the checkpoint initialization, and the empirical comparison showing that two-stage training marginally outperforms single-stage training while enabling more efficient experimentation.
  • Fourth, the fine-tuning stage with dynamic weight scheduling (Section 4.3), which is the paper's primary methodological contribution. This covers the construction of the high-quality 15K-hour subset, the four-group balancing scheme, the cosine weight schedule, and the catastrophic forgetting hypothesis.
  • Fifth, the nGPT encoder architecture and positional encoding experiments (Sections 2.1.2, 2.1.3, 6.3, 6.4.2), treated as a comparative study that illuminates the tradeoffs between data-hungry hyperspherical Transformers and speech-optimized FastConformers.
  • Sixth, the timestamp generation pipeline (Section 5), which is architecturally decoupled from the main model and involves a separate CTC model plus post-hoc forced alignment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a specialized speech encoder (FastConformer) with a carefully orchestrated multi-stage training and data balancing strategy can match or exceed much larger general-purpose Transformer models on multilingual ASR and AST, provided the training procedure actively counteracts corpus-domain dominance during the fine-tuning phase.

Training Data Balancing: The Two-Tier Sampling Policy

All training stages share a common sampling infrastructure described in Section 3.3.1. The goal is to transform a severely imbalanced raw dataset — where English ASR constitutes 40% of all data, VoxPopuli parliamentary recordings dominate 95%+ of the training material for most low-resource languages, and the X→En task has roughly half the hours of ASR or En→X (Figure 4) — into batches that expose the model to balanced language and corpus representations.

The policy operates in three sequential steps, applied per training batch:

Step 1: Corpus balancing within each language. For a specific language $l$, define $n(c)$ as the number of training hours in corpus $c$ (e.g., VoxPopuli = 50,000 hours for French, Common Voice = 2,000 hours for French), and $N_l = \sum_{c \in l} n(c)$ as the total hours available for language $l$ across all corpora. An unnormalized weight is computed as:

wc=(n(c)Nl)Ξ±w_c = \left(\frac{n(c)}{N_l}\right)^\alpha

where $w_c$ is the unnormalized sampling weight for corpus $c$ within language $l$, $n(c)$ is the number of hours in that corpus, $N_l$ is the total hours for language $l$ across all corpora, and $\alpha \in (0,1]$ is a subsampling factor controlling the trade-off between high-resource and low-resource corpora.

What it computes: a weight that is sublinear in the corpus's share of total language data. A corpus with 90% of the language's data receives weight $0.9^\alpha$; a corpus with 10% receives $0.1^\alpha$. At $\alpha = 0.5$ (the paper's setting), the ratio between these weights is $\sqrt{0.9} / \sqrt{0.1} = \sqrt{9} = 3$, compared to a ratio of $0.9 / 0.1 = 9$ under uniform weighting proportional to data volume. The exponent therefore compresses the dynamic range: large corpora are downweighted relative to small ones within the same language.

Why this form: if $\alpha = 1$, the probability of sampling a corpus is proportional to its hours β€” the model would see VoxPopuli 25Γ— more often than Common Voice for a language where VoxPopuli has 50K hours and Common Voice has 2K hours. If $\alpha = 0$, all corpora are equally weighted regardless of size β€” the model would see Common Voice as often as VoxPopuli, potentially overfitting the tiny corpus. $\alpha = 0.5$ provides a square-root compromise: the model sees larger corpora more often than smaller ones, but the ratio is compressed to prevent complete domination.

The normalized probability of sampling corpus $c$ within language $l$ is:

pc=wcβˆ‘c∈lwc,βˆ‘c∈lpc=1p_c = \frac{w_c}{\sum_{c \in l} w_c}, \quad \sum_{c \in l} p_c = 1

Step 2: Language balancing across the entire training set. After computing within-language corpus weights, cross-language weights are computed analogously:

wl=(n(l)Ntotal)Ξ²w_l = \left(\frac{n(l)}{N_{\text{total}}}\right)^\beta

where $n(l) = \sum_{c \in l} n(c)$ is the total hours for language $l$, $N_{\text{total}} = \sum_l n(l)$ is the global total across all languages, and $\beta \in (0,1]$ is an upsampling factor β€” named differently from $\alpha$ because its role is conceptually reversed: it increases the representation of low-resource languages, whereas $\alpha$ decreases the representation of high-resource corpora within a language. The paper sets $\beta = 0.5$.

What it computes: the same square-root compression but applied across languages rather than corpora. English, with ~285K ASR hours (~40% of total), receives weight proportional to $(0.4)^{0.5} \approx 0.63$ rather than $0.4$. A low-resource language with 0.1% of total data receives weight proportional to $(0.001)^{0.5} \approx 0.032$ rather than $0.001$. The ratio between English and the low-resource language shrinks from $0.4 / 0.001 = 400$ to $0.63 / 0.032 \approx 20$, a 20Γ— reduction in imbalance.

The normalized language probability is:

pl=wlβˆ‘lβ€²wlβ€²,βˆ‘lpl=1p_l = \frac{w_l}{\sum_{l'} w_{l'}}, \quad \sum_l p_l = 1

Step 3: Final entry distribution. The probability of selecting a specific training example from corpus $c$ in language $l$ is the product:

pc,l=plβ‹…pcp_{c,l} = p_l \cdot p_c

What it computes: the joint probability factorizes as "first pick a language, then pick a corpus within that language," where both choices are tempered by the square-root compression factors. This means a VoxPopuli recording in a low-resource language gets boosted twice: once because the language weight $p_l$ is elevated above its data-proportional value (low-resource upsampling), and once because its within-language corpus weight $p_c$ is driven by a ratio that is less extreme than the raw data sizes (corpus balancing).

Why this ordering: the paper explicitly contrasts this with Babu et al. (2021)'s procedure for XLS-R, which first upsamples languages within each corpus and then balances corpora as if each corpus were a separate language. The reversal here β€” corpus balancing first, then language balancing β€” "ensures that low-resource corpora within a language are not overshadowed before cross-language balancing is applied" (Section 3.3.1). If language balancing were applied first, a low-resource corpus within a high-resource language might never be sampled because the language-level balancing has already compressed the total probability mass for that language. Doing it second applies the compression after the within-language ratios have already been flattened.

Implementation detail: the paper uses the Lhotse library (Ε»elasko et al., 2021) to form batches according to $p_{c,l}$. The analysis shows that with $\alpha = \beta = 0.5$, batches consistently contain "at least 14 distinct language pairs" out of the possible combinations β€” a direct empirical check that the sampling policy is achieving its diversity goal.

Interaction with task balancing: for Canary-1B-v2's pre-training, each task direction is treated as a separate "language" entry — so "lv" (Latvian ASR), "lv→en" (Latvian-to-English translation), and "en→lv" (English-to-Latvian translation) are three separate entries that the above policy balances independently. This means the language $l$ in the formulas above actually refers to a tasked language pair, not just a language code. The imbalance across tasks (Figure 4: X→En has roughly half the hours of ASR and En→X) is therefore handled by the same $\beta$ upsampling mechanism, with X→En pairs receiving higher weight per hour than higher-resource task directions.

Data ordering across epochs: the paper does not explicitly state whether the sampling policy produces a fixed curriculum or random sampling without replacement, but the use of Lhotse dynamic bucketing with duration bins and OOMptimizer (Ε»elasko et al., 2024) for batch size maximization implies an online, stochastic sampling process where each batch is drawn independently according to $p_{c,l}$. This is consistent with the reported GPU utilization of ~95% throughout Stage 1 pre-training β€” the system is continuously feeding diverse batches without waiting for epoch boundaries.

Unified BPE Tokenizer Design

The tokenizer is a single SentencePiece (Kudo & Richardson, 2018) BPE model that maps text strings to sequences of subword tokens, shared across all 25 languages and all three tasks. This is a departure from Canary-v1's concatenated tokenizer, which maintained separate vocabularies for different language groups and merged them post-hoc.

Training data for the tokenizer. The paper constructs a balanced text corpus for SentencePiece training, motivated by the observation that an imbalanced text corpus leads to cross-lingual variation in compression rates β€” high-resource languages get more subword merges, producing shorter token sequences and artificially better downstream performance. The training text is drawn from two sources:

  1. The Enβ†’X portion of the training data β€” which Figure 5 shows is "uniformly balanced across non-English languages" with 20K–29K hours per language. This provides coverage of all 24 non-English target languages.
  2. English text from the YouTube subsets (YTC, YODAS of Granary) and NeMo ASR Set 3.0 β€” this is a deliberate domain diversification choice. The paper states this is "intended to prevent tokenizer vocabulary from overfitting to narrow-domain datasets (e.g., VoxPopuli parliamentary language)." YouTube data contains informal, conversational, and code-switched text, providing lexical coverage that parliamentary transcripts lack.

Vocabulary size experiments. The paper experiments with three sizes: 4,096, 8,192, and 16,384 tokens. All three include 1,162 special tokens specific to Canary model prompts: task-defining tokens like <|timestamps|>, <|notimestamps|>, language ID tokens (e.g., <|en|>, <|fr|>), and other placeholder tokens. The finding is that "larger vocabulary sizes yielded better downstream performance on both ASR and AST tasks" (Section 4.1), and 8,192 and 16,384 showed "similar compression rates, indicating an optimal balance between downstream performance and compression efficiency."

Compression rate analysis (Table 2). The compression rate is defined as the ratio of raw characters to BPE tokens β€” higher is better, meaning fewer tokens are needed to encode the same text. On the FLEURS dataset across all 25 languages:

  • GPT-4o tokenizer: average compression rate 3.15, standard deviation 1.53
  • Canary tokenizer (8,192 tokens): average 4.87, standard deviation 0.86
  • Canary tokenizer (16,384 tokens): average 5.78, standard deviation 0.91

The Canary tokenizers achieve both higher average compression rates and substantially lower standard deviations. The lower standard deviation is the critical finding: GPT-4o's tokenizer compresses some languages efficiently and others poorly (high variance), whereas the Canary tokenizer behaves more uniformly across languages. The paper attributes this to "balancing the amount of text per language in the overall corpus supplied to SentencePiece tokenizer training, which effectively mitigates drastic cross-lingual variations in compression rates."

Why a unified tokenizer over concatenated? The paper cites three reasons:

  1. Downstream performance: "preliminary experiments comparing unified and concatenated tokenizers in identical setups demonstrated that the unified tokenizer significantly outperformed the concatenated approach" (Section 4.1). No quantitative results are provided for this claim, but it is stated as the primary motivation.
  2. Code-switching: the Granary dataset "contains multilingual content where words from different languages appear within single-language datasets" (e.g., English loanwords in French speech, or a German speaker inserting English phrases). A concatenated tokenizer would need to decide which sub-vocabulary to use for each code-switched token; a unified tokenizer handles this naturally because all languages share the same token space.
  3. Downstream techniques: a "consistent lexical space" provides a "strong foundation for downstream techniques such as phrase and word boosting." Phrase boosting is a technique where the ASR decoder's output distribution is biased toward specific phrases (e.g., product names, technical terms) by increasing the probability of their constituent tokens. If different languages use different tokenizers, boosting a phrase that appears in multiple languages requires coordinating across tokenizer boundaries.

Special tokens and task conditioning. The 1,162 special tokens include task-defining tokens that are prepended to the decoder input during training and inference. The paper's Figure 9 shows example prompts:

  • ASR with timestamps: <|en|><|transcribe|><|timestamps|><|notimestamps|> followed by the audio
  • AST with timestamps: <|en|><|translate|><|timestamps|> followed by the audio

The model learns to condition its output format (timestamps vs. no timestamps, transcription vs. translation) on these special tokens, enabling a single model to serve all task combinations. The tokens are also used by the NFA pipeline: when timestamps are requested, the model's output is fed to the auxiliary CTC model for alignment.

Tokenization at inference: the decoder generates tokens autoregressively until a special end-of-sequence token is produced. The output token sequence is detokenized using the SentencePiece reverse mapping to produce the final text string. For timestamp generation, this text string is then fed (along with the audio) to the NFA pipeline β€” the tokenizer's role ends at producing the text.

Two-Stage Pre-Training

The pre-training procedure described in Section 4.2 is designed to establish a strong English-centric foundation before introducing the full task mixture. The key design choice is the three-part data decomposition of Stage 1: X→En translation (360K hours), English ASR (285K hours), and non-speech data (1.2K hours), trained for 150K steps before expanding to the full 1.7M-hour dataset in Stage 2 for an additional 100K steps.

Why X→En and English ASR in Stage 1: English serves as a "pivot" language that appears in two of the three tasks — as the source language in ASR and as the target language in X→En translation. By training the encoder to extract features useful for English transcription and the decoder to generate English text, Stage 1 builds representations that can later be adapted to other language directions. The paper's stated rationale is more implicit: the full dataset has En→X as the largest task component (675K hours), but starting with En→X would mean the decoder learns to generate text in 24 different languages simultaneously from scratch, which may be harder than learning a single target language (English) first.

Non-speech data in Stage 1: the inclusion of 1,200 hours of non-speech audio (silence, music, noise) paired with null output is positioned as a hallucination-reduction measure. The paper does not provide a hallucination metric, but the mechanism is standard: by training the model to produce no text when the audio contains no speech, the decoder learns to suppress output in silent or noisy conditions rather than confabulating plausible text.

Initialization from multilingual ASR checkpoint. Training does not start from random weights. The FastConformer encoder and Transformer decoder are initialized from "a FastConformer Hybrid RNN-T/CTC checkpoint that had been trained on four languages (English, Spanish, German, and French) for ASR task" (Section 4.2). This means:

  • The encoder already has learned acoustic features for four major languages before any Canary training begins β€” it is not learning to extract phonetically meaningful features from scratch.
  • The decoder already has some text generation capability for those four languages, though it will need to learn 21 additional ones and a completely new task (AST).
  • The hybrid RNN-T/CTC head is presumably discarded or not used in the Canary architecture (which uses an autoregressive Transformer decoder), but the shared encoder parameters provide a warm start.

Stage 1 configuration (150K steps). The training is performed on 64 NVIDIA A100 GPUs. The configuration:

  • Learning rate: 4e-4 with a minimum of 1e-6
  • Warm-up: 5,000 steps
  • Schedule: inverse square-root (lr(t) = base_lr * sqrt(warmup_steps) / sqrt(step) for steps beyond warm-up)
  • Optimizer: AdamW (Loshchilov & Hutter, 2019) with weight decay of 0.001
  • Data loading: training data is divided into duration bins using NeMo's 2D duration bucket estimation (which groups audio segments by their length to minimize padding within batches), combined with Lhotse dynamic bucketing
  • Batch size optimization: OOMptimizer (Ε»elasko et al., 2024) determines the maximum feasible batch size for each duration bucket by probing GPU memory, achieving ~95% GPU utilization

Stage 2 configuration (100K additional steps). Starting from the Stage 1 checkpoint, training continues on the full data blend (ASR + X→En + En→X) for 100K steps. The configuration changes:

  • Learning rate: 3e-4 (reduced from 4e-4, presumably because the model is closer to convergence and needs smaller updates)
  • Warm-up: 5,000 steps (a fresh warm-up period, because the data distribution has shifted significantly)
  • Other settings unchanged from Stage 1

Performance saturation: the paper notes that "performance saturation" was observed when training was extended beyond 100K steps in Stage 2 β€” additional training did not improve validation metrics. This establishes a practical upper bound on the pre-training budget.

Two-stage vs. single-stage comparison (Table 3). The paper also trained a single-stage baseline: same initialization from the multilingual ASR checkpoint, trained on the full data blend from the outset for 250K steps (matching the total 150K + 100K = 250K of two-stage training), with the same optimizer and learning rate schedule as Stage 1. The results (Table 3) show:

  • ASR (WER): two-stage wins on FLEURS (11.35 vs. 11.91) and MLS (7.40 vs. 7.74), single-stage wins on CoVoST2 (10.15 vs. 10.93). The differences are small β€” less than 1% absolute in all cases.
  • Xβ†’En (COMET): two-stage wins on both FLEURS (73.23 vs. 72.54) and CoVoST2 (72.88 vs. 71.64). The gaps are modest but consistent.
  • Enβ†’X (COMET): single-stage wins on FLEURS (83.03 vs. 82.63), two-stage wins on CoVoST2 (74.88 vs. 73.87). Again, small gaps.

The selection criterion for the two-stage checkpoint is explicitly stated: "X→En scores fell short of SOTA results, and the two-stage model performed slightly better in this direction." Since X→En was the weakest direction overall and the one where data imbalance is most severe (only 368K hours vs. 658K+ for the other tasks), even a small improvement is valued.

Efficiency argument for two-stage training. Beyond the marginal accuracy advantage, the paper highlights an experimental efficiency benefit: "in the two-stage regime the first stage already establishes a strong foundation. As a result, the second stage needs only around 100K additional steps on the full dataset to adapt and integrate task mixtures. This makes it possible to freely vary weighting schemes or data compositions in the second stage for experimentation purposes without incurring the cost of retraining the model for 250,000 steps each time." This is motivated by the computational cost β€” each full training run at this scale is expensive, so the ability to reuse the Stage 1 checkpoint for multiple Stage 2 experiments is valuable.

Stage 1+2 is still pre-training, not fine-tuning. The Section 4.2 process produces what the paper calls the "pre-trained checkpoint," which is then further refined in a separate fine-tuning stage (Section 4.3). The distinction matters: the pre-training stages use the full 1.7M-hour dataset (including pseudo-labeled, noisy, and domain-biased data), while fine-tuning uses a curated 15K-hour subset of high-quality data. The paper's Table 4 compares "pre-trained" against "fine-tuned," not "Stage 1" against "Stage 2."

Fine-Tuning with Dynamic Weight Scheduling

The fine-tuning stage (Section 4.3) represents the paper's primary methodological contribution and is where the largest performance gains are realized. The core insight is that pre-training on the full dataset β€” even with the temperature-based sampling policy from Section 3.3.1 β€” leaves residual biases from corpus dominance and quality imbalances that can only be addressed by selectively training on a high-quality subset with actively managed sampling weights.

Construction of the high-quality subset. The fine-tuning data is constructed through a filtering and capping process:

  1. Source selection: the subset draws from NeMo ASR Set 3.0 (human-annotated, ~227K hours total) and the YouTube portion of Granary (more diverse acoustics than VoxPopuli) for ASR and X→En, plus the supplementary En→X dataset. VoxPopuli-dominated sources (MOSEL's parliamentary recordings) are deprioritized but not eliminated — when a language pair has fewer than 200 hours of high-quality data, the remainder is filled with other sources "including VoxPopuli when necessary."
  2. Quality filtering for translation: the X→En and En→X translation data is filtered by Quality Estimation (QE) scores computed with the Unbabel/wmt22-comet-da model (Gowda et al., 2023), keeping only samples with QE > 0.85. This removes poorly translated pseudo-labeled pairs where the Granary pipeline may have produced low-quality translations.
  3. 200-hour cap per language pair: for each language pair (e.g., fr→en, en→de), exactly 200 hours are selected from the high-quality subset. If fewer than 200 hours exist, the remainder is filled from lower-quality sources. This cap enforces balanced representation: every language pair contributes the same maximum amount, preventing any single high-resource pair from dominating.
  4. Group construction: the capped data is divided into four equally weighted groups:
    • ASR (non-English): 4,800 hours (200h Γ— 24 non-English languages)
    • Xβ†’En: 4,800 hours (200h Γ— 24 source languages)
    • Enβ†’X: 4,800 hours (200h Γ— 24 target languages)
    • English ASR: 600 hours β€” unusually, this group is smaller because English ASR is already well-represented from pre-training (285K hours) and the cap reflects a deliberate choice to allocate fine-tuning budget to underrepresented directions.

Training procedure. Fine-tuning runs for 10,000 additional steps on the pre-trained checkpoint, using 4 A100 GPUs (significantly fewer than the 64 used for pre-training β€” the smaller dataset doesn't require as much parallelization). The configuration:

  • Learning rate: 2e-5 (two orders of magnitude lower than pre-training, appropriate for the smaller, cleaner dataset)
  • Minimum learning rate: 1e-6
  • Schedule: inverse square-root, no warm-up
  • Sampling: within each step, one of the four groups is selected with equal probability (0.25 each). Within the selected group, all language pairs are equally likely to be sampled (since each has exactly 200 hours of data, this is naturally enforced).

Dynamic weight scheduling. The paper introduces a refinement beyond uniform sampling: instead of fixing the data distribution at the start of fine-tuning, the sampling weights for languages and corpora transition gradually from an initial imbalanced distribution toward the target uniform distribution over the 10,000 steps. The schedule is a cosine function:

For each training step $t \in [0, T]$ where $T = 10{,}000$: wsource(t)=wstart+(wtargetβˆ’wstart)β‹…1βˆ’cos⁑(Ο€t/T)2w_{\text{source}}(t) = w_{\text{start}} + (w_{\text{target}} - w_{\text{start}}) \cdot \frac{1 - \cos(\pi t / T)}{2}

where $w_{\text{source}}(t)$ is the sampling probability for a specific data source at step $t$, $w_{\text{start}}$ is its initial probability (reflecting the natural data volume or a pre-computed imbalanced distribution), and $w_{\text{target}}$ is its target uniform probability (1 / number_of_languages_in_group for language balancing, or the square-root-balanced weight for corpus balancing).

What it computes: a smooth interpolation between the starting distribution and the target distribution, with the rate of change following a cosine curve β€” slow at the beginning and end, fastest in the middle. When $t = 0$, $w(t) = w_{\text{start}}$ because $\cos(0) = 1$, making the cosine factor $(1-1)/2 = 0$. When $t = T$, $w(t) = w_{\text{target}}$ because $\cos(\pi) = -1$, making the factor $(1-(-1))/2 = 1$. At $t = T/2$, the factor is 0.5, and the weight is halfway between start and target.

Why cosine over linear or exponential: the paper reports experimenting with linear and exponential schedules and finding that "cosine yielded the most stable and robust results" (Section 4.3). The cosine has the property that the derivative is zero at the endpoints β€” the weights change very slowly at the start (avoiding a sudden shock to the model's loss landscape) and at the end (allowing the model to settle into the balanced distribution). A linear schedule would apply constant pressure throughout, which might cause instability early when the learning rate is high. An exponential schedule would concentrate the change at one end.

The learning rate interaction. Figure 7 shows the key insight: the cosine weight schedule is overlaid on the inverse square-root learning rate schedule. As fine-tuning progresses:

  • The learning rate decays from 2e-5 toward 1e-6 (inverse square root)
  • The MOSEL (VoxPopuli-heavy) dataset weight decreases (red line, cosine downward)
  • The FLEURS (high-quality) dataset weight increases (blue line, cosine upward)

The paper explicitly states this is inspired by Parmar et al. (2024)'s finding that "high-quality subsets are introduced at points in training when the learning rate is most suitable to absorb a shift in data distribution." The physical interpretation: early in fine-tuning, the learning rate is high enough to make substantial parameter updates, so the model is still shown a distribution close to what it saw in pre-training (MOSEL-dominant). As the model converges and the learning rate drops, the distribution shifts toward high-quality clean data, making small, precise adjustments. If the shift happened early, the large learning rate combined with a radically different data distribution could destabilize training β€” the model would effectively "forget" its pre-trained representations.

Intra-group balancing during weight scheduling. Within each of the four groups, the paper applies the two-tier balancing policy from Section 3.3.1 with modified parameters: $\alpha = 0.2$ (corpus subsampling factor, more aggressive than the 0.5 used in pre-training) and $\beta = 0.5$ (unchanged language upsampling factor). The smaller $\alpha$ means corpora within a language are balanced more aggressively during fine-tuning β€” consistent with the goal of reducing VoxPopuli dominance. Over the 10K steps, these weights transition from their $\alpha = 0.2, \beta = 0.5$ computed start values toward a target uniform distribution (1/24 for non-English languages, implicitly 1.0 for the English-only group).

Results of fine-tuning (Table 4). The paper compares three configurations:

  • Pre-trained (two-stage, no fine-tuning)
  • Fine-tuned (static balancing): the four-group configuration with fixed uniform sampling throughout the 10K steps
  • Fine-tuned (weight scheduling): the four-group configuration with cosine weight scheduling

Key results:

  • ASR FLEURS WER: Pre-trained 11.03% β†’ Static FT 8.63% β†’ Scheduled FT 8.40%. This is a 25% relative reduction from pre-trained to scheduled FT. The scheduled version edges out the static version by 0.23% absolute.
  • ASR CoVoST2 WER: Static FT 12.91% β†’ Scheduled FT 10.81%. This is where scheduling provides the largest gain β€” a 1.5% absolute WER reduction. The paper hypothesizes this is because CoVoST2 is "the most spontaneous of our benchmarks" and "weight scheduling prevents 'catastrophic forgetting' of pre-trained knowledge, whereas fine-tuning only on 15k hours of clean data biases the model too strongly toward clearer speech."
  • Xβ†’En FLEURS COMET: Pre-trained 73.23 β†’ Static FT 79.30 β†’ Scheduled FT 79.30. A 6-point absolute gain from pre-training, with no difference between static and scheduled.
  • Xβ†’En CoVoST2 COMET: Scheduled FT 78.66 vs. Static FT 78.24. A small 0.42 advantage for scheduling.
  • Enβ†’X: essentially flat across all configurations β€” the pre-trained distribution for this direction was already well-balanced (Figure 5), so fine-tuning's rebalancing has minimal effect.

The catastrophic forgetting hypothesis for CoVoST2. The paper's explanation for the large ASR CoVoST2 gain with scheduling deserves scrutiny. The argument is: CoVoST2 contains spontaneous, noisy speech that is more similar to the diverse pre-training data (which includes YouTube, Common Voice, etc.) than to the clean fine-tuning data. Static fine-tuning on 15K hours of clean data causes the model to "forget" how to handle noisy speech β€” it overfits to the clean acoustic conditions. Weight scheduling mitigates this by maintaining exposure to the broader pre-training-style distribution early in fine-tuning, when the learning rate is high enough to cause significant forgetting, and only shifting to clean data later when the learning rate is too low to undo the pre-trained noisy-speech representations. This is a specific, testable hypothesis that is consistent with the data but not experimentally isolated β€” there is no ablation that varies only the schedule while keeping the data fixed.

Selection of the release model. The paper selects the weight-scheduled fine-tuned model as the primary Canary-1B-v2 release, citing that it "consistently extracted more performance from the available data."

Parakeet-TDT-0.6B-v3 Training

The companion model follows a simpler training recipe, reflecting its ASR-only scope. The architecture is a FastConformer encoder with 24 layers (same as Canary-1B-v2) and a TDT decoder (Xu et al., 2023) instead of a Transformer. The TDT decoder jointly predicts output tokens and their durations, making it more efficient for streaming ASR but unsuitable for the reordering required in translation.

Training stages:

  1. Initialization: from a multilingual CTC checkpoint pre-trained on the Granary ASR subset β€” analogous to Canary-1B-v2's initialization from a four-language ASR model, but with broader language coverage.
  2. Pre-training: 150K steps on 128 A100 GPUs, using only the ASR subset of the Canary training data (~658K hours across 25 languages), with language and corpus balancing at $\alpha = 0.5, \beta = 0.5$.
  3. Fine-tuning: 5K steps on 4 A100 GPUs, using 7,500 hours of high-quality NeMo ASR Set 3.0 data. The paper does not specify whether dynamic weight scheduling is used for this stage β€” the brevity of the description suggests uniform sampling may have been sufficient for the smaller model.

nGPT Encoder Architecture (Alternative)

The nGPT encoder is presented as a comparative study, not as the primary architecture. It adapts the Normalized GPT formulation (Loshchilov et al., 2024) β€” originally developed for language modeling β€” to the speech encoder setting. The paper claims this is "the first work to explore the nGPT architecture for speech processing tasks."

Core mechanism: hyperspherical normalization. In a standard Transformer, the output of each layer is: ht+1=ht+Attn(LN(ht))+FFN(LN(ht+Attn(LN(ht))))\mathbf{h}_{t+1} = \mathbf{h}_t + \text{Attn}(\text{LN}(\mathbf{h}_t)) + \text{FFN}(\text{LN}(\mathbf{h}_t + \text{Attn}(\text{LN}(\mathbf{h}_t))))

where $\mathbf{h}_t$ is the hidden state at layer $t$, $\text{Attn}$ is the multi-head self-attention function, $\text{FFN}$ is the feed-forward network, and $\text{LN}$ is layer normalization. The residual connections allow the norm of $\mathbf{h}_t$ to grow unboundedly across layers, which can cause training instability.

In nGPT, all vectors are constrained to lie on a unit hypersphere β€” meaning after every operation, the vector is normalized to have L2 norm equal to 1. The nGPT layer update becomes: ht+1=Normalize(ht+Ξ±Aβ‹…Attn(Normalize(ht))+Ξ±Fβ‹…FFN(Normalize(ht+Ξ±Aβ‹…Attn(Normalize(ht)))))\mathbf{h}_{t+1} = \text{Normalize}\left(\mathbf{h}_t + \alpha_A \cdot \text{Attn}(\text{Normalize}(\mathbf{h}_t)) + \alpha_F \cdot \text{FFN}(\text{Normalize}(\mathbf{h}_t + \alpha_A \cdot \text{Attn}(\text{Normalize}(\mathbf{h}_t))))\right)

where $\alpha_A$ and $\alpha_F$ are learnable scalar parameters controlling the magnitude of the attention and feed-forward contributions respectively. Because everything is normalized, $\alpha_A$ and $\alpha_F$ determine the angular step size on the hypersphere β€” how far the representation moves in each layer.

What this accomplishes: the residual connections are reinterpreted as first-order optimization steps on the hypersphere, where each layer computes an update direction and $\alpha$ controls the step size. This has two claimed benefits: training stability (bounded representations prevent exploding activations) and faster convergence (the model can reach target validation metrics in fewer steps than standard Transformers).

Implementation details for the nGPT encoder:

  1. Front-end: a "Linear subsampling front-end" reduces the temporal resolution of the input spectrogram. This is analogous to the FastConformer's convolutional subsampling but implemented differently β€” the paper does not specify whether it uses strided convolutions or simple linear projection with frame stacking.
  2. Multi-head attention: uses Rotary Position Embeddings (RoPE) by default, with standard scaled dot-product attention. The query, key, and projection weight matrices are explicitly normalized.
  3. Feed-forward network: a "gated two-stage feed-forward module with SiLU activation." This is the SwiGLU variant common in modern architectures: the input is projected to two intermediate representations, one is passed through SiLU (Sigmoid Linear Unit, $x \cdot \sigma(x)$), and the two are multiplied element-wise before a final projection. Gating provides a learnable mechanism for the network to suppress or amplify different feature dimensions.
  4. Weight normalization: in addition to activation normalization, the weight matrices themselves are normalized after each optimizer update β€” a second constraint that further stabilizes training.

Decoder in nGPT experiments: the paper retains a "conventional Transformer architecture" for the decoder in nGPT-based experiments, ensuring that any observed differences from the FastConformer baseline can be attributed to the encoder architecture rather than decoder changes.

Positional Encoding Design: RoPE, ALiBi, and Task-Dependent Behavior

The nGPT encoder experiments reveal an unexpected interaction between positional encoding choice and downstream task. This is explored in Section 6.4.2, which compares Rotary Position Embeddings (RoPE) and Attention with Linear Biases (ALiBi) for long-form ASR.

RoPE baseline. RoPE (Su et al., 2021) encodes position by applying a rotation to the query and key vectors before computing attention scores. For a token at position $i$, the query vector $\mathbf{q}_i$ and key vector $\mathbf{k}_j$ are rotated by angles proportional to $i$ and $j$ respectively, such that the dot product $\mathbf{q}_i^T \mathbf{k}_j$ depends on the relative position $i - j$ through trigonometric identities: qiTkj=qTRT(i)R(j)k=qTR(jβˆ’i)k\mathbf{q}_i^T \mathbf{k}_j = \mathbf{q}^T \mathbf{R}^T(i) \mathbf{R}(j) \mathbf{k} = \mathbf{q}^T \mathbf{R}(j - i) \mathbf{k}

where $\mathbf{R}(\theta)$ is a block-diagonal rotation matrix with sinusoidal entries at different frequencies. The key property: the attention score decays gracefully with relative distance because the rotation angles for distant positions are decorrelated. In language models, this provides length generalization β€” models trained on sequences of length 2K can handle 8K at inference by extrapolating the rotation schedule.

ALiBi adaptation for bidirectional encoders. ALiBi (Press et al., 2022) takes a simpler approach: add a static, position-dependent bias to the attention scores before the softmax: Attention(Q,K,V)=softmax(QKTdk+B)V\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}} + \mathbf{B}\right)\mathbf{V}

where $\mathbf{B}_{i,j} = -m \cdot |i - j|$ in the original causal formulation β€” tokens attend less to distant positions, with the penalty growing linearly with distance and the slope $m$ being a head-specific learned or fixed parameter.

The paper's contribution is adapting ALiBi for the bidirectional encoder by using a symmetric bias matrix (Figure 3): Bi,j=βˆ’mβ‹…βˆ£iβˆ’j∣\mathbf{B}_{i,j} = -m \cdot |i - j|

This is identical to the causal version mathematically β€” $|i-j|$ is symmetric by definition β€” but the conceptual shift is from "penalize future positions" to "penalize distant positions equally in both directions." The paper states this is "the first exploration of ALiBi positional embeddings for speech recognition and translation tasks."

Results: ALiBi outperforms RoPE for ASR, RoPE for translation (Figure 13). The long-form ASR experiment (Section 6.4.2, Figure 13) evaluates both encodings on the Earnings dataset at evaluation lengths of 20, 40, 60, 80, and 100 seconds:

  • Baseline RoPE: WER increases steadily from ~14% at 20s to ~21% at 100s.
  • Baseline ALiBi: WER is lower than RoPE at all lengths, increasing from ~11% at 20s to ~16% at 100s. ALiBi consistently outperforms across the full range.
  • Modified RoPE (reduced angular rotation): by reducing the angular rotation per position β€” effectively "stretching" the positional encoding to accommodate longer sequences without exceeding the angular range seen during training β€” WER improves significantly, tracking closer to the ALiBi baseline: ~12% at 20s, ~18% at 100s.
  • Modified ALiBi (reduced bias slope): by reducing the slope $m$ of the linear bias β€” making the penalty for distant tokens less severe β€” WER improves further: ~10% at 20s, ~14% at 100s.

Both modifications work by making the positional encoding more permissive of long-range attention at test time. For RoPE, this means reducing the rotation frequency so that positions 0–4000 (at 80ms resolution, ~320 seconds) map to the same angular range that positions 0–2000 mapped to during training. For ALiBi, this means reducing the slope so that a token 100 positions away receives a bias of $-m \cdot 100$ where $m$ is smaller, allowing it to still compete for attention.

Why ALiBi wins for ASR: the paper hypothesizes that "reducing emphasis on distant tokens is not detrimental for speech recognition, where local context dominates." Speech recognition is fundamentally a local task β€” to identify the word at time $t$, the model primarily needs the surrounding ~1-2 seconds of audio (phonetic context, coarticulation). Distant audio (words uttered 30 seconds ago) provides minimal information for the current transcription decision. ALiBi's built-in bias against distant tokens is therefore well-matched to the ASR task structure. RoPE, by contrast, preserves full attention across all distances (attenuated only by rotation decorrelation), which provides no benefit for ASR and may introduce noise from irrelevant distant context.

Why RoPE wins for translation (Section 6.3): the paper reports that for AST, "long-range context is more critical, and we found RoPE to be more effective." Translation requires understanding the full source sentence to produce correct target-language word order β€” you cannot translate a German sentence with the verb at the end without attending to the entire utterance. ALiBi's distance penalty actively suppresses the long-range attention that translation needs, while RoPE's rotation-based encoding preserves it. The paper does not provide a direct AST comparison figure for the positional encoding variants, but states it as an empirical finding.

Interaction with encoder architecture choice: the paper positions the nGPT + ALiBi combination as a compelling option for ASR-only models, while the nGPT + RoPE combination or the FastConformer (which uses convolutional positional encoding implicitly through its convolutional modules) is preferred for multi-task systems. The released Canary-1B-v2 model uses FastConformer, not nGPT, and the positional encoding question becomes moot because FastConformer's convolutional layers inherently provide local positional information β€” the subsampling convolutions capture relative timing at the 80ms timescale, and the depthwise separable convolutions within Conformer blocks capture fine-grained local patterns.

nGPT vs. FastConformer Scaling Comparison (Table 7)

The paper trains nGPT-based models at 1B and 3B parameter scales on the Granary datasets (single-stage, 100K steps) and compares them against FastConformer models with multi-stage training. The results (Table 7):

  • 3B nGPT (single-stage, 100K steps) vs. 1B FastConformer (two-stage + fine-tuning): nGPT achieves competitive or better results across all tasks, especially Xβ†’En COMET (82.25 vs. 78.66 on CoVoST2) β€” the task where FastConformer struggled most before fine-tuning. This supports the paper's characterization of nGPT as "a data-hungry model that thrives under large-scale training and quickly reaches competitive performance."
  • 1B FastConformer (two-stage + fine-tuning) vs. 3B nGPT: FastConformer surpasses nGPT in all tasks after multi-stage training and fine-tuning: better ASR (8.40 vs. 10.70 FLEURS WER), better Xβ†’En (79.30 vs. 79.03 FLEURS COMET), and better Enβ†’X (83.79 vs. 82.50 FLEURS COMET).

The key tradeoff: nGPT achieves strong single-stage performance — reaching 79.03 X→En COMET with 1B parameters in 100K steps, where FastConformer needed two pre-training stages plus fine-tuning to reach 79.30. But FastConformer benefits more from fine-tuning — after the full multi-stage pipeline, the 1B FastConformer outperforms the 3B nGPT, suggesting that FastConformer's architectural inductive biases (convolutional modules for local patterns, subsampling for efficiency) leave more room for improvement from data curation, while nGPT's hyperspherical constraints already extract most of what the data can provide in a single stage.

Timestamp Generation via NeMo Forced Aligner (NFA)

Timestamp generation is architecturally decoupled from the main Canary-1B-v2 model, running as a post-processing step after ASR or AST transcription is complete. This section explains how it works, why decoupling was chosen over in-model timestamp prediction, and what the limitations are for AST.

The problem: aligning output tokens to audio frames. In an attention-based encoder-decoder model, the cross-attention weights between the decoder and encoder provide a "soft alignment" β€” for each output token (a subword), the attention distribution over encoder time steps indicates which audio frames were most relevant to generating that token. However, this alignment is:

  • Probabilistic: the attention distribution is a softmax, meaning weight is spread across multiple frames rather than concentrated at a single boundary.
  • Non-monotonic: attention can "jump backward" β€” the model might attend to frame 50 for token 1, frame 30 for token 2, and frame 60 for token 3, producing a zigzag alignment path that doesn't correspond to the sequential nature of speech.

The goal of timestamp generation is to produce a monotonic, hard alignment where each word (or segment) is assigned a single start time and end time.

Why not in-model prediction? The paper discusses Whisper's and earlier Canary's approach of interleaving timestamp tokens (e.g., <|0.52|>) into the output text and training the model to predict them. This approach "shifts the alignment burden to the model's training process" but introduces "additional complexities" (Section 5.1.2):

  • Data curation requires timestamp-annotated training data, which is expensive to produce at scale.
  • The model must learn an additional task (timestamp prediction) alongside transcription, which "can create balancing issues and requires a more carefully tuned training regimen to avoid negatively impacting transcription accuracy."

The NFA approach avoids these issues by using a separate, specialized model for alignment.

NFA pipeline (Figure 8a for ASR, 8b for AST). The pipeline has three components:

  1. Canary-1B-v2 generates the transcription (ASR) or translation (AST) text given the input audio. This is the only model the user interacts with directly.
  2. Auxiliary Parakeet CTC model β€” a 600M-parameter ASR model with a CTC decoder, trained on the ASR subset of Canary-1B-v2's training data for 250K steps on 128 A100 GPUs, using the same unified tokenizer as Canary-1B-v2. This model is NOT the same as Parakeet-TDT-0.6B-v3 β€” it uses a CTC head rather than TDT, and its role is alignment, not standalone transcription.
  3. Viterbi forced alignment: given the audio and a reference text sequence (the Canary-1B-v2 output), the CTC model computes log-probabilities for each token at each time step. The Viterbi algorithm finds the most probable monotonic alignment path β€” the sequence of (token, time_step) pairs that maximizes the product of emission probabilities, subject to the constraint that the token sequence, when CTC-decoded (with blank tokens removed and repeated tokens merged), produces the reference text.

What Viterbi alignment computes conceptually: the CTC model outputs a matrix $P \in \mathbb{R}^{T \times V}$ where $T$ is the number of audio frames (at the CTC model's native temporal resolution, which depends on its own subsampling factor) and $V$ is the vocabulary size (16,384). For a reference text of length $L$ tokens, the Viterbi algorithm finds a path $\pi = [\pi_1, \pi_2, ..., \pi_T]$ where each $\pi_t \in \{1, ..., V\} \cup \{\text{blank}\}$ such that:

  • The path is consistent with the reference: after collapsing consecutive repeated tokens and removing blanks, $\pi$ produces exactly the reference text.
  • The path maximizes $\prod_{t=1}^T P_{t, \pi_t}$ (or equivalently, $\sum_t \log P_{t, \pi_t}$).
  • From the path, the start time of token $l$ is the first frame $t$ where $\pi_t$ corresponds to token $l$ (not blank), and the end time is the last such frame.

How this produces word/segment timestamps: the Viterbi alignment provides a frame-level alignment for each token (subword). To get word-level timestamps, the token boundaries are aggregated: the start time of a word is the start time of its first token, and the end time is the end time of its last token. For segment-level timestamps (e.g., sentence boundaries), the same aggregation is applied at a coarser granularity. The paper's Figure 9 shows example output with segment-level timestamps β€” each sentence or phrase pair gets a start and end time.

The AST alignment challenge. The critical question for AST is: can you force-align translated text to source-language audio? The alignment is inherently non-monotonic and cross-lingual β€” the English word "love" might appear at position 2 in the source and position 3 in the French translation, with no consistent temporal relationship.

The paper's empirical finding (Section 5.2) is that segment-level timestamps work adequately for the 25 European languages in their scope, despite the non-monotonic relationship:

"We therefore recommend using segment-level timestamps with translation outputs, as word-level timestamps can be inaccurate due to the non-monotonic nature of speech translation."

The mechanism: at the segment level (roughly sentence boundaries), the audio for "the English sentence" and the translated text "the French sentence" correspond to roughly the same time interval, even though the internal word ordering differs. The Viterbi alignment finds a path that maps French tokens to English audio frames in a way that respects the CTC monotonicity constraint β€” the path is forced to be monotonic (never going backward), so it assigns the French tokens to audio frames in the order they appear in the French text, but the actual audio content at those frames is English. The alignment is "wrong" at the word level but approximately correct at the segment level because the segment boundaries β€” where the speaker pauses or changes topic β€” are language-independent.

Limitations acknowledged by the paper:

  1. Typological distance: "the applicability of the NFA pipeline to typologically distant languages remains uncertain and warrants more thorough testing." European languages have relatively similar word order (SVO for most), which means the relationship between source audio timing and target text position is less distorted than it would be for, say, Japanese (SOV) or Arabic (VSO) translation. For these languages, segment-level timestamps might also fail because the syntactic reordering could shift entire clauses across segment boundaries.
  2. CTC model is ASR-only: the auxiliary Parakeet CTC model is "trained solely for ASR tasks and lacks AST knowledge" β€” it has never seen translation pairs. The Viterbi alignment is therefore using an ASR model's acoustic-phonetic knowledge to align non-native (translated) text to audio. This works only because the CTC model's output log-probabilities provide a generic speech-vs-silence signal that helps identify segment boundaries even when the specific token probabilities are wrong. The paper does not quantify how much worse the alignment is for AST compared to ASR.
  3. Word-level inaccuracy: the paper explicitly states word-level timestamps are unreliable for AST. This restricts the downstream applications β€” subtitling at the word level (karaoke-style highlighting) would require a different approach.

Why this approach over DTW on cross-attention weights? The paper discusses the alternative of applying Dynamic Time Warping (DTW) to the cross-attention matrix between the decoder and encoder (Section 5.1.1), as done in WhisperX (Bain et al., 2023). The NFA approach is preferred because:

  • The CTC model provides calibrated log-probabilities for every token at every frame, which the Viterbi algorithm can use to find a globally optimal monotonic path. Cross-attention weights are not calibrated probabilities β€” they are softmax outputs that may be diffuse or multi-modal.
  • The CTC model is purpose-trained for ASR and has learned acoustic-phonetic alignments during its own training. Cross-attention weights in an AED model are a byproduct of sequence-to-sequence training and are not explicitly optimized for alignment quality.
  • The NFA pipeline is decoupled: improvements to the CTC model or the alignment algorithm don't require retraining the main Canary model.

Figure 9: Example output with segment-level timestamps. The paper shows two examples:

  1. ASR: "Jeg stΓ₯r op klokken syy, tager et brusebad, drikker kaffe og spiser morgenmad." with timestamps like [0.0s - 4.2s] and [4.2s - 12.8s] for different segments.
  2. AST: Source audio in one language, translated text in another, with segment-level timestamps aligning the translated segments to the source audio segments.

The prompts used for these examples include special tokens: \<\|en\|\>\<\|transcribe\|\>\<\|timestamps\|\> for the ASR case and analogous tokens for translation. These trigger the model to produce timestamp-aware output (which is then fed to NFA) versus plain text output.

Non-Speech Training Data Integration

The paper includes non-speech audio in the training data as a robustness measure (Section 3.2). This is worth detailing because the mechanism is not obvious from just reading "we added non-speech data."

Data construction: non-speech audio samples (silence, music, environmental noise) are randomly assigned source-target language pairs, creating "task combinations" that cover all possible directions: X→X (ASR), En→X, and X→En. The total non-speech volume is 36,581 hours (Table 9).

Training signal: when a non-speech sample is presented to the model during training, the target text is an empty sequence or a special token indicating "no transcription." The model learns to:

  1. Recognize when the audio contains no speech (the encoder must produce a representation that signals "no phonetic content").
  2. Suppress text generation (the decoder must learn to produce the empty output token or a special end-of-sequence token immediately rather than generating hallucinated text).

Why random language pair assignment: by covering X→X, En→X, and X→En with non-speech data, the model learns that "no speech" should result in no output regardless of what task is requested. Without this, the model might correctly suppress output for ASR on silence (X→X, which was seen during training) but hallucinate for translation on silence (X→En, which was never paired with silence during training and whose training data always contains actual speech). The random assignment ensures coverage across the full task space.

Quantity: 36,581 hours out of 1.74M total hours is approximately 2.1% of the training data. This is a small fraction β€” similar to the approach in Whisper β€” and is positioned as a targeted intervention rather than a major data component.

Long-Form Inference Strategy

The paper addresses the practical challenge of transcribing or translating audio that exceeds the training segment length (typically 30–40 seconds). Section 6.4.1 describes a dynamic parallel chunking mechanism.

Chunking procedure:

  1. The input audio is segmented into chunks of 30–40 seconds, where the exact chunk length is chosen dynamically "to minimize padding in the final chunk." This means if the audio is 95 seconds long, it might be split into three chunks of 35, 35, and 25 seconds rather than three chunks of 30 seconds with 5 seconds of silence appended to the last one (which would waste computation).
  2. Adjacent chunks overlap by 1 second. This overlap provides context at chunk boundaries β€” without it, the encoder at the end of chunk $k$ would have no information about the beginning of chunk $k+1$, and the decoder might produce inconsistent or truncated output at the boundary.
  3. Chunks are processed in parallel as separate entries within a single batch. This is the key efficiency gain over sequential chunking: rather than processing chunks one at a time (which would have the same latency as the original long audio), all chunks are fed to the model simultaneously, leveraging GPU parallelism.
  4. Outputs are merged into a single hypothesis using Longest Common Subsequence (LCS) at the token level. In the 1-second overlap region, both chunk $k$ and chunk $k+1$ produce text. The LCS algorithm finds the token sequence that appears in both outputs, resolving the overlap by keeping only one copy. This is more robust than simple truncation because the model may produce slightly different tokenizations at the boundary even for the same audio.

Hierarchical extension for very long recordings: for audio longer than one hour, the recording is first split into hour-long blocks, and each block is independently processed using the chunking and merging procedure described above. This is a two-level hierarchy: hour-level segmentation (concatenated sequentially) and chunk-level batching (processed in parallel). The paper does not specify whether hour-level blocks overlap.

Results (Table 8): the parallel chunking approach with 1-second overlap is compared against a sequential chunking baseline (fixed-length chunks processed one by one through the model, outputs concatenated directly):

  • Earnings22: WER improves from 15.61% to 13.93%, RTFx increases from ~300 to ~1141 (~3.8Γ— throughput improvement).
  • TAL (10-hour test set): WER improves from 16.62% to 10.12%, RTFx increases from ~342 to ~778 (~2.3Γ— throughput improvement).

The WER improvement is attributed to the overlap strategy providing context at boundaries, while the throughput improvement is due to batch parallelism β€” processing all chunks together amortizes GPU kernel launch overhead and enables efficient matrix multiplications across the batch dimension.

Why 1-second overlap specifically: the paper does not justify the exact overlap duration, but 1 second at 80ms encoder resolution corresponds to roughly 12.5 encoder frames. This provides a modest amount of bidirectional context for the decoder at chunk boundaries β€” enough to resolve a word or short phrase that spans the boundary, but not so much that the overlap computation dominates the total cost.

Summary of Key Design Choices and Their Justifications

  • Two-tier sampling ($\alpha = 0.5, \beta = 0.5$) over proportional sampling: square-root compression prevents corpus and language dominance while maintaining some preference for larger reliable sources. Proportional sampling would drown low-resource languages; uniform sampling would overfit tiny corpora.
  • Two-stage pre-training (English pivot first, then full mixture) over single-stage: establishes a strong English-centric encoder-decoder foundation before introducing the rarer Enβ†’X task, with the side benefit of enabling efficient experimentation by reusing Stage 1 checkpoints.
  • Dynamic weight scheduling with cosine over static fine-tuning: prevents catastrophic forgetting of pre-trained noisy-speech representations when fine-tuning on clean data, by introducing the clean-data distribution gradually as the learning rate decays. Cosine is used over linear or exponential because its zero derivative at endpoints avoids distribution shift shocks.
  • Unified BPE tokenizer over concatenated: improves downstream accuracy (per preliminary experiments), naturally handles code-switching, and provides a consistent lexical space for phrase boosting. The balanced tokenizer training corpus prevents cross-lingual compression rate variation.
  • NFA over in-model timestamp prediction: decouples alignment from transcription, avoiding multi-task training interference and the need for timestamp-annotated training data. The auxiliary CTC model provides calibrated log-probabilities for Viterbi alignment, producing more reliable boundaries than DTW on cross-attention weights.
  • ALiBi for ASR encoder position encoding over RoPE: local-context bias matches the ASR task structure where distant audio provides minimal information. RoPE remains preferred for translation where long-range context is critical β€” this is the paper's primary architectural insight for positional encoding.
  • Parallel chunking with 1-second overlap over sequential chunking for long-form inference: exploits GPU batch parallelism for throughput, with overlap providing boundary context for the decoder and LCS merging resolving duplicated edge tokens.
  • Non-speech data at 2.1% of training volume over no non-speech data: provides hallucination suppression coverage across all task combinations (Xβ†’X, Enβ†’X, Xβ†’En) through random language pair assignment, without dominating the training distribution.

4. Key Insights and Innovations

Innovation 1: Dynamic Weight Scheduling as a Solution to Catastrophic Forgetting During Data Rebalancing

The paper's most conceptually distinctive contribution is not the idea of fine-tuning on high-quality data β€” that is standard practice β€” but rather the introduction of dynamic weight scheduling as a technique for transitioning between data distributions without erasing knowledge acquired from the pre-training distribution. The key insight is that when you change the data distribution matters as much as what you change it to, and that the learning rate schedule provides a natural signal for timing this transition.

Prior work on fine-tuning for domain adaptation typically switches to the target distribution abruptly at the start of fine-tuning. The model sees clean, curated data from step 1, and if this data is acoustically different from the pre-training mixture (e.g., clean read speech vs. noisy YouTube audio), the model's representations shift toward the fine-tuning domain and its performance on out-of-domain audio degrades β€” a well-known instance of catastrophic forgetting. The dominant mitigation has been to include a fraction of the original pre-training data in the fine-tuning mix (experience replay) or to freeze early layers, but these are static interventions that don't exploit the temporal dynamics of training.

The paper's innovation is to recognize that the inverse relationship between learning rate and plasticity creates a natural window for safe distribution shift: early in fine-tuning, when the learning rate is high, the model is plastic and large changes to the data distribution would cause large, potentially destructive parameter updates. Late in fine-tuning, when the learning rate has decayed, the model is near convergence and can absorb a distribution shift as a small refinement. A cosine schedule on the sampling weights β€” which changes slowly at first (protecting against early shock) and accelerates mid-training (when the learning rate has decayed sufficiently) β€” aligns the distribution shift with the model's decreasing plasticity.

The evidence for this mechanism comes from Table 4's CoVoST2 ASR results: the statically fine-tuned model achieves 12.91% WER (worse than the pre-trained model's 10.93%), while the weight-scheduled fine-tuned model achieves 10.81% WER β€” a 2.1% absolute improvement. The paper's interpretation is that static fine-tuning on clean data caused the model to "forget" how to handle the spontaneous, noisy speech in CoVoST2, while weight scheduling maintained contact with the broader pre-training distribution during the plastic phase of fine-tuning. This is not merely an empirical trick β€” it introduces a temporal dimension to data balancing that the field's prior focus on static sampling probabilities (e.g., temperature-based upsampling as in Babu et al., 2021) had missed entirely. The contribution is a design principle, not just a recipe: align data distribution transitions with the model's changing capacity for adaptation, mediated by the learning rate schedule.

Innovation 2: ALiBi > RoPE for ASR Encoders β€” A Task-Dependent Positional Encoding Result That Reverses the NLP Consensus

The paper contributes a finding that challenges a growing default in speech processing: that Rotary Position Embeddings (RoPE), which have become standard in large language models due to their length generalization properties, should be adopted for speech encoders. The nGPT experiments in Section 6.4.2 demonstrate that Attention with Linear Biases (ALiBi) consistently and substantially outperforms RoPE for long-form ASR, while RoPE remains superior for speech translation. This is not a minor hyperparameter difference β€” it is a task-dependent reversal of the NLP community's positional encoding consensus.

The NLP literature (Press et al., 2022; Su et al., 2021) has generally found that RoPE provides better length generalization than ALiBi for language modeling, and the speech community has largely imported RoPE as the default for Transformer-based speech encoders without systematically questioning whether the properties that make RoPE effective for autoregressive text generation β€” preserving full attention across all distances, allowing models to attend to tokens thousands of positions away β€” are actually beneficial for speech. The paper's insight is that they often are not.

The explanation the paper offers is task-structural: ASR is fundamentally a local task where the acoustic evidence for a phoneme or word resides within a ~1-2 second window. Distant audio provides at best redundant information and at worst distracting noise that interferes with attention-based feature aggregation. ALiBi's linear distance penalty β€” which progressively down-weights attention to distant tokens β€” is therefore not a compromise but a useful inductive bias that aligns the model's attention pattern with the task's information structure. RoPE, by preserving full attention across all distances, introduces noise without providing benefit.

The translation result provides the crucial counterpoint: when the task requires long-range dependencies (reordering words across a full sentence), RoPE's distance-preserving property becomes valuable and ALiBi's penalty becomes harmful. This dual finding transforms positional encoding from a generic architecture hyperparameter into a task-dependent design choice β€” you should choose your positional encoding based on whether your task requires local or global context, not based on what works best for language modeling.

The paper's adaptation of ALiBi for bidirectional encoders (using a symmetric bias matrix, Figure 3) also represents a minor architectural contribution β€” prior ALiBi work focused exclusively on causal (autoregressive) settings β€” but the conceptual contribution is the empirical demonstration that the optimal positional encoding for speech is not inherited from text.

Innovation 3: Externally Decoupled Timestamp Generation as a Principle for Multi-Task Speech Models

The paper makes an architectural choice that may initially appear to be an implementation detail but represents a conceptual stance with broader implications: timestamp generation should be handled by a separate, specialized model rather than integrated into the primary ASR/AST model as an additional prediction task.

The standard approach in Whisper and earlier Canary versions embeds timestamp prediction into the model's output space by training it to generate special time-code tokens interleaved with text tokens. This is architecturally elegant β€” a single model handles everything β€” but introduces a fundamental tension: the model must simultaneously optimize for transcription accuracy and temporal precision, and these objectives can conflict. The paper's choice to use an external NeMo Forced Aligner with an auxiliary CTC model decouples these objectives entirely. The Canary-1B-v2 model focuses solely on producing correct text; a separate, purpose-trained alignment model handles the temporal mapping.

The significance of this decoupling extends beyond the specific NFA implementation. It establishes a separation-of-concerns principle for speech models: when a task can be decomposed into a primary objective (what was said) and a secondary objective (when it was said), solving them with separate specialized models may be more robust than forcing a single model to balance both. The evidence for this is implicit β€” the paper reports that the NFA pipeline produces "consistently reliable timestamping performance" (Section 5.2) without the training instabilities and data curation challenges that Hu et al. (2025) documented for in-model timestamp prediction in earlier Canary versions.

A secondary conceptual contribution is the paper's empirical finding that CTC-based forced alignment works adequately for speech translation timestamps at the segment level, despite the inherently non-monotonic relationship between source audio and translated text. This is not obviously true a priori β€” one might expect that forcing a monotonic alignment between English audio and French text would produce meaningless results. The paper's hypothesis (that European languages' similar sentence-level structure makes segment boundaries language-independent) is a testable claim about cross-lingual speech structure that the speech translation community had not previously validated empirically. The paper explicitly flags the limitation for typologically distant languages, establishing a boundary condition for when this approach can be expected to work.

Innovation 4: The nGPT-FastConformer Comparison as a Diagnostic for Architecture-Data Interaction

The paper's comparison of nGPT and FastConformer encoders (Section 6.3, Table 7) reveals a pattern that is more interesting than either architecture's raw performance: the choice of encoder architecture determines how much benefit the model extracts from multi-stage training and data curation. This reframes architecture selection from a pure accuracy comparison to a question about the shape of the scaling curve and the interaction between architectural inductive biases and training procedure.

The finding is that nGPT — with its hyperspherical constraints and minimal speech-specific inductive biases — achieves strong performance in a single training stage with large-scale data (the 3B nGPT model reaches 79.03 X→En COMET on FLEURS in a single stage, nearly matching the multi-stage FastConformer's 79.30). But it benefits relatively little from fine-tuning: its hyperspherical normalization forces representations into a compact region of space from the start, leaving less room for the targeted refinements that fine-tuning provides. FastConformer, by contrast, starts weaker (the pre-trained model achieves only 73.23 X→En COMET) but gains substantially more from fine-tuning (to 79.30) — its convolutional inductive biases provide a flexible foundation that can be sharpened by curated data.

This is not merely an empirical observation but a diagnostic concept: architectures with strong built-in constraints (normalization, inductive biases) may converge faster and require less training procedure engineering, while more flexible architectures may underperform in simple training regimes but reward investment in multi-stage training and data balancing. The paper's choice of FastConformer for the final model reflects an engineering judgment that the available training procedure investment (two pre-training stages + weight-scheduled fine-tuning) could extract more value from FastConformer's flexibility than from nGPT's stability. This framing β€” architecture selection as a function of the training procedure budget β€” is a useful lens for the speech modeling community that goes beyond the typical "architecture A beats architecture B on benchmark C" comparison.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary evaluation benchmark is the FLEURS dataset (Conneau et al., 2022), covering all 25 supported languages for all three tasks (ASR, Xβ†’En, Enβ†’X). Supplementary evaluation uses CoVoST2 (Wang et al., 2020) and Multilingual LibriSpeech (MLS; Pratap et al., 2020), which cover subsets of the 25 languages. English ASR is additionally evaluated on the Hugging Face Open ASR Leaderboard datasets (Srivastav et al., 2023), which include multiple English test sets. Per-language coverage by task and dataset is detailed in Table 1.

  • Base model(s). The primary model is Canary-1B-v2, a 1-billion-parameter FastConformer encoder with a Transformer decoder trained on 1.7M hours of speech data. The companion model is Parakeet-TDT-0.6B-v3, a 600M-parameter FastConformer encoder with a TDT decoder trained on the ASR subset (~660K hours) and fine-tuned on 7,500 hours of NeMo ASR Set 3.0 data. Both models are chosen to represent the paper's claim that specialized speech encoders with careful training can match or exceed much larger general-purpose systems.

  • Metrics. For ASR, the primary metric is normalized Word Error Rate (WER, %) , computed using the Hugging Face Open ASR Leaderboard normalizers: the English normalizer for English test sets (handles numbers, abbreviations, currencies, punctuation removal, lowercasing) and a multilingual normalizer for non-English sets (replaces symbols with spaces, maps/strips diacritics to base forms, removes punctuation, lowercases). For AST, the primary metric is COMET (neural reference-based metric using Unbabel/wmt22-comet-da; Gowda et al., 2023), with BLEU (Papineni et al., 2002) reported in appendices for completeness. The paper argues COMET is more reliable than BLEU for multilingual settings because it uses pretrained multilingual encoders and human-annotated data to capture semantic similarity beyond n-gram overlap. For inference efficiency, RTFx is reported β€” the ratio of audio duration to processing time (higher is faster), computed as RTFx = audio_duration_seconds / inference_time_seconds. An RTFx of 749 means 1 second of audio is processed in 1/749 seconds.

  • Baselines. The paper compares against four families: (1) Whisper-large-v3 (1.55B parameters; Radford et al., 2023) β€” a general-purpose Transformer encoder-decoder trained on 680K hours of weakly supervised data; (2) Seamless-M4T-v2-large (2.3B) and Seamless-M4T-medium (1.2B; Barrault & Others, 2023) β€” unified multilingual speech/text translation models; (3) Voxtral-Mini-3B-2507 (3B; Liu et al., 2025) β€” an LLM-based multimodal system; and (4) Phi-4-multimodal-instruct (5.6B; Abouelenin et al., 2025) β€” a multimodal model with a Conformer speech encoder and Phi-4-Mini LLM backbone with LoRA adapters. The first two are closest in scale to Canary-1B-v2; the latter two are included to "highlight the trade-offs between accuracy and inference efficiency."

  • Generation budget / compute accounting. Inference efficiency is measured by RTFx rather than FLOPs. The paper does not use a generation budget (no beam search or multiple samples). All models are evaluated with greedy decoding (single output per input). The RTFx comparison is based on throughput on the same hardware, but the paper does not specify the GPU model used for the RTFx measurements in Table 5. The RTFx for Canary-1B-v2 (749) and Parakeet-TDT-0.6B-v3 (3332.74) are reported from the Hugging Face Open ASR Leaderboard, which standardizes evaluation on a single A100 GPU.

  • Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing. All evaluations are single-pass on the specified test sets. The pre-training vs. fine-tuning comparisons in Table 4 and Table 3 use fixed checkpoints evaluated once on each benchmark. The paper does not report confidence intervals, standard deviations across runs, or statistical tests for any result. For multilingual evaluations, results are reported under two settings: "all supported languages" (24 total, excluding Latvian which is not supported by the Seamless baselines) and "common languages" (6 languages shared by all compared models: en, fr, de, it, pt, es), with averages computed across datasets within each setting.

Main Quantitative Results

English ASR (HuggingFace Leaderboard)

The headline finding from Table 5: Canary-1B-v2 achieves an average WER of 5.56% on the Hugging Face Open ASR Leaderboard, outperforming Whisper-large-v3 (6.65%) while running approximately 10Γ— faster (RTFx 749 vs. ~74 for Whisper-large-v3). Parakeet-TDT-0.6B-v3 achieves 6.32% WER with RTFx of 3332.74 β€” roughly 54Γ— faster than Phi-4-multimodal-instruct (6.14% WER, RTFx ~61).

Table 5 reports per-dataset WER and the official leaderboard average. Whisper-large-v3 was included directly from the leaderboard; the paper does not re-evaluate it. The RTFx comparisons are leaderboard-reported values, which standardize measurement on an A100 GPU. The 10Γ— speedup claim comes from 749 / ~74 β‰ˆ 10.1. The 54Γ— speedup for Parakeet comes from 3332.74 / ~61 β‰ˆ 54.6.

This is a strong result but limited in one dimension: the evaluation covers only English ASR. The paper's core claim about multilingual coverage is not tested on the leaderboard benchmark. Additionally, the leaderboard normalizer used for non-English evaluation differs from the English normalizer (as described in Section 6.1), so these English results cannot be directly compared to the multilingual WER numbers in Figure 10, which use the multilingual normalizer.

Multilingual ASR Performance

Figure 10 presents ASR performance averaged across three datasets (FLEURS, CoVoST2, MLS) on both the 24-language and 6-language subsets. The headline numbers:

  • 24-language average WER: Canary-1B-v2 achieves ~8.1%, outperforming Whisper-large-v3 (~9.9%) and Seamless-M4T-medium (12.0%), while trailing Seamless-M4T-v2-large (~7.2%). Parakeet-TDT-0.6B-v3 achieves ~9.7% (11.52/9.78/7.83 on FLEURS/CoVoST/MLS), edging past Whisper-large-v3.

  • 6-language average WER: Canary-1B-v2 achieves 5.2%, outperforming Whisper-large-v3 (5.8%) and Seamless-M4T-medium (8.2%), and essentially matching Seamless-M4T-v2-large (5.3%). Parakeet-TDT-0.6B-v3 achieves 5.3% (4.37/4.79/6.74), matching Seamless-M4T-v2-large.

The pattern is consistent: Canary-1B-v2 substantially outperforms models of similar or larger size (Whisper-large-v3 at 1.55B, Seamless-M4T-medium at 1.2B) and remains competitive with the much larger Seamless-M4T-v2-large (2.3B). The 0.6B Parakeet model matches or exceeds the 1.55B Whisper on both language sets, supporting the efficiency claim.

Per-language WERs are provided in Appendix B (Tables 11 for FLEURS, 12 for MLS, 13 for CoVoST2). The paper does not highlight any language-specific failure cases, but these tables would reveal whether the average gains are evenly distributed or concentrated in specific languages.

Noise robustness (Table 6). Evaluated on LibriSpeech Clean with MUSAN music and noise samples added at varying SNRs:

  • Canary-1B-v2: WER drops from 2.18% at SNR 100 (essentially clean) to 2.01% at SNR 25, then rises to 3.88% at SNR 5 and 19.38% at SNR -5. The slight improvement at SNR 25 (2.18% β†’ 2.01%) is attributed to the model having been trained on noisy data (YouTube, VoxPopuli with background interpreter noise), allowing it to leverage mild noise as a regularizer.

  • Parakeet-TDT-0.6B-v3: consistently more robust, with WER ranging from 1.92% at SNR 100 to 1.96% at SNR 25, 2.57% at SNR 5, and 12.21% at SNR -5. At severe noise (SNR -5), Parakeet shows ~37% relative improvement over Canary-1B-v2 (12.21% vs. 19.38%).

This is an unexpected result: the smaller ASR-only model is more noise-robust than the larger multi-task model. The paper does not explain why β€” possible factors include the TDT decoder's architectural differences or the fine-tuning data composition for Parakeet, but this is not investigated.

AST Performance: X→En

Figure 11 presents X→En translation performance (COMET) on FLEURS and CoVoST2:

  • 24-language FLEURS: Canary-1B-v2 achieves 79.28 COMET, outperforming Whisper-large-v3 (76.51) and Seamless-M4T-medium (77.30) by substantial margins, while trailing Seamless-M4T-v2-large (81.71) by 2.43 points.

  • 24-language CoVoST2: Canary-1B-v2 achieves 78.14 COMET, outperforming Whisper-large-v3 (74.61) and Seamless-M4T-medium (75.64), and trailing Seamless-M4T-v2-large (80.26) by 2.12 points.

  • 6-language FLEURS: 82.41 COMET vs. 82.84 for Seamless-M4T-v2-large, 84.53 for Voxtral-Mini-3B-2507, and 84.06 for Phi-4-multimodal-instruct. The gap to the LLM-based models is 1.6–2.1 COMET points.

  • 6-language CoVoST2: 79.07 COMET vs. 80.16 for Seamless-M4T-v2-large, 78.90 for Voxtral-Mini-3B-2507, and 80.11 for Phi-4-multimodal-instruct. Notably, Canary-1B-v2 outperforms Voxtral-Mini-3B-2507 on the more spontaneous CoVoST2 data (+0.17 COMET) despite Voxtral being 3Γ— larger and LLM-based.

The paper emphasizes the CoVoST2 result as evidence of robustness: "On the more spontaneous and noisy CoVoST2 dataset, Canary-1B-v2 performs particularly well (79.07 vs. 76.86 for whisper-large-v3), increasing the gap over the whisper model." This is consistent with the fine-tuning story β€” the model's exposure to diverse acoustic conditions during pre-training and weight-scheduled fine-tuning makes it more robust to spontaneous speech than Whisper, which was trained primarily on web-collected data.

Per-language COMET and BLEU scores are provided in Appendix C (Tables 14 for FLEURS, 15 for CoVoST2).

AST Performance: En→X

Figure 12 presents En→X translation performance:

  • 24-language FLEURS: Canary-1B-v2 achieves 84.47 COMET, essentially matching Seamless-M4T-v2-large (84.85) β€” a difference of 0.38 points despite the 2.3Γ— parameter disadvantage.

  • 6-language FLEURS: 83.79 COMET vs. 83.56 for Seamless-M4T-v2-large, 82.20 for Phi-4-multimodal-instruct, and 83.56 for Voxtral-Mini-3B-2507. Canary-1B-v2 slightly outperforms all baselines.

  • 24-language CoVoST2: 80.03 COMET vs. 82.66 for Seamless-M4T-v2-large. The gap is larger here (2.63 points), and the paper notes this as a weakness.

  • 6-language CoVoST2: 78.37 COMET vs. 81.30 for Seamless-M4T-v2-large, 80.38 for Voxtral-Mini-3B-2507, and 80.75 for Phi-4-multimodal-instruct. Canary-1B-v2 lags all baselines.

The paper's explanation for the CoVoST2 En→X weakness: "while the En→X training data covers diverse domains, the X-language ASR pathways were more strongly shaped by narrower-domain data, leading the decoder to generalize less effectively to spontaneous CoVoST speech." This is a somewhat hand-wavy account — En→X is the task that was already well-balanced in pre-training (Figure 5), so the fine-tuning's rebalancing shouldn't have affected it much. The paper does not investigate further.

Per-language COMET and BLEU scores are provided in Appendix D (Tables 16 for FLEURS, 17 for CoVoST2).

nGPT vs. FastConformer Comparison

Table 7 compares three configurations:

  • 3B nGPT (single-stage, 100K steps): ASR FLEURS WER 10.70, Xβ†’En CoVoST2 COMET 82.25, Enβ†’X FLEURS COMET 82.50.
  • 1B FastConformer (pre-trained only, two-stage): ASR FLEURS WER 11.35, Xβ†’En CoVoST2 COMET 72.88, Enβ†’X FLEURS COMET 82.63.
  • 1B FastConformer (two-stage + fine-tuned): ASR FLEURS WER 8.40, Xβ†’En CoVoST2 COMET 78.66, Enβ†’X FLEURS COMET 83.79.

The comparison reveals: (1) the 3B nGPT in a single stage achieves X→En performance (82.25) that the 1B FastConformer cannot match even after two pre-training stages (72.88) — nGPT is genuinely more data-efficient for this task; (2) after fine-tuning, the 1B FastConformer closes most of the X→En gap (78.66 vs. 82.25) and surpasses nGPT on ASR (8.40 vs. 10.70 WER) and En→X (83.79 vs. 82.50). The paper's characterization — "nGPT is a data-hungry model that thrives under large-scale training and quickly reaches competitive performance, whereas FastConformer benefits more from fine-tuning" — is consistent with these numbers, but the table uses different training regimes (single-stage vs. multi-stage, different step counts), making the comparison more about training methodology choices than pure architecture comparison. A cleaner comparison would be single-stage 1B nGPT vs. single-stage 1B FastConformer vs. both after identical fine-tuning, which the paper does not provide.

Long-Form Inference

Table 8 compares parallel chunking (the paper's method) against sequential chunking on two datasets:

  • Earnings22: WER 13.93% (parallel) vs. 15.61% (sequential), RTFx ~1141 vs. ~300.
  • TAL (10h): WER 10.12% (parallel) vs. 16.62% (sequential), RTFx ~778 vs. ~342.

The parallel method improves both WER (due to 1-second overlap providing boundary context) and throughput (due to batch parallelism). The TAL improvement is particularly large β€” a 6.5% absolute WER reduction β€” suggesting that sequential chunking's lack of overlap was causing severe boundary artifacts on that dataset.

Positional encoding for long-form (Figure 13). The comparison between ALiBi and RoPE on the Earnings dataset across evaluation lengths (20–100 seconds) shows baseline ALiBi consistently lower WER than baseline RoPE at all lengths (e.g., ~16% vs. ~21% at 100s). Modifying ALiBi by reducing the bias slope improves it further (~14% at 100s), and modifying RoPE by reducing the angular rotation reduces its WER but not below the baseline ALiBi (~18% at 100s). Importantly, the figure reports WER only for the nGPT encoder with these positional encodings β€” the FastConformer (which uses convolutional positional encoding implicitly) is not in this comparison. The conclusion that "ALiBi remained more effective than RoPE in long-context ASR" is therefore specific to the nGPT architecture.

Ablation Studies and Robustness Checks

Two-stage vs. single-stage pre-training (Table 3): Comparing a model trained for 150K steps on X→En + English ASR followed by 100K steps on the full mixture against a model trained on the full mixture from the outset for 250K steps. The differences are small: ASR FLEURS WER 11.35 vs. 11.91 (two-stage wins by 0.56), X→En FLEURS COMET 73.23 vs. 72.54 (two-stage wins by 0.69), En→X FLEURS COMET 82.63 vs. 83.03 (single-stage wins by 0.40). The two-stage model was selected because it performed "slightly better" on the X→En direction, which was identified as the weakest. The narrow margins suggest that the two-stage regime is not fundamentally necessary but provides a modest advantage specifically for the data-scarce X→En task.

Static fine-tuning vs. weight-scheduled fine-tuning (Table 4): This is the critical ablation. Fine-tuning with static uniform sampling (four-group configuration with equal probability per group) yields: ASR FLEURS WER 8.63%, ASR CoVoST2 WER 12.91%, X→En FLEURS COMET 79.30, X→En CoVoST2 COMET 78.24, En→X FLEURS COMET 83.74. Fine-tuning with cosine weight scheduling yields: ASR FLEURS WER 8.40%, ASR CoVoST2 WER 10.81%, X→En FLEURS COMET 79.30, X→En CoVoST2 COMET 78.66, En→X FLEURS COMET 83.79.

The large difference is on ASR CoVoST2: 10.81% (scheduled) vs. 12.91% (static), a 2.1% absolute improvement. This is the paper's primary evidence for the catastrophic forgetting hypothesis — static fine-tuning actually degrades performance relative to the pre-trained model (10.93% WER on CoVoST2 from Table 3 vs. 12.91% after static fine-tuning in Table 4), while scheduled fine-tuning marginally improves it (10.81%). Other differences between static and scheduled are small (0.23 WER on FLEURS, 0.42 COMET on CoVoST2 X→En, 0.05 COMET on En→X).

Fine-tuning with QE filtering: The translation data in the fine-tuning subset is filtered by QE score > 0.85. There is no ablation comparing with and without this filter, so its contribution to the 6-point X→En COMET gain (73.23 → 79.30) cannot be separated from the effect of the balanced sampling itself. The gain could be driven primarily by removing poorly translated pseudo-labeled data rather than by the balancing strategy — the paper does not disentangle these.

Tokenization vocabulary size: Experiments with 4,096, 8,192, and 16,384 tokens showed that "larger vocabulary sizes yielded better downstream performance on both ASR and AST tasks." No quantitative results are reported. The 16,384-token vocabulary was selected, with 8,192 showing "similar compression rates" β€” the compression rate analysis in Table 2 suggests 16,384 provides higher average compression (5.78 vs. 4.87 characters per token) with only a slight increase in cross-lingual variance (0.91 vs. 0.86 standard deviation). The absence of downstream WER/COMET numbers for the vocabulary ablation is a gap.

Comet metric vs. BLEU: The paper reports BLEU in Appendices C and D but uses COMET as the primary metric throughout Section 6.2. The justification is well-articulated β€” BLEU "often underestimates semantic adequacy when translations diverge lexically or syntactically from reference sentences but remain valid" β€” but no correlation analysis between COMET and BLEU rankings on this specific data is shown. The claim that COMET is "more faithful" is supported by citation to prior metric evaluation literature, not by within-paper analysis.

RTFx measurement: The paper reports RTFx from the Hugging Face Open ASR Leaderboard (Table 5), which standardizes hardware (single A100 GPU) and evaluation protocol. However, RTFx for the multilingual evaluations (Figures 10–12) is not reported β€” the efficiency claims are limited to the English ASR evaluation. For AST, the paper cannot make throughput comparisons because it doesn't provide RTFx numbers for translation inference on any baseline. This is a limitation: the "matching performance with higher efficiency" narrative is only demonstrated for English ASR, not for the multilingual AST tasks that constitute half the paper's claimed contribution.

Parallel chunking overlap duration: The paper uses 1-second overlap in the parallel chunking method. There is no ablation varying the overlap duration (e.g., 0.5s, 2s) to determine whether the 1-second choice is optimal or merely adequate.

Symmetric ALiBi formulation: The adaptation of ALiBi from causal to symmetric (Figure 3) is motivated by the bidirectional encoder, but there is no comparison between symmetric ALiBi and the original causal ALiBi in the encoder. The paper assumes causal ALiBi would be inappropriate β€” which is likely correct β€” but doesn't demonstrate it.

Critical Assessment

The experiments demonstrate convincingly that Canary-1B-v2 can match or exceed Whisper-large-v3 on English and multilingual ASR while being faster, and that it achieves competitive multilingual AST performance against larger models. However, the paper's central claims about why these results are achieved β€” specifically the contribution of dynamic weight scheduling to the gains β€” are supported with thinner evidence than the headline performance numbers.

Claim: Dynamic weight scheduling prevents catastrophic forgetting and is responsible for the 25% WER reduction on FLEURS ASR and 6-point COMET gain on X→En. The evidence from Table 4 shows that the vast majority of the gain from fine-tuning comes from the transition to the high-quality subset itself, not from the scheduling. Static fine-tuning achieves 8.63% WER on FLEURS (from 11.03% pre-trained) versus 8.40% with scheduling — a gain of 2.40% from static, plus an additional 0.23% from scheduling. On X→En FLEURS COMET, the gain is entirely from the static approach (73.23 → 79.30), with scheduling adding nothing (79.30 → 79.30). The scheduling contribution is material only for ASR CoVoST2 (2.1% absolute WER improvement over static, which had actually degraded from pre-training). This is a real finding, but it's more specific than the paper's framing suggests: weight scheduling helps primarily for out-of-domain acoustic conditions (spontaneous/noisy speech) and provides minimal benefit for in-domain clean speech. The paper's abstract and conclusions present scheduling as a core contribution driving the overall gains, when the data shows it's a targeted fix for a specific failure mode.

Claim: Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10Γ— faster. Strongly supported by Table 5 (5.56% vs. 6.65% WER, RTFx 749 vs. ~74). This is the most robust finding in the paper. The caveat is that Whisper-large-v3's RTFx is leaderboard-reported and depends on the specific implementation used β€” the paper did not independently benchmark Whisper's throughput.

Claim: Canary-1B-v2 delivers "competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large." Supported with qualifications. For multilingual ASR, Canary-1B-v2 is actually better than the larger model on the 6-language subset (5.2% vs. 5.3%) and close on the 24-language average (8.1% vs. 7.2%). For Xβ†’En AST, Canary-1B-v2 is consistently behind Seamless-M4T-v2-large (79.28 vs. 81.71 on 24-language FLEURS COMET, 78.14 vs. 80.26 on CoVoST2) β€” a gap of 2–2.5 COMET points that is non-trivial. For Enβ†’X AST, the models are essentially tied on FLEURS (84.47 vs. 84.85) but Canary-1B-v2 lags on CoVoST2 (80.03 vs. 82.66). The claim of "competitive" performance is fair for ASR and Enβ†’X FLEURS, but oversells the Xβ†’En and Enβ†’X CoVoST2 results. The paper's own Table 4 shows that fine-tuning improved Xβ†’En dramatically (73.23 β†’ 79.30 COMET) but still left a gap to Seamless-M4T-v2-large β€” the architecture/training procedure does not close this gap.

Weakness: single evaluation run, no error bars. All results in Tables 3–8 and Figures 10–13 are reported as point estimates from single evaluation runs. Without confidence intervals or multiple seeds, differences of 0.2–0.5 WER or COMET β€” which the paper sometimes treats as meaningful (e.g., two-stage vs. single-stage selection) β€” could be noise. This is especially concerning for the CoVoST2 and MLS evaluations, where some language subsets are small. The paper does not report per-language sample counts for any evaluation benchmark.

Weakness: the nGPT comparison is confounded by training regime. The 3B nGPT is trained single-stage for 100K steps. The 1B FastConformer is trained two-stage for 250K steps plus 10K fine-tuning steps. The claim that "FastConformer benefits more from fine-tuning" cannot be separated from the fact that the FastConformer received more total training and a different procedure. A fair comparison would require identical training recipes at matched parameter counts β€” a 1B nGPT vs. 1B FastConformer both in single-stage, both after the same fine-tuning protocol. The paper does not provide this, making the nGPT findings suggestive rather than conclusive.

Weakness: the RTFx advantage is only demonstrated for English ASR, not AST. The paper builds a narrative around Canary-1B-v2 being a faster alternative to larger models for both ASR and AST, but the throughput numbers in Table 5 cover only the Hugging Face English ASR benchmark. AST inference speed is never reported for Canary-1B-v2 or any baseline. Since AST uses the same Transformer decoder and encoder, the speedup likely carries over, but the absence of numbers is a gap β€” especially because the non-autoregressive Seamless-M4T models might have different throughput characteristics for translation specifically.

Missing experiment: the interaction between QE filtering and data balancing in fine-tuning. The fine-tuning data is both QE-filtered (QE > 0.85) and balanced (200 hours per language pair). The 6-point X→En COMET gain could be primarily from removing low-quality translations rather than from balancing. A simple ablation — fine-tuning on the same total hours without QE filtering but with balancing — would isolate the filtering effect. Without it, the paper cannot attribute the gain to the balancing strategy.

Missing baseline: fine-tuning Whisper-large-v3 on the same high-quality subset. The paper compares Canary-1B-v2 after extensive fine-tuning on curated data against Whisper-large-v3 and Seamless-M4T-v2-large without comparable fine-tuning. If Whisper-large-v3 were fine-tuned on the same 15K-hour high-quality subset, it might close or reverse the accuracy gap. The paper's claim that the FastConformer architecture + training procedure is superior to the general-purpose Transformer approach would be stronger if the comparison included a similarly fine-tuned Whisper baseline.

Missing analysis: per-language breakdown of gains. The paper reports average WER/COMET across 24 or 6 languages but does not highlight which languages benefit most from fine-tuning. The per-language tables in the appendices contain this information, but the paper does not analyze them. Given the data imbalance analysis in Section 3.3 showing VoxPopuli dominance for low-resource languages, the hypothesis would be that low-resource languages benefit most from the rebalancing β€” the paper could have tested this explicitly and didn't.

Hard problems remain unsolved for En→X CoVoST2. The paper's own results show Canary-1B-v2 lagging Seamless-M4T-v2-large by 2.63 COMET on En→X CoVoST2 (80.03 vs. 82.66) and by 2.12 on X→En CoVoST2 (78.14 vs. 80.26). The fine-tuning gains are concentrated on FLEURS, which is cleaner and more controlled; the spontaneous speech gap persists. This suggests the paper's approach — like the example paper's finding that test-time compute doesn't help on genuinely hard problems — has a boundary: academic-clean speech benefits substantially, but challenging spontaneous speech remains dominated by larger pretrained models. The paper acknowledges this for En→X CoVoST2 but doesn't discuss it for X→En where the gap is also present.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for β€” and in a Deployment Setting It Would Dominate

The entire compute-optimal framework rests on the ability to place each prompt into one of five difficulty bins before deciding how to allocate the inference budget. The paper's method for doing so β€” generating 2,048 complete solutions per question and averaging either ground-truth correctness (oracle) or the process reward model's (PRM's) final-answer scores (predicted) β€” is extraordinarily expensive. The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference ... our experiments do not account for this cost largely for simplicity"

The consequence is that the paper's headline claim of "more than 4Γ— better efficiency over a standard best-of-N baseline" is computed after difficulty is already known, without amortizing the cost of learning it. Generating 2,048 samples per prompt consumes significantly more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment where difficulty estimation must be performed for every incoming prompt, the total cost would be difficulty estimation + strategy execution, and the former would dominate the latter. The 4Γ— figure is therefore best understood as an upper bound on achievable efficiency β€” a ceiling that cannot be approached without a radically cheaper difficulty estimation method.

What evidence exists: the paper provides none that quantifies the total cost including difficulty estimation. The predicted-difficulty-bins variant (Figures 4, 8) only removes the need for ground-truth answer checking β€” it still requires the full 2,048-sample generation pass with PRM scoring. The cost of this pass is never added to the reported generation budgets.

Mitigation status: the authors acknowledge the issue (Section 3.2) and flag it as an avenue for future work on "pretraining or finetuning models to directly predict difficulty of a question." No such model is developed or evaluated. A promising direction they do not explore is adaptive difficulty estimation: start with a small number of samples, assess the PRM score distribution, and allocate the remaining budget based on that signal, effectively amortizing difficulty estimation into the problem-solving process. Without such a scheme, the compute-optimal strategy as presented is not deployable at scale.

The 14Γ— Larger Model Baseline Is Systematically Weakened β€” It Is Neither Compute-Optimally Trained nor Given Test-Time Compute

The FLOPs-matched comparison in Section 7 pits PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14Γ— more parameters. This larger model is trained by scaling parameters only while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence: a Chinchilla-optimal model trained with the same total pretraining FLOPs β€” scaling both data and parameters equally β€” would likely outperform a parameter-only-scaled model. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy-to-medium questions with revisions at R β‰ͺ 1) may shrink or reverse against a properly compute-optimal larger pretrained model.

Compounding this, the larger model uses greedy decoding only β€” no majority voting, no best-of-N, no search of any kind. The paper's own results show that even modest test-time compute (best-of-4, best-of-16) produces substantial gains over greedy decoding (Figure 3, left). By denying the larger model any inference-time compute budget, the FLOPs-matched comparison conflates two distinct claims: (1) test-time compute can substitute for pretraining, and (2) all inference compute should go to the smaller model rather than being partially allocated to the larger one. A fairer comparison would give the larger model a test-time compute budget proportional to its share of the inference FLOPs β€” but Section 7 never runs this experiment.

What evidence exists: Figures 1 and 9 report the comparison as described, with the larger model evaluated at a single point (greedy decoding, no test-time compute). The paper is transparent about the parameter-only scaling choice but does not discuss the greedy-only decoding limitation.

Mitigation status: the authors flag the compute-optimal pretraining issue and defer it to future work (Section 7). The greedy-decoding limitation is not acknowledged at all. A proper FLOPs-matched comparison would allocate inference compute to both models in proportion to their per-token costs, and would train the larger model under a Chinchilla-optimal scaling regimen. Without this, the paper's central claim about the pretraining-inference tradeoff is directional rather than quantitative.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate β€” and the Mitigations Are Patches, Not Solutions

Section 6.1 describes a significant practical problem: during sequential revision, the model encounters correct answers in its context that it had produced earlier in the chain, and β€” because it was trained only on sequences where all in-context answers are incorrect followed by a correct target β€” it revises them into wrong answers. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naΓ―ve approach"

This is a direct consequence of the training data construction procedure (Section 6.1): the correct answer always appears at the end of the training trajectory, preceded by 0–4 incorrect answers. At test time, when the model's own output may contain correct intermediate answers, it encounters an out-of-distribution scenario β€” a correct answer in the context β€” and has never been trained on what to do with it. The model's learned behavior is "when you see an answer, revise it," regardless of whether it is correct.

The consequence: revision chains are inherently unstable. The system relies on post-hoc selection (majority voting or verifier-based selection across all revisions in the chain) to rescue correct answers that the model subsequently corrupts. This turns the revision model into a proposal generator whose individual steps are unreliable, rather than a process that monotonically improves answers. In applications where the number of revisions is limited (latency-constrained settings), the model might produce a correct answer at step 3, revert it at step 4, and never recover β€” and the post-hoc selection may fail if no mechanism remembers the step-3 answer.

What evidence exists: the paper quantifies the reversion rate (38%, Section 6.1) and shows that majority voting and verifier-based selection across the chain partially mitigate the problem (Figure 6, right). The ReST^EM experiment (Appendix K, Figure 16) provides additional circumstantial evidence: attempting to optimize the revision model with RL-style training amplified the reversion problem and caused performance to degrade substantially with sequential revisions β€” the fully sequential ReST^EM model achieves ~33.5% accuracy at 256 generations versus ~38.5% at an optimal sequential-to-parallel ratio. This suggests that the reversion issue is not merely a byproduct of offline data construction but a fundamental property of training on incorrect-to-correct trajectories that cannot be fixed by simply training longer.

Mitigation status: the paper applies within-chain selection (majority or verifier) to recover correct answers that get reverted, but this is a downstream patch, not a fix to the training procedure. The paper does not explore training the revision model to recognize when a solution is already correct and output a "no revision needed" token, nor does it train on trajectories that include correct intermediate answers. The reversion problem remains an open issue that limits the reliability of sequential revision at high step counts.

Hard Problems Remain Essentially Unsolved β€” Test-Time Compute Does Not Create Capability, Only Amplifies It

Across all methods β€” PRM search, iterative revisions, and their compute-optimal combinations β€” the hardest problem quintile (difficulty bin 5) shows near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budget levels (4 through 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, never approaching the larger model's performance.

This is a fundamental capability bound: test-time compute operates on the proposal distribution induced by the base model's parameters. If the base model's pass@1 on a problem class is near zero β€” meaning that among 2,048 independent samples, essentially none are correct β€” then no amount of search or refinement can produce a correct answer. The search algorithm is optimizing a verifier signal over a space that contains no correct solutions. The revision model is iterating on answers that were incorrect from the start.

The practical significance: for any deployment where the input distribution includes genuinely hard problems (problems outside the base model's effective capability range), test-time compute strategies provide zero value. The only path to improvement is pretraining a more capable base model or, as Section 7's FLOPs-matched analysis suggests, scaling pretraining compute. The paper is candid about this limitation (Section 7 takeaway box, Section 8), but the framing β€” "test-time compute can be more effective than pretraining" β€” should be understood as applying only to problems within the base model's capability frontier, not as a universal prescription.

What evidence exists: Figures 3 (right), 7 (right), 8, and 9 all show bin 5 performance effectively flat at near-zero accuracy. The paper explicitly states this finding but does not characterize what fraction of real-world prompts would fall into bin 5 for a typical base model β€” this fraction determines the practical ceiling on test-time compute's utility.

Mitigation status: none. The authors acknowledge that pretraining remains necessary for hard problems and do not claim otherwise. This is a fundamental limitation baked into the problem formulation rather than a fixable flaw. However, the paper does not provide any diagnostic for distinguishing hard problems from medium ones without the expensive 2,048-sample pass β€” a practitioner would need to know when to give up and escalate to a larger model, and the current difficulty estimation method is too costly for this purpose.

The Study Is Limited to a Single Domain (MATH) and a Single Model Family (PaLM 2-S*) β€” the Difficulty-Dependent Scaling Patterns May Not Transfer

All experiments in the paper use only the MATH benchmark (Hendrycks et al., 2021) consisting of 500 competition-level math problems, with PaLM 2-S* (Codey) as the base model. The authors argue this model is "representative of the capabilities of many contemporary LLMs" (Section 4), but this is an unverified assertion. Several aspects of the findings could be model-specific or domain-specific:

  • The PRM's over-optimization behavior (Figure 3, right β€” beam search hurting performance on easy problems at high budgets) depends on the verifier's calibration properties, which are a function of the base model's output distribution. A model with different error modes (e.g., one that makes systematic rather than random errors on easy problems) would produce a verifier with different failure characteristics, shifting the difficulty-dependent scaling curves.
  • The revision model's ability to improve (Figure 6, left β€” pass@1 improving from ~18% to ~24% over 20 revision steps) depends on the base model's capacity for in-context learning from its own incorrect outputs. This capacity varies substantially across model families and scales β€” a smaller or differently-architected model might not exhibit the same monotonically-improving revision trajectory.
  • The MATH benchmark consists exclusively of symbolic reasoning problems with unambiguous ground-truth answers. The difficulty estimation, PRM training (via Monte Carlo rollouts), and revision model training (via edit-distance-based pairing) all rely on the existence of verifiable correct answers. Extending the framework to domains with ambiguous, subjective, or multi-dimensional correctness (essay evaluation, dialogue quality, creative generation) would require fundamentally different verifier and difficulty estimation approaches.

What evidence exists: none that tests transfer. The paper contains no experiments on other reasoning benchmarks (e.g., GSM8K, MMLU reasoning subsets, coding benchmarks), no experiments with other base model families, and no discussion of how findings might differ for knowledge-recall tasks versus pure reasoning tasks.

Mitigation status: the authors do not claim cross-domain or cross-model generalization β€” they present specific findings on MATH with PaLM 2-S* and leave extension to other settings implicit. The limitation is in scope, not in overclaiming. However, practitioners considering adopting these strategies for, say, code generation or scientific QA would need to replicate the difficulty-dependent scaling analysis for their specific domain and model.

Sequential Revision Introduces Unresolved Latency vs. Accuracy Tension That the Paper Does Not Quantify

The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial β€” revision k+1 cannot be generated until revision k is complete because it conditions on revision k's full output text. Parallel best-of-N can, with sufficient hardware, be executed simultaneously. A strategy that allocates 128 generations as 64 sequential Γ— 2 parallel takes roughly 64Γ— longer in wall-clock time than one that allocates all 128 generations to parallel sampling, even though both consume the same total FLOPs.

The paper's compute-optimal policies (Figures 4, 8) systematically favor sequential-heavy strategies on easy problems (purely sequential revisions for difficulty bins 1–2; beam search at low beam widths for medium problems). In a latency-sensitive application β€” interactive assistants, real-time tutoring, on-device inference β€” these policies might be practically unusable regardless of their FLOPs-efficiency. A user waiting for a response cannot wait for 64 sequential revisions, even if those 64 revisions consume the same total GPU-seconds as 16 parallel samples.

The paper does not discuss latency at all. The cost model in Section 3.1 uses "number of generations N" as the budget unit without distinguishing between serial and parallelizable generations. The revision model's sequential structure (Section 6, Figure 5) is described as an advantage because it improves pass@1, but the serial dependency that makes it slower than parallel sampling is never acknowledged as a deployment constraint.

What evidence exists: none β€” latency is simply not measured or discussed. The paper does not report wall-clock times for any method, making it impossible for a practitioner to assess the latency-accuracy tradeoff.

Mitigation status: the authors do not address this limitation and do not suggest latency-aware variants of the compute-optimal policy. A latency-aware extension would need to introduce a second budget constraint (maximum wall-clock time) and optimize the sequential-to-parallel ratio subject to both a FLOPs budget and a time budget, which is a more complex optimization problem. The sequential revision approach's applicability to real-time settings remains an open question.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around multilingual speech models from a pretraining-scale-centric view toward one where training procedure design β€” specifically, the temporal dynamics of data balancing β€” is a first-class research object with outsized impact on final performance. The conceptual contribution is not that fine-tuning on high-quality data helps (that is standard practice across the field) but rather that when and how gradually you transition between data distributions during fine-tuning determines whether you improve or degrade out-of-domain performance, and that the learning rate schedule provides a principled signal for timing this transition.

The magnitude of this shift is more like a methodological reframing than a paradigm shift. It does not displace the dominant encoder-decoder architecture or the value of large-scale pretraining β€” Canary-1B-v2 still relies on 1.7M hours of pretraining data and a FastConformer encoder. But it changes what practitioners should optimize after pretraining. Prior to this work, the standard approach to multilingual model fine-tuning was to construct a balanced subset and train on it with fixed sampling weights. This paper demonstrates that this static approach can actively degrade performance on out-of-domain data: the statically fine-tuned model's CoVoST2 ASR WER of 12.91% is worse than the pre-trained model's 10.93% (Tables 3 and 4). The weight-scheduled variant recovers this loss and improves it to 10.81%, a 2.1% absolute gain over static fine-tuning.

The paper resolves a specific tension in the multilingual speech literature that prior work had not articulated clearly: the conflict between data quality and data diversity. High-quality fine-tuning data (NeMo ASR Set 3.0, FLEURS) is typically cleaner and more controlled than web-scale pretraining data (YouTube, VoxPopuli parliamentary recordings). Fine-tuning naively on this clean data improves in-domain performance but sacrifices robustness to spontaneous, noisy, or acoustically diverse speech β€” the model "forgets" how to handle the messy conditions it learned during pretraining. The dynamic weight scheduling solution is to maintain contact with the broader pretraining distribution during the plastic phase of fine-tuning (when the learning rate is high enough to cause significant representational change) and only shift fully to the clean distribution once the model has stabilized. This reframes the data balancing problem from a static optimization ("what proportions of each data source should I use?") to a dynamic one ("what proportions should I use at each point in training, given the current learning rate?").

The paper also shifts the positional encoding conversation in speech processing away from a NLP inheritance model. The finding that ALiBi outperforms RoPE for long-form ASR while RoPE outperforms ALiBi for translation (Section 6.4.2, Figure 13) challenges the growing default of adopting RoPE for all speech Transformer encoders. The task-dependence of this result β€” that the optimal positional encoding depends on whether the task requires local or global context β€” provides a more nuanced framework than "RoPE works best for Transformers." For ASR-only systems (like Parakeet-TDT-0.6B-v3), the paper suggests ALiBi may be the better choice, while multi-task systems supporting translation should either use RoPE or rely on convolutional positional encoding (as FastConformer does implicitly).

Finally, the paper's architectural decoupling of timestamp generation β€” moving from in-model prediction to an external forced aligner β€” establishes a separation-of-concerns principle that may influence future multi-task speech model design. Rather than asking a single model to simultaneously optimize transcription accuracy and temporal precision, the paper demonstrates that a dedicated alignment model produces reliable timestamps without interfering with the primary ASR/AST objectives. This could reduce the complexity of training recipes for future systems that need timestamping capability.

Follow-Up Research This Work Enables

Quantifying the forgetting-prevention mechanism through controlled plasticity ablations. The paper hypothesizes that weight scheduling prevents catastrophic forgetting by maintaining exposure to the pretraining distribution during the high-learning-rate phase of fine-tuning. A direct test would vary the scheduling function while holding all else constant: compare cosine scheduling against a step function (abrupt switch to target distribution at step 5,000), a linear ramp, and a reverse-cosine (fast at start, slow at end) on the CoVoST2 ASR benchmark. If the paper's hypothesis is correct, the step function should produce the worst CoVoST2 WER (matching or exceeding the static fine-tuning's 12.91%), the reverse-cosine should be intermediate, and the standard cosine should be best. Additionally, measuring per-layer representational similarity (e.g., CKA or centered kernel distance) between the pre-trained and fine-tuned checkpoints throughout training would reveal whether scheduling genuinely preserves pretrained features in early layers more than static fine-tuning does, or whether the benefit operates through some other mechanism.

Stress-testing the ALiBi-for-ASR finding across diverse speech conditions and encoder architectures. The paper demonstrates ALiBi's superiority for long-form ASR only with the nGPT encoder on the Earnings dataset (Figure 13). Two natural extensions would tighten the finding. First, evaluate the same ALiBi-vs-RoPE comparison on a FastConformer encoder (replacing or augmenting its convolutional positional encoding) across multiple ASR benchmarks including spontaneous speech (Switchboard, Fisher), noisy conditions (CHiME), and multiple languages. If ALiBi consistently outperforms RoPE for ASR across architectures, the finding becomes a general design principle rather than an nGPT-specific curiosity. Second, test whether the ASR-vs-translation task dependence holds for a broader set of language pairs, particularly those with different word-order typology: does RoPE's advantage for translation hold for SOV languages (Japanese, Turkish) where long-range reordering is even more extreme than for the European SVO languages tested in the paper? A negative result β€” finding that ALiBi also outperforms RoPE for translation into reordering-heavy languages β€” would clarify whether the task dependence is about "translation generally needs global context" or specifically about the reordering patterns of the tested language pairs.

Extending the NFA timestamp pipeline to typologically distant languages and quantifying alignment quality. The paper explicitly flags that the NFA pipeline's AST timestamp performance is untested for languages with divergent syntactic structures (Section 5.2). A direct follow-up would apply the NFA pipeline to, for example, English→Japanese or English→Arabic speech translation and measure word-level and segment-level timestamp accuracy against manually annotated ground-truth alignment. The paper's hypothesis — that segment-level timestamps work because segment boundaries are language-independent — would predict reasonable segment-level accuracy but poor word-level accuracy for these language pairs. Quantifying this degradation would establish the practical boundary of the NFA approach and determine whether a language-adaptive alignment strategy (e.g., using a translation-specific CTC model or incorporating syntactic reordering knowledge into the Viterbi path constraints) is necessary for broader deployment.

Training a lightweight difficulty predictor for the Parakeet-Canary model family. While this paper does not use a difficulty-aware test-time compute allocation strategy (unlike the example paper), the concept of routing decisions based on estimated input difficulty is directly applicable. A practical extension would train a lightweight classifier β€” perhaps a distilled version of the FastConformer encoder's intermediate representations or a separate small model β€” to predict whether a given audio segment will have high WER or low COMET (i.e., is "difficult" for the model). Inputs flagged as difficult could be routed to a larger model (Seamless-M4T-v2-large) or a human review queue, while easy inputs are handled by Canary-1B-v2 or Parakeet-TDT-0.6B-v3. The paper's per-language WER tables in Appendices B-D provide the ground-truth difficulty labels needed to train such a predictor. The key metric would be the cost-accuracy tradeoff: what fraction of inputs are routed to the large model, and how much does the cascaded system's aggregate WER/COMET improve over using Canary-1B-v2 alone?

Replicating the weight-scheduled fine-tuning approach on an LLM-based speech model. The paper positions Canary-1B-v2 against LLM-based systems (Voxtral-Mini-3B-2507, Phi-4-multimodal-instruct) but all baselines are evaluated off-the-shelf without comparable fine-tuning. A fairer comparison would apply the same weight-scheduled fine-tuning protocol (four-group balanced construction, 200-hour cap per language pair, cosine weight schedule, 10K steps) to an open-weight LLM-based speech model β€” for example, fine-tuning SALM or a Whisper-derived architecture on the same 15K-hour high-quality subset. The question is whether the 25% relative WER reduction and 6-point COMET gain that the paper achieves for FastConformer also materialize for other architectures, or whether these gains are partially attributable to FastConformer's specific inductive biases being well-suited to the fine-tuning data. A null result β€” weight scheduling does not help an LLM-based model β€” would suggest that the technique is architecture-dependent, while a positive result would establish it as a general fine-tuning strategy for multilingual speech models.

Practical Applications and Downstream Use Cases

Cost-efficient multilingual transcription and translation pipelines for enterprises. Organizations that operate across European markets β€” customer support centers handling calls in 10+ languages, media companies subtitling content for EU audiences, or government agencies processing multilingual meeting recordings β€” currently face a choice between deploying a large, slow model (Seamless-M4T-v2-large at 2.3B parameters) or managing a fragmented set of per-language models. Canary-1B-v2 provides a single-model solution that handles 25 languages for both ASR and AST with throughput approximately 10Γ— higher than Whisper-large-v3 (RTFx 749 vs. ~74 on English ASR, Table 5) while matching or exceeding its multilingual ASR accuracy (8.1% vs. 9.9% WER averaged across FLEURS, CoVoST2, and MLS for 24 languages, Figure 10). For a pipeline processing 100,000 hours of audio per month, the 10Γ— throughput difference translates directly to GPU-hour costs: roughly 10Γ— fewer A100 instances needed to maintain the same processing latency SLA. Parakeet-TDT-0.6B-v3 offers an even more aggressive cost profile for ASR-only deployments (RTFx 3332.74, ~54Γ— faster than Phi-4-multimodal-instruct at comparable accuracy per Table 5), relevant for applications like call transcription where translation capability is not needed.

On-device or edge ASR for European language voice assistants with competitive accuracy. The companion Parakeet-TDT-0.6B-v3 model achieves 6.32% WER on the Hugging Face Open ASR Leaderboard (Table 5) β€” within 0.18 percentage points of Phi-4-multimodal-instruct's 6.14% while running ~54Γ— faster β€” supporting a 0.6B-parameter model that can plausibly run on-device or at the edge for latency-sensitive voice assistant applications. The paper's noise-robustness results (Table 6) show that Parakeet-TDT-0.6B-v3 maintains WER below 2% down to SNR 25 and below 13% even at SNR -5 (severe noise), outperforming the larger Canary-1B-v2 under heavy noise conditions. For a voice assistant deployed in a car, factory floor, or public space β€” where background noise is the norm rather than the exception β€” this combination of small size, high speed, and noise robustness is directly deployable without requiring cloud round-trips.

Timestamped subtitle generation for EU parliamentary and media content. The paper's NFA timestamp pipeline (Section 5, Figure 8) produces segment-level timestamps for both ASR and AST output across 25 European languages, tested successfully on FLEURS and CoVoST2 benchmarks. For organizations that need to produce timestamped subtitles from multilingual audio β€” EU institutions generating meeting transcripts in 24 official languages, broadcasters creating accessible content, or video platforms complying with accessibility regulations β€” Canary-1B-v2 provides a single integrated solution rather than requiring separate ASR/AST models plus an external alignment step (as WhisperX does for Whisper models). The paper's specific recommendation β€” use segment-level timestamps for AST rather than word-level, due to the non-monotonic cross-lingual relationship (Section 5.2) β€” provides a practical guideline for subtitle generation workflows where word-level highlighting is less critical than accurate segment boundaries.

When to Prefer This Method

The paper explicitly positions Canary-1B-v2 and Parakeet-TDT-0.6B-v3 against three named alternatives β€” general-purpose Transformers (Whisper, SeamlessM4T), LLM-based systems (Voxtral, Phi-4-Multimodal), and the previous Canary v1 β€” with clear tradeoffs along the dimensions of throughput, language coverage, and task support:

  • Prefer Canary-1B-v2 over SeamlessM4T-v2-large or Whisper-large-v3 when: inference throughput is the binding constraint and your language coverage falls within the 25 supported European languages. Canary-1B-v2 matches or exceeds Whisper-large-v3 on multilingual ASR (8.1% vs. 9.9% WER, 24-language average) while running ~10Γ— faster, and delivers competitive Xβ†’En AST (79.28 vs. 81.71 COMET on 24-language FLEURS) at less than half the parameters of SeamlessM4T-v2-large. The tradeoff is a persisting 2–2.5 COMET-point gap to SeamlessM4T-v2-large on Xβ†’En translation and weaker Enβ†’X performance on spontaneous speech (80.03 vs. 82.66 COMET on 24-language CoVoST2).

  • Prefer Parakeet-TDT-0.6B-v3 over Canary-1B-v2 when: you need ASR only (no translation) and either throughput is paramount (RTFx 3332.74 vs. 749 for Canary-1B-v2, Table 5) or noise robustness matters more than absolute peak accuracy (Parakeet achieves 12.21% WER at SNR -5 vs. 19.38% for Canary-1B-v2, Table 6). The tradeoff is a small accuracy penalty on clean speech: 6.32% vs. 5.56% WER on English ASR leaderboard average.

  • Prefer Canary-1B-v2 over LLM-based systems (Voxtral-Mini-3B-2507, Phi-4-multimodal-instruct) when: inference cost per query dominates model selection, and you are willing to accept a small accuracy gap on some translation benchmarks. Phi-4-multimodal-instruct achieves 6.14% WER vs. Canary-1B-v2's 5.56% on English ASR (Canary wins) and 84.06 vs. 82.41 COMET on 6-language FLEURS Xβ†’En (Phi wins by 1.65 points), but runs at an estimated ~12Γ— slower throughput (RTFx ~61 vs. 749). For batch inference or high-volume production, the throughput gap favors Canary decisively.

  • Prefer SeamlessM4T-v2-large or LLM-based systems over Canary-1B-v2 when: your language coverage needs extend beyond the 25 European languages, or you require the highest possible Xβ†’En and Enβ†’X spontaneous-speech translation quality (CoVoST2 performance) and are willing to pay the throughput cost. Canary-1B-v2's largest gaps are on CoVoST2 Xβ†’En (78.14 vs. 80.26 COMET) and CoVoST2 Enβ†’X (80.03 vs. 82.66 COMET), both against SeamlessM4T-v2-large. For applications where spontaneous, conversational translation quality is the primary metric and languages outside Europe are required, the larger general-purpose models remain preferable.