Granary: Speech Recognition and Translation Dataset in 25 European Languages

🎯 Pitch

A massive pseudo-labeling effort finds you can train ASR models with half the data if you filter ruthlessly—and reveals that prior datasets left hallucinated transcripts and truncated audio unchecked. The result is an open, 643k-hour corpus spanning 25 European languages, plus the first large-scale open speech translation pairs.

1. Executive Summary

This paper introduces Granary, a large-scale open-source collection of pseudo-labeled speech datasets for automatic speech recognition and speech translation across 25 European languages, built by processing approximately 1 million hours of raw audio from YODAS, YouTube-Commons, and MOSEL through a pipeline that includes two-pass Whisper inference, language ID verification, hallucination filtering, and LLM-based punctuation restoration. The pipeline produces 643k hours of ASR transcriptions and 351k hours of X→En AST translation pairs, with the ASR filtering achieving a 60.7% retention rate. Models trained on Granary's filtered VoxPopuli subset match or slightly outperform those trained on the MOSEL baseline while using approximately 50% less data—for English, a FastConformer-L model trained on 14k Granary hours achieves comparable word error rates to one trained on 23.5k MOSEL hours, with a roughly 10% relative improvement on the FLEURS test set, establishing that more aggressive filtering yields higher-quality training data even for high-resource languages but only when the filtration pipeline addresses truncation, hallucinations, and language identification errors that prior pseudo-labeling efforts left uncorrected.

2. Context and Motivation

The Problem: Speech Data Scarcity in an Era of Data-Hungry Models

The central problem this paper addresses is deceptively simple to state but structurally difficult to solve: high-quality, human-annotated speech data is extremely scarce, yet modern speech models require enormous quantities of it to perform well. While this tension exists for all languages, it is particularly acute for low-resource languages, where manual annotation is even rarer and the economics of data collection are least favorable.

The paper frames this problem in the context of a broader trend affecting all of machine learning. The authors cite Villalobos et al. (2022), who project that human-generated text annotations may soon be exhausted as training data for large language models. The speech domain faces an analogous—and in some ways worse—situation. Unlike text, which can be crawled from the web in vast quantities, high-quality speech transcription requires humans to listen to audio and produce accurate word-level transcripts. This is slow, expensive, and fundamentally non-automatable in a way that text scraping is not. The result is a structural scarcity: the demand for training data from increasingly large speech foundation models far outstrips the supply of human-annotated speech.

This scarcity manifests as a practical bottleneck for speech technology deployment. Applications such as automatic speech recognition (ASR) and automatic speech translation (AST) underpin voice assistants, meeting transcription, content localization, and accessibility tools. For these systems to work reliably across languages, they need training data that captures the acoustic, linguistic, and contextual diversity of real-world speech. When such data is unavailable for a language—or available only in small quantities—the resulting models exhibit degraded accuracy, higher error rates, and systematic biases against speakers of those languages. The paper's focus on 25 European languages, including lower-resource ones like Croatian, Lithuanian, and Maltese, reflects this concern: even within a relatively well-studied continent, many languages lack sufficient annotated speech data to train competitive models.

Why This Problem Matters: The Gap Between Available Audio and Usable Labels

A crucial nuance the paper highlights is the asymmetry between raw audio availability and labeled data availability. There exists a vast amount of unlabeled speech data online—YouTube videos, podcasts, audiobooks, parliamentary recordings—that could, in principle, be used to train speech models. The bottleneck is not the absence of audio; it is the absence of reliable transcriptions paired with that audio. Bridging this gap—converting raw audio into high-quality training examples—is the fundamental challenge.

This matters for several reasons beyond academic interest:

1. Linguistic equity. Speech technology that works only for English and a handful of other high-resource languages exacerbates digital divides. Speakers of lower-resource languages are excluded from voice-based interfaces, automatic captioning, and translation services. The paper's explicit focus on 25 European languages—including several that are underrepresented in speech corpora—positions it as an effort to broaden the language coverage of usable training data.

2. The economics of data curation. Human annotation of speech at scale is prohibitively expensive. The paper notes this directly: "high-quality human-annotated speech data remains scarce due to the high cost and extensive effort required for curation." Pseudo-labeling—using a trained model to generate transcriptions automatically—offers a path to dramatically reduce this cost, but only if the pseudo-labels are reliable enough to serve as training targets. The quality-quantity tradeoff in pseudo-labeled data is the central tension the paper's pipeline aims to navigate.

3. Reproducibility and open science. Much of the pseudo-labeled data used to train state-of-the-art speech models remains proprietary. The paper cites Seamless (Barrault et al., 2023) and the work of Puvvada et al. (2024) as examples where pseudo-labeled data drives performance but is not publicly released. This limits reproducibility: researchers outside the organizations that produced the data cannot verify results, build on the pipelines, or adapt the data for new languages. By open-sourcing Granary and its pipeline, the paper aims to lower this barrier.

4. Enabling multi-task and multilingual models. The paper's motivation connects to a broader trend in machine learning: large, multi-task, multilingual models benefit from diverse training data. The more languages and tasks represented in training, the better these models generalize—but only if the data quality is sufficient. Providing pseudo-labeled data that covers both ASR (transcription) and AST (translation into English) across 25 languages enables training models that handle both tasks and many languages simultaneously, which the paper positions as a direction for future work.

Prior Approaches and Their Limitations

The paper identifies three categories of prior work and explains where each falls short. Understanding these limitations is essential for appreciating why Granary's pipeline is designed the way it is.

YouTube Caption-Based Datasets: YODAS and YouTube-Commons

The first category consists of large-scale datasets that derive labels from YouTube's built-in caption system. YODAS (Li et al., 2024) provides over 500,000 hours of audio across 100+ languages with labels sourced from YouTube subtitles. YouTube-Commons (Pleias, 2025) follows a similar approach, curating a massive open corpus with transcriptions from YouTube captions, though it is heavily skewed toward English (~70% of the data).

The fundamental problem with these datasets is label reliability. YouTube captions come from multiple sources: some are professionally created, some are auto-generated by YouTube's own ASR system, and some are user-submitted with no verification. The paper states bluntly that YODAS annotations "derive from YouTube subtitles, which are often unreliable" and that "even manually created captions lack guaranteed human verification." This means the transcriptions may contain errors, omissions, paraphrases rather than verbatim text, or timing misalignments with the audio. Using such labels as training targets for an ASR model risks teaching the model to reproduce these errors.

Additionally, YouTube-based datasets suffer from language identification (LID) inaccuracies. The paper reports that for some languages, LID errors lead to massive data loss: "only 20% retention for Bulgarian, Ukrainian." This means the majority of audio labeled as a given language may actually contain a different language, contain code-switching, or be misclassified entirely. Training on such mislabeled data would degrade performance for the target language.

Finally, these datasets contain significant noise—non-speech audio, music, overlapping speakers, poor recording quality—that requires extensive preprocessing before the data is usable for training.

Pseudo-Labeled Datasets: MOSEL

The second category is represented by MOSEL (Gaido et al., 2024), which takes a different approach. Rather than relying on YouTube captions, MOSEL uses Whisper-large-v3 (Radford et al., 2023) to pseudo-label existing speech corpora—specifically VoxPopuli (Wang et al., 2021) and LibriLight (Kahn et al., 2020)—producing approximately 950,000 hours of transcribed speech for EU languages.

MOSEL represents a significant advance over caption-based datasets because the labels are generated by a consistent, modern ASR model rather than scraped from heterogeneous sources. However, the paper identifies three specific failure modes that limit MOSEL's data quality:

1. Truncation errors. Whisper's default behavior segments audio into 30-second chunks for processing. When speech is segmented at arbitrary boundaries, utterances can be cut off mid-word or mid-sentence, producing incomplete transcriptions. The paper includes Figure 2, which shows a concrete example: a MOSEL transcription that is truncated mid-sentence compared to the same audio processed through Granary's pipeline, which produces the complete utterance. These truncated segments, when used as training data, teach the model that partial sentences are valid outputs—degrading its ability to produce complete transcriptions.

2. Hallucinations. Whisper, particularly in its turbo variant, is prone to generating text that was never spoken—a phenomenon the paper refers to as "hallucination." This includes repeated n-grams (the model gets stuck in a loop), implausibly long words, and commonly hallucinated phrases like "Thank you very much" or language-specific filler text (the paper gives the example of "Sous-titrage Société Radio-Canada" in French). MOSEL filters some hallucinations but the paper's analysis suggests these filters are insufficient—hallucinated transcriptions remain in the dataset and, when used as training targets, can cause the trained model to learn to produce hallucinated outputs.

3. Language identification errors. Whisper performs language identification as part of its inference process, but this identification is imperfect. When Whisper misidentifies the language of an audio segment, it may transcribe in the wrong language or produce garbled output. MOSEL retains some of these misidentified samples, which introduce cross-language contamination into the training data.

The paper's assessment of MOSEL is nuanced: it is the most directly comparable prior work and serves as the primary baseline for Granary's evaluation, but its data quality is limited by these three issues, and the paper's central claim is that addressing them through more aggressive filtering yields better training data even at reduced volume.

Human-Annotated Datasets and Community Efforts

The third category includes efforts like GigaSpeech (Chen et al., 2021), which provides 10,000 hours of human-transcribed audio. These datasets offer high label quality but are limited in scale and language coverage. The paper notes that such efforts "remain restricted to human-generated data and cover only a limited number of languages." The cost of human annotation makes it infeasible to scale this approach to the hundreds of thousands of hours needed for modern speech foundation models, particularly across many languages.

The AST Gap

For automatic speech translation specifically, the paper identifies an even starker gap. While ASR pseudo-labeling has received some attention (YODAS, MOSEL), "open-source projects tackling translation tasks—particularly in speech applications—are exceptionally sparse." Prior approaches to generating AST training pairs typically involved training text-based neural machine translation (NMT) models on the ASR outputs to produce translations—a multi-step process that compounds errors from both the ASR and NMT models. The paper argues that recent advances in large language models (LLMs) have made them more reliable for translation tasks, opening the possibility of using a single LLM to directly generate translation pairs from ASR transcripts—an approach the paper claims is "the first to the best of our knowledge" for speech translation data.

How Granary Positions Itself

The paper positions Granary not as a fundamentally new type of dataset, but as a quality-focused re-processing of existing open-source corpora through a carefully designed pipeline. The key differentiators from prior work are:

1. Aggressive, multi-stage filtering rather than volume maximization. Where MOSEL prioritizes retaining as much data as possible (publishing pseudo-labels for nearly all available audio), Granary's pipeline explicitly aims to "retain approximately 50% of the original" data while discarding the half that is likely to be low-quality. This is a deliberate design choice: the paper bets that training on a smaller, cleaner dataset will outperform training on a larger, noisier one. The empirical results in Tables 2 and 3 validate this bet—models trained on 14k Granary hours match or exceed those trained on 23.5k MOSEL hours.

2. Addressing truncation through better segmentation and voice activity detection. The paper identifies Whisper's fixed-length segmentation as a root cause of MOSEL's truncation errors and addresses it through two mechanisms: using alignment-based segmentation (Whisper timestamps for YouTube data, Parakeet TDT decoder timestamps for LibriLight) to segment at natural speech boundaries, and integrating Silero VAD with 400ms padding to "minimize truncated transcriptions" and "reduce hallucinations by focusing inference on detected speech regions."

3. Explicit hallucination filtering with language-specific phrase lists. Rather than relying solely on generic hallucination heuristics, the paper compiles "language-specific lists of commonly hallucinated phrases" by examining Whisper's outputs across languages. These lists—which include both language-specific filler (the French "Sous-titrage Société Radio-Canada") and cross-lingual patterns ("Thank you very much" appearing in non-English contexts)—are used to filter out hallucinated samples. This is a concrete, practical improvement over MOSEL's more generic filtering.

4. LLM-based punctuation and capitalization restoration. Whisper outputs are lowercased and lack punctuation, which limits their usefulness for training models that should produce properly formatted text. Rather than accepting this limitation, Granary uses Qwen 2.5-7B-Instruct to restore punctuation and capitalization, with a quality-control heuristic that retains the original pseudo-labels if the LLM's output deviates by more than 5% character error rate—preventing the LLM from introducing its own errors.

5. First open-source AST pseudo-labeling pipeline using LLMs. The paper's translation pipeline—using EuroLLM-9B to translate ASR transcripts into English, followed by quality estimation filtering with cometoid-wmt23—fills what the authors identify as a genuine gap. Prior work either kept AST data proprietary or used multi-step NMT pipelines rather than direct LLM-based translation.

6. Efficiency as a design constraint. The paper emphasizes that the entire pipeline is "designed for efficiency" and "processes vast amount of data within hours." The AST filtration pipeline is reported to filter the entire MOSEL dataset in 47 minutes using 8 nodes with 8 A100 GPUs each. This is not just a convenience claim—it reflects a design philosophy that data processing pipelines should scale to web-scale audio corpora without becoming computational bottlenecks themselves.

In summary, Granary positions itself as a practical engineering contribution that bridges the gap between the existence of raw speech data and the need for high-quality training labels, with specific technical improvements over the most comparable prior work (MOSEL) that collectively yield better data quality at reduced volume. The paper does not claim to introduce novel machine learning techniques; its contribution is the careful integration of existing tools (Whisper, Silero VAD, EuroLLM, Qwen, cometoid) into a pipeline whose filtering decisions are informed by systematic analysis of failure modes in prior pseudo-labeling efforts.

3. Technical Approach

3.1 Reader Orientation

Granary is a data processing pipeline — a sequenced set of computational steps — that takes raw, unlabeled speech audio from open-source corpora and produces high-quality text transcriptions and English translations suitable for training speech recognition and translation models. The core problem it solves is that existing pseudo-labeled speech datasets, while large, contain systematic errors (truncated transcriptions, model hallucinations, language misidentification) that degrade the quality of models trained on them; Granary's solution is an aggressive, multi-stage filtering pipeline that deliberately discards roughly 40% of the input data to produce a smaller but substantially cleaner dataset, with each filtering stage targeting a specific, empirically observed failure mode in prior pseudo-labeling efforts.

3.2 Big-Picture Architecture (Diagram in Words)

The Granary pipeline consists of two parallel tracks — one for ASR (automatic speech recognition) and one for AST (automatic speech translation) — that share a common preprocessing stage. The architecture can be understood as a sequence of seven major components, organized in a "box-and-arrows" flow:

Input Layer: Three open-source speech corpora — YODAS, YouTube-Commons (YTC), and MOSEL — collectively containing approximately 1 million hours of raw audio across 25 European languages. These corpora enter the pipeline in their original formats (various audio codecs, sample rates, and channel configurations).

Component 1 — Audio Standardization: Converts all audio files to a uniform format (FLAC or WAV, 16 kHz sample rate, mono-channel) and enforces a maximum duration of 40 seconds per segment. This is a mechanical step that ensures downstream tools receive consistent inputs.

Component 2 — Long-form Audio Segmentation: Takes potentially lengthy audio recordings (e.g., full YouTube videos, parliamentary sessions) and splits them into manageable segments aligned with speech boundaries. This component uses alignment-based approaches — Whisper timestamps for YouTube-derived data, Parakeet TDT decoder timestamps for LibriLight data — rather than fixed-length segmentation, which is the root cause of MOSEL's truncation errors.

Component 3 — Two-Pass ASR Inference: Runs Whisper-large-v3 via FasterWhisper in two passes over each audio segment. The first pass predicts the language ID; the second pass generates the transcription conditioned on that predicted language. Silero VAD (voice activity detection) with 400ms padding is integrated at this stage to focus inference on detected speech regions, reducing both truncation and hallucination.

Component 4 — Robust Data Filtration: Applies three categories of filters to the pseudo-labeled (audio, transcript) pairs: hallucination detection (repeated n-grams, implausibly long words, language-specific hallucinated phrases), character rate heuristics (rejecting segments with anomalously low or high text-to-duration ratios), and character set validation (rejecting any transcript containing characters outside the valid set for European languages). This is the stage where the majority of data is discarded — approximately 39.3% of input audio is filtered out, producing the 60.7% retention rate.

Component 5 — LLM-Powered Punctuation and Capitalization Restoration: Passes each surviving transcript through Qwen 2.5-7B-Instruct with a language-specific few-shot prompt that instructs the LLM to add proper punctuation and capitalization. A quality-control heuristic compares the LLM's output to the original Whisper transcript — if the character error rate exceeds 5%, the original lowercased transcript is retained instead, preventing the LLM from introducing semantic drift.

Component 6 — AST Translation Pair Generation (parallel track): Takes the ASR transcripts produced by Component 5 and feeds them to EuroLLM-9B, which translates each transcript from its source language into English using greedy decoding via vLLM, generating X→En translation pairs.

Component 7 — AST Filtration: Applies a second filtering pipeline specifically to the translation pairs, including length ratio filtering (from Moses), character histogram filtering, FastText language ID verification (to confirm the translation is actually English), and quality estimation filtering using the cometoid-wmt23 model via PyMarian. This is a GPU-accelerated pipeline implemented in NeMo-Curator.

Output Layer: Two datasets emerge — approximately 638,144 hours of filtered ASR data and 351,048 hours of X→En AST data, both with hallucinations, truncations, and language errors substantially reduced relative to the MOSEL baseline.

3.3 Roadmap for the Deep Dive

First, the pipeline's input data sources (YODAS, YTC, MOSEL) — what they are, why they were chosen, and the specific quality problems each introduces, since these problems directly determine the pipeline's design.
Second, audio standardization and segmentation — the mechanical preprocessing that converts heterogeneous inputs into uniform segments, focusing on why the segmentation algorithm matters and how the choice of alignment method (Whisper timestamps vs. Parakeet TDT) was made.
Third, the two-pass ASR inference engine — how Whisper-large-v3 is configured and deployed, including the FasterWhisper optimizations, the two-pass language ID + transcription protocol, and the integration of Silero VAD to reduce truncation.
Fourth, the three-category filtration system — hallucination detection (with language-specific phrase lists), character rate filtering, and character set validation, since this is where most data is discarded and where Granary's quality advantage over MOSEL is primarily achieved.
Fifth, the LLM-based punctuation restoration module — how Qwen 2.5-7B-Instruct is prompted, the 5% CER heuristic, and why this step matters for downstream model training.
Sixth, the AST pipeline — model selection (why EuroLLM-9B over alternatives), inference configuration, and the translation-specific filtration pipeline built in NeMo-Curator.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering paper whose core idea is that aggressive, failure-mode-specific filtering of pseudo-labeled speech data produces training sets that yield better models despite containing fewer total hours, and that this filtering must address four specific failure modes — truncation, hallucination, language misidentification, and missing punctuation — each of which requires a distinct technical intervention rather than a single generic quality filter.

Input Data Sources: What Gets Processed and Why

The pipeline ingests three open-source speech corpora, each with distinct characteristics and failure modes. These are not treated as interchangeable — the pipeline's segmentation strategy differs based on which corpus is being processed, and the filtering thresholds are language- and corpus-specific.

YODAS (Li et al., 2024) is the largest and noisiest source. It contains over 500,000 hours of audio across 100+ languages, with labels derived from YouTube subtitles. The paper identifies three specific quality problems with YODAS. First, the labels themselves are unreliable — they come from a mixture of professional captioning, auto-generated captions, and user-submitted captions, none of which are guaranteed to be verbatim transcriptions. Second, language identification in the metadata is often wrong, with the paper reporting that only 20% of Bulgarian and Ukrainian audio survives language verification — meaning 80% of the data labeled as these languages is actually something else. Third, YODAS contains substantial quantities of non-speech audio (music, silence, noise) that must be separated from speech before transcription.

YouTube-Commons (YTC) (Pleias, 2025) shares YODAS's fundamental limitation — transcriptions sourced from YouTube captions inherit the same reliability problems — but adds an additional complication: extreme language skew. The paper notes that YTC is "heavily skewed toward English (70% of data)," which limits its usefulness for the multilingual goals of Granary. Additionally, the paper reports that "due to download constraints, only a subset is currently processed, with the remainder planned for future work," meaning the Granary release does not include the full YTC corpus.

MOSEL (Gaido et al., 2024) differs from the YouTube-based corpora in a critical way: its labels are not scraped from captions but generated by Whisper-large-v3, a consistent, modern ASR model. MOSEL contains approximately 950,000 hours of pseudo-labeled speech drawn from VoxPopuli (Wang et al., 2021) — European Parliament recordings — and LibriLight (Kahn et al., 2020) — audiobook readings. Because MOSEL's labels come from a known model rather than heterogeneous human sources, they are more consistent than YouTube captions, but they inherit Whisper's specific failure modes: truncation at 30-second segment boundaries, hallucinations (particularly in the turbo variant), and language identification errors. MOSEL also provides metadata — hallucination flags and language ID predictions from Whisper — that Granary leverages in its own filtering.

The paper's choice to re-process MOSEL data rather than simply use it as-is reflects the central thesis: MOSEL's pseudo-labels contain systematic errors that Granary's pipeline can correct, and discarding the worst-quality segments yields better training data than retaining everything.

Audio Standardization and Segmentation

Before any inference or filtering occurs, all audio files undergo mechanical standardization. Every file is converted to either FLAC or WAV format at a sample rate of 16 kHz with a single (mono) channel. These choices reflect standard practice in speech processing: 16 kHz is sufficient to capture the frequency range of human speech (roughly 0–8 kHz, per the Nyquist theorem), and mono avoids unnecessary channel duplication for speech content. The paper also enforces a maximum duration of 40 seconds per output segment, citing Koluguri et al. (2024) for this threshold. This is not arbitrary — Whisper's architecture processes audio in 30-second windows, and keeping segments within a similar range avoids the need for additional chunking during inference.

Why segmentation matters beyond convenience. The segmentation step is the first place where Granary's design addresses a specific failure mode in prior work: truncation. MOSEL's approach — running Whisper on fixed 30-second chunks — means that utterances that span chunk boundaries get cut off mid-word or mid-sentence. These truncated transcriptions, when used as training targets, teach the ASR model that incomplete utterances are valid outputs, degrading its ability to produce complete transcriptions at test time.

Granary replaces fixed-length segmentation with alignment-based segmentation, which uses information from the transcription (pseudo-label) to find natural speech boundaries. For the YouTube-derived corpora (YODAS and YTC), this means running Whisper to generate word-level timestamps, then using those timestamps to split the audio at word boundaries rather than at arbitrary 30-second intervals. For the LibriLight subset of MOSEL, the paper uses the Parakeet TDT (Time-Duration-Transducer) decoder — specifically the parakeet-tdt_ctc-110m model — to generate what the paper describes as "high-quality segment-level alignment."

The paper reports an important empirical finding that validates this choice: during development, the authors compared ground-truth transcriptions to intermediate pseudo-labels and found that "pseudo-labels consistently improved segmentation results." In other words, even imperfect machine-generated transcriptions provide better alignment cues than no transcription at all, because they capture the approximate temporal location of words. This is a non-obvious result — one might expect that noisy labels would produce noisy segmentation — and it justifies the use of pseudo-labels as input to the segmentation algorithm itself.

The VAD integration. After segmentation, Silero VAD (Silero Team, 2024) — a pre-trained enterprise-grade voice activity detector — identifies which portions of each segment contain actual speech. The VAD output is padded by 400ms on each side before being passed to the ASR model. This padding serves two purposes: it prevents the ASR model from missing speech that occurs near segment boundaries, and it reduces hallucinations by ensuring that the ASR model receives context around speech regions rather than abrupt onsets. The paper states that this integration "minimized truncated transcriptions" and "reduced hallucinations by focusing inference on detected speech regions."

A practical detail: The paper notes that Whisper's segment-level timestamps were "poor at word alignment but effective for speech/non-speech detection." This is why the pipeline uses Whisper timestamps for the broad task of separating speech from silence, but relies on Parakeet TDT for the finer-grained task of word-level alignment when available. For approximately 60,000 hours of English data processed with the TDT decoder, the alignment was good enough to "eliminate the need for a second inference pass" — a meaningful computational saving.

Two-Pass ASR Inference with FasterWhisper

Once audio is segmented and speech regions are identified, the core ASR pseudo-labeling begins. The pipeline uses Whisper-large-v3 (Radford et al., 2023), accessed through the FasterWhisper reimplementation (SYSTRAN, 2023) which uses CTranslate2 for optimized inference. The choice of Whisper-large-v3 over alternatives is motivated by three factors: "its robust performance, multilingual capabilities, and open license." The paper explicitly acknowledges following MOSEL's best practices in adopting this model, while adding modifications to address its limitations.

Two-pass protocol. The inference proceeds in two sequential passes over each audio segment, following the approach established by MOSEL:

Pass 1 — Language Identification: Whisper processes the audio and predicts the language ID. This prediction is extracted and stored as metadata. Because Whisper is a multilingual model trained to transcribe in whatever language it detects, running transcription without specifying the language risks the model switching languages mid-utterance or defaulting to a high-resource language when uncertain. By first determining the language, the pipeline can constrain the second pass.

Pass 2 — Transcription: Whisper processes the same audio again, but this time with the language explicitly set to the predicted language ID from pass 1. This "conditioned" decoding improves transcription accuracy because it narrows the model's output distribution to tokens from the correct language. The inference uses a beam size of 5 (meaning Whisper explores 5 candidate hypotheses in parallel and selects the best) and a chunk batch size of 16 (processing 16 audio chunks simultaneously for throughput).

FasterWhisper modifications. The paper reports making "custom modifications to FasterWhisper source code to extract language IDs for each segment." In the standard FasterWhisper implementation, language ID is available only as a byproduct of the transcription pass. By modifying the code to extract and store language IDs explicitly, the pipeline gains the ability to filter samples where Whisper's predicted language does not match the expected language — a critical filtration step described in Section 3.1.3 of the paper.

The Silero VAD integration in detail. Silero VAD is not run as a separate pre-processing step — it is "integrated into the pipeline" during the two-pass inference. For each audio segment, Silero VAD first identifies the time intervals containing speech. These intervals, with 400ms padding added to each boundary, define the regions that Whisper processes. Audio outside these regions is excluded from inference entirely. This has two complementary effects on data quality:

Truncation reduction: By providing Whisper with speech-bounded audio rather than arbitrary 30-second chunks, the VAD integration ensures that utterances are not cut off mid-word at segment boundaries. The paper's Figure 2 directly illustrates this improvement — a MOSEL transcription truncated mid-sentence ("Mr President, I should like to thank the Commission for its") appears in Granary as the complete utterance ("Mr President, I should like to thank the Commission for its rapid response.").
Hallucination reduction: Whisper is known to hallucinate — generate text that was never spoken — when processing silence, noise, or non-speech audio. By restricting inference to regions where speech is actually detected, Silero VAD eliminates the primary trigger for these hallucinations. The paper reports this effect explicitly: the VAD integration "reduced hallucinations by focusing inference on detected speech regions."

Language ID Verification and Filtration

After the two-pass inference, each (audio, transcript) pair is associated with a predicted language ID from Whisper's first pass. The pipeline applies a language ID verification step that filters out samples where the predicted language does not match the expected language for that corpus.

The basic filter. For all corpora, the pipeline "eliminates data points where Whisper's predicted LID does not align with the target language." This is a binary decision: if Whisper thinks the audio is in German but the corpus metadata says it should be Croatian, the sample is discarded. This addresses the language identification inaccuracies that the paper identifies in YODAS, where metadata labels are often wrong — if Whisper disagrees with the metadata, the pipeline trusts Whisper (the more reliable signal) and removes the sample rather than risking cross-language contamination.

The VoxPopuli-specific refinement. For the VoxPopuli subset (which comes from European Parliament recordings), the paper applies an additional filter: samples where Whisper predicts multiple languages are removed. The paper explains that this is "common in the Voxpopuli dataset due to interpreter voices" — in parliamentary settings, one speaker may be heard in the original language while an interpreter speaks simultaneously in another language. These code-switched or overlapping-language segments are not suitable for training a monolingual ASR model.

Furthermore, for Granary's VoxPopuli processing, the paper applies a confidence threshold: samples where the language ID probability (lid_prob) is below 0.8 are filtered out. This means that Whisper must be at least 80% confident in its language identification for the sample to be retained. The threshold of 0.8 is a design choice that balances retention (higher thresholds discard more data) against purity (lower thresholds admit more misidentified samples). The paper does not report an ablation over this threshold, presenting it as the chosen operating point.

Why this matters. Language ID filtering is one of the dimensions where Granary's pipeline is more aggressive than MOSEL's. MOSEL publishes pseudo-labels for samples even when Whisper's language identification is uncertain, leaving it to downstream users to decide whether to filter. Granary makes this filtering decision as part of the pipeline, removing samples that would introduce noise into training. This contributes to the overall retention rate of 60.7% — language ID filtering is one of the filters that discards data.

Robust Data Filtration: The Three-Category System

This is the pipeline stage where the majority of filtration occurs and where Granary's approach most clearly diverges from MOSEL's. The paper describes "three primary metrics for conducting the filtration process," each targeting a different failure mode in pseudo-labeled data.

Category 1 — Hallucination Detection. Whisper, like many sequence-to-sequence models, can produce text that was never spoken. The paper identifies three distinct hallucination patterns:

Repeated n-grams: The model gets stuck in a loop, producing the same word or phrase multiple times (e.g., "the the the the"). The pipeline detects these by scanning transcripts for n-gram repetition beyond a threshold and flags them for removal.

Long words: The model generates implausibly long character sequences that do not correspond to real words in any language. The pipeline detects these by checking word lengths against language-specific thresholds.

Frequently hallucinated phrases: This is the most interesting and practically useful filter. The paper reports having "compiled language-specific lists of commonly hallucinated phrases" through custom modifications to Whisper-large-v3 and its turbo variant. These lists are generated by examining Whisper's outputs across many samples per language and identifying text patterns that appear systematically but are not actually present in the audio. The paper gives two concrete examples: the French phrase "Sous-titrage Société Radio-Canada" (a captioning credit that Whisper hallucinates in French audio) and the English phrase "Thank you very much" (which Whisper sometimes appends to the end of transcriptions, even when not spoken). These phrase lists are language-specific — what Whisper hallucinates in French differs from what it hallucinates in Polish — and the paper commits to releasing them as part of the open-source pipeline.

The hallucination detection operates as a set of binary flags: if any of the three hallucination patterns is detected for a given (audio, transcript) pair, the pair is removed from the dataset. This is a hard filter, not a soft weighting — the sample is either kept or discarded.

Category 2 — Character Rate Filtering. This filter examines the relationship between the length of the transcript (in characters) and the duration of the audio (in seconds). The paper uses "language- and corpus-specific heuristics" to define acceptable ranges for this ratio.

The intuition is straightforward: normal speech has a characteristic speaking rate that varies by language (some languages are more information-dense per syllable, some speakers talk faster) but falls within predictable bounds. If the character rate is anomalously low — meaning very few characters for a long audio segment — the audio likely contains mostly silence, music, or non-speech sounds that produced only a few words of transcription. If the character rate is anomalously high — meaning many characters for a short segment — the transcript likely contains hallucinated text, repetitions, or formatting artifacts.

The paper states that this filter eliminates "speech-transcription pairs with anomalously low or high character rates, assuming such samples may contain non-speech segments, poor pseudo-labels, or unusual speech patterns." The thresholds are language-specific (because character density per second of speech varies by language) and corpus-specific (because different recording conditions produce different characteristic silence ratios).

Category 3 — Character Set Filtering. This is a hard constraint: any transcript containing a character not in the allowed set for European languages is discarded. The paper defines "a comprehensive superset of over 300 characters and symbols" that covers the alphabets of all 25 represented European languages — this includes Latin characters with diacritics (e.g., č, ñ, ü), Cyrillic characters (for Bulgarian), Greek characters (for Greek), and common punctuation.

The character set filter serves two purposes. First, it catches transcripts where Whisper has output text in the wrong script — e.g., producing Cyrillic output for a language that uses Latin script, which indicates a language identification error that the LID filter may have missed. Second, it catches transcripts containing control characters, emoji, or other non-text symbols that can appear as artifacts in Whisper output but should not be present in training data for a speech recognition model.

Why three categories rather than one unified quality score? The paper's design choice to use three separate filters rather than a single composite quality metric reflects an engineering philosophy that different failure modes require different detection mechanisms. A hallucinated phrase cannot be detected by character rate (it may have perfectly normal length), and a language script error cannot be detected by hallucination pattern matching (it may be perfectly fluent but in the wrong language). By keeping the filters separate, each can be tuned independently for its specific failure mode, and the pipeline's behavior is interpretable — when a sample is discarded, the reason (hallucination, rate anomaly, or invalid character) is known.

LLM-Powered Punctuation and Capitalization Restoration

Whisper-large-v3 outputs lowercased text without punctuation. This is a deliberate design choice in Whisper's architecture — the model was trained on data where punctuation and capitalization were removed to simplify the tokenizer vocabulary and improve multilingual generalization. However, for downstream ASR training, punctuation and capitalization are often desirable: a model trained on all-lowercase, unpunctuated transcripts will produce all-lowercase, unpunctuated output at inference time, which is less readable and less useful for applications like subtitle generation or meeting transcription.

MOSEL does not address this limitation — its pseudo-labels are lowercased and unpunctuated. Granary adds a restoration step using Qwen 2.5-7B-Instruct, a 7-billion-parameter instruction-tuned language model.

The prompting strategy. For each language, the paper crafts a "language-specific prompt" that instructs Qwen to assess and correct the punctuation and capitalization of a given transcript. The prompt includes "multiple correction examples" — few-shot demonstrations showing the model what properly punctuated text looks like for that language. This is necessary because punctuation conventions differ across languages (e.g., quotation marks use different characters in different languages, decimal separators vary, sentence boundaries are marked differently).

The quality-control heuristic. An LLM asked to edit text can introduce its own errors — it might rephrase content, add words that were not in the original, or delete portions of the transcript. To prevent this, Granary implements a hard quality gate: after Qwen produces its punctuated output, the pipeline computes the character error rate (CER) between Qwen's output and the original Whisper transcript.

CER measures the minimum number of character insertions, deletions, and substitutions needed to transform one string into the other, normalized by the length of the reference string:

$\text{CER} = \frac{S + D + I}{N}$

where $S$ is the number of substituted characters, $D$ is the number of deleted characters, $I$ is the number of inserted characters, and $N$ is the total number of characters in the reference string (the original Whisper transcript).

What it computes: the fraction of characters in the original transcript that would need to be changed to produce the LLM's output. A CER of 0 means the LLM made no changes (identical output); a CER of 0.03 means 3% of characters were changed (e.g., adding a period and capitalizing the first letter); a CER of 0.15 means substantial modification.

The pipeline applies a threshold: if CER exceeds 5% (0.05), Qwen's output is discarded and the original Whisper pseudo-label is retained instead.

Why this form and threshold: CER (rather than word error rate) is appropriate because punctuation changes are character-level — adding a period or capitalizing a letter changes characters without changing words. The 5% threshold is designed to allow Qwen to add punctuation and capitalization (which typically changes 1-3% of characters in a normal sentence) while catching cases where the LLM rephrases content, adds hallucinated text, or deletes portions of the transcript. The paper explicitly acknowledges that "the quality of these modifications remains a subject for further testing" — this is a pragmatic heuristic, not a theoretically justified optimum.

A subtle design choice: The 5% threshold "allows Qwen to potentially correct typos or refine formulations." In other words, if Whisper made a small transcription error and Qwen's output accidentally corrects it (or produces a plausible alternative), that output is retained. This means the punctuation restoration step can serve double duty as a light error correction step. However, this also introduces a risk: Qwen might "correct" something that Whisper got right, introducing a new error. The CER threshold provides a bound on how much change is permitted.

AST Translation Pipeline: Model Selection and Inference

The automatic speech translation (AST) pipeline begins with the filtered, punctuated ASR transcripts from the previous stages and produces English translations, creating X→En training pairs for speech translation models.

Model selection. The paper reports benchmarking several candidate translation models to select the best one for pseudo-labeling. The candidates included:

Alma-13B-R (Xu et al., 2024): A 13-billion-parameter LLM fine-tuned for translation. The paper reports it "under-performs significantly in speech domain (despite achieving impressive results on WMT test sets)" — a finding that highlights the domain gap between formal written text (WMT benchmarks) and spoken language transcripts (Granary's input). Speech transcripts contain disfluencies, incomplete sentences, and colloquial expressions that differ from the edited text on which translation models are typically evaluated.
Qwen-2.5-7B (Yang et al., 2024): A 7-billion-parameter general-purpose LLM. The paper reports it "suffers from hallucination issues during pseudo-labeling" — generating translations that are fluent but do not correspond to the source text.
EuroLLM-1.7B and EuroLLM-9B (Martins et al., 2024): Multilingual language models specifically trained for European languages. The paper identifies EuroLLM-9B as the best-performing model after benchmarking on the Flores dataset (Goyal et al., 2022) covering all 24 translation directions of interest. The 1.7B variant was tested but the 9B version was selected for final use.
Riva-Megatron Any2Any 1B: An encoder-decoder NMT model. The paper does not report detailed performance for this model, including it in the benchmarked set but not in the final pipeline.

The paper also explicitly excludes several models from consideration: GPT-4o ("out of cost concerns" — using a paid API for millions of translations would be prohibitively expensive), TowerInstruct-13B and Aya-23 ("because of non-commercial licenses" — these could not be used in an open-source dataset release).

Inference configuration. The selected model, EuroLLM-9B, is run using the translation prompt from the model's official model card — a standard practice that ensures the model is prompted in the way its creators intended for translation tasks. The paper uses greedy decoding (temperature = 0, always selecting the most likely next token) via vLLM, an inference engine optimized for high-throughput LLM serving.

The choice of greedy decoding is explicitly justified: "We also experimented with beam search, which provided a slight improvement for EuroLLM-1.7B but had diminishing returns for the 9B model, making the added computational cost unjustifiable." This is a practical engineering decision — beam search with, say, a beam size of 5 would increase inference cost by roughly a factor of 5 while providing minimal quality improvement for the larger model. For a pipeline that must process hundreds of thousands of hours of transcripts, this cost-quality tradeoff favors greedy decoding.

AST Filtration Pipeline

The translation pairs (source transcript, English translation) produced by EuroLLM-9B are not used directly. A second, translation-specific filtration pipeline is applied, implemented in NeMo-Curator, which the paper describes as "a GPU-accelerated data curation toolkit." The use of GPU acceleration is important — the paper reports that the entire MOSEL dataset can be filtered in 47 minutes using 8 nodes with 8 A100 GPUs each, a speed that makes the filtration practical at scale.

The AST filtration consists of five sequential steps:

Step 1 — Length Ratio Filtering. This is a re-implementation of the length ratio filter from Moses, a standard statistical machine translation toolkit. For each translation pair, the pipeline computes the ratio of source length (in characters or words) to target length. Pairs where this ratio falls outside an acceptable range are discarded.

The motivation: in normal translation, source and target texts have roughly similar lengths — a 100-character English sentence typically translates to somewhere between 70 and 130 characters in another European language, depending on the language pair. If the ratio is extreme (e.g., the translation is 10× longer than the source, or 10× shorter), it signals either a hallucinated translation (the model generated much more text than the source contained), a truncated translation (the model stopped early), or a language mismatch (the model translated into the wrong language entirely).

The paper does not specify the exact ratio thresholds, stating that they follow the standard Moses implementation.

Step 2 — Character Histogram Filtering. Introduced by Fan et al. (2021) for multilingual machine translation, this filter examines the distribution of characters in the translation. The paper references this work but does not elaborate on the specific histogram criteria. The general principle is that legitimate text in a given language has a characteristic distribution of characters — for English, this includes a high frequency of letters 'e', 't', 'a', and low frequencies of characters from other scripts. Translations with anomalous character distributions (e.g., unusually high frequency of punctuation, blocks of repeated characters, mixture of scripts) are filtered out.

Step 3 — FastText Language ID Verification. The pipeline runs FastText (Joulin et al., 2016), a lightweight text classification model, on the English translation to verify that the output is actually English. This is analogous to the language ID verification in the ASR pipeline — if EuroLLM-9B generated output in the wrong language (e.g., leaving the source untranslated, or translating into French instead of English), FastText detects this and the pair is discarded.

The choice of FastText over a heavier model is motivated by speed — FastText can classify millions of sentences per second on GPU, making it suitable for pipeline-scale processing.

Step 4 — Quality Estimation Filtration. This is the most sophisticated filtration step. The pipeline runs cometoid-wmt23 (Gowda et al., 2023), a quality estimation (QE) model, on each translation pair. Unlike the previous filters, which check for mechanical anomalies, QE models are trained to predict the quality of a translation without access to a reference translation — they estimate how "good" the translation is based on the source text and the machine-generated translation alone.

The cometoid-wmt23 model is a distilled version of the COMET metric (a widely used neural metric for translation quality), trained specifically for quality estimation — predicting translation quality scores without a reference. The paper accesses it through the PyMarian interface (Gowda et al., 2024). Translations with low QE scores (indicating poor quality) are filtered out.

The paper does not specify the QE score threshold. In practice, the threshold is likely chosen to balance the retention rate against the desired quality level, and may be tuned per language pair.

Step 5 — Integration. The filtration steps are applied sequentially — a translation pair must pass all five filters to be retained in the final dataset. The paper emphasizes the efficiency of this pipeline: the entire MOSEL dataset (millions of translation pairs) can be processed in 47 minutes on 64 A100 GPUs. This speed is achieved through GPU acceleration in NeMo-Curator and the use of relatively lightweight models (FastText, cometoid) rather than large LLMs for the filtration steps.

Why a separate AST filtration pipeline? The ASR filtration pipeline (hallucination detection, character rate, character set) operates on transcripts in the source language and is concerned with transcription quality. The AST filtration pipeline operates on translation pairs and is concerned with translation quality — a transcript that is perfectly correct as ASR output may still produce a poor translation, and vice versa. The separation reflects the fact that translation introduces a new failure mode (mistranslation, hallucinated translation) that is orthogonal to the failure modes in transcription.

4. Key Insights and Innovations

Innovation 1: Aggressive, Failure-Mode-Specific Filtering Beats Volume Maximization for Pseudo-Labeled Speech Data

The paper's most intellectually distinctive contribution is not any individual filtering technique, but the counter-intuitive empirical finding that deliberately discarding roughly 40% of your training data can produce better models than keeping everything. This inverts the dominant assumption in pseudo-labeling — visible in MOSEL's decision to publish labels for nearly all available audio — that maximizing data volume is the primary objective and that filtering should be conservative to avoid wasting potentially useful examples.

What makes this finding genuinely novel rather than an obvious "quality over quantity" platitude is the specificity of the filtration. Granary does not apply a generic quality score and discard the lowest-ranked samples; it targets four distinct, empirically diagnosed failure modes — truncation, hallucination, language misidentification, and missing punctuation — each addressed by a different technical intervention calibrated to that failure mode. The hallucination filter detects repeated n-grams, implausibly long words, and language-specific hallucinated phrases compiled by systematically examining Whisper's outputs. The character rate filter rejects samples where the text-to-duration ratio is anomalous. The language ID filter discards samples where Whisper's predicted language conflicts with the corpus metadata. The punctuation restoration step adds structure that Whisper's training objective deliberately removed.

This is not a refinement of an existing approach; it is a fundamental re-framing of the pseudo-labeling problem. Prior work implicitly treated pseudo-labeling as a coverage problem — how much of the available audio can we transcribe? Granary re-frames it as a precision problem — how much of the transcribed audio is trustworthy enough to serve as a training target? The 60.7% retention rate is not a failure to process more data; it is the deliberate outcome of a design philosophy that treats filtration as the primary value-add rather than a post-processing cleanup step.

The evidence anchoring this claim is Tables 2 and 3: a FastConformer-L model trained on 14,000 Granary hours matches or slightly outperforms one trained on 23,500 MOSEL hours (for English), and the same pattern holds for Croatian, a lower-resource language. The roughly 10% relative improvement on FLEURS — the most reliable test set — demonstrates that the quality gain is real and not an artifact of test set overlap. The fact that this holds across both a high-resource language (English) and a low-resource one (Croatian) suggests the finding generalizes beyond the specific language or data volume.

Innovation 2: LLM-Based Punctuation Restoration as a Data Quality Intervention, with a Novel Quality Gate

While punctuation restoration itself is not novel — it is a standard post-processing step in ASR — the paper's use of an LLM for this purpose, combined with the 5% character error rate (CER) quality gate, represents an intellectually interesting design choice that reframes punctuation restoration from a cosmetic enhancement to a data quality intervention.

The standard approach in prior work (including MOSEL) is to accept Whisper's lowercased, unpunctuated output as-is, since punctuation is not needed for word error rate computation and adding it risks introducing errors. Granary challenges this assumption by arguing that punctuation and capitalization are training signal — a model trained on properly formatted text learns to produce properly formatted text, which matters for downstream applications like subtitle generation and meeting transcription. The restoration step is thus a deliberate bet that the benefit of having punctuation in the training data outweighs the risk of the LLM introducing errors during restoration.

What elevates this from an engineering detail to an intellectual contribution is the CER quality gate. Rather than trusting the LLM's output unconditionally or discarding it entirely, the pipeline computes the edit distance between the LLM's punctuated output and the original Whisper transcript, retaining the LLM's version only if it changed less than 5% of characters. This is a principled tradeoff mechanism: it allows the LLM to add punctuation and capitalization (which typically affect 1–3% of characters) while catching cases where the LLM rephrases, hallucinates, or deletes content (which produce larger CER values). The 5% threshold is not theoretically derived, but the concept — using a lightweight quality metric to gate a more powerful but less reliable model — is a transferable design pattern for any pipeline where an LLM edits machine-generated text.

The paper's candid acknowledgment that "the quality of these modifications remains a subject for further testing" is itself a contribution: it flags an open question about whether LLM-based text normalization helps or hurts downstream training, and provides the infrastructure (the CER-gated pipeline) for others to investigate.

Innovation 3: First Open-Source LLM-Based Pseudo-Labeling Pipeline for Speech Translation, with a Deliberate Model Selection Methodology

The paper identifies a genuine gap in the open-source ecosystem — "open-source projects tackling translation tasks — particularly in speech applications — are exceptionally sparse" — and fills it with a pipeline that uses a single LLM (EuroLLM-9B) to directly translate ASR transcripts into English, bypassing the multi-step NMT pipelines used in prior work.

The intellectual contribution here is less the use of an LLM for translation (which follows Finkelstein et al., 2024's precedent in the text domain) and more the systematic model selection methodology that justifies the choice. The paper benchmarks five candidate models — Alma-13B-R, Qwen-2.5-7B, EuroLLM-1.7B, EuroLLM-9B, and Riva-Megatron Any2Any 1B — and reports not just which model performed best, but why the alternatives failed in ways that are informative for future work. Alma-13B-R "under-performs significantly in speech domain (despite achieving impressive results on WMT test sets)" — a finding that quantifies the domain gap between formal written text benchmarks and spoken language transcripts. Qwen-2.5-7B "suffers from hallucination issues during pseudo-labeling" — a warning that general-purpose LLMs may produce fluent but unfaithful translations when applied to noisy ASR output. These negative results are contributions in their own right, telling future practitioners which models not to use and why.

The deliberate exclusion criteria are equally informative: GPT-4o is excluded for cost reasons (API pricing at scale), TowerInstruct-13B and Aya-23 for licensing reasons (non-commercial restrictions incompatible with open-source release). These are not technical decisions but deployment-relevant design constraints that the paper surfaces explicitly, making the pipeline's choices reproducible by others with similar constraints.

The AST filtration pipeline — length ratio, character histogram, FastText LID, and cometoid-wmt23 quality estimation, all GPU-accelerated in NeMo-Curator — is an engineering contribution (47 minutes to filter the entire MOSEL dataset on 64 A100 GPUs) that demonstrates how quality estimation models, previously used primarily for evaluation, can serve as practical data filtration tools at scale.

Innovation 4: Diagnosis of Whisper's Systematic Failure Modes as a Prerequisite for Effective Pseudo-Labeling

While the individual filtration techniques are not novel in isolation, the paper's systematic diagnostic framework — identifying four specific, recurring failure modes in Whisper's pseudo-labels and mapping each to a targeted intervention — represents an intellectual contribution that generalizes beyond this specific pipeline.

The four failure modes the paper diagnoses are:

Truncation — caused by Whisper's fixed 30-second segmentation window cutting utterances mid-word or mid-sentence. The intervention is alignment-based segmentation (Whisper timestamps or Parakeet TDT) plus Silero VAD padding.
Hallucination — Whisper generating text never spoken, manifesting as repeated n-grams, implausibly long words, or language-specific hallucinated phrases. The intervention is a three-part hallucination detector with hand-compiled phrase lists.
Language misidentification — Whisper predicting the wrong language, leading to garbled or wrong-language output. The intervention is a confidence-thresholded LID filter (discard if lid_prob < 0.8 or if multiple languages detected).
Missing punctuation and capitalization — Whisper's training objective produces lowercased, unpunctuated text. The intervention is LLM-based restoration with a CER quality gate.

What makes this diagnostic framework distinctive is that it is empirically grounded in the specific model being used for pseudo-labeling (Whisper-large-v3) rather than being a generic quality checklist. The paper's compilation of language-specific hallucinated phrases — "Sous-titrage Société Radio-Canada" in French, "Thank you very much" appearing in non-English contexts — is the product of systematic examination of Whisper's failure patterns, not an off-the-shelf filter. This is a transferable methodology: any team using any ASR model for pseudo-labeling could (and, the paper implies, should) perform a similar failure mode analysis on their specific model before designing their filtration pipeline.

The finding that pseudo-labels consistently improved segmentation results over ground-truth transcripts — a non-obvious result that the paper reports from its development experiments — further reinforces the value of this diagnostic approach. It suggests that understanding your pseudo-labeling model's behavior is not just about catching errors; it is about identifying where the model's outputs, even when imperfect, can be productively used as intermediate signals in the pipeline.

This diagnostic framework is a conceptual advance rather than a metric gain — it provides a template for how to think about pseudo-labeling quality that future work can apply to different models, languages, and domains.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training data source for the evaluation experiments is the VoxPopuli subset drawn from MOSEL [6], which provides pseudo-labeled transcriptions alongside hallucination features and language ID predictions generated by Whisper. The authors create a filtered version of these MOSEL transcriptions as a baseline. For Granary, they process the same VoxPopuli audio through their pipeline. Evaluation is conducted on three test sets: VoxPopuli [7] itself, FLEURS [28], and (for English only) Mozilla Common Voice (MCV). Additionally, English models are evaluated on the Hugging Face ASR leaderboard datasets [29], with results reported as "HF-Avg." Evaluations are performed both with and without punctuation and capitalization where applicable, though the paper only reports the standard WER numbers in its tables.
Base model(s). All experiments use the FastConformer encoder [14] coupled with a hybrid RNNT-CTC decoder [27], employing the Large model configuration with 120 million parameters. This model architecture is chosen as a modern, representative ASR architecture that is publicly available through the NeMo framework [30]. The paper uses this single architecture and scale for all comparisons, meaning the analysis measures data quality effects independent of model architecture choices. The choice of a 120M parameter model (rather than a smaller or larger variant) is not explicitly justified but represents a mid-range configuration that can be trained on the available computational budget (64 A100 80GB GPUs).
Metrics. The primary evaluation metric is Word Error Rate (WER), reported as a percentage. WER is the standard metric in ASR evaluation and measures the minimum number of word insertions, deletions, and substitutions needed to transform the model's hypothesized transcript into the reference transcript, normalized by the number of words in the reference. Lower WER indicates better performance. The paper does not report confidence intervals or statistical significance tests for WER differences.
Baselines. The primary baseline is MOSEL [6], which provides Whisper-large-v3 pseudo-labeled transcriptions for VoxPopuli audio. The authors "leveraged this information to create a filtered version of the MOSEL transcriptions for the VoxPopuli data," meaning they apply MOSEL's own published hallucination flags and language ID predictions to filter the MOSEL dataset. This establishes a fair comparison: both MOSEL and Granary are filtered versions of the same underlying VoxPopuli audio, but MOSEL uses its own (less aggressive) filtering while Granary uses the pipeline described in Section 3. No other ASR pseudo-labeling baselines (e.g., raw YODAS captions, unfiltered Whisper outputs) are evaluated, nor are models trained on the full unfiltered datasets.
Generation budget / compute accounting. All models are trained for 80,000 steps with a batch duration of approximately 10 hours per step, using 64 A100 80GB GPUs and a CosineAnnealing scheduler. The maximum learning rate is 1e-3 with 15,000 warmup steps and a constant weight decay of 1e-3. Model training uses the NeMo framework [30] and Lhotse dataset modules [31]. The key resource constraint for the experimental comparison is training data volume, not compute — the central question is whether a model trained on Granary's smaller-but-cleaner dataset (14,000 hours for English) matches a model trained on MOSEL's larger-but-noisier dataset (23,500 hours for English) given identical training recipes and compute budgets.
Cross-validation / statistical protocol. The paper does not report any cross-validation or statistical significance testing. For the Croatian experiment, where MOSEL published pseudo-labels for only 2,800 out of 8,000 available VoxPopuli hours, the authors "randomly sampled a comparable number of hours from Granary's VoxPopuli dataset in Croatian" to ensure a fair comparison. This random sampling is the only explicit statistical control mentioned. The absence of multiple training runs, standard deviations, or confidence intervals means the reported WER differences cannot be assessed for statistical reliability.

Main Quantitative Results

English ASR: Granary Matches MOSEL with ~40% Less Data

Table 2 presents the core comparison for English ASR. The MOSEL baseline uses 23,500 filtered hours from VoxPopuli; the Granary model uses 14,000 hours from the same underlying audio processed through the Granary pipeline. Both models share the same FastConformer-L architecture and training recipe.

The headline finding is that the Granary-trained model achieves comparable or slightly better WER across all test sets despite using approximately 40% less training data:

On VoxPopuli (the in-domain test set, drawn from the same distribution as the training data), Granary achieves a WER of 11.7% without punctuation/capitalization (PnC) and 13.1% with PnC, compared to MOSEL's 11.8% and 13.2% respectively — essentially identical performance.
On FLEURS, a clean, multi-domain test set considered highly reliable, Granary achieves 9.4% without PnC and 12.1% with PnC, compared to MOSEL's 10.4% and 13.2%. This represents roughly a 10% relative improvement (from 10.4% to 9.4%) — the paper's strongest evidence that Granary's data quality advantage is real.
On MCV (Mozilla Common Voice), Granary achieves 13.0% without PnC and 14.3% with PnC, compared to MOSEL's 13.1% and 14.6% — a marginal improvement.
On HF-Avg (the aggregated Hugging Face ASR leaderboard score), Granary achieves 10.11% without PnC and 11.44% with PnC, compared to MOSEL's 10.22% and 11.49% — a small but consistent advantage.

The pattern across test sets is informative: the largest gains appear on FLEURS, which is the cleanest and most carefully constructed test set, while performance on VoxPopuli (drawn from the same domain as the training data) is essentially tied. This suggests that Granary's filtration produces more generalizable training signal — the model learns patterns that transfer better to clean, diverse evaluation conditions — while MOSEL's larger but noisier dataset may contain spurious correlations that help on in-domain data but hurt on out-of-domain benchmarks.

Croatian ASR: The Pattern Holds for a Lower-Resource Language

Table 3 replicates the comparison for Croatian, using a matched number of hours (approximately 2,800) from both MOSEL and Granary. The Croatian result is particularly important because it tests whether the "less data, better quality" finding generalizes beyond English, where Whisper's pseudo-labeling quality is highest.

The results show the same pattern as English:

On VoxPopuli, Granary achieves 13.7% without PnC and 15.8% with PnC, compared to MOSEL's 17.5% and 19.2% — a substantial improvement of roughly 3.5–4 percentage points absolute.
On FLEURS, Granary achieves 10.4% without PnC and 13.2% with PnC, compared to MOSEL's 11.3% and 14.6% — a smaller but consistent advantage of roughly 1–1.5 percentage points.

The Croatian gains are larger in absolute terms than the English gains, which is consistent with the paper's argument that data quality matters more for lower-resource languages. Whisper's pseudo-labeling accuracy is lower for Croatian than for English, meaning there is more "bad data" for Granary's filtration to remove. The larger improvement on VoxPopuli (in-domain) versus FLEURS (out-of-domain) for Croatian is the opposite of the English pattern, suggesting that for lower-resource languages, the primary benefit is cleaning up the training distribution itself rather than improving out-of-domain generalization.

However, the Croatian experiment has an important limitation: the paper reports only two test sets (VoxPopuli and FLEURS) compared to four for English, and the total data volume (2,800 hours) is much smaller, making the comparison potentially noisier.

Data Retention and Processing Scale

Figure 3 reports the total hours per language after final filtering. English dominates with approximately 275,000 ASR hours, followed by German, French, and Spanish in the 50,000–100,000 hour range. Ukrainian has the least data with 932.67 ASR hours and 608.80 AST hours. Table 1 documents the overall processing scale: approximately 1 million input hours yielded 638,144 ASR hours (60.7% retention) and 351,048 AST hours. The retention rate varies by corpus, though the paper does not provide corpus-specific breakdowns beyond noting that YODAS has particularly low retention for some languages (e.g., 20% for Bulgarian and Ukrainian due to LID errors).

Ablation Studies and Robustness Checks

Segmentation method choice: The paper reports an ablation in the segmentation stage (Section 3.1.1) where the authors "experimented with multiple alignment methods, including VAD, NeMo Forced Alignment (NFA), Time-Duration-Transducer (TDT) decoder and Whisper timestamps." The key finding is that "when evaluating segmentation methods, we found no significant differences in final model performance." This is a negative result — the specific choice of segmentation algorithm (Whisper timestamps vs. Parakeet TDT) does not measurably affect downstream WER despite the paper's detailed discussion of their differences. The practical implication is that the choice was made on operational grounds (speed, availability, quality of word-level alignment) rather than because it materially changes model quality. This finding is not reported in a formal table but described in the text of Section 3.1.1.

Pseudo-labels vs. ground truth for segmentation: The paper reports an additional finding from the segmentation experiments: "Using ASR models (Parakeet & Whisper) for timestamp generation, we compared ground truth and intermediate transcripts, finding that pseudo-labels consistently improved segmentation results." This is the evidence that justifies using pseudo-labels (which are available for all audio) rather than ground truth (which is only available for a subset) as the basis for segmentation. The finding implies that pseudo-labels, even imperfect ones, provide a more consistent alignment signal than ground truth transcripts, possibly because ground truth may contain formatting, punctuation, or normalization differences that interfere with forced alignment. This is a practically important result — it means the pipeline does not need human-annotated transcripts for the segmentation step — but the paper does not quantify the magnitude of the improvement or report it in a formal comparison table.

Two-pass vs. single-pass ASR inference: The paper follows MOSEL's two-pass protocol (first pass for language ID, second pass for transcription conditioned on predicted language) but does not report an ablation comparing this to single-pass inference. The effectiveness of two-pass inference is treated as established by prior work (MOSEL) rather than experimentally verified in this paper.

Whisper-large-v3 vs. Whisper-turbo: The paper mentions that Whisper's turbo variant "is prone to hallucinations" and implies that this motivated the choice of Whisper-large-v3. However, no direct comparison of pseudo-labeling quality between the two variants is reported. The hallucination phrase lists were compiled from both variants, but their relative hallucination rates on this data are not quantified.

LLM-based punctuation restoration quality gate: The paper reports setting the CER threshold at 5% for deciding whether to retain Qwen's punctuated output or fall back to the original Whisper transcript, but does not ablate this threshold. The paper explicitly states that "the quality of these modifications remains a subject for further testing," acknowledging that the effectiveness of LLM-based punctuation restoration for downstream ASR training is not experimentally validated in this work. This is a meaningful gap — the downstream WER tables report results both with and without punctuation/capitalization (PnC), but these reflect evaluation-time normalization choices, not whether punctuation restoration during training helped or hurt model quality.

AST model selection: The paper reports benchmarking multiple translation models (Alma-13B-R, Qwen-2.5-7B, EuroLLM-1.7B, EuroLLM-9B, Riva-Megatron Any2Any 1B) on the Flores dataset covering all 24 translation directions. EuroLLM-9B was selected as the best-performing model. However, no downstream AST model training results are reported in this paper — the AST pipeline produces data, and the paper evaluates only ASR models trained on Granary data. The claim that Granary provides high-quality AST data is based on the intrinsic evaluation of the translation model on Flores and the cometoid-wmt23 quality estimation filtration, not on extrinsic evaluation of a speech translation model trained on the resulting data. This is a significant limitation: the paper's title and abstract claim to provide a dataset for "recognition and translation," but only the recognition portion is experimentally validated.

AST filtration ablation: The AST filtration pipeline applies five sequential filters (length ratio, character histogram, FastText LID, quality estimation). The paper does not report any ablation measuring the contribution of each individual filter or the effect of different QE score thresholds on downstream model quality. The 47-minute processing time for the full MOSEL dataset is reported, but the filtration parameters that determine the retention rate are not disclosed.

Granary vs. MOSEL filtering granularity: The comparison in Tables 2 and 3 uses a "filtered version" of MOSEL that applies MOSEL's own published hallucination flags and LID predictions. The paper does not compare against unfiltered MOSEL, which would show the raw quality difference between the two pseudo-labeling approaches before any filtering. This is a reasonable baseline choice (since unfiltered MOSEL would contain known-bad data), but it means the comparison is between two different filtering strategies applied to the same underlying Whisper outputs, not a comparison of the Whisper outputs themselves.

Effect of training data volume on the Granary advantage: The paper reports only a single Granary data point (14,000 hours for English, ~2,800 for Croatian) compared to MOSEL at its full filtered volume (23,500 and ~2,800 hours respectively). There is no learning curve showing how Granary model performance scales with data volume — would Granary-trained models continue to improve if trained on 23,500 hours of Granary-filtered data, or does the aggressive filtering already capture most of the useful signal at 14,000 hours? This missing experiment would help distinguish whether Granary's advantage is due to better data quality (which would scale) or better selection of the most informative subset (which might saturate).

Multi-task and multilingual model training: The paper states in its conclusion that "future work will focus on releasing multi-task, multilingual models trained on the complete Granary corpora." None of the experiments in this paper train on multiple languages simultaneously or on the combined ASR+AST data. The experiments are monolingual (English or Croatian) and ASR-only. This means the paper's claim to enable multi-task and multilingual training — a motivation stated in the abstract — is aspirational rather than demonstrated.

Critical Assessment

The experiments in this paper demonstrate a specific, valuable finding: for monolingual ASR model training using FastConformer-L on VoxPopuli data, Granary's pipeline produces training data that yields comparable or slightly better WER than MOSEL's filtering while using less data. This finding is supported by two language comparisons (English and Croatian) across multiple test sets.

However, there is a meaningful gap between what the experiments demonstrate and what the paper's framing implies. The title and abstract present Granary as a dataset for "recognition and translation across 25 European languages." The experiments validate English monolingual ASR, Croatian monolingual ASR, and nothing else. The following claims from the paper's framing are not experimentally supported:

Unsupported Claim 1: The AST data is high-quality. No speech translation model is trained on Granary's 351,048 hours of X→En AST data. The paper benchmarks translation models on Flores (a text dataset) and applies quality estimation filtering, but never demonstrates that a model trained on this data actually performs well at speech translation. Text translation quality on Flores does not guarantee speech translation quality on ASR-derived transcripts, given the domain gap the paper itself identifies (Alma-13B-R "under-performs significantly in speech domain despite achieving impressive results on WMT test sets"). This is the most significant missing experiment.

Unsupported Claim 2: The approach works across 25 languages. The evaluation covers only 2 of the 25 languages. Croatian was chosen as a low-resource representative, and the results are encouraging, but there is no evidence that the same data quality improvement holds for, say, Maltese (with very little data) or German (with Whisper performance closer to English). The paper's Figure 3 shows dramatically different data volumes per language after filtering — from 275,000 hours for English to 932 hours for Ukrainian — and it is plausible that Granary's filtration effectiveness varies with both Whisper's per-language accuracy and the amount of data available.

Unsupported Claim 3: Granary enables multi-task, multilingual models. No multi-task (ASR+AST) or multilingual model is trained. The paper's conclusion explicitly defers this to future work. The experiments provide evidence that monolingual ASR training benefits from Granary's filtration, but training dynamics for multi-task, multilingual models — where data quality, quantity, and language balance interact — are not explored.

Supported Claim: Aggressive filtering yields better training data than volume maximization, for these two languages. The English results (Table 2) show Granary at 14,000 hours matching MOSEL at 23,500 hours — a 1.7× data efficiency improvement. The Croatian results (Table 3) show a more dramatic improvement (3.5–4 absolute WER points on VoxPopuli) at matched data volume. This is the paper's central contribution and the evidence supports it, with the caveat that "better training data" is measured only through downstream ASR WER and only for monolingual FastConformer-L models.

What would strengthen the paper:

AST model training results: Train a speech translation model (e.g., a cascaded ASR+MT or end-to-end AST model) on Granary's X→En data and compare to a baseline trained on MOSEL transcripts run through the same EuroLLM-9B translation without Granary's filtration. This is the single most important missing experiment for validating the AST contribution.
Multi-language ASR results: Train a single multilingual model on Granary data for all 25 languages (or a representative subset) and compare to the same model trained on MOSEL data. This would validate the claim that Granary's quality improvements hold across many languages simultaneously.
Intermediate data volume points: Show WER as a function of training hours for both Granary and MOSEL to demonstrate whether Granary's advantage is consistent across data scales or saturates. The current comparison at exactly two points (14k vs. 23.5k for English, ~2.8k vs. ~2.8k for Croatian) leaves open the question of whether MOSEL would match Granary if trained on the same number of hours, or whether Granary would surpass MOSEL if given the same volume.
Multiple random seeds or standard deviations: The WER differences between Granary and MOSEL on English are small (e.g., 9.4% vs. 10.4% on FLEURS). Without any measure of variance, it is impossible to assess whether this 1 percentage point difference is statistically reliable or within the noise of training stochasticity.
Unfiltered baselines: Showing WER for models trained on completely unfiltered YODAS, YTC, or MOSEL data would contextualize how much of the improvement comes from any filtering (MOSEL's or Granary's) versus the specific Granary filtering choices. Currently, we know Granary outperforms MOSEL's filtering, but not the margin over raw pseudo-labels.

Genuine weaknesses that are not acknowledged:

Test set size and domain coverage: The paper uses standard test sets (VoxPopuli, FLEURS, MCV, HF leaderboard), but these are all relatively clean, read or prepared speech. No evaluation is reported on spontaneous, conversational, or noisy speech, which is where pseudo-labeling quality issues (hallucination, truncation) would most affect training. If Granary's filtration is removing precisely the challenging, noisy audio segments where Whisper struggles, the trained model may be worse — not better — on exactly the real-world conditions where ASR is hardest. This is not tested.
The pipeline's own cost is not accounted for: The paper emphasizes efficiency (47 minutes to filter MOSEL AST data) but does not factor the pipeline's computation cost into the data efficiency comparison. If Granary's pipeline requires significant GPU-hours for Whisper inference, Silero VAD, Qwen punctuation restoration, EuroLLM translation, and cometoid quality estimation, then the "50% less training data" claim should be weighed against the pipeline's processing cost. For a fair comparison, one could ask: given the same total compute budget (pipeline processing + model training), does Granary outperform simply training longer on MOSEL data?
The pipeline is validated only on data it was designed for: Granary's filtration was developed by examining Whisper-large-v3's failure modes on these specific corpora. The paper does not test whether the same pipeline (same hallucination phrase lists, same character rate thresholds, same LID confidence cutoff) transfers to a new, unseen corpus or a different ASR model for pseudo-labeling. This limits the claimed generality of the approach.

In summary, the experiments convincingly demonstrate that Granary's aggressive, failure-mode-specific filtering produces better monolingual ASR training data than MOSEL's lighter filtering, for FastConformer-L models on English and Croatian VoxPopuli data. The paper's broader claims — about AST quality, multi-language coverage, and multi-task training — remain unvalidated, and several important experimental dimensions (statistical reliability, spontaneous speech evaluation, pipeline cost amortization, filter transferability) are unexplored.

6. Limitations and Trade-offs

AST Data Quality Is Not Experimentally Validated

The paper's title and abstract position Granary as a dataset for both "recognition and translation," and the pipeline produces 351,048 hours of X→En AST translation pairs. However, no speech translation model is trained or evaluated on this data. The only validation of the AST pipeline is intrinsic: EuroLLM-9B was benchmarked on the Flores text translation dataset to select it over alternatives (Section 3.2.1), and the cometoid-wmt23 quality estimation model is used to filter translation pairs (Section 3.2.3). Neither of these validates that a speech translation model trained on Granary's pseudo-labeled AST pairs actually produces good translations.

This is a consequential gap because the paper itself provides evidence that text-domain translation performance does not transfer to the speech domain. During model selection, the authors report that Alma-13B-R "under-performs significantly in speech domain (despite achieving impressive results on WMT test sets)" — a finding that demonstrates the domain gap between formal written text benchmarks and ASR-derived transcripts. The same domain gap could affect EuroLLM-9B's translation quality on Granary's transcripts, and the cometoid-wmt23 quality estimation model was itself trained on text-domain data. Neither the Flores benchmark nor the QE filtration guarantees that the resulting translation pairs are suitable for training speech translation models.

The consequence is that a practitioner downloading Granary for speech translation has no evidence that training on this data will produce a working AST system. The data could contain systematic translation errors that are invisible to text-domain quality metrics — for instance, EuroLLM-9B might produce fluent English translations that are semantically unfaithful to the source when the ASR transcript contains disfluencies, colloquialisms, or Whisper errors. If such errors are common, a model trained on these pairs would learn to produce translations that sound natural but are incorrect.

What evidence exists in the paper: None. Section 4 ("Model Training and Evaluation") reports only ASR results. The AST data is described in Section 3.2 and its volume is reported in Table 1 and Figure 3, but no downstream AST model evaluation exists. This is the single largest gap between the paper's claims and its experimental evidence.

Mitigation status: The paper does not acknowledge this as a limitation. The conclusion states that "future work will focus on releasing multi-task, multilingual models trained on the complete Granary corpora," implying AST evaluation will come later, but the current paper makes claims about the AST data's value without supporting evidence. This limitation is fully unaddressed in the present work.

The Pipeline's Computational Cost Is Not Amortized in Efficiency Claims

The paper's headline finding is that models trained on Granary data achieve comparable performance to those trained on MOSEL data while using approximately 50% less training data — 14,000 Granary hours match 23,500 MOSEL hours for English (Table 2). However, this comparison does not account for the computational cost of running the Granary pipeline itself, which includes: Whisper-large-v3 two-pass inference on all audio (via FasterWhisper with beam size 5), Silero VAD processing, LLM-based punctuation restoration (Qwen 2.5-7B-Instruct), EuroLLM-9B translation (via vLLM), and cometoid-wmt23 quality estimation (via PyMarian). The paper reports that AST filtration alone takes 47 minutes on 64 A100 GPUs for the MOSEL dataset, but provides no total pipeline cost across all stages and corpora.

The consequence is that the claimed data efficiency is an incomplete accounting. A fair comparison would amortize pipeline processing FLOPs against training FLOPs: if Granary's pipeline costs X GPU-hours to produce 14,000 filtered hours of data, and MOSEL's lighter filtering costs Y GPU-hours to produce 23,500 hours, then the total compute (pipeline + training) should be compared, not just the training data volume. It is possible that Granary's total compute advantage is smaller than the 1.7× data efficiency suggests, or even reversed if the pipeline costs dominate. For a practitioner deciding whether to adopt this pipeline versus simply training longer on MOSEL data, the total cost of ownership is what matters, and the paper does not provide it.

Additionally, the 47-minute figure applies only to the AST filtration step on the MOSEL subset. The full pipeline includes: (1) audio format conversion for ~1 million hours of raw audio, (2) segmentation with Whisper or Parakeet TDT timestamp generation, (3) two-pass Whisper-large-v3 inference with Silero VAD integration, (4) hallucination detection with language-specific phrase matching, (5) character rate and character set filtering, (6) Qwen 2.5-7B-Instruct punctuation restoration, (7) EuroLLM-9B inference for translation, and (8) the AST filtration pipeline (five steps). Steps 3, 6, and 7 involve running large neural models and are likely to dominate the cost scale. Without reporting these costs, the pipeline's practicality at web scale is asserted but not demonstrated.

What evidence exists in the paper: Only the 47-minute figure for AST filtration on MOSEL (Section 3.2.3). No other pipeline stage costs are reported. The paper states the pipeline is "designed for efficiency" and "processes vast amount of data within hours" (abstract), but does not quantify "vast amount" or "within hours" with specific throughput numbers, GPU counts, or total FLOPs.

Mitigation status: The paper does not acknowledge the omission of pipeline cost accounting. The efficiency claims focus exclusively on training data reduction, not total compute. This is an important practical consideration that remains unaddressed.

Validation Is Limited to Two Languages and One Model Architecture

The paper's experimental validation covers 2 of the 25 languages in the dataset — English (high-resource) and Croatian (low-resource) — and uses a single model architecture (FastConformer-L, 120M parameters) with a single training recipe (80,000 steps, batch duration ~10 hours). This means the paper's central finding — that Granary's filtration produces better training data than MOSEL's — is established for exactly one architecture and two data points on the resource spectrum.

The consequence is uncertainty about generalization along several dimensions:

Across languages: Figure 3 shows dramatic variation in data volume after filtering — from 275,000 hours for English to 932 hours for Ukrainian. Whisper-large-v3's pseudo-labeling accuracy varies substantially by language (the paper acknowledges this in Section 3.1, noting "Whisper exhibits reduced accuracy in low-resource languages"). It is plausible that Granary's filtration effectiveness varies with Whisper's per-language accuracy: for languages where Whisper is already accurate (e.g., German, French), aggressive filtering may discard useful data with only marginal quality gains, while for languages where Whisper is error-prone (e.g., Maltese, Lithuanian), the filtration may be more beneficial but may also leave too little data to train a viable model. The Croatian result provides one data point suggesting the approach works for at least one lower-resource language, but Croatian is not the lowest-resource language in the set (Ukrainian, with 932 ASR hours, may behave differently).

Across model architectures and scales: The FastConformer-L (120M parameters) is a mid-sized model. The paper does not test whether Granary's data quality advantage persists for smaller models (where data quantity might matter more relative to quality) or larger models (where data quality issues might be amplified by the model's greater capacity to memorize noisy labels). The choice of a hybrid RNNT-CTC decoder is specific to the NeMo ecosystem; practitioners using other architectures (e.g., CTC-only, attention-based encoder-decoder, or fully autoregressive models) cannot infer whether Granary's advantage transfers.

Across training recipes: All models are trained for exactly 80,000 steps with fixed hyperparameters. If MOSEL-trained models benefit more from longer training (because the larger dataset provides more unique examples), or if Granary-trained models saturate earlier (because the smaller dataset is exhausted faster), the fixed training budget could bias the comparison. The paper does not provide learning curves showing WER as a function of training steps or data volume for either dataset.

What evidence exists in the paper: Tables 2 and 3 (English and Croatian ASR), plus the segmentation ablation reported in text (Section 3.1.1). No other languages, architectures, or training recipes are evaluated. The Croatian experiment uses a matched data volume (~2,800 hours), providing a comparison at equal scale but leaving open the question of whether Granary's filtration would outperform MOSEL if MOSEL were also restricted to the same number of hours, or whether Granary would outperform MOSEL if given the full 8,000 hours of Croatian VoxPopuli audio.

Mitigation status: The paper does not claim broader validation than it provides — the experiments are clearly described as covering English and Croatian. However, the title ("25 European Languages"), abstract, and Figure 3 (which prominently displays hour counts for all 25 languages) create an expectation of broader validation that the experiments do not fulfill. The conclusion's deferral of "multi-task, multilingual models trained on the complete Granary corpora" to future work implicitly acknowledges this gap.

Filtration Removes Challenging Audio That May Be Important for Robustness

Granary's filtration pipeline is designed to remove data points where Whisper produces low-quality pseudo-labels. The three-category filtration system (Section 3.1.4) removes samples with hallucinations, anomalous character rates, or invalid characters — all indicators that Whisper struggled with the audio. However, the audio that causes Whisper to struggle — noisy recordings, overlapping speakers, accented speech, fast speech, spontaneous disfluencies — is precisely the audio that is most important for training robust ASR models. By filtering out these challenging samples, Granary may produce a cleaner dataset that trains models with better WER on clean test sets (like FLEURS) while making them worse on the noisy, spontaneous, or conversational speech that real-world ASR systems encounter.

This is a fundamental tradeoff in data curation: filtering removes noise from the labels (improving training signal quality) but may also remove diversity from the input distribution (narrowing the range of acoustic conditions the model learns to handle). The paper provides evidence that the label-quality gains dominate for the test sets evaluated — Granary outperforms MOSEL on FLEURS (Table 2), which is the cleanest test set — but the evaluation does not include any test set that systematically varies acoustic difficulty (noise level, speaking rate, speaker overlap, conversational style).

The risk is that a practitioner training on Granary data for a production ASR system serving diverse real-world audio would find that the model performs well on clean, read speech (parliamentary recordings, audiobooks — the domains of VoxPopuli and LibriLight) but degrades sharply on the kinds of audio that were filtered out during dataset creation. The paper's character rate filter, for instance, removes samples with "anomalously low or high character rates," which could include fast speakers (who produce more characters per second than the typical rate, triggering the high-character-rate filter) or speakers in noisy environments (where VAD may capture silence, triggering the low-character-rate filter). If these filters systematically remove speakers with certain acoustic characteristics, the resulting model will have blind spots for those characteristics.

What evidence exists in the paper: None. All evaluation test sets (VoxPopuli, FLEURS, MCV, HF leaderboard) consist of relatively clean, prepared speech. VoxPopuli is European Parliament recordings (formal, read or semi-prepared speech by professional speakers in controlled acoustic conditions). FLEURS is read speech recorded in quiet environments. MCV consists of volunteer-read sentences, also typically in quiet conditions. The HF leaderboard aggregates similar standard benchmarks. No evaluation on spontaneous conversational speech, telephone-quality audio, or systematically degraded audio is reported.

Mitigation status: The paper does not acknowledge this tradeoff. The filtration criteria are justified entirely by their effect on pseudo-label quality (removing samples where Whisper's output is unreliable), without discussion of potential impacts on input distribution coverage. The FLEURS improvement suggests the approach works for clean speech generalization, but the robustness question is entirely open.

The Punctuation Restoration Quality Gate Is Untested for Downstream Impact

The LLM-based punctuation and capitalization restoration step (Section 3.1.5) uses a 5% character error rate (CER) threshold to decide whether to retain Qwen 2.5-7B-Instruct's punctuated output or fall back to Whisper's original lowercased transcript. The paper explicitly states that "the quality of these modifications remains a subject for further testing." This is a candid acknowledgment, but it leaves a significant uncertainty: does training on LLM-punctuated text actually improve the trained ASR model's ability to produce properly formatted output, or does it introduce subtle errors (incorrect punctuation placement, hallucinated words that passed the 5% CER gate) that degrade model quality?

The paper reports WER both with and without punctuation/capitalization (PnC) in Tables 2 and 3, but this reflects evaluation-time normalization choices (whether punctuation and capitalization are stripped before computing WER), not a comparison between models trained with and without the punctuation restoration step. We cannot tell from the reported numbers whether the Granary pipeline's punctuation restoration helped, hurt, or had no effect on model quality — we only know that models trained on Granary data (which includes this step) perform comparably to models trained on MOSEL data (which does not).

The risk for practitioners is two-fold. First, if Qwen occasionally introduces semantic errors that fall within the 5% CER threshold — for instance, changing "I have three cats" to "I have three cars" by "correcting" what it misinterprets as a typo — these errors would become training targets, teaching the ASR model to produce incorrect words. The 5% threshold is generous enough to permit such substitutions. Second, if punctuation restoration provides no benefit for downstream ASR quality, the computational cost of running Qwen 2.5-7B-Instruct on every transcript is pure waste — and this cost is not reported.

What evidence exists in the paper: None. The paper includes this step in the pipeline and describes the CER gate (Section 3.1.5), but no experiment isolates its effect on trained model quality. The explicit caveat that "the quality of these modifications remains a subject for further testing" is the paper's own acknowledgment of this gap.

Mitigation status: The paper identifies the limitation ("remains a subject for further testing") but provides no experimental mitigation. A simple ablation — training two models on Granary data, one with and one without punctuation restoration, and comparing their WER with and without PnC normalization — would resolve this uncertainty but is not reported.

7. Implications and Future Directions

How This Work Changes the Landscape

Granary is not a paradigm shift in speech technology — it does not introduce a new model architecture, a novel training objective, or a fundamentally different approach to speech recognition. Rather, it is best understood as a methodological correction to the dominant assumption in pseudo-labeling that data volume is the primary objective and that filtering should be conservative to avoid discarding potentially useful examples.

The paper's central empirical finding — that a model trained on 14,000 aggressively filtered hours matches or slightly outperforms one trained on 23,500 lightly filtered hours, with a roughly 10% relative improvement on the cleanest test set — is specific to one model architecture (FastConformer-L), two languages (English and Croatian), and one training recipe. But the conceptual reframing it demonstrates has broader implications for how the field approaches pseudo-labeled data curation.

From volume-maximization to precision-targeting. Prior to Granary, the dominant approach to pseudo-labeling speech data — exemplified by MOSEL's release of labels for nearly all available audio — treated the pseudo-labeling problem as one of coverage: can we transcribe as much of the available audio as possible, and let downstream users decide what to keep? Granary inverts this logic. Its pipeline is designed around the question: which transcribed segments are trustworthy enough to serve as training targets? The 60.7% retention rate is not a failure to process more data; it is the deliberate consequence of applying four distinct, failure-mode-specific filters (truncation, hallucination, language misidentification, missing punctuation) each calibrated to a diagnosed weakness in the specific pseudo-labeling model (Whisper-large-v3).

This reframing matters because it changes how practitioners should think about pseudo-labeling pipelines. Rather than building a pipeline that maximizes throughput and writes labels for everything, the Granary approach suggests building a pipeline that diagnoses and filters, where the filtration logic is the primary intellectual contribution — not an afterthought. The paper demonstrates that investing engineering effort in understanding your pseudo-labeling model's specific failure modes (compiling language-specific hallucination phrase lists, tuning character-rate thresholds per language and corpus, setting language-ID confidence cutoffs) yields better training data than simply running the model and keeping most of the output.

Reconciling conflicting intuitions about data quality vs. quantity. The paper resolves a tension that has been implicit in the pseudo-labeling literature but rarely addressed directly. On one hand, the success of large-scale weakly supervised models (Whisper itself was trained on 680,000 hours of weakly labeled web data) suggests that volume can compensate for label noise. On the other hand, the observation that filtered data often outperforms unfiltered data in downstream tasks suggests that quality matters independently of quantity. Granary provides a concrete case study at the intersection of these intuitions: for pseudo-labeled speech data from a specific model (Whisper-large-v3) on specific corpora (European Parliament recordings, audiobooks), aggressive, failure-mode-specific filtering outperforms lighter filtering at the same training compute budget, and the advantage holds across both a high-resource language (English) and a lower-resource one (Croatian).

This is not a universal law — the paper does not claim it generalizes to all models, languages, or domains — but it establishes a pattern that future work can test: when your pseudo-labeling model has systematic, diagnosable failure modes, targeting those failure modes with specific filters is likely more effective than applying generic quality thresholds or keeping everything.

What becomes more attractive as a research direction: Systematic failure mode analysis of pseudo-labeling models, language-specific and model-specific filtering heuristics, quality-gated LLM post-processing of machine-generated text, and quality estimation models repurposed as data filtration tools.

What becomes less attractive: The assumption that maximizing pseudo-labeled data volume is the primary objective, that generic quality scores can substitute for failure-mode-specific filtering, and that pseudo-labeling pipelines are interchangeable plug-and-play components rather than systems that must be tuned to the specific model and data distribution.

A limitation on the reframing's reach: The paper's diagnostic framework is tightly coupled to Whisper-large-v3. The hallucination phrase lists, the character-rate thresholds, and the LID confidence cutoff are all tuned to this specific model's behavior on these specific corpora. The paper does not test transfer to a different pseudo-labeling model (e.g., Whisper-large-v2, a different ASR architecture, or a future Whisper version with different failure modes). This means the specific pipeline is a point solution; the methodology — diagnose your model's failure modes, design targeted filters for each, validate with downstream training — is the transferable contribution, but the paper does not itself demonstrate this transfer.

Follow-Up Research This Work Enables

1. Extrinsic validation of Granary's AST data through downstream speech translation model training. The single most urgent experiment the paper leaves undone is training an actual speech translation model on Granary's 351,048 hours of X→En AST pairs and comparing it to a baseline. A strong follow-up would train a cascaded ASR+MT system (using the same FastConformer-L for ASR and a comparable-size NMT model for translation) on Granary data versus on the same ASR transcripts run through EuroLLM-9B without Granary's AST filtration. If Granary's data produces better BLEU or COMET scores, the AST contribution is validated. If not — if the QE filtration and translation model selection don't actually improve downstream translation quality — the paper's claim to provide translation data is undermined despite the careful model selection methodology. This experiment is tractable: the data exists, the models are standard, and the comparison is well-defined.

2. Multilingual and multi-task model training on the full 25-language Granary corpus. The paper evaluates only monolingual ASR for English and Croatian. A natural extension is to train a single multilingual ASR model on all 25 languages (or a representative subset spanning the resource spectrum from English at 275k hours to Ukrainian at 932 hours) and compare to the same architecture trained on equivalently filtered MOSEL data. This would test whether Granary's data quality advantage persists when languages with very different data volumes and Whisper accuracy levels are mixed in training — a scenario where data quality, quantity, and language balance all interact. The paper provides the data; the remaining work is training and evaluation infrastructure.

3. Robustness evaluation on spontaneous and noisy speech. Granary's evaluation uses only clean, prepared-speech test sets (VoxPopuli Parliament recordings, FLEURS read speech, MCV volunteer readings). A critical stress test would evaluate Granary-trained models on spontaneous conversational speech (e.g., Switchboard, CallHome, or a non-English conversational corpus), telephone-bandwidth audio, or systematically noise-degraded speech. The hypothesis to test: Granary's filtration removes audio segments where Whisper produces poor pseudo-labels, which are precisely the challenging acoustic conditions that real-world ASR systems must handle. If Granary-trained models underperform MOSEL-trained models on spontaneous or noisy speech — even while outperforming on clean speech — this would reveal a robustness tradeoff that the paper's current evaluation cannot detect. The experiment requires only standard test sets and the already-trained models.

4. Pipeline cost amortization analysis: total compute vs. training data volume. The paper reports that Granary achieves comparable WER with ~50% less training data, but does not report the computational cost of running the Granary pipeline itself (Whisper inference on ~1M hours, Qwen punctuation restoration, EuroLLM translation, cometoid quality estimation). A thorough follow-up would measure the total GPU-hours required for each pipeline stage and compute the total cost of ownership: pipeline processing + model training for Granary vs. equivalent model training on MOSEL data. If Granary's pipeline costs are comparable to or smaller than the training compute saved, the efficiency claim is strengthened. If the pipeline costs dominate, the practical advice becomes: "Granary produces better data, but at a higher total compute cost; use it when data quality is paramount, not when compute is the binding constraint." This analysis is an engineering measurement task, not a research contribution, but it is essential for practitioners deciding whether to adopt the pipeline.

5. Transfer of the diagnostic methodology to a different pseudo-labeling model. The paper's filtration is tightly coupled to Whisper-large-v3's specific failure modes. A test of the methodology's generality would apply the same diagnostic framework — identify systematic failure modes in the pseudo-labeling model's output, design targeted filters for each, validate with downstream training — to a different ASR model used for pseudo-labeling (e.g., Whisper-large-v2, a future Whisper version, or a non-Whisper architecture like Parakeet). If the resulting pipeline produces data that improves downstream models on its own test sets, the methodology is validated as transferable. If the same hallucination phrase lists, character-rate thresholds, and LID cutoffs fail on a new model — requiring a completely new diagnostic process — the methodology still transfers, but the specific filters are model-specific, which is an important finding about the limits of pipeline reuse.

6. Ablation of individual filtration stages on downstream WER. The paper applies four filtration stages (LID verification, hallucination detection, character rate filtering, character set filtering) plus punctuation restoration, but never measures the contribution of each stage independently. A straightforward ablation — train models on Granary data with each filter disabled in turn, and measure the WER impact — would reveal which filters matter most and which might be redundant. For instance, if character rate filtering and hallucination detection remove largely the same samples, one could be simplified or removed without quality loss. Conversely, if punctuation restoration actually hurts WER (by introducing LLM errors that pass the 5% CER gate), it should be dropped from the pipeline. This experiment requires training multiple models, but each is a standard FastConformer-L run, and the ablation logic is clean.

Practical Applications and Downstream Use Cases

1. Bootstrapping ASR for lower-resource European languages with limited annotation budgets. The Croatian result in Table 3 — where Granary achieves 13.7% WER on VoxPopuli compared to MOSEL's 17.5%, using the same number of training hours — suggests that organizations building ASR for languages where human-annotated data is scarce (Maltese, Lithuanian, Estonian, Latvian, etc.) should prioritize Granary's filtration pipeline over simply downloading and using MOSEL's pseudo-labels. The 3.8 percentage point absolute improvement on VoxPopuli for Croatian, if it generalizes to similarly low-resource languages, could mean the difference between a usable and unusable ASR system in deployment. The practical workflow is: download Granary's pre-processed data for the target language, train a FastConformer-L (or comparable architecture), and evaluate on in-domain test sets. The data is open-source and the training recipe is specified in the paper (80,000 steps, batch duration ~10 hours, CosineAnnealing scheduler, 64 A100 GPUs).

2. Data curation for speech foundation model pretraining. For teams building large, multilingual speech foundation models — where training data quality directly affects downstream task performance — Granary provides a pre-filtered dataset covering 25 languages with consistent processing (same ASR model, same filtration criteria, same punctuation restoration). This is valuable for pretraining because it avoids the heterogeneous label quality that comes from mixing YouTube captions (unreliable), human annotations (expensive, limited coverage), and raw Whisper outputs (hallucinations, truncation). A foundation model trained on Granary data starts from a higher-quality baseline than one trained on unfiltered or lightly filtered web data, potentially requiring less fine-tuning to reach target performance. The data volume (638k ASR hours) is sufficient for pretraining large models, and the language coverage (23 EU languages plus Ukrainian and Russian) provides broad multilingual signal.

3. Low-cost speech translation data generation without human translators. For applications requiring speech-to-text translation (e.g., meeting transcription for multilingual organizations, content localization for European media), Granary's 351k hours of X→En AST pairs provide a zero-human-cost starting point. While the paper does not validate this data through downstream AST training, the pipeline's design — LLM-based translation with quality estimation filtering — follows the pattern established by Finkelstein et al. (2024) for text translation, where LLM-generated parallel data outperformed web-crawled data. An organization needing En→X or X→Y translation data could adapt the pipeline: swap EuroLLM-9B for an LLM supporting the desired translation direction, apply the same QE filtration with cometoid-wmt23 (or a multilingual QE model), and validate with a small human-annotated test set. The GPU-accelerated filtration pipeline in NeMo-Curator makes this feasible at scale.

4. Curriculum learning and data selection research. Because Granary's filtration produces data with explicit quality signals — we know why each sample was retained or discarded (hallucination flag, character rate, LID confidence, CER after punctuation restoration) — it enables research on data ordering and selection strategies that go beyond random shuffling. A researcher could sort training data by quality indicators (samples with high LID confidence, low hallucination risk, character rates near the median for the language) and train models with curriculum learning — starting with the cleanest data, then introducing noisier samples. Alternatively, the filtered-out data (the ~40% that Granary discards) could be used for robustness training: after the model converges on clean data, fine-tune on the noisy data with a lower learning rate to improve performance on challenging acoustic conditions. These experiments are enabled by Granary's explicit filtration metadata, which MOSEL does not provide at comparable granularity.

When to Prefer This Method

The paper positions Granary against MOSEL as the primary alternative for pseudo-labeled European speech data. Based on the experimental results and acknowledged limitations, the decision criteria are:

Prefer Granary's filtered data over MOSEL's filtered data when:

Training monolingual ASR models for English or Croatian (directly validated), and plausibly for other European languages where Whisper-large-v3's pseudo-labeling accuracy is comparable.
The downstream evaluation domain is clean, prepared, or read speech (FLEURS, parliamentary recordings, audiobooks) — the only domains tested.
The binding constraint is training compute rather than pipeline processing compute — Granary uses ~40% less training data to reach comparable WER, but the pipeline's processing cost is unquantified.
Punctuation and capitalization in ASR output matter for the application (Granary restores them; MOSEL does not).
Data quality transparency matters — Granary's filtration stages produce interpretable rejection reasons, whereas MOSEL provides only Whisper's built-in hallucination flags.

Prefer MOSEL's filtered data (or other alternatives) when:

Training speech translation models — Granary's AST data is unvalidated by downstream training, and MOSEL's ASR transcripts can be independently translated with the translation model of your choice.
Training multilingual or multi-task models — the paper provides no evidence that Granary's data quality advantage persists when languages are mixed in training.
The downstream evaluation domain includes spontaneous, conversational, or noisy speech — Granary's filtration may remove the challenging audio that is most important for robustness, and this is untested.
Pipeline processing cost is a primary concern — Granary's pipeline involves multiple large-model inference steps (Qwen 2.5-7B, EuroLLM-9B, cometoid-wmt23) whose total cost is not reported, while MOSEL's labels are already published and downloadable.
The target language is not among the 25 covered by Granary, or is one of the lowest-resource languages in the set (Ukrainian at 932 ASR hours) where Granary's aggressive filtration may leave too little data to train a viable model — a concern the paper does not explore.