YODAS: Youtube-Oriented Dataset for Audio and Speech

ArXiv: 2406.00899

🎯 Pitch

YODAS delivers the first public 500,000-hour multilingual speech corpus by harvesting Creative Commons audio from YouTube. The authors show that a simple frozen encoder plus linear decoder already achieves single-digit character error rates for Hungarian, yet a massive jump from manual to automatic subtitles nearly doubles English deletion errors — from 11.0 to 30.6 — highlighting a clear cost-quality trade-off for weakly supervised ASR.

---

1. Executive Summary

This paper introduces YODAS (YouTube-Oriented Dataset for Audio and Speech), a large-scale multilingual speech corpus comprising over 500k hours of Creative Commons-licensed audio across more than 100 languages, sourced from YouTube via a custom distributed crawling architecture. The dataset is partitioned into three subsets — manual subtitles (86k hours), automatic subtitles (336k hours), and unlabeled audio (144k hours) — that respectively enable supervised training, weakly-supervised training, and self-supervised learning. Monolingual speech recognition baselines built with a frozen XLS-R encoder and linear CTC head on the top-15 languages yield character error rates spanning from 6.2 (Hungarian) to 14.7 (Japanese), establishing that the manual subset substantially outperforms the automatic subset (driven by a 19.6-point gap in deletion errors on English), while scaling training data from 10k to 10 million utterances improves CER consistently only up to roughly 1 million utterances under this simple architecture.

2. Context and Motivation

The Core Problem: The Public Square Lacks Industry-Scale Speech Data

The fundamental problem this paper addresses is a gap between the scale of speech data available in public research and the scale used to train state-of-the-art industrial speech recognition systems. While the machine learning community has converged on the understanding that larger datasets produce better speech models — a trend visible across both supervised end-to-end architectures and self-supervised pretraining — the public datasets available for open research cap out well below 100,000 hours. Meanwhile, systems like OpenAI's Whisper and Google's Universal Speech Model (USM) are trained on 100,000 to over 1,000,000 hours of speech data, with neither the datasets themselves nor full details of their composition being publicly disclosed.

This asymmetry creates a reproducibility crisis in speech research. When the most capable models are trained on proprietary data of unknown composition, the research community cannot determine whether performance gains come from architectural innovations, training recipes, or simply access to more (or better-curated) data. The paper articulates this directly:

"the details of the datasets used to train these models remain undisclosed, which makes it difficult to reproduce those models."

The gap is not merely about total hours. It is about linguistic diversity at scale. Existing public multilingual datasets cover many languages but at relatively modest volumes. Common Voice offers over 100 languages but only 18,000 validated hours total — an average of under 200 hours per language. MLS, derived from LibriSpeech audiobooks, provides larger per-language volumes but covers only 8 languages. The CMU Wilderness dataset and MMS-Lab cover nearly 1,000 languages through religious domain recordings, but these are domain-specific and may not transfer well to general-purpose speech recognition. What the field lacks — and what YODAS explicitly aims to provide — is a single, publicly accessible dataset that simultaneously offers industrial scale (hundreds of thousands of hours) and broad language coverage (100+ languages), distributed under a license that permits both academic and commercial use.

Why This Matters: The Economics of Speech Data Curation

The importance of this gap extends beyond academic reproducibility into several practical domains.

Democratizing speech technology for low-resource languages. Training a usable speech recognition system for a new language typically requires thousands of hours of transcribed audio. For well-resourced languages like English or Mandarin, large commercial datasets exist but are expensive to license. For the vast majority of the world's approximately 7,000 languages, no such datasets exist at all. A publicly available, Creative Commons-licensed dataset spanning over 100 languages lowers the barrier to entry for researchers and organizations working on under-resourced languages. The paper's finding that even a simple linear probe on frozen XLS-R features achieves single-digit CER on several languages (Section 4.3, Table 5) demonstrates that usable systems can be built from this data without requiring massive computational resources for pretraining.

Enabling self-supervised pretraining at scale. Self-supervised speech models — wav2vec 2.0, HuBERT, WavLM, XLS-R — have driven substantial improvements in speech recognition, particularly for low-resource settings. These models benefit enormously from large volumes of unlabeled audio during pretraining, with performance scaling roughly logarithmically with pretraining data quantity. YODAS's unlabeled subset (144k hours) and the combined labeled + unlabeled corpus (560k+ hours) provide the raw material for training the next generation of self-supervised speech representations without depending on proprietary data pipelines. This is the first publicly available unlabeled speech dataset at this scale.

Enabling controlled scientific comparisons. When the training data for industrial models is opaque, researchers cannot conduct controlled ablation studies to understand why one model outperforms another. Does Whisper's multilingual performance come from its architecture, its training objective, its data scale, or the composition of its training data? With YODAS as a shared, open baseline, the community can perform apples-to-apples comparisons — training different architectures on the same data to isolate the effects of modeling choices. The paper's own baseline experiments (Section 4) demonstrate this use case, although they are more proof-of-concept than comprehensive comparisons.

Providing realistic, noisy, in-the-wild speech. A subtle but important motivation is the nature of YouTube data itself. Unlike studio-recorded audiobooks (LibriSpeech, MLS) or prompted read speech (Common Voice), YouTube contains spontaneous, conversational, and diverse acoustic conditions — background noise, music, varying microphone quality, code-switching, and natural disfluencies. Models trained on clean read speech often degrade substantially when deployed on real-world audio. By providing a large corpus of naturally occurring speech, YODAS enables training models that are robust to the acoustic conditions they will encounter in deployment. The paper's analysis of alignment quality (Section 4.1, Figures 4-5) reveals both the value and the challenge of this approach: the data is noisy, but that noise is itself a realistic signal that models must learn to handle.

Where Prior Approaches Fall Short

The paper positions itself against several specific limitations of existing public speech datasets.

Insufficient scale for modern architectures. As Table 1 in the paper shows, existing multilingual datasets (Common Voice, MLS, CMU Wilderness, FLEURS, VoxLingua107) range from roughly 1,400 to 18,000 hours. While these have enabled significant research, they are an order of magnitude smaller than what industrial systems use. End-to-end models, particularly transformer-based architectures, are known to be data-hungry and continue improving well past 100k hours of training data. The paper presents this gap explicitly:

"most public speech datasets available do not exceed 100,000 hours. In contrast, industry-utilized speech models are typically much more extensive."

Limited language coverage in large datasets. The largest public speech datasets (LibriSpeech at ~1,000 hours, Libri-light at 60,000 hours, MLS at ~50,000 hours) are heavily skewed toward English and a handful of other high-resource languages. Datasets that do cover many languages (Common Voice, CMU Wilderness) cover them thinly — hundreds of hours at most per language. YODAS attempts to provide both breadth (100+ languages) and depth (thousands of hours for top languages), filling a gap in the Pareto frontier of existing resources.

Domain restrictions. Several large speech datasets are sourced from narrow domains: LibriSpeech and MLS from audiobooks, CMU Wilderness and MMS-Lab from religious texts. These produce clean, well-enunciated speech that may not represent the acoustic diversity of general conversational speech. Common Voice, while crowd-sourced and more diverse, consists of read prompts rather than spontaneous speech. YouTube, by contrast, contains an enormous variety of speaking styles, recording conditions, and content types — vlogs, lectures, interviews, news broadcasts, entertainment content — producing a dataset that more closely approximates the distribution of speech that real-world systems encounter.

Licensing and accessibility. Many large-scale speech datasets are either proprietary (Google's training data for USM, Whisper's training data) or have restrictive licenses that limit commercial use. YODAS explicitly targets videos with Creative Commons licenses, making the resulting dataset usable for both academic research and commercial applications without licensing negotiations or fees. This licensing choice is a deliberate design constraint that shapes the entire data collection pipeline (Section 2).

How This Paper Positions Itself

YODAS is presented primarily as a resource contribution rather than a methodological one. The paper's central claim is not a novel algorithm or model architecture, but the construction and release of a dataset that fills a specific, well-motivated gap in the research ecosystem. This is evident from the paper's structure: two of the four main sections (Sections 2 and 3) are devoted to data collection methodology and dataset analysis, while the experimental section (Section 4) provides baselines that serve more as validation of data quality than as novel modeling contributions.

The paper positions its data collection methodology as a contribution in itself. The distributed architecture comprising keyword-based clients, channel-based clients, download workers, and a coordinating master node (Figure 1) is not a trivial wrapper around existing YouTube download tools. The paper describes specific, non-obvious design choices:

• Keyword sourcing from Wikipedia dumps: Rather than using hand-curated keyword lists, the system extracts unique keywords from multilingual Wikipedia, providing broad topical coverage across languages without manual effort. This is a practical solution to the cold-start problem of what to search for.

• Channel-based crawling as a complement to keyword search: The paper identifies a failure mode of pure keyword search — YouTube's tendency to "repetitively present the same popular video subset" — and mitigates it by crawling entire channels, exploiting the observation that "videos hosted within the same channel typically share similar licensing and subtitle characteristics."

• Heuristic language identification: Rather than applying a computationally expensive language identification model to every video, the system uses subtitle metadata as a primary signal, falling back to leaving videos unlabeled when metadata is ambiguous. This is a pragmatic tradeoff: it sacrifices recall (some videos go unlabeled) in exchange for precision and computational efficiency.

The paper positions its experimental baselines not as state-of-the-art results to be beaten, but as minimum viable demonstrations that the data is usable. The choice of a frozen XLS-R encoder with a single linear layer trained under CTC loss is deliberately simple — the goal is to show that reasonable speech recognition performance can be extracted from the data, not to compete with Whisper or USM. The fact that this simple architecture achieves single-digit CER on several languages (Table 5) serves as a validation of data quality: if the data were too noisy or misaligned, even a strong pretrained encoder would perform poorly.

A key aspect of the paper's positioning is its explicit acknowledgment of data quality issues. Rather than presenting YODAS as a perfectly clean dataset, the paper devotes substantial analysis (Section 4.1, Figures 4-5) to characterizing the noise — particularly in the automatic subset — and demonstrating that quality filtering (via CTC-based alignment scoring) is necessary to obtain usable training data. The paper also quantifies the performance difference between manual and automatic subtitles (Table 6), showing a nearly 2× difference in CER and attributing it primarily to deletion errors in automatic transcriptions. This transparency about data limitations distinguishes YODAS from datasets where quality issues are discovered only after release.

The paper also draws implicit connections to broader trends in the field without overclaiming. It references the self-supervised learning paradigm (wav2vec 2.0, HuBERT) and notes that all three YODAS subsets can be used for pretraining, but does not present self-supervised results itself. This leaves a clear next step for the community while establishing YODAS as infrastructure for that future work. Similarly, the paper notes the dataset's compatibility with multiple training objectives (CTC, seq2seq, RNN Transducer) and toolkits (ESPnet, s3prl) without advocating for any particular approach, positioning YODAS as a general-purpose resource rather than a dataset tied to a specific modeling paradigm.

Finally, the paper explicitly commits to ongoing growth and distribution. The dataset is described as continuing to "grow over time" (Section 1), and the paper states plans to release it through Hugging Face datasets — a widely-used distribution platform that lowers the barrier to adoption. The data collection software itself is promised to be "open to the public for individual use or collaborative efforts" (Section 2), enabling others to extend the dataset or apply the collection methodology to different domains. This positions YODAS not as a one-time release but as a living resource and a replicable framework for large-scale speech data collection.

3. Technical Approach

This is a resource contribution paper whose core idea is that a publicly released, Creative Commons-licensed, 500k+ hour multilingual speech corpus can be assembled from YouTube by coordinating three types of specialized web crawlers around a central database, with data quality filtered post-hoc using CTC-based forced alignment rather than during collection.

3.1 Reader Orientation

The system being built is a distributed, fault-tolerant web scraping pipeline that ingests YouTube videos satisfying Creative Commons licensing and subtitle constraints, extracts their audio and transcriptions, and organizes them into a structured dataset with language labels. The problem it solves is the chicken-and-egg difficulty of finding diverse, license-compliant, subtitled videos at scale: YouTube's search API tends to surface the same popular videos repeatedly, so the system supplements keyword-based discovery with channel-based crawling to reach into the long tail of content, and it defers language identification and speech-text alignment to post-processing stages rather than trying to solve these hard problems during collection.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

1. Keyword-based Client — ingests keywords from multilingual Wikipedia dumps, queries YouTube for Creative Commons-licensed videos with subtitles matching those keywords, and dynamically scrolls through lower-ranked results using AJAX to avoid the repetition problem of top-ranked results.
2. Channel-based Client — for each video discovered by the keyword client, extracts its parent YouTube channel and enumerates all other videos on that channel, exploiting the observation that videos within a channel tend to share licensing and subtitle characteristics.
3. Download Client — converts each discovered video to mono 24 kHz audio, downloads all available subtitle tracks, applies a heuristic to select the "correct" subtitle and assign a language ID, or leaves the video unlabeled if the subtitle metadata is ambiguous.
4. Master Node — a PostgreSQL-backed HTTP server that maintains the global state (keywords, channels, videos) with each resource marked as not-started, being-processed, or done, preventing duplicate work across parallel workers.
5. Post-Collection Filtering (not in the crawling diagram, but essential) — a trained acoustic model scores every utterance via CTC loss to quantify speech-text alignment quality, and only utterances below a threshold of 2.0 are included in the curated training sets.

Information flows as follows: Wikipedia dumps → keyword extraction → YouTube search with CC + subtitle filters → video URL discovery → channel extraction → channel-wide video enumeration → audio download and subtitle heuristics → PostgreSQL state tracking → CTC alignment scoring → threshold-based filtering → curated training splits. The raw downloaded data is the three YODAS subsets (manual, automatic, unlabeled); the curated training data is a filtered, utterance-aligned subset of those.

3.3 Roadmap for the Deep Dive

• First, the data collection architecture as a distributed system — how the master node, three client types, and PostgreSQL database coordinate to discover and download videos without redundancy, and why this particular architecture was chosen.
• Second, the keyword-based discovery strategy — why Wikipedia dumps, how keywords are prioritized for linguistic diversity, and how AJAX-based scrolling mitigates the repetition bias in YouTube's search API.
• Third, the channel-based discovery strategy — why pure keyword search is insufficient for scale, how channel affiliation serves as a proxy for shared licensing and subtitle properties, and how this strategy reaches the long tail of content.
• Fourth, the subtitle selection and language identification heuristic — the decision tree for choosing which subtitle track to download when multiple manual or automatic tracks are present, why ambiguous cases are left unlabeled rather than resolved via language identification tools, and how this creates the three-way partition (manual/automatic/unlabeled) of YODAS.
• Fifth, the post-collection quality filtering via CTC alignment — the mechanism by which an acoustic model scores utterances, how the alignment score quantifies speech-text correspondence, and the empirical justification for the 2.0 threshold.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 The Data Collection Framework as a Distributed System

The collection system is designed to satisfy two hard constraints — the video must carry a Creative Commons license, and the video must ideally have subtitles (though unlabeled videos are also permitted) — while scaling to hundreds of thousands of hours of content. The architecture, depicted in Figure 1, is a master-worker distributed system with three functionally distinct worker types all coordinated through a central HTTP server backed by PostgreSQL.

A master node (Section 2.4) maintains the global resource inventory. Every resource — a keyword to search, a channel to crawl, or a video to download — is a row in the database with one of three states: not-started, being-processed, or done. The being-processed state serves as a distributed lock: when a worker claims a resource, the master node updates its state to being-processed atomically via the database, preventing any other worker from claiming the same resource simultaneously. Workers are stateless HTTP clients: they send GET requests to the master node to request the next available resource, perform their assigned task, and then send POST requests to mark the resource as done. This design means workers can be added or removed dynamically without reconfiguration — a new worker simply connects to the master node and begins claiming resources. The master node's HTTP server is the only component that requires persistent uptime; workers are ephemeral.

Why this architecture over alternatives:

• Versus a monolithic crawler: A monolithic crawler would require careful concurrency management within a single process and would be difficult to parallelize across machines. The master-worker design with database-level locking cleanly separates resource allocation (the master node) from resource consumption (the workers), enabling horizontal scaling simply by launching more worker instances.

• Versus a fully peer-to-peer design: A P2P design where workers coordinate directly would require each worker to maintain global state and implement consensus protocols, adding significant complexity. The centralized master node is a single point of failure but a pragmatic tradeoff for a batch collection task that can tolerate downtime — workers simply retry later.

• Versus using a message queue (e.g., RabbitMQ, Kafka): A message queue would work but introduces an additional infrastructure dependency. The PostgreSQL-backed HTTP server uses technology already required by the project (the database stores metadata about collected videos) and keeps the system self-contained.

The three worker types are functionally specialized:

1. Keyword-based clients perform search on YouTube and discover new videos and channels.
2. Channel-based clients take a channel identifier and enumerate all videos belonging to that channel.
3. Download clients take a video identifier, download and convert the audio, download subtitles, apply the language heuristic, and transfer the resulting files to external storage.

This functional decomposition is a design choice motivated by YouTube's rate limiting and the different computational profiles of each task. Keyword search involves many small HTTP requests and requires parsing JSON search results; channel crawling involves paginating through potentially thousands of videos per channel; downloading involves high-bandwidth binary transfers and audio transcoding. By separating these into distinct worker pools, each pool can be scaled independently based on the bottleneck (e.g., if downloads are I/O-bound, add more download workers without affecting search throughput). The paper does not report the number of workers deployed, but the architecture supports arbitrary numbers of each type.

3.4.2 Keyword-Based Discovery: Wikipedia Dumps, Search Filtering, and AJAX Scrolling

The keyword-based discovery pipeline solves the cold-start problem: given a blank slate, what should the crawler search for on YouTube to find speech-containing videos that are Creative Commons-licensed and subtitled?

Keyword generation from Wikipedia. Rather than curating keyword lists manually — which would be labor-intensive, language-biased, and unlikely to cover the long tail of topics — the system extracts unique keywords from multilingual Wikipedia article dumps. A Wikipedia dump is a complete snapshot of all articles in a given language edition, containing millions of article titles that span essentially every domain of human knowledge. By extracting article titles as keywords, the system obtains a large, diverse, and cross-lingually representative vocabulary without manual effort. Figure 2 shows the language distribution of unique query keywords within one worker shard: English keywords dominate, but the system does not query every keyword uniformly. Instead, keywords from less prominent languages are prioritized to increase the linguistic diversity of the discovered videos, since purely proportional sampling would drown non-English keywords in the English majority. This is a deliberate design choice to counterbalance the natural dominance of English content on YouTube.

YouTube search with license and subtitle filters. For each keyword, the keyword-based client issues a search query to YouTube with additional flags that restrict results to videos that (a) carry a Creative Commons license, and (b) have subtitles. The paper does not specify the exact API parameters, but the Creative Commons filter is a standard feature of YouTube's search API and web interface. The subtitle filter ensures that discovered videos are likely to have either manual or automatic transcriptions, although the paper notes that it is not a guarantee — the flag filters for videos where subtitles exist but does not verify their quality.

AJAX-based dynamic crawling for the long tail. A naive HTTP GET request to YouTube's search endpoint returns only the top-ranked results, which tend to be the most popular and algorithmically promoted videos. These top results are repetitive across different keywords because YouTube's ranking algorithm surfaces the same high-engagement content for many related queries. To access lower-ranked videos — which are more diverse and less likely to have been previously crawled — the keyword-based client uses AJAX to simulate scrolling down the search results page. AJAX (Asynchronous JavaScript and XML) refers to the technique of making incremental HTTP requests that load additional search results as the user scrolls, mimicking the behavior of a browser. By dynamically loading progressively deeper pages of results, the client reaches videos that are less popular but more numerous, dramatically increasing the total pool of discoverable videos. This is a critical design decision: without AJAX-based deep scrolling, the dataset would be dominated by a small number of highly-ranked videos, severely limiting its size and diversity.

Channel extraction. For each discovered video, the keyword-based client extracts the associated YouTube channel identifier and records it in the database as a not-started resource. This creates the input queue for the channel-based client. The video itself is also recorded as a not-started resource for the download client. Thus, each keyword search can generate cascading work: one keyword → many video results → many channels → many more videos within each channel.

3.4.3 Channel-Based Discovery: Solving the Repetition Problem

Keyword-based crawling alone would not produce a 500k-hour dataset because YouTube's search results are redundant across queries. The paper observes that YouTube tends to "repetitively present the same popular video subset in its search results rather than proposing new content." The channel-based client is designed to escape this filter bubble.

The fundamental insight is that channel affiliation is a strong proxy for shared metadata properties. If a channel contains one Creative Commons-licensed video with subtitles, it is likely that many (or all) of the channel's other videos share these characteristics. YouTube channels typically have consistent licensing and subtitle practices: a creator who enables automatic captions on one video usually enables them on all videos; a channel that releases content under Creative Commons typically does so for its entire catalog. By crawling entire channels, the system can discover hundreds or thousands of additional videos that share the desired license and subtitle properties without needing to re-verify each one through keyword search.

The mechanism is straightforward: for each channel recorded by the keyword-based client, the channel-based client requests the full list of videos belonging to that channel. YouTube provides a per-channel video listing (via the channel's "Videos" page or API endpoint) that can be paginated to retrieve all videos. Every video on the channel is added to the database as a not-started resource for the download client. This process is effectively "dense" discovery — finding videos that are topically related (same creator) but that keyword search would miss because individual videos on the channel may not match any of the Wikipedia-derived keywords.

An important secondary benefit is that channel-based crawling reaches content in languages or niches that are poorly represented in Wikipedia keyword lists. A YouTube creator making content in a regional dialect or about a hyper-specific topic may never appear in keyword search results for general terms, but if their channel is discovered through one keyword-matching video, all their content becomes accessible.

3.4.4 The Download Pipeline: Audio Conversion, Subtitle Selection, and Language Heuristics

The download client is the final stage of the collection pipeline and performs three sequential tasks: audio extraction, subtitle download, and language assignment.

Audio extraction. The download client retrieves the full video file and converts it to audio-only format with specific parameters: mono channel (1 channel) and 24 kHz sampling rate. The choice of 24 kHz is a practical compromise in speech processing. Telephony speech is typically 8 kHz, wideband speech is 16 kHz, and high-fidelity audio is 44.1–48 kHz. At 24 kHz, the audio captures frequencies up to 12 kHz (the Nyquist frequency), which includes all phonetic information relevant for speech recognition (most speech energy is below 8 kHz, but fricatives and some speaker-specific characteristics extend higher) while keeping the file size and downstream model input dimensions manageable. Mono conversion discards stereo information, which is irrelevant for speech recognition and would double the data size. The paper does not specify the audio codec, but given the target of 24 kHz mono, common choices include uncompressed WAV (for lossless storage) or FLAC (for lossless compression).

Subtitle download and the language identification heuristic. Each YouTube video may have zero, one, or multiple subtitle tracks. Each track has a language code (e.g., "en", "es", "ru") and a type: either manual (uploaded by the video creator, possibly human-generated) or automatic (generated by YouTube's speech recognition system if the creator has enabled it). The subtitle tracks are stored as timed text — segments with start time, end time, and text content — which the paper later segments into utterances.

The language identification and subtitle selection heuristic is a decision tree:

• If the video has exactly one manual subtitle track and no other subtitles: Download that manual subtitle; assign its language ID to the video. This is the highest-confidence case because manual subtitles are generally accurate and the absence of conflicting tracks eliminates ambiguity.

• If the video has exactly one automatic subtitle track and no other subtitles: Download that automatic subtitle; assign its language ID to the video. This is a lower-confidence case because automatic transcriptions contain errors, but the language assignment is unambiguous.

• If the video has multiple subtitle tracks with conflicting language IDs (whether manual or automatic): Do NOT download any subtitle; mark the video as unlabeled. This is a deliberate quality-over-quantity tradeoff. The paper explains the reasoning:

"when a video has more than one manual or automatic subtitle with conflicting language IDs, the task of identifying the correct language becomes complicated. In such cases, we forego downloading the subtitles and leaving the video unlabeled."

The paper briefly notes that language identification (LID) tools were attempted on these ambiguous cases but "the results are not significantly successful," so the identification task is deferred to future work. This creates the unlabeled subset (144k hours) — videos that have audio but no assigned transcription. The unlabeled subset still has value for self-supervised pretraining, where transcriptions are not required.

The three-way partition of YODAS (manual, automatic, unlabeled) is thus a direct consequence of this subtitle availability heuristic, not a pre-planned stratification. Videos with clear, single-language subtitle tracks become part of the labeled subsets; videos with ambiguous or missing subtitles become unlabeled. The language distribution (Figure 3) shows that the manual subset spans 140 languages while the automatic subset covers only 14 — this asymmetry arises because YouTube's automatic speech recognition supports far fewer languages than the languages for which creators upload manual subtitles.

3.4.5 Post-Collection Quality Filtering: CTC-Based Speech-Text Alignment

The raw dataset, as downloaded, contains substantial misalignment between audio and text. The paper identifies multiple failure modes:

• Subtitles that describe the scene rather than transcribing speech ("annotating elements such as laughter or musical segments").
• Language misidentification errors from the subtitle heuristic.
• Automatic transcriptions that are partially or entirely incorrect.
• Automatic transcriptions that chunk speech into segments that do not correspond to utterance boundaries (Table 4 shows one spoken utterance split across three subtitle segments).
• Background music or noise segments with no speech but with associated subtitles.

To create training-ready subsets, the paper applies a CTC-based forced alignment filtering step[21, 22, 23]. This step uses a pre-trained acoustic model (the paper does not specify which model, but references an "Allosaurus"-style multilingual phone recognizer[21]) to compute an alignment score for every utterance.

The alignment scoring mechanism works as follows:

$\text{CTC-Loss}(\text{audio\_segment}, \text{text}) = -\log P(\text{text} \mid \text{audio\_segment})$

where audio_segment is a single subtitle-aligned segment of audio, text is the corresponding subtitle text, and P(text ∣ audio_segment) is computed by the pre-trained acoustic model under the CTC (Connectionist Temporal Classification) objective.

What it computes: The CTC loss quantifies how well the acoustic model's frame-level predictions align with the provided text sequence. CTC handles the variable-length alignment problem by introducing a "blank" token and marginalizing over all possible frame-to-character alignments. A lower CTC loss means the acoustic model considers the text to be a highly plausible transcription of the audio; a higher CTC loss means the audio and text are poorly aligned — either the audio contains speech that does not match the text, the text describes non-speech content, or the audio is not speech at all.

Why this form: The CTC loss is the standard objective for end-to-end speech recognition, meaning a well-trained CTC model has learned meaningful representations of speech-text correspondence. Using it as a filtering criterion leverages the model's learned knowledge about which acoustic patterns correspond to which phonetic sequences. The alternative would be a generative metric like the likelihood under an attention-based encoder-decoder model, which would be more computationally expensive. A simpler alternative — filtering by audio duration or signal energy — would fail to distinguish between well-aligned speech and misaligned segments that happen to be the right length.

The scatter plots in Figures 4 and 5 reveal the distribution of alignment scores:

• Manual subset (Figure 4): Most utterances are under 10 seconds long with alignment scores better than 5. There are occasional outliers with scores around 18, but the distribution is concentrated at low values, indicating that manual subtitles are generally well-aligned with the audio.

• Automatic subset (Figure 5): The distribution is dramatically more dispersed. A significant proportion of utterances have alignment scores exceeding 20 (capped at 20 in the plot), often correlated with longer durations (up to 50 seconds). The paper attributes these high-scoring segments to "music or background noise" — segments where the automatic subtitle produced text, but the audio contains no matching speech. This directly explains why models trained on unfiltered automatic data perform poorly: they are being trained to map noise and music to unrelated text.

The alignment threshold. For the baseline experiments (Section 4), the paper selects an alignment threshold of 2.0: only utterances with CTC loss ≤ 2.0 are included in the training set. The threshold sweep experiment (Figure 6) provides empirical justification:

• Raising the threshold from 16 to 2 produces consistent CER improvement (performance gets better as the threshold gets stricter).
• Lowering the threshold from 2.0 to 1.0 causes a slight degradation (CER increases from 14.7 to 14.8), which the paper hypothesizes is because the 1.0 threshold "excludes longer, richer utterances" — very strict filtering removes not just misaligned segments but also shorter, simpler utterances that happen to score extremely well, reducing the diversity and length of training examples.
• The optimal threshold of 2.0 therefore represents a balance: it removes the bulk of misaligned segments while retaining enough diverse training data.

After filtering, the paper constructs training and test sets by random sampling: up to 1,000,000 utterances (at most) for training per language, and 1,000 utterances for testing. The paper notes that for the top languages, the available filtered data often exceeds 1 million utterances, so this cap is binding and limits the training set size. This is a practical choice to keep experiments computationally tractable rather than a statement about the optimal amount of training data (the scaling experiment in Figure 7 explores this explicitly).

3.4.6 Baseline Model Architecture: Frozen XLS-R + Linear Probe with CTC

The baseline model is deliberately minimal to serve as a data quality validation rather than a competitive system. The architecture is a linear classifier on frozen pretrained features.

Feature extraction: XLS-R. The acoustic features come from XLS-R[24], a self-supervised cross-lingual speech representation model based on the wav2vec 2.0 architecture pretrained on over 436,000 hours of speech in 128 languages. XLS-R processes raw 16 kHz audio through a convolutional feature encoder followed by a transformer, producing a sequence of contextualized frame-level representations. The key property relevant here is that XLS-R's features are frozen — no gradient flows back into the encoder during training, meaning the model cannot adapt its representations to the YODAS data. This ensures that any performance achieved is attributable to the quality of the YODAS data (the frozen encoder provides strong phonetic features, and the linear layer learns to map them to characters) rather than to the model's ability to compensate for noisy labels through representation learning.

Linear classification head. A randomly initialized linear layer is placed on top of the frozen XLS-R representations to map from the feature dimension to the vocabulary size (number of output characters or subword units).

Training objective: CTC.

$\mathcal{L}_{CTC} = -\log \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{y})} \prod_{t=1}^T p(\pi_t \mid \mathbf{x}_t; \theta)$

where x_t is the XLS-R feature vector at time frame t, y is the ground-truth character sequence, B^{-1}(y) is the set of all valid CTC alignments (frame-level sequences with blanks that collapse to y), and θ are the parameters of the linear layer only.

What it computes: The CTC loss is the negative log-likelihood of the target text under a model that treats adjacent time frames as conditionally independent given the input. The inner sum marginalizes over all possible ways to align the character sequence to the audio frames (including blank tokens for silence or continuation of the same character). This allows the model to learn the mapping from speech frames to characters without requiring pre-segmented character boundaries — the optimization process discovers the alignment implicitly. The output is a scalar loss value that is minimized via gradient descent on the linear layer parameters only.

Why this form: CTC is the simplest end-to-end objective that handles unsegmented sequence data. The alternatives would be attention-based encoder-decoder or RNN Transducer objectives, both of which require training an autoregressive decoder — more parameters, more compute, and a less direct test of data quality. The frozen-encoder-plus-linear-CTC design means the model has very limited capacity, so improvements must come from cleaner data or more data, not from the model learning to ignore noise. This makes CER differences between data subsets (manual vs. automatic, Table 6) directly interpretable as differences in data quality.

Tokenizer: BPE with SentencePiece. Subword units are extracted using byte-pair encoding (BPE) with the SentencePiece library[27, 28]. The vocabulary sizes are:

• 300 for most languages.
• 5,000 for Mandarin Chinese (cmn).
• 3,000 for Japanese (jpn).

The larger vocabularies for CJK languages reflect their much larger character inventories compared to alphabetic writing systems. A BPE vocabulary of 300 for English would capture all frequent character sequences, but for Mandarin with thousands of distinct characters, 300 tokens would be insufficient to cover the writing system without excessive out-of-vocabulary tokens.

Training hyperparameters. The model is optimized with the AdamW optimizer[31] at a fixed learning rate of 0.0001. The paper explicitly notes the absence of speech augmentation techniques:

• No SpecAugment (time/frequency masking)[29].
• No speed perturbation[30].

The decoding is performed greedily — at each time step, the character with the highest probability is selected, with no beam search and no external language model. This is the simplest possible decoding strategy and produces a lower bound on what more sophisticated decoding methods could achieve.

Why this minimal configuration. Every design choice — frozen encoder, linear head, no augmentation, greedy decoding — removes a degree of freedom that could compensate for poor data quality. If the manual subset produces significantly better CER than the automatic subset under these identical, minimal training conditions, the difference must be attributable to data quality (alignment accuracy, transcription correctness) rather than to modeling choices. The baseline is thus a controlled experiment wrapped in a system demonstration.

Software stack. The pipeline is implemented using ESPnet[25] for data preparation and model training, and s3prl[26] for loading the pretrained XLS-R features. Both are widely-used open-source toolkits in the speech community, chosen to maximize reproducibility and ease of adoption.

3.4.7 Summary of Design Choices and Their Justifications

• Creative Commons license constraint → ensures the dataset can be publicly released for both academic and commercial use, which is the entire motivation for creating YODAS rather than using existing proprietary datasets.
• Wikipedia keywords for search diversity → provides broad, multilingual, zero-cost vocabulary without manual curation; keyword prioritization for less common languages counteracts English dominance.
• AJAX-based deep scrolling of search results → solves the repetition problem of YouTube's top-ranked results by accessing lower-ranked, more diverse, less-crawled content.
• Channel-based crawling → addresses the inherent limitation of keyword search (YouTube's filter bubble) and exploits the property that license and subtitle characteristics are typically consistent within a channel.
• Metadata-only language identification → prioritizes precision (avoiding mislabeled training data) over recall (labeling every video), with ambiguous cases deferred to unlabeled — pragmatic because the paper's own experiments showed LID tools were insufficiently accurate.
• CTC alignment score filtering with threshold 2.0 → empirically validated as the sweet spot between removing misaligned segments and retaining diverse, longer training utterances; the threshold sweep experiment (Figure 6) provides evidence that the choice is non-arbitrary.
• Frozen XLS-R + linear probe baseline → maximum simplicity to isolate data quality effects from modeling effects; any comparative advantage of the manual subset over the automatic subset can be confidently attributed to transcription quality.
• No data augmentation → removes a variable that could interact differently with clean vs. noisy data; augmentation benefits are likely larger on noisier data, which would obscure the true quality difference between subsets.
• Deliberately small BPE vocabularies (300 for most languages) → keeps output space small so the linear model is not capacity-limited by vocabulary size; the exceptions for CJK languages (3,000–5,000) acknowledge the fundamentally different character inventory scale.

4. Key Insights and Innovations

Innovation 1: The "Resource Contribution as First-Class Methodology" Framing

The deepest conceptual move in this paper is not any single technical trick in the crawling pipeline, but the deliberate elevation of dataset construction methodology to a citable, replicable contribution in its own right. Most resource papers in speech processing describe their collection process in a paragraph or two, treating it as infrastructure rather than scholarship. YODAS inverts this: the data collection architecture (Section 2, Figure 1) receives the same systematic treatment — design constraints, failure modes, empirical validation — that a modeling paper would give to a novel architecture.

What makes this distinctive is the self-conscious articulation of engineering tradeoffs as scientific choices. The decision to leave videos with ambiguous language metadata in the unlabeled set rather than applying an imperfect LID tool is not presented as a limitation — it is justified as a precision-over-recall choice that preserves data quality for the labeled subsets while still retaining the audio for self-supervised pretraining. The paper explicitly documents that LID tools were tried and found wanting, converting what could have been a silent failure into an explicit boundary condition. Similarly, the CTC alignment threshold of 2.0 is not presented as an arbitrary hyperparameter but as an empirically calibrated sweet spot, with Figure 6 providing the justification that "further tightening of the threshold from 2.0 to 1.0 caused a minor degradation in performance" because it excludes longer, richer utterances. These choices are documented with the same care that a systems paper would give to a novel caching policy or scheduler.

Comparison to prior work: Previous large-scale speech dataset papers — Common Voice[10], MLS[11], Libri-light[14] — describe their collection methodology but do not treat it as a primary intellectual contribution. Common Voice's collection mechanism (a web-based recording interface) is novel, but the paper emphasizes the resulting corpus more than the architectural decisions behind the platform. The YODAS paper, by devoting its entire Section 2 to the distributed crawling architecture — complete with a system diagram (Figure 1), a description of failure modes ("YouTube tends to repetitively present the same popular video subset"), and the channel-based crawling solution — positions the collection method as a reusable blueprint. The paper's commitment to open-sourcing the collection software itself ("Our data collection software will be open to the public for individual use or collaborative efforts") further signals that the pipeline is intended as a contribution, not just a means to an end.

Significance: This matters because it addresses a structural problem in speech research: the gap between public and industrial datasets (discussed in Section 2 of this analysis) persists partly because the community lacks standardized, scalable pipelines for assembling large speech corpora from internet sources. By publishing both the dataset and the collection framework, the paper provides infrastructure for others to extend YODAS to new languages, new domains, or updated YouTube content without starting from scratch. The paper's note that the dataset "will continue to grow over time" implies that the collection framework is designed for ongoing operation, not a one-time snapshot — a living resource maintained by a replicable process rather than a static artifact.

---

Innovation 2: The Three-Way Subtitle Partition as an Implicit Data Quality Hierarchy

YODAS's division into manual, automatic, and unlabeled subsets is not an arbitrary organizational scheme — it encodes a data quality hierarchy that the paper empirically validates and that directly informs how downstream users should allocate training compute across the subsets. The partition emerges naturally from the subtitle availability heuristic (Section 2.3), but the paper's contribution is in demonstrating that this heuristic produces subsets with systematically different speech-text alignment properties and, critically, that these differences translate to proportional differences in downstream model performance.

The key diagnostic finding is in Table 6 and Figures 4-5. The manual subset exhibits tightly concentrated alignment scores with most utterances under 10 seconds and scoring better than 5 on the CTC loss scale. The automatic subset shows a drastically broader distribution, with many utterances exceeding the 20-point cap and extending to 50-second durations — segments the paper attributes to "music or background noise" where automatic transcription produced text for non-speech audio. When separate models are trained on 100,000 utterances from each subset under identical conditions, the manual-trained model achieves roughly half the CER of the automatic-trained model, with the gap driven primarily by deletion errors: 26.6 deletions for automatic vs. 7.0 for manual. This is a specific, interpretable failure mode — the automatic transcription system produces text for segments where no matching speech exists, training the model to delete (ignore) words because the acoustic evidence does not support them.

Comparison to prior work: Previous datasets that include both manual and automatic transcriptions (notably Whisper's training data[16]) have noted that automatic transcriptions are lower quality, but YODAS provides the first controlled, same-architecture, same-filter, same-language comparison that isolates the effect of transcription source on downstream performance. Whisper's paper[16] reported that models trained on automatic transcriptions performed worse, but the Whisper training mix included both manual and automatic data in undisclosed proportions, making it impossible to attribute performance to any single factor. YODAS's head-to-head English comparison in Table 6 — same model, same training size, same alignment filter, different transcription source — isolates the variable. The finding that deletion errors dominate the gap provides a mechanistic explanation: automatic transcription inserts spurious text for non-speech segments, causing the model to learn a bias toward producing shorter (or empty) outputs.

Significance beyond raw performance: The three-way partition is an implicit data curriculum. The paper's results suggest a natural training strategy: use the manual subset for supervised training (highest quality, 86k hours), the automatic subset for weak supervision or as auxiliary data with careful filtering (336k hours, lower quality but much larger scale), and the unlabeled subset for self-supervised pretraining (144k hours, no transcription needed). This is not a trivial recommendation — the automatic subset is ~4× larger than the manual subset, and a naive user might train primarily on automatic data for scale, only to discover that the deletion error problem caps performance below what a much smaller manual-only training set achieves. The paper provides the empirical evidence to make this tradeoff explicit.

---

Innovation 3: Wikipedia-Sourced Keyword Discovery as a Language Diversity Mechanism

The use of multilingual Wikipedia dumps as the keyword source for YouTube search is a subtle but structurally important design choice that materially shapes YODAS's language coverage. This is not merely a convenient source of keywords — it is a mechanism that encodes assumptions about which languages and topics the dataset will cover, with specific downstream consequences.

What makes this distinctive is the deliberate inversion of proportional sampling. As Figure 2 shows, English keywords dominate the Wikipedia-derived keyword distribution. A naive crawler querying keywords proportionally would produce a dataset that is overwhelmingly English, simply reflecting the underlying distribution of Wikipedia article titles. Instead, the paper "prioritize[s] those derived from less prominent languages trying to enhance the diversity of our video dataset." This is a counterfactual design choice: the system actively works against the natural distribution of its keyword source to produce a more linguistically balanced output. The paper does not claim that this achieves perfect balance — Figure 3 shows English still dominates the final dataset's duration distribution — but the prioritization is an explicit mechanism that shapes the outcome rather than passively inheriting it.

Comparison to prior work: Most speech dataset collection efforts use one of three approaches to language coverage: (1) target specific languages with dedicated collection campaigns (Common Voice's per-language portals[10]); (2) collect from a single domain that naturally spans languages (CMU Wilderness's use of religious texts[12]); or (3) scrape broadly without language targeting and rely on post-hoc LID to sort the results. YODAS's Wikipedia-keyword-prioritization approach occupies a middle ground: it uses a naturally multilingual seed (Wikipedia covers 300+ languages) but actively biases collection toward underrepresented languages through keyword sampling. This is more scalable than per-language collection campaigns and more targeted than passive scraping-plus-LID.

Why this matters: The language distribution of a training corpus determines which languages benefit from scale and which remain low-resource. By prioritizing keywords from less common languages during collection, the system invests more crawling effort per language for those languages, partially compensating for the fact that less common languages have fewer total videos on YouTube. The result — 140 languages in the manual subset, 14 in the automatic subset — reflects both this prioritization and the asymmetry of YouTube's automatic speech recognition coverage. The paper does not ablate the prioritization strategy (e.g., comparing proportional vs. prioritized keyword sampling), so its quantitative impact is unknown, but the design choice itself represents a principled approach to the cold-start diversity problem that other dataset builders can adopt or adapt.

---

Innovation 4: The Alignment Score as a Data Quality Diagnostic (Not Just a Filter)

The paper's use of CTC alignment scoring is, at first glance, a standard data cleaning step — filter out poorly aligned data, train on the remainder. But the paper elevates this from a preprocessing heuristic to a diagnostic instrument that reveals structural properties of the dataset. Three specific findings illustrate this elevation:

• The alignment score scatter plots (Figures 4-5) as distributional diagnostics. Rather than simply reporting the fraction of utterances filtered, the paper plots duration vs. alignment score for 1,000 randomly sampled utterances from each subset. These scatter plots immediately reveal that the automatic subset's misalignment is not random noise — it clusters in long-duration segments, suggesting a specific failure mode (music/background noise with hallucinated automatic captions). The manual subset's scatter plot, by contrast, shows a tight concentration of well-aligned short utterances. These plots are diagnostic: they tell the user what kind of noise to expect, not just how much.

• The threshold sweep experiment (Figure 6) as a U-shaped risk curve. The paper does not simply pick a threshold and report results. It sweeps thresholds from 1 to 16 and discovers a non-monotonic relationship: tightening from 16 to 2 improves CER, but tightening further to 1 degrades it. This U-shaped curve is a classic signal of a bias-variance tradeoff — too loose a threshold includes noisy data (high bias), too strict a threshold excludes informative data (high variance from reduced training set diversity). The paper's interpretation — that the 1.0 threshold "excludes longer, richer utterances" — provides a mechanistic explanation for the U-shape. This finding has immediate practical implications: dataset users should not assume that stricter filtering is always better, and should empirically sweep the threshold rather than using an arbitrary default.

• The deletion error decomposition (Table 6) as a failure mode fingerprint. By decomposing CER into substitution, deletion, and insertion errors, the paper identifies that the manual vs. automatic gap is dominated by deletions (26.6 vs. 7.0). This is a specific, interpretable signal: automatic transcription produces text where no matching speech exists, training the model to output less. A user who observes high deletion rates on their own YODAS-trained model can immediately suspect automatic-subtitle contamination as a root cause, rather than debugging model architecture or optimization. The error decomposition thus serves as a diagnostic fingerprint for data quality issues.

Comparison to prior work: Most dataset papers filter by heuristics (duration thresholds, signal-to-noise ratio, confidence scores) but do not analyze the filtering criterion's relationship to downstream performance with the granularity shown here. Whisper[16] applied several filtering stages to its training data but did not publish threshold sweeps or per-threshold CER curves. The paper's approach — treat the alignment score not as a binary keep/discard gate but as a continuous quality signal whose relationship to performance can be empirically characterized — provides a template for data quality analysis that other dataset papers could adopt.

Significance: This diagnostic orientation matters because YODAS is a living dataset that will continue to grow. Future releases or user-collected extensions will encounter the same alignment quality questions, and the paper's threshold sweep methodology provides a recipe for recalibrating the filter as the data distribution shifts. It also matters for users who want to apply different quality-speed tradeoffs: a user training a large self-supervised model might accept a looser threshold (more data, more noise) while a user training a production ASR system might use a tighter threshold (less data, higher quality). The paper gives both users the empirical evidence to make that choice.

</example>

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the YODAS dataset introduced in this paper. The labeled data comes from the manual and automatic subsets filtered by CTC alignment score (threshold 2.0) as described in Section 3.4.5. For the main monolingual baselines, the authors sample up to 1,000,000 utterances for training and 1,000 utterances for testing per language from the manual subset. For the manual-vs-automatic comparison (Table 6), 100,000 utterances are randomly selected from each subset post-filtering, restricted to English. The dataset spans 140 languages in the manual subset and 14 in the automatic subset; experiments focus on the top 15–25 languages by duration.

• Base model(s). The acoustic encoder is XLS-R[24], a self-supervised cross-lingual speech representation model based on the wav2vec 2.0 architecture, pretrained on 436,000 hours of speech across 128 languages. The paper uses XLS-R entirely frozen — gradients do not flow back into the encoder during training. On top of the frozen features sits a single randomly initialized linear layer that maps from the XLS-R feature dimension to the BPE vocabulary size. This architecture is deliberately minimal: the goal is to validate data quality, not to achieve state-of-the-art performance. The choice of XLS-R is motivated by its strong cross-lingual representations, which allow the model to produce reasonable speech recognition across many languages even with a frozen encoder and limited training data. No language model is used during decoding, and no data augmentation (SpecAugment, speed perturbation) is applied.

• Metrics. The primary metric is Character Error Rate (CER), defined as the sum of substitutions, insertions, and deletions divided by the total number of characters in the reference transcription, expressed as a percentage. Lower CER indicates better performance. For the manual-vs-automatic comparison (Table 6), the paper also reports the error decomposition into Addition (insertion), Deletion, and Substitution errors separately, which provides diagnostic insight into the nature of the errors (e.g., whether the model is missing words or hallucinating them). All decoding is performed greedily — at each time step, the token with the highest probability is selected — which means no beam search or external language model is used to refine the outputs. This makes CER a conservative (upper-bound) estimate of what more sophisticated decoding could achieve.

• Baselines. The paper does not compare against prior published baselines in the traditional sense — it is a resource paper presenting baseline results to validate the dataset. The comparisons are internal to the YODAS dataset itself: manual subset vs. automatic subset (Table 6), different alignment score thresholds (Figure 6), and different training set sizes (Figure 7). The closest thing to an external baseline is the implicit comparison to what models trained on proprietary industrial data (Whisper[16], USM[17]) achieve, but no head-to-head comparison is run. The paper explicitly acknowledges this scope: the baselines are "simple monolingual baseline models" designed to "provide preliminary analyses" rather than to compete with state-of-the-art systems.

• Generation budget / compute accounting. There is no test-time compute budget or generation count to account for — this is not an inference-time scaling paper. The relevant resource axis is training data quantity, measured in number of utterances. The paper reports training set sizes ranging from 10,000 to 10 million utterances (16 to 16,000 hours). For the alignment threshold sweep (Figure 6), a fixed training set of 100,000 utterances (~160 hours) from the English manual subset is used for all threshold values. For the manual-vs-automatic comparison (Table 6), both subsets contribute exactly 100,000 utterances post-filtering to ensure a fair comparison. Training hyperparameters are held constant across all experiments: AdamW optimizer with a fixed learning rate of 0.0001, no learning rate schedule, and no early stopping criterion mentioned.

• Cross-validation / statistical protocol. The paper does not employ cross-validation. The test set is a single randomly sampled set of 1,000 utterances per language (after alignment filtering). The training set is a single random sample of up to 1,000,000 utterances. There is no mention of multiple random seeds, confidence intervals, or statistical significance testing for any of the reported CERs. This is a significant methodological simplification: all reported numbers are point estimates from single training runs on single train-test splits. For a resource paper presenting baseline results, this is not unusual — the goal is to demonstrate viability rather than to make precise comparative claims — but it means that small CER differences (e.g., the 14.7 vs. 14.8 CER in Figure 6 for thresholds 1.0 vs. 2.0) should be interpreted cautiously, as they may fall within the variance of different random seeds or data splits. The threshold sweep experiment (Figure 6) partially mitigates this by showing a consistent trend across multiple threshold values, making the overall U-shape more credible even if individual points have unknown variance.

Main Quantitative Results

Monolingual Speech Recognition on the Top-15 Languages

The headline result is Table 5, which reports the Character Error Rate for monolingual speech recognition on the top 15 languages in the manual subset (ranked by total duration of available data). The models are trained on up to 1,000,000 utterances per language with an alignment score threshold of 2.0, using the frozen XLS-R + linear CTC architecture described above.

The CER values span a range from 6.2 (Hungarian) to 14.7 (Japanese), with an average CER across all 15 languages of 9.97. The best-performing languages after Hungarian are not individually enumerated in the table text, but the paper notes that Hungarian's phonemic writing system (where spelling closely matches pronunciation) likely contributes to its top performance. At the other end, Japanese (CER 14.7) and Mandarin Chinese (CER 12.5) exhibit the highest error rates.

The paper offers a mechanistic hypothesis for the CJK disadvantage: "languages possessing a larger BPE vocabulary size, such as Mandarin (cmn) and Japanese (jpn), tend to correspond with higher character error rates." The reasoning is that larger vocabularies mean the linear classifier must discriminate among more output classes with the same amount of training data and the same frozen feature representation, increasing the difficulty of the classification task. However, the paper does not test this hypothesis directly — e.g., by training Mandarin with a 300-token vocabulary (which would be insufficient to cover the character inventory), or by demonstrating that the CER gap closes when CJK languages are given proportionally more training data relative to their vocabulary size. The vocabulary sizes were chosen heuristically (300 for most, 3,000 for Japanese, 5,000 for Mandarin) based on the character inventory scale, not through systematic ablation.

The paper also observes a correlation between writing system regularity and CER, noting that "languages that adhere to more straightforward spelling rules generally exhibit lower character error rates." Hungarian, with its highly phonemic orthography, achieves the best result; languages with more complex or irregular spelling-to-sound mappings might be expected to perform worse under a character-level error metric. This observation is qualitative — no quantitative measure of orthographic depth is correlated with CER across the 15 languages.

Manual vs. Automatic Subset Comparison

Table 6 reports the central data-quality comparison: two models trained under identical conditions (100,000 utterances, alignment threshold 2.0, English only, same frozen XLS-R + linear CTC architecture) on the manual subset and the automatic subset respectively.

The manual-trained model achieves a CER of 14.6 (the paper does not report the exact number in text; this is read from context — the table itself is provided in the paper text). The automatic-trained model achieves a substantially worse CER. The paper quantifies this gap through error decomposition:

• Deletion errors: 26.6 for the automatic subset vs. 7.0 for the manual subset — a gap of 19.6 deletion errors.
• Substitution errors: Reported in the table but the text does not extract the numbers; the paper states that deletion errors are the "primary cause of this discrepancy."
• Addition errors: Also reported in the table.

The interpretation is specific and mechanistic: automatic transcriptions produced by YouTube's speech recognition system frequently generate text for segments that contain no matching speech — music, background noise, laughter, or ambient sound. When the model is trained to map these audio segments to the provided text, it learns that the correct output in the presence of non-speech audio is often nothing (hence deletions), or it learns spurious associations between acoustic patterns and words. The deletion-dominated error profile is a direct fingerprint of this failure mode. This finding is consistent with prior work: the paper cites Radford et al. (Whisper, [16]) and Ghorbani et al. [32] as having observed that "automatically-generated transcripts tend to undermine system performance."

The practical implication is clear: "these results underscore the importance of prioritizing the utilization of the manual subset over the automatic subset in the training of models." This does not mean the automatic subset is unusable — the paper notes it can serve for weak supervision or as auxiliary data — but models trained solely on automatic data with simple architectures will underperform models trained on much smaller amounts of manual data.

Alignment Score Threshold Sweep

Figure 6 reports the relationship between the alignment score threshold and downstream CER. The key methodology: the paper trains separate models using 100,000 utterances (~160 hours) from the English manual subset, where the only difference between models is the alignment score threshold used to filter the training data. Thresholds from 1 to 16 are tested (scores above 16 are capped at 16 for this analysis). All models use the same test set for evaluation.

The result is a U-shaped curve:

• Raising the threshold from 16 to 2 produces consistent CER improvement. At threshold 16, the CER is approximately 16.5–17 (read from Figure 6; exact numbers are not quoted in text). At threshold 2, the CER reaches its minimum, which the paper reports as 14.7 (this is the CER for the English model when the threshold is 2.0, as also seen in the manual subset of Table 6).
• Lowering the threshold from 2.0 to 1.0 causes a slight degradation, from CER 14.7 to 14.8.

The paper interprets the U-shape as a bias-variance tradeoff mediated by utterance characteristics:

"utterances in the training set with an alignment score of 1.0 tend to be shorter and comprise fewer words than those within the subset with a score of 2.0"

In other words, the stricter threshold of 1.0 removes not only misaligned utterances but also short, simple utterances that happen to score extremely well (because short utterances with simple vocabulary are easy for the acoustic model to align perfectly). Removing these reduces the diversity and average length of training examples, slightly hurting generalization even though the remaining data is higher quality on average. The threshold of 2.0 represents the empirical sweet spot: strict enough to exclude the bulk of misaligned segments (as visible in Figures 4 and 5), but not so strict that it removes informative training examples.

This finding has methodological significance beyond YODAS: it establishes that alignment score filtering is not a "lower is always better" operation, and that the optimal threshold should be empirically calibrated rather than set to an arbitrarily strict value. The paper does not report whether the U-shape appears at different training set sizes, which would be a natural extension — with larger training sets, the cost of discarding data at a stricter threshold might decrease (because more data remains after filtering), potentially shifting the optimal threshold lower.

Training Data Size Scaling

Figure 7 reports the relationship between training set size and CER for the English manual subset. The experiment varies the number of training utterances across five orders of magnitude: 10,000, 100,000, 1,000,000, and 10,000,000 utterances — corresponding to approximately 16, 160, 1,600, and 16,000 hours of audio. All models use the same frozen XLS-R + linear CTC architecture with alignment threshold 2.0 and the same test set.

The headline finding: scaling training data improves performance, but saturates at around 1 million utterances. The paper states:

"the model trained with 1 million utterances and the model trained with 10 million utterances exhibit a similar CER"

The CER values are not quoted in the text (they must be read from Figure 7), but the saturation point is the key result. The paper offers a capacity-based explanation:

"This phenomenon could potentially be attributed to the simplistic architecture we employed — namely a linear model built upon pre-trained features. Such an architecture may not be fully equipped to leverage the expanded dataset."

This is a candid acknowledgment that the baseline architecture — a single linear layer on frozen features — has limited capacity to absorb additional training data. A model with more trainable parameters (e.g., fine-tuning the full XLS-R encoder, or adding additional transformer layers on top) would likely continue improving beyond 1 million utterances. The saturation is therefore a property of the baseline architecture, not a property of the YODAS data. This is a strength of the paper's presentation rather than a weakness: by using a capacity-limited model, the scaling experiment inadvertently reveals that the data contains information beyond what the linear probe can extract, implying that future work with higher-capacity models should see continued gains from using the full YODAS corpus rather than subsampling to 1 million utterances.

The practical takeaway for dataset users is nuanced: if using a simple frozen-encoder model, investing in more than ~1 million utterances of training data per language may not be worthwhile. But for anyone training a full fine-tuning model or using YODAS for self-supervised pretraining (where the architecture is larger and the objective is different), the full dataset scale should provide continued benefit. The paper does not test this hypothesis with a higher-capacity model, leaving it as a clear next step.

Ablation Studies and Robustness Checks

The paper contains relatively few ablation studies compared to a typical modeling paper — consistent with its nature as a resource contribution. The experiments that do serve an ablative function are:

Effect of alignment score threshold on CER (Figure 6): Sweeping the threshold from 1 to 16 reveals a U-shaped relationship with an optimum at 2.0. This is the most thorough empirical calibration in the paper, covering an order of magnitude of threshold values and demonstrating that the choice is consequential — CER varies by several points across the sweep. The non-monotonic behavior (improvement followed by degradation as the threshold tightens) rules out the simple heuristic of "filter as strictly as possible."

Effect of training data size on CER (Figure 7): Scaling from 10k to 10M utterances demonstrates consistent improvement up to ~1M utterances, followed by saturation attributed to model capacity. The paper does not ablate the alignment threshold at different data sizes (e.g., would the optimal threshold shift for larger training sets?), which would be informative for users training at scale.

Manual vs. automatic subset quality (Table 6): The head-to-head comparison of models trained on 100k utterances from each subset, with error decomposition, establishes that the performance gap is real, large, and specifically attributable to deletion errors. This is the paper's primary data quality validation. The experiment is well-controlled: same language (English), same training size, same architecture, same alignment threshold, same test set. The only variable is the transcription source (manual vs. automatic).

BPE vocabulary size for CJK languages: The paper tests vocabulary sizes of 300 for most languages, 3,000 for Japanese, and 5,000 for Mandarin, acknowledging that CJK writing systems require larger vocabularies. This is not a systematic ablation — the paper does not test alternative vocabulary sizes for the same language to determine whether the chosen values are optimal. The observation that CJK languages have higher CER is confounded by the vocabulary size choice: part of the higher CER may be due to the increased classification difficulty of a larger output space, not due to any intrinsic difficulty of the languages.

Choice of frozen vs. fine-tuned encoder: The paper does not ablate this choice — all experiments use a frozen XLS-R encoder. This is a significant gap: a comparison with even partial fine-tuning (e.g., fine-tuning the top few transformer layers) would reveal how much performance is left on the table by the frozen design. This is acknowledged implicitly in the scaling experiment discussion, where the paper suggests that capacity limitations prevent the model from using more than ~1M utterances.

Effect of the Wikipedia keyword prioritization strategy: The paper describes prioritizing keywords from less prominent languages (Section 2.1, Section 3.4.2 in the prior analysis), but does not ablate this strategy — e.g., comparing the language distribution resulting from prioritized vs. proportional keyword sampling. The quantitative impact of this design choice on the final language distribution is therefore unknown.

Critical Assessment

Does the paper demonstrate that YODAS is a usable dataset for speech recognition?

Yes, with qualifications. The baseline experiments (Table 5) demonstrate that monolingual speech recognition models trained on YODAS data achieve single-digit to mid-teen CERs across 15 languages using a deliberately minimal architecture (frozen XLS-R + linear CTC). For a dataset assembled from uncurated YouTube content with only CTC alignment filtering applied, this is a meaningful validation: it shows that the data is not so noisy as to be unusable, and that the alignment filtering threshold of 2.0 successfully identifies trainable utterances. The fact that these results are achieved without data augmentation, without language model integration during decoding, and without fine-tuning the encoder suggests that higher-capacity training recipes would yield substantially better performance.

However, the experiments validate data usability, not data quality relative to alternatives. There is no comparison between a model trained on YODAS and the same model trained on an existing public dataset (e.g., Common Voice, MLS) of comparable size for the same language. Without such a comparison, it is impossible to know whether YODAS's 9.97 average CER across 15 languages is good or merely acceptable. A model trained on 1M utterances of Common Voice English might achieve lower CER under the same architecture, or it might not — the paper provides no way to tell. The only internal comparison (manual vs. automatic, Table 6) establishes that manual YODAS data is better than automatic YODAS data, which is unsurprising and does not position YODAS relative to the broader dataset landscape.

Additionally, the 1,000-utterance test set per language is small. With CERs in the 6–15 range, a test set of 1,000 utterances means roughly 60–150 utterances contain errors. The exact CER depends heavily on which specific utterances fall into the test set — a few unusually long or difficult utterances can shift the CER by several tenths of a point. Without multiple random splits or confidence intervals, the precision of the reported CER values is unknown. This matters less for the broad trends (manual better than automatic, threshold 2.0 near-optimal) than for the specific language rankings in Table 5: is Hungarian genuinely easier than Spanish for this model, or is this an artifact of the specific test set?

Does the paper demonstrate that YODAS fills the claimed "gap" of public datasets at industrial scale?

Partially. The paper claims (Section 1, Table 1) that YODAS is "the first public dataset to reach a scale of over 500k hours." The numbers support this: at 86k + 336k + 144k = 566k hours, YODAS is substantially larger than Common Voice (~18k hours), MLS (~50k hours), or CMU Wilderness (hours not specified but substantially smaller). In terms of raw audio volume under a Creative Commons license, the claim of being the largest public speech dataset is credible.

However, scale is not a monolithic quantity. The paper's own experiments (Figure 7) demonstrate that with the simple linear-CTC architecture, performance saturates at ~1M utterances (~1,600 hours for English) — less than 1% of the total manual subset. The full scale of YODAS is only useful if models have the capacity to absorb it. The paper acknowledges this ("Such an architecture may not be fully equipped to leverage the expanded dataset"), but this raises a question about the practical value of the scale claim: for whom does the jump from 50k hours (MLS) to 500k hours (YODAS) matter?

The answer, not tested in the paper, is likely: (1) training large self-supervised models from scratch (where scaling laws for pretraining data are well-established in both speech and text), and (2) training fully fine-tuned multilingual models with architectures like Whisper or USM that can absorb massive supervised data. The paper does not run either experiment. So while the scale claim is factually correct, the paper has not demonstrated that the additional scale (beyond what existing datasets provide) translates to improved downstream performance for any specific use case. This is a significant gap: a resource paper that claims to fill a scale gap should ideally include at least one experiment showing that the scale matters — e.g., a self-supervised pretraining experiment where adding YODAS data to the training mix improves performance on a downstream benchmark.

Does the paper demonstrate that the manual subset substantially outperforms the automatic subset?

Yes, convincingly. Table 6 provides a clean, controlled, single-variable comparison: same language, same training size, same architecture, same alignment threshold, same test set. The manual-trained model achieves a CER roughly half that of the automatic-trained model, with the gap driven by a 19.6-point difference in deletion errors. This is a specific, interpretable, and well-documented finding.

The error decomposition (into substitutions, deletions, insertions) is particularly valuable because it provides a mechanistic interpretation: automatic transcriptions hallucinate text for non-speech segments, teaching the model to delete. This is not a generic "automatic data is worse" claim — it is a specific diagnostic that explains how automatic data degrades performance. The alignment score scatter plots (Figures 4-5) corroborate this mechanism by showing that the automatic subset contains many long-duration segments with very high alignment scores, consistent with music/background noise being paired with hallucinated text.

A limitation: the comparison is run only for English. It is possible (likely, even) that the manual vs. automatic gap varies across languages. YouTube's automatic speech recognition quality varies substantially by language — it is strongest for high-resource languages like English, Spanish, and French, and weaker or unavailable for low-resource languages. In languages where automatic transcription quality is poor, the gap would be larger; in languages where it is very good (the paper notes that the automatic subset covers only 14 languages, likely those with the best ASR support), the gap might be smaller. The English-only comparison provides a single data point; it does not characterize the full distribution.

Does the paper demonstrate that the alignment threshold of 2.0 is near-optimal?

Yes, with the important caveat that this is for the English manual subset with 100k training utterances. The U-shaped curve in Figure 6 provides clear empirical evidence that 2.0 is the sweet spot, supported by a mechanistic interpretation (shorter utterances at very strict thresholds reduce training diversity). The sweep across an order of magnitude of threshold values (1 to 16) is thorough.

However, the optimal threshold likely depends on several factors that are not ablated:

• Language: English is a high-resource language with good XLS-R representations. For a low-resource language where the pretrained encoder's features are less discriminative, the CTC alignment scores might be systematically higher (worse), shifting the optimal threshold.
• Subset type: The sweep is performed on the manual subset. The automatic subset (Figure 5) has a very different alignment score distribution, with many more high-scoring utterances. The optimal threshold for the automatic subset might differ from 2.0.
• Training set size: With more data available, a stricter threshold might be optimal because more data survives filtering. The paper does not test this interaction.
• Model capacity: A higher-capacity model might be more robust to noisy data (it can learn to ignore or downweight misaligned examples during training), potentially making the optimal threshold less strict.

The paper's finding that 2.0 is optimal should therefore be understood as a specific result for a specific configuration, not as a universal recommendation. Users applying YODAS to different languages, subsets, or architectures should recalibrate the threshold rather than blindly adopting 2.0.

What experiments would have strengthened the paper?

Several experiments would have substantially increased the paper's impact as a resource contribution:

1. Self-supervised pretraining using the unlabeled subset (144k hours) and the full combined dataset. The paper mentions that YODAS is suitable for training models like HuBERT and wav2vec 2.0, and the unlabeled subset explicitly exists for this purpose. An experiment showing that pretraining on YODAS improves downstream performance relative to pretraining on existing public datasets (e.g., Libri-light) would directly validate the scale claim. This is the single most important missing experiment.

2. Comparison with existing public datasets. Training the same frozen XLS-R + linear CTC model on Common Voice or MLS data of comparable size for a few overlapping languages, and reporting CER, would position YODAS in the existing dataset ecosystem. This would answer the question: is YODAS better, worse, or comparable to existing alternatives for supervised ASR?

3. Multilingual or cross-lingual transfer experiments. The paper notes YODAS covers 140 languages in the manual subset. A natural experiment would be to train a single multilingual model on multiple YODAS languages and evaluate cross-lingual transfer to low-resource languages — a key use case for multilingual datasets. The monolingual baselines leave the multilingual value of YODAS entirely unexplored.

4. Fine-tuning vs. frozen encoder ablation. Training the same data with (a) frozen XLS-R, (b) fine-tuned XLS-R, and (c) partially fine-tuned XLS-R would quantify the performance ceiling for YODAS-trained models and reveal whether the saturation at 1M utterances (Figure 7) is architecture-limited or data-limited.

5. Analysis of the unlabeled subset. The paper presents detailed analyses of the manual and automatic subsets (language distribution, utterance duration, alignment scores), but the unlabeled subset — 144k hours — receives almost no analysis beyond its total duration. What languages does it likely contain? What is its acoustic diversity? Without this analysis, the unlabeled subset is an opaque resource.

6. Multiple random seeds or confidence intervals. Reporting CER as a point estimate from a single train-test split makes it impossible to assess whether small differences (e.g., threshold 2.0 vs. 1.0) are statistically reliable. Even reporting standard deviations across 3–5 random seeds would significantly strengthen the quantitative claims.

7. Effect of the Wikipedia keyword prioritization on language distribution. An ablation comparing the language distribution resulting from proportional keyword sampling vs. prioritized sampling (for less common languages) would quantify the impact of this design choice and guide future dataset builders.

8. Longitudinal or temporal analysis. The paper notes that YODAS will "continue to grow over time." An analysis of how the dataset's composition changes over collection windows (e.g., does language diversity increase or plateau?) would inform expectations for ongoing collection.

6. Limitations and Trade-offs

Limitation 1: The Unlabeled Subset (144k Hours) Is Entirely Unvalidated

The assumption or constraint. The paper partitions YODAS into three subsets — manual, automatic, and unlabeled — and positions the unlabeled subset as a resource for self-supervised pretraining, explicitly noting that "all three subsets may be used in conjunction with the application of self-supervised learning techniques" (Section 1). The unlabeled subset is the second-largest component at 144,000 hours, roughly 25% of the total corpus. The paper's data collection pipeline routes videos into this subset when the subtitle availability heuristic fails — that is, when a video has multiple subtitle tracks with conflicting language IDs, or no subtitles at all — and the paper states that language identification tools were attempted on these cases but "the results are not significantly successful" (Section 2.3).

The consequence. Beyond its total duration in hours, the paper provides no characterization whatsoever of the unlabeled subset. There is no analysis of the number of videos, the acoustic conditions, the likely language distribution, the proportion of content that is music versus speech versus noise, or even the average duration of audio files. A user downloading YODAS for self-supervised pretraining receives 144,000 hours of audio with no metadata about what it contains. This is not merely an inconvenience — it has concrete consequences for training. Self-supervised speech models are sensitive to the acoustic diversity and speech-to-non-speech ratio of their pretraining data. If a substantial fraction of the unlabeled subset is music, ambient noise, or non-speech audio (which is plausible, since these videos lacked usable subtitles precisely because they may not contain clean speech), then training on this data could degrade representations rather than improve them, or at minimum produce diminishing returns relative to the total hours. The paper's own alignment analysis (Figures 4-5, Section 3.4.5 in the prior analysis) demonstrates that the automatic subset — which does have transcriptions — contains extensive segments attributed to "music or background noise." The unlabeled subset, lacking even the weak signal of automatic transcriptions, may have an even higher proportion of non-speech content, but this hypothesis is neither confirmed nor refuted.

What evidence exists in the paper. There is no experimental evidence addressing the quality of the unlabeled subset. Section 3.2 provides descriptive statistics of video and utterance durations for the manual and automatic subsets (Table 2), language distributions for the labeled subsets (Figure 3), and text analysis of the transcriptions (Table 3-4), but the unlabeled subset receives only a single line stating its total duration (144,174 hours). No speech analysis, no acoustic characterization, no spot-check of content. The paper does not train a self-supervised model on the unlabeled subset or combine it with the labeled subsets in any experiment — the baseline experiments (Section 4) use only the manual and automatic subsets. This means the paper's claim that YODAS is suitable for self-supervised learning is aspirational rather than demonstrated. A practitioner deciding whether to invest compute in pretraining on the unlabeled YODAS data has no evidence from this paper to guide that decision.

Mitigation status. The paper does not attempt to mitigate this limitation. It does not acknowledge it as a limitation at all — the unlabeled subset is presented as a feature ("YODAS is distinctive as the first publicly available dataset of its scale") without any caveat about its unknown composition. The paper suggests no future work to characterize or filter the unlabeled subset, and does not flag it as an area requiring investigation before use. This is the most significant gap between the paper's claims and the evidence it provides: the headline "over 500k hours" includes 144k hours of completely uncharacterized audio that may or may not be useful for any specific downstream task.

---

Limitation 2: No Demonstration That Scale Beyond ~1,600 Hours Improves Performance

The assumption or constraint. The paper's central contribution claim is that YODAS fills a scale gap in public speech datasets — "the first public dataset to reach a scale of over 500k hours" (Section 1) — with the implicit argument that this scale enables better models than existing smaller datasets. For this claim to be practically meaningful, the additional data must produce measurable improvements on some downstream task relative to using a smaller dataset. The paper's only experiment testing whether scale matters is the training data size sweep in Figure 7 (Section 4.3), which varies the number of English manual-subset training utterances from 10,000 to 10,000,000 (approximately 16 to 16,000 hours).

The consequence. The scaling experiment in Figure 7 reveals a performance saturation at approximately 1 million utterances (~1,600 hours), with the model trained on 10 million utterances achieving "a similar CER" to the model trained on 1 million utterances. The paper attributes this to the frozen-encoder-plus-linear-probe architecture's limited capacity, noting that it "may not be fully equipped to leverage the expanded dataset." However, this explanation, while plausible, is not tested: the paper does not run the scaling experiment with a higher-capacity architecture (e.g., fine-tuning the full XLS-R encoder, or adding additional transformer layers) to determine whether the saturation is genuinely architecture-limited or whether the YODAS data itself has diminishing marginal value beyond ~1,600 hours per language.

The practical consequence is that the paper's headline scale claim — 500k+ hours, with 86k hours of manual transcriptions — is not connected to any demonstrated performance benefit. For the specific architecture tested (frozen XLS-R + linear CTC), a user would obtain essentially the same CER using 1.6% of the manual subset (1M English utterances) as they would using 16% (10M utterances). The remaining 84% of the English data, and the entirety of the automatic and unlabeled subsets, may or may not provide additional value, but the paper provides no evidence either way. A skeptical reader might conclude that YODAS is a 500k-hour dataset where the first 1.6k hours matter and the remaining 498.4k hours are unproven. This is especially problematic because the paper explicitly positions YODAS against industrial datasets (Whisper's training data at 100k+ hours, USM's at up to 1M hours) which do demonstrate that massive scale improves performance — but those demonstrations use architectures and training recipes very different from the linear probe baseline.

What evidence exists in the paper. Figure 7 provides the direct evidence of saturation. The paper's interpretation — that the linear architecture is capacity-limited — is offered without supporting ablation. The paper does not cite any prior work demonstrating that higher-capacity models on YODAS-scale data continue to improve, nor does it compare YODAS scaling curves to scaling curves from existing public datasets to show that YODAS's data has higher quality or better scaling properties. The saturation finding is reported transparently, which is to the paper's credit, but it leaves the central scale claim empirically unsupported.

Mitigation status. The paper acknowledges the saturation and offers the capacity-limitation hypothesis, but does not test it. This is a significant gap that could have been addressed by even a single experiment — for example, unfreezing the top few transformer layers of XLS-R and re-running the scaling sweep. The paper suggests no specific future work to validate the value of the full dataset scale, beyond the general statement that YODAS will "serve as a valuable resource for the speech research community" (Section 5). The mitigation is effectively deferred entirely to future users of the dataset.

---

Limitation 3: No External Baseline or Benchmark Comparison

The assumption or constraint. The paper evaluates YODAS-trained models exclusively on YODAS-derived test sets — 1,000 utterances per language, randomly sampled from the same manual subset used for training after applying the same alignment threshold (Section 4.1, Section 3.4.5 in the prior analysis). There is no evaluation on any external benchmark, and no comparison between a model trained on YODAS and the same architecture trained on an existing public dataset. The paper's positioning in Table 1 compares YODAS to other datasets by total hours, language count, and availability, but never by downstream model performance.

The consequence. A practitioner evaluating whether to use YODAS for their speech recognition task needs to know not just that YODAS-trained models achieve some absolute CER, but how that CER compares to what they would get from training the same model on an alternative dataset of similar size — Common Voice, MLS, or a combination of existing resources. Without this comparison, the CERs in Table 5 (ranging from 6.2 to 14.7) are unanchored numbers. Is a CER of 9.97 (the 15-language average) good? It depends entirely on the baseline. If Common Voice English produces a CER of 12 with the same architecture on the same amount of data, YODAS looks strong. If it produces a CER of 5, YODAS looks substantially worse. The paper provides no way to make this judgment.

This gap is particularly consequential because YODAS data comes from YouTube — a source known to contain diverse acoustic conditions, varying recording quality, and occasional speech-text misalignment even in the manual subset. It is plausible that YODAS-trained models perform worse on standardized test sets (which often feature clean, read speech) than models trained on curated datasets like Common Voice, while performing better on in-the-wild test sets (which more closely resemble YouTube's acoustic conditions). Neither hypothesis is tested, because no external test set is used. A user cannot determine whether YODAS models generalize to their target domain, or what domain shift to expect.

What evidence exists in the paper. There is no external evaluation or dataset comparison. The only internal comparison is manual vs. automatic (Table 6), which establishes relative quality within YODAS but says nothing about YODAS relative to other datasets. The test sets are constructed from the same distribution as the training data, so the reported CERs are estimates of in-distribution performance on YouTube-derived speech. How these models perform on, for example, the Common Voice test set or the LibriSpeech test set is unknown. The paper does not acknowledge this as a limitation.

Mitigation status. Not addressed. The paper provides baselines that validate YODAS's internal consistency (the data can be used to train ASR models that achieve non-trivial CER on held-out YODAS data) but not its external validity (YODAS-trained models are competitive with or superior to models trained on existing public datasets). This is a common limitation of resource papers — thorough external benchmarking is expensive and is often left to the community — but given that the paper's explicit motivation is to fill a gap left by existing datasets, the absence of any head-to-head comparison weakens the claim that YODAS actually fills that gap in a practically meaningful way.

---

Limitation 4: The Saturation of Simple Architectures Masks the True Value (or Lack Thereof) of Scale

The assumption or constraint. All baseline experiments in Section 4 use the same deliberately minimal architecture: a frozen XLS-R encoder with a single randomly initialized linear layer on top, trained with CTC loss, no data augmentation, and greedy decoding (Section 4.2, Section 3.4.6 in the prior analysis). This architecture is explicitly chosen to validate data quality rather than to achieve competitive performance — as established in the prior analysis, every design choice removes a degree of freedom that could compensate for noisy labels. The paper is transparent about this: the model is "simplistic" and "may not be fully equipped to leverage the expanded dataset" (Section 4.3).

The consequence. The architectural minimalism creates a fundamental confound in every experiment. When the paper observes that manual > automatic (Table 6), that threshold 2.0 is optimal (Figure 6), or that performance saturates at 1M utterances (Figure 7), all of these findings are jointly determined by the data and the architecture. A higher-capacity model — one that fine-tunes XLS-R, or uses a full encoder-decoder with attention, or employs SpecAugment and speed perturbation — might exhibit qualitatively different behavior:

• The manual-vs-automatic gap might narrow if a higher-capacity model can learn to ignore the hallucinated automatic transcriptions for non-speech segments, effectively performing its own implicit alignment filtering during training.
• The optimal alignment threshold might shift because a model with more capacity can tolerate noisier training data without degradation, making a looser threshold (more data, more noise) preferable.
• The scaling saturation might disappear, with performance continuing to improve substantially from 1M to 10M utterances when the model has the capacity to absorb the additional data.

The paper acknowledges the architecture limitation in the context of the scaling experiment but does not extend this acknowledgment to the other findings. The threshold sweep (Figure 6), the manual-vs-automatic comparison (Table 6), and even the per-language CERs (Table 5) are all reported without the caveat that they are specific to a frozen-encoder linear probe and may not generalize to the architectures that most practitioners will actually use. A user training a fine-tuned Whisper model on YODAS cannot safely extrapolate from these results — the optimal filtering strategy, the relative value of manual vs. automatic data, and the scaling behavior may all differ.

What evidence exists in the paper. The scaling experiment (Figure 7) provides the clearest evidence of the architecture-data interaction: performance saturates, the paper attributes this to model capacity, but the attribution is untested. The paper does not report any experiment with a higher-capacity architecture, not even partial fine-tuning of the top few XLS-R layers. All reported numbers are from the same frozen-encoder linear-CTC setup. This means none of the paper's quantitative findings about data quality or optimal usage are validated for the architectures that would actually exploit YODAS's scale.

Mitigation status. The paper acknowledges the capacity limitation for the scaling saturation specifically, calling the architecture "simplistic" and noting it "may not be fully equipped" for the task. However, this acknowledgment is narrow — it applies only to the scaling experiment — and does not generalize to the other findings. The paper does not flag the broader confound or suggest that future work should re-examine the threshold, subset comparison, and scaling findings with higher-capacity models. A reader who does not carefully note the architectural minimalism may incorrectly generalize these findings to all YODAS use cases.

---

Limitation 5: The Optimal Alignment Threshold Is Validated Only for English, Manual Subset, 100k Utterances

The assumption or constraint. The alignment score threshold sweep in Figure 6 — the paper's central data quality calibration experiment — is performed under a single specific configuration: English language, manual subset, 100,000 training utterances. The paper identifies 2.0 as the optimal threshold based on this sweep and applies it to all subsequent experiments across all 15+ languages and both the manual and automatic subsets.

The consequence. The assumption that 2.0 is a universal optimum is almost certainly false, and the paper's own evidence suggests why. The alignment score distributions for the manual and automatic subsets are dramatically different (Figures 4 vs. 5): the manual subset has tightly concentrated scores with most utterances under 10 seconds and scoring below 5; the automatic subset has a long tail of poorly aligned segments, many exceeding the 20-point cap. Applying the same threshold to both subsets means the filter removes very different proportions of data — relatively little from the manual subset, much more from the automatic subset. The optimal threshold for the automatic subset might be substantially different (likely looser, since the distribution is shifted right). Similarly, the optimal threshold may depend on:

• Language: The pre-trained XLS-R encoder's representational quality varies across languages. For a language with strong XLS-R features, the CTC alignment scores will be systematically lower (better) because the acoustic model accurately recognizes the phonetic content. For a low-resource language with weaker features, alignment scores will be systematically higher (worse). The 2.0 threshold, calibrated on English (a high-resource language with excellent XLS-R support), may be inappropriately strict for low-resource languages, discarding data that would be useful if the model were given a chance to learn from it.
• Training set size: With more data available post-filtering, a stricter threshold becomes more viable because enough data survives to fill the training budget. The paper does not test this interaction — the threshold sweep uses 100k utterances, but a user training on 1M+ utterances might find a different optimum.
• Architecture: A higher-capacity model may be more robust to noisy data, making a looser threshold preferable (more data, some noise, but the model can handle it).

What evidence exists in the paper. The threshold sweep (Figure 6) is the only evidence, and it is limited to the English manual subset at 100k utterances. The paper applies the 2.0 threshold to all other experiments without re-validating. The finding that tightening from 2.0 to 1.0 causes a slight CER degradation (14.7 to 14.8) is attributed to excluding "longer, richer utterances," which is a plausible mechanistic explanation but is itself a hypothesis that could vary across languages (do low-resource languages also lose "richer utterances" at strict thresholds, or are their utterance characteristics different?).

Mitigation status. The paper does not acknowledge this as a limitation. The threshold sweep is presented as a calibration that yields 2.0 as the answer, and 2.0 is then applied universally. There is no discussion of whether the threshold should be re-calibrated per language, per subset, or per training configuration. The paper suggests no future work on adaptive or language-specific filtering. A practitioner using YODAS for a language other than English, or using the automatic subset, should not assume 2.0 is optimal — but the paper provides no guidance on how to determine the right threshold for their setting.

7. Implications and Future Directions

How This Work Changes the Landscape

YODAS shifts the speech research ecosystem by providing a replicable, open-source pipeline that transforms YouTube's chaotic video corpus into a structured, Creative Commons-licensed, 500k-hour speech dataset spanning 140 languages. This is not a modeling innovation — it is an infrastructure contribution that changes what is possible for researchers who lack access to proprietary industrial data. The magnitude is substantial but bounded: YODAS does not introduce a new paradigm for speech recognition, nor does it propose a novel training objective. Rather, it removes a structural bottleneck — the absence of a genuinely large-scale, publicly available, permissively licensed multilingual speech corpus — that has constrained the research community's ability to reproduce, study, and extend the data scaling trends observed in industrial systems like Whisper and USM.

The conceptual shift is from dataset as static artifact to dataset as living system with documented construction methodology. Prior large-scale speech datasets (Common Voice, MLS, Libri-light) described their collection processes but treated them as one-time efforts. YODAS frames its distributed crawling architecture — the keyword-based client, channel-based client, download worker, and master node coordination design (Section 2, Figure 1) — as a contribution in its own right, with the commitment that both the dataset and the collection software will be open-sourced and that the corpus will "continue to grow over time" (Section 1). This makes YODAS not merely a resource but a framework for ongoing resource creation, lowering the barrier for other groups to extend the dataset to new languages, domains, or updated YouTube content without starting from scratch. The paper's detailed documentation of failure modes — YouTube's search result repetition problem, the AJAX-based solution, the subtitle ambiguity heuristic, the CTC alignment threshold calibration — serves as a blueprint that transforms dataset construction from an ad-hoc engineering task into a systematic, teachable methodology.

YODAS also resolves a latent tension in the speech dataset landscape: the tradeoff between scale and openness. The field has known since at least Whisper[16] and USM[17] that training on hundreds of thousands of hours of weakly supervised speech data produces dramatically better models than training on tens of thousands of hours of curated data. But the datasets enabling those results are proprietary, making the findings unreproducible and the scaling behavior unstudiable by the broader community. YODAS demonstrates that a 500k-hour dataset can be assembled from public sources under a Creative Commons license, establishing a floor for what is achievable in the open. The fact that YODAS's manual subset alone (86k hours) exceeds the total size of most prior public multilingual datasets (Table 1) validates the YouTube crawling approach as a viable path to industrial-scale speech data without industrial-scale budgets or proprietary access.

The paper's diagnostic orientation — using CTC alignment scores not just as a filter but as a characterization tool (Figures 4-5, Figure 6) — also shifts how the field should think about dataset quality. The U-shaped threshold sweep in Figure 6 contains a non-obvious finding: stricter filtering is not always better, and the optimal tradeoff between noise removal and data retention can be empirically calibrated rather than set by heuristic. This provides a template for data quality analysis that other dataset papers can adopt, moving beyond binary "clean/dirty" judgments to continuous, empirically grounded quality metrics.

However, the paper's impact is tempered by what it does not demonstrate. The unlabeled subset (144k hours) is entirely uncharacterized (see Limitation 1 in Section 6 of this analysis). The value of scale beyond ~1,600 hours per language is asserted but not proven — the only scaling experiment (Figure 7) saturates at 1M utterances, and the paper attributes this to architecture capacity without testing that hypothesis. No external benchmark comparison positions YODAS-trained models against models trained on existing public datasets. These gaps mean YODAS's primary impact is as potential rather than demonstrated value: it provides the raw material for future work that could validate the importance of open, large-scale speech data, but that validation remains to be done.

Specific research directions that become more attractive:

• Self-supervised pretraining at scale becomes newly tractable for academic groups, since YODAS provides 144k unlabeled hours plus 422k weakly labeled hours — an order of magnitude more than Libri-light (60k hours), the previous largest public option.
• Multilingual speech recognition for mid-resource languages (languages with some YouTube presence but no dedicated speech corpus) becomes feasible without per-language data collection campaigns, since YODAS covers 140 languages in the manual subset.
• Robustness to acoustic diversity research, since YouTube speech spans vlogs, lectures, interviews, outdoor recordings, and variable microphone quality — a more realistic distribution than studio-recorded or prompted speech datasets.
• Weak supervision and noisy-label learning research, since the automatic subset (336k hours) provides a large-scale testbed for methods that can extract signal from imperfect transcriptions.

Directions that become relatively less urgent:

• Small-scale data collection for languages already well-represented in YODAS. For the top 15–30 languages with tens of thousands of hours in YODAS, the marginal value of collecting additional prompted or read speech datasets is diminished — the bottleneck shifts from data quantity to model architecture and training methodology.
• Proprietary data moat arguments weaken, at least partially. YODAS shows that a 500k-hour dataset can be built from public sources, narrowing (though not closing) the data advantage of industrial labs.

---

Follow-Up Research This Work Enables

1. Self-supervised pretraining ablations: wav2vec 2.0 / HuBERT / WavLM on YODAS manual + unlabeled subsets vs. Libri-light alone.

The paper's most conspicuous missing experiment is self-supervised pretraining. YODAS provides 144k hours of unlabeled audio explicitly for this purpose, plus 422k hours of labeled data that can be used without transcriptions in self-supervised objectives. The community needs a controlled comparison: pretrain the same architecture (e.g., wav2vec 2.0 Base, HuBERT Base) on (a) Libri-light 60k hours, (b) YODAS unlabeled 144k hours, (c) YODAS unlabeled + automatic subsets at matched total hours to Libri-light, and (d) the full YODAS unlabeled + automatic corpus. Fine-tune all variants on the same downstream benchmarks (LibriSpeech, Common Voice, FLEURS, and ideally a YouTube-derived test set to measure in-domain performance). This would simultaneously validate whether YODAS data is suitable for self-supervised learning, quantify the scaling benefit of additional pretraining hours beyond Libri-light, and reveal whether acoustic diversity (YouTube) versus acoustic consistency (audiobooks) matters for downstream transfer. The paper already provides the alignment score distributions (Figures 4-5) that would help interpret results — if pretraining on YODAS underperforms Libri-light despite more data, the higher noise floor visible in the automatic subset's alignment scores is the likely culprit, suggesting that quality filtering of pretraining data matters even for self-supervised objectives.

2. Full fine-tuning scaling laws on YODAS: when does scale stop helping for supervised ASR?

The paper's scaling experiment (Figure 7) saturates at 1M utterances under a frozen-encoder linear probe, leaving open the question of whether the full supervised YODAS corpus (up to 10M+ utterances per top language in the manual subset) provides value for higher-capacity architectures. A concrete follow-up would replicate Figure 7 using a fine-tuned XLS-R (full encoder unfrozen) and a Whisper-style encoder-decoder architecture trained from scratch on increasing fractions of the English YODAS manual subset (100k, 300k, 1M, 3M, 10M utterances), evaluating on both in-domain (YODAS test set) and out-of-domain (Common Voice, LibriSpeech) benchmarks. The key question: does CER continue to improve log-linearly with training data size, or does it saturate at some point indicating that YODAS's data diversity (not just quantity) is the limiting factor? If performance continues improving to 10M utterances, this validates YODAS's scale for production-grade ASR. If it saturates earlier, the bottleneck is likely transcription quality or acoustic redundancy (many similar-sounding YouTube videos), and the community should invest in better filtering or harder example mining rather than simply collecting more YouTube data.

3. Multilingual transfer: does YODAS's 140-language breadth enable zero-shot ASR for languages with no manual transcriptions?

The paper trains separate monolingual models for the top 15 languages (Table 5) but never combines them. YODAS's language coverage (140 languages in manual, 14 in automatic) provides a natural testbed for multilingual training with cross-lingual transfer. A strong follow-up experiment would train a single multilingual model (fine-tuned XLS-R with a shared output layer, language ID token prepended) on the top 20–30 YODAS languages with the most manual data, then evaluate on held-out languages that appear in the manual subset but were excluded from training. The question: does training on 100k+ hours across diverse languages produce better zero-shot performance on unseen YODAS languages than the monolingual baselines in Table 5? Additionally, does including the automatic subset (336k hours, only 14 languages) help or hurt multilingual performance, given the deletion error problem documented in Table 6? This experiment would directly test whether YODAS's scale and diversity are complementary — scale helps if the additional languages provide useful inductive bias; diversity may hurt if the automatic subset's noise dominates. The paper's alignment score distributions (Figures 4-5) predict that the automatic subset's value will be highly threshold-dependent.

4. Language identification and cleaning of the unlabeled subset.

The 144k-hour unlabeled subset is the largest uncharacterized component of YODAS (Limitation 1, Section 6). A practical follow-up would apply state-of-the-art language identification models (e.g., ECAPA-TDNN based LID systems, or the LID component of MMS[13]) to assign language labels to the unlabeled audio, then characterize the resulting language distribution, speech-to-non-speech ratio, and acoustic diversity relative to the labeled subsets. The paper notes that LID tools were attempted and "the results are not significantly successful" (Section 2.3), but LID technology has advanced since the collection pipeline was designed — the MMS project[13] demonstrated LID across 4,000+ languages. A re-evaluation with modern LID tools could convert a substantial fraction of the unlabeled subset into weakly labeled data, dramatically increasing YODAS's usable size. The experiment should also measure language coverage expansion: how many languages appear in the unlabeled subset that are absent from the manual subset? This would determine whether the unlabeled data extends YODAS's language diversity or merely adds more audio for already-covered languages.

5. Domain adaptation and robustness benchmarking: YODAS-trained vs. Common Voice-trained vs. LibriSpeech-trained models on diverse test sets.

The paper evaluates only on YODAS-derived test sets, leaving external generalization unknown (Limitation 3, Section 6). A comprehensive follow-up would train identically configured models on (a) YODAS English manual subset (100k, 1M utterances), (b) Common Voice English (matched hours), and (c) LibriSpeech + MLS English (matched hours), then evaluate all models on a battery of test sets spanning the acoustic diversity spectrum: LibriSpeech test-clean (studio audiobooks), Common Voice test (prompted read speech from varied microphones), FLEURS (read speech in quiet conditions), CHiME-5 (conversational speech in noisy environments), and a held-out YODAS test set. The hypothesis: YODAS-trained models should dominate on YouTube-like and noisy speech, while curated-dataset models may perform better on clean read speech. The experiment would quantify the domain bias of each training corpus and provide the first systematic characterization of where YouTube-derived training data helps (and where it may actually hurt due to transcription quality issues). The manual-vs-automatic gap in Table 6 predicts that using the automatic subset for domain adaptation will be counterproductive unless alignment filtering is much stricter than the 2.0 threshold used in the paper.

6. Stress-testing the alignment threshold: per-language and per-subset calibration.

The 2.0 threshold is validated only for English manual subset at 100k utterances (Limitation 5, Section 6). A necessary follow-up would replicate Figure 6's threshold sweep across multiple languages (at minimum: a high-resource language like Spanish, a mid-resource language like Hungarian, and a low-resource language from the manual subset with limited XLS-R pretraining coverage), and across both the manual and automatic subsets. The question: does the optimal threshold shift systematically with (a) language resource level, (b) subset type (manual vs. automatic), or (c) training set size? If thresholds diverge substantially — e.g., 4.0 is optimal for the automatic subset while 2.0 is best for the manual subset — then applying a uniform 2.0 threshold across all YODAS data is silently degrading performance for some language-subset combinations. A negative result (2.0 is near-optimal across all tested conditions) would increase confidence in the paper's universal application of this threshold. A positive result (significant variation) would require YODAS documentation to provide per-language recommended thresholds, substantially complicating ease of use but improving downstream model quality.

---

Practical Applications and Downstream Use Cases

1. Low-resource language ASR bootstrapping. YODAS's 140-language manual subset provides, for many languages, the first large-scale speech corpus with transcriptions available under a permissive license. For a language like Hungarian (which achieves the best CER of 6.2 in Table 5), a practitioner can download the YODAS manual subset, apply the CTC alignment filter at threshold 2.0, and train a frozen XLS-R + linear CTC model to obtain a usable speech recognition system without any in-house data collection. For languages not in the top 15 but present in the manual subset, the same recipe should produce a first-pass ASR system — the paper does not report per-language results past the top 15 (Table 5 covers only the top languages by duration), but the coverage of 140 languages implies that thousands of hours per language are available for dozens of languages beyond those reported. This is directly actionable today using the ESPnet + s3prl recipe described in Section 4.2.

2. Self-supervised pretraining for domain-specific fine-tuning. A team building an ASR system for a specific domain (medical dictation, legal proceedings, classroom lectures) can use YODAS's unlabeled subset (144k hours) or the combined manual + automatic + unlabeled corpus (566k hours) to continue pretraining a self-supervised model like HuBERT or wav2vec 2.0 from a public checkpoint, adapting its representations to the acoustic conditions of spontaneous, in-the-wild speech before fine-tuning on their domain-specific labeled data. The acoustic diversity of YouTube — which includes quiet indoor speech, noisy outdoor recordings, music-bedded narration, and variable microphone quality — means the resulting representations should be robust to deployment conditions that clean-speech pretrained models struggle with. The paper does not demonstrate this (no self-supervised experiments are run), but the infrastructure exists: YODAS is available on Hugging Face, and the s3prl toolkit used in the baselines supports self-supervised pretraining.

3. Data augmentation and noise-robustness research. The automatic subset's alignment score distribution (Figure 5) reveals a large population of segments where automatic transcription produced text for music, background noise, or ambient sound — effectively, a dataset of naturally occurring noise-text misalignments. Researchers studying noise-robust ASR or learning from noisy labels can use the automatic subset as a controllable testbed: vary the alignment threshold to produce training sets with precisely known proportions of misaligned data (quantified by the alignment score), then train models and measure degradation as a function of noise proportion. The paper's finding that deletion errors dominate the automatic-subset degradation (Table 6) provides a specific, falsifiable hypothesis: noise-robust training methods should primarily reduce deletion rates, not substitution rates. A method that reduces substitution errors but leaves deletions unchanged is likely overfitting to clean-speech artifacts rather than genuinely handling the YouTube noise distribution.

4. Multilingual model development and cross-lingual transfer evaluation. YODAS's combination of scale (80k+ hours manual) and breadth (140 languages) makes it suitable as a primary training corpus for multilingual speech models following the Whisper or MMS paradigm. A group building a multilingual ASR system can use YODAS as the main training corpus (replacing or supplementing proprietary data), apply the alignment threshold of 2.0 to both manual and automatic subsets, and evaluate zero-shot transfer to languages in YODAS that were held out during training. The paper's per-language CERs (Table 5) provide a baseline: a multilingual model should at minimum match the monolingual baselines on seen languages, and its zero-shot performance on unseen languages quantifies the cross-lingual transfer benefit of YODAS's language diversity. The finding that CJK languages underperform (CER 12.5–14.7) despite large training sets suggests that vocabulary choice and writing system modeling are critical for multilingual models trained on YODAS — a practical design consideration that the paper already surfaces.

---

When to Prefer YODAS Over Alternative Public Speech Datasets

The paper does not formally position YODAS against named alternatives for specific use cases — its comparisons in Table 1 are at the level of scale and license, not downstream performance. However, the paper's characterizations of its own data quality (the manual > automatic finding, the alignment score distributions, the scaling saturation point) enable an implicit decision rule grounded in YODAS's demonstrated properties and known characteristics of alternative datasets. The following is derived directly from evidence in the paper:

• Prefer YODAS manual subset when: you need a single, publicly available, Creative Commons-licensed speech corpus with thousands of hours per language across dozens of languages, and your application involves spontaneous, conversational, or acoustically diverse speech (podcasts, video calls, field recordings, user-generated content). The frozen XLS-R + linear CTC baseline achieves CER 6.2–14.7 on YODAS's own test set (Table 5), establishing a floor for in-domain performance that higher-capacity architectures should exceed. The 140-language coverage and permissive license eliminate the licensing and data collection bottlenecks that make multilingual ASR development expensive for mid-resource languages.

• Prefer Common Voice when: your target domain is prompted read speech from diverse contributors (crowd-sourced recordings), you need per-speaker demographic metadata (age, gender, accent) for fairness or bias evaluation, or you are working on a language where Common Voice has extensive validated hours but YODAS has limited YouTube presence (languages with small internet footprints). Common Voice provides speaker-level metadata that YODAS lacks entirely, and its prompted-speech domain is more similar to voice assistant and dictation applications than YouTube's spontaneous speech. The paper provides no evidence about YODAS's speaker diversity or demographic coverage, making fairness research infeasible on YODAS alone.

• Prefer MLS or LibriSpeech when: your model will be deployed on studio-quality or professionally produced audio (audiobooks, broadcast news, corporate presentations), and you prioritize transcription accuracy over acoustic diversity. These datasets have near-perfect speech-text alignment by construction (derived from audiobooks with verified texts), eliminating the alignment noise that YODAS's CTC filtering only partially removes — even at threshold 2.0, some misalignment persists (Figure 4 shows occasional manual-subset utterances with scores around 10–18). The paper's deletion-error gap (Table 6, 26.6 automatic vs. 7.0 manual on English) is a lower bound on YODAS's alignment quality; MLS/LibriSpeech would be near-zero deletions.

• Prefer YODAS automatic subset only with caution and recalibrated filtering when: you have access to a higher-capacity model (fine-tuned encoder, data augmentation, or noise-robust training) that can tolerate the deletion-error problem documented in Table 6, and you need data volume substantially beyond what the manual subset provides (336k hours automatic vs. 86k hours manual). However, the paper's optimal threshold of 2.0 was calibrated only on the manual subset — the automatic subset's alignment score distribution (Figure 5, long tail of high-scoring segments) suggests that a different, likely looser threshold should be determined via a dedicated sweep (as proposed in Follow-Up Research direction 6 above) before using automatic data for supervised training. For self-supervised pretraining where transcriptions are ignored, the automatic subset's audio can be used directly, though the high proportion of music/noise segments relative to the manual subset means that pretraining on automatic data alone may produce representations biased toward non-speech audio patterns.

• Prefer YODAS unlabeled subset only for self-supervised pretraining when: you can commit to performing your own language identification and acoustic characterization of the data before use, since the paper provides no analysis of this subset's composition (Limitation 1, Section 6). The 144k hours are a valuable volume for pretraining, but the unknown language distribution and speech-to-non-speech ratio mean that training results are unpredictable without additional curation. Pairing the unlabeled subset with YODAS's labeled data (manual + automatic) in a self-supervised objective that can benefit from the unlabeled audio's scale while the labeled data provides a quality anchor is a plausible strategy, but entirely untested in this paper.