Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models

ArXiv: 2603.25750

🎯 Pitch

Overlapping speech isn't noise—it's the key to building AI that can interrupt and backchannel like a human. This paper shows that fine-tuning Moshi on data that preserves, rather than discards, conversational overlaps dramatically improves its ability to handle fluid, real-time dialogue. The authors release the entire scalable pipeline to curate such data from raw audio, solving a critical data bottleneck for full-duplex speech models.

---

1. Executive Summary

This paper introduces Sommelier, an open-source, scalable data processing pipeline designed to curate multi-turn conversational speech from in-the-wild audio for training full-duplex Speech Language Models. The pipeline combines rigorous speaker diarization via Sortformer, overlap speech separation using SepReformer with speaker identity matching, and ensemble-based ASR with ROVER voting across three models to reduce hallucinations, while preserving conversational dynamics like overlapping speech and backchanneling rather than discarding them. Validated by fine-tuning Moshi on Sommelier-processed data, the pipeline improves full-duplex interaction capabilities—achieving increases in Backchannel, Smooth Turn-Taking, and User Interruption scores on Full-Duplex-Bench 1.0, with the fine-tuned model demonstrating a shift from unresponsive behavior to actively processing user input—while operating at a practical real-time factor of 0.1746 on a single A100, establishing that high-fidelity full-duplex training data can be curated from single-stream recordings at scale only when the pipeline systematically preserves and disentangles overlapping speech regions rather than treating them as noise to be excised.

2. Context and Motivation

The Problem: We Cannot Build Full-Duplex Speech Models Without Full-Duplex Training Data

The paper addresses a fundamental bottleneck in the development of speech language models: the absence of large-scale, high-quality multi-speaker conversational data suitable for training full-duplex systems. While text-based LLMs have benefited enormously from web-scale text corpora and standardized preprocessing pipelines, the speech domain faces a different kind of scarcity. It is not that raw conversational audio does not exist — radio broadcasts, podcasts, interviews, and panel discussions are abundant. Rather, the problem is that current data processing pipelines for speech are designed for single-speaker, clean audio and systematically destroy the very conversational phenomena that full-duplex models need to learn.

This is not merely an inconvenience. Full-duplex speech models — systems that can listen and speak simultaneously, handling interruptions, overlapping speech, and backchanneling — represent the next frontier in human-computer interaction. Unlike turn-based systems where the user speaks, then the model processes, then the model responds, full-duplex systems engage in conversation the way humans do: with rapid turn exchanges, brief acknowledgments ("mm-hmm," "yeah"), simultaneous speech during enthusiastic agreement or competitive interruption, and the complex temporal choreography of who speaks when. The paper's opening framing in Section 1 makes the stakes clear:

"Progress toward full-duplex SLMs has been facing bottlenecked by the lack of high-quality conversational data suitable for duplex training. While Moshi leverages millions of hours of unsupervised audio for pre-training, these sources are largely single-stream and provide limited supervision for overlapping speech. Consequently, overlap robustness relies on relatively small high-fidelity conversational corpora such as Fisher, which is unlikely to meet the scale and diversity required for supervised fine-tuning."

The practical implication is stark: models can only learn to handle overlapping speech if they are trained on data where overlapping speech is faithfully represented, with both speakers' speech streams cleanly separated and transcribed. Data that flattens overlaps into a single muddied audio stream — or worse, discards overlapping regions entirely — trains models that are fundamentally incapable of full-duplex behavior.

This gap persists despite the existence of large-scale speech datasets. LibriSpeech (Panayotov et al., 2015) provides 1,000 hours of read audiobooks, GigaSpeech (Chen et al., 2021) contains 10,000 hours of multi-domain audio, and Emilia (He et al., 2024) aggregates massive multilingual speech. But as the paper notes in Appendix A.2:

"Existing speech datasets, despite their increasing volume, remain suboptimal for training full-duplex models that require rich interactional dynamics. Traditional ASR benchmarks like LibriSpeech and GigaSpeech are dominated by scripted read speech or solitary monologues, failing to capture the dynamic and interactive spontaneity of human dialogue."

Even when datasets do contain conversation — Fisher (Cieri et al., 2004) and Switchboard (Godfrey and Holliman, 1993), both telephone speech corpora — they are limited to narrowband telephony quality (8kHz), fixed two-party telephone interactions, and scales of a few thousand hours. These datasets were collected in controlled settings designed for ASR research, not for training models that must synthesize or understand the full acoustic richness of natural, in-person conversations.

The paper also highlights a critical transparency gap in the speech processing community. While text-based LLM research has benefited enormously from open data processing frameworks — Dolma (Soldaini et al., 2024), RedPajama (Weber et al., 2024), FineWeb (Penedo et al., 2024) — that allow researchers to understand exactly how training data was curated, the speech domain operates largely in the dark:

"Although model weights for Speech Language Models are frequently released, the intricate 'data recipes' required to curate high-quality pre-training corpora remain proprietary 'black boxes,' impeding the community's ability to reproduce results or improve upon existing strategies."

Sommelier positions itself as filling this transparency gap alongside the data curation gap: it is open-source, modular, and documented in sufficient detail for the community to adopt, adapt, and improve upon.

---

The Technical Challenges of Processing In-the-Wild Conversational Audio

Curating full-duplex training data from natural sources presents a cascade of interconnected technical challenges, each of which the paper addresses systematically. The central tension is between fidelity to conversational dynamics and pipeline robustness.

Challenge 1: Speaker Diarization Under Adversarial Conditions. In-the-wild audio contains frequent overlaps, rapid turn changes, and very short utterances — all of which are precisely the phenomena that matter most for full-duplex models, but which standard diarization tools struggle to handle. The paper notes in Appendix A.3:

"Standard tools like Pyannote often struggle in the complex acoustic environments of in-the-wild web videos. Crucially, these tools frequently misinterpret the overlaps and rapid turn-taking, essential features of full-duplex interaction, as segmentation errors or noise, thereby degrading the structural integrity of the conversational data."

This is not merely a matter of slightly degraded accuracy — it is a qualitative failure mode where the tool designed to identify who speaks when instead erases the evidence of simultaneous speech, making the data useless for its intended purpose. The paper's solution — adopting NVIDIA's Sortformer (Park et al.) over Pyannote 3.1 (Bredin, 2023) — is motivated by Sortformer's demonstrated superiority on short utterances and turn boundaries, as documented in Table 2.

Challenge 2: The Overlap Dilemma. When two speakers talk simultaneously in a single-stream recording, the data processor faces an uncomfortable choice. Figure 5 in the paper illustrates the four possible strategies for handling overlapping regions:

• Case 1 trims the overlap entirely, yielding clean non-overlapping segments but discarding potentially important speech content from both speakers during the overlap interval.
• Case 2 and Case 3 assign the overlap to one speaker or the other, which preserves full-length segments but contaminates one speaker's audio with the other's speech, causing ASR errors where transcripts mix or fail.
• Case 4 — the paper's approach — duplicates the overlapping region to both speakers and then applies speech separation to disentangle the two voices.

This is the paper's central design philosophy: preserve the overlapping speech as training data rather than discarding it, and invest the necessary computation to separate it cleanly. The tradeoff is computational — separation models add latency — but the paper argues (and validates in Section 3.3) that this investment is necessary for the resulting data to be useful for full-duplex training.

Challenge 3: ASR Hallucinations as a Data Contamination Problem. The paper identifies a failure mode that is well-known in ASR research but insufficiently addressed in speech data pipelines: hallucinations during silence or non-speech segments. The Whisper model family (Radford et al., 2022a), despite its strong overall performance, is prone to generating repetitive loops or nonsensical text when the input contains silence, noise, or music. The paper cites the "Careless Whisper" study (Koenecke et al., 2024b) and notes:

"These models are prone to hallucinations, particularly in silent or noisy segments, where they often generate repetitive or nonsensical text. Such artifacts introduce noise into the training signal, causing the downstream model to mimic these pathological behaviors."

This is a critical insight: ASR hallucinations are not just an evaluation metric problem — they become training data contamination that teaches downstream speech language models to produce hallucinations themselves. The paper's ensemble ASR approach with ROVER voting and n-gram filtering (Section 2.5) is explicitly designed to suppress these artifacts before they enter the training corpus.

Challenge 4: Background Music and Non-Speech Audio. Real-world audio sources — particularly radio broadcasts and podcasts — frequently contain background music, which is undesirable for training speech language models. The challenge is to remove music without degrading speech quality, and to do so only when necessary to avoid unnecessary processing. The paper's selective application of Demucs (Rouard et al., 2023; Défossez, 2021) based on PANNs (Kong et al., 2020) music probability estimation addresses this with a pragmatic threshold-based approach.

Challenge 5: Scalability. The paper is explicit about its industrial deployment context. Processing web-scale audio demands throughput that makes some sophisticated approaches impractical. Section 2.4 notes that SAM-Audio (Shi et al., 2025) was considered for music removal but excluded due to its high inference latency (RTF 0.73 on A100). The entire pipeline must operate at a real-time factor that makes processing thousands of hours of audio feasible under practical compute budgets.

---

The Limitations of Prior Data Processing Pipelines

The paper positions itself against a landscape where speech data processing pipelines exist but are fundamentally misaligned with the goals of full-duplex training.

The Emilia pipeline (He et al., 2024) is the most direct antecedent and receives the most attention. Emilia introduced a standardized pipeline for processing large-scale speech data from web sources, but it was designed for speech generation (TTS), not conversational understanding or full-duplex interaction. The paper's critique in Section 2.1 is implicit but clear: Emilia normalizes audio, applies VAD, and runs diarization — but it does not handle overlapping speech as content to be preserved. Instead, Emilia treats overlap as noise to be excised, which is appropriate for clean single-speaker TTS training data but destructive for conversational training data.

In Appendix A.2, the paper is explicit about this gap:

"While recent web-scale initiatives like WenetSpeech and Emilia have successfully aggregated massive datasets, their pipelines are heavily optimized for single-stream speech, thereby neglecting the concurrent dynamics required for full-duplex interaction. Crucially, their pre-processing pipelines treat overlapping speech as noise to be excised or ignored rather than a feature to be modeled."

This is not a criticism of Emilia per se — the paper acknowledges that Emilia was designed for a different purpose — but rather a diagnosis of why existing pipelines cannot simply be reused for full-duplex data curation.

The paper also identifies a tooling gap in the default choices that the community has converged on. Pyannote 3.1 (Bredin, 2023) has become the de facto standard for speaker diarization, adopted by pipelines including Emilia. But the paper presents quantitative evidence in Section 3.2 that Sortformer outperforms Pyannote specifically on the metrics that matter most for conversational data: short utterance handling and turn-boundary accuracy. This is not a marginal improvement — the gain on the critical DER (turn) metric is striking (0.006 vs. 0.051 in Table 2) — and it represents a deliberate choice to optimize the pipeline for the specific demands of full-duplex training data.

---

The Conceptual Shift: Overlap as Feature, Not Noise

Perhaps the most important intellectual contribution of the paper is a reframing of what constitutes "clean" conversational data. Traditional speech processing, rooted in ASR and TTS, optimizes for isolated, non-overlapping speech segments. In that paradigm, overlapping speech is noise — it degrades ASR accuracy and confuses speaker identification, so the rational thing to do is remove it.

The paper argues that for full-duplex SLMs, this logic is inverted. Overlapping speech is not an annoyance to be filtered — it is the most information-rich part of the signal. It carries the temporal dynamics of turn-taking, the acoustic signatures of agreement and interruption, and the paralinguistic cues that distinguish a fluid conversation from a disjointed exchange of monologues. A model trained on data where these regions have been excised or muddied will inevitably fail to reproduce natural conversational behavior at inference time.

This reframing is embedded in the paper's design philosophy statement in Section 2:

"Unlike traditional ASR pipelines that prioritize clean, non-overlapping speech, our design philosophy centers on preserving the chaotic yet rich dynamics of human dialogue, such as overlaps and backchannelings, while ensuring scalability for web-scale processing."

And again, more pointedly:

"Rather than stripping away speech overlaps and backchannelings, interruptions, and simultaneous speech that characterize real dialogue, we preserve them. This allows the duplex speech language model to learn not just what people say, but how conversations actually unfold."

This is a design decision with cascading consequences. It means the pipeline must include a speech separation module (SepReformer) and a speaker identity matching procedure (Section 2.3). It means the ASR system must produce accurate transcripts for speech that was artificially reconstructed from separation, which is inherently noisier than clean speech. It means validation metrics must include overlap-specific benchmarks (Full-Duplex-Bench) rather than just clean-speech accuracy. Every component of the pipeline is downstream of this fundamental choice to treat overlap as signal rather than noise.

---

The Validation Strategy: Not Just a Pipeline, But Proof That It Works

The paper does not merely describe a pipeline — it validates that the pipeline produces data that improves full-duplex model behavior. This is a crucial distinction. A pipeline description paper could measure only intermediate metrics (diarization error rate, ASR word error rate, separation quality) without ever demonstrating that downstream models benefit. The paper does report these intermediate metrics (Sections 3.2–3.4), but its central validation experiment is a full training run: fine-tuning Moshi (Défossez et al., 2024), a state-of-the-art full-duplex speech model, on 83 hours of Sommelier-processed data and evaluating on Full-Duplex-Bench 1.0.

The motivation for this validation choice is pragmatic. The full-duplex SLM field is nascent, and standard evaluation protocols are still emerging. The authors select Moshi because it represents the current best open-source full-duplex system. They select Full-Duplex-Bench (Lin et al., 2025b) because it provides standardized metrics for the conversational capabilities that matter: pause handling, backchanneling, turn-taking, and interruption handling. The fact that Sommelier-processed data improves Moshi's performance on these metrics — most notably in shifting the model from ignoring user input during turn-taking to actively processing it — is the paper's strongest evidence that its pipeline addresses a real need.

The paper also identifies a practical training consideration that emerged during this validation: turn-length sensitivity. The base Moshi model was trained on data with long single-speaker turns (over a minute), and the authors found that including such long turns in the fine-tuning data led to "unstable loss reduction and degrades performance, causing the model to become unresponsive" (Section 3.1). This led to a data filtering criterion — restricting turns to no more than 10 seconds and requiring at least three consecutive turns to form a valid training segment — that is not part of the pipeline per se but represents practical knowledge about what constitutes effective training data for full-duplex models. This finding, documented in Section 3.1, is itself a contribution to the emerging practice of full-duplex SLM training.

---

Positioning in the Broader Research Landscape

The paper positions Sommelier at the intersection of several research threads that have been largely separate:

Speech data curation. The lineage from WenetSpeech (Zhang et al., 2022) to Emilia (He et al., 2024) established the paradigm of web-scale automated speech processing. Sommelier extends this lineage by adding overlap handling and hallucination suppression, both motivated by the specific demands of full-duplex training.

Full-duplex speech models. Moshi (Défossez et al., 2024) and GPT-4o (Hurst et al., 2024) demonstrated that full-duplex interaction is achievable, but both relied on proprietary or limited data pipelines. Sommelier provides an open-source data pipeline that can support the broader development of such models.

Multi-speaker speech processing. Techniques like speech separation (SepReformer; Shin et al., 2024) and speaker diarization (Sortformer; Park et al.) have been developed for applications like meeting transcription, but have not been integrated into a comprehensive training data pipeline for conversational speech models. Sommelier bridges this gap by showing how these techniques can be composed into an end-to-end curation system.

Open-source AI infrastructure. By releasing the pipeline as open source (Section 4: "We release our pipeline to support reproducible industrial research and to accelerate progress toward natural, real-time human–AI interaction"), the paper positions itself alongside efforts like Dolma and FineWeb in the text domain, bringing the same transparency ethos to speech data processing.

The paper does not claim to have solved all problems in full-duplex data curation. The Limitations section acknowledges the pipeline's exclusive focus on speech (no non-speech acoustic event handling) and the inevitable fidelity loss from artificial separation compared to natively multi-channel recordings. But these limitations are framed as tradeoffs inherent in the approach, not as fatal flaws. The paper's contribution is to show that a practical, scalable, open-source pipeline can produce data that measurably improves full-duplex model performance — and to provide the community with a starting point that can be extended and improved.

3. Technical Approach

3.1 Reader Orientation

Sommelier is a modular, open-source audio processing pipeline designed to convert raw, in-the-wild multi-speaker recordings—podcasts, radio broadcasts, interviews—into cleanly separated, transcribed, and temporally aligned training data suitable for full-duplex speech language models. The core problem it solves is that standard speech processing pipelines treat overlapping speech as noise to be discarded, but full-duplex SLMs need precisely those overlapping regions as training examples to learn conversational dynamics like interruption handling and backchanneling; Sommelier's solution is to systematically detect, separate, and preserve overlapping speech regions while suppressing ASR hallucinations and background music, all within a single configurable pipeline where each module can be toggled on or off depending on the use case.

3.2 Big-Picture Architecture (Diagram in Words)

The pipeline consists of seven major components arranged in sequential order, as illustrated in Figure 1 of the paper:

1. Audio Standardization (§2.1) — Converts all input audio to a uniform format (16kHz, 16-bit, Mono) and applies loudness normalization to -20dBFS, ensuring downstream models receive consistent input regardless of the source recording's original format or volume level.

2. Voice Activity Detection (VAD) and Chunking (§2.2) — Splits long audio files into segments of less than five minutes by cutting at silence intervals detected by the Silero VAD model, preventing the speaker diarization model from running out of memory on lengthy recordings while preserving conversational context at segment boundaries.

3. Speaker Diarization (§2.2) — Identifies who speaks when using NVIDIA's Sortformer model, producing speaker-labeled time boundaries for each utterance. This step is the critical structural foundation for all subsequent overlap handling and speaker-specific processing.

4. Overlap Separation (§2.3) — Detects temporal regions where two speakers are talking simultaneously, extracts clean reference embeddings from each speaker's non-overlapping speech, applies SepReformer speech separation to disentangle the mixed audio into two candidate streams, and matches each separated stream to the correct speaker using cosine similarity against the reference embeddings.

5. Background Music Removal (§2.4) — Uses PANNs to estimate the probability of background music in each segment; if the probability exceeds 0.3, applies Demucs to extract the vocal track from the full audio context (rather than short segments) to improve separation quality, with selective application to avoid unnecessary speech degradation.

6. Ensemble ASR (§2.5) — Runs three distinct ASR models (Whisper-large-v3 as the primary backbone, plus Canary and Parakeet) on each speaker segment, aligns their word-level outputs using ROVER, applies majority voting (accept a word if predicted by at least two models; otherwise default to Whisper), extracts word-level timestamps from Whisper, and filters out hallucinations using an n-gram repetition detector.

7. Optional Context Captioning (Appendix E) — Generates rich metadata (emotion, gender, age group, situation descriptions) for each audio segment using Qwen3-Omni-Captioner with in-context learning, where the preceding two segments serve as audio prompts to capture conversational context like sarcasm.

Information flows linearly through the pipeline: raw audio enters → standardized format → split into chunks at silence boundaries → each chunk gets speaker-labeled time segments → overlapping regions are separated and recombined with non-overlapping speech → background music is selectively removed → three ASR models transcribe each speaker's segments → a final transcript is selected via ensemble voting → metadata is optionally generated. The output is a JSON file containing, for each utterance segment: start and end times, speaker ID, the final ensemble transcript, individual transcripts from each ASR model, word-level timestamps with confidence scores, and flags indicating whether the segment underwent separation or music removal.

3.3 Roadmap for the Deep Dive

• First, audio standardization and the VAD-based chunking strategy, since these establish the input format and segment size constraints that all downstream components depend on, and the five-minute limit is a practical engineering constraint with consequences for pipeline design.

• Second, the diarization model choice and the quantitative comparison between Sortformer and Pyannote 3.1, because speaker diarization is the structural backbone of the entire pipeline—every subsequent step (overlap detection, separation, speaker-specific processing) depends on accurate speaker boundaries, and the failure modes of standard diarizers on short utterances directly motivate the Sortformer adoption.

• Third, the overlap handling framework (the four cases in Figure 5 and the paper's selection of Case 4), the speech separation procedure using SepReformer, and the speaker identity matching algorithm, since this is the paper's central technical innovation and the component that distinguishes Sommelier from prior pipelines.

• Fourth, the background music removal strategy, including the PANNs detection threshold, the Demucs separation model, and the design choice to process full two-minute chunks rather than short segments for improved separation quality.

• Fifth, the ensemble ASR system, including the ROVER voting algorithm, the hallucination filtering via n-gram repetition detection, and the timestamp alignment procedure, since transcription quality and temporal precision directly determine the training signal that downstream SLMs receive.

• Sixth, the optional context captioning module (Appendix E), since it provides rich metadata that can enhance speech understanding and generation models, and the in-context learning approach for capturing conversational context is a design pattern that differs from standard per-segment captioning.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and pipeline paper whose core idea is that full-duplex SLM training data can be curated at scale from single-stream recordings by composing existing models (Sortformer, SepReformer, Whisper, Canary, Parakeet, Demucs, PANNs) into a modular pipeline that preserves overlapping speech as signal rather than treating it as noise, with the key innovations being the overlap handling procedure, the ensemble ASR hallucination suppression strategy, and the practical validation that downstream full-duplex model performance improves when trained on the resulting data.

---

Audio Standardization: The Foundation for Downstream Consistency

The pipeline begins by converting all input audio into a uniform representation to eliminate format variability as a source of downstream errors. This step follows the approach established by He et al. (2024) in the Emilia pipeline.

Format conversion. Using the pydub and librosa (McFee, 2025) libraries, all audio is converted to 16kHz sample rate, 16-bit depth, and single-channel mono. The choice of 16kHz is standard for speech processing: it captures the full frequency range of human speech (up to 8kHz, satisfying the Nyquist criterion for speech formants and fricatives) while keeping file sizes and computation manageable. 16-bit depth provides sufficient dynamic range (96dB theoretical) for speech without the storage overhead of 24-bit or 32-bit formats. Mono conversion is necessary because most downstream models—including the speech separation and ASR models—expect single-channel input, and stereo recordings of conversations (where different speakers are on different channels) are rare in in-the-wild sources.

Loudness normalization. All audio is normalized to -20 dBFS (decibels relative to full scale) using the ITU-R BS.1770 loudness standard. This value is chosen because it is the common target level for broadcast audio and provides sufficient headroom to prevent clipping during subsequent processing steps. Normalization is critical because downstream models are trained on data with consistent volume levels, and extreme variations in input loudness can cause ASR models to produce degraded transcriptions or hallucinate.

Why standardization matters. Without this step, the pipeline would need to handle arbitrary combinations of sample rates (8kHz telephone, 16kHz podcast, 44.1kHz music-derived speech, 48kHz professional recordings), bit depths, channel counts, and loudness levels. This variability would manifest as silent failures: an ASR model trained primarily on 16kHz speech might silently degrade on 8kHz input without explicit error, or a diarization model might produce spurious speaker changes at points where dramatic volume shifts occur. By enforcing a single canonical format at the pipeline entry point, all downstream components operate in a known, controlled regime.

---

VAD-Based Chunking: Memory Constraints as Design Driver

The practical constraint that drives this component is that speaker diarization models run out of GPU memory on audio files longer than approximately five minutes. This is not a fundamental limitation of the architecture but rather a consequence of the quadratic or near-quadratic attention complexity in transformer-based diarization models like Sortformer. Processing a 30-minute podcast in a single pass would require memory proportional to the square of the sequence length, exceeding the capacity of even large GPUs (the paper uses A100 80GB GPUs for pipeline processing).

The five-minute chunking strategy. The pipeline splits long audio into units of less than five minutes, but does so at silence boundaries rather than at arbitrary time points. The Silero VAD model (Team, 2024) is a pre-trained neural voice activity detector that classifies each short audio frame as speech or non-speech. The pipeline uses VAD predictions to identify natural pause points—moments where no speaker is talking—and cuts the audio at these boundaries.

Why cut at silence rather than arbitrary timestamps? A naive approach that splits every 295 seconds regardless of content would routinely cut in the middle of words, sentences, or speaker turns. This would create three cascading problems:

1. The diarization model would see a fragment of speech without context, potentially misidentifying the speaker or failing to detect a speaker change that occurs immediately after the cut.

2. The overlap separation module (which operates on diarized segments) would process artificially truncated utterances, potentially missing overlap that spans the cut boundary.

3. The ASR system would transcribe partial words at segment edges, producing garbled or nonsensical text that becomes training data contamination.

By cutting at VAD-detected silence, the pipeline ensures that each chunk begins and ends at natural conversational boundaries where no speech information is lost. If a chunk is approaching the five-minute limit but no silence is detected, the pipeline does cut at an arbitrary point, but this is the exception rather than the rule for conversational audio, which naturally contains frequent brief pauses.

A practical tradeoff. The five-minute constraint is an engineering compromise between memory feasibility and conversational context preservation. Longer chunks would retain more context (a diarization model can track speaker identities more reliably across a 10-minute conversation than across multiple 5-minute segments stitched together), but at the cost of requiring larger GPUs or more sophisticated memory management. The paper chooses 5 minutes as a practical threshold that fits within A100 80GB memory while being long enough to capture most conversational exchanges. The paper does not ablate this threshold, so it may represent a tuning opportunity for deployments with different hardware profiles.

---

Speaker Diarization: Sortformer vs. Pyannote 3.1

Speaker diarization answers the question "who is speaking when?" and produces a sequence of time-stamped speaker labels—the fundamental structural annotation that all subsequent processing depends on. The paper's key decision is to adopt NVIDIA's Sortformer (Park et al.) rather than the community-standard Pyannote 3.1 (Bredin, 2023; Plaquet and Bredin, 2023).

The Pyannote 3.1 baseline. Pyannote is an end-to-end neural diarization system built on the powerset multi-class cross-entropy loss (Plaquet and Bredin, 2023). It processes audio in sliding windows and, for each window, predicts which subset of speakers is active at each time frame. It has become the de facto standard for speaker diarization in the speech processing community, adopted by pipelines including Emilia (He et al., 2024). Its strengths are broad community adoption, well-characterized performance on standard benchmarks, and integration with the wider Pyannote ecosystem (voice activity detection, overlapped speech detection, speaker embedding extraction).

The Sortformer alternative. Sortformer takes a different architectural approach, using a permutation-resolved training objective that explicitly handles the ambiguity of which output slot corresponds to which speaker. Without this resolution, standard diarization models face a label permutation problem: if the model predicts "Speaker A" for a segment and "Speaker B" for another segment, but the ground-truth labels assign them in the opposite order, the loss function penalizes the model for what is actually a correct assignment. Sortformer's permutation-resolved loss addresses this by considering all possible speaker-label matchings and selecting the one that minimizes error, eliminating a source of training noise that particularly affects short utterances and rapid speaker changes.

Quantitative comparison (Table 2). The paper evaluates both models on the VoxConverse (Chung et al., 2020) common subset, which contains recordings with at most four speakers. The metrics are:

• DER (Diarization Error Rate): The fraction of speaker time that is incorrectly attributed, aggregating missed speech (speaker was talking but the model assigned silence), false alarm speech (model assigned a speaker but no one was talking), and speaker confusion (model assigned the wrong speaker). Computed with a standard forgiveness collar (typically 250ms around reference boundaries) to avoid penalizing minor timestamp imprecision.

• JER (Jaccard Error Rate): The average Jaccard distance between the reference and hypothesis speaker segments, which is more sensitive to boundary placement quality than DER because it directly compares segment overlap rather than using a collar.

The results show Sortformer improving on Pyannote across all global metrics:

Sortformer: DER 7.16% vs. Pyannote 8.40%, JER 14.69% vs. 17.68%

The absolute differences (1.24% DER, 2.99% JER) are notable but not dramatic on these aggregate measures. The critical differences emerge in the stress-test metrics:

• DER on short utterances (≤1.0s): Sortformer achieves 16.87% vs. Pyannote's 20.21% — a 3.34 percentage point improvement that directly measures better handling of brief interjections like "yeah," "mm-hmm," or "right."

• DER on turn-taking regions: Sortformer achieves 0.006 vs. Pyannote's 0.051 — nearly an order of magnitude improvement. The paper computes DER specifically in temporal windows around speaker change points (the brief moments when the active speaker changes), which is precisely where full-duplex models need accurate segmentation to learn turn-taking dynamics.

Why these specific metrics matter. The DER (turn) metric is the paper's strongest justification for adopting Sortformer. The difference between 0.051 and 0.006 might seem like a small absolute improvement, but it represents a fundamentally different failure mode: Pyannote is degrading diarization quality precisely at speaker boundaries—the moments when the conversational dynamics that full-duplex models need to learn are most salient—while Sortformer largely preserves boundary accuracy. A diarizer that misattributes 5% of speaker-change points will systematically create training examples where speaker transitions are blurred or incorrect, teaching downstream models to be less responsive to turn-taking cues.

The reliance on VoxConverse. The paper uses the VoxConverse common subset for evaluation, which is a standard benchmark but consists of YouTube videos with variable acoustic conditions. The absolute DER numbers (7–8%) indicate that even the best model makes non-trivial errors, and these errors propagate: a diarization mistake that assigns a segment to the wrong speaker will cause that speaker's audio to be included in the wrong speaker's concatenated stream, potentially creating nonsensical training examples where a single speaker seems to argue with themselves.

---

Overlap Handling: The Central Innovation

This section constitutes the paper's core technical contribution and the component with no analog in standard speech processing pipelines. The design philosophy is articulated in Section 2 and implemented in Section 2.3 with Figure 5 providing the conceptual framework.

The four-case taxonomy of overlap handling. When two speakers' utterances overlap in time, the data processor must decide how to assign the overlapping audio region to the output segments. Figure 5 illustrates the four possible strategies:

• Case 1: Discard the overlap. Trim each speaker's segment to end or begin at the overlap boundary, yielding two non-overlapping segments but losing all speech content during the overlap interval. For a 3-second utterance where the last 1.5 seconds overlap with the other speaker, Case 1 discards the final 1.5 seconds of each speaker's speech. This is the cleanest approach from an audio quality perspective (no mixed speech in either segment) but destroys potentially critical conversational content—for instance, the exact moment when one speaker interrupts another is the most information-rich signal for learning interruption dynamics.

• Case 2: Assign overlap to the first speaker. Extend the first speaker's segment through the overlap and truncate the second speaker's segment to start after the overlap. This preserves full-length audio for Speaker 1 but contaminates Speaker 2's segment with missing speech, which will cause ASR errors where the transcript cuts off or misses words. It also means Speaker 2's audio does not contain the full context of their utterance, potentially making their speech harder to interpret.

• Case 3: Assign overlap to the second speaker. The mirror of Case 2: preserve Speaker 2's full utterance and truncate Speaker 1's segment at the overlap boundary. Same contamination issue, different victim.

• Case 4: Duplicate and separate. Assign the overlapping region to both speakers and then use a speech separation model to disentangle the two voices within that region, producing two cleaned streams that are concatenated with each speaker's non-overlapping speech. This is the paper's approach and the only one that preserves full utterance information for both speakers.

Why Cases 2 and 3 are dangerous. The paper makes an explicit statement about their risks:

"Cases 2 and Case 3 assign the overlapping speech to one side, which risks ASR errors where utterances mix or transcripts fail."

The "mix" failure is when the ASR model, presented with audio containing two simultaneous voices, produces a transcript that combines words from both speakers into a single nonsensical sentence. The "fail" failure is when the ASR model, unable to parse the mixed audio, produces a completely hallucinated transcript or outputs silence/null. Both failures contaminate the training data: the downstream SLM learns from examples where mixed speech maps to garbled text, which is exactly the opposite of the desired behavior where overlapping speech should correspond to natural conversational dynamics.

Why Case 4 with separation is the right choice for full-duplex training. The paper's justification is implicit but can be reconstructed: full-duplex models need to learn to produce and understand speech in overlapping conditions. Training data that preserves both speakers' full utterances—including the overlapping portions, cleaned via separation—gives the model examples of what simultaneous speech sounds like from each speaker's perspective. A model trained on Case 1 data might never encounter overlapping speech examples and would fail to handle interruptions at inference time. A model trained on Case 2 or 3 data would have corrupted examples that associate overlapping speech with transcription errors.

The cost of Case 4 is computational: the separation model adds latency (0.0013 RTF according to Table 5) and the separated audio is inevitably slightly degraded compared to natively non-overlapping speech. The paper acknowledges this tradeoff in the Limitations section: "the resulting audio fidelity is inevitably slightly inferior to datasets that are originally recorded with distinct, isolated channels (Oracle), as the artificial separation process may introduce minor acoustic artifacts." But Section 3.3 demonstrates that the degradation is modest and that separated audio achieves UTMOS scores close to the Oracle upper bound.

The separation procedure in detail (Figure 2). The overlap handling component has two phases, illustrated in Figure 2 of the paper.

Phase 1: Reference embedding extraction (Figure 2a). Before separating any overlapping speech, the pipeline identifies non-overlapping segments from each speaker that are longer than 2 seconds. These serve as "reference samples" — clean audio that is definitively from a single speaker and can be used to identify which separated output stream belongs to which speaker. A speaker embedding model Memb processes these reference segments to produce embedding vectors:

eref1 = Memb(a0), eref2 = Memb(a1)

where a0 and a1 are non-overlapping audio segments from two different speakers, and each e is a fixed-dimensional vector that encodes the acoustic characteristics of that speaker's voice (pitch, timbre, speaking rate, accent). The choice of using non-overlapping segments longer than 2 seconds is important: segments shorter than this might not provide enough acoustic material for the embedding model to extract a reliable speaker representation, particularly for speakers with variable vocal patterns.

Phase 2: Separation and identity matching (Figure 2b). The overlapping audio region aoverlap is fed into the speech separation model Msep, which performs two-speaker source separation. The specific model used is SepReformer (Shin et al., 2024), which the paper chose over alternatives for reasons not fully detailed but likely related to its asymmetric encoder-decoder architecture that balances separation quality and computational efficiency. The separation model produces two candidate audio streams, acand1 and acand2, that represent the model's best estimate of what each speaker's voice sounded like during the overlap interval. These candidates are permutation-invariant: the model does not know which candidate corresponds to which speaker, only that it has separated the mixture into two distinct streams.

To resolve the permutation ambiguity, the pipeline computes cosine similarity between each candidate's speaker embedding and the reference embeddings from Phase 1:

S1 = cosSim(Memb(acand1), eref1)
S2 = cosSim(Memb(acand1), eref2)

If S1 > S2, candidate 1 is assigned to the first speaker (the one whose reference audio produced eref1); otherwise, candidate 1 is assigned to the second speaker. Candidate 2 then goes to the remaining speaker. The use of cosine similarity rather than Euclidean distance is standard in speaker verification — cosine similarity normalizes for embedding magnitude, making it robust to variations in the reference embedding that might arise from different segment lengths or recording conditions.

Final assembly. The separated overlap segments are concatenated with each speaker's non-overlapping speech to produce the final single-speaker segments. For a speaker whose utterance spans [tstart, t2] where the overlap occurs from t1 to t2, the final audio is: non-overlapping speech from tstart to t1, concatenated with the separated audio from t1 to t2. This produces a continuous stream that preserves the full duration and content of the original utterance while removing the interfering speaker's voice.

A critical implementation detail: inputting only the duplicated overlap region, not the full segment. The paper states:

"We find that inputting only the duplicated part into separation model works better than using the entire segment."

This is an empirical finding that improves separation quality. When the full segment (including non-overlapping portions) is fed into the separation model, the model must both preserve the clean portions and separate the overlapping portions, which is a harder task than separating an entirely-mixed signal. By isolating only the overlapping region for separation, the model can focus its capacity on the hard part of the problem, and the clean non-overlapping portions are preserved without any processing degradation.

When overlap becomes backchanneling. Appendix C and Figure 4 illustrate a special case of overlap: backchanneling, where one speaker produces a short utterance that is fully contained within another speaker's longer utterance (e.g., a listener saying "mm-hmm" while the speaker continues talking). The four-case taxonomy in Figure 4 for backchanneling is analogous to Figure 5 for partial overlap, with the same tradeoffs and the same Case 4 selection. Backchanneling is particularly important for full-duplex models because it represents the most common form of simultaneous speech in natural conversation, and the pipeline's ability to preserve these brief acknowledgments as cleanly-separated audio streams directly enables models to learn appropriate backchanneling behavior.

---

Background Music Removal: Selective Processing for Speech Purity

Many in-the-wild audio sources—particularly radio broadcasts, podcasts with intro/outro music, and dramas—contain background music (BGM) that is undesirable for training speech language models. The challenge is threefold: detect which segments contain music, remove the music without degrading speech quality, and do so without wasting computation on segments that do not need processing.

Detection: PANNs with a threshold. The pipeline uses PANNs (Pre-trained Audio Neural Networks; Kong et al., 2020), a family of convolutional neural networks trained on AudioSet for audio event classification. PANNs processes each audio segment and outputs a probability that background music is present. The paper applies a threshold of 0.3 on this probability: segments scoring below 0.3 are passed through unchanged, while segments scoring above 0.3 are processed by the Demucs music separation model.

Why 0.3? The paper does not provide an ablation study for this threshold, but the choice represents a bias toward recall over precision: it is better to unnecessarily process a few music-free segments (wasting some computation but not degrading speech quality) than to miss a music-containing segment (letting music-contaminated speech into the training data). The 0.3 threshold is relatively low—a model predicting 30% probability of music presence is not confident—which suggests the authors prioritize clean training data over minimizing unnecessary processing.

Separation: Demucs with full-context input. Segments flagged by PANNs are processed by Demucs (Rouard et al., 2023; Défossez, 2021), a hybrid transformer-based source separation model originally developed for music source separation (separating vocals, drums, bass, and other instruments from mixed audio). The paper uses Demucs in its vocal extraction mode, which outputs only the vocal (speech) track while discarding the instrumental accompaniment.

The critical design choice is that Demucs processes full two-minute audio chunks rather than short segments in isolation:

"We find that feeding the entire audio context into Demucs yields substantially better separation performance than processing short segments in isolation."

This is a data-level insight about how source separation models work. Demucs uses temporal context (the surrounding audio) to build a model of the background music's statistical properties, which makes it easier to identify and remove consistent musical elements (a repeating bassline, a persistent pad synth) from the foreground speech. When processing only a 5-second segment, the model has insufficient context to distinguish a brief musical phrase from speech prosody—a sung note might be confused with an emphatic spoken word. Full two-minute chunks provide enough musical context for the model to learn the separation effectively. After separation, only the portions of the output that correspond to the flagged segments are extracted, preserving the non-music segments without processing overhead.

Selective vs. universal application. The paper explicitly advocates for selective application:

"Since music removal can degrade speech quality, we selectively apply it only to segments identified by PANNs, minimizing unnecessary processing."

This is a pragmatic recognition that all separation models introduce some artifacts. The Demucs vocal extraction is not perfect—it can remove speech harmonics that overlap with music frequencies, introduce slight distortions in voice timbre, or fail to completely remove soft background music—and applying it to clean speech would only degrade audio quality without any benefit. The selective approach implements a simple cost-benefit analysis: the cost of potential music contamination in training data is higher than the cost of minor speech degradation from the separation model, but only for segments that actually contain music.

Alternative considered but rejected: SAM-Audio. The paper mentions considering SAM-Audio (Shi et al., 2025) for music removal but excludes it because:

"its high inference latency (RTF 0.73 on A100) limits its scalability for large datasets."

An RTF (Real-Time Factor) of 0.73 means that processing one hour of audio takes approximately 44 minutes of GPU time on an A100. For a pipeline targeting thousands of hours of audio, this would be the dominant computational cost, making the pipeline infeasible. Demucs, by contrast, achieves substantially lower latency (the pipeline's total RTF is 0.1746, of which Demucs is a fraction), making the selective application approach practical at scale.

---

Ensemble ASR: Combating Hallucinations Through Model Diversity

Automatic Speech Recognition is the final processing step that converts each speaker's separated audio into text transcripts, which become the training labels for downstream speech language models. The paper argues that single-model ASR, even with state-of-the-art architectures like Whisper-large-v3, is insufficiently reliable for large-scale training data curation because hallucinations—repetitive loops, nonsensical text, fabricated content during silence—will contaminate the training corpus and cause downstream models to learn pathological behaviors.

The ROVER ensemble algorithm. The paper employs the Recognizer Output Voting Error Reduction (ROVER) algorithm (Fiscus, 1997), a classic technique from the ASR literature that combines outputs from multiple speech recognition systems to produce a single, more accurate transcript. The specific implementation uses three models:

• Whisper-large-v3 (Radford et al., 2022a): The primary backbone. A large transformer-based encoder-decoder ASR model trained on 680,000 hours of weakly supervised multilingual data.
• Canary-1b-v2 (Sekoyan et al., 2025): A 1-billion-parameter encoder-decoder model optimized for efficient multilingual ASR and speech translation.
• Parakeet-tdt-0.6b-v3 (Sekoyan et al., 2025): A 600-million-parameter model using a Transducer-based architecture with time-depth separable convolutions, also optimized for multilingual ASR.

The voting procedure. ROVER operates at the word level:

1. The three transcripts are aligned using dynamic programming (edit distance) to establish a one-to-one correspondence between words across the three model outputs. This alignment handles insertions, deletions, and substitutions—if one model outputs "I am ready" and another outputs "I'm ready," the alignment recognizes that "I am" in the first transcript corresponds to "I'm" in the second.

2. For each aligned position, ROVER applies prioritized majority voting. If at least two of the three models agree on a word, that word is selected. If all three models disagree (a 1-1-1 split where each model outputs a different word), the system defaults to Whisper's prediction.

3. The selected words are concatenated to form the final transcript.

This is a practical engineering choice grounded in the observation that different ASR architectures make different types of errors. Whisper, trained on web-scale weakly supervised data, is strong on common speech patterns but prone to hallucinating during silence—generating repetitive text like "Yeah. Yeah. Yeah." or "Thank you. Thank you. Thank you." Canary and Parakeet, trained with different architectures and data mixtures, may have different failure modes: Canary might be more conservative during silence (outputting nothing instead of hallucinating) but less accurate on fast or accented speech, while Parakeet might handle technical vocabulary better but struggle with disfluencies. The ensemble exploits this diversity: a hallucination from one model is unlikely to be replicated by the other two, so majority voting suppresses it.

The hallucination filter: n-gram repetition detection. Even after ROVER ensembling, residual hallucinations can persist, particularly when two of the three models share a failure mode or when the hallucination is subtle. The paper applies a RepetitionFilter that discards any sample containing excessive n-gram repetition:

"Residual hallucinations are further pruned using a RepetitionFilter that discards samples with excessive n-gram (n = 15) repetitions (count ≥5)."

An n-gram of length 15 means a sequence of 15 consecutive words. The filter checks whether any 15-word sequence appears 5 or more times in the transcript, which would indicate a pathological repetitive loop (e.g., a model generating "I think that this is a very important point" on repeat). This approach follows the methodology of Udandarao et al. (2025), who showed that n-gram repetition filtering is effective for removing ASR hallucinations from speech-language pretraining data.

The choice of n=15 and count≥5 is specific and likely tuned to the expected length of hallucinated repetitions. Shorter n-grams (n=5) would risk filtering legitimate repeated phrases (e.g., a news anchor saying "Let me repeat that" and actually repeating a 5-word phrase), while longer n-grams (n=30) might miss hallucinations that repeat shorter phrases. A threshold of 5 repetitions means the same 15-word sequence must appear at least five times before the sample is discarded, which is a fairly conservative filter that only removes clear cases of pathological repetition.

Timestamp extraction and temporal alignment. Beyond the transcript text, the pipeline extracts word-level timestamps via Whisper's built-in timestamp prediction capability. Whisper-large-v3 can output not just the transcribed text but the estimated start and end time of each word, which is crucial for streaming speech language models that require synchronization between audio tokens and text tokens. The paper emphasizes this requirement:

"Precision in temporal alignment is critical for modern streaming speech language models, which typically require strict synchronization between audio and text tokens."

Full-duplex SLMs like Moshi operate on continuous audio streams with aligned text, generating speech output in real-time while processing incoming speech. If the word-level timestamps are inaccurate—e.g., a word is timestamped 200ms later than it actually occurs—the model will learn mismatched audio-text alignments, producing speech that sounds temporally disjointed or generating text at the wrong moments during inference.

Quantitative ASR evaluation (Table 4). The paper evaluates the ensemble against the single-model Whisper-large-v3 baseline on three standard ASR benchmarks:

• LibriSpeech test-clean: Read audiobook speech in quiet conditions. Whisper achieves 3.63% WER, the ensemble achieves 2.04% — a 43.8% relative improvement.

• LibriSpeech test-other: Read audiobook speech in noisy conditions. Whisper achieves 6.26% WER, the ensemble achieves 3.92% — a 37.4% relative improvement.

• TEDLIUM3 test: Spontaneous lecture speech with variable recording quality. Whisper achieves 12.19% WER, the ensemble achieves 10.66% — a 12.6% relative improvement.

The pattern is revealing: the ensemble's advantage is largest on clean read speech (where the baseline is already quite good) and smaller on spontaneous lecture speech (where all models struggle more). This suggests that the ensemble's primary value is in suppressing rare but severe failures (hallucinations, dropped words, misrecognitions of clear speech) rather than in improving average performance on challenging acoustic conditions. On noisy or spontaneous speech, all three models may make similar errors, so voting provides less benefit.

The paper also acknowledges the inference time cost: the ensemble requires approximately three times longer than single-model Whisper, primarily because Canary is the slowest of the three models, and running all three models sequentially (or with concurrent loading overhead) adds latency. This is a classic accuracy-latency tradeoff: the paper chooses higher transcription quality at the cost of processing speed, which is acceptable for offline data curation where throughput can be scaled with more GPUs but accuracy directly affects downstream model quality.

Qualitative example of hallucination suppression. In Appendix F, the paper provides a concrete example of the value of ensemble ASR. In the transcribed JSON example, a segment of speech is transcribed by each model:

• Whisper: "You're the emcee on the show, Ira. Oh, great. Ira, are you Ira? Ira?"
• Parakeet: "Oh, you're the MC on the show, Ira. Oh, great. Ira Iron."
• Canary: "Oh you're the MC on the show I read about Oh great I read I read I read"

The Whisper output shows hallucination: it repeats "Ira" and adds "Ira, are you Ira? Ira?" — a loop that fabricates content not present in the audio. The Parakeet output hallucinates "Iron." The Canary output hallucinates "I read I read I read." All three models fail in different ways. The ROVER ensemble, by requiring agreement, produces a cleaner transcript that avoids all three hallucination patterns. The paper explicitly notes this as a success case in Section 3.4:

"Qualitatively, we observed hallucinations in the Whisper outputs, such as repetitive generation (e.g., 'Yeah., Yeah., Yeah...'). We confirmed that our method successfully corrected these errors by selecting the accurate transcript provided by Canary (e.g., 'Yeah, big decision for Dan')."

Why ensemble ASR matters for downstream training. This is the paper's most important insight about data quality and is worth stating explicitly. When an ASR system hallucinates and that hallucination enters the training data, the downstream speech language model learns to map the audio features that triggered the hallucination (e.g., silence, noise, overlapping speech) to the hallucinated text. This creates a self-reinforcing pathology: the trained model will itself produce hallucinated text when encountering similar audio during inference, because it has been trained to do exactly that. By suppressing hallucinations at the data curation stage—before they enter the training corpus—the ensemble ASR approach prevents this contamination cascade entirely. The paper's framing of hallucinations as "training signal contamination" rather than merely an evaluation metric issue is a key contribution to data-centric AI for speech.

---

Context-Aware Captioning: Preserving Conversational Semantics

The final optional component, described in Appendix E, generates rich metadata for each audio segment, including emotion, gender, age group, and situation descriptions. This is not a core pipeline component (the paper frames it as supplementary), but it represents an additional capability that enhances the training data's utility for speech understanding and generation models.

The model: Qwen3-Omni-Captioner. The paper uses Qwen3-Omni-Captioner (Xu et al., 2025b), a multimodal model capable of processing both text and audio, to generate structured captions describing the acoustic and paralinguistic properties of each speech segment.

The problem with per-segment captioning. A naive approach that captions each segment in isolation fails to capture conversational context. For example, an isolated segment might sound angry, but in the context of preceding segments, it might be clear that the speaker is being sarcastic or joking. The paralinguistic meaning of speech depends on conversational context, and captioning segments independently discards this context.

The solution: in-context learning with audio prompts. The paper implements context-aware captioning by providing the two preceding audio segments as prompts when captioning the current segment:

"For consecutive audio segments a1, a2, and a3, we calculate the conditional probability P(C3 | I, a1, a2) to generate the caption C3 for a3."

Here, I is the instruction prompt to the captioner model, a1 and a2 are the two preceding audio segments concatenated as audio prompts, and C3 is the generated caption for the current segment a3. The model conditions on the prior conversational context (both the audio and any previously generated captions) when generating the caption, allowing it to incorporate contextual information like "speaker continues their sarcastic tone from the previous segment" or "speaker becomes increasingly agitated across the conversation."

Why in-context learning rather than fine-tuning. This approach exploits the Qwen3-Omni-Captioner's native in-context learning capability without requiring task-specific fine-tuning. The model was trained to process multi-turn interactions, so feeding it prior segments as context is a natural extension of its training paradigm. This keeps the pipeline modular: the captioning component can be upgraded to a new model without retraining, as long as the new model supports audio prompts.

The value of rich metadata. While the paper does not directly validate that these captions improve downstream model performance (the Moshi fine-tuning experiments in Section 3.1 do not use the captions), the motivation is that paralinguistic metadata—emotion, age, gender, acoustic environment—enables speech language models to learn richer representations of speech beyond text content. Models like CapSpeech (Wang et al., 2025a) and Ditto-TTS (Lee et al., 2025) use such metadata for style-controllable text-to-speech, while omni-modal models like Qwen-Omni (Xu et al., 2025a) and Ming-UniAudio (Yan et al., 2025) use it to understand the full communicative intent of speech, not just the words.

---

Latency and Throughput: Feasibility at Scale

Section 3.5 provides a detailed breakdown of pipeline latency on a single A100 80GB GPU, processing a 120-second audio sample (Table 5):

The Real-Time Factor (RTF) is the ratio of processing time to audio duration: an RTF of 0.1746 means the pipeline processes audio approximately 5.7 times faster than real-time. The ASR stage dominates, accounting for 66% of total processing time (13.91 out of 20.95 seconds). The SepReformer separation component is remarkably cheap (0.15 seconds for 120 seconds of audio), confirming that the decision to handle overlaps via separation rather than deletion does not impose a substantial computational penalty.

With 23GB of peak GPU memory usage, the paper notes that three concurrent processes can run on a single A100, effectively reducing the RTF to 0.0443 per GPU. For industrial-scale processing:

"Consequently, processing 10,000 hours of audio using eight A100 GPUs would take approximately 55 hours, demonstrating the practical feasibility of our approach."

This back-of-the-envelope calculation (10,000 hours × 0.0443 RTF / 8 GPUs = 55.4 hours) is important because it establishes that the pipeline is genuinely scalable, not merely described as such. Ten thousand hours is a realistic target for training data volume—Moshi used millions of hours for pretraining, but the supervised fine-tuning stage (where Sommelier data would be used) typically requires orders of magnitude less data.

The optional FlowSE denoising step appears in the latency table but is not discussed in the main text. FlowSE (presumably a speech enhancement model) adds 4.99 seconds of processing time (RTF 0.0416) and can be excluded to reduce total RTF to approximately 0.133. The paper does not describe this component in detail, so its exact function and performance characteristics are unclear from the provided text.

4. Key Insights and Innovations

Innovation 1: Reframing Overlapping Speech from Noise to Training Signal

The paper's most fundamental intellectual contribution is not any single algorithm or model choice, but a paradigm shift in what constitutes "clean" conversational training data. The standard assumption in speech processing — inherited from ASR research where overlapping speech degrades word error rates, and from TTS data curation where isolated single-speaker utterances are the gold standard — is that simultaneous speech from multiple speakers is contamination to be removed. Pipelines like Emilia (He et al., 2024) and WenetSpeech (Zhang et al., 2022) treat overlap as noise to be excised, and this was entirely rational for the applications they served: speech recognition and speech synthesis.

Sommelier argues that for full-duplex SLMs, this logic is precisely inverted. Overlapping speech is not contamination — it is the most information-dense region of the signal, carrying the temporal dynamics of turn-taking, the acoustic signatures of interruption and agreement, and the paralinguistic cues that distinguish fluid conversation from disjointed monologue exchanges. A model trained on data where overlap has been removed or flattened into a single mixed stream will fundamentally lack the capability to engage in natural full-duplex interaction, because it has never seen examples of what simultaneous speech sounds like from each speaker's perspective.

This reframing is not merely aspirational — it is operationalized in a taxonomy of overlap handling strategies (the four cases in Figure 5) that makes explicit what prior pipelines did implicitly. Cases 2 and 3 (assigning overlap to one speaker) are revealed as producing training examples that teach downstream models pathological behaviors: mapping mixed audio to garbled text, or learning to ignore the second speaker entirely. Case 1 (discarding overlap) produces clean audio but at the cost of erasing the very conversational phenomena the model needs to learn. The paper's selection of Case 4 — duplicate the overlap and separate — is the direct consequence of treating overlap as signal: if you value the information in overlapping speech, you must invest computation to disentangle it rather than delete it.

What makes this a genuine innovation rather than an obvious observation is that the full-duplex SLM community had not yet articulated this design principle as a data curation requirement. Prior full-duplex models like Moshi (Défossez et al., 2024) relied on proprietary data pipelines whose design choices were opaque. The paper's explicit statement that "Rather than stripping away speech overlaps and backchannelings, interruptions, and simultaneous speech that characterize real dialogue, we preserve them" (Section 2) establishes a diagnostic criterion that future full-duplex data pipelines can be evaluated against: does this pipeline preserve or destroy conversational dynamics?

The downstream validation experiment (Section 3.1) provides the crucial evidence that this reframing has practical consequences. Moshi fine-tuned on Sommelier-processed data shows substantially improved Backchannel, Smooth Turn-Taking, and User Interruption scores compared to the base model (Table 1). The base model's low latency on these metrics was not a sign of efficiency — it was a sign of pathological non-responsiveness, where the model simply continued speaking through user input. The fine-tuned model's increased latency reflects active processing of interruption and backchannel cues. This confirms that the data processing philosophy directly translates to behavioral differences in the trained model.

Innovation 2: The Ensemble ASR Hallucination Filter as a Data Contamination Firewall

The paper identifies and systematically addresses a failure mode that is well-known in the ASR literature but insufficiently operationalized in speech data curation pipelines: ASR hallucinations as a training data contamination vector. The standard approach in the field is to select a single high-performing ASR model (typically Whisper-large-v3) and trust its output as ground-truth transcription for downstream training. The paper argues that this is dangerously insufficient because even state-of-the-art models produce hallucinations — repetitive loops, nonsensical text, fabricated content during silence — that, when included in training data, teach downstream SLMs to produce the same pathological behaviors.

This is a diagnostic insight that changes how practitioners should think about data quality. The problem is not merely that hallucinations reduce transcript accuracy (a few percent WER on a benchmark) — it is that they create a self-reinforcing contamination cascade. If a training example maps silence or noise to the text "Yeah. Yeah. Yeah.", the downstream SLM learns to associate those acoustic conditions with that text pattern. At inference time, when the model encounters similar silence or noise, it will produce "Yeah. Yeah. Yeah." because that is exactly what it was trained to do. The paper's framing of hallucinations as "training signal contamination" rather than merely an evaluation metric issue elevates data curation from a quality-improvement task to a model-safety concern.

The solution — a three-model ROVER ensemble (Whisper, Canary, Parakeet) with prioritized majority voting and n-gram repetition filtering — is effective not because any single model is hallucination-free, but because the three models hallucinate differently. Section 3.4 provides concrete evidence: Whisper hallucinates repetitive loops, Parakeet hallucinates word substitutions, Canary hallucinates entirely different phrases. The ensemble exploits this diversity through majority voting, where a hallucination from one model is unlikely to be replicated by the other two. The n-gram filter (n=15, count≥5) serves as a safety net for the rare cases where two models share a hallucination pattern.

What distinguishes this from standard ASR ensemble work is the explicit connection to downstream model behavior. The paper's qualitative example (Section 3.4) of Whisper generating "Yeah., Yeah., Yeah..." and the ensemble correcting it to "Yeah, big decision for Dan" is not just an ASR accuracy improvement — it is a demonstration that the pipeline prevents a specific hallucination pattern from entering the training corpus and potentially infecting the downstream SLM. The quantitative WER improvements (Table 4: 37% relative reduction on LibriSpeech test-other, 3.92% vs. 6.26%) are substantial but secondary to this contamination-prevention argument.

Innovation 3: Practical Difficulty Taxonomy as a Bridge Between Data Curation and Model Training

The paper makes an empirically grounded contribution that bridges the gap between data processing and model training: the discovery that full-duplex SLM training dynamics impose specific requirements on the data format that go beyond audio quality and transcription accuracy. During the Moshi fine-tuning experiments (Section 3.1), the authors discovered that including long single-speaker turns (over a minute) in the training data caused "unstable loss reduction and degrades performance, causing the model to become unresponsive." This led to explicit data filtering criteria — restricting turns to no more than 10 seconds, requiring at least three consecutive turns to form a valid training segment, and assigning only one speaker to the left channel of stereo training data — that are not about audio quality but about training stability and model behavior.

This is a surprisingly underexplored area in the speech SLM literature. While the text LLM community has extensively studied the effects of data composition, sequence length, and repetition on training dynamics, the speech SLM community has not yet developed a comparable body of knowledge about what makes training data "good" beyond clean audio and accurate transcription. The paper's documentation of turn-length sensitivity and turn-sequence requirements represents some of the first public empirical evidence about the relationship between conversational data structure and full-duplex model trainability.

The significance extends beyond these specific findings. The paper establishes a feedback loop between data curation and model training that the field currently lacks: data processing pipelines should be designed with knowledge of how downstream models will use the data, and model training should inform data filtering criteria. The fact that these filtering rules emerged during fine-tuning rather than being designed a priori suggests that the optimal data format for full-duplex training is not obvious and requires empirical discovery — a finding that implicitly argues for more open data pipelines that enable the community to collectively accumulate this knowledge.

Innovation 4: A Modular, Togglable Architecture as an Epistemic Tool

While the pipeline's modularity might appear to be an engineering convenience rather than an intellectual contribution, the paper's design choice to make every component independently togglable serves a deeper purpose: it transforms Sommelier from a fixed recipe into an epistemic tool for understanding which processing steps matter and under what conditions. The paper states in Section 2 that "each component can be toggled or reconfigured, allowing researchers to adapt the trade-off between data purity and conversational authenticity," which frames the pipeline as an instrument for exploring the data-quality landscape rather than a black-box solution.

This is a departure from the dominant paradigm in speech data curation, where pipelines are optimized for a single use case (ASR training, TTS training) and their internal design choices are opaque to downstream users. Emilia (He et al., 2024) produces clean single-speaker speech; its design decisions are fixed. Sommelier's architecture allows researchers to ask counterfactual questions: What happens to downstream full-duplex performance if we disable the overlap separation module and use Case 1 (discard overlap) instead? How much does the ensemble ASR improve downstream model behavior versus single-model Whisper? What is the optimal PANNs music detection threshold for a specific domain?

The paper does not answer all of these questions — the validation experiments focus on the full pipeline — but the architecture is designed to enable the community to investigate them. The latency breakdown in Table 5 is particularly instructive: by showing that SepReformer contributes only 0.0013 RTF to the total pipeline, the paper makes visible that overlap handling is computationally cheap, removing a potential objection to its adoption. Similarly, showing that the ASR ensemble dominates pipeline latency (0.1159 RTF out of 0.1746 total) focuses attention on the component where accuracy-latency tradeoffs are most consequential.

This design philosophy draws from the open-source data curation movement in text-based LLMs (Dolma, RedPajama, FineWeb) but adapts it to the specific technical demands of speech processing. In the text domain, the primary unit of processing is the document, and the primary quality concerns are deduplication, toxicity filtering, and factual accuracy. In the speech domain, the unit is the conversational turn, and quality concerns span acoustic fidelity, speaker attribution, temporal alignment, and transcription accuracy — a qualitatively different problem space that requires a different kind of modular transparency. Sommelier's contribution is to demonstrate that this transparency is achievable and valuable for speech, not just text.

5. Experimental Analysis

Evaluation Methodology

Dataset. The pipeline is evaluated using several datasets, each serving a distinct purpose. For overlap separation quality (Section 3.3), the authors synthesize 900 two-speaker mixtures from the LibriSpeech (Panayotov et al., 2015) test utterances by varying Signal-to-Interference Ratio (SIR ∈ {0, 5, 10} dB) and overlap ratio (ρ ∈ {0.2, 0.5, 1.0}), yielding nine distinct conditions covering easy to severe overlap scenarios. For diarization evaluation (Section 3.2), they use the VoxConverse (Chung et al., 2020) common subset (recordings with at most four speakers), a standard benchmark for speaker diarization in the wild. For ASR evaluation (Section 3.4), they employ LibriSpeech test-clean and test-other (read audiobooks in quiet and noisy conditions) and TEDLIUM3 (Hernandez et al., 2018) test set (spontaneous lecture speech). The downstream full-duplex validation (Section 3.1) uses 83 hours of Sommelier-processed data drawn from a diverse set of conversational domains (Figure 6 shows the category distribution, including Lifestyle 16.6%, Religion/Spirituality 12.3%, Sports 10.3%, Education 8.8%, and others), evaluated on Full-Duplex-Bench 1.0 (Lin et al., 2025b) and additionally on Full-Duplex-Bench 1.5 (Lin et al., 2025a) for the more comprehensive overlap handling evaluation.

Base model(s). The downstream validation fine-tunes Moshi (Défossez et al., 2024), specifically the moshiko-pytorch-bf16 variant, using LoRA with rank 128 on 8 A100 GPUs for 2,000 steps with batch size 16 and learning rate 2e-6 (Table 9). Moshi is chosen because it represents the current best open-source full-duplex speech language model, making it the natural testbed for evaluating whether Sommelier-processed data improves full-duplex capabilities. The other models in the pipeline are all pre-trained, off-the-shelf components: Sortformer (Park et al.) for diarization, SepReformer (Shin et al., 2024) for speech separation, Whisper-large-v3 (Radford et al., 2022a), Canary-1b-v2, and Parakeet-tdt-0.6b-v3 (Sekoyan et al., 2025) for ASR, PANNs (Kong et al., 2020) for music detection, and Demucs (Rouard et al., 2023; Défossez, 2021) for music removal. None of these models are fine-tuned by the authors; the pipeline composes them as fixed components.

Metrics. The paper uses diverse metrics across its component evaluations. For diarization (Section 3.2, Table 2): DER (Diarization Error Rate, the fraction of speaker time incorrectly attributed), JER (Jaccard Error Rate, the average Jaccard distance between reference and hypothesis speaker segments, more sensitive to boundary quality), and stress-test variants DER(≤1.0s) on short utterances and DER(turn) on windows around speaker change points. For overlap separation (Section 3.3, Tables 3, 6, 7, 8): WER (Word Error Rate, measuring transcription intelligibility of separated speech), SI-SDR (Scale-Invariant Signal-to-Distortion Ratio, measuring acoustic separation quality), STOI (Short-Time Objective Intelligibility, measuring how intelligible the separated speech is), and UTMOS (Saeki et al., 2022) (a learned MOS predictor measuring perceptual naturalness). For ASR (Section 3.4, Table 4): WER on standard test sets. For full-duplex interaction (Section 3.1, Table 1, 10, 11): Full-Duplex-Bench metrics including TOR (Turn-Over Rate) for pause handling, backchannel frequency and JSD (Jensen-Shannon Divergence of backchannel timing), turn-taking latency and Candor TOR, interruption TOR and latency, and audio quality metrics (STOI, PESQ, SI-SDR, UTMOS) for overlap scenarios.

Baselines. For each subsystem, the paper establishes clear baselines. In diarization (Table 2), the baseline is Pyannote 3.1 (Bredin, 2023; Plaquet and Bredin, 2023), the community-standard diarization model also used by Emilia (He et al., 2024). In overlap separation (Tables 3, 6, 7, 8), the baseline ("Original" or "Base") extracts time segments directly from the mixed audio without separation; the upper bound ("Oracle") uses clean source signals before mixing. In ASR (Table 4), the baseline is single-model Whisper-large-v3 (Radford et al., 2022a). For full-duplex evaluation (Table 1), the baseline is the unmodified base Moshi model (Défossez et al., 2024) without Sommelier-processed fine-tuning data.

Generation budget / compute accounting. The pipeline's compute is measured as processing latency and throughput. Section 3.5 (Table 5) provides a detailed breakdown of processing time per 120-second audio sample on a single A100 80GB GPU: VAD + Sortformer 1.91s (RTF 0.0159), SepReformer separation 0.15s (RTF 0.0013), ASR ensemble 13.91s (RTF 0.1159), and optional FlowSE denoising 4.99s (RTF 0.0416). The total RTF is 0.1746, meaning the pipeline runs approximately 5.7× faster than real-time. With 23GB peak memory, three concurrent processes can run per GPU, reducing effective RTF to 0.0443. The paper uses this to estimate that processing 10,000 hours of audio takes ~55 hours on 8 A100 GPUs. For the downstream Moshi fine-tuning, compute is measured in training steps (2,000) and total data duration (~83 hours) processed through the full pipeline.

Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense for its component evaluations — each subsystem is evaluated on standard held-out test sets. For the full-duplex validation (Section 3.1), the fine-tuned Moshi is evaluated on Full-Duplex-Bench 1.0 and 1.5, which provide standardized test scenarios designed specifically for full-duplex evaluation. The overlap separation experiments (Section 3.3) use systematically synthesized mixtures across nine conditions (3 SIR levels × 3 overlap ratios) totaling 900 samples, ensuring coverage of diverse overlap scenarios. No k-fold or repeated-trial protocols are reported, meaning statistical confidence intervals for the downstream full-duplex metrics are not available.

---

Main Quantitative Results

Diarization Model Comparison (Section 3.2, Table 2)

The diarization component evaluation directly compares Sortformer against the Pyannote 3.1 baseline on the VoxConverse common subset. The headline finding is that Sortformer outperforms Pyannote across all metrics, with the gap being most pronounced on the measures that directly impact conversational data quality.

On aggregate metrics, Sortformer achieves DER 7.16% vs. Pyannote's 8.40% (a 1.24 percentage point improvement) and JER 14.69% vs. 17.68% (a 2.99 percentage point improvement). These are meaningful but not dramatic gains on standard evaluation.

The critical differences emerge on the stress-test metrics that measure performance on precisely the conversational phenomena most valuable for full-duplex training data. On short utterances (≤1.0s), Sortformer achieves DER 16.87% vs. Pyannote's 20.21%, representing a 3.34 percentage point improvement in correctly attributing brief interjections like backchannels. This matters because backchannel utterances ("mm-hmm," "yeah," "right") are typically under one second, and a diarizer that misses or misattributes them at a 20% rate will systematically corrupt the training signal for backchannel-related conversational dynamics.

The most striking result is on the DER(turn) metric, which specifically evaluates diarization accuracy at speaker change boundaries. Sortformer achieves 0.006 vs. Pyannote's 0.051 — nearly an order of magnitude improvement. While both numbers are small in absolute terms, the relative difference is substantial: Pyannote makes errors at roughly 5% of turn-taking boundaries, while Sortformer makes errors at approximately 0.6%. Since turn boundaries are precisely where the conversational dynamics of interruption, smooth turn-taking, and backchannel timing are encoded, a 5% boundary error rate means a non-trivial fraction of training examples will have incorrect speaker transitions, potentially teaching the downstream model to be insensitive to turn-taking cues or to respond at inappropriate times.

These results support the paper's decision to adopt Sortformer over the community-standard Pyannote for the specific demands of full-duplex training data curation, though the absolute error rates (7.16% DER for Sortformer) indicate that even the best model leaves room for improvement.

---

Overlap Separation Quality (Section 3.3, Tables 3, 6, 7, 8; Figure 3)

The overlap separation evaluation addresses the central question of the pipeline design philosophy: can speech separation restore sufficient audio quality from single-stream overlapping speech to serve as usable training data for full-duplex models?

The headline result from Table 3 is that the separation module ("Sep") consistently and substantially outperforms the baseline ("Ori," which extracts audio directly from the mixed signal without separation) across all four metrics (WER, STOI, UTMOS, and the additional SI-SDR in the detailed tables), with the gap widening dramatically under the most challenging conditions — high overlap ratios. The separation module achieves UTMOS scores that closely approximate the Oracle upper bound (clean source audio before mixing), suggesting that the separated speech is perceptually natural enough for training purposes.

The overlap ratio is the dominant factor, not SIR. The paper explicitly states: "While variations in the Signal-to-Interference Ratio (SIR) have a limited impact on performance, the overlap ratio serves as the critical determinant of task difficulty." The bottom row of Figure 3 shows this visually: WER for both speakers increases sharply as the overlap ratio moves from 0.2 to 1.0 under the baseline condition, while the separation module largely flattens this curve.

The detailed breakdown by speaker (Tables 6, 7, 8) reveals asymmetric benefits: Speaker 2 (the quieter or secondary speaker in the mixture, depending on SIR) benefits far more from separation than Speaker 1 (the louder or primary speaker). In the worst-case condition — 0 dB SIR and 1.0 overlap ratio — the WER for Speaker 2 improves from 0.444 (nearly half of words incorrect) under the baseline to 0.138 under separation (Table 6), while Speaker 1 improves from 0.535 to 0.175. This asymmetry makes intuitive sense: the primary speaker's speech already dominates the mixture, so the baseline transcription is partially intelligible, while the secondary speaker's speech is largely masked by the primary speaker's voice, making separation essential for recovering their utterance. For full-duplex training data, where both speakers' speech must be accurately transcribed during overlaps, this recovery of the secondary speaker's speech is critical.

The UTMOS scores (Table 8) are particularly encouraging for the pipeline's goal of producing training-quality data. Even under the most adverse condition (0 dB SIR, 1.0 overlap ratio), the separation module achieves UTMOS of 3.05 (Speaker 1) and 2.99 (Speaker 2), compared to the Oracle's 3.81 and 3.86, and substantially above the baseline's 1.73 and 1.67. UTMOS is a learned predictor of Mean Opinion Score on a scale roughly from 1 to 5; scores near 3.0 indicate "fair" to "good" quality, while scores near 1.7 indicate "bad" quality. The separation module thus transforms audio from "bad" to "fair/good" naturalness, which the paper argues is sufficient for training data — not as good as natively clean audio, but dramatically better than mixed audio and good enough to provide useful training signal.

A nuanced finding on WER at low overlap ratios. The tables reveal that at ρ = 0.2 and ρ = 0.5, the baseline actually shows competitive (and in some cases lower) WER compared to the separation module. For example, at 5 dB SIR and ρ = 0.2, the baseline achieves WER 0.080 for Speaker 1 vs. the separation module's 0.088 (Table 6). The paper explicitly explains this (Section 3.3): "This comparison is inherently unfair: the Baseline extracts each speaker's time segment directly from the mixed signal, meaning that at 20% and 50% overlap ratios, the majority of each segment contains clean, non-overlapping speech." Specifically, at ρ = 0.2, the actual overlap constitutes only ~15% of each speaker's segment, leaving ~7 seconds of clean audio that an ASR system can transcribe accurately. The separation module introduces minor artifacts even in the non-overlapping portions, explaining the slightly higher WER. This finding does not undermine the case for separation — at high overlap ratios where separation is needed most, the module provides massive gains — but it correctly calibrates expectations: the pipeline's main value is at ρ ≥ 0.5, where baseline transcription fails entirely.

---

ASR Ensemble Performance (Section 3.4, Table 4)

The ASR evaluation compares single-model Whisper-large-v3 against the three-model ROVER ensemble (Whisper + Canary + Parakeet) on standard benchmarks. The headline is that the ensemble reduces WER across all test sets, with the largest relative gains on clean speech.

On LibriSpeech test-clean, the ensemble achieves WER 2.04% with standard deviation ±6.50%, compared to Whisper's 3.63% ±9.37% — a relative improvement of approximately 43.8%. On LibriSpeech test-other (noisy conditions), the ensemble achieves 3.92% ±8.92% vs. Whisper's 6.26% ±11.63% — a relative improvement of approximately 37.4%. On TEDLIUM3 (spontaneous lecture speech), the ensemble achieves 10.66% ±11.73% vs. Whisper's 12.19% ±12.31% — a relative improvement of approximately 12.6%, notably smaller than on the LibriSpeech splits.

The pattern — larger gains on clean read speech, smaller gains on spontaneous speech — suggests that the ensemble's primary value lies in suppressing rare but severe hallucinations (repetitive loops, fabricated content during silence or noise, word-level substitution errors) rather than in improving performance on consistently challenging acoustic conditions. On LibriSpeech test-clean, Whisper's error rate is already low (3.63%), but the standard deviation (±9.37%) is large relative to the mean, indicating that most utterances are transcribed nearly perfectly while a small fraction suffers from catastrophic failures. The ensemble reduces these tail failures — the WER drops to 2.04% with a reduced standard deviation of ±6.50%. On TEDLIUM3, where spontaneous speech, disfluencies, and variable acoustics challenge all models more uniformly, the three models make similar errors, and voting provides less benefit.

The inference time tradeoff is explicitly quantified: the ensemble requires approximately "three times longer than the baseline," with latency "primarily attributed to the inference speed of Canary, the slowest model among the three." This 3× slowdown (from ~0.36s to ~1.33s per sample in Table 4) is the cost of the hallucination suppression that the ensemble provides. For offline data curation — where transcription runs once and the output feeds into downstream training that may run for days — this tradeoff is strongly favorable: paying a 3× computational premium to prevent hallucination contamination from entering the training corpus is cheap relative to the cost of training a model on corrupted data and discovering the problem only at evaluation time.

The qualitative example cited in Section 3.4 is worth examining closely. The paper reports: "We observed hallucinations in the Whisper outputs, such as repetitive generation (e.g., 'Yeah., Yeah., Yeah...'). We confirmed that our method successfully corrected these errors by selecting the accurate transcript provided by Canary (e.g., 'Yeah, big decision for Dan')." This is a concrete illustration of the diversity-of-failure-modes argument: Whisper hallucinates a repetitive loop during what appears to be a brief pause or non-speech segment; Canary correctly transcribes the actual speech ("big decision for Dan"); majority voting selects the correct transcript because at least one other model (presumably Parakeet) agrees with Canary. The example also demonstrates that the n-gram repetition filter (n=15, count≥5) serves as a backup for cases where ensemble voting alone might not catch a shared hallucination pattern — the repetitive "Yeah." sequence would trigger the repetition filter and cause the sample to be discarded if the ensemble failed to correct it.

---

Full-Duplex-Bench 1.0 Results (Section 3.1, Table 1)

The central validation experiment tests whether Sommelier-processed data improves actual full-duplex model behavior by fine-tuning Moshi on 83 hours of pipeline-processed speech and evaluating on Full-Duplex-Bench 1.0. The results must be interpreted with care because Full-Duplex-Bench metrics are not all oriented in the same direction — some are better when lower, others when higher — and the base Moshi exhibits pathological behavior that artificially inflates certain metrics.

Backchannel handling. The fine-tuned model achieves Backchannel TOR of 0.291 vs. base Moshi's 0.000 — a shift from zero backchannel engagement to active backchannel behavior. The base model's TOR of 0.000 indicates that it never produced a full turn during backchannel-eligible windows, which the paper interprets not as success but as failure to engage: "the base model exhibited notably short latencies simply because it failed to engage in backchanneling or interruption handling, reflecting suboptimal behavior where the model continued speaking regardless of user input." The fine-tuned model's TOR of 0.291 still represents infrequent but non-zero backchanneling. The Frequency metric (backchannels normalized by total audio duration) increases from 0.001 to 0.052 — a substantial relative increase but still a low absolute backchannel rate. The JSD (Jensen-Shannon divergence between model and human backchannel timing) improves from 0.957 to 0.630, indicating that the backchannels produced are more naturally timed relative to human conversational patterns, though still far from human-level timing (JSD of 0 would indicate perfect distributional match). The Candor metric improves from 0.941 to 1.000, indicating the model now successfully responds to all backchannel-eligible moments.

Smooth Turn-Taking. The fine-tuned model achieves Candor TOR of 1.000 (up from 0.265) and Latency of 0.344s (down from 0.765s). The Candor improvement from 0.265 to 1.000 is dramatic — the base model failed to take turns in nearly three-quarters of eligible opportunities, while the fine-tuned model takes every turn — and again reflects the base model's pathological non-responsiveness. The latency improvement from 0.765s to 0.344s represents a shift toward more fluid turn transitions. The TOR metric improves from 1.000 to 0.858, which seems counterintuitive (a decrease in TOR) until one understands that TOR in this context (different from Candor TOR) measures synthetic TOR; the paper's interpretation is that the fine-tuned model produces more human-like turn-taking with fewer inappropriate turn grabs.

User Interruption. The fine-tuned model achieves Interruption TOR of 3.684 (up from 0.257) — an increase of over 14× that the paper explicitly interprets as positive: "the increased latency can be interpreted positively, as it indicates that the model is now actively processing user input and preparing appropriate responses for backchannels and interruptions." The base model's low TOR reflects the fact that it ignored user interruptions entirely, continuing to speak regardless of input. The GPT-4o relevance score improves from 0.257 to 1.065 (on a 0–5 scale), indicating that the fine-tuned model's interruption responses are more semantically relevant to what the user said, though 1.065 out of 5 is still quite low.

Pause Handling. This is the one area where the fine-tuned model does not meaningfully improve. The Synthetic TOR metric is 1.000 for both base and fine-tuned models (both handle pauses without incorrectly treating them as turn boundaries), and Candor TOR is 1.000 (up from 0.985) — a negligible improvement. The paper hypothesizes that the Moshi architecture or the absence of prompt audio (as used in Personaplex; Roy et al., 2026) may fundamentally limit pause handling improvements, placing the burden on model architecture rather than training data.

Interpreting the latency increases as positive. This is perhaps the most counterintuitive result and requires careful framing. The base Moshi model exhibits latency of 0.265s for turn-taking and 0.257s for interruption handling — seemingly "fast" responses. The fine-tuned model's latency increases to 0.344s and 3.684s respectively. The paper argues convincingly that the base model's low latency reflects non-responsive behavior: the model simply continues its own speech without processing user input, resulting in zero computational delay. The fine-tuned model's higher latency reflects actual processing — detecting the user's speech, generating an appropriate response, and timing its delivery appropriately. The increase from 0.257 to 3.684 for interruption handling is particularly stark, but reflects the model shifting from ignoring interruptions entirely to actively listening and formulating responses, which takes time.

---

Full-Duplex-Bench 1.5 Results (Appendix D.3, Tables 10, 11)

The evaluation on Full-Duplex-Bench 1.5 (Lin et al., 2025a) provides a more granular picture of the fine-tuned model's behavior across four distinct overlap scenarios: Background Speech, Talking to Other, User Backchannel, and User Interruption.

Audio quality improvements. Table 10 shows substantial and consistent improvements in post-processing audio quality metrics. Most notably, the SI-SDR for the Background Speech scenario improves from 5.43 dB (base Moshi) to 20.76 dB (fine-tuned) — a 15.3 dB improvement that suggests the fine-tuned model produces much cleaner speech output when background speech is present. The PESQ scores improve across all scenarios: from 2.55 to 3.30 (Talking to Other), from 1.60 to 3.01 (User Backchannel), and from 2.87 to 3.27 (User Interruption). The UTMOS improvements are moderate (e.g., 1.86 to 1.87 for Background Speech, 2.65 to 2.58 for User Interruption), suggesting that while intelligibility metrics (STOI, PESQ) improve substantially, the perceptual naturalness of the output speech is more constrained by the base Moshi architecture. STOI scores improve from 0.79 to 0.98 (Background Speech), from 0.63 to 0.91 (User Backchannel), and from 0.94 to 0.97 (User Interruption), indicating dramatically better intelligibility of the output speech across all overlap scenarios.

Behavioral ratio shifts. The behavior classification ratios at the bottom of Table 10 reveal shifts in how the model handles overlap. The "Background Speech" scenario shows a reduction in "UNK" (unknown/unresponsive) behavior from 0.75 to 0.60, and an increase in "RESP" (responsive) behavior from 0.15 to 0.28. The "User Backchannel" scenario shows the most dramatic behavioral shift: UNK drops from 0.92 to 0.72 (still high, but substantially improved), and RESP increases from 0.01 to 0.08. The model is still far from perfect — it remains unresponsive in 72% of backchannel scenarios — but 8% responsiveness is infinitely better than 1% in a domain where the base model was essentially non-functional.

Latency analysis (Table 11). The latency breakdown separates stop latency (time from user speech onset to model speech cessation) and response latency (time from user speech offset to model speech resumption). The fine-tuned model achieves substantially lower stop latencies across all scenarios: 0.68s vs. 1.02s (Background Speech), 0.82s vs. 1.13s (Talking to Other), 0.70s vs. 1.30s (User Backchannel), 0.89s vs. 1.30s (User Interruption). Response latencies improve even more dramatically: 0.73s vs. 2.90s (Background Speech), 0.84s vs. 3.22s (Talking to Other), 1.12s vs. 2.38s (User Backchannel), 0.66s vs. 1.99s (User Interruption). The response time improvements of 2–4× across all scenarios demonstrate that the fine-tuned model is not just producing better audio quality but also engaging in more responsive turn-taking behavior.

Sample count shifts. An interesting pattern in Table 11 is the shift in the number of samples classified under "Stop" vs. "Resp." For the base model, many scenarios produce more "Stop" samples than "Resp" samples (e.g., Background Speech: 150 Stop, 89 Resp), indicating that the model frequently ceases speaking but then fails to subsequently respond. For the fine-tuned model, this ratio flips dramatically: Background Speech shows 44 Stop samples but 192 Resp samples. This confirms the behavioral interpretation from Section 3.1 — the base model often stops speaking (perhaps detecting speech) but then does nothing, while the fine-tuned model stops speaking and then generates an appropriate response.

---

Pipeline Latency and Scalability (Section 3.5, Table 5)

The latency breakdown quantifies the computational feasibility of the pipeline at production scale. On a single A100 80GB GPU processing a 120-second audio sample:

• VAD + Sortformer: 1.91s (RTF 0.0159)
• SepReformer Separation: 0.15s (RTF 0.0013)
• ASR Ensemble: 13.91s (RTF 0.1159)
• FlowSE Denoising (optional): 4.99s (RTF 0.0416)
• Total: 20.95s (RTF 0.1746)

The ASR ensemble consumes 66.4% of total processing time (13.91s out of 20.95s), making it the clear bottleneck. This is expected — running three large ASR models on each audio segment, with alignment and voting, is inherently more expensive than any other stage. The SepReformer separation is remarkably efficient at 0.15s (0.7% of total time), validating the paper's implicit claim that overlap handling does not impose a substantial computational burden relative to the rest of the pipeline. The total RTF of 0.1746 means audio is processed approximately 5.7× faster than real-time.

With peak GPU memory usage of 23GB (less than one-third of the A100's 80GB capacity), the paper demonstrates that three concurrent processes can run on a single GPU, reducing the effective per-GPU RTF to 0.0443 (the pipeline processes audio approximately 22.6× faster than real-time per GPU). The paper then projects that processing 10,000 hours of audio on eight A100 GPUs would take approximately 55 hours, establishing feasibility for web-scale data curation. The implied throughput is approximately 182 hours of audio processed per GPU per 55-hour window, or about 3.3 hours of audio per GPU-hour — practical for curated fine-tuning datasets (tens to hundreds of hours) and feasible for larger-scale pretraining data (thousands of hours) with sufficient GPU resources.

---

Ablation Studies and Robustness Checks

Model-level ablation: Pyannote 3.1 vs. Sortformer for diarization (Table 2). The choice of diarization model is effectively ablated by direct comparison. Sortformer reduces DER from 8.40% to 7.16%, JER from 17.68% to 14.69%, and — most critically — DER(turn) from 0.051 to 0.006. This is the strongest ablation in the paper because it tests the default community choice (Pyannote) against the alternative (Sortformer) on metrics specifically relevant to the pipeline's intended use case. The result demonstrates that the default assumption — Pyannote is adequate for all speech processing tasks — does not hold for full-duplex data curation, where boundary accuracy and short-utterance handling are paramount.

Separation vs. no-separation for overlapping speech (Tables 3, 6, 7, 8). The overlap separation evaluation is structured as an ablation across nine conditions (3 SIR levels × 3 overlap ratios), comparing baseline (direct extraction from mixed audio), separation (SepReformer with speaker identity matching), and oracle (clean source audio). Across all conditions, separation improves over baseline, with the gap widening at high overlap ratios. At ρ = 1.0 and 0 dB SIR, UTMOS improves from 1.73 to 3.05 (Speaker 1) and 1.67 to 2.99 (Speaker 2), demonstrating that separation transforms unusably degraded audio into fair-to-good quality suitable for training. The ablation also reveals the asymmetry noted earlier: Speaker 2 (the quieter speaker) benefits far more, confirming the intuition that separation is most critical for recovering the speech that would otherwise be masked entirely.

Single-model ASR vs. ensemble ASR (Table 4). The ensemble ASR is ablated by comparing single-model Whisper-large-v3 against the three-model ROVER ensemble. The ensemble reduces WER across all test sets, with the largest relative gains on LibriSpeech test-clean (43.8% relative improvement). The ablation demonstrates that the ensemble's benefit is robust across acoustic conditions (clean speech, noisy speech, spontaneous lecture speech), but is most pronounced where the baseline is already strong, suggesting that the ensemble's primary value is in suppressing rare catastrophic failures (hallucinations) rather than improving average-case transcription accuracy.

Context size for Demucs music separation (Section 2.4). While not a formal ablation table, Section 2.4 describes a finding that amounts to an implicit ablation: feeding full two-minute audio chunks into Demucs yields "substantially better separation performance than processing short segments in isolation." This is an empirical result about how source separation models leverage temporal context — the model can better identify and remove consistent musical elements when it has sufficient surrounding audio to build a statistical model of the music's acoustic properties.

Model selection: SAM-Audio excluded due to latency (Section 2.4). The paper explicitly reports that SAM-Audio (Shi et al., 2025) was considered for music removal but excluded because its RTF of 0.73 on an A100 would make it the dominant computational cost, limiting scalability. This is a practical decision point that amounts to a negative result: a more sophisticated model was rejected because its computational cost outweighed its potential quality benefits for the target throughput requirements.

ReSTˢᵐ revision model results (Appendix K, Figure 16). Not applicable to this paper — the reference example mentions ReSTˢᵐ, but Sommelier does not use or evaluate revision models. There is no ReSTˢᵐ experiment in Sommelier.

Difficulty bin sensitivity (not applicable). The reference example discusses difficulty bin sensitivity for compute-optimal allocation. Sommelier does not employ difficulty estimation or adaptive allocation strategies. The pipeline processes all audio uniformly regardless of conversational difficulty, and no difficulty-dependent ablation is conducted. This is a notable omission: the paper does not investigate whether the pipeline's performance varies with characteristics of the input audio (number of speakers, overlap density, background noise level, recording quality), which would be valuable knowledge for deployment.

Turn-length filtering as implicit ablation (Section 3.1). One of the most practically important findings is not presented as a formal ablation but emerges from the training methodology: long single-speaker turns (over one minute) in the training data cause "unstable loss reduction and degrades performance, causing the model to become unresponsive." The paper's solution — restricting turns to ≤10 seconds and requiring at least three consecutive turns — is essentially a data ablation demonstrating that training data format directly impacts model behavior in non-obvious ways. This is a robustness check in the broadest sense: it shows that the pipeline's output must be further filtered — not just processed — to be effective for training, and that the filtering criteria are specific to the downstream model's training dynamics.

---

Critical Assessment

The Sommelier experiments collectively demonstrate that the pipeline improves intermediate metrics (diarization error, separation quality, ASR accuracy) and that full-duplex model behavior improves when fine-tuned on pipeline-processed data. However, the gap between what the experiments demonstrate and what the paper claims — and the experiments that were not run — deserves careful scrutiny.

Does the pipeline actually produce data that enables full-duplex training? The Moshi fine-tuning experiment (Tables 1, 10, 11) provides affirmative evidence, but the evidence is narrower than it appears. The fine-tuning uses only 83 hours of data — a small amount by pretraining standards, though reasonable for supervised fine-tuning — and evaluates on benchmarks (Full-Duplex-Bench 1.0 and 1.5) that, while designed for full-duplex evaluation, are themselves synthetic test suites rather than real conversational interactions. The paper does not evaluate the fine-tuned model in genuinely interactive settings with human users, nor does it compare against Moshi fine-tuned on an equivalent amount of data processed through a simpler pipeline (e.g., Emilia-style processing without overlap handling). This means the experiments demonstrate that Sommelier-processed data improves over the untrained base model, but cannot isolate how much of the improvement comes from the pipeline's distinctive features (overlap preservation, ensemble ASR, selective music removal) versus merely having 83 hours of additional conversational fine-tuning data regardless of how it is processed.

The strongest evidence for the pipeline's design philosophy would be a controlled experiment comparing Moshi fine-tuned on Sommelier-processed data versus Moshi fine-tuned on the same raw audio processed through a simpler pipeline that discards overlaps (Case 1), uses single-model ASR, and defaults to Pyannote diarization. This experiment would directly test whether preserving overlaps matters — the paper's central claim. The paper does not run this experiment.

Do the component-level improvements aggregate to pipeline-level benefit? The paper evaluates each subsystem independently — diarization accuracy on VoxConverse, separation quality on synthetic LibriSpeech mixtures, ASR accuracy on LibriSpeech and TEDLIUM3 — and each shows improvement over baselines. However, the subsystems are evaluated on different data than the actual pipeline processes (LibriSpeech read speech vs. in-the-wild conversational audio), and the downstream Moshi fine-tuning experiment cannot attribute improvements to specific components. An ablation study that toggles individual pipeline components (e.g., disable overlap separation and use Case 1 instead of Case 4; use single-model Whisper instead of ensemble ASR) and measures downstream Moshi performance would reveal which components matter most for the final model quality. The paper's modular architecture deliberately enables such experiments (Section 2: "each component can be toggled or reconfigured"), but the paper does not report them.

The scale gap between pipeline evaluation and industrial ambition. The paper positions Sommelier as an industrial-scale data processing pipeline, projecting throughput of 10,000 hours in ~55 hours on 8 A100 GPUs. However, the actual validation uses only 83 hours of processed data, and the paper provides no evidence that the pipeline's performance characteristics (error rates, latency) hold at larger scale. Pipeline failures that are rare at 83-hour scale — a diarization model crashing on a particularly challenging recording, an ASR model entering a degenerate state during a long segment, a memory leak in the audio separation module — could become significant at 10,000-hour scale. Large-scale data curation pipelines typically encounter a long tail of edge cases (unusual audio formats, extreme recording conditions, very long or very short utterances) that stress-test robustness in ways that small-scale validation does not.

The pipeline's end-to-end error propagation is not measured. Each component introduces errors: diarization misattributes speakers (DER 7.16% for Sortformer), speech separation introduces acoustic artifacts (WER degradation at low overlap ratios), and ensemble ASR still has non-zero error rates (2.04–10.66% WER across datasets). These errors compound: a diarization error at a turn boundary feeds incorrectly attributed audio to the separation module, which produces artifacted speech, which the ASR system transcribes imperfectly. The paper evaluates each component in isolation but does not measure the end-to-end error rate on the final pipeline output — what fraction of utterance segments in the processed data have incorrect speaker labels, degraded audio quality, or transcription errors? This would be measurable by manually annotating a sample of pipeline-processed data and comparing against ground truth, but the paper does not report such an analysis. Without it, the cumulative quality of the pipeline's output remains unknown.

The absence of human evaluation. All evaluation metrics are automated: WER for ASR, DER/JER for diarization, STOI/PESQ/SI-SDR/UTMOS for audio quality, and Full-Duplex-Bench metrics for conversational behavior. While automated metrics are appropriate for component-level evaluation, the claim that Sommelier produces "high-quality training corpora for full-duplex SLMs" (Section 2) ultimately rests on the quality of the training signal that downstream models receive. A small-scale human evaluation — having annotators listen to pipeline-processed segments and rate their naturalness, speaker clarity, and transcription accuracy — would provide a more direct measure of output quality than the proxy metrics reported. The paper does not include any human evaluation.

Validation on only one downstream model (Moshi) and one benchmark suite (Full-Duplex-Bench). The paper's most important validation experiment tests a single full-duplex model on a single evaluation framework. This leaves open the question of whether Sommelier-processed data would benefit other full-duplex architectures (e.g., GPT-4o-style systems, Personaplex) or generalize to other conversational domains beyond the benchmark scenarios. Full-Duplex-Bench 1.0 and 1.5 are synthetic test suites, not natural conversations, and performance on these benchmarks may not fully capture real-world interaction quality. The paper acknowledges the nascent state of full-duplex evaluation (the benchmarks were released in 2025, contemporaneous with the paper), but the single-model, single-benchmark validation limits the generalizability of the findings.

Practical deployment considerations not addressed. The paper's difficulty estimation cost for compute-optimal allocation is not relevant here, but an analogous concern applies: the pipeline's configuration overhead for a new domain or language is not discussed. Each component has thresholds and hyperparameters (PANNs music probability threshold of 0.3, VAD silence detection sensitivity, Demucs context window size of 2 minutes, overlap separation requirement of >2s reference segments) that were presumably tuned on the authors' target data distribution (English-language podcasts and radio broadcasts). Deploying Sommelier on a new domain (e.g., multilingual call center recordings, courtroom proceedings, multi-party meetings with >2 speakers) would require re-tuning these thresholds, and the paper provides no guidance on sensitivity or tuning methodology.

The positive results are genuine but limited in scope. The improvement in Full-Duplex-Bench metrics (Table 1: Backchannel TOR 0→0.291, Interruption TOR 0.257→3.684) represents a genuine shift from non-functional to partially functional conversational behavior, which is a meaningful improvement. However, the absolute scores remain low: the fine-tuned model's backchannel frequency is 0.052 (producing a backchannel approximately once every 19 seconds of audio), and the GPT-4o relevance score for interruption responses is 1.065 out of 5. The paper frames these as improvements over a non-functional baseline, which is accurate, but they also indicate that 83 hours of Sommelier-processed data is insufficient to produce a model with natural conversational dynamics. This is not a failure of the pipeline — it likely reflects the inherent difficulty of full-duplex interaction — but the paper's rhetoric in the introduction and conclusion ("high-quality training corpora," "accelerate progress toward natural, real-time human–AI interaction") implies a larger impact than the quantitative results actually demonstrate.

Overall, the experiments provide credible evidence that Sommelier improves over simpler processing approaches on component-level metrics and that the processed data can improve full-duplex model behavior relative to an untrained base model. The absence of controlled experiments comparing Sommelier-processed data against data processed through simpler pipelines, the lack of end-to-end error measurement, the small scale of downstream validation (83 hours, single model, single benchmark suite), and the limited absolute performance of the fine-tuned model all mean that the paper's strongest claims about the pipeline's superiority for full-duplex data curation are supported more by the design philosophy than by direct experimental evidence.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in Headline Pipeline Efficiency

The assumption or constraint. The entire pipeline operates uniformly on all input audio — it does not estimate conversational difficulty or adapt its processing strategy per recording. However, this is not itself the limitation. The deeper issue is that the pipeline's validation experiment (Section 3.1) applies domain-specific filtering criteria — restricting turns to ≤10 seconds, requiring at least three consecutive turns, assigning only one speaker to the left channel — that were discovered empirically during training and are not part of the automated pipeline. The paper states in Section 3.1:

"We find that long turn-taking in the training data for Moshi, specifically when a single speaker holds the floor for too long (more than a minute), leads to unstable loss reduction and degrades performance, causing the model to become unresponsive."

And further:

"We selected segments from the Sommelier-processed data where each turn lasted no more than 10 seconds. We defined a valid region as a sequence of at least three consecutive turns, truncating the region if an utterance exceeding 10 seconds appeared."

The consequence. The pipeline produces segments that may be unsuitable for downstream training without additional filtering, and the filtering criteria are not part of the pipeline itself — they were discovered ad hoc during fine-tuning. A practitioner deploying Sommelier to curate training data for a different full-duplex model (not Moshi) or a different domain would not know a priori what turn-length constraints, channel assignments, or minimum-turn-sequence requirements their model needs, and the pipeline provides no mechanism for discovering or enforcing such constraints. The consequence is that the pipeline's output is not directly usable as training data without a manual, model-specific filtering and experimentation step that the paper does not formalize or automate.

What evidence exists in the paper. The evidence is entirely empirical and negative. Section 3.1 reports that including long turns caused "unstable loss reduction and degrades performance." The paper's solution — the 10-second turn limit, three-turn minimum, and channel assignment rules — is presented as a practical finding, not a pipeline feature. Figure 6 shows that the filtered dataset spans diverse domains (Lifestyle 16.6%, Religion/Spirituality 12.3%, Sports 10.3%), but no experiment tests whether the same filtering criteria would be necessary or sufficient for a different downstream model, a different domain, or a different language. The paper does not ablate the filtering criteria — e.g., testing whether 15-second or 5-second turn limits produce different training dynamics — meaning the sensitivity of downstream training to these filtering choices is unknown.

Mitigation status. The paper does not attempt to formalize or automate the filtering criteria. It presents the findings as discovered knowledge (which they are — this is genuinely useful information for the community) but does not integrate them into the pipeline as a configurable or learnable component. The paper suggests no future work on automating data suitability filtering for downstream training. A practitioner would need to replicate the discovery process — train a model, observe training instability, hypothesize data quality causes, filter, retrain — for each new model or domain, which partially undermines the pipeline's claimed value as a turnkey data curation solution.

---

Only One Downstream Model and One Benchmark Family Are Tested

The assumption or constraint. The paper validates the pipeline's utility by fine-tuning exactly one downstream model — Moshi (Défossez et al., 2024) — and evaluating on one benchmark family — Full-Duplex-Bench 1.0 and 1.5 (Lin et al., 2025b,a). The paper implicitly assumes that Sommelier-processed data is broadly useful for full-duplex SLM training, but provides evidence from only this single model-benchmark pair. The paper does not claim broader validation, but the introduction frames the contribution as addressing "community-wide data scarcity" for full-duplex models generally.

The consequence. It is unknown whether Sommelier-processed data would benefit other full-duplex architectures. Moshi is a specific architecture using a multi-stream neural audio codec with text-audio alignment — other full-duplex approaches (e.g., GPT-4o-style systems (Hurst et al., 2024), Personaplex (Roy et al., 2026), or cascaded ASR-LLM-TTS systems with duplex modifications) may have different data format requirements, different sensitivity to separation artifacts, or different training dynamics that respond differently to Sommelier-processed data. A practitioner building a non-Moshi full-duplex system cannot conclude from this paper that Sommelier-processed data will improve their model's performance.

Furthermore, Full-Duplex-Bench 1.0 and 1.5 are synthetic test suites — they evaluate model behavior in controlled, single-turn-mini-scenarios rather than extended natural conversations. The benchmarks measure specific capabilities (pause handling, backchanneling, turn-taking, interruption) in isolation. A model that scores well on Full-Duplex-Bench may still produce unnatural conversational flow in multi-minute interactions where context accumulation, topic drift, and emotional dynamics matter. The paper does not evaluate the fine-tuned model in genuinely interactive settings with human users or on natural conversational transcripts.

What evidence exists in the paper. Sections 3.1, D.2, and D.3 present the Full-Duplex-Bench results. Table 1 shows improvements on Backchannel, Turn-Taking, and Interruption metrics, but the absolute scores remain low (backchannel frequency 0.052, GPT-4o relevance score 1.065/5.0). Tables 10 and 11 show audio quality and latency improvements on Full-Duplex-Bench 1.5. No experiments test other full-duplex models, other evaluation protocols, or interactive human evaluation. The paper does not compare Moshi fine-tuned on Sommelier-processed data against Moshi fine-tuned on the same 83 hours of data processed through a simpler pipeline (e.g., Emilia-style), which would isolate the pipeline's distinctive contributions.

Mitigation status. The paper does not address this limitation. The conclusion states that the pipeline is released "to support reproducible industrial research and to accelerate progress toward natural, real-time human–AI interaction" (Section 4), implying that community adoption will provide the missing multi-model validation. This is a reasonable open-source strategy — release the tool, let the community test it broadly — but it means the paper's claims of pipeline utility rest on a single validation point. There is no suggestion of future work to test additional model architectures or evaluation protocols.

---

End-to-End Output Quality Is Not Measured — Only Component-Level Metrics Are Reported

The assumption or constraint. The paper evaluates each pipeline subsystem independently — diarization on VoxConverse (Table 2), speech separation on synthetic LibriSpeech mixtures (Tables 3, 6, 7, 8), and ASR on LibriSpeech and TEDLIUM3 (Table 4) — but never measures the cumulative error rate of the full pipeline on its actual output data. Each component independently shows better-than-baseline performance on its respective test set, but these test sets differ from the in-the-wild conversational audio the pipeline processes (VoxConverse YouTube videos, LibriSpeech audiobooks, TEDLIUM3 lectures vs. the podcasts, radio broadcasts, and interviews that Sommelier targets). The paper implicitly assumes that component-level improvements aggregate to pipeline-level quality, and that performance on standard benchmarks transfers to in-the-wild data.

The consequence. Component errors compound, and the paper provides no estimate of the end-to-end error rate. A Sortformer diarization error (DER 7.16%) that misattributes a speaker at a turn boundary will feed incorrectly labeled audio to the overlap separation module. The separation module will attempt to separate speech that may not actually overlap, producing artifacted audio. The ensemble ASR will transcribe this artifacted audio, potentially producing lower-quality transcriptions than if the diarization had been correct. Further downstream, a training example with a misattributed speaker label, separation artifacts, and a slightly degraded transcript may be worse for model training than if that segment had been discarded entirely.

The paper's component-level metrics also use different data distributions than the pipeline's actual inputs. The overlap separation evaluation uses synthetic mixtures from LibriSpeech — read audiobook speech with controlled overlap ratios and SIR levels. Real podcast overlaps involve spontaneous speech, variable recording quality, background noise, and acoustic environments that differ substantially from clean audiobook recordings. The ASR evaluation uses LibriSpeech and TEDLIUM3, which do not contain the music, overlapping speech, and variable recording conditions of in-the-wild sources. Performance on these benchmarks may not transfer to the pipeline's actual operating regime.

What evidence exists in the paper. The paper's only evidence about output quality comes from the downstream Moshi fine-tuning experiment (Section 3.1), which demonstrates that 83 hours of pipeline-processed data improves full-duplex behavior. This is indirect evidence that the cumulative pipeline output is useful, but it does not measure the end-to-end error rate. A controlled experiment would manually annotate a sample of the 83-hour training corpus — checking speaker labels, transcript accuracy, and audio quality for a few hundred utterance segments — and report the fraction that are correct. The paper does not conduct any such annotation or report any end-to-end metric. The pipeline's output JSON format (Appendix F) includes individual ASR model transcripts and flags for separation and music removal, suggesting that the authors designed the output to enable such quality assessment, but they do not report summary statistics on these quality indicators across the 83-hour dataset.

Mitigation status. The paper does not acknowledge this gap. The individual component evaluations are presented as sufficient evidence of pipeline quality. A practitioner using Sommelier would need to conduct their own end-to-end validation on their target domain — a reasonable expectation for production deployment, but one that makes the paper's claims of pipeline quality less directly actionable. The modular, togglable architecture (Section 2) does enable component-level quality diagnostics (if separation artifacts are suspected, disable separation and compare), but the paper provides no methodology or guidance for conducting such diagnostics.

---

The Pipeline Is Validated Only on English-Language Conversational Audio

The assumption or constraint. Sommelier is designed, evaluated, and validated exclusively on English-language audio. All component evaluations use English-language benchmarks: VoxConverse (English YouTube videos), LibriSpeech (English audiobooks), TEDLIUM3 (English lectures), and Full-Duplex-Bench (English conversational scenarios). The ASR models (Whisper-large-v3, Canary-1b-v2, Parakeet-tdt-0.6b-v3) are multilingual, but their performance on non-English speech — and specifically on non-English conversational speech with overlaps — is not evaluated. The diarization model (Sortformer) is evaluated only on English-language data. The speech separation model (SepReformer) and music removal model (Demucs) are acoustic models that are in principle language-agnostic, but their training data distributions and failure modes on non-English speech are unknown. The paper makes no claims about multilinguality, but also provides no caution about English-only scope.

The consequence. A practitioner seeking to curate full-duplex training data in a non-English language — or in a code-switched, multilingual conversational setting — cannot infer from this paper that Sommelier will produce usable output. The failure modes are predictable but unquantified: the ASR ensemble's word-level voting assumes the three models produce comparable transcriptions, which may not hold for low-resource languages where model quality varies dramatically; diarization models trained primarily on English speech may misattribute speakers with unfamiliar phonetic patterns; speech separation models may confuse languages with similar acoustic profiles. The consequence is that the paper's contribution, while valuable for the English-speaking research community, does not address the global need for full-duplex training data — a gap that is particularly acute given that conversational dynamics (overlap patterns, backchannel norms, turn-taking timing) vary cross-culturally.

What evidence exists in the paper. The paper provides no multilingual evaluation. The language field in the pipeline output JSON (Appendix F) indicates "language": "en" for all example segments, and no other language codes appear. The dataset statistics (Figure 6) do not include a language breakdown, implying the 83-hour training corpus is entirely or predominantly English. The paper does not discuss multilinguality as a limitation or future work direction.

Mitigation status. Not addressed. The paper makes no claims about multilinguality, but also issues no scope limitation warning. A practitioner working in a non-English language would need to independently evaluate every pipeline component — ASR accuracy, diarization quality, separation fidelity, music detection reliability — on their target language before trusting the pipeline's output for training. The modular architecture does allow swapping in language-specific models (e.g., replacing Whisper with a target-language ASR system), but the paper provides no guidance on how to do so or how the pipeline's design assumptions might need to change for non-English conversational dynamics.

---

The Pipeline Cannot Recover Speech Quality Lost from Artificial Separation — And This Loss Is Not Bounded

The assumption or constraint. The speech separation module (SepReformer; Shin et al., 2024) produces estimated clean speech streams from overlapping audio. The paper acknowledges in the Limitations section:

"Although our overlap separation module effectively disentangles simultaneous speakers from single-stream recordings, the resulting audio fidelity is inevitably slightly inferior to datasets that are originally recorded with distinct, isolated channels (Oracle), as the artificial separation process may introduce minor acoustic artifacts."

The key word is "slightly" — the paper characterizes the degradation as minor, but the evidence shows it is condition-dependent and can be significant.

The consequence. The separation module introduces acoustic artifacts whose severity depends on the overlap ratio, SIR, and acoustic similarity of the speakers. Table 3 shows that at ρ = 1.0 and 0 dB SIR, the separation module achieves UTMOS of 3.05 (Speaker 1) and 2.99 (Speaker 2), compared to the Oracle's 3.81 and 3.86. The gap from Oracle — approximately 0.8–0.9 UTMOS points — represents a non-trivial degradation in perceptual naturalness. At ρ = 1.0 and 10 dB SIR (the most favorable condition), the gap narrows to ~0.7 UTMOS points for Speaker 1 and ~1.2 for Speaker 2. These numbers are from synthetic LibriSpeech mixtures with controlled acoustic conditions — real-world overlaps with background noise, room reverberation, and dissimilar microphone qualities would likely produce larger Oracle-Separation gaps.

More critically, the separation artifacts may introduce systematic acoustic signatures that downstream models could learn as spurious features. If the separation model consistently produces a particular spectral distortion when separating female-male overlaps, a downstream SLM trained on these separated examples might learn to associate that distortion pattern with overlapping speech, potentially affecting its own speech generation quality during simulated full-duplex interactions. The paper does not investigate whether separation artifacts manifest as trainable patterns that downstream models might exploit in undesirable ways.

What evidence exists in the paper. The UTMOS gaps in Tables 3 and 8 quantify the perceptual degradation. At ρ = 1.0 and 0 dB SIR, the WER for Speaker 2 under separation is 0.138 vs. Oracle's 0.048 (Table 6) — nearly 3× higher. The STOI gap (Table 7) is narrower (e.g., 0.918 vs. 1.000 at 0 dB SIR, ρ = 1.0 for Speaker 1), suggesting intelligibility is largely preserved even when naturalness degrades. The paper does not investigate whether separation artifacts propagate as trainable features to downstream models. The Moshi fine-tuning experiment (Section 3.1) demonstrates that the separated data is useful — the model improves over the untrained baseline — but does not compare against a model trained on natively multi-channel data to quantify the performance cost of separation artifacts.

Mitigation status. The paper acknowledges the limitation in the Limitations section but makes no attempt to mitigate it. There is no discussion of whether improved separation models (beyond SepReformer) could close the Oracle gap, whether domain-specific fine-tuning of the separation model on the target audio distribution would help, or whether downstream models are robust to the specific types of artifacts that SepReformer produces. The paper treats the Separation-Oracle gap as an inherent cost of the single-stream-to-multi-stream approach, which is accurate, but provides no guidance on when this cost becomes prohibitive. A practitioner whose target domain involves heavily overlapping speech (e.g., heated debates, multi-party meetings with frequent interruptions) would need to independently assess whether the separation quality is sufficient for their use case.

---

Selective Background Music Removal Lacks Sensitivity Analysis for the Detection Threshold

The assumption or constraint. The pipeline applies Demucs music removal only to segments where PANNs (Kong et al., 2020) estimates a background music probability exceeding a fixed threshold of 0.3 (Section 2.4). The paper provides no ablation, sensitivity analysis, or domain-specific tuning of this threshold. The implicit assumption is that 0.3 is a reasonable universal threshold, balancing the risk of missed music (false negatives) against the risk of unnecessary speech degradation from Demucs processing (false positives).

The consequence. The 0.3 threshold directly controls two competing failure modes that affect downstream data quality. If the threshold is too high (e.g., 0.7), segments with quiet or intermittent background music will pass through unprocessed, and the music-contaminated speech will enter the training corpus — potentially teaching the downstream SLM to associate musical timbres with speech patterns or to generate music-like artifacts during speech synthesis. If the threshold is too low (e.g., 0.1), segments without music will be unnecessarily processed by Demucs, introducing separation artifacts that degrade speech quality — as the paper acknowledges: "since music removal can degrade speech quality, we selectively apply it only to segments identified by PANNs." The optimal threshold depends on the PANNs model's calibration on the target audio distribution (podcast music vs. radio drama background scores vs. interview ambient noise), the Demucs model's artifact profile on clean speech, and the downstream model's sensitivity to music contamination vs. separation artifacts — none of which are characterized in the paper.

The paper also notes a context-size effect for Demucs — "feeding the entire audio context into Demucs yields substantially better separation performance than processing short segments in isolation" — but does not specify the minimum effective context length or investigate whether the two-minute chunk size is universally appropriate or domain-dependent. For audio sources with very short music segments (e.g., podcast stingers lasting 2–3 seconds), the two-minute context window might include mostly non-music audio, potentially reducing separation effectiveness.

What evidence exists in the paper. None. The paper states the threshold value (0.3) and the context window size (2 minutes) as implementation details without ablation or justification. The pipeline output JSON (Appendix F) includes a "demucs": true/false flag per segment, enabling post-hoc analysis of how frequently Demucs was applied, but the paper does not report these statistics for the 83-hour training corpus. Without knowing what fraction of segments triggered Demucs processing, a practitioner cannot assess whether the threshold is operating in a reasonable regime for their domain.

Mitigation status. The paper does not address this limitation or suggest future work on threshold tuning. The modular architecture allows swapping PANNs for a different music detection model or adjusting the threshold, but the paper provides no methodology for selecting an appropriate threshold for a new domain. A practitioner deploying Sommelier on e.g., call center recordings (typically music-free) vs. music radio shows (nearly continuous background music) would need to independently calibrate this parameter, potentially through trial and error on downstream model performance — an expensive and indirect optimization loop.

7. Implications and Future Directions

How This Work Changes the Landscape

Sommelier does not advance a new model architecture or training algorithm—the pipeline composes existing, off-the-shelf models. Its impact is infrastructural: it establishes that the data curation bottleneck for full-duplex speech models is tractable with current tools, provided those tools are composed with the right design philosophy. This is less a paradigm shift than a methodological correction to the speech data processing community's default assumptions about what constitutes "clean" data.

The correction is this: the standard pipeline design principle—treat overlapping speech as noise to be excised—was inherited from ASR and TTS use cases where it was entirely appropriate. ASR systems optimize for transcription accuracy, and overlapping speech degrades WER; TTS systems need isolated single-speaker utterances to learn clean voice synthesis. These applications legitimately benefit from discarding overlap. But full-duplex SLMs have the opposite data requirement: they need overlapping speech as training examples precisely because their task is to produce and understand speech during overlaps. A pipeline optimized for ASR or TTS systematically destroys the training signal that a full-duplex model needs most.

Sommelier's contribution is to operationalize this insight as an end-to-end, open-source pipeline with measurable component-level performance and a downstream validation experiment. The paper's most understated finding is not any particular metric but the demonstration that the design philosophy—preserve overlap, separate it, transcribe both streams, filter hallucinations—is practically achievable at scale. Table 5's latency breakdown shows that the overlap separation component adds only 0.0013 RTF to the pipeline, meaning the principled choice to handle overlaps properly rather than discard them carries essentially zero computational penalty versus the dominant ASR stage. This removes a potential objection that principled overlap handling is too expensive for production use.

The paper also reconciles a tension that has been implicit in the speech processing community's tool choices. Pyannote 3.1 became the default diarization model through a combination of strong benchmark performance, open-source availability, and integration with a broader ecosystem—a virtuous cycle of adoption. But the paper's Table 2 demonstrates that Pyannote's error rate on turn boundaries (DER(turn) = 0.051) is nearly an order of magnitude worse than Sortformer's (0.006), and that this gap matters specifically for conversational data where turn boundaries encode the dynamics that full-duplex models need to learn. The lesson is not "always use Sortformer"—it is that default tool choices optimized for general benchmarks can be silently misaligned with specialized downstream requirements, and that data curation pipelines for new model paradigms (full-duplex SLMs) need to re-examine component choices from first principles rather than inheriting them from adjacent fields (ASR, TTS).

Looking forward, Sommelier makes several research directions more attractive. The demonstration that single-stream recordings can be converted into separated, transcribed conversational data at practical throughput makes web-scale full-duplex data curation a realistic engineering target rather than a speculative goal. The pipeline's modular, togglable architecture (Section 2) enables controlled experiments that isolate the contribution of individual processing steps—experiments that were previously infeasible without building a complete pipeline from scratch. The discovery that downstream training dynamics are sensitive to data formatting details like turn length (Section 3.1) opens a new axis of data quality research that the speech SLM community has barely begun to explore.

What becomes less attractive, or at least more difficult to justify without evidence, is the dismissal of overlap as noise that has been the default in speech data pipelines. A paper proposing a new speech dataset or processing pipeline that discards overlapping speech without discussing the implications for full-duplex training would now need to contend with Sommelier's demonstration that overlap can be preserved at minimal computational cost. The burden of proof has shifted: the default should be to preserve conversational dynamics unless there is a specific, justified reason to remove them. Sommelier does not make overlap preservation mandatory—there are legitimate use cases where it is unnecessary or undesirable—but it makes the cost of discarding overlap visible and the alternative demonstrably feasible.

---

Follow-Up Research This Work Enables

Controlled ablation of pipeline components on downstream full-duplex behavior. The paper's central validation experiment (Section 3.1) demonstrates that Moshi fine-tuned on the full Sommelier pipeline improves over the untrained base model. But the experiment cannot attribute improvements to specific pipeline design choices. A direct follow-up would train Moshi on three versions of the same 83-hour dataset: (1) full Sommelier pipeline, (2) Sommelier with overlap separation disabled (using Case 1 from Figure 5, discarding overlaps), and (3) Sommelier with single-model Whisper ASR instead of ensemble ASR. Evaluating all three on Full-Duplex-Bench 1.0 and 1.5 would quantify how much of the performance gain comes from overlap preservation versus hallucination suppression versus the baseline benefit of having additional conversational fine-tuning data. This experiment would directly test the paper's central design claim—that preserving overlaps matters—and would produce a cost-benefit analysis for each component that guides practitioners deciding which pipeline stages to enable for their specific use case.

Training data format sensitivity analysis for full-duplex SLMs. The paper's discovery that turn length exceeding one minute causes training instability in Moshi (Section 3.1) is a single data point that hints at a broader, underexplored question: what properties of conversational training data affect full-duplex model training dynamics? A systematic study would sweep turn-length limits (5s, 10s, 15s, 30s, no limit), minimum consecutive turns (1, 2, 3, 5), speaker balance (equal vs. skewed speaker proportions), and overlap density (fraction of segments containing overlap) across multiple full-duplex architectures (Moshi, open-source alternatives as they emerge) and measure both training stability (loss curves, gradient norms) and downstream interaction quality (Full-Duplex-Bench, interactive human evaluation). This would transform the ad-hoc filtering criteria from Section 3.1 into a principled understanding of data quality requirements for full-duplex training, analogous to how the text LLM community has studied the effects of data composition, sequence length, and repetition on pretraining stability (though for a fundamentally different modality and task). Sommelier's modular architecture and JSON output format make it practical to generate systematically varied training datasets for such a study.

End-to-end error propagation measurement. The paper evaluates each pipeline component independently on standard benchmarks (VoxConverse for diarization, LibriSpeech for ASR and separation, TEDLIUM3 for ASR), but never measures the cumulative error rate on the pipeline's actual output. A crucial follow-up would randomly sample ~500 utterance segments from a Sommelier-processed dataset (drawn from the same podcast/radio distribution used for the Moshi fine-tuning), have human annotators verify speaker labels, transcription accuracy, and audio quality for each segment, and report the fraction of segments that are fully correct versus partially degraded versus unusable. This would produce an end-to-end quality estimate that no component-level metric can provide: what percentage of the training examples fed to the downstream model are actually correct? Sub-analyses would trace errors back to their source component—is the dominant failure mode diarization errors propagating to mislabeled separation, or ASR hallucinations surviving ensemble filtering, or separation artifacts degrading transcription? This measurement would also reveal whether the error sources are independent (errors from different components affect different segments) or compounding (errors cascade through the pipeline), which has direct implications for where to invest improvement effort.

Cross-lingual and cross-cultural generalization of pipeline components. The paper validates Sommelier exclusively on English-language audio, but full-duplex interaction norms—backchannel timing, interruption patterns, overlap frequency, acceptable pause durations—vary substantially across languages and cultures. A cross-lingual stress test would process conversational audio in 3–5 typologically diverse languages (e.g., Japanese with its particle-based backchannel system, Italian with high overlap frequency, Finnish with long pause tolerances) through the pipeline and measure: (a) whether Sortformer's diarization accuracy degrades on non-English speech (since the model's training data distribution is not characterized in the paper), (b) whether the Whisper+Canary+Parakeet ensemble maintains its hallucination-suppression advantage across languages, (c) whether SepReformer's separation quality is language-independent (it should be, since separation is acoustic, but cross-lingual artifacts are possible), and (d) whether a downstream full-duplex model trained on Sommelier-processed data in Language X exhibits culturally appropriate conversational timing. This study would determine whether Sommelier is genuinely "community-wide" infrastructure (as the paper claims) or English-only infrastructure requiring substantial adaptation for global use.

Training a full-duplex model from scratch on web-scale Sommelier-processed data. The paper's Moshi fine-tuning experiment uses 83 hours of data—enough for supervised fine-tuning but orders of magnitude less than what full-duplex pretraining might require. A scaling experiment would process 5,000–10,000 hours of in-the-wild conversational audio through Sommelier (the paper projects this is feasible in ~55 hours on 8 A100 GPUs; Section 3.5) and use the resulting data to pretrain or extensively fine-tune a full-duplex model, then evaluate whether conversation quality continues to improve with data scale or plateaus. This is the experiment that would determine whether Sommelier truly addresses the "community-wide data scarcity" identified in the introduction—if a model trained on 10,000 Sommelier-processed hours achieves conversational quality approaching human-level on Full-Duplex-Bench or interactive evaluation, the pipeline is a validated solution to the data bottleneck. If performance plateaus at a few hundred hours, the bottleneck lies elsewhere (model architecture, training objective, or the inherent difficulty of full-duplex interaction), and Sommelier's value is limited to fine-tuning rather than pretraining.

---

Practical Applications and Downstream Use Cases

Curating fine-tuning data for open-source full-duplex models. The most immediate practical application is using Sommelier to generate supervised fine-tuning datasets for Moshi and future open-source full-duplex models. The paper demonstrates that 83 hours of Sommelier-processed data measurably improves Moshi's backchannel, turn-taking, and interruption handling (Table 1: Backchannel TOR from 0.000 to 0.291, Interruption TOR from 0.257 to 3.684). A practitioner with access to domain-specific conversational audio—call center recordings, meeting transcripts, podcast archives, interview collections—can run Sommelier on 50–200 hours of this data and fine-tune an open-source full-duplex model for their domain. The pipeline's RTF of 0.1746 on a single A100 (Table 5) means processing 200 hours costs approximately 35 GPU-hours—a one-time curation cost that is negligible compared to the fine-tuning compute. The fine-tuned model would inherit domain-appropriate conversational norms (e.g., a model fine-tuned on customer service calls would learn service-appropriate backchannel frequency and interruption handling) that a generic model would lack. This is the speech-domain analog of how practitioners currently fine-tune text LLMs on domain-specific corpora, but with the additional requirement that the corpus must be formatted as separated, transcribed, utterance-level segments with speaker labels—exactly the output format Sommelier produces.

Building evaluation datasets for full-duplex benchmarking. Full-Duplex-Bench 1.0 and 1.5 are synthetic evaluation suites designed to test specific conversational capabilities in controlled scenarios. While useful for standardized comparison, they do not evaluate models on natural conversational flow over extended interactions. Sommelier enables the creation of held-out evaluation sets from real conversational audio by processing long-form conversations (e.g., 30-minute podcast episodes, hour-long meeting recordings) and reserving segments for evaluation. Because Sommelier produces word-level timestamps and speaker labels (Appendix F, JSON output format), an evaluation set can include metrics beyond those in Full-Duplex-Bench: turn-taking latency distribution, backchannel timing accuracy relative to human ground truth, interruption appropriateness (does the model interrupt at conversationally appropriate moments?), and long-range conversational coherence (does the model maintain topic and speaker-state awareness across multiple turns?). An evaluation set constructed from 20–50 hours of diverse conversational audio would complement the synthetic benchmarks and provide a more ecologically valid measure of full-duplex model quality.

Data generation for self-improving spoken dialogue systems. The paper's validation focuses on supervised fine-tuning of a pretrained full-duplex model, but the same data could enable iterative self-improvement loops similar to those used in text-based LLM alignment (RLHF, constitutional AI). A full-duplex model deployed in an interactive setting could record its conversations (with user consent), and periodic batches of recorded interactions could be processed through Sommelier to create new training examples that include the model's own behavior during overlaps, its backchannel timing, and its interruption responses. Fine-tuning on this self-generated data would create a feedback loop where the model's conversational behavior is refined based on its actual interaction patterns. Sommelier's overlap handling is particularly important for this use case because real human-model conversations will contain overlaps and interruptions that a simpler pipeline would discard, and the model needs to learn from how it handled (or mishandled) these moments. The pipeline's throughput (10,000 hours in ~55 hours on 8 A100 GPUs; Section 3.5) makes it feasible to process conversation logs at the scale of a deployed service, not just a one-time curation effort.