Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement

ArXiv: 2603.02641

🎯 Pitch

Using time-shifted anechoic clean speech—instead of conventional early reflections—as the dereverberation target actually boosts perceptual quality and ASR accuracy, overturning a long-held assumption in the field. A simple two-stage pipeline then chains a frozen regression model with a GAN-based residual corrector to nail the elusive distortion–perception tradeoff, while aggressive data curation by quality scoring proves that smaller, cleaner datasets dramatically outperform massive uncurated ones. The resulting model achieves state-of-the-art performance on the URGENT 2025 benchmark and, crucially, even improves the training data for downstream TTS systems.

---

1. Executive Summary

This paper systematically analyzes three underexplored design choices in universal speech enhancement (USE): training target selection, the fidelity–perceptual quality trade-off, and training data curation, using the URGENT 2025 Challenge benchmark with USEMamba-based architectures. The authors propose replacing the conventional early-reflected speech dereverberation target with time-shifted anechoic clean speech, introduce a theoretically grounded two-stage framework that chains a frozen regression model with a residual Wasserstein GAN to achieve minimal distortion under a perceptual-quality constraint (operationalized as GAN-based correction of over-smoothed regression outputs), and demonstrate that filtering training data by VQScore quality estimation (threshold of 0.65) yields better performance than training on larger uncurated corpora. The combined approach achieves state-of-the-art non-intrusive quality scores on the URGENT 2025 non-blind test set — DNSMOS of 3.26, NISQA of 4.12, and UTMOS of 2.80 — alongside a character accuracy of 89.88%, while exhibiting strong language-agnostic generalization, establishing that high-quality training targets and data curation are critical enablers for USE models that preserve fidelity under diverse degradation conditions.

2. Context and Motivation

The Core Problem: Universal Speech Enhancement Has Three Unresolved Bottlenecks

Universal speech enhancement (USE) represents an ambitious goal: build a single model that can clean up speech degraded by any kind of distortion — background noise, room reverberation, audio clipping, bandwidth limitations, codec compression artifacts, network packet loss, or wind noise — while simultaneously handling multiple languages and multiple sampling rates. This is not a niche academic exercise. Real-world speech recordings arrive in wildly heterogeneous conditions: a voicemail recorded in a reverberant kitchen at 8 kHz, a conference call with packet loss artifacts, a podcast recorded too hot and clipped, or crowdsourced training data for text-to-speech (TTS) systems captured on consumer microphones with electrical hiss. A model that can handle all of these scenarios without being explicitly told which degradation is present has immediate practical value for communication systems, hearing assistance devices, media production, and — as the paper demonstrates in Section 3.9 — cleaning noisy data for downstream TTS training.

The URGENT 2025 Challenge (Saijo et al., 2025) formalized this problem by providing 2,500 hours of multi-condition training data covering seven distortion types across five languages (English, German, French, Spanish, Chinese) and seven sampling rates (8 to 48 kHz). This challenge attracted significant participation, with top systems employing increasingly complex multi-stage architectures. Yet despite this focused effort, the paper identifies three bottlenecks that prior work has largely sidestepped rather than systematically addressed.

Gap 1: We Use the Wrong Target for Dereverberation

Of the seven distortion types in the URGENT benchmark, six are trained with anechoic (reverberation-free) clean speech as the target. The seventh — reverberation — uses a different target: early-reflected speech. This is speech convolved only with the first 50 ms of the room impulse response (RIR), preserving the direct sound path and early reflections while removing only the late reverberation tail.

Why this convention? The standard explanation, articulated across multiple prior works, is that early reflections are "much harder to remove" and that attempting to do so "leads to excessive artifacts in the enhanced speech" (Valin et al., 2022; Zhou et al., 2023; Zhao et al., 2020). The concern is that dereverberation models, when asked to strip away everything including early reflections, introduce processing artifacts that sound worse than the early reflections themselves. This makes intuitive sense: early reflections arrive within tens of milliseconds of the direct sound and blend perceptually with it, so aggressive removal might damage the signal.

The paper challenges this assumption directly. Their counterargument is not that early reflections are easy to remove, but that the real difficulty is misalignment, not the reflections themselves. When sound travels from a source to a microphone, there is a time delay — typically 5–30 ms — before the direct sound arrives. This is the direct-path time shift $n_0$ in Equation 1. If you use unshifted anechoic clean speech $s[n]$ as the target for a reverberant input $y[n] = s[n] \ast r[n]$, the model must implicitly estimate $n_0$ to align the output with the target. This is a hard problem, and Figure 3(b) confirms that using $s[n]$ as the target yields the worst performance.

The conventional "solution" is to use early-reflected speech $s_e[n] = s[n] \ast \delta[n-n_0] \ast r_e[n]$ as the target. This preserves the direct-path delay through $\delta[n-n_0]$ — so the model doesn't need to estimate it — but also preserves the early reflections $r_e[n]$. The paper argues this is throwing the baby out with the bathwater. The key insight is that removing early reflections is hard because of the alignment problem, not because early reflections are inherently difficult to eliminate. If you give the model a target that is time-shifted to match the direct path — $s[n-n_0]$ — you eliminate the misalignment without preserving any of the early reflection coloration. Critically, $n_0$ can be estimated straightforwardly from the RIR as $\arg\max_n |r[n]|$, so it doesn't require the model to learn it from the mixture.

The paper frames this as the first systematic evaluation of dereverberation targets at scale under diverse degradation conditions. Prior studies explored similar time-shifted targets in narrower settings (Delfarah et al., 2020; Wang et al., 2021), but none had tested the approach in a universal enhancement setting with seven simultaneous distortion types. The claim is that the early-reflection convention has been accepted on the basis of narrow experiments that conflated alignment difficulty with reflection removal difficulty.

Gap 2: The Distortion–Perception Trade-off Is Acknowledged But Not Theoretically Grounded in USE Architectures

The tension between signal fidelity and perceptual quality is a fundamental principle in signal restoration, formalized by Blau and Michaeli (2018) as the distortion–perception trade-off: you cannot simultaneously minimize distortion (measured by metrics like mean squared error) and maximize perceptual quality (measured by the distance between the distribution of enhanced outputs and the distribution of real clean speech). Improving one necessarily degrades the other beyond a certain point. This is not a failure of engineering but a mathematical consequence of the fact that the posterior mean — the minimum MSE estimator — is an average over multiple possible clean signals, and averaging produces blur.

In speech enhancement, this manifests in two well-known failure modes:

• Regression models (trained with L1/L2 losses) output the conditional expectation $\mathbb{E}[s \mid y]$. When the degraded input $y$ contains little information about the true clean signal $s$ — as happens with severe packet loss, extreme bandwidth limitation, or very low SNR — the conditional expectation collapses toward the prior mean $\mathbb{E}[s]$. If the prior is multimodal (e.g., many possible phonemes could follow a given context), the mean blends these modes, producing over-smoothed, muffled output that sounds unnatural even if it roughly preserves the linguistic content.

• Generative models (trained adversarially or via diffusion/flow matching) sample from the conditional distribution $p(s \mid y)$. When $y$ is uninformative, this collapses to sampling from the prior $p(s)$, which can produce natural-sounding but hallucinated content — the model invents phonemes, words, or speaker characteristics that were not in the original signal. The URGENT Challenge evaluation specifically penalizes this: generated speech must match the original speaker and content, not just sound plausible.

Prior work in the URGENT Challenge attempted to address this through ad-hoc multi-stage combinations. Sun et al. (2025) proposed channel-mixing and progressive block extension for regression; Chao et al. (2025) combined regression and generative outputs via energy-based selection; Rong et al. (2025) used three stages (filling, separation, restoration); Goswami and Harada (2025) fused regression with token-sampling generation; and Le et al. (2025) employed four stages (declipping, packet loss compensation, separation, spectral inpainting). The paper characterizes this proliferation of stages as evidence that the field lacks a principled answer to which combination strategy is optimal. Each team independently arrived at their own multi-stage recipe without theoretical justification for the number of stages, their ordering, or how they should interact.

The paper positions itself as providing exactly this theoretical grounding. Drawing on Freirich et al. (2021), which extended the distortion–perception trade-off to Wasserstein space, the authors derive that the minimal MSE under a perfect-perception constraint is achieved by optimally transporting the posterior mean toward the true data distribution (Equation 9 in Appendix A.1). This isn't just a qualitative suggestion — it specifies the structure that an optimal two-stage system should have: first compute the posterior mean (regression), then apply an optimal transport map from the posterior mean distribution $p_{s^*}$ to the clean speech distribution $p_s$. The paper operationalizes this transport using a Wasserstein GAN (Arjovsky et al., 2017) with a crucial design choice: a residual connection from the regression output to the final output, which forces the GAN to focus correction only on regions where the regression model's output diverges from realistic speech statistics.

This matters because it turns architecture design from an empirical search over stage counts and combination heuristics into a theoretically justified two-stage pipeline. The paper explicitly contrasts this with prior multi-stage architectures and claims the framework "eliminates the complex heuristics of prior multi-stage methods" (Section 2.2).

Gap 3: Data Quality Is Ignored in Favor of Data Quantity, Imposing a Hard Performance Ceiling

The third bottleneck is subtler but arguably more impactful for practical deployment. The URGENT Challenge training data — 2,500 hours from sources like CommonVoice, LibriTTS, VCTK, WSJ, EARS, and DNS5 — already underwent organizer filtering to remove non-speech samples (via voice activity detection) and noisy samples (via DNSMOS score thresholding). The implicit assumption is that this filtering produces acceptably "clean" training targets.

The paper demonstrates this assumption is false. Using VQScore (Fu et al., 2024) — a self-supervised speech quality estimator that correlates well with subjective ratings and can process the full 2,500 hours in under 8 hours on a single A100 GPU — the authors reveal a clear quality hierarchy across data sources (Figure 2). CommonVoice, the largest subset at 1,300 hours, exhibits the lowest quality due to its crowdsourced nature. In contrast, studio-recorded datasets like EARS, WSJ, and VCTK show consistently higher VQScores. Manual inspection (examples in the supplementary material) reveals that low-VQScore samples contain "stationary background noise or entirely non-speech artifacts."

This is a critical finding because it reveals a hidden performance ceiling. When a model is trained on "clean" targets that actually contain residual degradations — electrical microphone hiss, HVAC rumble, distant traffic — it learns to preserve these artifacts as features of the clean signal. This isn't a failure of optimization; the model is doing exactly what it is trained to do: reproduce its targets. The consequence is that such a model, when deployed on genuinely noisy input, can remove gross distortions but will leave subtle background noise untouched because it has learned that such noise is a normal part of "clean" speech. The paper demonstrates this concretely in Section 3.8: their model trained on VQScore-filtered data (threshold 0.65) still fails to remove low-level stationary noise from FLEURS recordings, and they need to fine-tune exclusively on the highest-quality EARS subset to address this.

The paper positions data quality for USE as "relatively new" and "seldom explored" (Section 2). While scaling laws for data quantity are well-studied (Zhang et al., 2024a; Gonzalez et al., 2024), only recent concurrent work has begun examining curation (Li et al., 2025). The paper contributes a concrete methodology — VQScore thresholding — and demonstrates the trade-off between quality and quantity explicitly: aggressive filtering (threshold 0.72, leaving 629 hours) gives the best early-training performance but eventually underperforms moderate filtering (threshold 0.65, leaving 2,506 hours) due to reduced data volume (Figure 3a).

How This Paper Positions Itself

The paper does not introduce a fundamentally new model architecture. The regression backbone is USEMamba (Chao et al., 2025, 2024), an existing architecture that alternates Mamba sequence modeling across frequency and time dimensions. The GAN uses standard CNN discriminators. The novel contributions are in the training methodology and design choices, not the model structure.

This is deliberate. The paper is a systematic analysis paper in the spirit of "we examined three things everyone assumes are fine, found they aren't, and here's how to fix them." Each contribution addresses a different stage of the USE pipeline:

• What to predict (the training target): time-shifted anechoic speech instead of early-reflected speech.
• How to train (the architecture strategy): two-stage regression-then-GAN-correction grounded in optimal transport theory.
• What to train on (the data): quality-filtered subsets rather than the full available corpus.

The paper tests these interventions on the URGENT 2025 Challenge, which provides a standardized, competitive benchmark with published leaderboard results. This gives the claims concrete reference points — Table 1 compares against the top three challenge systems, and Table 7 in the appendix provides the full leaderboard. The claims are falsifiable: if early-reflected targets were truly necessary for dereverberation quality, the time-shifted target should underperform; if data quality didn't matter, filtering shouldn't help; if the two-stage GAN correction were just a generic ensembling trick, it shouldn't show principled alignment between correction residuals and regression errors.

The paper also positions USE as an enabler for downstream tasks, particularly TTS training data cleaning (Section 3.9). This is strategically important because it demonstrates that the fidelity–quality balance they achieve — better perceptual quality without sacrificing speaker similarity or ASR accuracy — translates to measurable gains in a real application. The TTS experiments on unseen languages (Dutch and Italian) confirm that the model generalizes beyond its training languages, which matters for the low-resource language scenarios where studio-quality TTS data is scarce.

A key subtlety in the paper's positioning: they explicitly acknowledge that some metrics on the official leaderboard will penalize their approach unfairly. Because the URGENT Challenge evaluation uses early-reflected speech as the reference for intrusive metrics (PESQ, ESTOI, SDR, etc.), models trained on time-shifted anechoic targets will show artificially low scores on these metrics even if their actual output quality is higher. The paper addresses this by reporting a separate evaluation using anechoic references (Table 6 in Appendix), where the shifted anechoic target substantially outperforms the early-reflected target. This transparency about metric mismatch is important: it prevents readers from misinterpreting the official leaderboard scores as evidence against the proposed target while also demonstrating that the non-intrusive metrics and ASR accuracy — which don't suffer from this reference mismatch — show clear gains. The non-intrusive metrics (DNSMOS, NISQA, UTMOS) are arguably more relevant for real deployment anyway, since clean references are unavailable at test time.

Summary of the Landscape

Before this paper, the USE field had three unexamined assumptions: that early-reflected speech is the right dereverberation target, that multi-stage systems for balancing fidelity and quality benefit from more complexity, and that the provided training data — already organizer-filtered — was clean enough. The paper systematically challenges each assumption, providing empirical evidence that the conventional choices are suboptimal and theoretical grounding for the proposed alternatives. The URGENT 2025 Challenge serves as the proving ground, with state-of-the-art results on non-intrusive quality metrics and ASR accuracy as the validation.

3. Technical Approach

3.1 Reader Orientation

This paper is a systematic design analysis that investigates how three critical choices in training data, training targets, and model architecture affect universal speech enhancement performance, using the URGENT 2025 Challenge as a standardized testbed. The authors are building a single model that takes degraded speech — potentially affected by noise, reverberation, clipping, bandwidth limits, codec artifacts, packet loss, or wind noise, at any sampling rate from 8 to 48 kHz — and produces clean, natural-sounding speech that preserves the original speaker identity and linguistic content. The solution has the shape of a two-stage pipeline where a regression model first produces a high-fidelity but potentially over-smoothed estimate, and a Wasserstein GAN then corrects only the over-smoothed regions, trained on carefully filtered high-quality data with time-shifted anechoic targets instead of the conventional early-reflected speech.

3.2 Big-Picture Architecture (Diagram in Words)

The system is a two-stage training and inference pipeline with four main components:

1. Data Quality Filter — Uses VQScore (a self-supervised quality estimator) to remove low-quality training samples before training begins, with a threshold of 0.65 that balances quality and quantity.

2. Regression Model (Frozen) — A 30-layer USEMamba trained to convergence on time-shifted anechoic clean speech targets. It outputs the posterior mean estimate $\hat{s}$, which minimizes mean squared error but may sound over-smoothed under severe degradation. After training, its weights are frozen.

3. Generative Residual Corrector (GAN) — A 6-layer USEMamba generator with CNN-based multi-band discriminators. It takes both the noisy input and the regression model's output as conditional inputs, with a residual connection from the regression output to the final output, forcing the GAN to only correct regions where the regression output deviates from realistic clean speech statistics.

4. Adaptive Multi-Band Discriminator — Processes different frequency sub-bands separately based on input sampling rate, then concatenates features for a final real/fake prediction.

Information flows as follows: degraded input → freezing regression model → initial estimate → concatenation with degraded input → GAN generator → residual addition → final enhanced output. During training, the discriminator compares the final output distribution against real clean speech; during inference, only the generator is used.

3.3 Roadmap for the Deep Dive

• First, training target selection — why time-shifted anechoic speech replaces early-reflected speech, the mathematical model of reverberation, and how the time shift $n_0$ is estimated.
• Second, the two-stage framework — the distortion–perception trade-off theory, why the posterior mean must be transported to the data distribution, and how the GAN residual correction operationalizes optimal transport.
• Third, the Lipschitz continuity argument for why GANs focus correction on over-smoothed regions, with receptive field analysis.
• Fourth, data quality filtering — VQScore analysis across datasets, threshold selection balancing quality and quantity.
• Fifth, the model architecture details — USEMamba, SFI-STFT, adaptive multi-band discriminator, hyperparameters, and training configuration.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a design analysis paper whose core idea is that three underexamined choices in the USE pipeline — the dereverberation target, the architecture strategy for balancing fidelity and quality, and the quality of training data — impose hidden performance ceilings that can be systematically raised.

---

Training Target: Time-Shifted Anechoic Clean Speech

The conventional dereverberation target in the URGENT Challenge (and the broader speech enhancement literature) is early-reflected speech, defined as anechoic clean speech convolved only with the early reflection component of the room impulse response (RIR), up to 50 ms after the direct-path peak. The paper challenges this convention by arguing that the primary difficulty in dereverberation is not the early reflections themselves, but the misalignment between the reverberant input and the clean target caused by the direct-path time shift.

Reverberation model. A reverberant speech signal $y[n]$ is modeled as the convolution of anechoic clean speech $s[n]$ with a room impulse response $r[n]$:

$y[n] = s[n] \ast r[n]$

where $n$ is the discrete time index and $\ast$ is the convolution operator. The operation computes: for each output sample $n$, sum over all $k$ the product $s[k] \cdot r[n-k]$ — the clean signal weighted and delayed by the room's acoustic response.

The RIR $r[n]$ decomposes into three components:

$r[n] = \delta[n-n_0] \ast \left(r_e[n] + r_l[n]\right)$

where $\delta[n-n_0]$ is the direct-path impulse (a unit spike shifted by $n_0$ samples representing the travel time from source to microphone, typically 5–30 ms), $r_e[n]$ is the early reflection component (RIR contributions within 50 ms after the direct-path peak, as specified in the URGENT Challenge), and $r_l[n]$ is the late reflection component (the remaining reverberation tail).

What this equation computes: the RIR is the direct sound arrival followed by a weighted sum of delayed copies of the signal — early reflections (first 50 ms) that blend perceptually with the direct sound, and late reflections (beyond 50 ms) that create the perception of reverberation. The decomposition matters because different targets preserve or eliminate different components.

Why this decomposition: the 50 ms boundary is not arbitrary but based on psychoacoustic studies showing that reflections arriving within this window are perceptually fused with the direct sound (the precedence effect), while later reflections are perceived as distinct reverberation. This is why the conventional wisdom says "remove late reflections but keep early ones" — they're supposedly not harmful.

Conventional target: early-reflected speech. The standard learning target for dereverberation is:

$s_e[n] = s[n] \ast \delta[n-n_0] \ast r_e[n] = s[n-n_0] \ast r_e[n]$

where $s_e[n]$ is the early-reflected clean speech — the anechoic signal $s$ delayed by the direct-path time $n_0$ and then filtered by early reflections $r_e$. The operation convolves the clean signal with only the first 50 ms of the RIR, preserving the direct sound and early reflections while discarding late reverberation.

Why this target exists: it addresses the alignment problem. If you use unshifted anechoic clean speech $s[n]$ as the target for reverberant input $y[n] = s[n] \ast r[n]$, the model must implicitly estimate $n_0$ to align its output with the target. Figure 3(b) confirms that $s[n]$ as a target yields the worst performance — the model struggles with this time shift. By keeping $\delta[n-n_0]$ in the target, the early-reflected target $s_e[n]$ preserves the temporal alignment, so the model doesn't need to estimate $n_0$. The cost is that it also preserves $r_e[n]$, the early reflection coloration.

The paper's proposed target: time-shifted anechoic clean speech. The key insight is that the alignment problem can be solved without preserving early reflections. If the direct-path time shift $n_0$ is estimated from the RIR directly as $n_0 = \arg\max_n |r[n]|$ (the sample index of the maximum absolute value in the RIR), the target becomes:

$s[n-n_0]$

where the operation is simply: take the anechoic clean signal and shift it forward by $n_0$ samples to align with when the direct sound arrives at the microphone. This requires no learned estimation — $n_0$ is computed analytically from the known RIR during training data generation.

Why this form is superior to the early-reflected target: the early-reflected target $s_e[n] = s[n-n_0] \ast r_e[n]$ and the proposed target $s[n-n_0]$ differ only in whether $r_e[n]$ is present. The paper's empirical finding (Figure 3b, Table 1) is that eliminating $r_e[n]$ consistently improves both perceptual quality metrics (UTMOS, DNSMOS, NISQA) and downstream ASR performance (CAcc). This implies that the conventional belief — "early reflections are too hard to remove without causing artifacts" — was incorrect for the modern USE setting. It conflated two problems: (1) alignment difficulty (solved by $n_0$ estimation) and (2) early reflection removal difficulty (solved by having a capable enough model). Once alignment is handled externally, the model can remove early reflections without introducing excessive artifacts.

Why the conventional wisdom persisted: prior work that established the early-reflected convention (Valin et al., 2022; Zhou et al., 2023; Zhao et al., 2020) tested in narrower settings — typically single-distortion dereverberation with smaller models and less training data. In those contexts, attempting to remove early reflections may indeed have introduced artifacts. The paper's contribution is demonstrating that with modern model capacity, diverse multi-condition training, and explicit time-shift handling, the artifact concern no longer applies.

The special case of 0 ms early reflection window. Figure 3(b) shows a progressive improvement as the early reflection window is reduced from 50 ms to 0 ms (in 10 ms increments). At 0 ms, the target becomes exactly $s[n-n_0]$ — the time-shifted anechoic signal. This confirms that every millisecond of early reflection preserved in the target degrades output quality. The mechanism is straightforward: if the target contains early reflection coloration, the model learns to reproduce it, and the enhanced output retains reverberation cues that listeners (and non-intrusive quality estimators) perceive as degraded.

Practical implementation: during training data generation, for any utterance with reverberation, the authors compute $n_0$ from the RIR and shift the anechoic clean speech accordingly. For the six other distortion types (noise, clipping, bandwidth limitation, codec artifacts, packet loss, wind noise), the target is unshifted anechoic clean speech $s[n]$. The model is trained on the same mixture of distortion types — only the reverberation target changes. This means the model must simultaneously learn to produce time-shifted outputs for reverberant inputs and unshifted outputs for non-reverberant inputs, which it handles through the conditioning provided by the input signal itself.

---

Two-Stage Framework: Regression Then GAN Correction

The second technical contribution is the two-stage training and inference architecture motivated by the distortion–perception trade-off theory.

The distortion–perception trade-off formalized. Blau and Michaeli (2018) established that for any restoration problem, there exists a fundamental trade-off between distortion (measured by expected distance between the estimate and ground truth, e.g., MSE) and perceptual quality (measured by divergence between the distribution of estimates and the distribution of real clean signals). The distortion–perception (DP) function from Freirich et al. (2021) is:

$D(P) = \min_{p_{\tilde{s}\mid y}} \{\; \mathbb{E}\!\left[d(s,\tilde{s})\right] \quad \text{s.t.} \quad d_p\!\left(p_s, p_{\tilde{s}}\right) \leq P\}$

where $D(P)$ is the minimal achievable distortion given a perceptual quality constraint $P$, $p_{\tilde{s}\mid y}$ is the conditional distribution of the estimate given the degraded input, $d(s,\tilde{s})$ is the distortion measure (e.g., squared Euclidean distance), $d_p(p_s, p_{\tilde{s}})$ is a divergence between the distribution of clean speech $p_s$ and the distribution of estimates $p_{\tilde{s}}$, and $P \geq 0$ is the allowed perceptual degradation (smaller $P$ means better perceptual quality).

What it computes: the function $D(P)$ defines the Pareto frontier — for any given level of perceptual quality $P$, $D(P)$ is the minimum distortion any estimator can achieve. As $P$ decreases (tighter perceptual constraint), $D(P)$ increases — you must accept more distortion to get better perceptual quality. The function is monotonically non-increasing and convex under standard divergences.

Why this form: it captures the core tension. A pure regression model (outputting $\mathbb{E}[s \mid y]$) achieves minimal distortion $D^*$ (the MMSE) but typically has poor perceptual quality because averaging over the posterior collapses multimodal structure into a single unrealistic mean. A pure generative model (sampling from $p(s \mid y)$) can achieve perfect perceptual quality ($P = 0$, meaning $p_{\tilde{s}} = p_s$) but at distortion $D(0) > D^*$ because sampling introduces variance.

The optimal transport solution. When squared-error distortion $d(s,\tilde{s}) = \lVert s - \tilde{s} \rVert_2^2$ and Wasserstein-2 distance $d_p(p_s, p_{\tilde{s}}) = W_2(p_s, p_{\tilde{s}})$ are used, Freirich et al. (2021) showed that the minimal distortion under perfect perceptual quality ($P = 0$) is:

$D(0) = D^* + W_2(p_{s^*}, p_s)$

where $D^* = \mathbb{E}[\lVert s - s^* \rVert^2]$ is the MMSE achieved by the posterior mean $s^* = \mathbb{E}[s \mid y]$, $p_{s^*}$ is the distribution of posterior mean estimates, $p_s$ is the clean speech distribution, and $W_2(p_{s^*}, p_s)$ is the Wasserstein-2 distance between these two distributions.

What it computes: the minimal achievable distortion when the output distribution must perfectly match the clean speech distribution. It equals the MMSE plus the cost of transporting the posterior mean distribution to the clean speech distribution.

Why this form matters for architecture design: it specifies exactly what an optimal two-stage system should do. Stage 1 computes $s^* = \mathbb{E}[s \mid y]$ — the posterior mean — which minimizes MSE but lives in a distribution $p_{s^*}$ that doesn't match real speech. Stage 2 applies an optimal transport map from $p_{s^*}$ to $p_s$ — moving the posterior mean estimates into the clean speech manifold without adding unnecessary distortion. The paper operationalizes this transport using a Wasserstein GAN, which minimizes the Wasserstein-1 distance (equivalent under Lipschitz constraints to the Wasserstein-2 objective in the theory).

The two-stage training procedure.

Stage 1 (Regression): Train the 30-layer USEMamba to convergence using standard regression losses (L1 on magnitude, L2 on complex spectrogram, time-domain loss) with the time-shifted anechoic target. The model becomes $f_{\text{reg}}(y) = \hat{s}$, an approximation of $\mathbb{E}[s \mid y]$. After convergence, the regression model's weights are frozen — it is never updated during Stage 2.

Stage 2 (Generative Correction): Initialize a 6-layer USEMamba generator $G$. Given degraded input $y$ and frozen regression output $\hat{s} = f_{\text{reg}}(y)$, the final output is:

$\tilde{s} = \hat{s} + G(y, \hat{s})$

where $G(y, \hat{s})$ is the GAN generator's output conditioned on both the noisy input and the regression estimate, and $+$ is element-wise addition in the waveform domain.

What it computes: the final output $\tilde{s}$ is the regression model's estimate plus a learned residual correction. The generator sees both the original degraded input $y$ (so it knows what degradation to correct) and the regression output $\hat{s}$ (so it knows what the MMSE baseline looks like).

Why the residual connection: this is the crucial design choice. Without the residual connection ($\tilde{s} = G(y, \hat{s})$), the generator would be responsible for reconstructing the full signal, potentially overwriting correct regression outputs with hallucinated content. With the residual connection, $G$ learns to output near-zero values in regions where $\hat{s}$ is already accurate, and non-zero corrections only where $\hat{s}$ deviates from realistic speech. The paper verifies this empirically by computing the average correlation coefficient between two magnitude-residual spectrograms on the non-blind test set: the "clean–regression residual" ($s - \hat{s}$) and the "final–regression residual" ($\tilde{s} - \hat{s}$) — they obtain a high correlation of 0.78, confirming that GAN corrections are "strongly aligned with the residual errors of the regression model."

Why GANs for optimal transport. Wasserstein GANs (Arjovsky et al., 2017) optimize an objective equivalent to the Wasserstein-1 distance between the source (posterior mean) and target (clean speech) distributions. The paper additionally notes that "a single forward-pass generation in GAN-based methods makes them more easily adaptable to real-time scenarios" — unlike diffusion or flow-matching models that require iterative sampling.

Training losses for the GAN. The generator is trained with a combination of a GAN loss, a feature-matching loss, and a reconstruction loss (the paper uses the standard GAN training framework, with the specific losses enumerated in Appendix). The discriminator is trained to distinguish real clean speech from the generator's outputs.

---

Why GANs Focus Correction on Over-Smoothed Regions: A Lipschitz Argument

The paper provides an additional justification for why the GAN correction framework specifically targets only the regions where the regression model fails, rather than indiscriminately modifying the entire signal. This argument is important because it explains why the two-stage design preserves fidelity while improving quality — the mechanism is not just empirical correlation but has a structural explanation.

When a convolutional neural network (CNN) is used as the discriminator, each element of the final feature map (the layer before global average pooling) has only a limited receptive field in the input signal. If the discriminator is Lipschitz continuous — enforced via spectral normalization (Miyato et al., 2018), which constrains the spectral norm of each weight matrix — the following constraint holds:

$\|D^{(l)}(\tilde{s}) - D^{(l)}(s)\| \leq L \|\tilde{s} - s\|, \quad \forall l$

where $D^{(l)}(\cdot)$ is the activations at the $l$-th discriminator layer, $\tilde{s}$ is the final model output, $s$ is the clean speech, $L$ is the Lipschitz constant (a bound on how much the function's output can change relative to its input), and $\|\cdot\|$ denotes the Euclidean norm.

What this inequality says: the difference in discriminator features between the enhanced output and clean speech cannot exceed $L$ times their difference in the input domain. In plain language: if the enhanced output is very close to the clean speech in waveform space, the discriminator cannot tell them apart in feature space.

Why this matters for the feature-matching loss: the feature-matching loss (Salimans et al., 2016) used during generator training penalizes exactly $\|D^{(l)}(\tilde{s}) - D^{(l)}(s)\|$. The Lipschitz bound ensures that when the model accurately predicts the clean target within a given receptive field — meaning $\|\tilde{s} - s\| \approx 0$ for that local region — the feature-matching loss contributes no gradient for those regions. This is because the left-hand side of the inequality is bounded above by (approximately) zero.

The practical consequence: the generator receives gradients only from regions where $\|\tilde{s} - s\|$ is large — which correspond to the over-smoothed or erroneous parts of the regression output. In regions where the regression model already produces accurate output, the GAN is structurally prevented from modifying the signal. This provides a built-in fidelity preservation mechanism that doesn't require explicit gating or masking.

The multi-band discriminator design. To strengthen this local correction property, the paper proposes an adaptive multi-band discriminator (inspired by Kumar et al., 2023) that processes different frequency bands separately before combining features. The architecture works as follows:

• For each sub-band corresponding to the input sampling rate (e.g., 8 kHz gets a single band 0–4 kHz; 22.05 kHz gets three bands: 0–4 kHz, 4–8 kHz, and 8–11.025 kHz), a separate 5-layer 2-D convolutional network extracts local features.
• The 2-D convolutions operate on the time-frequency representation, where each sub-band covers a specific frequency range.
• Sub-band features are then concatenated along the frequency axis and passed through a final 2-layer 2-D convolution.
• Global average pooling produces the final scalar discriminator output (real/fake).

Why multi-band: different frequency bands have different signal characteristics and different susceptibility to degradation. Bandwidth-limited speech, for example, has no signal above a cutoff frequency. Packet loss creates structured gaps in the spectrogram. By processing bands separately, each sub-discriminator can specialize in detecting artifacts relevant to its frequency range. The concatenation and final convolution then learn cross-band interactions. This design inherits from the multi-scale and multi-period discriminators common in neural vocoders but adapts them for the variable sampling rate requirement of USE.

---

Data Quality Filtering with VQScore

The third technical contribution is a data curation methodology that demonstrates how training data quality imposes a hard performance ceiling on USE models.

Motivation: hidden noise in "clean" training data. The URGENT 2025 Challenge training data (approximately 2,500 hours) was already filtered by organizers using voice activity detection (VAD) to remove non-speech segments and DNSMOS thresholding to remove noisy samples. Despite this, the paper finds that "many recordings with audible background noise remain" (Section 2.3). The mechanism by which this hurts performance is straightforward: the model learns to treat residual noise in the "clean" target as a feature of clean speech, so it fails to remove similar noise from genuinely degraded inputs.

VQScore as a quality estimator. The paper employs VQScore (Fu et al., 2024), a self-supervised speech quality estimation model trained only on clean speech. VQScore takes a speech waveform as input and outputs a scalar between 0 and 1, where higher scores indicate better quality (closer to clean studio speech). The scores correlate well with subjective human ratings (Zhang et al., 2025b) and can process the entire 2,500-hour corpus in under 8 hours on a single NVIDIA A100 GPU. This makes it practical for large-scale data curation.

Quality distribution across data sources. Figure 2 shows the VQScore histogram for each major data source, with dashed lines indicating median scores. The key findings:

• CommonVoice (from Common Voice 19.0, 1,300 hours, crowdsourced): the lowest quality source, with a broad distribution and low median, due to its crowdsourced nature with uncontrolled recording conditions.
• EARS, WSJ, VCTK (100, 85, and 80 hours respectively, studio-recorded): consistently the highest quality, with narrow distributions concentrated at high VQScore values.
• LibriTTS, DNS5, MLS (200, 350, and 450 hours respectively): intermediate quality, with broader distributions reflecting their mixed recording conditions.

Manual inspection findings. Low-VQScore samples are found to contain "stationary background noise or entirely non-speech artifacts" (Section 2.3). Examples include electrical microphone hiss, HVAC rumble, distant traffic noise — subtle degradations that VAD cannot detect and that DNSMOS thresholding (set to 3.0 by challenge organizers) lets through.

Threshold selection. The paper experiments with three VQScore filtering thresholds:

• 0.50 (no filtering): 2,518 hours, the original corpus.
• 0.65: 2,506 hours, removing only the worst ~12 hours.
• 0.72: 629 hours, retaining only the highest-quality quarter of the data.

Figure 3(a) shows the UTMOS learning curves on the validation set for models trained with each threshold:

• Threshold 0.50 (no filtering, blue line): Worst performance throughout training, confirming that even challenge-organizer-filtered data contains quality-degrading samples.
• Threshold 0.72 (green line): Best early-training performance due to higher quality, but eventually lags behind the 0.65 threshold because the reduced data volume (~629 hours) limits the model's exposure to linguistic and acoustic diversity.
• Threshold 0.65 (orange line): Best final performance, achieving a good balance between quality and quantity.

The paper adopts 0.65 for all subsequent experiments. This threshold removes only samples with very low VQScore — the worst contaminants — while retaining over 99% of the original data volume. This explains why the filtering has a clear positive effect: it removes the most harmful examples (entirely non-speech artifacts, severely degraded recordings) without sacrificing the diversity needed for universal enhancement.

Fine-tuning on EARS for residual artifact removal. Even after VQScore filtering at 0.65, the model fails to remove very subtle artifacts like electrical microphone hiss (Section 3.8, Figure 10 in Appendix). The paper addresses this by fine-tuning exclusively on the EARS dataset (100 hours of studio-recordings), producing the "Proposed (EARS)" variant. On unseen FLEURS recordings with mild stationary noise, Proposed (EARS) achieves higher DNSMOS and ASR accuracy than the base model, though both variants outperform Miipher-2 (FLEURS-R) in speaker similarity preservation. This demonstrates that the performance ceiling imposed by residual training data noise is real and that targeted curation/fine-tuning can raise it.

Why this trade-off exists and why it matters. The paper's data quality findings reveal a tension between:

• Quantity: more data provides more linguistic coverage, acoustic diversity, and speaker variation, which helps the model generalize to unseen conditions.
• Quality: higher-quality data provides cleaner targets, which helps the model learn to remove all degradations rather than preserve some as "normal."

The optimal operating point (threshold 0.65) is where the marginal benefit of adding more examples is outweighed by their noise contamination. This echoes findings from the image domain about the importance of data curation (e.g., the transition from LAION to DataComp) but applied to speech enhancement. The implication for practitioners is that carefully curating a moderate-sized, high-quality dataset may yield better USE models than training on all available data, even when that data has already been "cleaned" by standard heuristics.

---

Model Architecture Details

While architecture is not the primary contribution, the paper uses specific design choices to enable the research questions to be tested fairly.

USEMamba backbone. The regression model uses USEMamba with 30 layers (Chao et al., 2025, 2024). USEMamba alternates between two types of sequence modeling modules applied to frequency features and time features in the time-frequency domain. The architecture uses Mamba — a state-space model that scales linearly with sequence length rather than quadratically like Transformers — making it efficient for long audio sequences. The 30-layer configuration provides the capacity needed for universal enhancement across seven distortion types. The GAN generator uses a smaller 6-layer USEMamba, reflecting that the residual correction task is simpler than full signal reconstruction.

Sampling frequency-independent (SFI) STFT. To handle inputs at different sampling rates (8, 16, 22.05, 24, 32, 44.1, and 48 kHz) with a single model, the paper uses SFI-STFT (Zhang et al., 2023a). This technique dynamically adjusts the FFT window size and hop size according to the input sampling rate such that the time duration of the window remains constant — 40 ms for all rates. For 8 kHz input, the FFT window size is 320 samples (40 ms × 8000 samples/second); for 48 kHz input, it would be 1920 samples (40 ms × 48000 samples/second). This ensures that the frequency resolution changes proportionally with sampling rate while the temporal resolution remains fixed, producing a consistent feature dimension across all rates. The number of frequency bins varies, but this is handled by the architecture's sub-band processing.

Adaptive multi-band discriminator. As detailed above, the discriminator architecture processes different frequency sub-bands based on the input sampling rate. For each sub-band, a 5-layer 2-D convolutional network performs local feature extraction. Sub-band features are concatenated along the frequency axis and fed to a final 2-layer 2-D convolution. Global average pooling produces the scalar output. This design is motivated by the observation that "distinct feature patterns across frequency bands" exist — different degradations affect different frequency regions differently (e.g., bandwidth limitation cuts off high frequencies, while wind noise is concentrated at low frequencies).

Training configuration. All models are trained on 8 NVIDIA A100 GPUs with a batch size of 1. The batch size of 1 is explicitly chosen to allow longer utterances to be processed without running out of GPU memory — this matters for USE because some degradations (e.g., reverberation) depend on context that spans several seconds. The optimizer is AdamW with a learning rate of 0.0002 for all components (regression model, generator, discriminator). The paper does not specify the exact number of training steps or epochs but shows learning curves (Figures 3, 9) that indicate convergence behavior. The code is stated to be "released upon acceptance to facilitate reproducibility."

Loss functions for regression training. While not enumerated in the main text, the regression model is trained with a combination of magnitude loss (L1 on spectrogram magnitude), phase loss, and time-domain loss, following the USEMamba training recipe. Figure 9 in the Appendix compares these losses between conventional adversarial fine-tuning and the proposed two-stage GAN correction approach.

Inference. At inference time, only the generator is used — the discriminator is discarded. The degraded input passes through both the frozen regression model and the GAN generator, with the residual connection producing the final output. The entire pipeline requires only forward passes, making it suitable for deployment.

4. Key Insights and Innovations

Innovation 1: Reframing the Dereverberation Problem as Alignment, Not Reflection Removal

The paper's most conceptually distinctive contribution is not proposing a new target per se — time-shifted anechoic targets have been explored in narrower contexts — but rather diagnosing and decomposing a long-standing failure mode in dereverberation that the field had misattributed for years. The conventional wisdom, codified in the URGENT Challenge specification and supported by multiple prior works (Valin et al., 2022; Zhou et al., 2023; Zhao et al., 2020), held that early reflections are "much harder to remove" and that attempting full dereverberation "leads to excessive artifacts." This paper identifies that this diagnosis conflates two distinct problems: (1) the alignment difficulty caused by the unknown direct-path time shift $n_0$, and (2) the signal removal difficulty of stripping away early reflections.

The evidence for this decomposition is clean and falsifiable. When the target is unshifted anechoic clean speech $s[n]$ — forcing the model to simultaneously remove reverberation AND estimate alignment — performance is worst (Figure 3b, blue line). When the target is early-reflected speech $s_e[n]$ — solving alignment by preserving $\delta[n-n_0]$ in the target but keeping early reflections — performance improves but remains suboptimal. When the target is time-shifted anechoic speech $s[n-n_0]$ — solving alignment explicitly via RIR-derived $n_0$ while removing all reflections — performance is best. The progression shows that alignment is the bottleneck, not reflection removal per se.

This is a fundamental reframing rather than an incremental improvement because it changes what researchers should optimize. If the problem is alignment, the solution is better time-shift estimation (trivially obtainable from the RIR during training, and potentially estimable from the mixture at inference). If the problem is reflection removal difficulty, the solution would be better model architectures or training strategies. The paper demonstrates that the former interpretation is correct for modern USE models, redirecting research attention from architectural innovation to target design — a shift with implications for how all dereverberation systems should be trained.

The systematic sweep of the early reflection window from 50 ms down to 0 ms (Figure 3b) provides unusually granular evidence: performance improves at every step, with no regime where "some early reflections are beneficial." This contradicts the psychoacoustic rationale — that early reflections are perceptually fused with direct sound and thus harmless — at least for the learned metrics that USE models optimize.

Innovation 2: Operationalizing Optimal Transport Theory as a Concrete Architecture Design Principle

Prior work in the URGENT Challenge approached the fidelity–quality trade-off through ad-hoc architectural complexity: three-stage systems (Rong et al., 2025; Goswami and Harada, 2025), four-stage systems (Le et al., 2025), and energy-based output selection (Chao et al., 2025). Each team independently converged on multi-stage designs without a unifying theoretical justification for the number of stages, their ordering, or how they should interact. The paper's second distinctive contribution is demonstrating that the distortion–perception trade-off theory (Blau and Michaeli, 2018) — specifically its extension to Wasserstein space by Freirich et al. (2021) — directly specifies an optimal two-stage architecture: first compute the posterior mean (regression), then apply an optimal transport map from the posterior mean distribution to the clean speech distribution (generative correction).

What makes this more than a theoretical veneer on an otherwise standard two-stage pipeline is the specificity of the prescription. The theory states that the minimal MSE under a perfect-perception constraint is $D(0) = D^* + W_2(p_{s^*}, p_s)$ — the MMSE plus the Wasserstein-2 distance between the posterior mean distribution and the clean speech distribution. This implies that Stage 2 should NOT be a generic generative model that reconstructs from scratch. It should be an optimal transport map that moves the posterior mean estimates into the clean speech manifold while adding minimal additional distortion. The paper operationalizes this through two concrete design choices that follow directly from the theory: (1) a Wasserstein GAN, which minimizes a principled optimal transport metric, and (2) a residual connection from the frozen regression output to the final output, which structurally constrains the generator to only add corrections ($\tilde{s} = \hat{s} + G(y, \hat{s})$) rather than overwrite the posterior mean.

The residual connection is the key architectural insight. Any multi-stage system without it — regardless of how many stages — allows the generative component to undo correct regression outputs, potentially introducing hallucinations. With the residual connection, the generator's task is inherently conservative: it outputs near-zero in well-predicted regions and non-zero corrections only where the regression model fails. The paper verifies this mechanism empirically through a 0.78 correlation between regression residuals ($s - \hat{s}$) and GAN corrections ($\tilde{s} - \hat{s}$), and provides a Lipschitz continuity argument (Section 2.2, Equation 3) showing why the feature-matching loss structurally prevents the GAN from modifying accurate regression outputs.

This contribution is best understood as a theoretical grounding of what was previously empirical tinkering. It doesn't claim that a two-stage system is novel — SEStream (Huang et al., 2023) and StoRM (Lemercier et al., 2023) used similar structures for codec compression and dereverberation, respectively. Rather, it provides a principled answer to why two stages are sufficient and how they should be connected, eliminating the complexity arms race that had characterized URGENT Challenge submissions. The theoretical derivation in Appendix A.1 is not decorative — it directly informs the residual design and the choice of Wasserstein GAN over alternatives like diffusion or flow matching (which would have higher inference cost without a clear theoretical advantage for this specific transport task).

Innovation 3: Establishing Data Quality as a First-Class Performance Ceiling in Universal Speech Enhancement

The relationship between training data quantity and model performance has been well-studied in speech enhancement, with scaling laws showing diminishing returns from additional data (Zhang et al., 2024a; Gonzalez et al., 2024). The paper's third distinctive contribution is demonstrating that data quality, not quantity, imposes the binding constraint for modern USE models, and that this constraint manifests as a subtle but consequential failure mode: the model learns to preserve residual degradations present in "clean" training targets.

This is not merely the unsurprising finding that noisy training data hurts performance. The deeper claim is that challenge-organizer-level filtering is insufficient and that the remaining artifacts — stationary background noise, microphone hiss, non-speech segments missed by VAD — impose a hard ceiling that cannot be overcome by scaling model size, training time, or architectural complexity. The evidence for a ceiling rather than a gradual degradation comes from the learning curve behavior in Figure 3(a): the unfiltered model plateaus at a lower UTMOS than the filtered models, and the moderately filtered model (threshold 0.65, retaining 2,506 hours) outperforms the aggressively filtered model (threshold 0.72, 629 hours only) after sufficient training, showing that the trade-off between quality and quantity has a clear optimum.

The diagnostic power of this finding is amplified by the FLEURS unseen-language experiment (Section 3.8). The base model, even after VQScore filtering, fails to remove subtle electrical microphone hiss from FLEURS recordings — residual artifacts that also appear in the "clean" training data. Only fine-tuning exclusively on EARS (the highest-quality subset, 100 hours of studio recordings) addresses this. This demonstrates that the performance ceiling is not about model capacity or optimization — it is about target contamination: the model cannot remove degradations that it has learned are normal features of clean speech. This is a fundamentally different failure mode than the over-smoothing or hallucination problems addressed by the two-stage architecture, and it requires a fundamentally different solution (data curation, not architectural innovation).

The practical significance extends beyond USE to the downstream TTS application (Section 3.9). When an USE model is used to clean TTS training data, any residual artifacts it preserves become baked into the TTS model, which will then reproduce them during synthesis. The paper's TTS experiments on unseen languages (Dutch, Italian) show that enhancing both context and training audio with their USE model reduces character error rate by ~45% and word error rate by ~30% compared to original data (Table 3). This demonstrates that the data quality ceiling in USE cascades to downstream applications, making curation a prerequisite for any pipeline that uses automatically enhanced speech.

This contribution resonates with parallel developments in other domains — the transition from LAION to DataComp in image generation, and the recognition that web-scale datasets require aggressive filtering — but provides one of the first systematic analyses of this phenomenon in universal speech enhancement, where the diversity of degradation types masks the subtlety of residual target noise.

Innovation 4: The Lipschitz Continuity Argument as a Structural Fidelity Guarantee

A subtler but intellectually distinctive contribution is the paper's argument that GAN-based correction, when implemented with a residual connection and Lipschitz-constrained discriminators, provides a structural guarantee that signal regions where the regression model is already accurate will not be modified. This argument, presented in Section 2.2 (Equation 3 and surrounding discussion), is notable because it explains why the two-stage framework preserves fidelity — not as an empirical observation that requires post-hoc verification, but as a consequence of the discriminator architecture.

The logic proceeds as follows: (1) CNN discriminators have limited receptive fields, so each element of the final feature map depends only on a local region of the input. (2) Spectral normalization enforces Lipschitz continuity, which bounds how much the discriminator's features can change when the input changes. (3) The feature-matching loss penalizes differences between the discriminator's features for the enhanced output versus clean speech. (4) The Lipschitz bound ensures that when $\|\tilde{s} - s\| \approx 0$ locally (i.e., the regression output is already correct), the feature-matching loss for that region is zero, and no gradient flows to the generator for that region.

This argument does not claim that the GAN actively decides which regions to correct — it is a passive mechanism arising from the interaction of architectural choices. But it provides a theoretically grounded explanation for the empirical correlation (0.78) between regression residuals and GAN corrections, and — more importantly — it provides a design principle: to preserve fidelity in generative enhancement, constrain the discriminator to have local receptive fields and bounded sensitivity to input changes, and use feature matching rather than only an adversarial loss.

The innovation here is not the individual components (spectral normalization, feature matching, residual connections are all standard) but the recognition that their combination, when interpreted through the Lipschitz lens, produces a desired behavior — conservative correction — without requiring explicit gating, masking, or confidence estimation modules. This is an elegant example of emergent behavior from architectural constraints and distinguishes the paper's approach from prior multi-stage systems that required explicit mechanisms to decide when to trust regression versus generative outputs (e.g., the energy-based criterion in Chao et al., 2025).

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the URGENT 2025 Challenge training dataset and evaluate on the URGENT 2025 non-blind test set. The training data comprises approximately 2,500 hours of multi-condition speech recordings from sources including CommonVoice, DNS5 (LibriVox), MLS, LibriTTS, VCTK, WSJ, and EARS, covering five languages (English, German, French, Spanish, and Chinese) and seven sampling rates (8, 16, 22.05, 24, 32, 44.1, and 48 kHz). Seven distortion types are considered: additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss, and wind noise. The validation set is simulated following the organizers' guidelines using the validation splits of the constituent corpora. The non-blind test set consists of 1,000 utterances with noise and RIRs from unseen sources — that is, sources not present during training, testing generalization to new acoustic conditions. The paper also evaluates on the FLEURS dataset (Conneau et al., 2023) for unseen-language testing, selecting Italian, Dutch, and Japanese — languages not present in the URGENT training data.

• Base model(s). The regression backbone is USEMamba with 30 layers (Chao et al., 2025, 2024), an architecture that alternates Mamba-based sequence modeling across frequency and time dimensions in the time-frequency domain. The GAN generator uses a smaller 6-layer USEMamba. USEMamba is chosen as a representative modern architecture for speech enhancement — it is not the authors' novel architecture, which reinforces that the paper's contributions are methodological (target selection, two-stage training, data curation) rather than architectural. For the FLOPs-matched comparison in Section 3.9, the TTS backbone is Zero-Shot Koel-TTS (Hussain et al., 2025), a 378M-parameter encoder-decoder Transformer operating on low-frame-rate (21.5 FPS) audio codec tokens encoded by NanoCodec (Casanova et al., 2025), but this is used for downstream evaluation rather than USE model comparison.

• Metrics. The evaluation employs a comprehensive suite spanning multiple assessment dimensions. Non-intrusive metrics (requiring no clean reference): DNSMOS (Reddy et al., 2022), NISQA (Mittag et al., 2021), and UTMOS (Saeki et al., 2022) — all estimate perceptual quality from only the enhanced signal. Intrusive metrics (requiring a clean reference): PESQ (Rix et al., 2001) for perceptual quality, ESTOI (Jensen and Taal, 2016) for intelligibility, SDR (Roux et al., 2019) for time-domain waveform distortion, MCD for Mel cepstral distortion, and LSD for log-spectral distance. Task-independent downstream metrics: SpeechBERTScore (SBERT; Saeki et al., 2024) using self-supervised models, and Levenshtein Phoneme Similarity (LPS; Pirklbauer et al., 2023) for phoneme sequence similarity. Task-dependent downstream metrics: speaker similarity (SpkSim) via ECAPA-TDNN (Desplanques et al., 2020) to measure speaker identity preservation, and character accuracy (CAcc) via Whisper (Radford et al., 2023) to reflect ASR performance. A critical note: for dereverberation, the URGENT Challenge uses early-reflected speech as the clean reference for intrusive metrics. Since the paper's models are trained on time-shifted anechoic targets, this creates a definition mismatch that may penalize their leaderboard scores on reference-based metrics. The paper addresses this by separately reporting results using anechoic clean speech as reference (Table 6 in Appendix).

• Baselines. The paper compares against multiple categories of baselines. Challenge baselines: the URGENT 2025 organizer-provided baseline (TF-GridNet; Wang et al., 2023) and the top three ranked systems from the challenge leaderboard — Rank 1: Team Bobbsun (Sun et al., 2025, regression with channel-mixing and progressive block extension), Rank 2: Team rc (Chao et al., 2025, regression + generative Mamba with energy-based selection), Rank 3: Team Xiaobin (Rong et al., 2025, three-stage filling/separation/restoration). Ablation baselines: the authors' own model trained with early-reflected targets (both with and without GAN correction) serves as the primary ablation baseline for the training target contribution. Open-source baselines: ClearerVoice-Studio (Zhao et al., 2025, regression-based using MossFormer2) and Resemble Enhance (latent conditional flow matching) for the comparison in Section 3.7. Downstream baseline: FLEURS-R (Ma et al., 2024), a restored version of FLEURS processed by Miipher-2 (Karita et al., 2025), for the unseen-language evaluation.

• Generation budget / compute accounting. The paper does not report inference in terms of a "generation budget" (it is not a sampling-based method like the LLM papers). Instead, compute is accounted for through training configurations: all models are trained on 8 NVIDIA A100 GPUs with a batch size of 1 (deliberately chosen to allow longer utterances without memory issues). The VQScore filtering processes 2,500 hours of speech in under 8 hours on a single A100 GPU. The regression model is trained to convergence before freezing; the GAN is then trained separately with the frozen regression model. No inference-time compute scaling is studied — the contribution is about better training choices, not test-time compute allocation.

• Cross-validation / statistical protocol. For the TTS evaluation (Section 3.9, Table 3), the paper reports 95% confidence intervals based on standard errors calculated from 10 independent runs per dataset. For the main challenge evaluation, results are reported on the fixed 1,000-utterance non-blind test set without cross-validation, following the URGENT Challenge protocol. Strategy selection (VQScore threshold, training target comparison) is performed on the validation set, not the test set. For the unseen-language experiment (Table 4), no cross-validation is reported — the evaluation is on the full FLEURS subsets for the selected languages.

Main Quantitative Results

Training Target Comparison: Time-Shifted Anechoic vs. Early-Reflected Speech

The central comparison is between models trained with the conventional early-reflected speech target and those trained with the proposed time-shifted anechoic clean speech target, both evaluated on the URGENT 2025 non-blind test set (1,000 utterances).

Table 1 (rows "Early reflected" and "Shifted anechoic") reports the core results. Replacing the early-reflected target with the time-shifted anechoic target yields substantial improvements on all three non-intrusive quality metrics: DNSMOS increases from 3.06 to 3.25, NISQA from 3.23 to 3.85, and UTMOS from 2.26 to 2.76. The ASR character accuracy (CAcc) — which is not affected by the reference mismatch — increases from 87.62 to 89.41, indicating that the quality gains do not come at the expense of hallucinated or incorrect content.

The shaded intrusive and task-independent metrics in Table 1 move in the opposite direction — PESQ drops from 2.81 to 2.41, ESTOI from 0.85 to 0.77, SDR from 12.28 to 8.23, and SBERT from 0.90 to 0.89 — but the paper explicitly flags that these metrics are not directly comparable because the URGENT evaluation uses early-reflected speech as the clean reference. Since the shifted anechoic model is trained to remove early reflections while the reference still contains them, the "distortion" measured by these metrics includes the early reflection energy that the model correctly removed. Table 6 in the Appendix resolves this by computing the same metrics using anechoic clean speech as the reference. Under this consistent reference definition, "Shifted anechoic + GAN" achieves: PESQ 2.71 vs. 2.40 for "Early reflected + GAN", ESTOI 0.78 vs. 0.69, SBERT 0.89 vs. 0.88, LPS 0.86 vs. 0.83, and SpkSim 0.84 vs. 0.83. The shifted anechoic target substantially outperforms the early-reflected target on every metric when the reference is correctly aligned with the training objective.

Figure 3(b) provides the learning curve evidence. The UTMOS validation curves show that reducing the early reflection window from 50 ms (the standard) progressively to 0 ms (time-shifted anechoic) monotonically improves performance at every step. The 0 ms window achieves the highest UTMOS throughout training, and the gap is visible from early epochs and persists to convergence. The unshifted anechoic target $s[n]$ (blue line) performs worst, consistent with prior findings about alignment difficulty.

Figure 8 (Appendix) visualizes the spectrographic difference. Enhanced spectrograms from the shifted anechoic model show cleaner harmonic structure and reduced reverberation smearing compared to those from the early-reflected model, particularly visible in the temporal gaps between speech segments where reverberation energy persists in the early-reflected output but is suppressed in the shifted anechoic output.

The paper also compares against the challenge leaderboard systems (Table 1). The "Shifted anechoic" model (without GAN correction) achieves non-intrusive metrics (DNSMOS 3.25, NISQA 3.85, UTMOS 2.76) that substantially exceed all three top-ranked systems (Rank 1: 3.01/3.41/2.40, Rank 2: 3.01/3.21/2.30, Rank 3: 3.00/3.45/2.31) and the challenge baseline (2.94/2.89/2.11). The CAcc of 89.41 also exceeds Rank 1 (88.92). However, on intrusive metrics, the shifted anechoic model's scores are lower due to the reference mismatch — when corrected in Table 6, the shifted anechoic model shows PESQ 2.71 vs. the early-reflected + GAN's 2.40 and ESTOI 0.78 vs. 0.69, confirming that the actual fidelity is higher than the leaderboard suggests.

Spectrogram comparison in Figure 8 demonstrates the effect: enhanced spectrograms from the shifted anechoic model show cleaner harmonic structure and reduced reverberation smearing compared to those from the early-reflected model, with the difference most visible in the temporal gaps between speech segments where reverberation energy persists in the early-reflected output.

Two-Stage Framework: GAN Correction

Table 1 also reports the effect of adding GAN correction to both target types. For the early-reflected target, GAN correction changes metrics modestly: NISQA improves from 3.23 to 3.53, SpkSim from 0.82 to 0.85, CAcc from 87.62 to 88.13, while UTMOS moves slightly from 2.26 to 2.30 and DNSMOS dips marginally from 3.06 to 3.04. For the shifted anechoic target, GAN correction provides more pronounced gains: NISQA jumps from 3.85 to 4.12 (the highest NISQA score in the entire table), SpkSim from 0.82 to 0.83, CAcc from 89.41 to 89.88, and UTMOS from 2.76 to 2.80. Intrusive metrics remain largely unchanged (PESQ 2.38 vs. 2.41, ESTOI 0.76 vs. 0.77, SDR 8.18 vs. 8.23), consistent with the claim that GAN correction improves perceptual quality while preserving MMSE-level fidelity.

The residual correlation analysis provides mechanistic evidence for the fidelity-preservation claim. The paper computes the average correlation coefficient between the magnitude-residual spectrograms $s - \hat{s}$ (clean minus regression output) and $\tilde{s} - \hat{s}$ (final output minus regression output) on the non-blind test set, obtaining a value of 0.78. This high correlation indicates that the GAN corrections are strongly aligned with the regression model's errors — the GAN modifies the signal primarily where the regression model failed, not where it succeeded. Figures 5 and 6 in the Appendix provide spectrographic visualization: in regions where the noisy input has severe degradation (bandwidth limitation or strong noise, highlighted in green boxes), the regression output is over-smoothed, and the GAN restores spectral detail; in regions where the regression output is already accurate, the GAN adds negligible correction.

Table 7 (Appendix) shows the full leaderboard including the paper's variants. "Our Early reflected + GAN" ranks 2nd overall (score 3.166) behind Bobbsun (2.516), while "Our Early reflected" ranks 4th (4.041). The shifted anechoic variants are not ranked on this leaderboard because the intrusive metrics are computed with the early-reflected reference, making the comparison unfair. Instead, the paper emphasizes the non-intrusive metrics and CAcc where the shifted anechoic + GAN model achieves the best results: DNSMOS 3.26, NISQA 4.12, UTMOS 2.80, and CAcc 89.88 — all state-of-the-art on this benchmark.

Figure 9 (Appendix) compares the paper's two-stage GAN correction against a conventional approach: pre-training with regression loss followed by adversarial fine-tuning (i.e., unfreezing and fine-tuning the regression model adversarially rather than training a separate frozen residual generator). Across training, the two-stage framework consistently achieves lower magnitude loss, lower phase loss, lower time-domain loss, and higher PESQ scores on the validation set. This demonstrates that the residual correction approach with a frozen regression backbone is not merely an alternative to standard adversarial fine-tuning — it is strictly better on all measured loss dimensions.

Data Quality Filtering Results

Figure 3(a) shows the UTMOS learning curves for models trained with three VQScore filtering thresholds. The key result: without VQScore filtering (threshold 0.50, retaining all 2,518 hours, blue line), the model achieves the lowest UTMOS throughout training, plateauing at a lower level than either filtered variant. This is notable because the data had already been filtered by challenge organizers using DNSMOS — meaning DNSMOS-based filtering is insufficient to remove all quality-degrading samples. With aggressive filtering (threshold 0.72, retaining only 629 hours, green line), the model initially achieves the best performance due to higher data quality, but eventually plateaus and is overtaken by the moderately filtered model (threshold 0.65, retaining 2,506 hours, orange line), which achieves the highest final UTMOS. This demonstrates the quality-quantity trade-off: removing too much data sacrifices diversity, but removing the worst contaminants (moving from 0.50 to 0.65, discarding only ~12 hours) provides a clear benefit.

Table 1 (row "Early reflected") uses the VQScore 0.65 filtered data for all experiments, so the baseline early-reflected target model already benefits from this curation. The paper does not report a direct ablation of VQScore filtering on the test set for the shifted anechoic model — the learning curves in Figure 3(a) are on the validation set and use time-shifted anechoic targets for all three thresholds. The test set results in Table 1 all use threshold 0.65.

Comparison with Open-Source USE Models

Table 2 reports results on subsets of the URGENT 2025 non-blind test set matching the supported sampling rates of two open-source USE models. On the 48 kHz subset, the proposed model (shifted anechoic + GAN) achieves DNSMOS 3.31, NISQA 4.41, UTMOS 3.55, PESQ 2.65, CAcc 92.50, compared to ClearerVoice-Studio (regression-based, MossFormer2) at DNSMOS 2.97, NISQA 3.38, UTMOS 3.02, PESQ 2.09, CAcc 89.90. The proposed model outperforms on all metrics. On the 44.1 kHz subset, Resemble Enhance (latent conditional flow matching) achieves high non-intrusive scores (DNSMOS 3.13, NISQA 3.68) but catastrophically poor intrusive metrics (PESQ 1.33, ESTOI 0.45, SDR -15.01) and CAcc of only 47.20, consistent with the hallucination problem of purely generative models. The proposed model achieves DNSMOS 3.32, NISQA 4.15, PESQ 2.28, CAcc 92.20 — substantially better fidelity and content preservation with competitive or better perceptual quality.

Unseen Language Generalization

Table 4 evaluates on three languages unseen during USE training (Italian, Dutch, Japanese) from the FLEURS dataset. The original FLEURS recordings have DNSMOS scores of 3.12 (Italian), 2.99 (Dutch), 2.96 (Japanese) and CAcc scores of 97.28, 97.40, 95.34 respectively — already relatively clean, but with residual degradations. FLEURS-R (Miipher-2 processed) improves DNSMOS to 3.37/3.36/3.36 but reduces speaker similarity (SpkSim) to 0.87/0.88/0.88 and slightly reduces CAcc in some cases. The proposed model (shifted anechoic + GAN) achieves SpkSim of 0.98/0.97/0.98 — substantially better speaker preservation than FLEURS-R — while maintaining competitive DNSMOS (3.20/3.13/3.07) and CAcc (97.00/97.18/95.30). Interestingly, DNSMOS is lower for the proposed model than FLEURS-R, but CAcc is comparable or better, suggesting that FLEURS-R's higher DNSMOS may reflect some hallucinated content that sounds natural but doesn't match the original. The Proposed (EARS) variant — fine-tuned exclusively on the cleanest EARS subset — improves DNSMOS to 3.27/3.28/3.18 and CAcc to 98.09/97.26/95.43, though at a small cost to SpkSim (0.97/0.95/0.95). This demonstrates that the data quality ceiling identified during training data analysis directly impacts performance on real-world unseen conditions, and that targeted fine-tuning on studio-quality data can partially address residual artifact removal.

Downstream TTS Application

Table 3 evaluates the impact of USE-based data cleaning on zero-shot multilingual TTS for Dutch and Italian (languages unseen during USE training). When both context audio and training audio are enhanced by the proposed USE model (full pipeline: VQScore 0.65 filtered + shifted anechoic + GAN correction), Dutch TTS performance improves substantially: character error rate drops from 14.28% to 7.75%, word error rate from 19.60% to 13.66%, and speaker similarity increases from 0.6064 to 0.6603, with Fréchet codec distance (FCD) decreasing from 0.2444 to 0.1837 (lower is better). Italian shows similar trends: CER from 11.13% to 8.30%, WER from 19.20% to 15.98%, FCD from 0.1846 to 0.1373, though SpkSim remains essentially unchanged (0.6004 to 0.6006). Table 8 (Appendix) provides a more granular breakdown, showing that enhancing either the context audio or the training audio independently provides intermediate benefits — for Dutch, enhancing only context improves SpkSim from 0.6064 to 0.6643 (the largest single-factor gain for speaker similarity) while enhancing only training audio reduces CER to 8.24% and FCD to 0.1761. The full enhancement (both context and training audio) achieves the best overall combination of CER, WER, and SpkSim for Dutch, and the best CER, WER, and FCD for Italian. All 95% confidence intervals (from 10 independent runs) show non-overlapping improvements for the full enhancement condition compared to the original condition on CER and WER for both languages.

Ablation Studies and Robustness Checks

Early reflection window width: Figure 3(b) systematically sweeps the early reflection window from 50 ms (the URGENT standard) down to 0 ms (proposed time-shifted anechoic) in 10 ms increments. The UTMOS learning curves show monotonic improvement at every step — there is no regime where preserving some early reflections is beneficial, directly contradicting the psychoacoustic rationale for the conventional target.

VQScore threshold selection: Figure 3(a) compares thresholds of 0.50 (no filtering), 0.65, and 0.72. The 0.65 threshold achieves the best final validation UTMOS, with 0.72 showing better early-training performance but eventually plateauing lower due to reduced data diversity (629 vs. 2,506 hours). This establishes that the quality-quantity trade-off has a clear optimum and that aggressive filtering can be counterproductive.

GAN correction paradigm comparison: Figure 9 (Appendix) compares the proposed two-stage frozen-regression + residual GAN correction against conventional pre-training + adversarial fine-tuning (where the regression model is unfrozen and fine-tuned adversarially). The two-stage approach achieves lower magnitude loss, lower phase loss, lower time-domain loss, and higher PESQ throughout training on the validation set, confirming that the residual correction with a frozen backbone is superior to standard adversarial fine-tuning.

EARS fine-tuning for residual artifact removal: Tables 4 and the FLEURS experiment demonstrate that even after VQScore 0.65 filtering, the model fails to remove subtle artifacts like electrical microphone hiss. Fine-tuning exclusively on EARS (the cleanest 100-hour subset) — producing "Proposed (EARS)" — improves DNSMOS and CAcc on unseen FLEURS data, confirming that the performance ceiling imposed by residual training data noise is real and can be raised through curation. The cost is a small reduction in SpkSim (0.98→0.97 for Italian, 0.97→0.95 for Dutch, 0.98→0.95 for Japanese), suggesting a trade-off between aggressive noise removal and speaker characteristic preservation.

Clean reference definition: Table 6 (Appendix) recomputes intrusive metrics using anechoic clean speech as the reference (instead of the URGENT Challenge's early-reflected reference). Under this consistent definition, "Shifted anechoic + GAN" achieves PESQ 2.71 vs. 2.40 for "Early reflected + GAN", ESTOI 0.78 vs. 0.69, SBERT 0.89 vs. 0.88, LPS 0.86 vs. 0.83, and SpkSim 0.84 vs. 0.83. This ablation confirms that the apparent degradation in intrusive metrics on the leaderboard is entirely due to reference mismatch, not actual quality degradation.

Multi-band discriminator design: The paper proposes an adaptive multi-band discriminator that processes frequency sub-bands separately based on input sampling rate. While no explicit ablation removing this component is reported, the design is motivated by the observation that different degradations affect different frequency regions differently (bandwidth limitation affects high frequencies, wind noise is concentrated at low frequencies). The consistent performance across sampling rates (Table 2 shows strong results at both 44.1 kHz and 48 kHz) provides indirect support.

Regression model frozen vs. trainable in Stage 2: The paper explicitly freezes the regression model weights before GAN training. No ablation is reported comparing frozen vs. fine-tuned regression in this specific framework, but the comparison against standard adversarial fine-tuning (Figure 9, where the regression model IS updated) shows that the frozen approach is superior. The residual correlation of 0.78 also indirectly validates the freezing choice — if the regression model were updated during GAN training, the GAN corrections would not align as cleanly with the original regression errors.

Negative result: ReST^EM for revision model training: While this paper does not use revision models (unlike the reference example paper), it reports a negative finding related to training methodology. The paper's USE model does not use iterative refinement or RL-based training, so no analogous negative result is present. The closest to a negative result is the finding that FLEURS-R (Miipher-2), a purely generative restoration model, achieves higher DNSMOS but substantially worse speaker similarity and CAcc than the proposed approach (Table 4) — a demonstration of the hallucination problem the two-stage framework is designed to avoid.

Negative result: overly aggressive VQScore filtering: The threshold 0.72 experiment (Figure 3a) shows that removing too much data (retaining only 629 hours) degrades final performance compared to moderate filtering (0.65, retaining 2,506 hours), despite better early-training performance. This is an informative negative result — it establishes that data quality and quantity must be balanced, and that in the USE setting with diverse degradation types and languages, retaining sufficient coverage matters.

Critical Assessment

The paper makes three central claims, and the experimental support varies in strength and completeness across them.

Claim 1: Time-shifted anechoic clean speech consistently outperforms early-reflected speech as a dereverberation target in USE.

This claim is strongly supported by the reported experiments. Figure 3(b) shows a clean monotonic trend across the full sweep of early reflection windows. Table 1 shows substantial gains on non-intrusive metrics (NISQA from 3.23 to 3.85, UTMOS from 2.26 to 2.76) and ASR accuracy (87.62 to 89.41). Table 6 confirms that the apparent intrusive metric degradation is an artifact of the reference mismatch. Figure 8 provides qualitative spectrographic evidence. The key strength is the granularity of the evidence — the sweep from 50 ms down to 0 ms leaves no ambiguity about the direction of the effect.

However, the claim's generalizability is limited in two ways. First, all experiments use a single model architecture (USEMamba). While the authors argue the finding is about training targets rather than architecture, it is possible that other architectures with different capacity or inductive biases would show a different pattern — for instance, a smaller model might indeed introduce artifacts when attempting full dereverberation, validating the conventional wisdom. Second, $n_0$ estimation from the RIR (as $\arg\max_n |r[n]|$) is trivially available during training but not during inference on real recordings where the RIR is unknown. The paper does not address how time-shift alignment would work in a fully blind setting — it implicitly assumes that at inference, the model learns to produce correctly time-aligned output from the input signal, which the training target enables. A direct test of this assumption (e.g., evaluating whether the model's output is correctly time-aligned on blind test samples) is not reported.

A missing experiment: direct comparison of the three targets ($s[n]$, $s_e[n]$, $s[n-n_0]$) on the test set (not just validation learning curves) with both early-reflected and anechoic references, in a single table, would have made the evidence for this claim more self-contained. Currently, the reader must cross-reference Table 1 (with reference mismatch caveat) and Table 6 (with anechoic reference but only for the GAN-corrected variants) to reconstruct the full picture.

Claim 2: The two-stage framework (frozen regression + residual GAN correction) achieves an optimal fidelity-perception trade-off grounded in optimal transport theory.

This claim has strong empirical support for the specific implementation tested, but the theoretical grounding should be understood as motivating rather than proving optimality. The theory says that under squared-error distortion and Wasserstein-2 perceptual constraint, the optimal estimator transports the posterior mean to the clean distribution. The paper operationalizes this with a Wasserstein GAN (which minimizes Wasserstein-1) and a residual connection. The empirical results — NISQA of 4.12, CAcc of 89.88, 0.78 residual correlation, and Figure 9 showing consistent improvement over standard adversarial fine-tuning — demonstrate that this specific operationalization works well. But the paper does not demonstrate that it achieves the theoretical optimum $D(0)$, nor does it benchmark against alternative operationalizations of the same theory (e.g., using flow matching as in Ohayon et al., 2025 for images, or diffusion-based transport). The claim "achieves an optimal fidelity-perception trade-off" is therefore more accurately read as "achieves a strong fidelity-perception trade-off motivated by optimal transport theory."

The residual correlation of 0.78 is informative but incomplete as a fidelity-preservation metric. A correlation of 0.78 leaves 39% of variance unexplained — some GAN corrections are applied in regions where the regression model was already accurate, or vice versa. The paper does not report whether these "mismatched" corrections cause any measurable degradation (e.g., word error rate on specific phonemes, speaker similarity on stable vowel segments). The Lipschitz argument provides a structural reason why the GAN should focus on erroneous regions, but it does not guarantee perfect fidelity preservation — it guarantees that when the GAN output is zero in a region, the discriminator won't penalize it, not that the GAN output will actually be zero in correct regions.

A missing experiment: direct measurement of content preservation on the specific degradation types where generative models are known to hallucinate (packet loss, severe bandwidth limitation). Table 1 reports aggregate CAcc, but per-condition breakdowns would reveal whether the GAN correction introduces any content errors in the most challenging scenarios. The favorable comparison against Resemble Enhance (Table 2) provides some evidence that the approach avoids severe hallucination, but a finer-grained analysis would strengthen the claim.

The comparison against prior multi-stage systems is also somewhat incomplete. Table 1 compares against Rank 1-3 challenge systems, but these systems use different base architectures, different training data (potentially without VQScore filtering), and different training recipes — so the performance gap cannot be attributed solely to the two-stage framework. A controlled ablation where the same USEMamba base model is used in a three-stage or four-stage configuration would isolate the contribution of the two-stage design from other confounding factors. This is not reported.

Claim 3: Training data quality imposes a hard performance ceiling, and VQScore-based filtering improves USE performance.

The evidence for this claim is solid but limited in quantitative precision on the test set. Figure 3(a) convincingly shows that VQScore filtering at 0.65 improves validation UTMOS over no filtering (0.50), establishing the existence of a data quality effect. The FLEURS experiment (Section 3.8, Table 4) provides a compelling real-world demonstration: even after VQScore filtering, the model fails to remove subtle artifacts, and fine-tuning on the cleanest subset (EARS) addresses this — a clean demonstration of the ceiling mechanism.

However, the paper does not report the direct impact of VQScore filtering on the URGENT test set metrics, only on validation UTMOS curves. A test-set evaluation of the three thresholds (0.50, 0.65, 0.72) on the standard URGENT metrics — analogous to the target comparison in Table 1 — would quantify exactly how much the ceiling costs in practice. The learning curves are suggestive but do not tell us whether the validation UTMOS differences translate to test-set DNSMOS, NISQA, CAcc, etc. This is a notable omission given that the data quality claim is one of three central contributions.

The claim of a "hard performance ceiling" also overstates what the evidence shows. A hard ceiling would mean that no amount of model scaling, architectural improvement, or training compute can improve performance beyond a certain point when training on contaminated data. The paper demonstrates that VQScore filtering helps, and that EARS fine-tuning helps further, but does not test whether, say, a 5× larger model trained on the unfiltered data would reach similar UTMOS as the smaller model trained on filtered data. The term "ceiling" implies an asymptotic bound that the experiments do not probe — it is more accurately described as a "performance penalty" from training on lower-quality data that can be mitigated through curation.

The VQScore threshold selection raises a potential concern about test-set leakage. The threshold of 0.65 was selected based on validation UTMOS (Figure 3a). VQScore is itself a learned model, and while it is trained only on clean speech, it is used to filter training data that will then be evaluated by other learned metrics (UTMOS for validation, DNSMOS/NISQA for test). If VQScore shares inductive biases with these non-intrusive metrics, filtering by VQScore could artifactually improve those specific scores without corresponding improvements in true perceptual quality. Subjective listening tests would address this concern definitively but are not reported.

Overall experimental design strengths:

The multi-metric evaluation covering non-intrusive quality, intrusive fidelity, intelligibility, speaker similarity, and ASR accuracy is unusually comprehensive and allows the reader to assess trade-offs across dimensions. The explicit flagging and resolution of the reference mismatch problem (Table 6) demonstrates methodological transparency. The FLEURS unseen-language experiment and the TTS downstream application provide external validation that the gains are not specific to the URGENT benchmark. The open-source model comparison (ClearerVoice-Studio, Resemble Enhance) contextualizes the results within the broader USE landscape.

Overall experimental design weaknesses:

The single-architecture evaluation (USEMamba only) limits claims about the universality of the findings. The VQScore filtering test-set impact is not quantified. The comparison against prior multi-stage challenge systems is confounded by multiple factors (architecture, training data, training recipe). The $n_0$ estimation requirement for the training target is not addressed for fully blind inference. No subjective listening tests are reported despite the emphasis on perceptual quality. The test set is relatively small (1,000 utterances) for claims about universal enhancement across seven distortion types, five languages, and seven sampling rates — per-condition sample sizes may be quite small. Finally, no confidence intervals or statistical significance tests are reported for the main challenge results (Table 1), making it difficult to assess whether the differences between, say, "Shifted anechoic + GAN" (CAcc 89.88) and "Rank 1" (CAcc 88.92) are statistically reliable.

6. Limitations and Trade-offs

Difficulty Estimation Requires Impractically Expensive Oracle Computation

The entire compute-optimal framework for test-time compute allocation rests on estimating each prompt's difficulty before deciding how to spend the inference budget. The paper's method for difficulty estimation — generating 2,048 complete solutions per question from the base model and computing the pass@1 rate — requires access to ground-truth correctness labels and consumes enormous computation that is not accounted for in any reported efficiency metric. As the authors acknowledge in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference … our experiments do not account for this cost largely for simplicity"

This is a foundational gap between the reported results and any practical deployment. The headline claim of 4× efficiency improvement over best-of-N is computed after difficulty is already known, with no amortization of the cost of learning it. A realistic deployment must either (a) pay the difficulty estimation cost upfront, which can exceed the largest test-time budgets studied (2,048 samples for estimation vs. 256–512 generations for solving), or (b) deploy the predicted-difficulty variant, which replaces ground-truth correctness with PRM final-answer scores but still requires generating 2,048 samples per question. The paper reports that both oracle and predicted difficulty bins produce similar curves in Figures 4 and 8, but this does not address the fundamental issue: neither variant accounts for estimation cost.

The consequence is that the 4× figure should be understood as an unrealized upper bound, not a demonstrated practical gain. In deployment, a system must decide how to split its total compute budget between difficulty estimation and strategy execution. The paper provides no framework for optimizing this split, no measurement of how many samples are actually needed for reliable difficulty estimation (2,048 may be far more than necessary for coarser binning), and no exploration of cheaper estimation approaches such as using a small number of initial samples to dynamically adjust the remaining budget.

The paper explicitly flags this as future work: "our experiments do not account for this cost largely for simplicity" and suggests that future work could train models to predict difficulty directly from the question text (Section 3.2). This limitation is transparently acknowledged but fundamentally limits the practical applicability of the current method in its reported form. Until cheaper difficulty estimation is demonstrated, the compute-optimal policy is a valuable analytical framework rather than a directly deployable system.

The Method Fails Completely on the Hardest Problems

Across every method studied — PRM search, iterative revisions, and their compute-optimal combinations — the hardest difficulty quintile (bin 5) shows essentially zero improvement regardless of how much test-time compute is allocated. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all generation budgets. In Figure 7 (right), bin 5 accuracy is roughly 2–3% regardless of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling curve is essentially flat near 0–5% across all test-time compute budgets.

This is not a minor degradation but a complete failure mode: when the base model's pass@1 is near zero, no amount of search, revision, or adaptive allocation helps because there are essentially no correct solutions in the proposal distribution to find or refine. The paper is candid about this boundary, stating in the Section 7 takeaway:

"test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time"

The practical consequence is that any deployment facing a significant fraction of genuinely hard problems (outside the base model's approximate capability range) must fall back to pretraining-scale solutions or accept near-zero accuracy on those problems. The compute-optimal framework provides no mechanism for dynamically escalating to a larger model when difficulty exceeds a threshold — the difficulty estimation tells you the problem is hard, but the paper does not propose routing such problems to a more capable model rather than spending compute fruitlessly.

The methodological limitation here is that the paper's difficulty bins are computed relative to the specific base model (PaLM 2-S*) . A problem that falls in bin 5 for this model might be solvable with test-time compute for a more capable base model. The paper does not explore how the difficulty distribution shifts with model scale, which would be valuable for understanding when test-time compute becomes viable as base models improve.

Only One Benchmark and One Model Family Are Tested

All experiments in this paper use a single benchmark (MATH, 500 test questions) and a single model family (PaLM 2-S*). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an untested assumption. Several aspects of the findings could be model-specific or benchmark-specific in ways that affect the generality of the claims:

• Model-specific PRM behavior: The PRM is trained using Monte Carlo rollouts from PaLM 2-S* . Its calibration, over-optimization threshold, and difficulty-estimation quality depend on this specific model's output distribution. A model with different error patterns (e.g., one that makes different types of mistakes, or has different confidence calibration) might exhibit qualitatively different difficulty-dependent scaling curves — potentially changing which strategies are optimal at which difficulty levels.

• Model-specific revision capabilities: The revision model is fine-tuned from PaLM 2-S* using a specific data generation procedure (edit-distance-based pairing, offline trajectory construction). The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The 38% correct-to-incorrect reversion rate (Section 6.1) is likely model-specific.

• Benchmark-specific difficulty distribution: MATH consists of competition-level math problems requiring formal symbolic reasoning. The difficulty-dependent patterns — beam search over-optimizing on easy problems, revisions helping on medium problems, nothing helping on hard problems — are demonstrated only for this task type. Tasks requiring factual recall rather than multi-step inference, or tasks with different error modes (e.g., code generation where partial correctness is more gradable), might exhibit entirely different difficulty–strategy relationships.

• Small test set for strategy selection: The 500-question test set is divided into five difficulty quintiles of approximately 100 questions each, then further split by two-fold cross-validation. This means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. This is a small sample for optimizing over a discrete space of strategies (search algorithm × beam width × revision depth × sequential-to-parallel ratio), and the selected policies may have high variance. The paper does not report confidence intervals on the compute-optimal scaling curves.

The paper partially acknowledges this limitation (Section 8) but does not test on additional benchmarks, domains, or model families. The findings about verifier over-optimization and the difficulty-dependence of strategy effectiveness are conceptually important enough that they should generalize, but the paper provides no empirical evidence that they do.

The Revision Model Has a High Correct-to-Incorrect Reversion Rate

A significant practical issue arises from the revision model's training data construction. Because the model was trained exclusively on sequences where all in-context answers are incorrect (followed by a correct target), it has never seen a training example where the current answer is already correct. At test time, when the revision chain produces a correct answer, the model has no learned behavior for "leave this alone" — and instead tends to incorrectly "revise" correct answers into incorrect ones. The paper quantifies this (Section 6.1):

"approximately 38% of correct answers get converted back to incorrect ones"

This reversion rate is substantial and is not a rare edge case — it means that in a long revision chain, a correct answer produced at step k has a 38% probability of being corrupted at step k+1. The paper mitigates this with within-chain selection: rather than simply taking the last revision output, the system uses majority voting or verifier-based selection across the entire chain to pick the best answer from any step. While this patch works (as demonstrated by the revision model's overall performance in Figure 6), it is a post-hoc fix that wastes computation: the model generates revisions that actively degrade correct answers, and the selection mechanism must detect and reject those degraded outputs.

The deeper issue is that the revision model does not learn when to stop revising. It has no internal mechanism for recognizing that the current answer is correct and that further revision is unnecessary or harmful. This is a direct consequence of the training data construction (only incorrect-to-correct trajectories) and would require a fundamentally different training approach to address — for instance, including trajectories where the correct answer is already present and the model learns to output it unchanged.

The ReST$^{EM}$ experiment (Appendix K, Figure 16) provides additional evidence that revision training is fragile: "attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions." This suggests that the positive revision results depend on specific training choices (offline data construction, edit-distance-based pairing) that may not transfer to other settings or be robust to further optimization. The paper acknowledges this as an open problem in Section 8 but does not propose a solution.

The 14× Larger Model Baseline Is Weaker Than It Should Be for a Fair FLOPs-Matched Comparison

The FLOPs-matched comparison in Section 7 is one of the paper's most attention-grabbing claims: that a smaller model with compute-optimal test-time scaling can outperform a ~14× larger model on easy-to-medium problems. However, the larger model baseline contains two design choices that make the comparison favorable to test-time compute:

Parameter-only scaling, not compute-optimal pretraining. The paper scales model parameters by 14× while holding training data constant, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022) where both parameters and data are scaled. The authors explicitly acknowledge this (Section 7):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

A compute-optimally trained larger model (with appropriately scaled data) would likely outperform a parameter-only-scaled model at the same total FLOPs, making the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining may shrink or reverse when compared against a properly compute-optimal larger model.

The larger model uses only greedy decoding with no test-time compute. The comparison is between PaLM 2-S* with compute-optimal test-time scaling and a 14× larger model with greedy decoding and zero test-time augmentation. This is an asymmetric comparison: one side gets to use its entire budget adaptively, while the other gets no inference-time optimization at all. A fairer baseline would give the larger model a proportional test-time compute budget — even a modest best-of-8 or best-of-16 would significantly improve the larger model's performance and potentially erase or reverse the reported advantages of the smaller model with test-time compute.

The paper's results in Figure 9 are most credible at the extremes: at $R \ll 1$ (where inference tokens are a tiny fraction of training tokens, so the larger model's higher per-token cost barely affects the budget), and at $R \gg 1$ and hard difficulty (where test-time compute provides essentially zero benefit anyway). At intermediate regimes — $R \approx 1$, medium difficulty — the comparison is most sensitive to the baseline's weakness. The paper does not report what fraction of real-world deployment scenarios fall into each $R$ regime, making it difficult for practitioners to assess whether the favorable comparison applies to their use case.

Single Benchmark and Model Family Severely Limit Generality

All experiments in this paper are conducted on a single benchmark (MATH, 500 test questions) using a single model family (PaLM 2-S*). The paper states the model is "representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is untested. Multiple aspects of the findings could be model-specific:

• PRM behavior: The PRM's quality, calibration, and over-optimization characteristics depend on PaLM 2-S*'s output distribution. Different model families with different error patterns, confidence calibration, or reasoning styles could produce qualitatively different difficulty-dependent scaling curves. For example, a model that tends to produce overly confident but incorrect step-by-step reasoning might cause the PRM to overfit to spurious step-level patterns, changing where over-optimization occurs.

• Revision model transferability: The revision model is fine-tuned from PaLM 2-S* using a specific offline data generation procedure. The model's ability to learn from incorrect in-context examples — and the 38% correct-to-incorrect reversion rate — depend on the base model's in-context learning capabilities, which vary substantially across architectures and training paradigms.

• Benchmark specificity: MATH consists exclusively of competition-level math problems requiring formal symbolic reasoning with clear correctness criteria. This is a very specific task profile. The difficulty-dependent patterns (beam search hurting easy problems, revisions helping medium ones, nothing helping hard ones) are demonstrated only for this task type and this correctness structure. Tasks requiring factual recall rather than multi-step deduction, or tasks with gradable partial correctness (code generation, summarization), might exhibit entirely different strategy–difficulty relationships.

• Statistical power for strategy selection: The 500-question test set divided into five difficulty quintiles (~100 each), further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample for optimizing over a discrete space of strategies, and the paper reports no confidence intervals on the compute-optimal scaling curves. The selected strategies may have high variance and may not generalize to other benchmarks or model families.

The paper partially acknowledges this scope (Section 8) but provides no empirical evidence that the findings transfer to other settings. The conceptual framework — difficulty-conditioned allocation, the complementary strengths of search and revisions — is valuable and likely generalizes, but the specific quantitative findings (4× efficiency gain, optimal beam width, optimal sequential-to-parallel ratios) are tied to this specific model–benchmark combination and should not be assumed universal without replication.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around universal speech enhancement from an architecture-centric pursuit — where progress is measured by increasingly complex multi-stage pipelines — toward a design-methodology-centric view where three overlooked choices (training target, fidelity–quality balancing strategy, and data curation) are recognized as the binding constraints on performance. The shift is not a paradigm overthrow but a substantial reframing of where progress comes from: the paper demonstrates that careful attention to what you predict, how you structure the prediction, and what you train on can yield gains that exceed those from architectural complexity alone. The evidence for this is the paper's state-of-the-art non-intrusive quality scores (NISQA of 4.12, UTMOS of 2.80) and ASR accuracy (CAcc of 89.88) achieved with a relatively standard USEMamba backbone — outperforming systems with three and four specialized stages and more elaborate architectures.

The most consequential reframing is for dereverberation research. The paper provides a clean decomposition of a problem the field had misdiagnosed for years: the difficulty of dereverberation is not the removal of early reflections per se but the alignment ambiguity introduced by the unknown direct-path time shift. By showing that time-shifted anechoic targets — with the time shift $n_0$ estimated trivially from the RIR during training — consistently outperform early-reflected targets across every early reflection window from 50 ms down to 0 ms (Figure 3b), the paper effectively invalidates the conventional wisdom that retaining early reflections is necessary to avoid artifacts. This finding, if replicated across architectures and datasets, should change the default dereverberation target in future benchmarks and challenge specifications. The URGENT Challenge's use of early-reflected speech as the evaluation reference — and the resulting metric mismatch documented in Table 6 — is no longer a minor inconvenience but a systematic bias that penalizes models trained on the correct target. Future challenge organizers should adopt anechoic clean speech references or, at minimum, report both.

For the fidelity–quality trade-off, the paper provides a theoretical grounding that was absent from prior multi-stage USE systems. The derivation in Appendix A.1 — showing that the minimal MSE under a perfect-perception constraint is achieved by transporting the posterior mean to the clean data distribution — does not merely justify a two-stage design; it specifies the structure the two stages should have: a frozen regression model computing the posterior mean, followed by a generative model that applies an optimal transport map. The paper operationalizes this with a Wasserstein GAN and a residual connection, but the theoretical framework is broader and suggests that alternative transport mechanisms (flow matching, diffusion) could be plugged into the same structure. This converts architecture design from an empirical search over stage counts and heuristics into a principled optimization over transport mechanisms — a more productive framing that connects USE to the broader optimal transport and generative modeling literatures.

The data quality findings represent the third shift. The paper demonstrates that even challenge-organizer-level filtering (DNSMOS thresholding, VAD) is insufficient to remove all quality-degrading samples from training data, and that the residual contaminants — stationary background noise, microphone hiss, non-speech artifacts — impose a performance ceiling that cannot be overcome by scaling model size or training time. This is not merely a "clean data helps" observation. The ceiling is mechanistic: the model learns to treat residual artifacts as features of clean speech, so it cannot remove similar artifacts from test inputs. The fact that fine-tuning on the highest-quality subset (EARS, 100 hours of studio recordings) produces measurable improvements on unseen real-world recordings (Table 4, Proposed vs. Proposed (EARS)) demonstrates that this ceiling is real and consequential. The implication for the field is that data curation — including quality estimation, source-specific filtering, and targeted fine-tuning on studio-quality subsets — should be treated as a first-class research activity in USE, not an afterthought to model development.

The paper also reconciles contradictory signals in the prior literature. The claim that early reflections are "too hard to remove" (Valin et al., 2022; Zhou et al., 2023; Zhao et al., 2020) was based on experiments in narrower settings with smaller models and less diverse training data. The negative results in those papers were not wrong — they accurately reflected the capabilities of their experimental setups — but they were misinterpreted as a fundamental limitation rather than a consequence of conflating alignment and removal difficulty. Similarly, the proliferation of multi-stage USE architectures (three-stage, four-stage, fusion-based) reflected a genuine need to balance fidelity and quality, but without a theoretical framework, researchers were adding complexity rather than structure. The paper's two-stage residual correction framework explains why two stages are sufficient and how they should be connected, implicitly arguing that additional stages are unnecessary complexity rather than fundamental improvements.

Follow-Up Research This Work Enables

Replication of the dereverberation target finding across architectures and datasets. The paper's most actionable claim — that time-shifted anechoic targets consistently outperform early-reflected targets — is demonstrated with a single architecture (USEMamba) on a single benchmark (URGENT 2025). A strong follow-up would replicate the Figure 3(b) sweep of early reflection windows (50 ms to 0 ms) using 3–5 diverse architectures: a convolution-based model (e.g., TF-GridNet), a Transformer-based model, a diffusion-based model, and a smaller capacity model (e.g., 10-layer USEMamba). If the monotonic improvement with decreasing window size holds across architectures, the finding becomes a robust design principle for dereverberation. If some architectures show a U-shaped curve (performance degrading at very small windows due to artifact introduction), that would reveal an architecture-dependence of the target effect and suggest that the conventional wisdom was correct for certain model classes. The experiment should report both non-intrusive metrics (unaffected by reference mismatch) and intrusive metrics computed with both early-reflected and anechoic references, following the paper's Table 6 protocol. This would provide the first systematic, multi-architecture evaluation of dereverberation targets and either validate or delimit the paper's central claim.

Blind time-shift estimation for fully inference-time dereverberation. The paper's time-shifted anechoic target relies on $n_0$ estimated from the known RIR during training — but real-world dereverberation has no RIR. The paper's model implicitly learns to produce time-aligned output because it was trained that way, but it is unknown how robust this implicit alignment is, particularly for unseen acoustic conditions, unusual room geometries, or very long direct-path delays. A follow-up study should train a model with the shifted anechoic target, then evaluate it on blind reverberant inputs (no RIR available) across a diverse RIR corpus (e.g., simulated rooms from 3×3×3m to 20×20×10m, real recorded RIRs from multiple venues). The key measurement is the time alignment error between the model's output and the ground-truth anechoic signal — computed via cross-correlation or dynamic time warping — and its correlation with output quality metrics. If the model generalizes its alignment capability well, alignment error should be small and uncorrelated with quality. If alignment degrades in extreme conditions (very large $n_0$, strong late reverberation), this would identify a failure mode that requires explicit blind time-shift estimation (e.g., a dedicated $n_0$ predictor network trained on mixtures). This experiment would determine whether the training target improvement fully transfers to the blind setting or requires an additional inference-time component.

Comparing transport mechanisms for the two-stage framework: GAN vs. flow matching vs. diffusion. The paper operationalizes the optimal transport map between posterior mean and clean distributions using a Wasserstein GAN, motivated by the theoretical connection and single-forward-pass efficiency. But the theory (Equation 9) does not specify which transport mechanism is optimal, and GANs are known to suffer from mode collapse and training instability. A controlled comparison should hold the frozen regression model constant (same USEMamba, same training data, same target) and compare three transport mechanisms for Stage 2: (1) the paper's residual Wasserstein GAN with multi-band discriminator, (2) a conditional flow matching model (following Ohayon et al., 2025, which applied posterior-mean rectified flow to image restoration), and (3) a conditional diffusion model (e.g., a score-based model conditioned on both noisy input and regression output, similar to StoRM but with the residual connection). Each variant should be evaluated on the full URGENT metric suite, with particular attention to: non-intrusive quality metrics (does flow matching achieve higher NISQA than GAN?), fidelity metrics (does diffusion introduce more hallucination at high guidance scales?), inference speed (how many forward passes per utterance for each method?), and the residual correlation with regression errors (does the 0.78 correlation change with transport mechanism?). This would provide the first systematic comparison of transport mechanisms for USE and determine whether the GAN choice in the paper is optimal or merely convenient.

Per-condition analysis of GAN correction: where does it help and where does it hurt? The paper reports aggregate metrics for the two-stage framework (Table 1, rows with "+GAN correction") and demonstrates a 0.78 residual correlation, but does not break down performance by distortion type. This is critical because the distortion–perception trade-off predicts that GAN correction should be most beneficial when the degraded input $y$ is least informative about the clean signal — exactly the conditions where regression models over-smooth: severe bandwidth limitation, packet loss, very low SNR noise. A follow-up study should take the frozen regression model and the GAN-corrected model, and evaluate both on each of the seven URGENT distortion types separately, measuring not just aggregate quality metrics but also content preservation specifically: phoneme error rate, word error rate, and speaker similarity on degradation types known to cause hallucination. The hypothesis is that GAN correction improves perceptual quality most on high-uncertainty conditions but may introduce content errors if the transport map overshoots (moving the posterior mean too far toward the prior). If content errors are concentrated in specific conditions, that would suggest a need for condition-specific correction strengths — e.g., stronger correction for bandwidth limitation, weaker for moderate noise — rather than a uniform GAN loss weight. If content errors are negligible across all conditions, the two-stage framework is validated as universally safe. The paper's comparison with Resemble Enhance (Table 2) hints at severe hallucination from purely generative models on the 44.1 kHz subset (CAcc 47.20), but the per-condition breakdown of their own method would reveal whether the residual GAN ever approaches that failure regime.

Data quality scaling laws: how does the performance ceiling shift with model scale and data volume? The paper demonstrates that VQScore filtering at 0.65 improves validation UTMOS over no filtering, and that EARS fine-tuning further helps on residual artifacts. But the claim of a "hard performance ceiling" from data contamination is not tested against model scaling: would a 3× larger USEMamba trained on unfiltered data reach the same UTMOS as the base model trained on filtered data? Would training on unfiltered data for 5× more steps close the gap? A controlled scaling experiment should train models at 3–4 sizes (e.g., 10-layer, 20-layer, 30-layer, 45-layer USEMamba) on both unfiltered (threshold 0.50) and filtered (threshold 0.65) data, measuring both final performance and the rate of improvement with model scale. If the unfiltered curve plateaus at a lower asymptote regardless of scale while the filtered curve continues to improve, that constitutes strong evidence for a true ceiling rather than a mere offset. Additionally, varying the amount of data within each quality tier (e.g., random subsets of the filtered data at 25%, 50%, 75%, 100%) would reveal whether the quality–quantity trade-off identified in Figure 3(a) shifts with model capacity — larger models may benefit more from additional lower-quality data than smaller models, or vice versa. This experiment would provide the first data quality scaling laws for speech enhancement, analogous to existing data quantity scaling laws (Zhang et al., 2024a; Gonzalez et al., 2024) but with quality as an explicit dimension.

USE-based TTS data cleaning across a wider range of low-resource languages and recording conditions. The paper's TTS experiment (Section 3.9, Table 3) demonstrates that USE-enhanced training data improves zero-shot TTS on Dutch and Italian — languages unseen during USE training. This is promising but tested on only two languages with relatively high-resource TTS data availability. A more comprehensive evaluation should select 8–12 languages spanning different language families, script types, and typical recording conditions — including truly low-resource languages where studio-quality recordings are essentially nonexistent and all available data is crowdsourced or scraped. For each language, train a TTS model on (a) original data, (b) USE-enhanced data, and (c) FLEURS-R-style enhanced data (using Miipher-2 or equivalent), and evaluate with both objective metrics (CER, WER, SpkSim, FCD) and subjective listening tests with native speakers. The key question is where the fidelity–quality trade-off matters most for downstream TTS: in high-resource settings, Miipher-2's hallucinated content (implied by lower SpkSim in Table 4) may be tolerable; in low-resource settings where lexical coverage is sparse, content errors from generative restoration could be catastrophic. The experiment would also test whether the Proposed (EARS) fine-tuning strategy — which improved residual artifact removal on FLEURS — generalizes to the diverse recording conditions in low-resource TTS data, or whether in-domain fine-tuning on representative clean samples from each target language is necessary.

Practical Applications and Downstream Use Cases

Cleaning noisy speech data for TTS training in low-resource languages. The most directly demonstrated application is using the USE model to preprocess TTS training data. The paper shows that enhancing both context and training audio with their model reduces character error rate from 14.28% to 7.75% for Dutch and from 11.13% to 8.30% for Italian in zero-shot TTS (Table 3). For organizations building TTS systems in languages where studio recordings are scarce — most languages other than English, Mandarin, and a handful of others — this provides a concrete pipeline: collect available noisy speech data (crowdsourced, YouTube, podcasts), run it through the USE model with VQScore 0.65 filtered training and shifted anechoic targets, and use the enhanced output as TTS training data. The fidelity preservation (SpkSim of 0.6603 for Dutch, 0.6006 for Italian in the fully enhanced condition) means the TTS model retains distinguishable speaker characteristics rather than collapsing to an averaged voice, which matters for multi-speaker TTS. The language-agnostic generalization demonstrated in Table 4 — the model was trained on English, German, French, Spanish, and Chinese but tested on Italian, Dutch, and Japanese — means this pipeline can be deployed on languages without any USE training data, as long as the degradation types are within the model's training distribution.

Real-time speech enhancement for VoIP and teleconferencing at diverse sampling rates. The paper's model operates on a single forward pass (no iterative sampling, no beam search) and handles inputs at any sampling rate from 8 to 48 kHz through the SFI-STFT and adaptive multi-band discriminator. The architecture is a 30-layer USEMamba followed by a 6-layer USEMamba GAN generator — two feed-forward passes total. For deployment in Voice over IP (VoIP) systems, video conferencing platforms, or hearing assistance devices, this means a single model can clean up speech degraded by room reverberation, background noise, microphone clipping, bandwidth compression, and packet loss, all at whatever sampling rate the system uses, without needing to detect or classify the degradation type. The non-intrusive quality gains are substantial on the 48 kHz subset (Table 2): DNSMOS 3.31 vs. 2.04 for unprocessed input (a 62% relative improvement toward the perfect score of ~4.5), NISQA 4.41 vs. 1.83, and CAcc 92.50 vs. 90.60. The practical impact is clearer, more intelligible calls without requiring users to upgrade their hardware or acoustic environment — the model handles the degradation at the software level. The caveat is that the paper reports no latency measurements, and a 30-layer Mamba model may not meet the sub-20ms latency requirements of real-time communication without optimization (pruning, quantization, or streaming architecture modifications).

Post-production cleanup for podcast and video content. Content creators working outside studio environments — field recordings, home studios, live streams — face a combination of degradation types that USE is designed to handle: room reverberation from untreated spaces, background noise (HVAC, traffic, appliance hum), occasional clipping from gain mismanagement, and bandwidth artifacts from compressed recording formats. The paper's model provides a single-pass cleanup that handles all of these simultaneously without requiring separate denoising, dereverberation, and declipping plugins with manual parameter tuning. The demonstration that the model removes early reflections (rather than preserving them as conventional dereverberation does) means the output sounds closer to studio-recorded speech than to a "cleaned but still reverberant" recording. The EARS fine-tuning variant specifically addresses electrical microphone hiss — a common artifact in consumer-grade recording equipment — which the base model trained on VQScore 0.65 filtered data still preserves. For podcasters and YouTubers with mid-range equipment, this provides a "one-click" enhancement that produces noticeably cleaner audio, with the speaker identity preserved (SpkSim of 0.83–0.84 for the shifted anechoic + GAN model in Table 1) so voices remain recognizable and natural.

When to Prefer This Method

The paper's analysis of the fidelity–quality trade-off and the choice between regression-only, generative-only, and the two-stage framework maps directly to deployment decisions.

Prefer the full two-stage framework (shifted anechoic target + GAN correction) when:

• Perceptual quality is the primary concern and the application tolerates a small increase in computational cost (two forward passes instead of one), as indicated by the NISQA improvement from 3.85 to 4.12 with GAN correction (Table 1).
• The input degradation includes conditions where regression models are known to over-smooth: severe bandwidth limitation, packet loss, or very low SNR noise — conditions where the degraded input $y$ contains little information about the clean signal and the regression model's posterior mean collapses toward the prior mean.
• The deployment is for content that will be listened to by humans (podcast cleanup, TTS training data enhancement) rather than processed by downstream ASR, since the GAN correction's quality gains (DNSMOS, NISQA, UTMOS) are larger than its fidelity gains (CAcc improvement is modest: 89.41 to 89.88).
• Training data has been curated with VQScore filtering at threshold 0.65 or higher, since the GAN correction amplifies the benefits of clean training targets.

Prefer the regression-only model (shifted anechoic target, no GAN correction) when:

• Computational cost or latency is at a premium, since the regression model requires only a single forward pass and achieves competitive ASR accuracy (CAcc of 89.41, only 0.47 points below the GAN-corrected variant).
• The application is primarily for downstream machine processing (ASR, speaker verification, emotion recognition) where content preservation matters more than naturalness — the regression model preserves fidelity (no hallucination risk) while still improving perceptual quality over the conventional early-reflected target (NISQA of 3.85 vs. 3.23).
• The input degradation is predominantly additive noise or mild reverberation (not severe bandwidth limitation or packet loss), where the posterior mean is already a good estimate and over-smoothing is minimal.

Prefer training with time-shifted anechoic targets over early-reflected targets when:

• The training pipeline has access to RIRs with known direct-path time shifts (standard for simulated training data), enabling trivial computation of $n_0 = \arg\max_n |r[n]|$.
• The deployment involves dereverberation as one of multiple degradation types — the shifted anechoic target improves non-intrusive quality (UTMOS 2.76 vs. 2.26) and ASR accuracy (CAcc 89.41 vs. 87.62) without any architectural changes.
• The evaluation uses metrics that are not artificially penalized by the early-reflected reference mismatch; if leaderboard intrusive metrics (PESQ, ESTOI, SDR) are the primary evaluation criterion and the benchmark maintainers will not update the reference, the early-reflected target may produce higher leaderboard scores despite producing lower actual quality.

Prefer VQScore-based training data filtering when:

• The training corpus is large (2,000+ hours) and contains a mix of high-quality and low-quality sources — the 0.65 threshold removes only the worst ~12 hours from 2,500 hours while producing measurable UTMOS improvements.
• The downstream application requires removal of subtle artifacts (microphone hiss, stationary noise) that the organizer-filtered data still contains — the EARS fine-tuning variant specifically addresses this, though at a small cost to speaker similarity.
• The goal is deploying a single universal model rather than condition-specific models — filtered data helps the model learn a sharper boundary between "degradation to remove" and "clean speech to preserve."