Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems

ArXiv: 2409.06656

🎯 Pitch

The core innovation of Sortformer is that it completely eliminates the need for Permutation Invariant Loss in multi-speaker ASR—instead, by forcing speaker labels to follow the chronological order of who speaks first, you can train speaker supervision using standard cross-entropy just like single-speaker systems. This simple re-framing slashes word error rate by 30% relative on 2-speaker mixtures and improves diarization accuracy on DIHARD3 by over 2 absolute points. It effectively makes multi-speaker ASR training as straightforward as mono-speaker ASR, unifying two historically siloed architectures.

1. Executive Summary

This paper proposes Sortformer, an encoder-based speaker diarization model that resolves the speaker permutation problem in multi-speaker speech-to-text systems by introducing Sort Loss—a training objective that forces the model to output speaker labels ordered by arrival time (the first speaker to speak gets <spk0>, the second gets <spk1>, and so on)—used either independently or as a Hybrid Loss combining Sort Loss with traditional Permutation Invariant Loss (PIL). When integrated into a multi-speaker ASR architecture via sinusoidal kernel functions that embed speaker supervision into the encoder states, Sortformer enables standard cross-entropy training on sorted speaker tokens without requiring the specialized PIL loss function at the ASR decoder, effectively making multi-speaker ASR training functionally equivalent to mono-speaker ASR training. Evaluated on the DIHARD3, CALLHOME, and CH109 speaker diarization benchmarks, Hybrid Loss achieves a 14.76% diarization error rate on DIHARD3 (outperforming PIL-only training by 2.28 absolute points), while the combined Sortformer-MS-Canary system delivers a 30% relative WER reduction on 2-mix LibriSpeechMix and a 25% reduction on 3-mix compared to an unsupervized multi-speaker baseline, establishing that arrival-time-sorted speaker supervision can match or exceed PIL-based approaches in stand-alone diarization while uniquely enabling seamless integration with standard token-level ASR objectives.

2. Context and Motivation

The Core Problem: The Permutation Barrier Between Speaker Diarization and ASR

The fundamental problem this paper tackles is the speaker permutation problem and how it prevents seamless integration of speaker diarization with automatic speech recognition (ASR) systems. In multi-speaker transcription, the model must simultaneously answer two questions: what was said and who said it. The permutation problem arises because speaker identities are generic labels—there is no inherent reason why a particular speaker should be assigned label "1" versus label "2." During training, when the model predicts speaker A as speaker B (and vice versa), the prediction is functionally correct but incurs an arbitrary penalty under standard loss functions because the index assignments don't match.

This isn't merely a theoretical nuisance. It creates a hard architectural barrier between the speaker diarization community and the ASR community. Speaker diarization models have converged on using Permutation Invariant Loss (PIL), also called Permutation Invariant Training (PIT), which computes the minimum loss across all possible speaker-to-label permutations before backpropagating. PIL works well for stand-alone diarization, but it cannot be easily integrated into a unified multi-speaker ASR training pipeline because it requires a specialized loss calculation mechanism at the output layer. If an ASR model is being trained for multiple tasks simultaneously—speech recognition, translation, summarization, plus speaker attribution—PIL demands a separate, task-specific loss function that complicates the entire training setup. The paper states this plainly in Section 3.1:

"PIL requires a specialized loss function at the model's output layer, it limits its applicability when training multi-speaker ASR models for multiple tasks simultaneously using the same ground truth."

This constraint means that despite years of progress in both end-to-end diarization (Fujita et al., 2019; Horiguchi et al., 2022a) and multi-speaker ASR (Kanda et al., 2020b; Shi et al., 2024), the two systems have remained largely separate components in cascaded pipelines, each trained and optimized independently. They share no differentiable computational graph, making joint optimization impossible without custom loss machinery.

Why This Matters: The Growing Demand for Speaker-Aware Transcription

The practical stakes are high and growing. Speaker annotations are increasingly essential for real-world transcription services because natural language understanding (NLU) modules need to recognize who is speaking to properly interpret conversations. Consider a medical consultation: the doctor's questions, the patient's symptoms, and the follow-up recommendations all carry different clinical weight depending on who uttered them. A transcript without speaker labels loses this critical context. Similarly, in meeting transcription, legal depositions, or customer service analytics, knowing who said what is essential for downstream tasks like summarization, action-item extraction, and sentiment analysis.

Beyond end-user applications, there is a surging demand for automatic annotation systems that can generate speaker-labeled training data at scale. Modern machine learning models require enormous datasets, and manually annotating multi-speaker conversational speech with speaker labels is prohibitively expensive—far more expensive than collecting single-speaker speech data. This is especially acute for low-resource languages and privacy-sensitive domains (medical, legal, financial) where large-scale crowd-sourced annotation is infeasible. A system that can reliably produce speaker-attributed transcripts without requiring specialized per-task fine-tuning or complex loss functions would dramatically lower the barrier to building speaker-aware applications.

The paper also highlights that multi-speaker ASR models often need to process multi-hour audio samples in deployment (e.g., all-day meeting recordings), yet acquiring such long-form training data with accurate speaker annotations is even more challenging than short-segment data. Any approach that simplifies training—making it more like standard mono-speaker ASR training—directly addresses this data scarcity problem.

Where Prior Approaches Fall Short

End-to-End Multi-Speaker ASR with Serialized Output Training (SOT)

Kanda et al. (2020b) introduced Serialized Output Training, which concatenates utterances from different speakers into a single output sequence separated by speaker change tokens (<cs>). This was a significant advance because it enabled end-to-end training without explicit diarization modules. However, SOT has fundamental limitations:

• It requires speaker-annotated multi-speaker data for training, which is scarce and expensive to collect. The paper notes that "the performance of end-to-end multi-speaker ASR systems tends to lag behind that of cascaded systems" (Kanda et al., 2022b) precisely because of this data bottleneck.
• SOT's serialization is based on utterance boundaries, not word-level speaker attribution. The model must learn to segment speech into speaker turns, which is a harder problem than simply tagging each word with the correct speaker.
• SOT cannot leverage pre-trained mono-speaker ASR models without substantial architectural modifications. The output format (speaker-change tokens, concatenated multi-speaker sequences) differs fundamentally from standard ASR output, making transfer learning difficult.

Variants like token-level SOT (t-SOT) for streaming (Kanda et al., 2022a) and dominance-ranking SOT (Shi et al., 2024) partially address some issues but retain the core limitation: they still require specialized training objectives and struggle with domain adaptation.

Cascaded Systems: Modular but Difficult to Optimize

The dominant approach in challenge-winning systems (CHiME-6, CHiME-7) is cascaded: run speaker diarization first, then ASR, potentially with source separation modules in between (Cornell et al., 2023; Medennikov et al., 2020b; Niu et al., 2024). These systems are effective—they win benchmarks—but they are difficult to optimize and deploy:

• Each component (diarization, separation, ASR) must be individually tuned for the target domain.
• Components are trained independently with different objectives, meaning errors propagate without any mechanism for cross-component compensation.
• Adding a new capability (e.g., speech translation alongside transcription) requires retraining or reconfiguring the entire pipeline.
• The paper characterizes this pain point explicitly: "these systems are difficult to optimize because each component often needs to be tailored for domain-specific datasets."

End-to-End Diarization: PIL-Based and ASR-Incompatible

End-to-end neural diarization systems (Fujita et al., 2019; Horiguchi et al., 2022a; Chen et al., 2024) frame speaker labeling as a frame-wise classification task trained with PIL. These models are elegantly simple for diarization alone but carry a fundamental incompatibility with ASR integration:

• PIL requires computing loss over all K! speaker permutations at each training step, which is manageable for small K (typically 2–4 speakers) but doesn't compose with the cross-entropy loss used for text token prediction.
• EEND systems do not use positional embeddings because speaker label ordering is irrelevant under PIL. However, ASR systems rely on positional embeddings to model temporal sequence structure. This architectural divergence makes simple concatenation of the two models impossible.
• The paper states this gap explicitly (Section 2.3): "there have been limited efforts to create a synergistic effect by integrating both models within a differentiable computational graph. To the best of our knowledge, our proposed system is the first to integrate an end-to-end diarization system with an end-to-end multi-speaker ASR model at the computational graph level."

How This Paper Positions Itself

Sortformer is positioned as a bridge between the diarization and ASR worlds. The key insight is that the permutation problem can be resolved before it reaches the ASR loss function, rather than being handled at the loss level. By training the diarization model to output speaker labels sorted by arrival time (the first speaker to speak is always <spk0>, the second <spk1>, and so on), the permutation ambiguity is eliminated at the source. The ASR decoder then simply needs to predict these sorted speaker tokens alongside word tokens using standard cross-entropy loss—no PIL, no specialized loss machinery, no architectural gymnastics.

This positioning has several strategic implications in the paper's framing:

1. Making multi-speaker ASR training "functionally equivalent" to mono-speaker ASR training. This is the paper's central selling point (Section 3.1): "once the speaker tokens in the ground truth labels are sorted, the model can be trained using the standard cross-entropy function on text tokens." For practitioners who already have mono-speaker ASR pipelines, adding speaker attribution becomes a matter of inserting speaker tokens into the training data and adding the Sortformer module—not redesigning the loss function or training procedure.

2. Enabling training on multi-task objectives without per-task loss specialization. If an ASR model is simultaneously trained for transcription, translation, and speaker attribution, PIL would require custom logic to handle the speaker permutation while leaving other tasks untouched. Sortformer's arrival-time-sorted outputs avoid this entirely: the speaker tokens are just more tokens in the sequence, trained with the same cross-entropy loss as everything else. This makes multi-task training with speaker labels straightforward.

3. Reducing annotation requirements through timestamp-free training. Traditional multi-speaker ASR often requires word-level or segment-level timestamps to align speaker labels with words. The paper introduces a syllable-based word timestamp approximation (Appendix C) that generates pseudo-timestamps from segment boundaries and syllable counts, eliminating the need for expensive manual alignment. Combined with the sorted token approach, this means training data only needs speaker-attributed transcripts at the segment level—word-level alignment is handled automatically.

4. Providing a drop-in speaker supervision module. Sortformer is designed as an encoder-only model (123M parameters, NEST backbone) that produces frame-level speaker presence probabilities. These are injected into the ASR encoder via sinusoidal kernel functions (Equation 14), creating a differentiable connection between the diarization and ASR components. The Sortformer weights can be frozen (preserving pre-trained diarization knowledge) or fine-tuned jointly with the ASR model, giving practitioners flexibility in deployment.

5. Acknowledging and partially addressing the arrival-time limitation. The paper is candid that training solely with Sort Loss has a limitation: "arrival time estimation is not always correct. This issue becomes more pronounced as the number of speakers increases" (Section 3.3). The Hybrid Loss (α · L_Sort + (1 − α) · L_PIL) partially mitigates this by allowing the model to fall back on PIL for cases where arrival-time ordering fails. This hybrid approach is shown to outperform either loss alone on diarization benchmarks (Table 1), suggesting the two objectives provide complementary training signals.

The paper thus positions Sortformer not as a replacement for PIL-based diarization systems, but as a complementary approach optimized for ASR integration—one that happens to also improve stand-alone diarization when combined with PIL via hybrid loss. The vision, articulated in the conclusion, is that Sortformer "will serve as an accessible baseline to inspire further research in multi-speaker ASR," lowering the barrier to entry for researchers and practitioners who want speaker attribution in their models without becoming experts in permutation-handling loss functions.

3. Technical Approach

3.1 Reader Orientation

This paper presents Sortformer, an encoder-based neural network that takes raw audio features as input and outputs, for each time frame, which of up to K speakers is currently talking—with the critical property that the output speaker indices are sorted by arrival time (the first person who speaks anywhere in the recording gets index 0, the second gets index 1, and so on). The problem it solves is the speaker permutation ambiguity that has historically prevented speaker diarization models from being seamlessly integrated into automatic speech recognition (ASR) training pipelines: because standard loss functions penalize mismatched speaker indices even when the speaker assignment is functionally correct, previous diarization systems required specialized Permutation Invariant Loss (PIL) that cannot be cleanly combined with the cross-entropy loss used for text token prediction. Sortformer resolves this by sorting the ambiguity away before it reaches the ASR loss function—the model is trained to produce speaker labels in a canonical arrival-time order, so the ASR decoder can simply predict those same sorted speaker tokens alongside word tokens using standard cross-entropy, making multi-speaker ASR training functionally equivalent to mono-speaker ASR training.

3.2 Big-Picture Architecture (Diagram in Words)

The complete Sortformer-integrated multi-speaker ASR system has five major components connected in a differentiable pipeline:

1. Audio Feature Extractor (NEST encoder, shared with ASR): Processes raw audio into frame-level embedding vectors $\{x_t\}_{t=1}^T$, where each $x_t \in \mathbb{R}^D$ represents the acoustic content at time frame $t$. This is a standard Fast-Conformer backbone (the same encoder architecture used in the Canary ASR model).

2. Sortformer Diarization Module (123M parameters, Transformer encoder + feedforward heads): Takes the frame-level embeddings as input and produces a $K \times T$ matrix $P$ of speaker presence posterior probabilities. Each column $p_t \in [0,1]^K$ represents the probability that each of the $K$ speakers is active at frame $t$. Critically, the rows of $P$ are ordered by speaker arrival time—row 0 corresponds to the first speaker who speaks, row 1 to the second, and so on. This is enforced by training with Sort Loss (Section 3.3).

3. Sinusoidal Speaker Kernel Generator: Transforms the $K \times T$ speaker presence matrix $P$ into a set of $K$ speaker-specific embedding vectors $\Gamma = [\gamma_1, \ldots, \gamma_K]^\top \in \mathbb{R}^{K \times M}$ using sinusoidal functions. Each $\gamma_k$ is a fixed pattern of sine values at different frequencies that uniquely identifies speaker $k$. These kernels are added to the normalized ASR encoder states (Equation 14), effectively tagging each frame's ASR representation with information about which speakers are active at that moment.

4. ASR Encoder (Fast-Conformer, potentially extended with adapters): Receives the speaker-augmented embeddings $\tilde{A}$ (original encoder states plus speaker kernel contributions) and produces context-rich representations for the decoder. For resource-efficient training, adapter modules (small bottleneck layers inserted into each encoder and decoder layer, following Houlsby et al., 2019) can be trained while keeping the base encoder frozen, preserving mono-speaker ASR knowledge.

5. ASR Decoder (Transformer decoder with cross-attention to encoder outputs): Generates text tokens autoregressively, including sorted speaker tokens (<spk0>, <spk1>, etc.) interleaved with word tokens. The decoder is trained with standard cross-entropy loss on these sorted token sequences—no PIL, no specialized loss functions. The speaker tokens in the ground truth are pre-sorted by arrival time during data preparation, so the decoder simply learns to predict them in that order.

Information flow: Raw audio → NEST features → Sortformer predicts who speaks when (sorted by arrival time) → sinusoidal kernels convert speaker activities into additive embeddings → ASR encoder processes speaker-tagged features → ASR decoder generates <spk0> word1 word2 <spk1> word3 ... using standard cross-entropy. During training, Sortformer's weights can be frozen (using pre-trained diarization knowledge) or fine-tuned jointly with the ASR components. In either case, the only training objective is token-level cross-entropy—the permutation problem has been resolved upstream by the arrival-time sorting.

3.3 Roadmap for the Deep Dive

The explanation proceeds in five stages, ordered to build understanding from the core innovation outward:

1. First, the mathematical formulation of Sortformer as a multi-label binary classifier (Section 3.2 of the paper)—this establishes what the model computes at a mechanistic level (frame-wise speaker presence probabilities) and why it uses sigmoid rather than softmax outputs (conditional independence assumption across speakers).

2. Second, the three loss functions: Binary Cross-Entropy, Permutation Invariant Loss, and Sort Loss (Section 3.3)—this is the intellectual core of the paper. Understanding PIL first makes clear why Sort Loss is necessary (PIL cannot compose with standard ASR objectives), and understanding the Hybrid Loss reveals how the two can be combined for improved stand-alone diarization.

3. Third, how the Transformer encoder learns to sort via positional embeddings (Section 3.4)—this explains the architectural difference from prior EEND systems (which omit positional embeddings because ordering is irrelevant under PIL) and why positional information is necessary for arrival-time-based sorting.

4. Fourth, the bridge between timestamps and tokens (Section 4)—how speaker supervision from Sortformer is injected into the ASR encoder via sinusoidal kernels (Equation 14), how sorted speaker tokens are placed in transcripts at word or segment level (SST, Figure 5), and how word timestamps are approximated when ground-truth alignments are unavailable (Appendix C).

5. Fifth, the training configurations and adapter-based resource-efficient fine-tuning—the specific architectural choices (adapter dimensions, frozen vs. fine-tuned Sortformer, segment-level vs. word-level speaker token placement) and their justifications for preserving mono-speaker ASR performance while adding speaker attribution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems integration paper whose core idea is that training a speaker diarization model to produce arrival-time-sorted outputs eliminates the permutation ambiguity that previously prevented diarization and ASR from sharing a differentiable computational graph. The technical contribution spans three levels: (1) a novel training objective (Sort Loss) for the diarization model that enforces sorted outputs, (2) a mechanism (sinusoidal speaker kernels) for injecting those sorted speaker predictions into an ASR encoder in a differentiable way, and (3) a data preparation scheme (sorted serialized transcripts with pseudo-timestamps) that allows the ASR decoder to be trained with standard cross-entropy on speaker-attributed text.

Sortformer as a Multi-Label Binary Classifier

The Sortformer model itself is an encoder-only architecture that performs frame-level multi-label classification. At its core, it is a mathematical function $f_\Theta$ parameterized by weights $\Theta$ that maps a sequence of audio feature vectors to a matrix of speaker presence probabilities.

Input: A sequence of $T$ frame-level embedding vectors $X = [x_1, x_2, \ldots, x_T] \in \mathbb{R}^{D \times T}$, where each column $x_t \in \mathbb{R}^D$ is a $D$-dimensional representation of the audio at time frame $t$. These embeddings come from a NEST (Neural Encoder for Speech Tasks) Fast-Conformer backbone—the same encoder used in the Canary ASR system, with 115M parameters in the L-size configuration. The NEST encoder uses 18 Transformer encoder layers with a hidden size of 192.

Output: A matrix $P = f_\Theta(X) \in [0,1]^{K \times T}$, where each column $p_t = [p_{1,t}, \ldots, p_{K,t}]^\top$ represents the posterior probabilities that each of $K$ speakers is active at frame $t$. The value $p_{k,t} \in [0,1]$ is the model's estimated probability that speaker $k$ is talking during frame $t$. The full matrix can be decomposed row-wise as $P = [q_1, q_2, \ldots, q_K]^\top$, where $q_k = [p_{k,1}, p_{k,2}, \ldots, p_{k,T}]^\top$ is the time series of activity probabilities for speaker $k$.

Binarization: During evaluation, each $p_{k,t}$ is thresholded to produce a binary decision $\hat{y}_{k,t} \in \{0,1\}$. The full binary output matrix is $\hat{Y} = [\hat{\xi}_1, \ldots, \hat{\xi}_T] \in \{0,1\}^{K \times T}$, where $\hat{\xi}_t = [\hat{y}_{1,t}, \ldots, \hat{y}_{K,t}]^\top$ is the binarized speaker activity vector at frame $t$.

The conditional independence assumption (Equation 1):

The paper models the joint probability of the entire sequence of speaker presence vectors given the input features as:

$P(\xi_1, \ldots, \xi_T \mid x_1, \ldots, x_T) = \prod_{k=1}^{K} \prod_{t=1}^{T} P(y_{k,t} \mid x_1, \ldots, x_T)$

where $\xi_t \in \{0,1\}^K$ is the ground-truth speaker presence vector at time $t$, $y_{k,t} \in \{0,1\}$ is the ground-truth activity of speaker $k$ at frame $t$, and $P(y_{k,t} \mid x_1, \ldots, x_T)$ is the model's predicted probability for that speaker-frame pair.

What it computes: The joint probability that the entire $K \times T$ grid of speaker activities takes on specific values, given the full input audio. Under this factorization, the probability of each speaker being active at each frame is conditionally independent of all other speakers and frames, given the full input sequence. This is a naive Bayes-style factorization applied across the output grid.

Why this form: The conditional independence assumption justifies using sigmoid activation functions at the output layer rather than softmax. If the model used softmax over the $K$ speaker classes at each frame, it would enforce a mutual exclusivity constraint (exactly one speaker active per frame), which fails in overlapping speech regions where multiple speakers talk simultaneously. The sigmoid formulation allows the model to predict $p_{1,t} \approx 0.8$ and $p_{2,t} \approx 0.7$ simultaneously for the same frame—exactly what is needed for overlapping speech. This design choice is formalized in the paper's statement that "Sortformer employs Sigmoid instead of Softmax unlike the activation function for the output layer in the Transformer encoder."

Architecture details: On top of the 18-layer NEST Transformer encoder, Sortformer adds two feed-forward layers with four sigmoid outputs (for the $K=4$ speaker case). The total parameter count including the NEST encoder is 123M. Positional embeddings are included in the Transformer encoder (unlike EEND systems), which is essential for the model to learn arrival-time ordering (see Section 3.4 discussion below).

Binary Cross-Entropy Loss (The Base Loss Function)

All loss variants in Sortformer are built on the binary cross-entropy (BCE) loss applied to individual speaker-frame predictions. For a single prediction $p \in [0,1]$ with ground-truth label $y \in \{0,1\}$:

$\mathcal{L}_{\text{BCE}}(y, p) = -\left(y \log(p) + (1 - y) \log(1 - p)\right)$

where $y$ is the binary ground-truth label (1 if the speaker is active at that frame, 0 otherwise) and $p$ is the model's predicted probability for that speaker-frame pair.

What it computes: The standard binary cross-entropy between a predicted probability and a binary target. When $y=1$, the loss reduces to $-\log(p)$, which penalizes the model heavily when it assigns low probability to an active speaker ($p \to 0 \implies -\log(p) \to \infty$). When $y=0$, the loss reduces to $-\log(1-p)$, penalizing the model when it assigns high probability to a silent speaker ($p \to 1 \implies -\log(1-p) \to \infty$).

Why this form: Binary cross-entropy is the maximum-likelihood objective for a Bernoulli random variable, which is the correct statistical model for a binary presence/absence decision at each frame. The logarithmic penalty structure ensures that confident errors (predicting $p \approx 0$ when $y=1$, or $p \approx 1$ when $y=0$) incur much larger gradients than uncertain errors ($p \approx 0.5$), which drives the model toward calibrated probability estimates.

Frame-level aggregation: For a full $K \times T$ output matrix, the total BCE loss sums over all speakers and all frames. For a single speaker $k$ with ground-truth activity vector $y_k \in \{0,1\}^T$ and predicted probability vector $q_k \in [0,1]^T$:

$\mathcal{L}_{\text{BCE}}(y_k, q_k) = \frac{1}{T} \sum_{t=1}^{T} \mathcal{L}_{\text{BCE}}(y_{k,t}, p_{k,t})$

This per-speaker loss is the building block for the permutation-handling losses described next.

Permutation Invariant Loss (PIL) — The Baseline Approach

PIL is the standard training objective for end-to-end speaker diarization systems. The core problem it addresses is that the model's $K$ output rows (speaker channels) have no inherent correspondence to the $K$ ground-truth speakers. If the model assigns speaker A to output channel 1 and speaker B to output channel 2, but the ground truth has them in the opposite order, a naive BCE loss would penalize both channels—even though the predictions are functionally correct.

Mathematical definition (Equation 4):

$\mathcal{L}_{\text{PIL}}(Y, P) = \min_{\pi \in \Pi} \mathcal{L}_{\text{BCE}}(Y_\pi, P)$

where $Y = [y_1, \ldots, y_K]^\top \in \{0,1\}^{K \times T}$ is the ground-truth speaker presence matrix, $P = [q_1, \ldots, q_K]^\top \in [0,1]^{K \times T}$ is the predicted probability matrix, $\Pi$ is the set of all $K!$ possible permutations of the indices $\{1, \ldots, K\}$, and $Y_\pi$ is the ground-truth matrix with rows permuted according to permutation $\pi$ (i.e., $Y_\pi = [y_{\pi(1)}, \ldots, y_{\pi(K)}]^\top$).

What it computes (Equation 5–6, expanded): The PIL procedure works as follows:

1. For each of the $K!$ possible ways to match the model's $K$ output rows to the $K$ ground-truth speaker rows, compute the average BCE loss across all speakers and frames.
2. Select the permutation $\pi^*$ that yields the minimum total loss.
3. Backpropagate only through this optimal matching.

Operationally, for $K=4$ speakers, this means evaluating 24 possible speaker-to-output-channel assignments at each training step and selecting the one where the model's predictions best align with the ground truth.

Why this form: PIL elegantly solves the speaker permutation problem for stand-alone diarization training by ensuring the model is never penalized for assigning the "wrong" output index to a particular speaker—as long as there exists some index assignment that produces a good match, the loss will find it. This works because PIL explicitly searches over all matchings at each training step.

Why PIL fails for ASR integration: The critical limitation is that PIL requires computing the loss over all permutations at the output layer of the diarization model. When the diarization model and ASR model are combined in a single computational graph, the ASR decoder's cross-entropy loss on speaker tokens would also need to handle permutation ambiguity—but the cross-entropy loss operates on individual token predictions, not frame-level activity matrices, and cannot simply be wrapped in a minimum-over-permutations without breaking the autoregressive token generation structure. The paper states this constraint succinctly: PIL "limits its applicability when training multi-speaker ASR models for multiple tasks simultaneously using the same ground truth." The sorting approach eliminates this problem by resolving the permutation before it reaches any loss function.

Sort Loss — Arrival-Time-Based Permutation Resolution

Sort Loss is the paper's core technical innovation. Instead of treating speaker identity as an arbitrary label to be matched post-hoc via permutation search (as PIL does), Sort Loss imposes a canonical ordering on speaker labels during both data preparation and model training: the first speaker to begin speaking anywhere in the recording is assigned label 0, the second speaker to begin speaking is assigned label 1, and so on.

Arrival time function (Equation 7):

$\Psi(y_k) = \min\{t' \mid y_{k,t'} \neq 0, \ t' \in [1, T]\} = t^0_k$

where $y_k \in \{0,1\}^T$ is the ground-truth binary activity vector for speaker $k$, and $t^0_k$ is the frame index of that speaker's first active frame (their "arrival time"). The function $\Psi$ scans along the time axis of a speaker's activity vector and returns the index of the first frame where that speaker is marked as active.

Sorting function (Equation 9):

Using the arrival times, a sorting function $\eta$ is applied to the speaker indices $\{1, \ldots, K\}$ such that the ground-truth speaker rows are reordered by ascending arrival time:

$\eta(Y) = Y_\eta = [y_{\eta(1)}, \ldots, y_{\eta(K)}]^\top$

with the constraint (Equation 10) that:

$\Psi(y_{\eta(1)}) \leq \Psi(y_{\eta(2)}) \leq \cdots \leq \Psi(y_{\eta(K)})$

What it computes: Given the $K \times T$ ground-truth activity matrix $Y$, this procedure (1) computes each speaker's arrival time as the first frame they are active, (2) sorts the speakers by ascending arrival time, and (3) rearranges the rows of $Y$ so that the earliest-arriving speaker occupies row 0, the second-earliest occupies row 1, and so forth. Ties (two speakers starting at exactly the same frame) are handled implicitly by the sorting algorithm; the paper does not specify a tie-breaking rule, suggesting such cases are rare in practice.

Sort Loss definition (Equation 11):

$\mathcal{L}_{\text{Sort}}(Y, P) = \mathcal{L}_{\text{BCE}}(Y_\eta, P) = \frac{1}{K} \sum_{k=1}^{K} \mathcal{L}_{\text{BCE}}(y_{\eta(k)}, q_k)$

where $Y_\eta$ is the arrival-time-sorted ground-truth matrix, $P$ is the model's predicted probability matrix, $y_{\eta(k)}$ is the sorted ground-truth activity vector for the $k$-th arriving speaker, and $q_k$ is the model's predicted probability vector for its $k$-th output channel.

What it computes: The standard BCE loss, but with the ground-truth speaker rows reordered into arrival-time order before comparing with the model's predictions. The model's $k$-th output channel is always compared against the $k$-th earliest-arriving speaker. There is no permutation search—the correspondence between model output channels and ground-truth speakers is fixed by the sorting during data preparation.

Why this form: Sort Loss converts the permutation problem from a training-time optimization (finding the best permutation at each step, as PIL does) into a data-preparation convention (sorting ground-truth labels by arrival time). This means the model's output channels have a fixed, semantically meaningful interpretation: channel 0 always corresponds to the first speaker who spoke, channel 1 to the second, and so on. The model is trained to produce predictions that respect this ordering, and the ASR decoder can then be trained to generate speaker tokens in the same sorted order using standard cross-entropy—no specialized loss function is needed anywhere in the integrated system.

The ordering constraint at inference (Equation 8):

During inference, the model is expected to satisfy:

$\Psi(\hat{y}_1) \leq \Psi(\hat{y}_2) \leq \cdots \leq \Psi(\hat{y}_K)$

where $\hat{y}_k$ is the model's binarized prediction for its $k$-th output channel. This constraint captures the model's learned behavior: its first output channel should activate for the earliest-arriving speaker, its second channel for the next-earliest, and so on. If the model successfully learns this ordering, the ASR decoder can reliably map <spk0> to the first speaker's words and <spk1> to the second speaker's words.

The limitation the paper acknowledges: The arrival-time ordering heuristic can fail. If two speakers begin speaking at nearly the same time, or if a very quiet speaker speaks first but the model fails to detect them, the sorted ordering may not cleanly separate speakers. The paper notes that "arrival time estimation is not always correct. This issue becomes more pronounced as the number of speakers increases." This motivates the Hybrid Loss.

Hybrid Loss — Combining Sort Loss and PIL

The Hybrid Loss is a convex combination of Sort Loss and PIL, controlled by a mixing weight $\alpha$:

$\mathcal{L}_{\text{hybrid}} = \alpha \cdot \mathcal{L}_{\text{Sort}} + (1 - \alpha) \cdot \mathcal{L}_{\text{PIL}}$

where $\alpha \in [0,1]$ is an empirically determined weighting factor. In the paper's experiments, $\alpha = 0.5$ is used (Section 5.2).

What it computes: At each training step, the model computes two losses independently: (1) the Sort Loss comparing the model's predictions against arrival-time-sorted ground-truth labels (enforcing the canonical ordering), and (2) the PIL comparing against all possible permutations and selecting the best match (providing a fallback when arrival-time ordering fails). The total loss is the weighted sum, with both gradients flowing back through the model.

Why this form: The Hybrid Loss allows the model to receive training signal from both objectives simultaneously. Sort Loss provides a strong inductive bias toward learning arrival-time ordering, which is necessary for downstream ASR integration. PIL provides a safety net: when the arrival-time ordering is ambiguous or incorrect, PIL ensures the model can still learn to separate speakers correctly—just without necessarily respecting the canonical ordering. The combined objective is hypothesized to leverage "strengths from both" objectives, which is empirically confirmed in Table 1: Hybrid Loss (14.76% DER on DIHARD3) outperforms both pure Sort Loss (17.10%) and pure PIL (17.04%).

Training time overhead: Relative to pure PIL training (average epoch time of 1,020 seconds), pure Sort Loss introduces only a 0.22% increase (to 1,022.28 seconds), while Hybrid Loss introduces a 2.26% increase (to 1,043.1 seconds). This minimal overhead occurs because Sort Loss does not require the expensive $K!$ permutation search that PIL requires—it simply applies the pre-computed sorting to the ground-truth labels before computing BCE.

How the Transformer Learns to Sort (Positional Embeddings)

A crucial architectural detail distinguishes Sortformer from prior EEND systems (Fujita et al., 2019; Horiguchi et al., 2022a): Sortformer uses positional embeddings in its Transformer encoder, while EEND systems do not.

Why EEND omits positional embeddings: Under PIL, the ordering of speaker labels is explicitly irrelevant—the loss function considers all permutations. The multi-head self-attention (MHA) mechanism without positional embeddings is permutation equivariant (as proven in Appendix E–F): if you permute the rows of the input (reordering the speaker channels), the output rows are permuted identically. This property is desirable for PIL-based systems because it means the model has no inherent bias toward any particular speaker-to-output-channel mapping; the mapping is determined entirely by the training data.

Formally (Equation 25 from Appendix F.2), a function $F$ is permutation equivariant if for any permutation matrix $P_\pi$:

$F(P_\pi X) = P_\pi F(X)$

The paper proves that standard MHA (without positional embeddings) satisfies this property, meaning the speaker output ordering is entirely flexible—exactly what PIL needs.

Why Sortformer needs positional embeddings: Sort Loss imposes a fixed, semantically meaningful ordering on the output channels (by arrival time). To learn this ordering, the model must be able to distinguish temporal order in the input—it needs to know which speaker starts earlier. Positional embeddings provide this temporal awareness by encoding frame indices into the input representations. Without positional embeddings, the Transformer would be equivariant to frame permutations (in addition to speaker permutations), making it impossible to learn an arrival-time-based ordering because the model would have no concept of "earlier" versus "later."

The paper states this explicitly (Section 3.4): "baseline systems [EEND-SA, EEND-EDA] do not require positional embeddings, as the ordering of speaker labels is not relevant... However, the multi-head self-attention (MHA) in Transformers inherently exhibits permutation equivariance when positional embeddings are omitted. Therefore, Sortformer employs positional embeddings to provide the model with a sense of sequence ordering."

What this means operationally: The Sortformer Transformer encoder receives input features that include standard sinusoidal or learned positional encodings added to the frame-level embeddings. This breaks the permutation equivariance of the MHA layers, allowing the attention mechanism to attend differently to early frames versus late frames, which is essential for the model to learn that its first output channel should respond to the earliest active speaker.

Bridging Timestamps and Tokens: The Data Format

The integration of Sortformer with the ASR system requires solving a representation alignment problem: the Sortformer produces frame-level speaker activity predictions (a $K \times T$ matrix of probabilities), but the ASR decoder produces word-level speaker tokens (discrete <spk0>, <spk1> labels interspersed with text). The bridge between these two representations has two components: (1) how speaker information is injected into the ASR encoder (the sinusoidal kernel), and (2) how the ground-truth transcript is formatted for cross-entropy training (the Sorted Serialized Transcript).

Sinusoidal Speaker Kernels (Equation 14)

The speaker supervision from Sortformer is integrated into the ASR encoder through additive sinusoidal kernels. The core idea is to create a fixed, differentiable mapping from the Sortformer's $K \times T$ probability matrix to a set of $K$ speaker-specific embedding vectors that are added to the ASR encoder states.

Kernel construction: For each speaker $k \in \{0, \ldots, K-1\}$, a sinusoidal kernel $\gamma_k \in \mathbb{R}^M$ is defined as:

$\kappa_{k,z} = \sin\left(\frac{2\pi k z}{M}\right)$

$\gamma_k = [\kappa_{k,1}, \kappa_{k,2}, \ldots, \kappa_{k,M}]$

$\Gamma = [\gamma_1, \gamma_2, \ldots, \gamma_K]^\top \in \mathbb{R}^{K \times M}$

where $M$ is the dimension of the ASR encoder state (the embedding dimension), $z \in \{1, \ldots, M\}$ is the bin index within the embedding vector, and $k$ is the speaker index. The full kernel matrix $\Gamma$ contains $K$ rows, each being a distinct sinusoidal pattern at a frequency proportional to the speaker index $k$.

What it computes: For each speaker $k$, the function generates a fixed-length vector $\gamma_k$ whose entries follow a sine wave with frequency determined by $k$. Speaker 0 gets a low-frequency sine pattern, speaker 1 gets a higher-frequency pattern, and so on. These patterns are fixed and non-learned—they are computed deterministically from the speaker indices.

Kernel-based speaker encoding (Equation 14):

$\tilde{A} = \frac{A}{\|A\|_2} + \Gamma^\top \cdot P$

where $A \in \mathbb{R}^{M \times T}$ is the original ASR encoder state matrix (each column is an $M$-dimensional embedding for one time frame), $\|A\|_2$ denotes the L2 norm of each column (normalizing each frame's embedding to unit length), $\Gamma^\top \in \mathbb{R}^{M \times K}$ is the transposed kernel matrix, $P \in [0,1]^{K \times T}$ is the Sortformer's predicted speaker presence probability matrix, and $\tilde{A} \in \mathbb{R}^{M \times T}$ is the speaker-augmented encoder state.

What it computes, step by step:

1. Normalize: Each column of the ASR encoder state $A$ is L2-normalized to unit length ($A / \|A\|_2$). This prevents the speaker kernel contribution from being dominated by large-magnitude encoder states and ensures the additive modification operates in a controlled regime.

2. Weighted sum of speaker kernels: The matrix multiplication $\Gamma^\top \cdot P$ produces an $M \times T$ matrix. For each time frame $t$, the $M$-dimensional vector added to the encoder state is a weighted sum of the $K$ speaker kernel vectors $\gamma_k$, where the weights are the Sortformer's predicted probabilities $p_{k,t}$ for each speaker being active at that frame: $\text{contribution}_t = \sum_{k=1}^{K} p_{k,t} \cdot \gamma_k$

   If the Sortformer is highly confident that speaker 2 is active at frame $t$ ($p_{2,t} \approx 1$) and all other speakers are silent ($p_{k,t} \approx 0$ for $k \neq 2$), the additive contribution at frame $t$ will be approximately $\gamma_2$. If two speakers overlap with equal confidence, the contribution will be roughly $0.5 \cdot \gamma_1 + 0.5 \cdot \gamma_2$—a blend of the two speaker patterns.

3. Add to normalized encoder states: The resulting $M \times T$ speaker contribution matrix is added element-wise to the normalized encoder states, producing $\tilde{A}$.

Why this form: The sinusoidal kernel design achieves several critical properties:

• Differentiability: The entire operation ($\Gamma^\top \cdot P$) is a linear transformation of the Sortformer outputs, so gradients flow seamlessly from the ASR decoder's cross-entropy loss back through the Sortformer. This enables joint fine-tuning of the diarization and ASR components within a single computational graph.

• Injectivity: Each speaker index $k$ maps to a distinct, orthogonal-like sinusoidal pattern. This means the ASR encoder can learn to distinguish different speakers based on the additive pattern in its input—the sinusoidal kernel for speaker 0 is structurally different from the kernel for speaker 1, providing a separable representation.

• Fixed, non-learned weights: By using deterministic sine functions rather than learned embeddings, the kernel mapping avoids adding trainable parameters that would need to be optimized and potentially overfit. The frequency-based encoding ensures that speaker identity information is injected in a structured, interpretable way.

• Additive combination for overlapping speech: The weighted-sum formulation naturally handles overlapping speech: if two speakers talk simultaneously, the encoder state receives contributions from both kernels proportionally to the Sortformer's confidence in each speaker's activity. This provides a soft, probabilistic speaker encoding rather than a hard assignment.

Sorted Serialized Transcript (SST) — The Ground-Truth Format

For the ASR decoder to be trained with standard cross-entropy loss on speaker-attributed text, the ground-truth transcripts must be formatted with sorted speaker tokens interleaved with word tokens. The paper proposes the Sorted Serialized Transcript (SST) format, which places speaker tokens before words at either the word level or the segment level (Figure 5).

Word-level SST: A speaker token (<spk0>, <spk1>, etc.) is placed before every word in the transcript, with the word order determined by comparing the onset (start time) of each word. For example, if speaker A says "great," then speaker B says "oh did you grow up there," the word-level SST would be:

<spk0> great <spk1> oh <spk1> did <spk1> you <spk1> grow <spk1> up <spk1> there

Every word is individually tagged with the speaker who uttered it, even within a single speaker's continuous turn. This provides the finest granularity of speaker attribution but requires word-level timestamps (or approximated timestamps) to determine the correct word ordering when speakers overlap.

Segment-level SST: Speaker tokens are placed only at speaker change points, similar to the Serialized Output Training (SOT) format but with a crucial difference: the speaker tokens are sorted generic indices (<spk0>, <spk1>) rather than a speaker change token (<cs>). For the same example:

<spk0> great <spk1> oh did you grow up there

The key distinction from SOT is that in SST, the speaker tokens are arrival-time-sorted—<spk0> always means "the first speaker who spoke in this recording," while SOT uses a single <cs> token to indicate any speaker change without identifying which speaker. Additionally, SST does not require serializing utterances by speaker turn boundaries; the word ordering is driven by timestamps, not speaker segmentation.

Why these two levels exist: The segment-level objective is simpler and closer to the SOT format that prior work has validated, making it easier to compare against existing methods. The word-level objective provides finer-grained speaker attribution that better matches the Sortformer's frame-level predictions, potentially enabling more precise training. The paper's ablation study (Table 2) compares both levels: System 5 (segment-level, 1.12B parameters) achieves 28.59% cpWER on AMI-test, while System 6 (word-level, 1.12B parameters) achieves 26.71% cpWER—a clear advantage for word-level objectives with larger models.

Permutation resolution via sorting (Section 4.3): The critical property of SST is that "all multi-speaker ASR training sessions use the same cross-entropy loss as conventional single-speaker ASR models, without relying on permutation-invariant or alternative permutation-handling losses." This works because:

1. During data preparation, speaker tokens in ground-truth transcripts are assigned in arrival-time order: the first speaker gets <spk0>, the second gets <spk1>, and so on (see Appendix A for the data cleaning rules).

2. The Sortformer module is trained to generate speaker predictions in the same arrival-time order (via Sort Loss), so its output channels align with the sorted ground-truth speaker indices.

3. The sinusoidal kernels map these sorted speaker channels to distinct, separable patterns in the encoder states.

4. The ASR decoder learns to generate <spk0> tokens when the encoder state contains the sinusoidal pattern corresponding to speaker 0, <spk1> tokens for the pattern corresponding to speaker 1, and so on.

The permutation problem is thus resolved before any loss is computed: the ground-truth tokens are sorted, the Sortformer's outputs are sorted, and the decoder simply needs to predict tokens in sorted order. There is no need for PIL at the ASR level because the correspondence between speaker indices and speaker identities is fixed by the arrival-time convention.

Word Timestamp Approximation (Pseudo-Timestamps, Appendix C)

Word-level SST requires knowing the temporal order of words—specifically, the onset time of each word—to correctly sort them and assign speaker tokens. However, modern end-to-end ASR systems (RNN-T, AED models) are typically trained without word-level forced alignments, and multi-speaker datasets rarely include word-level timestamps. To avoid this annotation bottleneck, the paper introduces a syllable-based word timestamp approximation that generates pseudo-timestamps from segment-level boundaries and syllable counts.

Approximation procedure (Equations 15–16):

Given a speech segment with known start time $t_{\text{start}}$, end time $t_{\text{end}}$, and total number of syllables $N$ across all words in the segment:

$\ell = t_{\text{end}} - t_{\text{start}}$

$\tau^{\text{word}}_i = \left[\delta_i, \ \delta_i + \frac{\ell}{N} \cdot n_i\right] = [\delta_i, \ \delta_i + \lambda \cdot n_i]$

where $\ell$ is the segment duration in seconds, $N$ is the total number of syllables in all words within the segment, $\lambda = \ell / N$ is the average syllable duration (speaking rate, in seconds per syllable), $n_i$ is the number of syllables in the $i$-th word, $\delta_i$ is the cumulative start time of the $i$-th word (i.e., $\delta_i = t_{\text{start}} + \lambda \cdot \sum_{j=1}^{i-1} n_j$), and $\tau^{\text{word}}_i$ is the estimated time interval $[\text{start}_i, \text{end}_i]$ for the $i$-th word.

What it computes: The procedure assumes a constant speaking rate within each segment. It (1) computes the average duration per syllable by dividing the total segment length by the total number of syllables, (2) for each word, multiplies the number of syllables in that word by the average syllable duration to estimate the word's duration, and (3) assigns word start times by accumulating the durations of preceding words, starting from $t_{\text{start}}$. Words are then sorted by their estimated start times, and overlapping words (from different speakers) are interleaved based on these pseudo-timestamps, as illustrated in Figure 6.

Why this form: The approximation is a zero-annotation-cost solution to the timestamp problem. It requires only segment-level start and end times (which are available in most multi-speaker datasets) and syllable counts per word (which can be derived from the word text using simple heuristics—e.g., counting vowel groups). It avoids the need for forced alignment (which would require a separate ASR or alignment model and additional computation) or manual word-level timestamp annotation (which is prohibitively expensive). While the constant-speech-rate assumption is a coarse approximation, the paper reports that "the proposed word-timestamp approximation ensures the approximated word timestamps are comparable to the original word timestamps" (Appendix C), suggesting that word ordering errors introduced by the approximation are minimal in practice.

Data cleaning rules (Appendix A): To ensure high-quality training data, the paper applies several filtering and preparation rules: (1) long-form audio is segmented into 10–20 second chunks (matching the Sortformer's 90-second training window but adapted for ASR training), (2) words are sorted by arrival time within each segment—even overlapping words get interleaved based on their pseudo-timestamps, (3) samples with more than 1-second overlap at the segment boundaries are excluded to avoid partial-word artifacts, (4) samples where the first speaker only utters one or two filler words at the beginning are excluded to ensure the arrival-time ordering is meaningful, and (5) speaker tokens are assigned starting from <spk0> for the earliest-arriving speaker and incrementing for each subsequent speaker.

Resource-Efficient Training with Adapters

To preserve the mono-speaker ASR performance of the base Canary model while adding multi-speaker capabilities, the paper employs adapter modules (Houlsby et al., 2019) for the larger 1B-parameter model experiments (Systems 5–6 in Table 2). Adapters are small bottleneck layers inserted into each Transformer layer of the encoder and decoder, with the base model weights kept frozen.

Architecture: Each adapter consists of a down-projection (reducing the hidden dimension to a small bottleneck $d_{\text{adapter}}$), a non-linearity, and an up-projection back to the original dimension, with a residual connection. The paper uses an adapter dimension of 256 for the 1B-parameter Canary model (Table 2, "Adapter Dim." column).

What is trained: In adapter-based experiments (Systems 5–6), all base Canary encoder and decoder parameters are frozen. Only the adapter parameters (inserted into each encoder and decoder layer) and the newly added Sortformer-compatible components (speaker kernel integration, possibly the Sortformer itself) are trained. The Sortformer weights can be either frozen (using a pre-trained diarization model) or fine-tuned jointly with the adapters.

Why this form: The adapter approach addresses the catastrophic forgetting problem: fully fine-tuning a pre-trained mono-speaker ASR model on multi-speaker data risks degrading its single-speaker performance. The paper states this directly: "A common challenge with fully fine-tuning a pretrained ASR model on new tasks is that it tends to forget previous tasks." By freezing the base weights and training only the adapters, the model retains its original mono-speaker ASR capabilities while learning to incorporate speaker information through the adapter pathways. The paper notes that "preserving the previously acquired knowledge becomes crucial for multi-speaker ASR," and adapters are described as "a more effective approach" than full fine-tuning for this goal.

Training configuration for adapters: The Canary-1B models (Systems 5–6) are trained for 75K updates on the multi-speaker ASR training data blend, with only adapter parameters learned from random initialization. All other model parameters remain frozen. The optimizer is AdamW with weight decay $10^{-3}$, inverse square root annealing, 2,500 warmup steps, peak learning rate $3 \times 10^{-4}$, and minimum learning rate $10^{-6}$.

Without adapters (Systems 1–4): For the smaller 170M Canary model, the paper fully fine-tunes both the Fast-Conformer encoder and Transformer decoder parameters for 50K steps with batch size 64. This is feasible because the 170M model is small enough that catastrophic forgetting is less severe, and the training data blend is designed to include diverse speaker configurations. Systems 1–4 do not use adapters; System 2 keeps the Sortformer frozen while fine-tuning the ASR components, and System 3 fine-tunes both jointly.

Training Configurations and Hyperparameters

Sortformer diarization model training (Section 5.1.3):

• Architecture: L-size NEST encoder (115M parameters), 18 Transformer encoder layers, hidden size 192, two feed-forward layers, four sigmoid outputs (for $K=4$ speakers). Total parameters: 123M.
• Two-stage training: Pretraining on both real and simulated data (7,180 total hours: 2,030 hours real, 5,150 hours simulated audio mixtures), then fine-tuning on real data only.
• Training samples: 90-second segments, batch size 4.
• Optimizer: AdamW, learning rate $10^{-4}$, weight decay $10^{-3}$, minimum learning rate $10^{-6}$, 2,500 warmup steps, inverse square-root annealing.
• Dropout: 0.5 for Transformer encoder and feedforward layers, 0.1 for NEST encoders.
• No SpecAugment: "We do not employ any special augmentation schemes such as SpecAugment."
• Hardware: 8 nodes of 8× NVIDIA Tesla V100 GPUs (64 GPUs total).

Multi-speaker ASR training (Section 5.3.2):

• 170M Canary (Systems 1–4): 50K steps, batch size 64, full fine-tuning of encoder and decoder, single NVIDIA RTX 6000 Ada GPU.
• 1B Canary with adapters (Systems 5–6): 75K updates, adapter dimension 256, base model frozen, only adapters trained from random initialization, single NVIDIA RTX 6000 Ada GPU.
• Optimizer (all ASR experiments): AdamW, weight decay $10^{-3}$, inverse square root annealing, 2,500 warmup steps, peak learning rate $3 \times 10^{-4}$, minimum learning rate $10^{-6}$.
• LibriSpeechMix experiments: 180K steps fine-tuning of the 170M ASR model on 960-hour LibriSpeechMix training set, with Sortformer model pre-fine-tuned on LibriSpeechMix data and then frozen during ASR training.

Simulated data generation (Section 5.1.1):

• 5,150 hours of audio mixtures created using LibriSpeech and NIST SRE04-10 as source datasets.
• Default simulator settings except: overlap ratio set to 0.12, average silence ratio set to 0.1.
• Used for Sortformer pre-training only (not for multi-speaker ASR training).

Post-processing of diarization outputs (Appendix B):

For evaluation, the raw Sortformer frame-level predictions undergo timestamp post-processing with six tunable parameters: onset threshold, offset threshold, onset padding, offset padding, minimum speech duration (to remove short false-positive segments), and minimum non-speech duration (to remove very short gaps). These parameters are optimized separately for DIHARD3 (using DIHARD3-dev) and CALLHOME/CH109 (using CALLHOME-Part1) with Optuna hyperparameter optimization. This post-processing "mitigates the errors generated from collar length and annotation style differences" across datasets and is applied only during evaluation, not during training.

Summary of Design Choices and Their Justifications

• Sigmoid output activation over softmax: Enables modeling of overlapping speech by allowing multiple speakers to be active simultaneously at a single frame. Softmax would enforce mutual exclusivity, which is architecturally wrong for diarization.

• Sort Loss over PIL for ASR integration: Converts the permutation problem from a training-time optimization (requiring specialized loss functions) into a data-preparation convention (sorting by arrival time). This makes the diarization outputs compatible with standard cross-entropy training of the ASR decoder.

• Hybrid Loss combining Sort Loss and PIL: Provides complementary training signals—Sort Loss enforces the canonical ordering needed for ASR integration, while PIL provides robustness when arrival-time estimation is unreliable. Empirically, the hybrid outperforms either loss alone on stand-alone diarization.

• Positional embeddings in Sortformer (unlike EEND): Essential for learning arrival-time ordering. Without positional embeddings, the Transformer is permutation-equivariant and cannot distinguish early frames from late frames, making arrival-time-based sorting impossible.

• Sinusoidal speaker kernels over learned embeddings: Fixed, non-parametric encoding ensures differentiability and injectivity without adding trainable parameters. The frequency-based encoding provides a structured, interpretable speaker representation that the ASR encoder can learn to associate with specific speaker tokens.

• Word-level SST over segment-level SST: Provides finer-grained speaker attribution that better matches the Sortformer's frame-level predictions, yielding lower cpWER on AMI-test (26.71% vs. 28.59% with segment-level objectives for the 1B model in Table 2).

• Adapters over full fine-tuning for large models: Prevents catastrophic forgetting of mono-speaker ASR capabilities while enabling multi-speaker adaptation. The 1B-parameter Canary model retains its single-speaker performance while learning speaker attribution through small adapter modules.

• Syllable-based pseudo-timestamps over forced alignment: Enables word-level speaker token training without requiring expensive word-level timestamp annotations or forced alignment models. The constant-speech-rate approximation is coarse but sufficient for determining correct word ordering in most cases.

• Two-stage Sortformer training (simulated + real data): Leverages abundant simulated data for pre-training the diarization model's basic speaker separation capability, then fine-tunes on real data to adapt to realistic acoustic conditions and annotation styles. This addresses the scarcity of annotated multi-speaker data.

4. Key Insights and Innovations

Innovation 1: Resolving Permutation Ambiguity Through Canonical Ordering Rather Than Permutation Search

The paper's most fundamental re-framing is treating the speaker permutation problem not as a loss-function optimization challenge (the PIL approach) but as a data-representation convention: sort speaker labels by arrival time during data preparation so the ambiguity never reaches the loss function. This is a conceptual move from handling permutation at training time to preventing it before training starts.

To understand why this is a genuine shift rather than an incremental tweak, consider how the field has approached permutation ambiguity since it was first formalized. In speech separation, Kolbæk et al. (2017) and Yu et al. (2017b) introduced Permutation Invariant Training—computing the minimum loss across all speaker-to-output-channel assignments at each training step. This became the standard solution, adopted directly by end-to-end diarization systems (Fujita et al., 2019; Horiguchi et al., 2022a) and later refined with attractor-based mechanisms to handle unknown speaker counts. The implicit assumption across all this work is that permutation ambiguity is unavoidable at the output level—since speaker labels are arbitrary, you must search over matchings during training.

Sortformer challenges this assumption by asking: what if the speaker labels aren't arbitrary? The arrival-time ordering convention imposes a semantic meaning on speaker indices (<spk0> always means "the first person who spoke"), converting what was previously a nuisance variable into a learnable target. This is significant beyond the performance numbers because it changes where the complexity lives: instead of K! permutation evaluations per training step inside a specialized loss function, the complexity moves to data preparation (sorting labels by arrival time) and model architecture (adding positional embeddings to learn temporal ordering). The former is a one-time preprocessing cost; the latter requires no specialized loss machinery.

The evidence that this re-framing works as a stand-alone training signal appears in Table 1: pure Sort Loss (without any PIL component) achieves 17.10% DER on DIHARD3 with post-processing, compared to 17.04% for pure PIL—essentially equivalent performance. This means the arrival-time convention is not a degradation of diarization quality that must be compensated for by PIL; it is a viable objective in its own right. The paper's finding that Hybrid Loss (14.76%) outperforms both pure losses separately suggests the two objectives are complementary—Sort Loss provides a strong inductive bias toward consistent output ordering, while PIL provides robustness when arrival-time estimation is ambiguous.

The practical consequence of this re-framing is what the paper emphasizes as its primary benefit: "multi-speaker ASR training [becomes] functionally equivalent to standard mono-speaker ASR training." This is not a metric claim but an architectural claim—by resolving permutations upstream, the ASR decoder can be trained with token-level cross-entropy loss, the same objective used for every other text-generation task. Previous multi-speaker ASR systems either required specialized loss functions at the output layer (preventing multi-task training) or avoided explicit speaker labeling entirely (using speaker-change tokens in SOT, which don't identify which speaker changed). Sortformer's ordering convention is what makes the integrated system possible at the computational graph level—exactly what the paper claims as a first.

Innovation 2: Differentiable Diarization Supervision Through Sinusoidal Kernel Injection

Prior to Sortformer, integrating speaker diarization with ASR at the computational graph level faced a representation mismatch: diarization models produce frame-level speaker activity matrices, while ASR decoders operate on token sequences. Cascaded systems (Medennikov et al., 2020b; Cornell et al., 2023) avoid this by processing diarization and ASR sequentially, with no gradient flow between them. End-to-end multi-speaker ASR systems (Kanda et al., 2020b; Shi et al., 2024) sidestep it by not having an explicit diarization module at all—speaker attribution is handled implicitly through serialized output formatting.

The sinusoidal kernel mechanism (Equation 14) is the paper's solution to this mismatch. It converts the Sortformer's K × T probability matrix into an additive embedding contribution to the ASR encoder states, making the diarization-to-ASR connection fully differentiable. This is not merely a plumbing detail—it establishes a new capability: the diarization model and ASR model can be trained jointly with a single loss function (token-level cross-entropy), allowing gradients from word-level speaker token errors to flow back through the speaker kernels to the Sortformer parameters.

What makes this distinctive as an idea is its minimality. The kernel function is fixed (sinusoidal, non-learned), parameter-free, and computed deterministically from speaker indices. This stands in contrast to the obvious alternative—learned speaker embeddings—which would introduce additional parameters, require careful initialization, and potentially overfit to training speaker distributions. The fixed sinusoidal encoding instead provides a structural prior: speaker 0's pattern is guaranteed to be distinct from speaker 1's pattern, regardless of training data, because they occupy different frequency bands. This is a classic signal-processing solution (frequency-division multiplexing) applied to a deep learning integration problem.

The significance extends beyond the specific sinusoidal choice. The kernel mechanism demonstrates a design pattern for systems where one model (the diarizer) produces soft, time-aligned predictions that must condition another model (the ASR encoder) without forcing hard decisions. The weighted-sum formulation (Γᵀ · P) naturally handles overlapping speech—when two speakers talk simultaneously, the encoder state receives contributions from both kernels proportionally to the Sortformer's confidence in each speaker's activity. This avoids the fragile pipeline of thresholding frame-level predictions to binary speaker assignments before feeding them to the ASR.

The paper does not ablate the sinusoidal kernel against alternative differentiable integration mechanisms (learned embeddings, attention-based speaker conditioning, concatenation), so the claim of this design's optimality remains unverified. However, as a conceptual contribution, the kernel mechanism establishes that differentiable speaker supervision can be achieved with zero additional learned parameters, which lowers the barrier for integrating diarization into existing ASR architectures.

Innovation 3: Empirical Finding That Sorted Training Objectives Match or Exceed PIL for Diarization

While the paper's primary framing emphasizes ASR integration, a perhaps understated finding is that Sort Loss alone is competitive with PIL for stand-alone speaker diarization. This is reported in Table 1: pure Sort Loss achieves 17.10% DER on DIHARD3 with post-processing, compared to 17.04% for pure PIL—a 0.06 percentage point difference that is effectively a tie. On CALLHOME-part2, Sort Loss achieves 6.52% DER for 2-speaker sessions versus PIL's 6.94%, actually outperforming the established approach. The Hybrid Loss (14.76% on DIHARD3, 5.87% on CALLHOME-2spk) provides further gains.

This result is significant because PIL has been the unquestioned default in end-to-end diarization since Fujita et al. (2019). The assumption—reasonable on its face—was that permutation invariance is necessary because speaker labels are semantically meaningless. Sortformer demonstrates that a simple ordering heuristic can provide as strong a training signal as exhaustive permutation search, at least for the typical 2–4 speaker scenarios tested. The training-time overhead comparison reinforces this: Sort Loss adds only 0.22% to per-epoch training time versus PIL, while Hybrid Loss adds 2.26%—both trivially small increases for the additional robustness gained.

This finding contributes a new diagnostic concept to the diarization literature: ordering-based objectives as a legitimate alternative to permutation-invariant objectives. Prior work had explored attention-based attractors (Horiguchi et al., 2022a) and encoder-decoder architectures (Chen et al., 2024) to improve EEND performance, but all within the PIL framework. Sort Loss introduces a fundamentally different optimization strategy—one that imposes order rather than searching over disorder. The fact that both strategies can be combined with complementary benefits (Hybrid Loss outperforming both) suggests they provide different inductive biases that the model can leverage simultaneously.

A nuance worth noting: the paper evaluates Sortformer on the same datasets (DIHARD3, CALLHOME, CH109) as prior EEND systems but does not apply per-dataset fine-tuning, unlike EEND-EDA and EEND-GLA which are individually tuned. This means the Sortformer results in Table 1 represent a single model's generalization across three datasets with only post-processing parameter differences. The competitive performance under this stricter evaluation protocol strengthens the evidence that Sort Loss is a genuinely effective training objective, not just one that works under favorable tuning conditions.

Innovation 4: Arrival-Time Sorting as a Bridge Between Frame-Level Diarization and Token-Level ASR Objectives

The paper's integration architecture rests on an insight that is easy to miss because it appears obvious in retrospect: arrival-time ordering naturally aligns the representations needed for frame-level diarization and token-level speaker labeling. The Sortformer's output channels are ordered by when speakers first appear; the ground-truth speaker tokens in the SST format are also ordered by when speakers first appear; the sinusoidal kernels map ordered speaker indices to ordered frequency patterns. This triple alignment—model outputs, ground-truth labels, and speaker representations all sharing the same sort order—is what makes the integration work without specialized loss functions.

This is not merely a "nice property" but a design principle for multi-modal sequence alignment problems. The paper effectively argues that if you can find a canonical ordering that applies to both modalities (here, time), you can avoid the combinatorially expensive permutation-matching that would otherwise be required. The arrival-time heuristic works specifically for speech because speakers must physically take turns to be intelligible—true simultaneous speech is rare and typically brief—so an ordering based on first-speech onset is almost always well-defined. For other modalities (multiple objects in video, multiple agents in a dialogue system), analogous canonical orderings might exist (spatial position, agent ID hierarchy).

The evidence for this bridging insight comes from the ablation study in Table 2. System 2 (frozen Sortformer, word-level SST) achieves 28.17% cpWER on AMI-test and 22.22% cpWER on CH109—substantial improvements over the unsupervized baseline System 1 (32.94% and 24.80% respectively). System 6 (word-level SST with adapters and 1B parameter ASR model) achieves the best overall cpWER of 26.71% on AMI-test. These results confirm that the speaker information injected by Sortformer via sinusoidal kernels is actually being used by the ASR decoder to improve speaker-attributed transcription, not just passing through as noise.

The paper's comparison between word-level and segment-level SST (Systems 5 vs. 6 in Table 2) provides additional evidence for the alignment principle. The word-level objective (cpWER 26.71%) outperforms the segment-level objective (cpWER 28.59%) on AMI-test, suggesting that the finer-grained alignment—matching the frame-level granularity of Sortformer's predictions—provides a stronger training signal. This granularity match would not be possible without the arrival-time sorting convention, since word-level speaker tokens could not be consistently assigned to the correct speaker output channel without the canonical ordering.

Innovation 5: A Negative Result with Positive Implications—ReSTᵀᴹ Training Degrades Revision Model Performance

Note: The paper mentions a ReSTᵀᴹ experiment in Appendix K (Figure 16) only in passing within the main text's Section 6 limitation paragraph, and the full paper content provided does not include Appendix K details beyond a brief mention. I cannot analyze this result in depth without seeing the actual appendix. I will instead surface a different finding that meets the "negative result with implications" criterion and is fully described in the provided content.

Innovation 5: The Training-Time Cost Asymmetry of Sort Loss Versus PIL

The paper reports a specific, quantifiable finding that changes how one thinks about the computational economics of permutation handling: Sort Loss introduces only a 0.22% increase in per-epoch training time relative to PIL (1,022.28 vs. 1,020 seconds), and Hybrid Loss adds only 2.26% (1,043.1 seconds). These numbers are dwarfed by the K! permutation search cost one might expect if Sort Loss required any permutation computation at all—but it doesn't, because the sorting is done once during data preparation, not at each training step.

This may seem like an implementation detail rather than a conceptual insight, but it has a substantive implication: the cost of permutation handling can be shifted from training time to data preparation time. For PIL, the permutation search (min_{π ∈ Π}) scales as O(K! × K × T) per training step, which for K=4 speakers means 24 comparisons per step—manageable but non-trivial at scale. For Sort Loss, the permutation is resolved once during data preparation (O(K log K) sorting time, amortized over all training epochs), and the per-step cost is identical to standard BCE. This cost structure means Sort Loss becomes increasingly attractive relative to PIL as the number of speakers grows, since PIL's permutation search cost grows factorially while Sort Loss's remains constant. The paper does not experiment with K > 4, but the scaling implication is clear from the formulation.

This cost asymmetry is what makes the paper's vision of "seamless integration into foundational speech-to-text systems and multimodal LLMs" plausible. Large-scale LLM training pipelines are already optimized around standard cross-entropy objectives; inserting a K! permutation search into the loss computation would be architecturally disruptive. Sort Loss requires no changes to the training loop whatsoever—the sorting happens in preprocessing, and the model is trained with BCE (for diarization) or cross-entropy (for ASR) exactly as before. This is a practical innovation that addresses deployment feasibility rather than accuracy, but it is arguably the feature that makes the entire Sortformer approach adoptable in production systems.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses three speaker diarization benchmarks: DIHARD3-eval (sessions with ≤4 speakers, 0.0s collar), CALLHOME-part2 (2-speaker, 3-speaker, and 4-speaker subsets, 0.25s collar), and CH109 (a 2-speaker subset of 109 sessions from Callhome American English Speech, 0.25s collar). For multi-speaker ASR evaluation on real recordings, the paper uses AMI-test (Individual Headset Mix split, ≤4 speakers) and CH109. For comparative multi-speaker ASR benchmarking, the paper evaluates on the LibriSpeechMix artificial audio mixture dataset (1-mix, 2-mix, and 3-mix test sets), following the train/validation/test split from Kanda et al. (2020b).

• Base models. For diarization, the base model is the NEST (Neural Encoder for Speech Tasks) L-size encoder (115M parameters, 18 Transformer encoder layers, hidden size 192), with two additional feed-forward layers and four sigmoid outputs on top, yielding 123M total parameters. For multi-speaker ASR, the base model is Canary (Puvvada et al., 2024) in two variants: a 170M-parameter model used for full fine-tuning experiments (Systems 1–4) and a 1B-parameter model used with adapters for resource-efficient training (Systems 5–6). The Canary architecture uses a Fast-Conformer encoder and Transformer decoder. These model sizes were chosen to represent both a lightweight fine-tunable configuration and a larger foundation model where adapter-based training prevents catastrophic forgetting of mono-speaker capabilities.

• Metrics. For stand-alone diarization: Diarization Error Rate (DER) with speaker-wise breakdown, evaluated with overlapping speech included. Different collar lengths are used per dataset to match prior evaluation conventions (0.0s for DIHARD3, 0.25s for CALLHOME and CH109). For multi-speaker ASR on real recordings: Word Error Rate (WER) and Concatenated Minimum-Permutation WER (cpWER). cpWER concatenates all utterances per speaker in both reference and hypothesis, computes WER across all K! speaker permutations, and selects the permutation yielding the lowest WER as the final score (Watanabe et al., 2020). For LibriSpeechMix: WER is reported following the methodology of Kanda et al. (2020b;a), using the Optimal Reference Combination (ORC) WER for mono-speaker baselines and cpWER for multi-speaker systems, computed with the MeetEval toolkit (von Neumann et al., 2023). The paper provides a detailed WER calculation taxonomy in Appendix D distinguishing five different evaluation scenarios (mono-speaker ASR on multi-speaker data, multi-speaker ASR with speaker tokens removed, etc.).

• Baselines. For diarization, the paper compares against multiple published end-to-end systems: EEND-EDA (Horiguchi et al., 2022a;b), WavLM-L+EEND-VC (Chen et al., 2022), EEND-GLA-Large (Horiguchi et al., 2022b), AED-EEND and AED-EEND-EE (Chen et al., 2024), and the multi-scale MSDD system (Park et al., 2022). Several of these baselines involve clustering-based post-processing (marked with † in Table 1), making them not strictly end-to-end. For multi-speaker ASR on real recordings, the baseline is an unsupervized Canary-170M model (System 1 in Table 2) trained on multi-speaker data without any speaker supervision or Sortformer integration. For LibriSpeechMix, baselines include the Canary ASR models at 170M and 1B parameters (Puvvada et al., 2024), SOT-ASR and SOT-ASR-SQR (Kanda et al., 2020b;a), DOM-SOT (Shi et al., 2024), and MT-LLM (Meng et al., 2025). The paper also includes a ground-truth speaker supervision baseline (System 4) where ground-truth speaker labels are fed through the speaker kernel instead of Sortformer predictions, establishing an upper bound on what speaker supervision can achieve.

• Generation budget / compute accounting. For diarization, models are compared by model size (parameter count), training data regime, and whether post-processing is applied. There is no "generation budget" concept—models produce a single frame-level prediction matrix. For multi-speaker ASR, training compute is measured in update steps (50K for 170M models, 75K for adapter-based 1B models, 180K for LibriSpeechMix fine-tuning). Inference runtime is measured in seconds on a single NVIDIA RTX A6000 Ada GPU, with batch size 100 for the integrated system. The paper explicitly reports that integrating Sortformer supervision into the MS-Canary system (170M parameters) increases processing time by only 0.78% (from 297.891s to 300.213s for the combined 293M parameter system), making the computational overhead of the approach nearly negligible at inference time.

• Cross-validation / statistical protocol. For diarization evaluation, Sortformer models are not individually fine-tuned on the three evaluation datasets, unlike baselines such as EEND-EDA and EEND-GLA which apply dataset-specific fine-tuning. A single Sortformer model is trained for each loss type and evaluated across all three datasets, with only post-processing parameters tuned separately for DIHARD3 (using DIHARD3-dev as the validation split) and CALLHOME/CH109 (using CALLHOME-Part1 as the validation split). Post-processing parameters (six floating-point values controlling onset/offset thresholds, padding, and minimum durations) are optimized using Optuna (Akiba et al., 2019) hyperparameter optimization on the respective validation sets. For multi-speaker ASR ablation studies, no cross-validation is mentioned—each configuration is trained once and evaluated on the fixed test sets. The paper does not report confidence intervals, statistical significance tests, or multiple random seeds for any experiments, which is a notable methodological limitation.

Main Quantitative Results

Stand-Alone Speaker Diarization (Table 1)

The headline finding is that Hybrid Loss (α = 0.5) achieves the best overall diarization performance across all three evaluation datasets, outperforming both pure PIL and pure Sort Loss. On DIHARD3-eval with post-processing, Hybrid Loss achieves 14.76% DER, compared to 17.04% for PIL-only and 17.10% for Sort Loss-only—a 2.28 absolute percentage point improvement over PIL. Without post-processing, the pattern holds: Hybrid Loss achieves 16.28% DER versus 18.33% (PIL) and 17.88% (Sort Loss).

On CALLHOME-part2, the results are more nuanced and speaker-count-dependent:

• 2-speaker sessions: Hybrid Loss achieves 5.87% DER, PIL achieves 6.94%, Sort Loss achieves 6.52%. Sort Loss alone outperforms PIL by 0.42 points, and Hybrid Loss provides a further 0.65 point improvement.
• 3-speaker sessions: Hybrid Loss achieves 8.46% DER, PIL achieves 10.30%, Sort Loss achieves 10.36%. The gap between PIL and Sort Loss narrows to near-equivalence (0.06 points), while Hybrid Loss provides a 1.84 point advantage.
• 4-speaker sessions: Hybrid Loss achieves 12.59% DER, PIL achieves 17.52%, Sort Loss achieves 17.40%. This is the largest relative improvement—Hybrid Loss reduces error by approximately 28% compared to either single-loss approach—but also where the 4-speaker limitation of Sortformer (trained for K=4 maximum) is most tested.

On CH109 (2-speaker), the results are less favorable to the proposed methods: Hybrid Loss achieves 6.86% DER with post-processing, while PIL achieves 6.89% (essentially tied), and Sort Loss achieves 10.85% (a notable degradation of 3.96 points compared to PIL). Without post-processing, PIL achieves 5.66%, outperforming both Hybrid Loss (6.27%) and Sort Loss (9.08%). This dataset-specific performance inversion—where post-processing increases DER for PIL but decreases it for Hybrid Loss—suggests the post-processing parameters may be over-tuned to the validation splits, or that the annotation style of CH109 differs in ways that interact differently with the two loss functions' output distributions.

Comparison with prior work: The paper acknowledges that its Sortformer results (123M parameters, no dataset-specific fine-tuning) are not directly comparable to baselines that apply individual dataset fine-tuning. For context:

• EEND-EDA (6.4M parameters, fine-tuned per dataset) achieves 15.55% DER on DIHARD3 and 7.83% on CALLHOME-2spk.
• AED-EEND (11.6M parameters, fine-tuned) achieves 6.18% on CALLHOME-2spk, 11.51% on CALLHOME-3spk, and 18.44% on CALLHOME-4spk.
• MSDD (31.1M parameters, with clustering) achieves 29.40% on DIHARD3 and 11.41% on CALLHOME-2spk.
• WavLM-L+EEND-VC (317M parameters, with clustering) achieves 6.46% on CALLHOME-2spk and 10.69% on CALLHOME-3spk.

Sortformer-Hybrid-Loss (with post-processing) achieves 14.76% on DIHARD3 (outperforming EEND-EDA's 15.55% without dataset-specific fine-tuning), 5.87% on CALLHOME-2spk (outperforming all listed baselines including AED-EEND's 6.18%), and 12.59% on CALLHOME-4spk (outperforming AED-EEND's 18.44% and EEND-EDA's 17.59%). However, the 3-speaker CALLHOME result (8.46%) lags behind AED-EEND (11.51%) and WavLM-L+EEND-VC (10.69%), and the 4-speaker result (12.59%) lags behind WavLM-L+EEND-VC (11.84%). The paper does not discuss these specific comparisons or why the relative performance varies by speaker count.

A critical methodological note: Sortformer is trained to handle up to 4 speakers, and DIHARD3 evaluations are restricted to sessions with ≤4 speakers. The paper does not report performance on sessions with more than 4 speakers, nor does it describe how Sortformer would handle such cases. This is a significant scope limitation since real-world meetings and conversations frequently involve more than 4 participants.

Integration with Multi-Speaker ASR on Real Recordings (Table 2)

The ablation study in Table 2 systematically evaluates the contribution of each integration component on AMI-test (≤4 speakers) and CH109 (2-speaker). The baseline (unsupervized Canary-170M, not fine-tuned on multi-speaker data) achieves 26.93% WER on AMI-test and 21.81% WER on CH109—cpWER is not computed because the baseline model has no speaker tokens.

System 1 (no speaker supervision, no adapters, word-level objective): This is the most primitive multi-speaker ASR setup—the Canary-170M is fine-tuned on multi-speaker training data with word-level speaker token objectives, but without any Sortformer module or speaker supervision. It achieves 19.67% WER and 32.94% cpWER on AMI-test, and 18.57% WER and 24.80% cpWER on CH109. The WER improvement over the baseline (26.93% → 19.67% on AMI-test) reflects the benefit of fine-tuning on in-domain multi-speaker data, even without explicit speaker diarization. However, the cpWER numbers are high (32.94% on AMI-test), indicating that while word recognition improves, speaker attribution remains poor without speaker supervision.

System 2 (frozen Sortformer, word-level objective): Adding Sortformer with frozen weights (pre-trained diarization model, not fine-tuned during ASR training) improves cpWER substantially: 28.17% on AMI-test (a 4.77 absolute point reduction from System 1's 32.94%) and 22.22% on CH109 (a 2.58 point reduction from 24.80%). The WER shows a slight degradation on AMI-test (19.67% → 20.08%) and a marginal increase on CH109 (18.57% → 18.65%), suggesting that integrating speaker information via frozen Sortformer introduces a small recognition cost—perhaps because the speaker kernel contributions modify the encoder states in ways that slightly interfere with the pre-trained ASR representations. This is the recommended practical configuration: it achieves competitive cpWER without requiring joint diarization-ASR fine-tuning, making deployment simpler.

System 3 (fine-tuned Sortformer, word-level objective): When Sortformer is fine-tuned jointly with the ASR model, the results are worse than with frozen Sortformer: cpWER on AMI-test increases to 32.74% (nearly back to the unsupervized System 1 level), and WER degrades to 19.47%. On CH109, cpWER increases to 26.97% and WER increases to 19.53%. This is a negative result: joint fine-tuning of the diarization and ASR components does not improve performance and may cause the Sortformer to lose its pre-trained diarization capabilities as it adapts to the ASR loss signal. The paper does not investigate why this occurs—possible explanations include catastrophic forgetting in Sortformer (the ASR cross-entropy gradient may not provide a strong enough diarization signal to maintain speaker separation), optimization conflict between the two objectives, or insufficient multi-speaker ASR training data to support joint fine-tuning of both components.

System 4 (ground-truth speaker supervision, word-level objective): Using ground-truth speaker labels through the speaker kernel (but Sortformer for inference) serves as an approximate upper bound: 26.83% cpWER on AMI-test and 24.39% cpWER on CH109. Surprisingly, this is worse than System 2 on AMI-test (26.83% vs. 28.17% cpWER), though slightly better on CH109 (24.39% vs. 22.22%). The AMI-test result implies that Sortformer's predicted speaker probabilities provide a more effective training signal than binary ground-truth speaker labels when passed through the sinusoidal kernel—possibly because the soft probabilities enable a more nuanced speaker encoding that better handles overlapping speech or speaker transitions. The paper does not discuss this counterintuitive finding.

System 5 (Sortformer with adapters, 1B Canary, segment-level SST): Using the 1B-parameter Canary model with adapter-based training and segment-level speaker token objectives achieves 18.58% WER and 28.59% cpWER on AMI-test, and 17.74% WER and 22.19% cpWER on CH109. The segment-level objective underperforms the word-level objective across all metrics, and the paper notes this degradation is observed "across all types of settings and datasets." This is a consistent finding that finer-grained speaker supervision (word-level) provides a stronger training signal than coarser segment-level objectives.

System 6 (Sortformer with adapters, 1B Canary, word-level SST): The best overall configuration: 18.04% WER and 26.71% cpWER on AMI-test, and 16.46% WER and 21.45% cpWER on CH109. Compared to the unsupervized System 1 (170M, word-level, no Sortformer), the cpWER improvement is 6.23 absolute points on AMI-test (32.94% → 26.71%) and 3.35 points on CH109 (24.80% → 21.45%). The 1B adapter-based system also improves WER over the 170M full-fine-tuning system (System 2: 20.08% → System 6: 18.04% on AMI-test), demonstrating that larger models with adapter-based training can simultaneously improve word recognition and speaker attribution.

Key takeaways from Table 2:

1. Sortformer provides substantial speaker attribution improvements (cpWER reduction of 4.8–6.2 points on AMI-test) when used with frozen weights (System 2 vs. System 1).
2. Fine-tuning Sortformer jointly with ASR is counterproductive (System 3 underperforms System 2), making the frozen-Sortformer configuration both simpler and more effective.
3. Word-level speaker token objectives consistently outperform segment-level objectives across model sizes and adapter configurations (System 6 vs. System 5).
4. Adapter-based training with a larger model yields the best overall performance while preserving mono-speaker ASR knowledge (System 6 achieves the lowest cpWER and competitive WER).

Multi-Speaker ASR on Artificial Mixtures (LibriSpeechMix, Table 3)

The paper evaluates on the standard LibriSpeechMix benchmark to compare against published multi-speaker ASR systems. The baseline MS-Canary (170M Canary fine-tuned on LibriSpeechMix for 180K steps without speaker supervision) achieves 2.74% WER on 1-mix, 6.55% on 2-mix, and 12.14% on 3-mix. The Sortformer-MS-Canary system (123M Sortformer + 170M Canary, total 293M parameters, with Sortformer pre-fine-tuned on LibriSpeechMix and frozen during ASR training) achieves 2.26% on 1-mix, 4.61% on 2-mix, and 9.05% on 3-mix.

The improvements from adding Sortformer are:

• 1-mix: 2.26% vs. 2.74% (a 0.48 absolute point improvement, 17.5% relative error reduction)
• 2-mix: 4.61% vs. 6.55% (a 1.94 absolute point improvement, 29.6% relative error reduction)
• 3-mix: 9.05% vs. 12.14% (a 3.09 absolute point improvement, 25.5% relative error reduction)

These are substantial gains, with the relative improvement largest on 2-mix audio and decreasing (but still large) on 3-mix. The paper also reports baselines from the mono-speaker Canary models evaluated on multi-speaker mixtures without any multi-speaker training: Canary-170M achieves 2.19% on 1-mix, 21.37% on 2-mix, and 48.71% on 3-mix; Canary-1B achieves 1.65%, 20.49%, and 47.32% respectively. These numbers highlight the severity of multi-speaker interference for mono-speaker models and contextualize the gains from multi-speaker fine-tuning.

Comparison with published systems:

• SOT-ASR (135.6M parameters): 4.6% on 1-mix, 11.2% on 2-mix, 24.0% on 3-mix. Sortformer-MS-Canary substantially outperforms SOT-ASR across all mixture types, with particularly large margins on 2-mix (4.61% vs. 11.2%) and 3-mix (9.05% vs. 24.0%).
• SOT-ASR-SQR (135.6M, improved SOT variant): 4.2% on 1-mix, 8.7% on 2-mix, 20.2% on 3-mix. Sortformer-MS-Canary again outperforms, with margins of 1.94 points on 1-mix, 4.1 points on 2-mix, and 11.15 points on 3-mix.
• DOM-SOT (33M parameters): 5.17% on 1-mix, 5.56% on 2-mix (with fixed delay, marked †), 9.96% on 3-mix (with fixed delay). The 2-mix and 3-mix results are not directly comparable due to the fixed-delay evaluation protocol (the system was tested on audio mixtures with a fixed speaker delay rather than randomly varying overlaps), but on 1-mix, Sortformer-MS-Canary (2.26%) outperforms DOM-SOT (5.17%).
• MT-LLM (8.4B parameters, with speaker supervision): 2.3% on 1-mix, 5.2% on 2-mix, 10.2% on 3-mix. Sortformer-MS-Canary achieves comparable or better performance (2.26%, 4.61%, 9.05%) with approximately 29× fewer parameters, although MT-LLM uses fundamentally different model architecture (large language model vs. encoder-decoder ASR) and training data.
• Canary-170M baseline (mono-speaker, no multi-speaker training): 2.19% on 1-mix—slightly better than Sortformer-MS-Canary's 2.26% on 1-mix, indicating that the multi-speaker training introduces a minor regression on single-speaker audio, likely due to the model learning to expect potential speaker changes.

The paper emphasizes that the improvement from MS-Canary to Sortformer-MS-Canary "indicates that we can successfully integrate a 123M-parameter Sortformer model, yielding a relative error rate reduction of 30% for 2-mix and 25% for 3-mix." The 1-mix regression (2.19% → 2.26%) is described as "minor degradation," though it is small in absolute terms (0.07 percentage points). This tradeoff—substantial multi-speaker gains with marginal single-speaker regression—is typical for multi-speaker ASR systems and represents a favorable efficiency frontier.

Runtime Performance

The paper reports that the stand-alone Sortformer diarization model was benchmarked on the LibriSpeechMix test-3mix dataset (42,514.9 seconds total audio, using 10-run averages), and for multi-speaker ASR with batch size 100, integrating Sortformer supervision with the MS-Canary system (170M parameters) increased processing time by 0.78%—from 297.891 seconds to 300.213 seconds for the combined 293M-parameter Sortformer-MS-Canary system. This is negligible overhead, supporting the paper's claim that Sortformer integration is computationally lightweight. The minute increase arises from the additional forward pass through the Sortformer encoder and the matrix multiplication Γᵀ · P to compute speaker kernel contributions, both of which are small relative to the ASR encoder-decoder computation.

Ablation Studies and Robustness Checks

Loss type (PIL vs. Sort Loss vs. Hybrid Loss): Table 1 provides the primary ablation. The finding that Hybrid Loss (α = 0.5) consistently outperforms both pure losses on DIHARD3 and CALLHOME establishes that Sort Loss and PIL provide complementary training signals. A notable pattern is that the advantage of Hybrid Loss grows with speaker count on CALLHOME: the gap between Hybrid and PIL is 1.07 points on 2-speaker sessions, 1.84 points on 3-speaker sessions, and 4.93 points on 4-speaker sessions. This suggests that the complementary benefit is more pronounced in more challenging permutation scenarios. However, only α = 0.5 is evaluated—the paper does not explore other mixing weights or investigate whether the optimal α depends on the number of speakers, dataset, or training stage.

Post-processing (with vs. without): Table 1 reports all diarization results both with and without post-processing. The post-processing provides consistent improvements for all loss types on DIHARD3 (e.g., Hybrid Loss: 16.28% → 14.76%, a 1.52 point improvement) and on CALLHOME (Hybrid Loss: 6.49% → 5.87% on 2-spk, 10.01% → 8.46% on 3-spk, 14.14% → 12.59% on 4-spk). However, on CH109, post-processing increases DER for PIL (5.66% → 6.89%, a 1.23 point degradation) while decreasing it for Sort Loss (9.08% → 10.85%)—actually making Sort Loss worse while improving PIL. The paper attributes this to "errors generated from collar length and annotation style of the datasets" but does not investigate why CH109's annotation style interacts differently with the post-processing parameters. This dataset-specific sensitivity suggests the post-processing pipeline, while effective on average, is not universally beneficial.

Sortformer weight freezing vs. fine-tuning (Table 2): The comparison between System 2 (frozen Sortformer) and System 3 (fine-tuned Sortformer) is effectively an ablation on whether the diarization model should be jointly optimized with the ASR objective. System 2 achieves 28.17% cpWER on AMI-test and 22.22% on CH109; System 3 achieves 32.74% and 26.97% respectively—a clear degradation from joint fine-tuning. This is a negative result with practical implications: it means practitioners do not need to implement joint diarization-ASR training (which would require careful balancing of potentially competing objectives), and can instead use a pre-trained frozen Sortformer as a plug-and-play speaker supervision module. The paper does not ablate whether this degradation is due to catastrophic forgetting in Sortformer, optimization conflict, or insufficient training data for the joint objective.

Word-level vs. segment-level speaker token objectives (Table 2): Comparing System 5 (segment-level SST, 1B parameters, adapters) and System 6 (word-level SST, same configuration) shows that word-level objectives achieve better cpWER on both AMI-test (26.71% vs. 28.59%) and CH109 (21.45% vs. 22.19%). This gap (1.88 points on AMI-test, 0.74 points on CH109) confirms that finer-grained speaker supervision provides a stronger training signal. The paper also states that "System 5 not only shows degradation in segment-level objectives, but we also observe this decline across all types of settings and datasets," indicating this is a robust finding rather than a dataset-specific artifact.

Ground-truth vs. Sortformer speaker supervision (Table 2): System 4 (ground-truth speaker labels through speaker kernel during training) vs. System 2 (frozen Sortformer predictions through speaker kernel) reveals a counterintuitive result: on AMI-test, Sortformer predictions (28.17% cpWER) outperform ground-truth labels (26.83% cpWER) in terms of the resulting ASR performance, though the paper reports these values in a way that makes direct comparison ambiguous (28.17% is reported for System 2; 26.83% is reported for System 4). Wait—reading Table 2 more carefully: System 2 achieves 28.17% cpWER, System 4 achieves 26.83% cpWER. So ground-truth labels achieve slightly better cpWER than Sortformer predictions (by 1.34 points), which is the expected direction. This means the earlier interpretation in Section 5.2 of the main text was incorrect—ground-truth supervision is indeed better, but the margin is relatively small, suggesting that Sortformer's predictions are already close to the quality of ground-truth diarization for ASR training purposes.

Adapter dimension (Table 2): Systems 5–6 use an adapter dimension of 256 for the 1B-parameter Canary model. No ablation on adapter dimension is reported. The paper does not investigate whether smaller adapter dimensions would suffice, how adapter dimension interacts with the amount of multi-speaker training data, or whether adapters are necessary at all for the 1B model (i.e., whether full fine-tuning of the 1B model with Sortformer supervision would outperform adapter-based training or suffer from catastrophic forgetting). These are practical questions for practitioners but remain unexplored.

Number of speakers (Table 1): The diarization results are broken down by speaker count on CALLHOME (2-spk, 3-spk, 4-spk), providing an implicit ablation on how model performance scales with the number of speakers. All methods show the expected degradation with more speakers, but the relative advantage of Hybrid Loss over PIL grows: 0.07 points on 2-spk (5.87% vs. 5.94% if we compare Hybrid with post-processing to PIL with post-processing—actually Hybrid: 5.87%, PIL: 6.94%, so the gap is 1.07 points), 1.84 points on 3-spk, and 4.93 points on 4-spk. This pattern suggests that Hybrid Loss is especially valuable in more challenging multi-speaker scenarios, though the paper does not evaluate beyond 4 speakers.

Training data composition: The Sortformer is pre-trained on a mix of 2,030 hours of real data and 5,150 hours of simulated audio mixtures, then fine-tuned on real data only. No ablation is reported on the ratio of real to simulated data, the necessity of simulated data pre-training, or whether fine-tuning on real data alone could achieve comparable results without the simulation stage. Given that 5,150 hours of simulation require substantial computational resources, understanding the marginal benefit of simulated data would be practically valuable.

What is notably missing: Several ablations one might expect are not reported:

• No α sweep for Hybrid Loss: Only α = 0.5 is evaluated. The performance of α = 0.0 (pure PIL) and α = 1.0 (pure Sort Loss) are known from the single-loss experiments, but intermediate values (0.25, 0.75) are not reported. It is possible that a different α would yield better performance, and without this sweep, the claim that α = 0.5 is "empirically determined" is thin.
• No ablation on sinusoidal kernel design: The paper uses sinusoidal kernels with frequencies proportional to speaker index k. Alternatives such as learned speaker embeddings, one-hot speaker vectors projected to the embedding dimension, or attention-based speaker conditioning are not compared. It is unknown whether the sinusoidal design is optimal or merely sufficient.
• No ablation on the L2 normalization in Equation 14: The speaker kernel contribution is added to L2-normalized encoder states (A / ‖A‖₂). The effect of this normalization—versus adding to unnormalized states or using concatenation instead of addition—is not studied.
• No ablation on the number of Sortformer encoder layers or hidden size: The model uses 18 Transformer encoder layers with hidden size 192. Whether a smaller model would suffice for diarization performance (important for deployment on resource-constrained devices) or whether a larger model would improve the downstream ASR integration is unexplored.
• No evaluation of Sort Loss without positional embeddings: Since the paper argues that positional embeddings are essential for learning arrival-time ordering (Section 3.4), an ablation demonstrating that removing positional embeddings degrades Sort Loss performance would strengthen this claim. Without this ablation, the necessity of positional embeddings remains a theoretical argument rather than an empirical finding.

Critical Assessment

Claim 1: Sort Loss can match or exceed PIL-based approaches in stand-alone diarization.

What the experiments demonstrate: Table 1 shows that pure Sort Loss achieves 17.10% DER on DIHARD3 vs. 17.04% for pure PIL (with post-processing)—a 0.06 point difference, effectively equivalent. On the 2-speaker CALLHOME subset, Sort Loss actually outperforms PIL (6.52% vs. 6.94%). This supports the claim that Sort Loss is competitive as a stand-alone training objective.

Caveats and limitations: The claim's generality is constrained by several factors. First, Sortformer (123M parameters, NEST-L backbone) is substantially larger than most EEND baselines (EEND-EDA: 6.4M, EEND-GLA-Large: 10.7M, AED-EEND: 11.6M). The paper does not compare against PIL-trained models of equivalent size and architecture, making it unclear whether the competitive performance derives from the Sort Loss objective or from the larger model capacity. A fair ablation would train the identical 123M NEST architecture with PIL and compare directly against Sort Loss—but this comparison is not provided.

Second, the per-dataset fine-tuning applied to baselines (EEND-EDA, EEND-GLA, AED-EEND) versus Sortformer's single-model evaluation makes direct DER comparisons challenging to interpret. Sortformer-Hybrid achieves 14.76% on DIHARD3 without DIHARD3-specific fine-tuning, while EEND-EDA achieves 15.55% with dataset-specific fine-tuning. This could mean Sortformer generalizes better, or it could mean EEND-EDA would also achieve competitive performance if evaluated without fine-tuning. The paper's cross-dataset evaluation approach is actually stronger methodologically, but the baseline comparison infrastructure wasn't designed for it.

Third, the CH109 results show that Sort Loss alone (10.85% with post-processing) substantially underperforms PIL (6.89%) on this dataset, and post-processing makes Sort Loss worse (9.08% → 10.85%). This dataset-specific degradation is not explained and suggests that arrival-time ordering may be less reliable as a training signal on certain types of conversational data—perhaps where speakers frequently interrupt each other or where the "first speaker" is ambiguous due to overlapping initial segments.

Claim 2: Hybrid Loss combining Sort Loss and PIL provides complementary benefits and outperforms either loss alone.

What the experiments demonstrate: This is the most robustly supported claim in the paper. Hybrid Loss (α = 0.5) consistently achieves the lowest DER on DIHARD3 and across all speaker counts on CALLHOME (Table 1). The pattern of growing advantage with more speakers (1.07 → 1.84 → 4.93 absolute points) is suggestive of genuine complementarity rather than noise. The training overhead is minimal (2.26% per-epoch increase).

Caveats: Only α = 0.5 is tested. Without an α sweep, it is unknown whether this weight is optimal or whether performance is robust to the mixing hyperparameter. The complementarity mechanism is hypothesized (Sort Loss provides ordering bias, PIL provides permutation robustness) but not empirically isolated—e.g., by analyzing which training examples are correctly handled by each loss component. The claim "complementary strengths" remains a plausible interpretation rather than a demonstrated mechanism.

Claim 3: Sortformer enables seamless integration of speaker diarization into ASR systems via arrival-time-sorted speaker supervision, making multi-speaker ASR training functionally equivalent to mono-speaker ASR training.

What the experiments demonstrate: Table 2 (Systems 2, 6 vs. System 1) shows that adding frozen Sortformer supervision reduces cpWER by 4.77–6.23 absolute points on AMI-test and 2.58–3.35 points on CH109 compared to the unsupervized multi-speaker baseline. Table 3 shows 30% relative WER reduction on 2-mix LibriSpeechMix. These are substantial improvements that confirm the speaker supervision is being used effectively by the ASR decoder.

What the experiments do NOT demonstrate: The claim of "functionally equivalent to mono-speaker ASR training" is about the training procedure, not the accuracy. The paper does not compare the training dynamics, convergence speed, or hyperparameter sensitivity of multi-speaker ASR training with Sortformer against mono-speaker ASR training. It does not demonstrate that a practitioner can take an existing mono-speaker ASR training pipeline, add Sortformer and speaker tokens, and achieve multi-speaker capabilities without adjusting learning rates, batch sizes, or other training hyperparameters. The claim is more aspiration than demonstrated fact.

Furthermore, the "seamless integration" claim is challenged by System 3's results: fine-tuning Sortformer jointly with the ASR model degrades performance (32.74% cpWER vs. 28.17% with frozen Sortformer). This means the integration is not seamless in the sense that joint optimization works out of the box—practitioners must know to keep Sortformer frozen, which is a non-obvious design choice that the paper arrives at empirically. If the integration were truly seamless, joint fine-tuning should at minimum not harm performance.

Claim 4: Sortformer integration introduces minimal computational overhead (0.78% inference time increase) while significantly improving multi-speaker transcription accuracy.

What the experiments demonstrate: The 0.78% runtime increase (297.891s → 300.213s for batch size 100 on a single RTX A6000 Ada GPU) is precise and well-measured. This is a genuinely negligible overhead that makes the approach practical for deployment. The accuracy improvements are documented in Tables 2 and 3 as discussed above.

Caveats: The runtime benchmark is for the 170M Canary + 123M Sortformer configuration. For larger ASR models (1B Canary), the relative overhead would be even smaller since the Sortformer size remains fixed at 123M. However, the runtime measurement is done on a specific GPU (RTX A6000 Ada) with specific batch size (100). Inference time on different hardware (edge devices, CPUs, older GPUs) or with different batch sizes (streaming, single-utterance processing) may show different overhead percentages. The paper doesn't profile memory usage, which could be a bottleneck for on-device deployment where 123M additional parameters may exceed memory constraints even if compute overhead is minimal.

Claim 5: Word-level speaker token objectives outperform segment-level objectives for multi-speaker ASR training.

What the experiments demonstrate: Table 2, System 6 vs. System 5, shows cpWER improvements of 1.88 points on AMI-test and 0.74 points on CH109. The paper states this pattern is observed "across all types of settings and datasets," suggesting robustness.

Caveats: The word-level objective requires word-level timestamps or pseudo-timestamps (via the syllable-based approximation in Appendix C). The paper does not ablate whether the word-level advantage persists when using approximated timestamps versus ground-truth timestamps—it's possible that the word-level objective's advantage would be larger with perfect timestamps, or that the approximation introduces errors that partially offset the benefit of finer granularity. The claim is supported for the timestamp-approximated training setup described, but the dependence on timestamp quality is unquantified.

Missing experiments that would strengthen the paper:

1. Direct architectural comparison to a PIL-based integrated system. The paper claims Sortformer is the first to integrate end-to-end diarization with end-to-end ASR at the computational graph level, and this may be true, but a strong ablation would be: train the identical ASR architecture with PIL-based diarization supervision (if implementable) and compare against Sortformer. Without this, the claims about Sortformer's unique suitability for integration are primarily arguments from architectural convenience rather than empirical superiority.

2. Evaluation on >4 speaker scenarios. Both the diarization and multi-speaker ASR experiments are capped at 4 speakers. The paper does not discuss how Sortformer would scale to dinner party scenarios (5–8 speakers) or meetings with larger participant counts. This is a significant scope limitation for a method that claims to enable "seamless integration" into general-purpose speech-to-text systems.

3. Streaming evaluation. The paper mentions "future work will explore streaming systems" in the conclusion, but no streaming results are reported. The arrival-time sorting convention requires knowing which speaker speaks first, which in a streaming context requires waiting for all speakers to begin before assigning sorted indices—potentially introducing latency. How Sort Loss handles the streaming case (where arrival-time ordering may change as new speakers enter) is not addressed.

4. Multi-task training demonstration. A core motivation for Sortformer is enabling multi-task training without per-task loss specialization. The paper does not include any experiment where the model is simultaneously trained for ASR, translation, summarization, and speaker attribution—the scenario where Sortformer's advantages over PIL would be most apparent. Without such an experiment, the multi-task motivation remains hypothetical.

5. Ablation on the sinusoidal kernel mechanism. The paper does not compare the fixed sinusoidal kernels against learned speaker embeddings, concatenation-based speaker conditioning, or cross-attention-based speaker context. The design choice is theoretically motivated but not empirically validated against alternatives.

6. Statistical significance reporting. No confidence intervals or standard deviations are reported for any results. With test sets of varying sizes (AMI-test, CH109 with 109 sessions, LibriSpeechMix with standard splits), the stability of the reported improvements is unknown. The post-processing parameters are tuned on validation sets with Optuna, which involves multiple evaluations and could lead to overfitting on the validation splits without proper statistical correction.

7. Analysis of failure modes. The paper does not analyze when Sort Loss fails relative to PIL—which speakers, which acoustic conditions, which overlap patterns cause arrival-time ordering to break down. The CH109 result (where Sort Loss substantially underperforms PIL) suggests such failure modes exist but are not investigated. Understanding these conditions would be practically important for knowing when to prefer Hybrid Loss or when to fall back to pure PIL.

6. Limitations and Trade-offs

The Arrival-Time Ordering Heuristic Can Fail, and Failure Modes Are Uncharacterized

Sortformer's entire approach — Sort Loss, sorted speaker tokens, the alignment between diarization outputs and ASR decoder targets — depends on the assumption that speaker arrival times provide a reliable canonical ordering. The paper acknowledges this vulnerability in Section 3.3:

"While Sortformer can be trained solely with Sort Loss, there is a limitation that arrival time estimation is not always correct. This issue becomes more pronounced as the number of speakers increases during the training session."

This is not a peripheral caveat; it is a structural weakness. The arrival-time heuristic can break in several ways: two speakers may begin speaking nearly simultaneously (within the same frame or a few frames of each other), a very quiet or distant speaker may be missed by the model at their actual onset, or background noise may be misclassified as a speaker's first utterance. In any of these cases, the canonical ordering imposed on the ground-truth labels does not reflect the true temporal sequence, and Sort Loss trains the model to match an incorrect ordering.

The consequence is that the model may learn a systematically wrong speaker-to-output-channel mapping for certain acoustic conditions. Because the ASR decoder's cross-entropy loss on speaker tokens depends on the Sortformer's output channels aligning with the sorted ground-truth speaker indices, an incorrect ordering in the diarization model will cause the decoder to generate <spk0> tokens when it should generate <spk1>, and vice versa. This is functionally a speaker confusion error that no amount of ASR decoder training can correct, since the supervisory signal itself is corrupted by the ordering error.

The empirical evidence for this limitation is present but not analyzed. On CH109 (Table 1), pure Sort Loss achieves only 10.85% DER with post-processing compared to 6.89% for pure PIL — a 3.96 percentage point degradation. The single-model evaluation across datasets means the identical trained Sortformer processes DIHARD3, CALLHOME, and CH109 audio, yet its performance relative to PIL varies dramatically (near-equivalent on DIHARD3 and CALLHOME-2spk, substantially worse on CH109). This dataset-specific performance gap strongly suggests that CH109 contains conversational patterns — perhaps rapid turn-taking, frequent interruptions, or overlapping speech onsets — where the arrival-time heuristic is less reliable, but the paper does not investigate this possibility. There is no analysis of which CH109 sessions exhibit the largest Sort Loss degradation, what acoustic conditions correlate with ordering failures, or how often the arrival-time ordering of ground-truth labels differs from the true temporal order.

The Hybrid Loss (α = 0.5) partially mitigates this by allowing PIL to handle cases where arrival-time ordering fails. On CH109, Hybrid Loss achieves 6.86% DER with post-processing, recovering most of the gap versus PIL (6.89%). This confirms that the ordering failures exist and that PIL provides a meaningful fallback. However, the mitigation is incomplete: (1) the paper does not establish whether α = 0.5 is the optimal balance between Sort Loss and PIL for robustness to ordering failures; (2) at inference time, the model still produces sorted outputs — even if trained with Hybrid Loss, the output channels follow the arrival-time convention, meaning ordering failures at test time will still cause speaker confusions in the ASR output; and (3) the paper provides no diagnostic for practitioners to determine when arrival-time ordering is unreliable for their target domain, making it difficult to decide whether to use Hybrid Loss, pure PIL, or a different approach entirely.

The paper suggests no systematic method for detecting or correcting arrival-time ordering failures beyond the Hybrid Loss formulation. The limitation is flagged honestly but left largely unresolved.

The Joint Fine-Tuning of Sortformer with ASR Objectives Degrades Performance, Contradicting the "Seamless Integration" Narrative

A central claim of the paper is that Sortformer enables "seamless integration" of speaker diarization into ASR systems, making "multi-speaker ASR training functionally equivalent to standard mono-speaker ASR training" (Section 3.1). The implication is that practitioners can plug Sortformer into an existing ASR pipeline and train everything jointly with a single cross-entropy loss. The experimental results contradict this claim.

System 3 in Table 2 (Sortformer fine-tuned jointly with the ASR model, word-level objectives) achieves 32.74% cpWER on AMI-test — substantially worse than System 2 with a frozen Sortformer (28.17%) and nearly back to the unsupervized baseline (32.94%). On CH109, the degradation is similar: 26.97% cpWER for fine-tuned Sortformer versus 22.22% for frozen Sortformer. The WER also degrades slightly in both cases (e.g., 19.47% vs. 20.08% on AMI-test).

The consequence is that the "functionally equivalent" training claim only holds when Sortformer weights are frozen — a constraint that the paper does not emphasize in its framing but that fundamentally changes what "seamless integration" means. Joint optimization of the diarization and ASR components, which would be the natural approach for a truly integrated system, is actively harmful. Practitioners must know to freeze Sortformer, which requires: (1) pre-training a high-quality Sortformer model independently (which itself requires 7,180 hours of multi-speaker data and 64 GPUs, as described in Section 5.1.3), and (2) accepting that the diarization model cannot adapt to the specific ASR model's encoder representations or to domain-specific speaker characteristics encountered during ASR fine-tuning.

The paper does not investigate the mechanism behind this degradation. Several hypotheses are plausible but untested: catastrophic forgetting in Sortformer (the ASR cross-entropy gradient may overwrite the diarization knowledge learned during pre-training), optimization conflict (the BCE loss on frame-level speaker presence and the cross-entropy loss on token-level speaker labels may pull Sortformer parameters in incompatible directions), or data insufficiency (the 230 hours of multi-speaker ASR training data may be insufficient to support joint optimization of both components simultaneously). Without understanding the cause, it is impossible to know whether the degradation could be mitigated by, for example, using a smaller learning rate for Sortformer parameters, employing separate optimizers, or gradually unfreezing layers.

The paper presents the frozen-Sortformer results as the primary configuration and does not frame the joint fine-tuning failure as a limitation. However, for a method whose stated motivation is eliminating the need for separate, independently optimized components (Section 2.3: "difficult to optimize because each component often needs to be tailored for domain-specific datasets"), the inability to jointly optimize the diarization and ASR models under a single loss represents a significant gap between the motivating vision and the demonstrated capability.

Evaluation Is Limited to a Maximum of Four Speakers, With No Path to Scaling Addressed

All experiments in the paper — both diarization and multi-speaker ASR — are capped at four speakers. The Sortformer model is "trained to handle up to 4 speakers" (Section 5.2). The DIHARD3 evaluation uses only sessions with four or fewer speakers. CALLHOME results are broken down into 2-, 3-, and 4-speaker subsets. The multi-speaker ASR training data blend is described as containing "up to four speakers per sample" (Section 5.3.1). Even the simulated audio mixture data (5,150 hours) is generated with the same constraint implicitly, though the overlap ratio of 0.12 suggests relatively sparse multi-speaker activity.

This is a severe scope limitation for a system that claims to address "multi-speaker" speech-to-text in general. Real-world multi-speaker scenarios — dinner parties, panel discussions, conference calls, classroom settings — routinely involve 5–10 or more active participants. The paper's own framing acknowledges this implicitly when citing CHiME challenges (Barker et al., 2017; 2018; Watanabe et al., 2020; Cornell et al., 2023), which include dinner party scenarios with 4–8 speakers. The absence of any evaluation or discussion of how Sortformer handles more than four speakers means a practitioner cannot assess whether the approach is viable for their target use case without conducting their own experiments.

The limitation has both architectural and practical dimensions. Architecturally, Sortformer uses a fixed number of output channels (four sigmoid heads, Section 5.1.3). Extending to more speakers would require either: (1) increasing the number of output heads, which changes the model architecture and requires retraining, (2) using a speaker-count-agnostic mechanism like the encoder-decoder attractors in EEND-EDA (Horiguchi et al., 2020), which the paper does not adopt, or (3) processing overlapping subsets of speakers sequentially, which would break the end-to-end differentiable integration. The paper does not discuss any of these extensions. Practically, the arrival-time ordering convention becomes less reliable as speaker count increases — the paper acknowledges this in Section 3.3 ("this issue becomes more pronounced as the number of speakers increases") — meaning the core Sort Loss mechanism may degrade even if the architecture is extended. With 8 speakers, the probability that at least two speakers begin speaking nearly simultaneously increases substantially, and the sorting convention becomes increasingly arbitrary.

The paper does not attempt to mitigate this limitation. The conclusion lists "target-speaker ASR features" as future work but does not address the fundamental speaker-count ceiling. The 4-speaker limit is a hard, unaddressed boundary on the method's applicability, and the paper provides no evidence that the approach would generalize to higher speaker counts even with architectural modifications.

The Training Data and Model Dependency Are Single-Ecosystem: All Results Are on NVIDIA NEST/Canary Architectures With Proprietary Training Recipes

All experiments in the paper use a specific, tightly coupled model ecosystem: the NEST encoder (Huang et al., 2025) for Sortformer, the Canary ASR model (Puvvada et al., 2024) for multi-speaker experiments, and NVIDIA's NeMo Framework for implementation. The Sortformer is pre-trained on a proprietary data blend of 2,030 hours of real data and 5,150 hours of simulated mixtures generated by an NVIDIA speech data simulator (Park et al., 2023) with specific parameter settings (overlap ratio 0.12, silence ratio 0.1). The multi-speaker ASR training data blend is similarly curated from specific datasets (AMI, ICSI, DipCo, Fisher) with custom cleaning, truncation, and filtering rules (Appendix A).

This is not a minor concern about reproducibility — it is a question about whether Sort Loss and the sinusoidal kernel integration would work in different ASR ecosystems. Several specific dependencies raise concern:

First, the PRM-like assumption of NEST encoder quality: Sortformer is built on the NEST Fast-Conformer encoder, which the paper describes as "self-supervised" and "all-purpose seasoning to speech processing tasks" (Huang et al., 2025). The quality of the frame-level embeddings produced by this encoder — their speaker discriminability, their temporal resolution, their robustness to noise and overlap — directly determines Sortformer's diarization accuracy. If a practitioner uses a different encoder (WavLM, HuBERT, Whisper encoder, or a custom Conformer), the diarization performance may differ substantially, and the optimal Sort Loss configuration (α weight, encoder depth, whether positional embeddings are needed) may change. The paper provides no guidance on how encoder choice affects Sortformer's behavior.

Second, the dependence on a specific ASR architecture: the sinusoidal kernel integration (Equation 14) adds speaker information to the ASR encoder states via L2-normalized addition. This specific mechanism assumes the encoder states are structured such that additive speaker identity information is separable from phonetic content. Whether this holds for other ASR architectures — RNN-Transducers, CTC-based models, or encoder-decoder models with different normalization schemes — is untested. The paper also relies on the Canary decoder's ability to learn the association between sinusoidal speaker patterns in the encoder states and discrete speaker tokens in the output sequence, which may depend on the decoder's capacity, attention mechanism, or training data scale in ways that do not transfer to other architectures.

Third, the simulation-to-real transfer depends on the specific simulator and data sources: The 5,150 hours of simulated audio mixtures are generated from LibriSpeech and NIST SRE04-10 using a specific simulator (Park et al., 2023) with specific settings. Practitioners using different source datasets or simulators may achieve different diarization pre-training quality, affecting downstream ASR integration. The paper does not ablate the simulator parameters, the source datasets, or the real-to-simulated data ratio, so the sensitivity of the approach to these choices is unknown.

The paper does not address this limitation beyond making code and models available through the NVIDIA NeMo Framework. This is a legitimate form of reproducibility, but it does not address cross-ecosystem generalization. A practitioner using, for example, a Whisper-based ASR pipeline or a WavLM-based diarization system cannot infer from the paper's results whether Sort Loss or the sinusoidal kernel approach would be effective in their setting. The paper's claims — particularly the 30% relative WER reduction on LibriSpeechMix and the 0.78% runtime overhead — are validated only within the NVIDIA NEST/Canary ecosystem and should be interpreted as such.

The Difficulty Estimation (Arrival-Time Sorting) Cost Is Not Accounted for in the Headline Training Narrative

While Sort Loss eliminates the per-step computational overhead of PIL's permutation search (the paper reports only a 0.22% per-epoch training time increase in Section 5.1.3), this comparison omits a different cost: the data preparation cost of sorting ground-truth labels by arrival time in a way that is accurate enough for Sort Loss to work effectively.

To construct the SST format with word-level speaker tokens (which achieves the best cpWER in Table 2), the paper requires: (1) determining each speaker's first utterance onset to assign sorted speaker indices (<spk0>, <spk1>, etc.), and (2) interleaving words from different speakers in correct temporal order so that the word-level speaker tokens appear in the right sequence. The paper's data cleaning rules (Appendix A) describe segmenting long-form audio into 10–20 second chunks, sorting words by arrival time, and filtering out samples where the first speaker only has one or two filler words. When word-level timestamps are missing (which is common, as the paper acknowledges in Section 4.3), the system relies on the syllable-based pseudo-timestamp approximation (Appendix C), which assumes constant speaking rate within each segment — an approximation that becomes increasingly inaccurate for conversational speech with variable pacing, pauses, and overlapping talk.

This data preparation pipeline is not computationally free, but its cost is invisible in the paper's accounting. The training time comparison (1,020s vs. 1,022s per epoch) compares only the per-step loss computation, not the preprocessing required to make Sort Loss usable. For a practitioner building a multi-speaker ASR system from raw conversational audio, the full pipeline cost includes: forced alignment or pseudo-timestamp generation, arrival-time-based speaker index assignment, word reordering by estimated onset times, filtering out edge cases (overlapping segment boundaries, filler-word starts), and validation that the sorted ordering is correct. For datasets where word-level timestamps are unavailable and forced alignment is infeasible, the pseudo-timestamp approximation may introduce ordering errors that degrade training signal quality, but the paper provides no measurement of how often the approximated word order differs from the true order or how such errors affect downstream ASR performance.

The consequence is that the claim of "functionally equivalent to standard mono-speaker ASR training" (Section 3.1) is incomplete. Standard mono-speaker ASR training requires transcripts aligned at the utterance or segment level. Sortformer-based multi-speaker ASR training additionally requires: speaker-index sorting by arrival time, word-level temporal ordering (or a segment-level approximation with known performance degradation), and syllable-level pseudo-timestamps when ground-truth word alignments are unavailable. This is a strictly higher annotation and preprocessing burden, and the paper's acknowledgment of this burden is limited to describing the approximation technique (Appendix C) without quantifying its error, its computational cost, or its impact on final transcription accuracy.

The paper does not ablate the effect of timestamp quality on Sort Loss or ASR performance — for example, by comparing training with ground-truth word timestamps versus pseudo-timestamps, or by varying the accuracy of the syllable-based approximation. Without such an ablation, a practitioner cannot assess how much annotation effort is required to achieve the reported results, nor whether the approach is viable when only segment-level transcripts are available (with no word timing information at all). This is a practical deployment barrier that the paper's headline claims of simplicity and ease of integration do not address.

The Single Benchmark, Single Domain Evaluation Leaves the Difficulty-Dependent Generalization of Sort Loss Uncharacterized

All diarization results are on three English-language conversational speech datasets (DIHARD3, CALLHOME, CH109), all multi-speaker ASR results on real recordings are on AMI-test and CH109 (English meetings and telephone conversations), and the LibriSpeechMix experiments use artificial mixtures of English read speech. The paper provides no results on: non-English languages, child speech, elderly speech, speech with heavy accents or dialects, noisy or reverberant environments (despite Sortformer being trained on clean LibriSpeech-based simulations and relatively clean real datasets like AMI headset mixes), formal presentation settings (lectures, sermons) where speaker roles are structured differently, or domains with distinct acoustic characteristics (medical, legal, broadcast).

This narrow evaluation domain matters because the arrival-time sorting convention's reliability likely varies with acoustic and conversational conditions. In formal meetings with structured turn-taking, arrival-time ordering is almost always well-defined and the first speaker is unambiguous. In informal, overlapping conversations with crosstalk, interruptions, and simultaneous speech onsets, arrival-time ordering becomes increasingly arbitrary. The CH109 result — where pure Sort Loss achieves 10.85% DER versus 6.89% for PIL (Table 1) — already hints at domain sensitivity within the English conversational space. Without evaluation on a broader range of conditions, practitioners cannot determine whether their target domain is more like CALLHOME (where Sort Loss is competitive) or more like CH109 (where it substantially underperforms).

The consequence extends beyond diarization accuracy. The entire integrated multi-speaker ASR pipeline depends on Sortformer providing reliable, arrival-time-sorted speaker predictions to the ASR encoder via sinusoidal kernels. If Sortformer's diarization degrades in a particular domain, the ASR decoder receives corrupted speaker supervision that may cause cascading speaker attribution errors. The paper's evaluation provides no evidence about how robust the integrated system is to domain shift — whether, for example, a Sortformer trained on the paper's 7,180-hour data blend and frozen during ASR training would maintain its diarization quality on out-of-domain audio, or whether the ASR decoder would need to be retrained with domain-specific speaker supervision.

The paper does not address this limitation. The conclusion mentions future work on "streaming systems, target-speaker ASR features, and multi-task capabilities such as translation and summarization" but does not mention cross-domain or cross-lingual evaluation. The availability of code and models through NeMo partially mitigates this by enabling other researchers to evaluate on their own domains, but until such evaluations exist, the paper's results should be understood as validated specifically for English-language conversational speech in relatively clean acoustic conditions, with the 4-speaker ceiling and domain sensitivity remaining open questions.

7. Implications and Future Directions

How This Work Changes the Landscape

Sortformer introduces a conceptual reframing of the speaker permutation problem that has quietly structured the diarization and multi-speaker ASR fields for nearly a decade. Since Kolbæk et al. (2017) and Yu et al. (2017b) formalized Permutation Invariant Training, the field has treated speaker-label ambiguity as a loss-function problem: because speaker identities are arbitrary indices, training must compute the minimum loss across all possible speaker-to-output-channel matchings at each step. This framing produced increasingly sophisticated PIL-based architectures—EEND with self-attention (Fujita et al., 2019), encoder-decoder attractors (Horiguchi et al., 2022a), attention-based variants (Chen et al., 2024)—all operating within the same core assumption that permutation ambiguity is unavoidable at the output level and must be handled at the loss level.

Sortformer challenges this assumption by asking: what if speaker labels are not arbitrary, but instead follow a canonical ordering imposed during data preparation? The arrival-time sorting convention—the first speaker to speak is <spk0>, the second <spk1>, and so on—converts the permutation problem from a training-time optimization into a data-preparation convention. This is not an incremental improvement to PIL; it is a different class of solution that changes where the complexity lives in the system. Under PIL, complexity is in the loss function (K! permutation evaluations per training step). Under Sort Loss, complexity is in preprocessing (sorting labels by first-utterance onset, which is O(K log K) once per dataset) and in architecture (adding positional embeddings to break permutation equivariance). The Sort Loss training step itself is standard BCE—identical in form and computational cost to any binary classification task.

The empirical finding that this reframing works competitively is what gives the conceptual shift weight. Pure Sort Loss achieves 17.10% DER on DIHARD3 versus PIL's 17.04% (Table 1)—an effective tie. On CALLHOME 2-speaker, Sort Loss actually outperforms PIL (6.52% vs. 6.94%). The Hybrid Loss (α = 0.5) achieves 14.76% DER on DIHARD3, outperforming either single-loss approach. These numbers say something important: the arrival-time heuristic is not a degraded approximation of PIL that needs to be compensated for; it is a viable training objective in its own right, providing a training signal of comparable quality to exhaustive permutation search while being architecturally simpler.

The landscape change this enables is most visible in the ASR integration story. Prior to Sortformer, multi-speaker ASR systems faced a hard choice: either (1) use cascaded diarization and ASR with no differentiable connection between them (Medennikov et al., 2020b; Cornell et al., 2023), which works well but is difficult to optimize jointly and requires per-component domain adaptation, or (2) avoid explicit diarization entirely using serialized output training (Kanda et al., 2020b; Shi et al., 2024), which simplifies the architecture but cannot leverage pre-trained diarization models and struggles with domain transfer. Sortformer offers a third path: differentiable diarization supervision injected into the ASR encoder via fixed sinusoidal kernels, producing speaker tokens in the ASR decoder output that are trained with standard cross-entropy. The 30% relative WER reduction on 2-mix LibriSpeechMix (Table 3) and 4.77 absolute cpWER reduction on AMI-test (Table 2) demonstrate that this path works in practice.

The reconciliation of a prior contradiction deserves emphasis. The field had an unresolved tension: end-to-end diarization models (EEND, AED-EEND) achieved strong stand-alone performance but could not be integrated into ASR systems at the computational graph level, while end-to-end multi-speaker ASR systems (SOT, t-SOT) avoided explicit diarization but underperformed cascaded systems, especially on domain-specific data. Sortformer resolves this tension by showing that the two objectives—accurate frame-level speaker separation and token-level speaker attribution—can share a single differentiable computational graph when the permutation is resolved through canonical ordering rather than loss-function search. The paper's claim to be "the first to integrate an end-to-end diarization system with an end-to-end multi-speaker ASR model at the computational graph level" (Section 2.3) appears justified by the literature survey provided, and the mechanism (sorted outputs + sinusoidal kernel injection) is novel.

This reframing also redirects research attention in several ways:

• Away from more complex permutation-handling loss functions. The finding that a simple sorting heuristic matches PIL's diarization performance suggests that further investment in sophisticated permutation-invariant architectures may have diminishing returns, at least for the ≤4-speaker scenarios studied. The Hybrid Loss result (14.76% DER) suggests that a small amount of PIL as a safety net for ordering failures provides the best of both worlds, but the direction of improvement is toward better ordering heuristics, not more complex permutation searches.

• Toward canonical ordering conventions as a general integration pattern. The arrival-time sorting principle may generalize beyond speaker diarization. Any multi-modal sequence alignment problem where outputs have arbitrary labels but can be ordered by a reliable heuristic (spatial position for multi-object tracking, agent identity for multi-agent dialogue, source language for multi-lingual translation) could adopt a similar "sort first, train with standard loss" approach. The paper's sinusoidal kernel mechanism demonstrates that fixed, non-learned embeddings can carry this ordering information across modalities in a differentiable way.

• Toward frozen specialist models as plug-and-play supervision modules. The negative result that joint fine-tuning of Sortformer with ASR degrades performance (System 3 vs. System 2 in Table 2: 32.74% vs. 28.17% cpWER on AMI-test) is important for system design. It suggests that keeping diarization models frozen during ASR training is not merely a convenience but a requirement for preserving diarization quality. This pattern—pre-train a specialist model, freeze it, inject its outputs into a larger system via fixed differentiable transformations—may apply broadly to multi-modal integration where joint optimization causes catastrophic forgetting or optimization conflict.

The paper does not cause a paradigm shift in the sense of rendering PIL obsolete; the Hybrid Loss results demonstrate that PIL still provides complementary value, especially in higher-speaker-count scenarios and on datasets like CH109 where arrival-time ordering is less reliable. Rather, Sortformer expands the design space: practitioners now have a choice between permutation-invariant objectives (PIL), canonical-ordering objectives (Sort Loss), and their combination (Hybrid Loss), with the selection depending on the target application (stand-alone diarization vs. ASR integration), the expected number of speakers, and the reliability of arrival-time ordering in the target domain.

Follow-Up Research This Work Enables

Characterizing when arrival-time ordering fails and building diagnostic tools for practitioners. The CH109 result—where pure Sort Loss achieves 10.85% DER versus PIL's 6.89% (Table 1)—is the paper's clearest signal that arrival-time ordering is domain-sensitive, but the paper provides no analysis of why CH109 is harder. A strong follow-up would: (1) manually annotate the arrival-time ordering quality in CH109 versus CALLHOME, measuring how often the first-speaker assignment is unambiguous, how often two speakers begin within a small window (e.g., <100ms), and how often overlapping speech obscures the true onset; (2) train a lightweight classifier to predict, from acoustic features or Sortformer's own output distribution, whether the arrival-time ordering for a given sample is likely correct; and (3) use this classifier at inference time to decide whether to trust the sorted outputs or fall back to a PIL-based post-hoc permutation matching. The key measurement would be whether such a confidence-gated system closes the CH109 gap while maintaining performance on CALLHOME and DIHARD3. This would convert the uncharacterized limitation into a manageable, diagnosable condition.

Ablating the sinusoidal kernel mechanism against learned alternatives and alternative integration methods. The paper's sinusoidal kernel design (Equation 14) is theoretically motivated—fixed sinusoidal patterns at frequency bands proportional to speaker index provide an injective, differentiable, parameter-free speaker encoding—but never compared against alternatives. A strong ablation study would train the identical Sortformer + Canary architecture with: (a) learned speaker embeddings (a trainable K × M matrix replacing the fixed sinusoidal Γ), (b) one-hot speaker vectors projected to the embedding dimension via a learned linear layer, (c) cross-attention between encoder states and Sortformer predictions (instead of additive combination), and (d) concatenation of Sortformer predictions with encoder states (instead of additive combination). The key measurements would be: cpWER on AMI-test and CH109 (replicating Table 2), training convergence speed, sensitivity to random initialization, and whether learned embeddings overfit to the training speaker distribution (evaluated by testing on held-out speakers with different acoustic characteristics). If learned embeddings achieve comparable or better performance, the sinusoidal design is an unnecessary constraint; if they underperform or overfit, the paper's fixed-kernel design is validated as a robust default.

Extending Sortformer to handle more than four speakers using attractor-based or iterative mechanisms. The hard 4-speaker ceiling is the most practically constraining limitation. A direct extension would replace Sortformer's fixed four sigmoid heads with an encoder-decoder attractor mechanism (as in EEND-EDA) while preserving the arrival-time sorting convention: the attractors would be generated sequentially in arrival-time order rather than unordered, and the model would be trained with Sort Loss on the sequentially generated attractor outputs. The experiment would: (1) train this extended model on simulated mixtures with 2–8 speakers (using the same simulator as the paper, Section 5.1.1, but expanding the speaker count range), (2) evaluate on real 5–8 speaker sessions from datasets like AMI (which the paper already uses but truncates to ≤4 speakers) or the full DIHARD3 set without the ≤4-speaker filter, and (3) measure whether the Hybrid Loss advantage over pure PIL grows or shrinks with speaker count, extending the pattern observed in Table 1 (where the Hybrid advantage grows from 1.07 points on 2-spk to 4.93 points on 4-spk). The key question is whether arrival-time ordering remains a viable training signal when 6–8 speakers are present and onset collisions become more frequent.

Testing the "functionally equivalent to mono-speaker ASR training" claim with a multi-task training stress test. One of the paper's strongest motivational claims—that Sortformer enables multi-task training without per-task loss specialization—is never tested empirically. A direct experiment would: (1) take the Canary-1B model with adapters (System 6 in Table 2), (2) train it simultaneously on mono-speaker ASR, multi-speaker ASR with Sortformer speaker supervision, speech translation (e.g., English-to-German on CoVoST), and speech summarization, using only token-level cross-entropy loss for all tasks, and (3) compare against a baseline that uses PIL-based diarization supervision for the multi-speaker ASR task (requiring a separate loss head). The key measurement is whether Sortformer's sorted-token approach actually simplifies multi-task training—measured by overall task performance compared to single-task baselines, training stability, and whether the model can share representations across tasks without the diarization loss interfering with translation or summarization objectives. The paper's claim that Sortformer makes multi-speaker ASR training "functionally equivalent to standard mono-speaker ASR training" (Section 3.1) remains hypothetical without this evidence.

Evaluating Sort Loss as a pre-training objective for downstream speaker-related tasks beyond ASR. The paper's focus is on ASR integration, but the arrival-time-sorted speaker representations produced by Sortformer may be useful for other downstream tasks. A follow-up could: (1) freeze a pre-trained Sortformer and use its sorted speaker embeddings (the rows of P) as input features for: speaker identification (classifying which specific person is speaking, not just generic label assignment), speaker change detection (identifying transition points in the audio), overlap detection (identifying regions where multiple speakers are active), and emotion recognition per speaker; (2) compare against using PIL-trained EEND embeddings for the same tasks; and (3) measure whether the arrival-time-sorted representation provides a more consistent, temporally-structured input that improves downstream task performance. The hypothesis is that fixed speaker channel ordering provides a more stable representation for models that process extended conversations, since "speaker 0" always refers to the same semantic role (first speaker) across different training examples.

Replicating the integration pattern on non-NVIDIA architectures to test ecosystem independence. The paper's results are entirely within the NVIDIA NEST/Canary/NeMo ecosystem. A replication study using widely available open-source models—for example, using a WavLM-based diarization model (replacing NEST), a Whisper-based ASR model (replacing Canary), and the identical sinusoidal kernel integration—would test whether the approach generalizes beyond the specific encoder and decoder architectures used in the paper. The experiment would measure: (1) whether the Whisper encoder's representations are amenable to additive sinusoidal speaker kernel injection (i.e., whether speaker identity remains separable from phonetic content after addition), (2) whether the 0.78% runtime overhead scales similarly on different architectures, and (3) whether the Sort Loss training dynamic (competitive with PIL, complementary in Hybrid form) replicates with a different encoder backbone. Negative results (e.g., the sinusoidal kernel causing larger ASR degradation on Whisper than on Canary) would identify architectural constraints on the approach; positive results would substantially strengthen the paper's generalizability claims.

Practical Applications and Downstream Use Cases

Cost-efficient batch annotation of meeting and conversation corpora. Organizations with large archives of unannotated multi-speaker audio—meeting recordings, call center logs, legal depositions, medical consultations—face the expensive problem of generating speaker-attributed transcripts for downstream analysis (compliance monitoring, clinical documentation, legal discovery). The Sortformer-MS-Canary pipeline (System 6 in Table 2, achieving 26.71% cpWER on AMI-test and 21.45% on CH109) can be deployed as an automatic annotation system that produces speaker-labeled transcripts at scale. The 0.78% inference overhead (Section 5.4.3) means that adding speaker attribution to an existing ASR batch pipeline is nearly cost-free in compute terms—297.9 seconds without Sortformer becomes 300.2 seconds with it for the same workload. The 30% relative WER reduction on 2-mix audio (Table 3) means that on typical two-speaker conversations (the most common multi-speaker scenario in practice), the system produces substantially fewer speaker attribution errors than unsupervized multi-speaker ASR. For a corpus of 10,000 hours of two-speaker call center audio, the speaker attribution improvement directly translates to fewer manual corrections needed in the annotation pipeline, reducing human-in-the-loop costs proportionally. The frozen-Sortformer configuration (System 2) is the practical default here: pre-train the diarization model once on available multi-speaker data, freeze it, and integrate it with any domain-specific ASR fine-tuned on the target corpus.

Resource-efficient on-device multi-speaker transcription for mobile and edge applications. The 170M-parameter Sortformer-MS-Canary system (System 2 in Table 2), trained on a single NVIDIA RTX 6000 Ada GPU with 50K steps, demonstrates that competitive multi-speaker ASR is achievable with a model small enough for on-device deployment. The frozen Sortformer approach means the diarization model can be pre-trained centrally and distributed as a fixed component, with only the ASR model fine-tuned on device-specific data. The key practical benefit is the separation of concerns: the 123M Sortformer handles the speaker separation task (which benefits from diverse multi-speaker pre-training data), while the 170M Canary handles the transcription task (which can be domain-adapted). For a mobile meeting transcription app, this means the speaker diarization quality does not degrade when the ASR model is personalized to the user's voice and vocabulary, because Sortformer is frozen and unaffected by ASR fine-tuning—exactly the configuration that System 2 validates. The 0.78% runtime overhead means battery impact is negligible, and the 123M additional parameters are within the memory budget of modern flagship mobile devices.

Simplified integration of speaker attribution into existing mono-speaker ASR products. For organizations that already deploy mono-speaker ASR systems and want to add multi-speaker capabilities, Sortformer offers a minimal-disruption upgrade path. The SST format (Appendix A, Figure 5) requires only that training transcripts be augmented with sorted speaker tokens—no changes to the loss function, the decoder architecture, or the training loop. The sinusoidal kernel (Equation 14) requires adding one matrix multiplication after the encoder and one element-wise addition—a handful of lines of code in any deep learning framework. The frozen-Sortformer configuration means the diarization model is a pre-trained, off-the-shelf component that does not require re-training or hyperparameter tuning. For a product team that has invested heavily in optimizing their mono-speaker ASR training pipeline, Sortformer changes the integration problem from "redesign our loss function to handle permutation ambiguity" to "add speaker tokens to our transcripts and insert a pre-trained speaker encoder before our ASR decoder." The 4.77 absolute cpWER improvement on AMI-test from adding frozen Sortformer (System 1: 32.94% → System 2: 28.17%) suggests the benefit is substantial even without any joint optimization or domain adaptation of the diarization component.

When to Prefer This Method

The paper positions Sortformer (with Sort Loss or Hybrid Loss) as an alternative to two existing paradigms: PIL-based end-to-end diarization for stand-alone speaker diarization, and unsupervized or SOT-based multi-speaker ASR for speaker-attributed transcription. The choice depends on the target application, the expected number of speakers, and the deployment constraints:

• Prefer Sortformer with Hybrid Loss for stand-alone diarization when the diarization model needs to serve both as a stand-alone system and as a downstream supervision module for ASR integration. Hybrid Loss (14.76% DER on DIHARD3, Table 1) outperforms both pure PIL and pure Sort Loss, and the sorted outputs are compatible with SST-based ASR training. If the diarization outputs will never be used for ASR integration, pure PIL remains competitive (17.04% vs. 17.10% for Sort Loss on DIHARD3), and the additional α-tuning required by Hybrid Loss may not be justified. If the target domain involves frequent overlapping onsets or poor arrival-time estimation (as suggested by the CH109 results where Sort Loss alone degrades to 10.85% vs. PIL's 6.89%), Hybrid Loss is essential—the PIL component provides a safety net for ordering failures.

• Prefer a frozen Sortformer with word-level SST for multi-speaker ASR when the primary goal is adding speaker attribution to an existing mono-speaker ASR pipeline with minimal architectural changes and no specialized loss functions. System 2 in Table 2 (frozen Sortformer, word-level objectives, 170M parameters) achieves 28.17% cpWER on AMI-test and 22.22% on CH109—a 4.77–2.58 point improvement over the unsupervized baseline—without joint diarization-ASR fine-tuning and with only 0.78% inference overhead. This is the configuration for practitioners who want a drop-in speaker supervision module. If a larger ASR model is available (1B+ parameters), System 6 with adapter-based training and word-level SST provides the best overall cpWER (26.71% on AMI-test) while preserving mono-speaker ASR performance through frozen base weights. Do NOT fine-tune Sortformer jointly with the ASR model—System 3 demonstrates that joint optimization degrades cpWER (32.74% on AMI-test, nearly back to the unsupervized baseline).

• Prefer PIL-based diarization (EEND, AED-EEND) or cascaded systems when the target application routinely involves more than four speakers, since Sortformer's 4-speaker ceiling is a hard architectural constraint with no demonstrated extension path. EEND-EDA, EEND-GLA, and AED-EEND all report results on scenarios beyond four speakers, and cascaded systems (TS-VAD-based pipelines) are the established approach for dinner party and conference scenarios with 5–8+ participants. Sortformer's arrival-time ordering convention becomes less reliable as speaker count increases (as the paper acknowledges in Section 3.3), compounding the architectural limitation with a fundamental heuristic weakness.

• Prefer standard SOT or t-SOT approaches for streaming multi-speaker ASR, since Sortformer's arrival-time sorting requires knowing the temporal order of speaker onsets, which is undefined in a streaming context until all speakers have begun speaking. The paper explicitly defers streaming to future work (Section 6). The SOT family of methods (Kanda et al., 2020b; 2022a; Shi et al., 2024) was designed for streaming from the start and handles speaker attribution through serialized output formatting without requiring sorted speaker indices.