Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering

🎯 Pitch

A streaming diarization model that eliminates permutation resolution entirely by predicting speakers in arrival-time order, and a smart caching mechanism actually makes the online system outperform its offline counterpart on standard benchmarks—even at ultra-low 0.32 s latency.

1. Executive Summary

This paper introduces Streaming Sortformer, a streaming extension of the offline Sortformer speaker diarization model that preserves its core property of predicting speakers in arrival-time order. The key mechanism is the Arrival-Order Speaker Cache (AOSC) (a memory buffer that stores frame-level acoustic embeddings ordered by speaker index, dynamically selecting the highest-scoring frames for each speaker based on past predictions), which eliminates the explicit permutation resolution step required by conventional speaker-tracing buffers. The streaming system with AOSC fine-tuning achieves state-of-the-art diarization error rates on benchmark datasets—notably matching or outperforming the offline Sortformer on DIHARD III Eval (19.02% vs. 21.39% at 10 s latency with post-processing) and CALLHOME (10.09% vs. 11.26%)—demonstrating that the approach stays competitive even at 0.32 s latency (19.32% on DIHARD III), establishing that arrival-time ordering enables robust streaming diarization without attractors or permutation resolution operations, though the system is primarily designed for up to four speakers and shows expected degradation on 5+ speaker scenarios.

2. Context and Motivation

The Core Problem: Making Speaker Diarization Work in Real-Time Without Losing Track of Who's Speaking When

The fundamental challenge this paper tackles is deceptively simple to state but devilishly hard to solve: how do you build a speaker diarization system that processes audio as it arrives, identifies speakers in real-time, and correctly tracks the identity of each speaker across an entire conversation without ever confusing who is who? Speaker diarization — the task of answering "who spoke when?" in a multi-speaker audio recording — is a prerequisite for virtually every downstream speech application that involves more than one person talking. Without accurate diarization, transcription becomes an undifferentiated stream of words with no indication of which speaker uttered which sentence, making it useless for meeting notes, legal depositions, medical dictations, or broadcast subtitling.

The paper opens with a pointed observation about the state of the field: while automatic speech recognition (ASR) has matured to the point where offline and online (streaming) systems perform comparably, the gap between offline and streaming speaker diarization remains dramatically wider. This is not merely a matter of engineering polish — it reflects fundamental algorithmic challenges that emerge when the system cannot look at the entire conversation before making decisions about speaker identity.

Why This Gap Matters: Real-World Applications Demand Streaming

The paper emphasizes a set of applications that fundamentally require streaming operation:

Live captioning of broadcasts, lectures, or parliamentary proceedings, where captions must appear with minimal delay and correctly attribute speech to different speakers as they alternate.
Virtual meetings (Zoom, Teams, etc.), where real-time speaker labels enable features like automatic transcription with speaker tags, speaker-specific volume controls, and real-time meeting analytics.
Conversational analytics for call centers or sales calls, where identifying who is speaking and when enables downstream analysis of interaction patterns, talk-time ratios, and customer sentiment by agent versus customer.
Human-robot interaction, where a robot must identify which human is addressing it and respond appropriately in real-time, without waiting for the interaction to end before processing.

In all of these scenarios, latency is not optional — the system must produce speaker labels as the audio arrives, with delays measured in seconds or fractions of a second. An offline system that requires the entire recording before producing output is simply not deployable.

Beyond latency, a subtler but equally important challenge is scale: real conversations can stretch for hours (think of a multi-hour meeting, a courtroom proceeding, or a dinner party recording). Offline systems that process the entire audio at once hit fundamental limits from self-attention mechanisms whose computational and memory costs grow quadratically with sequence length. As the paper notes (end of Section 1):

"Additionally, its ability to process long audio recordings is constrained by the maximum input length that the self-attention mechanism can handle, limiting its scalability for extended conversations."

This means streaming is not just about latency — it is also the key to handling arbitrarily long recordings that would otherwise exceed hardware memory or simply become computationally infeasible.

Prior Approaches: A Landscape of Progress and Persistent Limitations

The paper situates itself within a rich lineage of end-to-end neural diarization (EEND) systems, which have progressively displaced traditional modular approaches (clustering of speaker embeddings) over the past several years. Understanding this lineage is essential because each prior approach addresses one piece of the puzzle while leaving others unsolved.

First Generation: Fixed-Speaker EEND (Fujita et al., 2019)

The original EEND formulation framed diarization as a straightforward multi-class classification problem: for each time frame, predict which of $S$ speakers is active. The model outputs $S$ sigmoid activations — one per speaker — and is trained with permutation-invariant training (PIT) loss [1, 2, 3]. This elegantly handled overlapping speech (multiple speakers active in the same frame) without any explicit overlap detection module.

However, this formulation had a crippling limitation: the number of output speakers $S$ was fixed by the model architecture. A model trained for 4 speakers could not handle a 5-person meeting, and a model trained for 10 speakers wasted capacity on 2-speaker phone calls. Real conversations have variable and unknown numbers of speakers, making fixed-output architectures impractical for general deployment.

Second Generation: Attractor-Based EEND (Horiguchi et al., 2020–2022)

To handle variable speaker counts, Horiguchi et al. introduced EEND with encoder-decoder attractors (EEND-EDA) [6, 7]. The key idea: rather than fixing the output dimension, the model uses an LSTM encoder-decoder to generate "attractor" vectors — one per speaker — from the input features themselves. These attractors serve as speaker-specific queries that are used to compute per-speaker activations. The model can generate as many attractors as there are speakers in the recording by predicting a stop token.

This was a major advance, and the authors refined it further with global and local attractors [8], enabling diarization for an "unlimited" number of speakers. However, the attractor mechanism introduces architectural complexity — an additional LSTM encoder-decoder on top of the core diarization model — and requires the model to learn to count speakers, which is itself a challenging subtask.

Third Generation: Streaming Adaptations (Xue et al., 2021; Liang et al., 2024)

The transition from offline to streaming introduced a new set of challenges. The block-wise approach (BW-EDA-EEND) [10] simply processes audio in sequential chunks and outputs per-chunk predictions. The fundamental problem is immediately apparent: speaker permutation invariance across chunks. Within each chunk, the model assigns speaker labels arbitrarily (Speaker A in chunk 1 might correspond to output dimension 1, but the same Speaker A in chunk 2 might correspond to output dimension 3 because the model has no inherent concept of speaker identity across chunks). Without resolving these permutations, the output is incoherent — the same speaker gets assigned different labels in different time segments.

The speaker-tracing buffer (STB) [11, 12] was the key innovation to address this. The STB stores a buffer of past audio features along with their predictions and, at each step, evaluates all possible permutations of current-chunk predictions against the buffer predictions, selecting the permutation that maximizes correlation. Equations (3)–(6) in the paper formalize this: the STB function produces a permuted buffer prediction $\mathbf{P}_n^{buf}$ and updated buffer $\mathbf{B}_n$ , and a permutation $\psi$ is chosen via:

$\psi = \operatorname*{argmax}_{\phi \in \text{perm}(S)} \text{CC}(\mathbf{P}_n^{buf}, \phi(\mathbf{\widehat{P}}_n^{buf}))$

where CC is the correlation coefficient. This works but has significant drawbacks:

Computational cost: evaluating all $S!$ permutations becomes expensive as the speaker count grows (though practical systems typically evaluate only a subset).
Buffer management complexity: the buffer must store and update both audio features and predictions, and the correlation maximization step requires careful implementation.
Permutation ambiguity: when speakers have similar activity patterns or when the buffer contains mostly silence, correlation-based matching can fail, leading to speaker identity swaps.

More recent systems — FS-EEND [13] and its improved version LS-EEND [14] — moved to non-autoregressive self-attention-based architectures and achieved state-of-the-art online diarization performance. These systems also use speaker appearance order to resolve permutations, but they still rely on attractors and explicit permutation resolution operations.

The Sortformer Innovation and Its Offline Limitation

Most recently, Sortformer [18] introduced a fundamentally different approach. Rather than using attractors or post-hoc permutation resolution, Sortformer is trained with a novel loss function — Sort Loss — that forces the model to output speakers in arrival-time order. The first speaker to appear in the recording is always assigned to output dimension 1, the second speaker to dimension 2, and so on. This is a hybrid loss: Sort Loss (binary cross-entropy computed over sorted target labels) combined with conventional permutation-invariant loss. The architecture itself is relatively simple — a self-supervised pretrained NEST encoder [19] based on the Fast-Conformer architecture [20], followed by a stack of Transformer encoder layers, outputting four sigmoid activations.

The arrival-time ordering property is the key insight that this paper builds upon. It means that the model's output dimensions have a stable, semantically meaningful ordering across the entire recording — Speaker 1 always refers to the first-arriving speaker, Speaker 2 to the second-arriving, etc. This is in stark contrast to standard EEND, where output dimension 1 in one chunk has no guaranteed relationship to output dimension 1 in another chunk.

However, the original Sortformer has two critical limitations that this paper explicitly addresses:

It is an offline model that relies on full-length self-attention over the entire input, making it unsuitable for streaming applications where audio arrives incrementally.
It cannot process long recordings beyond the maximum input length of its self-attention mechanism, limiting its scalability for extended conversations.

Where Prior Approaches Fall Short: The Missing Pieces

To understand why this paper's approach is significant, we need to identify exactly what gaps remain in the existing literature at the time of this work:

No streaming diarization system achieves both simplicity and arrival-time ordering. While FS-EEND and LS-EEND [13, 14] have pushed streaming performance to state-of-the-art levels, they retain significant architectural complexity: they require attractor mechanisms (self-attention-based, but still attractors) and explicit permutation resolution operations. Their use of speaker appearance order is a post-hoc arrangement, not an inherent property of the model's training objective. A system that natively predicts speakers in arrival-time order — and therefore eliminates attractors and permutation resolution entirely — would be simpler, potentially more robust, and easier to integrate into multi-task architectures.

Speaker-tracing buffers require explicit permutation resolution. The STB approach [11, 12] works, but it adds computational overhead (correlation computations across permutations) and a potential failure mode (incorrect permutation matching) that is entirely separate from the core diarization task. Every chunk boundary introduces a decision point where the system can make an irreversible error — swapping two speakers — that propagates through the rest of the recording.

Offline Sortformer handles variable-length recordings poorly. Even ignoring the streaming requirement, the self-attention mechanism in offline Sortformer imposes a hard maximum input length. For recordings that exceed this length (long meetings, dinner party recordings), the model must either truncate (losing information) or use some form of chunked processing (which reintroduces the permutation problem across chunks). The paper explicitly notes this:

"the offline Sortformer's underperformance on long recordings due to a mismatch with 90-second training samples"

The performance gap between offline and streaming diarization remains large. The paper opens with this observation, and it is worth quantifying: Table 1 shows that existing streaming systems achieve DERs of 19–25% on DIHARD III Eval (a challenging multi-domain dataset), while offline systems are in the 14–22% range. For comparison, the offline-offline gap in ASR is often a few percent relative — here we see absolute differences of 5–10 percentage points in some conditions. This gap is what the paper aims to narrow.

How This Paper Positions Itself

The paper's framing is explicit and specific: it takes the Sortformer architecture (with its arrival-time ordering property) and asks, "Can we make this streaming in a way that preserves — and leverages — its key advantage over EEND-based systems?"

The answer involves a deliberate design choice that distinguishes this work from prior streaming EEND systems. Rather than adding attractors (like [8, 13, 14]) or building a complex permutation resolution mechanism (like [12]), the paper proposes an Arrival-Order Speaker Cache (AOSC) that:

Stores acoustic embeddings (from the NEST pre-encoder) rather than just predictions, giving the model richer information about past speakers.
Orders embeddings by speaker index, which corresponds to arrival-time order due to Sortformer's training, so no permutation resolution is needed — the ordering is architecturally enforced.
Dynamically selects which frames to keep based on the model's past prediction scores, ensuring that the most informative frames for each speaker are preserved while stale or low-confidence frames are evicted.

The key claim, stated explicitly in the introduction, is that this approach "achieves superior performance without relying on self-attention attractors [13, 14], local or global attractors [8], or permutation resolution operations for speaker-tracing buffers [12, 8]." In other words, the arrival-time ordering property of Sortformer makes the AOSC sufficient — the ordering inherent in the model's predictions eliminates the need for the additional machinery that prior streaming systems required.

The paper also acknowledges its relationship to the speaker-tracing buffer concept ("our approach builds on the STB concept" — Section 1) while distinguishing AOSC as a specialized design that aligns with Sortformer's core idea. It is not claiming to invent the buffer concept from scratch, but rather to show that the buffer can be dramatically simplified when paired with a model that natively orders speakers by arrival time.

Finally, the paper situates itself within a broader vision: integrating streaming speaker diarization into multi-speaker speech processing tasks like ASR, speech translation, and summarization. Sortformer [18] was originally motivated by the goal of seamless diarization-ASR integration, and this streaming extension is a step toward making that integration work in real-time. The conclusion explicitly states this ambition:

"We aim to integrate the streaming Sortformer diarizer into various multi-speaker speech processing tasks, including ASR, speech translation, and summarization, enabling real-world applications such as broadcasting and meeting transcription."

3. Technical Approach

3.1 Reader Orientation

The Streaming Sortformer is a real-time speaker diarization system that processes audio as it arrives and labels each time frame with which speakers are active, producing output with minimal latency. It solves the problem of maintaining consistent speaker identities across sequential chunks of streaming audio — the core challenge where standard chunk-wise processing would arbitrarily reorder which speaker corresponds to which output dimension at each chunk boundary. The solution takes a fundamentally different "shape" from prior streaming diarization systems: rather than adding explicit permutation resolution machinery (correlation matching) or attractor mechanisms on top of the diarization model, it leverages the arrival-time ordering property of the Sortformer architecture — the model is trained to always output the first speaker to appear on dimension 1, the second on dimension 2, and so on — and pairs this with a dynamically managed speaker cache that stores acoustic embeddings in this same arrival-order indexing, eliminating the permutation problem entirely by architectural design rather than post-hoc correction.

3.2 Big-Picture Architecture (Diagram in Words)

The Streaming Sortformer system consists of five major components that interact at each processing step:

Input Buffer — receives the raw audio for the current chunk plus a small amount of right-context (future frames) to improve prediction accuracy at chunk boundaries. This is the "what the system can currently hear" component.
FIFO Queue — a fixed-length first-in, first-out buffer holding several preceding chunks of raw audio features (Mel-spectrograms), providing broader temporal context for the current chunk without requiring full-length self-attention over the entire recording history. When frames exit the FIFO, they are processed by the AOSC update mechanism.
Arrival-Order Speaker Cache (AOSC) — the central novel component. It stores a compressed representation of previously observed speakers, consisting of frame-level acoustic embeddings extracted from the NEST pre-encoder (not raw audio features and not just predictions). These embeddings are ordered by speaker index (first-arriving speaker first, second-arriving second, etc.) and are dynamically pruned to retain only the highest-scoring frames per speaker based on the model's past confidence scores. This cache is concatenated with the FIFO queue content and the input buffer to form the full input sequence for the current step.
Sortformer Model — the same architecture as offline Sortformer: a NEST pre-encoder (Fast-Conformer-based self-supervised model) followed by a stack of Transformer encoder layers, outputting four sigmoid activations (one per potential speaker). It processes the concatenated [speaker_cache, fifo_queue, input_buffer] sequence and produces per-frame speaker activity predictions for the current chunk.
Cache Update Mechanism — a deterministic algorithm (not a learned component) that takes the model's predictions for the current chunk, computes speaker scores per frame, and selects which frames' NEST embeddings to keep in the AOSC for future steps. It enforces that each known speaker retains a minimum number of frames, prioritizes recent frames, and inserts silence embeddings as speaker separators.

The information flow per step is: audio arrives → appended to input buffer → [AOSC embeddings, FIFO queue features, input buffer features] are concatenated → fed into Sortformer → produces predictions for current chunk → cache update mechanism selects which frames to retain in AOSC → FIFO queue advances, evicted frames are compressed into AOSC → next step.

3.3 Roadmap for the Deep Dive

First, the Arrival-Order Speaker Cache (AOSC) — what it stores, how it differs from a standard speaker-tracing buffer, and why the arrival-order ordering eliminates permutation resolution. This is the core concept the rest of the system depends on.
Second, the cache update mechanism — the step-by-step algorithm (Steps 1–7 from Section 3.2) that dynamically selects which frames to retain, including how speaker scores are computed, how silence is handled, and how minimum speaker representation is enforced. This is where the "dynamic" intelligence of the cache lives.
Third, the streaming inference procedure with FIFO queue — how the FIFO queue provides temporal context, how it interacts with the cache update cycle, and how the system produces final per-chunk predictions without needing a separate permutation step. This completes the equation-level description from Section 3.3 (Equations 7–9).
Fourth, the training procedure — how the offline Sortformer checkpoint is fine-tuned with the AOSC mechanism, including the sequential windowing strategy, the speaker permutation augmentation, the right-context limitation, and all hyperparameter values. This explains how the model learns to work with the cache.
Fifth, design choice analysis — why AOSC stores NEST embeddings rather than raw features or predictions, why the update mechanism uses log-likelihood-style scoring rather than raw sigmoid values, why silence embeddings are appended per-speaker, and why the approach can eliminate attractors. This ties the technical details back to the paper's central claims.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a streaming adaptation paper whose core idea is that the arrival-time ordering property of Sortformer enables a dramatically simpler streaming architecture — an ordered speaker cache without permutation resolution — that matches or exceeds the performance of more complex attractor-based streaming systems.

Arrival-Order Speaker Cache (AOSC): What It Stores and Why Ordering Matters

The AOSC is a fixed-length memory buffer that stores frame-level acoustic embeddings extracted from the NEST pre-encoder module — the intermediate representations produced after the convolutional downsampling layers but before the Transformer encoder stack. The paper designates this buffer as $\mathbf{B}_n$ , where $n$ indexes the current processing step.

What is stored in the cache. Unlike the speaker-tracing buffer (STB) from prior work [11, 12], which stores raw audio features and/or predicted speaker activity probabilities, the AOSC stores the NEST pre-encoder outputs — rich, learned acoustic embeddings that encode speaker identity information. This is a critical design choice: these embeddings represent what the NEST model has learned about "who is speaking" in each frame, compressed into a representation optimized for the downstream Transformer layers. By caching these embeddings rather than raw Mel-spectrogram features, the AOSC provides the Sortformer model with a semantically meaningful representation of past speakers — one that the model was trained to interpret — rather than raw acoustic data that requires additional processing to extract speaker-relevant features.

How embeddings are ordered. The embeddings in the cache are arranged by speaker index, where speaker index corresponds to arrival-time order. Specifically: the embeddings for the first speaker to appear in the recording occupy the earliest positions in the cache, followed by the embeddings for the second-arriving speaker, then the third, and finally the fourth (since Sortformer is designed for up to four speakers). This ordering is not determined by a separate matching algorithm — it is architecturally enforced by Sortformer's training objective. Because Sortformer was trained with Sort Loss to output speaker 1 as the first-arriving speaker, speaker 2 as the second-arriving, etc., the cache can simply store embeddings in output-index order and the model will correctly interpret speaker index 1 as "the first person who started talking in this conversation." There is no separate step that reorders the cache — the ordering is inherent in the model's prediction structure.

Why this eliminates permutation resolution. In a standard EEND-based streaming system with STB, each chunk's output has speakers assigned to arbitrary output dimensions. The system must compute the correlation between current predictions and buffer predictions across all possible speaker permutations (Equation 5 in the paper) to determine which output dimension in the current chunk corresponds to which speaker in the history. This is an explicit optimization step — $\psi = \operatorname*{argmax}_{\phi \in \text{perm}(S)} \text{CC}(\mathbf{P}_n^{buf}, \phi(\mathbf{\widehat{P}}_n^{buf}))$ — that must be performed at every chunk boundary. In Streaming Sortformer, this step disappears entirely. Because the model always predicts speakers in arrival-time order, and the cache stores embeddings in this same order, the output dimension indexing is consistent across chunks by construction. Dimension 1 always means "first-arriving speaker," whether that speaker is currently active or not. This is the central simplification that distinguishes Streaming Sortformer from prior streaming diarization systems.

Cache size and compression. The AOSC has a maximum capacity of $M$ frames (188 frames in the reported configurations, corresponding to 15 seconds of audio at the 80 ms effective frame rate after NEST's 8× downsampling). When the total number of frames across all observed speakers exceeds $M$ , the cache update mechanism must compress the representation — selecting which $M$ frames to retain and which to discard. The compression algorithm is the subject of the next section. The key point here is that the cache size is fixed regardless of how many speakers are active or how long the conversation has been running, making the system's memory and computational requirements bounded and predictable.

Comparison with speaker-tracing buffer. The STB [11, 12] produces $\mathbf{P}_n^{buf}$ — a sequence of prediction probabilities corresponding to the buffer — and updates $\mathbf{B}_n$ with raw audio features. The AOSC replaces prediction-storing with embedding-storing: rather than caching the model's outputs (which are low-dimensional — just 4 sigmoid values per frame) and using them for correlation matching, it caches the high-dimensional NEST embeddings (which contain rich speaker-discriminative information) and relies on the model's arrival-order training to maintain consistent indexing. This shift from "cache predictions and match them" to "cache embeddings in a fixed order" is the architectural insight that the paper's title highlights.

Cache Update Mechanism: The Step-by-Step Algorithm

The cache update mechanism — denoted $\text{AOSC}()$ in Equation 8 — is the algorithmic core that determines which frames' NEST embeddings are retained in the speaker cache and which are discarded when the cache reaches capacity. It is a deterministic, non-learned procedure that operates on the model's own predictions to make retention decisions. The paper provides a 7-step description in Section 3.2.

Inputs and outputs of the update function. The AOSC update function (Equation 8) takes two inputs:

$[\mathbf{P}_{n-1}^{buf}, \mathbf{P}_{n-1}]$ — the concatenation of previous buffer predictions and the previous chunk's predictions. These are the model's sigmoid outputs (speaker activity probabilities) for every frame currently in the system's memory.
$[\mathbf{B}_{n-1}, \mathbf{C}_{n-1}]$ — the concatenation of the previous cache embeddings and the previous chunk's raw features (before NEST encoding). Wait — correction needed: careful reading shows that $\mathbf{B}_{n-1}$ already contains NEST embeddings (stored from earlier steps), and $\mathbf{C}_{n-1}$ is the raw Mel-spectrogram features of the previous chunk. The update function needs to process these raw features through NEST to get embeddings for compression. Actually, the paper's notation is slightly ambiguous here — but the clearer interpretation from the algorithm description is: the update function receives the predictions for all frames under consideration and the NEST embeddings for those frames (since the cache stores NEST embeddings), selects which frames to keep, and returns the compressed sequence.

The function outputs:

$\mathbf{P}_n^{buf}$ — the predictions corresponding to the frames kept in the cache (needed for future scoring during subsequent updates).
$\mathbf{B}_n$ — the updated speaker cache containing the selected NEST embeddings, ordered by speaker index.

Step 1: Computing speaker scores. For each frame in the input sequence, the mechanism computes a speaker score $S_i$ for each speaker $i$ :

$S_i = \log P_i + \sum_{j \neq i} \log(1 - P_j)$

where $P_i \in [0, 1]$ is the model's predicted sigmoid activation for speaker $i$ at that frame, and $P_j$ are the predictions for all other speakers $j \neq i$ .

What it computes: For a given frame and a given speaker $i$ , this score is the log-likelihood of the hypothesis "speaker $i$ is active and all other speakers are inactive." The first term $\log P_i$ is high when the model is confident that speaker $i$ is speaking; the second term $\sum_{j \neq i} \log(1 - P_j)$ penalizes frames where other speakers are also predicted to be active, since $\log(1 - P_j)$ becomes very negative when $P_j$ is close to 1. Frames that are clearly single-speaker (one $P_i$ near 1, all others near 0) receive high scores for that speaker; frames with overlapping speech (multiple $P_i$ near 1) receive lower scores because the sum of penalty terms drags them down; frames with all $P_i$ near 0.5 receive intermediate scores.

Why this form: This is a log-likelihood ratio formulation. The alternative — simply using raw sigmoid values $P_i$ — would not penalize overlapping speech: a frame where both speaker 1 and speaker 2 have $P = 0.9$ would get the same score as a frame where only speaker 1 has $P = 0.9$ . The log-likelihood formulation explicitly downweights ambiguous frames (overlap, uncertainty) in favor of frames that provide "clean" exemplars of each speaker — exactly what you want for building a speaker representation cache. This is the same formulation used in speaker identification scoring and reflects a deliberate choice to prioritize discriminative frames for each speaker.

Step 2: Detecting and handling silence. The mechanism identifies frames where the model assigns low probability to all speakers — these are silence frames. The paper does not specify an exact threshold, but the description implies that frames where $P_i < 0.5$ for all $i$ (checked in Step 3) are considered non-speech. For these silence frames, the mechanism computes an average silence embedding — the mean of the NEST embeddings over all detected silence frames — which is stored and used later as a filler embedding.

Why this matters: Silence is a natural "punctuation" in conversations — it marks speaker transitions. By explicitly identifying and storing a silence embedding, the cache can insert silence tokens between different speakers' embeddings, signaling to the model that there is a boundary between one speaker stopping and another starting. Without this, the concatenation of speaker embeddings would appear as continuous speech with abrupt speaker changes, making it harder for the model to track transitions.

Step 3: Disabling non-speech scores. For any frame where $P_i < 0.5$ , the corresponding score $S_i$ is set to $-\infty$ . This means: if the model is not even 50% confident that speaker $i$ is active in this frame, do not consider this frame as a candidate for speaker $i$ 's cache representation. This is a hard threshold — frames where the model thinks speaker $i$ is probably absent are completely excluded from being retained for that speaker.

Why 0.5: The sigmoid midpoint is the natural decision boundary — below 0.5, the model implicitly predicts "this speaker is more likely absent than present." Using a hard $-\infty$ rather than a gradual penalty ensures that low-confidence frames never displace higher-confidence frames in the cache, regardless of how the score-boosting in later steps works.

Step 4: Prioritizing recent frames. For frames that correspond to newly added embeddings (embeddings being considered for inclusion in this update step, not already in the cache), their scores are increased by $\delta > 0$ (set to $\delta = 0.05$ in the training configuration).

Why this matters: This is a recency bias. Without this term, a very clean exemplar from early in the conversation could permanently occupy cache space, even if the speaker's voice characteristics have changed (e.g., due to Lombard effect, distance from microphone, or emotional state). The $\delta = 0.05$ bonus is small — it gives a slight edge to recent frames — but over many update cycles, it ensures that the cache gradually refreshes, with older frames being evicted in favor of comparable-quality newer frames. This is an explicit design choice: the cache should track the current state of each speaker, not just store the historically best exemplars.

Step 5: Ensuring minimum speaker representation. For each speaker, the $K$ highest-scoring frames have their scores increased by $\Delta > 0$ . The paper's training configuration applies this step twice with different parameters:

Strong boosting: $K = 33$ frames per speaker boosted by $\Delta = -2\log(0.5)$ . The value $-2\log(0.5) = 2\log 2 \approx 1.386$ — a substantial score increase. This ensures that at least 33 frames per speaker are strongly favored for retention.
Weak boosting: $K = 66$ frames per speaker boosted by $\Delta = -\log(0.5) = \log 2 \approx 0.693$ — a moderate boost applied to a larger set of frames.

After both boosting steps, effectively at least 33 frames per speaker are nearly guaranteed retention (unless displaced by other speakers' strongly-boosted frames), and an additional 33 frames (up to 66 total) receive a moderate advantage.

Why two-tier boosting: The strong boost ( $K = 33$ ) guarantees a minimum speaker representation — even a speaker who has been silent for a long time retains at least 33 frames (approximately 2.64 seconds at 80 ms frame rate) in the cache, preventing the model from "forgetting" about them entirely. The weak boost ( $K = 66$ ) provides a larger buffer zone — if cache space permits, up to 66 frames (approximately 5.28 seconds) per speaker are favored. This two-tier design reflects a practical trade-off: you need enough frames per speaker for the model to recognize them when they resume speaking, but cache space is limited, so beyond a minimum you want to prioritize the highest-quality exemplars across all speakers rather than forcing equal representation.

Why these specific numbers: The values $K = 33$ and $K = 66$ correspond to approximately 2.6 and 5.3 seconds of speech respectively (at 80 ms per frame). For speaker diarization, this is roughly the minimum amount of speech needed to characterize a speaker's voice — a few seconds of clean speech is sufficient for speaker discrimination, while less than a second would be too short. The boosting magnitudes $-2\log 0.5$ and $-\log 0.5$ are chosen to be large relative to the range of typical $S_i$ values, ensuring the boosting dominates over raw score differences for the top frames.

Step 6: Appending silence embeddings. For each speaker, $A$ scores of $+\infty$ are appended, with these scores corresponding to the average silence embedding computed in Step 2. With $A = 3$ in the reported configuration, this means three "virtual frames" of silence are inserted after each speaker's retained embeddings.

Why this matters: This creates an explicit speaker transition marker in the cache. When the concatenated cache is fed into the model, it sees: [speaker 1 embeddings] → [silence] → [speaker 2 embeddings] → [silence] → ... The silence embeddings signal to the Transformer attention layers that there is a discontinuity — speaker 1's segment has ended, and speaker 2's segment is a separate speech event, not a continuation. This is crucial for the model to correctly predict speaker transitions at chunk boundaries. The $A = 3$ frames of silence correspond to 240 ms — long enough to be a perceptible pause, short enough not to waste cache capacity.

Step 7: Selecting and ordering the final cache. All scores (original $S_i$ , boosted scores, and $+\infty$ silence scores) are concatenated across all speakers. The $M$ highest-scoring frames are selected, and their corresponding NEST embeddings are returned as the updated cache $\mathbf{B}_n$ , while preserving their original temporal order within each speaker. For frames that received $+\infty$ or $-\infty$ scores, the average silence embedding is used instead of the original NEST embedding.

What "preserving order" means here: The returned cache sequence is: [speaker 1's retained frames in temporal order] → [speaker 1's silence frames] → [speaker 2's retained frames in temporal order] → [speaker 2's silence frames] → ... This is the arrival-order ordering: speaker 1 (first to arrive) first, then speaker 2 (second to arrive), etc. The temporal ordering within each speaker's segment is preserved so the model can observe the natural evolution of that speaker's voice.

Dynamic cache sizing per speaker. An important consequence of the score-based selection: the number of frames retained per speaker is not fixed — it is determined dynamically by the relative scores. A speaker who has been consistently active and whose frames have high scores may occupy a larger fraction of the cache; a speaker who has been silent for an extended period will have mostly low-scoring frames (which get boosted by Step 5) and may occupy only the minimum $K$ frames. This dynamic allocation is a deliberate design choice — it allows the cache to adapt to the conversation's speaker activity patterns without requiring a separate speaker-counting or activity-tracking module.

Why the update is applied when the FIFO queue evicts frames. The paper notes (Section 3.3) that the AOSC update is performed not after every individual chunk but when frames exit the FIFO queue. This means the update period is larger than the chunk size — the FIFO queue accumulates several chunks before older frames are pushed out and compressed into the cache. This serves two purposes: (1) it is computationally more efficient (fewer updates), and (2) it allows the cache update to observe a longer segment of audio, making better decisions about which frames are most representative of each speaker.

Streaming Inference Procedure with FIFO Queue

The full streaming inference procedure is formalized in Equations 7–9 and illustrated in Figure 2. Here we walk through the complete sequence of operations that occurs at each processing step.

Initialization (Equation 7). For the very first chunk $\mathbf{C}_0$ , there is no history — the speaker cache is empty ( $\mathbf{B}_0 = \emptyset$ ). The Sortformer model processes only the current chunk:

$\mathbf{P}_0 = \text{Sortformer}(\mathbf{C}_0)$

where $\mathbf{C}_0$ is the first chunk of input Mel-spectrogram features, $\text{Sortformer}()$ is the full model (NEST pre-encoder + Transformer encoder stack), and $\mathbf{P}_0 \in \mathbb{R}^{c \times 4}$ is the model's prediction — four sigmoid values per frame for up to four speakers, ordered by arrival time. This is a standard forward pass with no cache augmentation.

Subsequent steps: cache update (Equation 8). For all chunks $n = 1, 2, \ldots$ , the system first updates the AOSC using the previous chunk's predictions and features:

$\mathbf{P}_n^{buf}, \mathbf{B}_n = \text{AOSC}([\mathbf{P}_{n-1}^{buf}, \mathbf{P}_{n-1}], [\mathbf{B}_{n-1}, \mathbf{C}_{n-1}])$

where:

$[\mathbf{P}_{n-1}^{buf}, \mathbf{P}_{n-1}]$ is the concatenation of all previous cache predictions and the predictions from chunk $n-1$ . This gives the update mechanism the model's confidence scores for every frame that is a candidate for inclusion in the cache.
$[\mathbf{B}_{n-1}, \mathbf{C}_{n-1}]$ is the concatenation of the previous cache embeddings and the raw features of chunk $n-1$ . Wait — this needs clarification. The notation $\mathbf{C}_{n-1}$ specifically refers to raw Mel-spectrogram features, but the cache stores NEST embeddings. The actual implementation must first pass $\mathbf{C}_{n-1}$ through the NEST pre-encoder to get embeddings, then concatenate with $\mathbf{B}_{n-1}$ , then apply the 7-step selection algorithm on the pooled set. The paper's notation is slightly compressed here — the $\text{AOSC}()$ function internally handles the NEST encoding.
The output $\mathbf{P}_n^{buf}$ is the sequence of predictions corresponding to the frames that were retained in the cache, preserving the alignment between embeddings and their scores for the next update cycle.
The output $\mathbf{B}_n$ is the updated cache containing $M$ (or fewer, if total frames are below capacity) NEST embeddings ordered by speaker index.

What the cache update produces operationally. After this step, $\mathbf{B}_n$ contains a compressed history of all speakers observed up to and including chunk $n-1$ . The first segment of $\mathbf{B}_n$ represents the first-arriving speaker's most characteristic frames, followed by silence embeddings, then the second-arriving speaker's frames, and so on. The total length of $\mathbf{B}_n$ is at most $M$ frames (188 in the reported configuration), regardless of how much audio has been processed.

Subsequent steps: prediction with cache context (Equation 9). The system now forms the full input for the current chunk by concatenating the cache (context from the past) with the current chunk features:

$[\_, \mathbf{P}_n] = \text{Sortformer}([\mathbf{B}_n, \mathbf{C}_n])$

where:

$[\mathbf{B}_n, \mathbf{C}_n]$ is the concatenation of the speaker cache embeddings and the current chunk's raw Mel-spectrogram features. This forms a single sequence: [NEST_embeddings_of_past_speakers, mel_features_of_current_chunk].
$\text{Sortformer}()$ processes this concatenated sequence. The NEST pre-encoder transforms $\mathbf{C}_n$ into embeddings (the cache $\mathbf{B}_n$ already contains NEST embeddings, so they pass through the Transformer layers directly — they do not go through the pre-encoder again).
The output is split into two parts: $\_$ (discarded) represents the model's predictions for the cache portion of the input (these are not needed, since the cache frames are in the past and their diarization decisions were already made), and $\mathbf{P}_n$ represents the predictions for the current chunk.

Why the cache portion predictions are discarded. The model produces predictions for every frame in the input sequence, including the cache frames. These cache-frame predictions are "backward-looking" — they re-predict speaker activity for frames that have already been diarized. The system discards these because (a) they would overwrite already-committed decisions, and (b) the cache frames exist only to provide context, not to be re-labeled. Only $\mathbf{P}_n$ — the predictions for the new, not-yet-diarized chunk — are kept and added to the output stream.

The critical absence of a permutation step. Compare this to the STB procedure (Equations 3–6 in the paper), which requires the explicit permutation resolution:

$\psi = \operatorname*{argmax}_{\phi \in \text{perm}(S)} \text{CC}(\mathbf{P}_n^{buf}, \phi(\mathbf{\widehat{P}}_n^{buf}))$

and the subsequent permutation application $\mathbf{P}_n = \psi(\mathbf{\widehat{P}}_n)$ . In Streaming Sortformer, Equation 9 produces $\mathbf{P}_n$ directly — there is no $\psi$ , no $\text{CC}()$ correlation computation, no $\operatorname*{argmax}$ over permutations. The prediction $\mathbf{P}_n$ is already in the correct speaker order because the model was trained to output speakers in arrival-time order, and the cache provides the same ordering. The permutation resolution is baked into the architecture, not computed as a post-processing step.

Integration with the FIFO queue. Figure 1 illustrates that the FIFO queue sits alongside the AOSC and the input buffer. The FIFO queue stores raw Mel-spectrogram features from several preceding chunks, organized as a first-in, first-out buffer. When forming the input for the current step, the system concatenates [AOSC_embeddings, FIFO_queue_features, input_buffer_features]. The FIFO queue provides an intermediate temporal context — more than just the AOSC's compressed history but less than the full recording. The AOSC provides long-term speaker identity memory; the FIFO queue provides short-term acoustic context (prosody, room acoustics, recent word-level patterns) that helps the model make better predictions at chunk boundaries.

Why the FIFO queue matters for the update cycle. As noted in Section 3.3, the AOSC update is triggered when frames exit the FIFO queue, not after every individual chunk. The FIFO queue acts as a staging area: frames first enter the input buffer, then move to the FIFO queue (providing context for subsequent chunks), and finally — when they reach the end of the FIFO and are evicted — they are processed by the AOSC update mechanism. This means the cache update operates on frames that are already "in the past" by several chunk periods, allowing the update mechanism to observe a longer window of each speaker's activity before deciding which frames to retain. The update period and FIFO queue size are linked — with a 144-frame update period and 188-frame FIFO queue in the 1.04-second latency configuration (Table 2), the system updates the cache roughly once every 144 frames (~11.5 seconds of audio).

Latency components. The paper distinguishes between:

Input buffer latency — the amount of right-context (future frames) included in the input buffer beyond the current chunk. This is what Table 2 reports as "Latency" (e.g., 1.04 seconds, 0.32 seconds). It includes the chunk itself plus the right context.
Algorithmic latency — the computation time, reported as Real-Time Factor (RTF). For example, RTF = 0.093 means processing 1 second of audio takes 0.093 seconds of computation. This is separate from the input buffer delay and is not included in the "latency" figures.
Total end-to-end latency — the sum of input buffer delay plus computation time. The paper does not report this combined figure explicitly but provides both components for calculation.

Training Procedure: Fine-tuning Offline Sortformer with AOSC

The training procedure is designed to teach the offline Sortformer model to work effectively with the AOSC mechanism. It is a fine-tuning stage that starts from a pretrained offline Sortformer checkpoint and introduces the cache during training so the model learns to interpret and utilize the compressed speaker history.

Base model configuration. The starting point is an offline Sortformer model following the configuration from [18] with specific modifications:

NEST encoder: Uses a 109M-parameter version (rather than the original 115M), trained on multilingual data with 128-dimensional Mel-spectrogram inputs (rather than the original 80-dimensional). The NEST pre-encoder performs 8× downsampling via convolutional layers, reducing the frame rate from 10 ms (Mel-spectrogram frame step) to 80 ms (effective prediction step).
No global feature normalization: Feature normalization is removed because it is "unsuitable for streaming" — in a streaming setting, you cannot compute global mean and variance over the entire recording (since you haven't seen it yet). The model must work with unnormalized features or with local normalization computed over the available context.
Transformer encoder: A stack of Transformer layers on top of the NEST pre-encoder, outputting four sigmoid activations corresponding to up to four speakers.
Total parameters: 117M parameters.

Fine-tuning data handling. Training samples are 90-second audio segments. During fine-tuning, these are processed in sequential 15-second windows — the 90-second segment is split into 6 windows, and the model processes them sequentially, updating the AOSC at each window boundary. This simulates the streaming inference setup: the model sees only 15 seconds at a time but has access to the compressed history of previous windows through the cache.

Speaker cache size during training. The cache size is set to $M = 188$ frames, which at 80 ms per frame equals 15.04 seconds of audio. This means the cache can hold a compressed representation of roughly one window's worth of speech — a deliberate match between cache capacity and the processing window size. When the cache is full, the update mechanism must compress 15+ seconds of speaker activity into 15 seconds of selected embeddings.

AOSC update parameters (the same as inference). The cache update mechanism during training uses the same hyperparameters that will be used at inference:

$A = 3$ — three silence frames appended per speaker.
$\delta = 0.05$ — recency bonus for newly added frames.
Strong boosting: $K = 33$ frames per speaker, $\Delta = -2\log(0.5)$ .
Weak boosting: $K = 66$ frames per speaker, $\Delta = -\log(0.5)$ .

(These were described in detail in the cache update mechanism section above.)

Data augmentation strategies. The paper introduces two training augmentations specifically designed to make the model robust for streaming operation:

1. Random speaker permutation in the cache. At each streaming step during training, all speakers in the speaker cache are randomly permuted. This means the cache order is intentionally scrambled — speaker 1's embeddings might be placed in the position that normally corresponds to speaker 3, etc. The model must learn to ignore the cache's speaker ordering during training and rely on the content of the embeddings themselves, not their position, to identify speakers.

Why this matters: This is a counterintuitive but clever choice. At inference time, the cache is always in arrival-time order. Why train with random permutations? The answer is robustness to cache errors: if the cache update mechanism ever makes a mistake (e.g., retaining a frame from the wrong speaker, or if a speaker's voice changes significantly), the model should not blindly trust that the first cache segment is speaker 1. By randomly permuting during training, the model learns to use the cache as a set of speaker-discriminative features rather than as an ordered sequence — it must extract "who is who" from the acoustic content, not from the position. This is a form of data augmentation that prevents the model from overfitting to the cache ordering, making it more robust when the cache is imperfect.

The paper also notes that this is a replacement for standard augmentation techniques: "Notably, we do not apply common data augmentation techniques such as SpecAugment [35] or RIR+Noise augmentation [36]." The random speaker permutation serves as the primary augmentation, targeting the streaming-specific failure mode (over-reliance on cache ordering) rather than acoustic robustness.

2. Right-context limitation. The self-attention mechanism in the Transformer encoder is restricted to look only 7 frames (560 ms at 80 ms per frame) into the future, and this restriction is applied with 50% probability for each training batch. When the restriction is not applied, the model can attend to the full right context within the 15-second window.

Why this matters: In a true streaming setting, the model has limited right context — it can only look a short distance into the future (the "input buffer" in Figure 1). During inference with 1.04-second latency, for example, the right context is 7 frames (Table 2: "Right Context = 7"). Training with a mixture of full-context and limited-context batches teaches the model to work well with limited right context while still benefiting from the stronger training signal of full-context batches. The 50% probability means half the training steps use full context (easier learning) and half use limited context (streaming adaptation), balancing between learning speed and streaming performance.

Training infrastructure. All training runs use a batch size of 4, distributed across 64 NVIDIA Tesla V100 GPUs. This is a large distributed training setup, reflecting the computational demands of processing 90-second audio segments through the NEST + Transformer architecture with AOSC updates.

Training objective. The paper does not explicitly restate the loss function for fine-tuning, but it is the same Sort Loss + permutation-invariant loss combination used to train the original offline Sortformer [18]. The key point is that the model is fine-tuned end-to-end with the AOSC in the loop — gradients flow through the Sortformer layers based on how well the model uses the cache context to make correct predictions for the current window.

Design Choice Analysis: Why These Specific Mechanisms?

This section connects the technical details to the paper's central claims, explaining why each design choice was made and what alternatives it supersedes.

Why store NEST embeddings rather than raw features? The AOSC could have stored raw Mel-spectrogram features (like STB does with audio features) or model predictions (like STB stores prediction sequences). Storing NEST embeddings is the intermediate choice that balances information richness (embeddings contain speaker-discriminative features learned from large-scale self-supervised pretraining) with computational efficiency (the pre-encoder has already been applied, so the Transformer layers don't need to recompute representations for cached frames). Raw features would require re-running the entire model on cached frames at each step — wasteful and against the point of caching. Predictions alone would discard the rich acoustic information the model needs to distinguish speakers, reducing the cache to essentially the same function as STB's correlation matching.

Why log-likelihood scoring rather than raw sigmoid values? As discussed in Step 1 of the update mechanism, the log-likelihood formulation $S_i = \log P_i + \sum_{j \neq i} \log(1 - P_j)$ explicitly penalizes overlapping speech and ambiguous frames. An alternative — simply using $P_i$ as the score — would treat a frame where speaker 1 has $P_1 = 0.9$ and speaker 2 has $P_2 = 0.9$ as equally good for speaker 1 as a frame where $P_1 = 0.9$ and $P_2 = 0.1$ . The log-likelihood formulation correctly identifies that the first frame is ambiguous (both speakers are active — it's overlapping speech) while the second is a clean exemplar of speaker 1. This is essential for building a cache that stores discriminative frames — frames that uniquely identify one speaker in contrast to all others.

Why append silence embeddings per-speaker? Without silence markers, the concatenation of speaker embeddings would look like continuous speech with abrupt voice changes — imagine hearing "Hello, my name is Alice" immediately followed by "I think we should reconsider the budget" in Bob's voice, with no pause between them. The model's self-attention would treat this as a single utterance with a mid-sentence speaker change, making it harder to learn that these are separate speech events. The silence embeddings (240 ms of "silence" represented by the average silence embedding) create an explicit acoustic boundary that helps the Transformer's positional encoding and attention patterns recognize the speaker transition. This is an elegant way to encode a structural property (speaker boundaries) into the cache format without modifying the model architecture.

Why use both strong and weak boosting for minimum speaker representation? If the cache only used strong boosting ( $K = 33$ frames), speakers who are currently inactive but well-represented by their top 33 frames would be adequately tracked, but there would be a sharp cliff — frame 34 and beyond receive no boost and could be easily displaced. The two-tier boosting (strong for 33, weak for the next 33) creates a soft margin: the cache prefers to keep ~66 frames per speaker if space allows, but can compress down to 33 under memory pressure without losing the speaker entirely. This is essentially a priority queue with two priority levels rather than a hard per-speaker quota — it gives the cache flexibility to adapt to highly asymmetric conversations (e.g., one dominant speaker and several sporadic speakers) while preventing any speaker from being completely forgotten.

Why can this system eliminate attractors? Attractors in EEND-based systems [6, 7, 8, 13, 14] serve two purposes: (1) they count the number of speakers (the decoder stops generating attractors when all speakers are accounted for), and (2) they provide speaker-specific queries for computing per-speaker activations. Streaming Sortformer eliminates attractors because both functions are handled by other mechanisms:

Speaker counting is handled by the fixed four-output architecture — Sortformer does not count speakers dynamically; it always outputs four channels and relies on the model to output near-zero probabilities for non-existent speakers. For recordings with fewer than four speakers, the unused channels simply produce low activations. For recordings with more than four speakers (tested on DIHARD III's 5+ speaker subsets), the model tracks the four most dominant speakers.
Speaker-specific queries are provided by the AOSC — the cached NEST embeddings serve as "memory queries" that the Transformer's self-attention can use to compare the current chunk's speakers against past speakers. The attention mechanism naturally computes similarity between current-frame embeddings and cached speaker embeddings, and because the cache is ordered by speaker index, the model can learn to align current speaker identities with cache positions.

Why Sortformer's arrival-time ordering is the key enabler. Without arrival-time ordering, the AOSC would face the same permutation problem as STB: when adding new speakers or when updating the cache, there would be no guarantee that speaker 1 in the cache corresponds to speaker 1 in the model's current output. The arrival-time training ensures that the model's internal representation of "speaker index 1" is stable — it always means "the first person who started speaking in this recording." The cache and the model share this convention, so no matching step is needed. This is the fundamental simplification: the training objective (Sort Loss) creates a shared naming convention between the cache and the model, making the permutation resolution problem disappear.

Summary of Design Choices and Their Justifications

NEST embeddings in cache rather than raw features or predictions: Provides rich speaker-discriminative information without requiring re-encoding, balancing information content and computational cost.
Log-likelihood scoring for frame selection: Penalizes overlapping speech and ambiguous frames, ensuring the cache stores clean exemplars of each speaker that are useful for discrimination.
Recency bonus $\delta = 0.05$ : Enables gradual cache refresh, ensuring the representation tracks current speaker characteristics rather than freezing on historically best frames.
Two-tier minimum speaker boosting ( $K = 33, 66$ ): Creates a soft margin for per-speaker frame retention, preventing speaker forgetting while allowing asymmetric cache allocation.
Silence embeddings with $A = 3$ frames per speaker: Creates explicit speaker boundary markers in the cache format, helping the Transformer model learn speaker transitions.
Dynamic per-speaker frame counts determined by scores: Allows the cache to adapt to conversation dynamics without a separate activity-tracking module.
Random speaker permutation augmentation during training: Prevents the model from overfitting to cache position, making it robust to cache imperfections at inference time.
Right-context limitation with 50% probability: Balances training signal strength (full context) with streaming adaptation (limited context), enabling the model to work with 7-frame right context at inference.
FIFO queue as staging area for cache updates: Decouples the update period from the chunk size, allowing the update mechanism to observe longer speaker segments before compression decisions.
No attractors, no permutation resolution: The arrival-time ordering property eliminates two major sources of architectural complexity from prior streaming diarization systems, making the system simpler and potentially more robust.

4. Key Insights and Innovations

Innovation 1: Arrival-Time Ordering as a Permutation-Resolution Mechanism — Eliminating the Need for Explicit Speaker Matching

The core intellectual contribution of this paper is not the streaming architecture per se, but the recognition that a training objective can replace an inference-time algorithm. Specifically, Sortformer's Sort Loss — which forces the model to output speakers in arrival-time order — functions as a distributed permutation resolution mechanism that operates during training rather than during inference. This is a fundamentally different approach to the permutation problem than any prior streaming diarization work.

What the field did before. Every prior streaming EEND system — from BW-EDA-EEND [10] to the speaker-tracing buffer approaches [11, 12] to the most recent FS-EEND and LS-EEND [13, 14] — treated permutation resolution as an inference-time operation. The model produces chunk-wise predictions with arbitrary speaker ordering, and a separate algorithm (typically correlation maximization across all $S!$ permutations) must determine which output dimension in the current chunk corresponds to which speaker identity in the history. This algorithm is external to the model — it operates on the model's outputs, not within the model's computation. It is a post-hoc patch that corrects for a property the model was never trained to control.

The dominant assumption was that permutation invariance is necessary for training — the model cannot know, a priori, which speaker will appear first in a recording, so it must be trained with permutation-invariant loss that treats all output-speaker assignments as equivalent. The permutation resolution step at inference time was seen as the inevitable cost of this training freedom.

What this paper does differently. Sortformer's Sort Loss breaks this assumption. It trains the model to learn a canonical ordering — first-arriving speaker on channel 1, second-arriving on channel 2, etc. — that is stable across the entire recording. The permutation resolution is not performed by an external algorithm; it is learned by the model during training and expressed in its forward pass. At inference time, the model's output dimensions are already correctly ordered, and the AOSC simply respects this ordering by storing and retrieving embeddings in the same index scheme.

This is not just a performance improvement — it is a conceptual reframing of the permutation problem. Rather than asking "how do we match speaker identities across chunks after the model makes predictions?" it asks "can we train the model to produce predictions that are already identity-consistent across chunks?" The answer, demonstrated by the results, is yes — and the payoff is the elimination of an entire class of inference-time machinery.

Why this is distinctive beyond the mechanism. The AOSC is the mechanism (described in Section 3), but the innovation is the recognition that arrival-time ordering makes the buffer problem dramatically simpler. In a standard STB system, the buffer stores both features and predictions, and must run a correlation maximization at every chunk boundary. In Streaming Sortformer, the buffer stores embeddings in a fixed order and requires no matching step at all — this is evident from comparing Equation 9 (a single forward pass with no permutation step) to Equations 5–6 (explicit $\operatorname*{argmax}$ over permutations). The difference is not a cleverer matching algorithm; it is that the matching is unnecessary by construction.

Evidence. The paper does not provide an ablation directly comparing AOSC with and without permutation resolution — doing so would require comparing against a fundamentally different architecture (STB-based EEND). However, the structural evidence is clear: Table 1 shows that Streaming Sortformer-AOSC at 10-second latency (19.02% DER on DIHARD III Eval with post-processing) outperforms EEND-EDA + FW-STB (25.09%) and EEND-GLA-Large + BW-STB (20.73%), both of which require explicit permutation resolution. The performance is competitive with or superior to attractor-based systems (FS-EEND, LS-EEND) that also use explicit ordering mechanisms. The paper's claim is not just about DER — it is about achieving these results without attractors and without permutation resolution operations, as explicitly stated in the introduction.

Significance beyond performance. This innovation matters because it generalizes. Any streaming architecture that must maintain consistent identity across chunks faces a permutation problem — not just diarization, but also multi-speaker ASR, speaker-attributed translation, or any task where multiple parallel output channels correspond to tracked entities. The insight that a training objective can enforce channel-meaning consistency across time, eliminating the need for runtime matching, is applicable beyond this specific model and task. It suggests a design principle: embed the matching into the training signal rather than solving it at inference time.

This is a fundamental shift from an incremental improvement. Prior streaming diarization work (BW-EDA-EEND, STB, FS-EEND, LS-EEND) iteratively refined the permutation resolution algorithm — better correlation metrics, more efficient permutation search, integration with attractors. This paper eliminates the need for such an algorithm entirely, which is a different kind of advance: a simplification that comes from a better problem formulation, not from a better solution to the old formulation.

Innovation 2: Embedding-Level Speaker Cache with Score-Based Dynamic Allocation — Memory as a Learned Representation, Not a Raw Buffer

The second conceptual contribution is the design of the AOSC as a semantically compressed speaker memory rather than a raw feature buffer. Prior speaker-tracing buffers store what the system has seen (audio features) and what it has predicted (speaker probabilities). The AOSC stores what the model has learned about each speaker — NEST embeddings that encode speaker-discriminative information in a representation space the downstream Transformer layers are trained to interpret.

What the field did before. Speaker-tracing buffers [11, 12] store raw Mel-spectrogram features and model predictions. The buffer serves as a literal memory — "here is what the audio looked like and what we thought about it" — and the correlation matching operates on this literal representation. FS-EEND and LS-EEND [13, 14] similarly store frame-level predictions for attractor-based matching. In all cases, the buffer contents are the inputs and outputs of the model, stored verbatim.

The shift to embedding-level caching. The AOSC stores NEST pre-encoder outputs — intermediate representations that have already been partially processed by the model. This is a representational memory: rather than storing the raw sensory data (Mel spectrograms) and re-processing it at each step, the system stores the model's internal encoding of that data, which is optimized for the downstream task. When these embeddings are concatenated with current-chunk features and fed into the Transformer layers, the self-attention mechanism can directly compare current frame embeddings against cached speaker embeddings in a shared representational space. This is a more efficient and potentially more robust form of memory because the representations are already aligned with the model's task-specific feature space.

This choice has a deeper implication: it blurs the line between memory and computation. In a raw-feature buffer, the memory is passive — it stores data, and the model must compute representations from scratch each time it accesses the memory. In an embedding cache, the memory is partially pre-computed — the model's own encoding work from past steps is preserved and reused. This is analogous to the difference between caching raw database rows versus caching pre-computed query results: the latter trades storage generality for access speed and relevance.

The dynamic allocation mechanism as learned attention. The score-based update mechanism (Steps 1–7 in Section 3.2) determines which frames to retain based on the model's own confidence scores. This creates a feedback loop: the model's past predictions control which acoustic evidence is preserved for future predictions. Frames that the model identifies as "clean exemplars" of a speaker (high $S_i$ , indicating single-speaker confidence) are preferentially retained; ambiguous frames (overlap, low confidence) are evicted. The cache is not a passive recording of the past — it is an actively curated memory shaped by the model's own certainty.

The dynamic per-speaker sizing — where each speaker occupies a variable number of frames based on their score distribution — is particularly noteworthy. Prior STB approaches typically use fixed per-speaker buffer allocations. The AOSC's dynamic allocation means the system automatically devotes more memory to speakers who are currently active or whose representation needs refreshing, without requiring a separate speaker-tracking or activity-detection module. This is a form of implicit attention: the update mechanism implicitly "attends" to the most informative frames for each speaker, using the model's own confidence as the attention weight.

Evidence. The paper demonstrates that AOSC works even without fine-tuning (Offline Sortformer-AOSC in Table 1), showing that the embedding-level representation is intrinsically useful — the model can exploit cached NEST embeddings even though it was never trained to do so. However, performance degrades substantially without fine-tuning (27.58% vs. 14.79% on DIHARD III Eval at 10 s latency), confirming that the model benefits from learning to use the cache. The fine-tuned streaming model then recovers performance close to or better than the offline baseline, demonstrating that the cache provides effective long-term speaker memory.

Significance. The embedding-cache design suggests a more general principle for streaming neural architectures: store intermediate representations, not raw inputs. This is applicable to any task where the model compresses raw data through a series of transformations — store the representation at the stage where the information is most compact while still being useful for downstream layers. The dynamic, score-based curation further suggests that the model's own confidence can serve as a memory management policy, creating a tight coupling between what the model knows and what it remembers.

This is an incremental refinement conceptually (it builds on STB rather than replacing the buffer concept), but a significant practical advance because it (a) reduces per-step computation (no re-encoding of cached frames), (b) provides richer speaker representations (learned embeddings vs. raw features), and (c) enables dynamic per-speaker sizing without additional machinery.

Innovation 3: Training Augmentation as Architecture-Aware Regularization — Random Speaker Permutation and Right-Context Limitation as Targeted Robustness Interventions

The paper's training augmentation strategy is notable not for what it adds but for what it replaces and what it specifically targets. Standard speech processing pipelines typically apply acoustic data augmentation — SpecAugment (time/frequency masking) [35] and RIR+Noise augmentation (simulated room acoustics and background noise) [36] — to improve robustness to acoustic variability. This paper drops both and instead introduces two augmentations that are specific to the streaming-with-cache architecture: random speaker permutation in the cache, and probabilistic right-context limitation.

What makes this distinctive. These augmentations are not generic regularization — they are architectural robustness interventions designed to prevent the model from learning spurious shortcuts that the cache and streaming setup might otherwise enable.

Random speaker permutation addresses a specific failure mode: the model could learn to rely on cache position rather than acoustic content to identify speakers. If the cache is always ordered [speaker 1, speaker 2, speaker 3, speaker 4] during training, the model might develop positional heuristics — "the first segment of the cache is the first speaker" — rather than truly learning to match current speech to cached speaker characteristics based on voice similarity. At inference time, any cache corruption (e.g., a frame from speaker 2 accidentally stored in speaker 1's segment, or a speaker with an ambiguous voice) would break these brittle positional heuristics. By randomly permuting the cache during training, the model is forced to use content-based speaker matching — it must extract "who is speaking now" from the acoustic evidence and match it against the cached embeddings based on voice characteristics, not cache position.

This is a subtle but important distinction: the paper is not just adding noise to make training harder (standard augmentation). It is breaking a spurious correlation (cache position ↔ speaker identity) that the training setup would otherwise create. The target is the interaction between the architecture and the training data, not the acoustic variability of the data itself.

Right-context limitation addresses a different shortcut: the model could learn to rely on future audio to disambiguate speakers at chunk boundaries. In offline training with full self-attention, the model can "look ahead" to see that a speaker continues speaking after the current frame, or that a different speaker is about to start. In streaming inference, this future context is severely limited (7 frames = 560 ms in the 1.04 s latency configuration; 1 frame = 80 ms at 0.32 s latency). By probabilistically restricting right context during training (50% of batches use only 7 frames of right context), the model learns to make predictions with minimal future information while still benefiting from the stronger training signal of full-context batches.

The replacement of standard augmentation is itself a claim. The paper explicitly states: "Notably, we do not apply common data augmentation techniques such as SpecAugment or RIR+Noise augmentation." This implies a methodological claim: for streaming diarization with a speaker cache, the primary robustness challenge is not acoustic variability (which the self-supervised NEST encoder may already handle well) but architectural shortcut learning. The targeted augmentations address the specific failure modes that the cache-and-streaming setup introduces, while generic acoustic augmentation addresses a different (and perhaps less critical) robustness dimension.

Evidence for effectiveness. The paper does not provide an ablation study on these augmentations, which is a limitation. We cannot directly observe how much DER degrades without random permutation or without right-context limitation. However, the strong streaming performance — particularly the relatively small degradation from 10 s to 1.04 s to 0.32 s latency (Table 1: DIHARD III Eval DER goes from 19.02% to 18.97% to 19.32% with post-processing) — is indirect evidence that the model has learned to work with limited context, since the offline model without these augmentations would presumably degrade more sharply when context is reduced.

Significance. This innovation is methodological rather than architectural. It demonstrates a design philosophy: identify the specific shortcuts that your architecture enables during training, and design augmentations that specifically break those shortcuts. This is a more surgical approach to regularization than applying generic data augmentation, and it is likely applicable to other streaming neural architectures that combine memory buffers with limited context. The idea that "the memory buffer's ordering creates a spurious correlation that must be broken during training" is a transferable insight for any system that couples ordered memory with sequence processing.

This is an incremental conceptual contribution (targeted augmentation is not novel in itself) but a practically significant choice because it suggests that standard augmentation recipes may be suboptimal for streaming architectures, and that architecture-aware augmentation design can be more effective than generic approaches.

Innovation 4: Empirical Finding That Streaming Diarization Can Match Offline Performance — Redefining the Performance Ceiling for Real-Time Systems

Table 1 contains a result that, while not foregrounded as an innovation, challenges a fundamental assumption in the field: that streaming diarization must be substantially worse than offline diarization. On DIHARD III Eval, Streaming Sortformer-AOSC at 10-second latency achieves 19.02% DER with post-processing, compared to 21.39% for offline Sortformer — the streaming system actually outperforms its offline counterpart by 2.37 percentage points absolute. On CALLHOME (all speaker counts), the gap is similarly favorable: streaming at 10 s latency achieves 10.09% vs. offline's 11.26%.

Why this is surprising. The paper's own introduction establishes that "the performance gap between offline and streaming speaker diarization is significantly wider than the gap observed between offline and online ASR systems." The field expectation — reflected in the prior systems listed in Table 1 — is that streaming imposes a substantial accuracy penalty. BW-EDA-EEND, EEND-EDA + FW-STB, EEND-GLA-Large + BW-STB, FS-EEND, and LS-EEND all operate in the 19–25% DER range on DIHARD III, while the offline Sortformer sits at 21.39% — and these prior streaming systems were fine-tuned per-dataset, while Streaming Sortformer is a single model evaluated across all datasets without dataset-specific fine-tuning.

The streaming system surpassing the offline system is not just "closing the gap" — it is inverting the expected relationship. This is not a claim the paper makes explicitly as an innovation, but it is an empirical finding with significant implications.

The likely explanation — and why it matters. The paper provides a plausible mechanism for this inversion (Section 4.3):

"This likely stems from the offline Sortformer's underperformance on long recordings due to a mismatch with 90-second training samples. In contrast, streaming Sortformer avoids this issue with its fixed inference window."

Offline Sortformer was trained on 90-second segments. When applied to longer recordings (DIHARD III Eval includes recordings of varying lengths, some substantially longer than 90 seconds), the model encounters sequence lengths it was never trained on, and the full-length self-attention mechanism may struggle to generalize. Streaming Sortformer, by design, always processes a fixed window (the cache + FIFO queue + input buffer) regardless of total recording length. Its effective receptive field is bounded and matches its training distribution.

This is a diagnostic finding: the offline model's performance ceiling on long recordings is not a fundamental accuracy limit of the architecture but an artifact of the training-segment duration. The streaming system, by enforcing a fixed processing window, acts as a form of architectural regularization against sequence-length extrapolation errors. In other words, the streaming constraint — normally seen as a limitation — actually improves robustness to long recordings by preventing the model from operating outside its trained context length.

Why this is a conceptual contribution, not just a result. This finding reframes the relationship between offline and streaming diarization. It suggests that, at least for models with bounded training segment lengths, streaming is not just a latency-constrained approximation of offline processing — it can be a strictly better processing paradigm for long recordings. The fixed window size enforces consistency between training and inference conditions that offline processing (with variable-length inputs) violates.

This is analogous to findings in other sequence-processing domains where chunked or sliding-window approaches outperform full-sequence models on long inputs — the chunked approach avoids the distribution shift from training-length to inference-length sequences. The specific mechanism here (chunked processing via AOSC + FIFO) is not the innovation; the finding that this mechanism can exceed offline performance is.

Evidence and caveats. The streaming system does not universally outperform offline — for 5+ speakers on DIHARD III, offline achieves 51.51% while streaming achieves 41.45% at 10 s (actually better, though both are poor), and on CALLHOME 2-speaker subsets, offline maintains a small edge (5.37% vs. 5.27% at 10 s with post-processing). The inversion is most pronounced on the challenging, multi-domain DIHARD III dataset with variable-length recordings — exactly where the offline model's length mismatch would be most severe. This supports the proposed mechanism.

The caveat is that offline Sortformer's length limitation is specific to this architecture; other offline systems (e.g., clustering-based approaches, or EEND with chunked processing) may not exhibit this failure mode. The finding is not that "streaming is universally better than offline" but that a streaming architecture can match or exceed a specific offline architecture when that offline architecture has length generalization limitations.

Significance. This finding is empirically significant but not theoretically novel. It does not propose a new concept — it demonstrates that a specific limitation of offline models (training-inference length mismatch) can be overcome by streaming design. The practical implication is substantial: for deployments where recordings vary widely in length, a streaming system may be not just acceptable but preferable to an offline system, even if latency is not a strict requirement. This inverts the standard "streaming as a necessary compromise" framing.

5. Experimental Analysis

Evaluation Methodology

Dataset. The system is evaluated on three standard speaker diarization benchmarks: DIHARD III Eval [25] (a challenging multi-domain dataset with diverse recording conditions), CALLHOME Part2 [30] (two-speaker telephone conversations, using the standard Kaldi two-fold split where Part1 is training and Part2 is evaluation), and CH109 (a two-speaker subset of 109 sessions from the Callhome American English Speech dataset [34]). For training, the authors use 5150 hours of simulated mixtures plus 2030 hours of real multi-talker speech drawn from Fisher English Training Speech [22], AMI Corpus (IHM, Lapel Mix, SDM) [23, 24], DIHARD III Dev [25], VoxConverse-v0.3 [26], ICSI [27], AISHELL-4 [28], NIST SRE 2000 CALLHOME Part1 [30], AliMeeting (near and far microphones) [31], and DiPCo [32] with forced alignment-based RTTMs from [33]. All training data is segmented into 90-second chunks with up to four speakers and an 8-second shift between consecutive segments. For AliMeeting specifically, an offline Sortformer model is used to filter out segments with high insertion rates, addressing unannotated portions of recordings.
Base model(s). The starting point is an offline Sortformer model [18] with modifications: a 109M-parameter multilingual NEST encoder [19] (replacing the original 115M-parameter English-only version) operating on 128-dimensional Mel-spectrogram features (replacing the original 80-dimensional), with global feature normalization removed to support streaming. The NEST encoder is based on the Fast-Conformer architecture [20] and performs 8× downsampling (10 ms input frame step → 80 ms prediction step). On top of this sits a stack of Transformer encoder layers [21] outputting four sigmoid activations corresponding to up to four speakers, trained with a combination of Sort Loss and permutation-invariant loss. The total model size is 117M parameters. This model was chosen as representative of contemporary diarization capabilities and as the base for the streaming adaptation.
Metrics. The primary metric is Diarization Error Rate (DER), the standard metric in the field, which jointly accounts for missed speech (speaker present but not detected), false alarm speech (silence labeled as speech), and speaker confusion (correct detection but wrong speaker assignment). DER is reported as a percentage. Evaluation conditions vary by dataset: DIHARD III Eval uses 0 s collar tolerance (no forgiveness for boundary errors), while CALLHOME Part2 and CH109 use 0.25 s collar tolerance (consistent with prior work on these datasets). The paper applies different post-processing parameters per dataset group — one set tuned on DIHARD III Dev for DIHARD III Eval, and another set tuned on CALLHOME Part1 for CALLHOME Part2 and CH109 — to account for variations in collar length and annotation style. All evaluations include overlapping speech.
Baselines. The paper compares against an extensive set of published streaming and offline diarization systems, all evaluated on the same or overlapping benchmark sets. Streaming systems include: BW-EDA-EEND [10] (block-wise EEND with 10 s latency), EEND-EDA + FW-STB [12, 8] (frame-wise STB with attractors, 1 s latency), EEND-GLA-Large + BW-STB [8] (global-local attractors with block-wise STB, 1 s latency), FS-EEND + VCT [13] (frame-wise streaming with variable chunk training, 1 s latency), and LS-EEND [14] (long-form streaming EEND, 1 s latency). The offline baseline is Offline Sortformer [18] (the same architecture without streaming modifications). Additionally, three variants of the proposed system are evaluated: Offline Sortformer (full-context processing), Offline Sortformer-AOSC (AOSC applied at inference only, without fine-tuning), and Streaming Sortformer-AOSC (fully fine-tuned streaming system).
Generation budget / compute accounting. The paper does not measure compute in FLOPs or generations (as typical in LLM papers). Instead, it reports latency as the primary resource metric, defined as the input buffer delay (the amount of right-context included beyond the current chunk) and measured in seconds. Three latency configurations are evaluated: 10 s, 1.04 s, and 0.32 s (detailed in Table 2). Algorithmic computation cost is reported separately as Real-Time Factor (RTF) — the wall-clock time to process a recording divided by its duration — measured with batch size 1 on an NVIDIA RTX 6000 Ada Generation GPU. RTF values are 0.005 (10 s latency), 0.093 (1.04 s), and 0.180 (0.32 s). The paper notes that the declared latency values do not include algorithmic latency from computation.
Cross-validation / statistical protocol. There is no explicit cross-validation or statistical significance testing reported. The paper states that no dataset-specific fine-tuning is performed — all DERs in Table 1 are achieved with a single model evaluated across all three benchmark datasets without per-dataset adaptation. Post-processing parameters are tuned separately for DIHARD III Dev vs. CALLHOME Part1 but are applied consistently within each evaluation group. For the AliMeeting training data filtering, an offline Sortformer model is used to remove segments with high insertion rates. The evaluation is on fixed standard test splits (DIHARD III Eval, CALLHOME Part2, CH109) as used in prior work.

Main Quantitative Results

The experimental results are organized around a single comprehensive evaluation table (Table 1) that reports DER across all three benchmark datasets, three latency configurations (plus offline baselines), and with/without post-processing. The results are further broken down by speaker count subsets (≤4 spk, ≥5 spk for DIHARD III; 2, 3, 4, 5, 6 spk for CALLHOME). Table 2 provides the parameter details for each latency configuration.

Streaming Sortformer Achieves State-of-the-Art Performance at Low Latency

The headline finding is that Streaming Sortformer-AOSC matches or exceeds the performance of prior streaming systems across all evaluated benchmarks while using a simpler architecture (no attractors, no permutation resolution). The key numbers from Table 1, focusing on the 10-second latency configuration with post-processing (the most favorable setup for the proposed system):

DIHARD III Eval (all speakers): 19.02% DER for Streaming Sortformer-AOSC vs. 25.09% for EEND-EDA + FW-STB, 20.73% for EEND-GLA-Large + BW-STB, 19.61% for LS-EEND, and 21.39% for offline Sortformer. The streaming system outperforms its own offline counterpart by 2.37 percentage points.
DIHARD III Eval (≤4 speakers): 13.67% DER — the lowest among all systems compared, including LS-EEND (13.96%) and offline Sortformer (14.17%).
DIHARD III Eval (≥5 speakers): 41.45% DER. Note that all systems degrade substantially on 5+ speaker subsets; LS-EEND achieves 42.98%, offline Sortformer achieves 51.51%. Streaming Sortformer is competitive here despite being optimized for ≤4 speakers.
CALLHOME Part2 (all speakers): 10.09% DER vs. 14.93% (EEND-EDA + FW-STB), 14.29% (EEND-GLA-Large + BW-STB), and 12.11% (LS-EEND). Again, streaming outperforms offline Sortformer (11.26%).
CALLHOME Part2 (2-speaker): 4.82% DER — the single best result in Table 1, marginally worse than offline Sortformer's 4.86% but within the range of measurement noise.
CH109 (2-speaker): 5.09% DER vs. 12.82% (Offline Sortformer-AOSC without fine-tuning at 10 s). No prior streaming systems are reported on CH109.

Evidence that fine-tuning with AOSC is essential. Comparing Offline Sortformer-AOSC (AOSC applied at inference only, no fine-tuning) to Streaming Sortformer-AOSC (fine-tuned with AOSC) at 10 s latency without post-processing on DIHARD III Eval: 21.59% vs. 14.79% — fine-tuning improves DER by 6.8 percentage points absolute. The gap is even larger on CALLHOME Part2 (all speakers): 18.54% vs. 11.10% — a 7.4 point improvement. This confirms that the model must learn to use the cache; simply inserting cached embeddings into an offline-trained model yields substantially worse performance.

Performance Degradation with Reduced Latency Is Surprisingly Small

Table 1 reports results at three latency levels: 10 s, 1.04 s, and 0.32 s (all with post-processing unless noted). The key finding is that reducing latency by 30× (from 10 s to 0.32 s) degrades DER by only 0.3 percentage points on DIHARD III Eval (19.02% → 19.32%), with a slightly larger impact on CALLHOME Part2 (10.09% → 11.50%):

DIHARD III Eval (all speakers, with post-processing):
- 10 s: 19.02%
- 1.04 s: 18.97% (actually marginally better than 10 s — likely within noise)
- 0.32 s: 19.32% (only 0.30 points worse than 10 s)
DIHARD III Eval (≤4 speakers, with post-processing):
- 10 s: 13.67%
- 1.04 s: 13.32%
- 0.32 s: 13.43% (0.24 points better than 10 s)
CALLHOME Part2 (all speakers, with post-processing):
- 10 s: 10.09%
- 1.04 s: 10.79% (0.70 points worse)
- 0.32 s: 11.50% (1.41 points worse than 10 s)

The DIHARD III results are particularly striking — the 0.32 s and 1.04 s configurations actually show marginally better DER than the 10 s configuration in some subsets. This suggests that, contrary to expectation, the additional right-context provided by the 10 s latency setup does not translate into better diarization accuracy — and may even introduce noise or over-reliance on distant context that hurts performance. The CALLHOME results show the expected monotonic degradation with reduced latency, but the magnitude is modest (1.41 points from 10 s to 0.32 s).

Without post-processing, the pattern is similar but with slightly higher absolute DERs. On DIHARD III Eval (all speakers):

10 s: 14.79% (without post-processing) vs. 19.02% (with post-processing — note: post-processing here increases DER, which is unusual; this is because DIHARD III uses 0 s collar, and post-processing may introduce boundary errors)
1.04 s: 14.57%
0.32 s: 14.63%

Streaming Outperforms Offline on Challenging Multi-Domain Data

A non-obvious and significant result: Streaming Sortformer-AOSC consistently achieves lower DER than Offline Sortformer on DIHARD III Eval and CALLHOME, despite the inherent information disadvantage of streaming. Quantitatively (all with post-processing):

DIHARD III Eval (all): 19.02% (streaming, 10 s) vs. 21.39% (offline) — streaming wins by 2.37 points
DIHARD III (≤4): 13.67% (streaming) vs. 14.17% (offline) — streaming wins by 0.50 points
DIHARD III (≥5): 41.45% (streaming) vs. 51.51% (offline) — streaming wins by 10.06 points
CALLHOME (all): 10.09% (streaming) vs. 11.26% (offline) — streaming wins by 1.17 points
CH109 (2-spk): 4.82% (streaming) vs. 4.86% (offline) — essentially tied

The authors attribute this to the offline model's inability to handle recordings longer than its 90-second training segments — the streaming system's fixed window size provides better length generalization (discussed in Section 4.3). This is consistent with the largest streaming advantage appearing on DIHARD III (which includes variable-length recordings, some substantially longer than 90 seconds) and the smallest on CH109 (two-speaker telephone calls, likely shorter and closer to the training distribution).

Performance on Subsets Beyond the Design Scope (5+ Speakers)

The system is designed for up to four speakers, but Table 1 reports results on 5+ speaker subsets to test behavior beyond the design scope. The results show:

DIHARD III ≥5 speakers: Streaming Sortformer-AOSC achieves 41.45% DER (10 s, with post-processing), which is competitive with or better than prior systems: LS-EEND achieves 42.98%, EEND-GLA-Large + BW-STB achieves 45.17%. Offline Sortformer achieves 51.51% — substantially worse.
CALLHOME speaker-count breakdown: DER increases with speaker count as expected. At 10 s with post-processing: 2-spk 4.82%, 3-spk 10.01%, 4-spk 11.22%, 5-spk 20.34%, 6-spk 26.97%. The 4-to-5 speaker jump is the largest (9.12 points), consistent with the 4-speaker design limit. However, even at 5–6 speakers, performance is competitive with or better than LS-EEND (24.63% at 5-spk, 27.89% at 6-spk).

This demonstrates that the system gracefully degrades rather than catastrophically failing when the speaker count exceeds the design limit. It tracks the four most dominant speakers effectively and presumably assigns marginal speakers to one of the four channels or drops them.

Latency Configuration Details (Table 2)

Table 2 specifies the parameters for each latency configuration:

Latency	Chunk Size	Right Context	FIFO Queue	Update Period	Speaker Cache	RTF
10.0 s	124 frames	1 frame	124 frames	124 frames	188 frames	0.005
1.04 s	6 frames	7 frames	188 frames	144 frames	188 frames	0.093
0.32 s	3 frames	1 frame	188 frames	144 frames	188 frames	0.180

Key observations:

The speaker cache size is constant at 188 frames (15.04 seconds) across all latency configurations — the long-term memory budget does not change with latency.
The FIFO queue is substantially larger (188 frames) in the low-latency configurations than in the 10 s configuration (124 frames). At low latency, the FIFO queue provides the bulk of the temporal context, compensating for the tiny chunk size (3–6 frames).
The update period (144 frames for 1.04 s and 0.32 s) is larger than the chunk size, meaning the AOSC is updated less frequently than new chunks arrive. This reduces computational overhead and allows the update mechanism to observe longer speaker segments before compression.
RTF scales with reduced latency: 0.005 (10 s) → 0.093 (1.04 s) → 0.180 (0.32 s), reflecting the increased frequency of model forward passes. However, even at 0.32 s latency, the RTF of 0.180 means the system runs at roughly 5.6× real-time on a single GPU — well within real-time constraints.

Ablation Studies and Robustness Checks

The paper is notably light on formal ablation studies — there is no dedicated ablation section, no table systematically varying individual AOSC hyperparameters, and no experiment isolating the contribution of each component (FIFO queue, cache update mechanism, training augmentations). The primary "ablation" is the comparison between three system variants (Offline Sortformer, Offline Sortformer-AOSC without fine-tuning, Streaming Sortformer-AOSC with fine-tuning) in Table 1. However, several implicit ablations and robustness checks can be extracted:

Fine-tuning with AOSC vs. applying AOSC at inference only: This is the closest to a core ablation. Offline Sortformer-AOSC (AOSC applied to the offline checkpoint without any fine-tuning) achieves 27.58% DER on DIHARD III Eval (10 s, no post-processing) and 18.54% on CALLHOME Part2 (all speakers), compared to 14.79% and 11.10% respectively for Streaming Sortformer-AOSC with fine-tuning. The gap — 12.79 and 7.44 percentage points — confirms that the model must be explicitly trained to use the cache; the cached NEST embeddings are not immediately interpretable by an offline-trained model in a zero-shot manner. This establishes that the streaming performance is not trivially achievable by just concatenating cached embeddings. However, Offline Sortformer-AOSC at 1.04 s latency performs even worse (29.46% DER on DIHARD III, 22.58% on CALLHOME), suggesting that shorter processing windows exacerbate the model's inability to interpret the cache without fine-tuning.
Effect of post-processing: Table 1 reports results both with (✓) and without (✗) timestamp post-processing. The post-processing consists of six operations: onset thresholding, offset thresholding, onset padding, offset padding, and removal of short silences or speech segments below specified thresholds. The effect is dataset-dependent: on DIHARD III Eval, post-processing increases DER for the streaming system (14.79% → 19.02% at 10 s), while on CALLHOME and CH109, it decreases DER (11.10% → 10.09% on CALLHOME all speakers at 10 s). This is because DIHARD III uses 0 s collar tolerance — any boundary adjustments from post-processing that shift segment boundaries, even slightly, incur collar errors. CALLHOME's 0.25 s collar is more forgiving of boundary modifications. This is not presented as an ablation per se, but it demonstrates that post-processing effects are highly dependent on evaluation protocol and should not be assumed to be universally beneficial.
Single model across all datasets vs. per-dataset fine-tuning: The paper explicitly states that "unlike most studies that fine-tune their models separately for each evaluation dataset, our DERs... are achieved with a single model, without any dataset-specific fine-tuning across all evaluation sets." This is a robustness claim: the system generalizes across diverse recording conditions (telephone, meeting, dinner party, broadcast) without per-domain adaptation. The comparison systems (BW-EDA-EEND, EEND-EDA + FW-STB, FS-EEND, LS-EEND) may have been fine-tuned per-dataset, which would make the comparison somewhat unfair in favor of prior work. The paper does not re-evaluate prior systems under the same single-model constraint, so the magnitude of this advantage cannot be quantified, but the claim of single-model generalization is credible given the diversity of the evaluation sets.
Training data filtering (AliMeeting): The use of an offline Sortformer model to filter AliMeeting training segments with high insertion rates is a data cleaning step rather than an ablation. Its impact on final performance is not isolated. This is noted as a potential confound: if the offline model used for filtering has systematic biases, those biases could propagate into the training data and affect the streaming model's behavior on similar domains.
Effect of offline model's training-segment length on long recordings: This is a diagnostic finding rather than a controlled ablation, but it functions as one. The offline Sortformer was trained on 90-second segments. When evaluated on DIHARD III recordings (which include longer sessions), it underperforms relative to the streaming system. This implicitly ablates the training-inference length match — the streaming system, by using fixed 15-second windows during training and inference, maintains consistent length distributions that the offline system violates. This is not experimentally isolated (no experiment varies the offline training segment length to show DER improvement), but the mechanism is logically coherent and consistent with the pattern of results (largest streaming advantage on the dataset with the longest recordings).
Missing ablations (noted as limitations): The paper does not ablate:
- The contribution of the FIFO queue vs. the AOSC alone (how much does the intermediate context buffer matter?).
- The individual AOSC hyperparameters (boosting strength $\Delta$ , number of boosted frames $K$ , recency bonus $\delta$ , silence frames $A$ ).
- The random speaker permutation augmentation during training (what happens without it?).
- The right-context limitation during training (what happens if trained with full context only or limited context only?).
- The cache size $M$ — all experiments use 188 frames (15 seconds). How does performance scale with cache size?
- The NEST embedding choice vs. alternative representations (raw features, bottleneck features, predictions-only).
- The log-likelihood scoring vs. simpler alternatives (raw sigmoid, max probability).
- Performance with more than 4 output channels — the system is capped at 4 speakers by design. What if the architecture supported 6 or 8 channels?

These missing ablations represent significant gaps in the experimental validation. The paper's core claims — that AOSC eliminates the need for permutation resolution and that arrival-time ordering is the key enabler — are supported structurally (by the absence of permutation steps in Equations 7–9 vs. 2–6) and by competitive overall performance, but not by direct ablation experiments that would isolate the contribution of each mechanism.

Critical Assessment

Claim 1: Streaming Sortformer-AOSC achieves state-of-the-art streaming diarization performance

Supported, with caveats about comparison fairness. Table 1 shows Streaming Sortformer-AOSC at 10 s latency with post-processing achieving 19.02% DER on DIHARD III Eval, which is lower (better) than LS-EEND (19.61%), FS-EEND (no DIHARD III number reported in the table — only CALLHOME numbers at specific speaker counts), EEND-GLA-Large + BW-STB (20.73%), and EEND-EDA + FW-STB (25.09%). On CALLHOME, the margin is clearer: 10.09% vs. 12.11% (LS-EEND), 14.29% (EEND-GLA-Large + BW-STB), and 14.93% (EEND-EDA + FW-STB). These numbers support the claim.

However, the comparison is complicated by several factors:

Post-processing and per-dataset tuning: The paper applies different post-processing parameters per dataset group (one set for DIHARD III, another for CALLHOME/CH109), while it's unclear whether prior systems used comparable post-processing optimization. Without post-processing, the DER on DIHARD III is 14.79% (substantially better), which actually widens the gap over prior systems if they lacked similar post-processing. But the post-processing pipeline's effect is large and inconsistent (it helps on CALLHOME but hurts on DIHARD III), making it difficult to isolate the model's intrinsic performance from the post-processing optimization.
Single model vs. per-dataset fine-tuning: The paper emphasizes that its results come from a single model without per-dataset adaptation, which is a significant practical advantage but makes numeric comparisons against potentially per-dataset-tuned baselines unfair in the opposite direction — the proposed system might be at a disadvantage relative to baselines that were optimized per-dataset. If prior systems were fine-tuned per-dataset, the Streaming Sortformer's competitive performance despite being "handicapped" by single-model evaluation is a stronger result. However, the paper does not explicitly state whether each baseline was per-dataset fine-tuned, leaving this ambiguity unresolved.
Missing 0.32 s latency results for baselines: The paper evaluates at three latency levels but prior systems are reported only at their published latencies (typically 1 s for FS-EEND, LS-EEND, and STB-based systems; 10 s for BW-EDA-EEND). The 0.32 s latency results (19.32% on DIHARD III) cannot be compared against prior work. The claim of "robustness at low latency" is internally consistent across the three latency configurations but lacks external comparison at the lowest latency point.
NEST encoder version difference: The paper uses a "more advanced 109M-parameter version trained on multilingual data with 128-dimensional Mel-spectrograms" compared to the original Sortformer's 115M-parameter English-only encoder. The offline Sortformer baseline also uses this improved encoder, so the offline-vs-streaming comparison is fair, but comparisons against prior systems (which use different encoder architectures) partially reflect encoder quality improvements rather than architectural innovations.

Claim 2: The arrival-time ordering eliminates the need for permutation resolution and attractors

Supported structurally, but not experimentally isolated. The paper's strongest evidence for this claim is architectural: compare Equations 7–9 (Streaming Sortformer) to Equations 2–6 (STB-based EEND). The former has no $\operatorname*{argmax}$ over permutations, no correlation coefficient computation, no $\psi$ permutation application step. The AOSC stores embeddings in speaker-index order, and the model predicts in speaker-index order, so the output is already correctly permuted by construction. This is a clean structural argument.

However, an experiment that directly tests this claim is missing. To demonstrate that arrival-time ordering (and not the NEST embedding representation, or the specific cache update mechanism, or the fine-tuning protocol) is responsible for eliminating permutation resolution, one would need to compare:

Streaming Sortformer with AOSC (arrival-time ordering, no permutation step) vs.
A modified version with explicit permutation resolution added (to show that it doesn't help, or that it matches performance, confirming the permutation step is unnecessary).

This experiment is not performed. The competitive performance against STB-based systems (which do use permutation resolution) provides indirect evidence — if permutation resolution were necessary, Streaming Sortformer without it should perform worse — but the comparison is confounded by all the other architectural differences (NEST embeddings vs. raw features, Sort Loss training vs. permutation-invariant training, different encoder architectures).

Similarly, the claim of eliminating attractors is supported by the absence of attractors in the architecture (no LSTM encoder-decoder, no self-attention attractor mechanism), but there is no ablation showing that adding attractors would not improve performance. The comparison against attractor-based systems (EEND-EDA, FS-EEND, LS-EEND) shows Streaming Sortformer is competitive, but this reflects the entire system, not specifically the attractor-vs-no-attractor choice.

Claim 3: Performance degrades gracefully with reduced latency

Well-supported by the latency sweep. Table 1 provides DER at three latency levels (10 s, 1.04 s, 0.32 s) across all evaluation conditions. The degradation is minimal — on DIHARD III Eval (all speakers, with post-processing): 19.02% → 18.97% → 19.32%. The 1.04 s configuration actually shows the best DER on several subsets, suggesting that the optimal latency might be lower than 10 s. The CALLHOME results show a clearer trend (10.09% → 10.79% → 11.50%), but the 1.41 percentage point degradation from 10 s to 0.32 s is modest relative to the 30× latency reduction.

The granularity of the latency sweep (only three points, covering a 30× range) is somewhat coarse — there could be non-monotonic behavior between 0.32 s and 1.04 s that is not captured. Additionally, the paper does not explore the extreme low-latency regime (e.g., 100 ms or 50 ms), which would be relevant for applications like live captioning with strict delay requirements. The RTF at 0.32 s (0.180) suggests that even lower latencies might be computationally feasible, but the input buffer delay (which defines "latency" here) cannot go below 1 frame (80 ms) without fundamentally changing the architecture.

Claim 4: A single model generalizes across diverse datasets without per-dataset fine-tuning

Supported by the evaluation protocol, but lacking comparative baselines. The paper explicitly evaluates a single model on all three benchmarks and reports these results. The diversity of the benchmarks — DIHARD III (multi-domain, challenging acoustics, variable speaker counts), CALLHOME (telephone speech, 2–6 speakers), CH109 (American English telephone, 2 speakers) — covers a meaningful range of conditions. The fact that a single model achieves competitive or superior performance across all of them is a genuine practical advantage.

However, the claim would be stronger if it included a comparison where the same model (without per-dataset fine-tuning) is evaluated against prior systems also evaluated without per-dataset fine-tuning. If prior systems were fine-tuned per-dataset, the single-model Streaming Sortformer's competitive performance is impressive; if they were not, the comparison is fair but less informative. The paper does not clarify this for the baseline systems.

Genuine Experimental Weaknesses

No ablation of core AOSC hyperparameters. The paper specifies cache size (188 frames), boosting parameters ( $K=33, 66$ , $\Delta=-2\log 0.5$ , $-\log 0.5$ ), recency bonus ( $\delta=0.05$ ), and silence frames ( $A=3$ ) without any sensitivity analysis. Given the complexity of the 7-step update mechanism, it is likely that performance is sensitive to some of these choices, but the reader cannot assess which ones matter or how to tune them for new domains.
No comparison against a same-architecture STB baseline. The paper's central claim is that arrival-time ordering simplifies the buffer mechanism. A direct comparison against a modified Sortformer that uses explicit permutation resolution (like STB) instead of relying on arrival-time ordering would cleanly isolate the contribution. Without this, the reader cannot distinguish between "Sortformer is a better diarization model" and "arrival-time ordering simplifies the buffer."
Single model family (NEST + Transformer). All experiments use the NEST pre-encoder with Fast-Conformer architecture. It is unclear whether the findings — particularly the streaming-vs-offline performance inversion — are specific to this architecture or generalizable to other encoder architectures (e.g., standard Conformer, HuBERT, WavLM).
Limited latency granularity below 0.32 s. The lowest latency evaluated (0.32 s) leaves open the question of whether the system works at ultra-low latencies (sub-100 ms). The RTF of 0.180 suggests compute is not the bottleneck, but the architecture's minimum latency is bounded by the 80 ms frame rate — with a chunk size of 3 frames (240 ms at 80 ms per frame) and right context of 1 frame (80 ms), the theoretical minimum input buffer delay is 320 ms, which the 0.32 s configuration achieves. Going lower would require architectural changes (higher temporal resolution, smaller downsampling factor).
No evaluation on long-form benchmark beyond DIHARD III. While DIHARD III includes variable-length recordings, some of which exceed 90 seconds, the paper does not evaluate on dedicated long-form diarization benchmarks (e.g., VoxConverse test set, AMI full meetings without segmentation, or dinner party recordings from CHiME). The claim that streaming handles long recordings better than offline is supported only by the DIHARD III results and the logical argument about training-inference length mismatch.
Post-processing confounds DER interpretation. The post-processing pipeline is tuned per-dataset and has a large, inconsistent effect (improves CALLHOME, degrades DIHARD III). The raw DER without post-processing (14.79% on DIHARD III at 10 s) is substantially better than the post-processed DER (19.02%), which means the model's frame-level predictions are more accurate than the final diarization output suggests. This is an artifact of the 0 s collar on DIHARD III and raises questions about whether DER is the right metric for comparing systems that use different post-processing strategies.
Absence of confidence intervals or statistical testing. With 500 test recordings in DIHARD III Eval and an unknown number in CALLHOME Part2 and CH109, DER differences of 1–2 percentage points may or may not be statistically significant. The paper provides no error bars, standard deviations, or significance tests, making it difficult to assess whether, for example, the 19.02% vs. 19.61% gap over LS-EEND is reliable or within sampling noise. Given the small absolute differences between top systems, this is a meaningful omission.
Training data filtering using the offline model introduces potential bias. The paper filters AliMeeting training data using an offline Sortformer model to remove segments with high insertion rates. If the offline model has systematic errors (e.g., systematically inserting speakers in certain acoustic conditions), the training data will be biased toward segments the offline model handles well, potentially inflating the streaming model's apparent generalization. This is not an ablation that was run, and the filtering effect is unquantified.
The 4-speaker limit is a hard architectural constraint, not a learned limitation. Sortformer outputs exactly four channels. Performance on 5+ speaker subsets (41.45% on DIHARD III) is presented as a positive result, but this reflects a fundamental limitation: the system cannot track a fifth speaker as a distinct entity. In a 5-person meeting, one speaker is always either merged with another or dropped entirely. The paper acknowledges this in the conclusion ("we plan to extend the system to handle up to eight speakers") but the current system's applicability is limited to scenarios where the speaker count is known to be ≤4, which excludes many real-world meetings.

6. Limitations and Trade-offs

6.1 No Ablation of Core AOSC Hyperparameters or Design Choices

The assumption or constraint. The AOSC update mechanism (Section 3.2) is specified with a rich set of hyperparameters: cache size $M = 188$ frames (15 seconds), two-tier minimum speaker boosting ( $K = 33$ and $K = 66$ frames with boosts $\Delta = -2\log 0.5$ and $-\log 0.5$ ), recency bonus $\delta = 0.05$ , silence frames per speaker $A = 3$ , and the log-likelihood scoring formula (Equation 10). The paper provides no sensitivity analysis, hyperparameter sweep, or ablation for any of these values. The choice to store NEST embeddings rather than alternative representations (raw features, bottleneck features, predictions only) is similarly un-ablated — we do not know whether the embedding-level representation is genuinely superior or merely sufficient.

The consequence. A practitioner attempting to deploy Streaming Sortformer on a new domain or with a different encoder architecture cannot know which hyperparameters are critical to tune versus which are robust defaults. For example, the cache size of 188 frames (15 seconds at 80 ms frame rate) may be insufficient for domains with longer speaker turns or more speakers competing for cache space, or it may be unnecessarily large for domains with short utterances, wasting memory and increasing the effective sequence length fed to the Transformer. Similarly, the two-tier boosting scheme is complex — if the strong-boost parameter $K = 33$ is too small, speakers with brief appearances may be forgotten; if too large, the cache may be dominated by minimum-representation frames at the expense of high-quality exemplars. Without ablation evidence, these remain guesses.

The recency bonus $\delta = 0.05$ is particularly consequential: it determines how quickly the cache forgets older frames in favor of newer ones. If a speaker's voice characteristics are stable (e.g., same microphone, same distance, same speaking style), a low recency bonus preserves the historically best exemplars; if the speaker's voice changes (e.g., moving around the room, Lombard effect), a higher bonus is needed to track the drift. The paper's single $\delta = 0.05$ value is asserted without evidence of its appropriateness across recording conditions.

What evidence exists in the paper. None. There is no ablation study (no table, no figure, no paragraph) that varies any AOSC hyperparameter and reports the resulting DER. The only "ablation" is the coarse comparison between Offline Sortformer-AOSC (no fine-tuning) and Streaming Sortformer-AOSC (with fine-tuning) in Table 1, which establishes that fine-tuning matters but does not isolate the contribution of any specific cache mechanism. The paper also includes no comparison against alternative cache contents (e.g., raw features instead of NEST embeddings, prediction vectors instead of embeddings, or fixed per-speaker allocation instead of score-based dynamic allocation). These are not minor omissions — the AOSC is the paper's central contribution, and its internal design choices are presented as a completed recipe rather than as hypotheses that were tested.

Mitigation status. Not addressed. The paper provides no guidance for hyperparameter tuning, no default ranges, and no suggestion that these values are dataset-specific or task-specific. A practitioner must either use the reported values exactly (hoping they generalize) or perform their own hyperparameter search without the benefit of knowing which dimensions are most important.

6.2 The Arrival-Time Ordering Claim Is Structurally Evident but Not Experimentally Isolated

The assumption or constraint. The paper's central architectural claim is that Sortformer's arrival-time ordering eliminates the need for explicit permutation resolution and attractor mechanisms. This is presented as the key advantage over prior streaming EEND systems — compare Equations 7–9 (no $\operatorname*{argmax}$ over permutations, no attractor computation) to Equations 2–6 (explicit correlation-based permutation resolution $\psi = \operatorname*{argmax}_{\phi \in \text{perm}(S)} \text{CC}(\mathbf{P}_n^{buf}, \phi(\hat{\mathbf{P}}_n^{buf}))$ ). The arrival-time ordering is the mechanism that enables this simplification.

The consequence. The paper cannot disentangle why Streaming Sortformer achieves competitive performance from what makes it competitive. There are multiple confounding differences between Streaming Sortformer and the prior STB-based systems it compares against: the base model architecture (Sortformer vs. EEND-based), the encoder (NEST Fast-Conformer vs. various alternatives), the training objective (Sort Loss + PIT vs. PIT-only), the cache contents (NEST embeddings vs. raw features + predictions), the cache management (score-based dynamic allocation vs. fixed-size FIFO), and — only then — the presence or absence of explicit permutation resolution. When Streaming Sortformer outperforms EEND-EDA + FW-STB on DIHARD III (19.02% vs. 25.09% at 10 s with post-processing), we cannot attribute this to arrival-time ordering specifically. The improvement could equally reflect the NEST encoder's stronger representations, Sort Loss as a better training objective, or the embedding-level cache providing richer speaker information — all of these are confounded with the permutation-resolution claim.

A direct ablation would be: compare Streaming Sortformer-AOSC (arrival-time ordering, no permutation step) against an otherwise identical system that adds an explicit permutation resolution step after the Sortformer predictions (to test whether the permutation step is actually unnecessary) or that removes arrival-time ordering (to test whether it is actually necessary). Neither experiment appears in the paper.

What evidence exists in the paper. Only structural evidence (the absence of permutation operations in the equations) and competitive overall performance (Table 1). The paper does not provide:

An experiment where explicit permutation resolution is added to Streaming Sortformer (showing it doesn't help, confirming the permutation step is redundant).
An experiment where arrival-time ordering is disabled (e.g., training with PIT-only and using STB matching, keeping the same architecture and cache).
A comparison against a version of Sortformer trained with PIT-only and using STB for streaming, which would isolate the Sort Loss contribution from the cache and architecture contributions.

Mitigation status. Not addressed experimentally. The paper relies on the structural argument and the competitive DER numbers to support the claim, but a skeptical reader could reasonably conclude that Sortformer (with NEST embeddings and score-based cache management) is simply a stronger base diarization model, and the arrival-time ordering property — while elegant — is not the primary driver of the performance. The authors do not acknowledge this as a limitation or propose experiments to resolve it.

6.3 Four-Speaker Hard Limit Is an Architectural Constraint, Not a Learned Capacity

The assumption or constraint. Sortformer is designed with exactly four output channels — four sigmoid activations, one per potential speaker — and this is a hard architectural limit baked into the model's final layer dimension. The paper acknowledges this explicitly in Section 1:

"the original Sortformer is an offline model that relies on full-length self-attention, making it unsuitable for streaming applications. Additionally, its ability to process long audio recordings is constrained by the maximum input length that the self-attention mechanism can handle"

and in Section 4.3:

"while Sortformer is primarily designed and optimized for scenarios with up to 4 speakers, we also evaluate its performance on benchmarks with 5+ speakers to understand the system's behavior beyond its primary design scope."

However, the four-speaker constraint is more fundamental than the streaming limitation, which the paper addresses. Extending to more speakers would require retraining the entire model with additional output channels — it is not a configurable parameter that can be adjusted at inference time.

The consequence. For any real-world deployment with more than four speakers — a 5-person meeting, a panel discussion with 6 participants, a dinner party with 8 people — the system cannot track all speakers as distinct identities. It must either merge multiple speakers into one output channel (producing speaker confusion errors), drop speakers entirely (producing missed speech errors), or both. The 5+ speaker results in Table 1 quantify this: on DIHARD III Eval ≥5 speakers, DER is 41.45% (10 s with post-processing), compared to 13.67% for ≤4 speakers — a 27.78 percentage point gap. On CALLHOME, DER jumps from 11.22% at 4 speakers to 20.34% at 5 speakers and 26.97% at 6 speakers. These numbers are substantially worse than the ≤4 speaker regime and degrade rapidly as the speaker count increases beyond the design limit.

The paper presents the 5+ speaker results as evidence that the system "gracefully degrades" and "is able to accurately and robustly track the four most dominant speakers" (Section 4.3). But "tracking the four most dominant speakers" means that in a 5-person meeting, one person's speech is systematically misattributed or lost — and which person that is depends on their speaking time and prominence, not on their importance to the conversation. For applications like meeting transcription or legal deposition analysis, missing a speaker entirely is often a catastrophic failure, not an acceptable degradation.

Furthermore, the system provides no mechanism to signal which speaker was dropped or merged. The diarization output simply assigns all speech frames to one of four channels, without indicating that a fifth speaker was present but could not be represented. A downstream consumer of the diarization output (e.g., a meeting analytics system or a transcription display) has no way to know that the output is incomplete.

What evidence exists in the paper. Table 1 reports DER broken down by speaker count for DIHARD III (≤4 vs. ≥5) and CALLHOME (2, 3, 4, 5, 6 speakers). The steep degradation at 5+ speakers is clearly evident. The paper does not provide a detailed error analysis of the 5+ speaker case (e.g., what fraction of errors are missed speech vs. speaker confusion, whether the dropped speaker is consistently the least-talkative one, whether speaker identity swaps occur at the 4–5 speaker boundary).

Mitigation status. The conclusion (Section 5) explicitly acknowledges this limitation and states:

"For future work, we plan to extend the system to handle up to eight speakers, broadening its applicability."

This is an architectural change, not a hyperparameter adjustment — it requires redesigning the output layer and retraining. The paper provides no interim workaround for the current 4-speaker system (e.g., a mechanism to detect that a fifth speaker is present and route to a fallback system, or a clustering-based post-processing step to recover dropped speakers). Until this extension is realized, Streaming Sortformer's practical applicability is limited to scenarios where the maximum speaker count is known and bounded by four — which excludes a significant fraction of real-world meeting and conversation scenarios.

6.4 The Performance Comparison Against Prior Systems Is Confounded by Encoder Quality, Post-Processing, and Per-Dataset Tuning Differences

The assumption or constraint. The paper compares Streaming Sortformer-AOSC against an extensive list of prior streaming diarization systems (BW-EDA-EEND, EEND-EDA + FW-STB, EEND-GLA-Large + BW-STB, FS-EEND, LS-EEND) in Table 1. The headline claim — state-of-the-art streaming diarization performance — depends on these comparisons. However, several confounds make it difficult to attribute the performance differences to the proposed architecture rather than to other factors.

The consequence. Three specific confounds undermine the comparison:

Encoder quality difference. The paper uses "a more advanced 109M-parameter version [of NEST] trained on multilingual data with 128-dimensional Mel-spectrograms" (Section 4.2), upgrading from the original Sortformer's 115M-parameter English-only encoder. While the offline Sortformer baseline also uses this improved encoder (making the offline-vs-streaming comparison fair), the prior systems being compared against (BW-EDA-EEND from 2021, EEND-EDA + STB from 2021, FS-EEND and LS-EEND from 2024) use different encoder architectures entirely. Any performance gap could partially reflect the NEST encoder's quality rather than the AOSC or arrival-time ordering mechanisms. The paper does not provide a comparison where all systems use the same encoder, which would isolate the architectural contributions.
Single model vs. per-dataset fine-tuning. The paper states that "unlike most studies that fine-tune their models separately for each evaluation dataset, our DERs reported in Table 1 are achieved with a single model, without any dataset-specific fine-tuning across all evaluation sets" (Section 4.3). This is both a strength (demonstrating generalization) and a confound for the comparison. If prior systems were fine-tuned per-dataset — which is common practice in diarization research — then Streaming Sortformer is being compared against baselines that were optimized for each specific evaluation domain, making the comparison unfairly favorable to the baselines. The paper does not clarify whether each baseline was evaluated under the same single-model constraint. If some baselines were per-dataset tuned, Streaming Sortformer's competitive or superior performance despite being "handicapped" is a stronger result than the raw numbers suggest; if they were not, the comparison is fair but the paper's emphasis on single-model generalization is less distinctive than claimed.
Post-processing pipeline differences. The paper applies a 6-step timestamp post-processing pipeline (onset/offset thresholding and padding, short segment removal) with parameters tuned separately for DIHARD III Dev vs. CALLHOME Part1 (Section 4.3). The effect is large and inconsistent: on DIHARD III Eval, post-processing increases DER for Streaming Sortformer-AOSC (14.79% → 19.02% at 10 s), while on CALLHOME, it decreases DER (11.10% → 10.09%). The paper does not specify whether prior systems used comparable post-processing, nor does it re-evaluate prior systems with the same post-processing pipeline. A system with a strong post-processing pipeline tuned aggressively on the development set could achieve lower DER than a system with better frame-level accuracy but weaker post-processing. Without controlling for post-processing, the DER numbers in Table 1 conflate model quality with post-processing optimization.

What evidence exists in the paper. The paper reports DER with and without post-processing for its own system (Table 1, ✓ and ✗ columns), showing the magnitude of the post-processing effect. For prior systems, only a single DER number is reported, and their post-processing status is unspecified. The encoder difference is acknowledged (Section 4.2 describes the "more advanced" NEST encoder) but not flagged as a confound for external comparisons. The per-dataset fine-tuning issue is mentioned in Section 4.3 as a distinguishing feature of this work, but the baseline systems' training protocols are not documented.

Mitigation status. Partially acknowledged through transparency (reporting both raw and post-processed DER, stating the single-model evaluation protocol), but not resolved experimentally. A fair comparison would re-evaluate at least one prior system using: (a) the same encoder, (b) the same single-model constraint, and (c) the same post-processing pipeline. Alternatively, the paper could explicitly caveat the comparison by noting that any performance advantage over prior systems may partially reflect improved encoder quality rather than the proposed streaming mechanisms.

6.5 No Evaluation on Dedicated Long-Form Benchmarks to Substantiate the Streaming-Over-Offline Claim

The assumption or constraint. A striking result in Table 1 is that Streaming Sortformer-AOSC outperforms Offline Sortformer on DIHARD III Eval (19.02% vs. 21.39% at 10 s with post-processing) and on CALLHOME (10.09% vs. 11.26%). The paper attributes this to the offline model's length generalization failure:

"This likely stems from the offline Sortformer's underperformance on long recordings due to a mismatch with 90-second training samples. In contrast, streaming Sortformer avoids this issue with its fixed inference window." (Section 4.3)

This is a post-hoc explanation, not a tested hypothesis. It implies that Streaming Sortformer should show its largest advantage on recordings that substantially exceed the offline model's training segment length (90 seconds), and little or no advantage on recordings that fall within the training distribution.

The consequence. The claim that streaming processing is a strictly better paradigm for long recordings — not just a latency-acceptable approximation — requires evidence beyond two datasets where the recording lengths are not reported or systematically varied. DIHARD III Eval is a multi-domain dataset that includes recordings of varying lengths, but the paper does not break down results by recording duration. Some DIHARD III recordings may be under 90 seconds (where the offline model should perform well), and others may be substantially longer (where the offline model should degrade). Without a duration-stratified analysis, the reader cannot assess whether the streaming advantage is genuinely driven by length generalization or by some other factor (e.g., the streaming model's fine-tuning with AOSC providing a stronger training signal, or the DIHARD III domain being better matched to the streaming training distribution).

What evidence exists in the paper. The only evidence is the aggregate DER comparison in Table 1, which shows streaming outperforming offline on DIHARD III (where recordings are likely longer and more variable) and CALLHOME (where telephone conversations are typically shorter and the gap is smaller — 10.09% vs. 11.26%). On CH109 (2-speaker telephone, likely the shortest recordings), streaming and offline are essentially tied (4.82% vs. 4.86%). This pattern is consistent with the length-generalization hypothesis but is far from conclusive — it could equally reflect that streaming fine-tuning provides a general regularization benefit (not just length-related), that DIHARD III's acoustic conditions happen to be better handled by the fine-tuned model, or that the offline model's 90-second training segments were suboptimal for DIHARD III's domain regardless of length.

The paper does not evaluate on dedicated long-form diarization benchmarks where recording lengths are extreme and the length-generalization claim could be definitively tested. Examples include:

VoxConverse [26] test set (recordings ranging from 10 seconds to over 20 minutes).
AMI full meetings without segmentation (typically 20–40 minutes).
DiPCo dinner party recordings (15–30 minutes of continuous multi-speaker conversation).
CHiME-6/CHiME-8 dinner party scenarios (multi-hour recordings).

These benchmarks are standard in the diarization community and would provide clear evidence for or against the length-generalization claim. The paper's training data includes VoxConverse, AMI, and DiPCo (Section 4.1), making their exclusion from evaluation notable — the datasets are available and were used for training.

Mitigation status. Not addressed. The length-generalization explanation is presented as a plausible hypothesis but is not experimentally verified. The paper does not:

Report recording duration distributions for the evaluation sets.
Provide a duration-stratified DER breakdown (e.g., DER for recordings <2 min, 2–5 min, >5 min).
Evaluate on standard long-form benchmarks (VoxConverse test, full AMI meetings, DiPCo).
Vary the offline model's training segment length to test whether longer segments close the gap.

Without this evidence, the claim that streaming is better than offline for long recordings — rather than merely competitive — remains speculative. A practitioner with long-form recordings (multi-hour meetings, court proceedings) cannot confidently choose Streaming Sortformer over an offline alternative based on this evidence.

6.6 The AOSC Training and Inference Procedure Assumes Availability of 90-Second Segments for Fine-Tuning and Does Not Address Cold-Start or Very-Short-Conversation Performance

The assumption or constraint. The fine-tuning procedure (Section 4.2) processes 90-second audio samples in sequential 15-second windows, updating the AOSC at each window boundary. This assumes that training data can be segmented into 90-second chunks and that inference will operate on recordings of comparable or greater length. The cache is trained with a size of 188 frames (15 seconds) and expects to accumulate speaker representations over multiple processing windows.

The consequence. Two regimes are poorly characterized by the current evaluation:

Cold-start performance on very short recordings (e.g., under 15 seconds). In the first processing window, the cache is empty ( $\mathbf{B}_0 = \emptyset$ in Equation 7), and the model processes only the current chunk without any cached speaker history. The system has no prior information about speaker identities and must simultaneously detect who is speaking and begin building cache representations. The paper does not evaluate or analyze performance on short recordings where the cache never reaches capacity or where only one or two cache update cycles occur. In applications like real-time command processing, short query-response interactions, or snippet-level diarization (e.g., diarizing individual video clips from social media), the cold-start regime dominates, and the paper provides no evidence of how well the system performs.
Very long recordings where the cache must represent speakers over extended periods. The cache size is 188 frames (15 seconds) regardless of recording length. For a 1-hour meeting, the cache still holds at most 15 seconds of speaker exemplars. The recency bonus ( $\delta = 0.05$ ) ensures gradual refresh, but it may be insufficient for speakers whose voice characteristics drift substantially over long periods (e.g., due to fatigue, emotional changes, or varying distance from the microphone). Conversely, the recency bonus might be too aggressive if speakers are consistent but appear sporadically — a speaker who talks for 10 seconds every 10 minutes might find their exemplars gradually displaced by more recent frames from other speakers, even if the original exemplars were high-quality. The paper does not evaluate on recordings longer than DIHARD III's typical durations or analyze speaker-specific cache retention over long timescales.

What evidence exists in the paper. None directly. Table 1 reports aggregate DER across entire evaluation sets without breakdowns by recording duration. The offline-vs-streaming comparison (discussed in Limitation 6.5) provides indirect evidence that streaming handles longer recordings better than the offline model — but this evaluates the system's behavior on recordings longer than 90 seconds, not its cold-start performance or its behavior on multi-hour sessions. The paper includes no analysis of the first-window error rate vs. later-window error rate, which would quantify the cold-start penalty.

The training data construction (90-second segments with 8-second shifts) means the model rarely encounters sequences shorter than 90 seconds during training. The minimum effective recording length for which the system's behavior is well-characterized is approximately one full training window (15 seconds) plus enough preceding context to build a meaningful cache.

Mitigation status. Not addressed. The paper does not:

Evaluate on short recordings (sub-15-second) to quantify cold-start DER.
Report per-window error rates to show whether early windows have systematically higher DER.
Analyze cache retention for individual speakers over extended recordings.
Discuss whether the cache update mechanism's hyperparameters (particularly the recency bonus $\delta$ and the boosting strengths) should be adjusted for recordings of different lengths.
Propose alternative cache initialization strategies for the cold-start regime (e.g., using a universal background speaker embedding, or pre-populating the cache with speaker profiles if available).

A practitioner deploying Streaming Sortformer for an application with predominantly short interactions (e.g., voice assistant queries with multi-speaker turns) or extremely long sessions (e.g., full-day conference diarization) faces unknown performance characteristics outside the paper's demonstrated operating range.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological reframing rather than a paradigm shift: it demonstrates that a training objective (Sort Loss, with its arrival-time ordering) can replace what was previously an inference-time algorithm (explicit permutation resolution across chunks). The significance is not in proposing a new diarization architecture — the Sortformer model already existed — but in showing that the streaming challenge for diarization can be simplified by moving the identity-tracking burden from the inference procedure into the training signal.

The paper resolves a tension that has existed since the earliest streaming EEND work [10, 11, 12]: chunk-wise processing inevitably scrambles speaker identities across chunk boundaries, and the dominant solution — correlation-based permutation matching — adds algorithmic complexity and a potential failure mode (incorrect matching) that compounds over time. The field implicitly accepted that this was the price of streaming. Streaming Sortformer shows that the price may be unnecessary if the model is trained to produce a canonical speaker ordering that is stable across chunks.

This does not render prior STB and attractor-based approaches obsolete — those methods handle variable speaker counts (EEND-EDA can generate attractors for arbitrarily many speakers), while Sortformer is architecturally capped at four. But it does redirect research attention: for applications with a known or bounded speaker count (≤4), there is now a simpler path to streaming diarization that eliminates an entire class of inference-time machinery. The paper makes the attractor-based approach [8, 13, 14] — which was the state of the art for streaming — appear as one valid solution among several, rather than the default architecture.

The paper also reframes the relationship between offline and streaming diarization through an empirical finding that is not its central claim but has significant implications: streaming processing can outperform offline processing on long recordings when the offline model's training segments are shorter than the inference recordings. This inverts the standard framing of "streaming as a necessary compromise for latency" and suggests that bounded context windows can serve as a regularizer against training-inference length mismatch — a finding that generalizes beyond diarization to any sequence-processing architecture with self-attention over variable-length inputs.

Finally, the paper redirects the research priority for streaming diarization away from better permutation resolution algorithms and toward better speaker memory representations. The AOSC's design — storing NEST embeddings rather than raw features, using score-based dynamic allocation rather than fixed quotas, appending silence markers as structural punctuation — suggests that the quality of the speaker memory may matter more than the mechanism used to align it with current predictions. This is a shift from an algorithmic focus (how to match) to a representational focus (what to remember).

Research directions that become more attractive:

Training objectives as inference-time simplifiers. Sort Loss eliminates permutation resolution in diarization; analogous objectives could simplify other streaming tasks where identity must be tracked across chunks (multi-speaker ASR, speaker-attributed translation, multi-agent dialogue tracking). The design principle — embed the alignment into the training signal — is transferable.
Embedding-level caching for streaming architectures. The AOSC's choice to store intermediate representations (NEST embeddings) rather than raw inputs is a generalizable pattern for any streaming model with a pre-encoder + downstream processor architecture. Storing pre-computed representations trades storage generality for computational efficiency and representational richness.
Score-based dynamic memory management. Using the model's own confidence scores to curate a memory buffer — as the AOSC does with its log-likelihood scoring and per-speaker boosting — is applicable to streaming attention, neural memory, and state-space models where a fixed-size memory must represent variable-length history.

Research directions that become less attractive:

Incremental improvements to permutation resolution for bounded-speaker streaming. If arrival-time ordering can eliminate the problem entirely (for ≤4 speakers), further optimization of correlation-based matching for similar bounded-speaker scenarios is lower-impact. The effort is better spent on extending arrival-time training to higher speaker counts or on improving the memory representation quality.
Attractor-only architectures for streaming with bounded speakers. The paper demonstrates competitive performance without attractors, suggesting that the attractor mechanism — while powerful for handling variable speaker counts — is not necessary for scenarios where the maximum speaker count is known and small. Resources may be better directed at improving the simpler Sortformer-style architecture for these settings.

Follow-Up Research This Work Enables

1. Directly test whether arrival-time ordering eliminates the need for permutation resolution by adding it back. The paper's central architectural claim — that no permutation resolution step is needed — is supported structurally (Equations 7–9 lack a permutation step) but not experimentally isolated. A clean ablation would take the Streaming Sortformer-AOSC system and add an explicit STB-style permutation resolution step on top of the Sortformer predictions, evaluating whether this improves, degrades, or leaves DER unchanged. If DER is unchanged (the permutation step is truly redundant), the claim is experimentally confirmed. If DER degrades (the permutation step introduces errors by occasionally mismatching correctly-ordered outputs), the claim is strengthened — the arrival-time ordering is not just sufficient but protective. If DER improves (the permutation step catches residual ordering errors), the claim is partially undermined, and the paper's structural argument would need qualification. This experiment requires no new architecture — just a post-processing module that computes correlation between cache predictions and current-chunk predictions and optionally permutes the output.

2. Scale the AOSC to eight speakers and measure how performance degrades with speaker count. The conclusion explicitly plans this extension, and the paper provides the baseline to measure against. The key question is not just whether 8-speaker Sortformer works, but how the AOSC's dynamic per-speaker allocation scales: with 8 speakers competing for the same 188-frame cache, average frames per speaker drops from ~47 (for 4 speakers, ignoring silence frames) to ~24 (for 8 speakers). Does the two-tier boosting mechanism (K = 33, 66) need to be adjusted? Does the cache size need to increase, or can 188 frames adequately represent 8 speakers if the score-based selection is sufficiently discriminative? A strong follow-up would evaluate DER as a function of speaker count from 2 to 8 on a controlled benchmark (e.g., simulated meetings with fixed-per-speaker speaking time), reporting not just aggregate DER but per-speaker DER to detect whether late-arriving or less-talkative speakers are systematically forgotten.

3. Ablate the individual AOSC hyperparameters to identify which design choices matter. The 7-step update mechanism has at least six tunable parameters (cache size M, strong/weak boosting K₁, K₂, Δ₁, Δ₂, recency bonus δ, silence frames A), and none are ablated. A well-designed ablation study would vary each parameter independently while holding others at the paper's values and measure DER on DIHARD III Dev. The most consequential questions: (a) Does the log-likelihood scoring (Equation 10) outperform simpler alternatives like raw sigmoid values or max-probability scoring? This tests whether the penalty for overlapping speech is actually necessary or just well-motivated. (b) How does DER scale with cache size M? If performance saturates at, say, 94 frames (7.5 seconds), the cache could be halved with no accuracy loss, reducing sequence length and computational cost. (c) Is the two-tier boosting necessary, or would a single K = 50 with intermediate Δ perform equivalently? The two-tier design is complex — if a simpler scheme matches performance, the method simplifies without loss.

4. Stress-test the streaming system on dedicated long-form benchmarks with duration-stratified analysis. The paper's claim that streaming outperforms offline on long recordings (Section 4.3) is supported only by aggregate DER on DIHARD III and CALLHOME, without duration breakdowns. A rigorous test would evaluate Streaming Sortformer-AOSC and Offline Sortformer on VoxConverse test (recordings from 10 seconds to >20 minutes), full AMI meetings (20–40 minutes without segmentation), and DiPCo dinner party recordings (15–30 minutes continuous). For each benchmark, report DER stratified by recording duration (e.g., <2 min, 2–5 min, 5–15 min, >15 min) to test whether the streaming advantage grows with duration as the length-mismatch hypothesis predicts. If streaming doesn't show an increasing advantage with duration, the paper's explanation is incorrect, and the streaming-over-offline result reflects some other factor (e.g., AOSC fine-tuning providing stronger regularization independent of length).

5. Measure cold-start performance and first-window error rate to characterize the cost of empty cache. The AOSC starts empty at the beginning of every recording, and the paper provides no analysis of how DER varies over the first few processing windows. A simple analysis would report per-window DER — the DER computed only on frames within the first window, second window, etc. — to quantify the cold-start penalty and how quickly performance stabilizes. If the first window has substantially higher DER (e.g., 30% vs. 15% for later windows), this identifies a deployment constraint for short interactions. A follow-up could test cache initialization strategies: pre-populating the cache with a universal background speaker embedding, with speaker profiles from a previous recording of the same participants, or with a "warmup" period where the system runs in higher-latency mode to build cache before switching to low-latency streaming. The cold-start regime is particularly important for the low-latency configurations (0.32 s, 1.04 s) where the system is most likely to be deployed in real-time interactive applications with short utterances.

6. Replace the AOSC's hand-designed scoring with a learned memory controller. The 7-step cache update mechanism is entirely hand-designed — the scoring formula, boosting strengths, silence insertion, and selection logic are heuristics. A natural extension is to make the cache update learnable: train a lightweight controller network that takes the current cache state and model predictions as input and outputs retention/eviction decisions for each frame. The controller could be trained end-to-end with the Sortformer using REINFORCE or a continuous relaxation (e.g., Gumbel-softmax over retention probabilities), with the objective being final diarization accuracy rather than a proxy scoring heuristic. This would test whether the hand-designed mechanism captures the optimal policy or whether a learned controller can discover better retention strategies — for example, learning to retain frames that are dissimilar to already-cached frames (diversity) rather than just high-scoring frames, or learning domain-specific retention policies (retaining more frames in noisy conditions, fewer in clean conditions). The paper's setting — a fixed architecture with a differentiable diarization objective — makes this a tractable next step.

Practical Applications and Downstream Use Cases

1. Real-time meeting transcription with speaker labels for ≤4-person meetings. The system achieves 10.09% DER on CALLHOME (all speaker counts) and 11.22% DER on 4-speaker CALLHOME at 10 s latency with post-processing, dropping only modestly to 10.79% at 1.04 s latency. For small meetings — a standard 4-person video call, a medical consultation with doctor + patient + family member, or a legal deposition with interviewer + deponent + two attorneys — this is deployment-ready accuracy with latency low enough for live caption display. The single-model generalization across diverse acoustic conditions (telephone, meeting room, dinner party) means a single deployment can handle participants calling in from different environments without per-session calibration. The RTF of 0.093 at 1.04 s latency indicates the system runs at ~11× real-time on one GPU, making it feasible to run alongside ASR on the same hardware in a meeting platform.

2. Live broadcast captioning with speaker attribution for panel discussions. Broadcast panels typically involve 2–4 speakers (host + 1–3 guests) with structured turn-taking and occasional overlap. The DIHARD III Eval results (19.02% DER on all recordings, 13.67% on ≤4 speakers at 10 s latency) demonstrate robustness to the diverse acoustic conditions typical of broadcast (studio microphones, field recordings, phone-in guests). The arrival-time ordering property is particularly valuable here: new guests who join mid-broadcast are automatically assigned to the next available speaker index, and the AOSC begins tracking them from their first utterance without requiring manual configuration. The 0.32 s latency configuration (19.32% DER) pushes latency below the ~2-second threshold typically considered acceptable for live captioning, making real-time speaker-attributed subtitles achievable.

3. Voice assistant and smart speaker multi-user interaction. In a household setting with multiple registered users, a smart speaker must identify who is speaking during multi-turn interactions (e.g., a family conversation where different members issue commands or ask questions in sequence). The CH109 results (5.09% DER at 1.04 s latency, 5.41% at 0.32 s latency) on two-speaker telephone speech provide a lower bound on expected accuracy for the simpler two-speaker case. The cold-start behavior (which the paper does not analyze) is critical here — utterances are typically short (a few seconds each) and separated by silence, meaning the cache may empty between interactions. A deployment would need to combine the Streaming Sortformer with a speaker identification module that persists cache state across sessions, but the per-utterance diarization accuracy demonstrated on CH109 suggests that the core frame-level discrimination is sufficient for high-accuracy multi-user interaction.

4. Offline batch processing of long-form recordings where latency is not required. Counterintuitively, the paper's finding that streaming outperforms offline on long recordings (19.02% vs. 21.39% on DIHARD III) makes Streaming Sortformer a viable choice even for offline batch diarization of long meetings or court proceedings. A scenario: a legal technology company needs to diarize thousands of hours of deposition recordings, many of which are 2–4 hours long with 2–4 speakers. Running the offline model would require either truncating recordings to 90 seconds (losing long-range context and potentially missing speakers who appear only in later segments) or implementing a separate chunking-and-stitching pipeline with permutation resolution — exactly the problem Streaming Sortformer solves. The streaming system can process arbitrarily long recordings in a single pass with bounded memory, achieving lower DER than the offline model without chunk-boundary stitching complexity. The RTF of 0.005 at 10 s latency means processing 1000 hours of audio takes ~5 GPU-hours — entirely practical for batch processing.

When to Prefer This Method

The paper explicitly positions Streaming Sortformer against attractor-based streaming EEND systems (EEND-EDA + STB [12], EEND-GLA-Large [8], FS-EEND [13], LS-EEND [14]) and against its own offline counterpart. The following decision rules emerge from the paper's evidence and acknowledged limitations:

Prefer Streaming Sortformer-AOSC over attractor-based streaming systems (FS-EEND, LS-EEND, EEND-GLA-Large) when: the maximum speaker count is known and ≤4, architectural simplicity is valued (no attractor mechanism to train or tune), and inference should run without explicit permutation resolution. The paper demonstrates competitive or superior DER on all benchmarks (DIHARD III, CALLHOME, CH109) with a simpler architecture.
Prefer Streaming Sortformer-AOSC over Offline Sortformer when: recordings substantially exceed the offline model's training segment length (90 seconds), or streaming latency (≤10 s, scalable to 0.32 s) is required. Streaming outperforms offline on DIHARD III (19.02% vs. 21.39%) and matches or exceeds on CALLHOME (10.09% vs. 11.26%), with the advantage growing on longer recordings.
Prefer attractor-based systems (LS-EEND, EEND-GLA-Large) over Streaming Sortformer when: the speaker count is unknown or exceeds 4. Sortformer's 4-speaker architectural limit is hard — on DIHARD III ≥5 speakers, DER is 41.45%, which may be unacceptable for applications where missing a speaker is a critical failure. Attractor-based systems handle variable speaker counts up to "unlimited" speakers without architectural modification.
Prefer Offline Sortformer over Streaming Sortformer when: the maximum speaker count is ≤4, recordings are consistently ≤90 seconds, and latency is unconstrained. On CH109 (2-speaker, short telephone calls), offline and streaming are essentially tied (4.86% vs. 4.82%), so offline's simpler deployment (no cache management, no FIFO queue) may be preferable at equivalent accuracy. However, the paper demonstrates only a single offline model configuration — the advantage of offline over streaming in this regime is small and may vanish with further tuning.