REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation

🎯 Pitch

A speech translator learns to wait only when the next chunk of audio genuinely reduces uncertainty about what to say—improving the quality-latency trade-off by up to 21% over prior methods, without altering the underlying translation model. The key insight is that the frozen model’s own predictions on partial vs. full audio provide a perfect, intrinsic signal for training the streaming policy.

1. Executive Summary

This paper introduces Regularized Entropy INformation Adaptation (REINA), a new loss function for training an adaptive READ/WRITE policy that converts a non-streaming Speech-to-Text Translation (S2TT) model into a simultaneous speech translation (SimulST) model. Using a 445M-parameter architecture built on Whisper Medium and trained on 130k hours of open-source and synthetic data across French, Spanish, and German (both from and into English), REINA trains a lightweight policy network by maximizing the covariance between a learned heuristic and an information-theoretic estimate of the information gained about the next target token by waiting for more audio — operationalized as the difference in log-probabilities from the frozen S2TT decoder when conditioned on partial versus full input. The approach pushes the reported Pareto frontier of the latency/quality tradeoff, achieving up to 21% improvement in Normalized Streaming Efficiency (NoSE) — a metric the paper introduces to measure streaming performance as the area under the AL/BLEU curve normalized by non-streaming BLEU — over prior divergence-guided methods like DiG-SST, establishing that mutual-information-based policy training yields stronger streaming results than divergence-based alternatives even when the underlying non-streaming model is identical.

2. Context and Motivation

The Core Problem: Balancing Translation Quality and Latency in Streaming Speech Translation

The fundamental challenge this paper tackles is deceptively simple to state but remarkably difficult to solve: when translating speech in real-time, how does a system decide when it has heard enough audio to emit the next translated word versus when it should wait for more input? This is the READ/WRITE decision problem at the heart of Simultaneous Speech Translation (SimulST).

Unlike offline Speech-to-Text Translation (S2TT), where a model has access to the complete utterance before producing any output, SimulST must interleave receiving audio chunks with generating translated text. The system receives audio incrementally — say, in 0.25-second chunks — and after each chunk, it faces a binary choice: READ (consume another audio chunk, thereby increasing latency but gaining potentially crucial context) or WRITE (emit the next target-language token immediately, reducing latency but risking a translation error due to insufficient context).

This trade-off is inherently difficult because different language pairs encode information in different word orders. For example, translating from English ("the cat that I saw yesterday") to German ("die Katze, die ich gestern gesehen habe") places the main verb at the clause end — requiring significantly more audio context before the translator can confidently produce the German verb. A fixed policy that always waits for, say, 3 words of input before emitting output (the "wait-k" strategy) would be suboptimal: it would force unnecessary waiting on language pairs with similar word orders and risk emitting premature translations on pairs with divergent syntax.

The paper formalizes this problem in Section 3.1: given partial audio $a_t$ at frame $t$ and previously emitted tokens $s_1, s_2, \ldots, s_n$ , the system must decide whether to emit token $s_{n+1}$ (WRITE) or wait for audio frame $t+1$ (READ). The goal is to learn a policy that maximizes translation quality while minimizing the latency with which each token is emitted.

Why This Problem Matters: Real-World Impact and Theoretical Significance

The motivation is simultaneously practical and foundational:

Real-world deployment. Conversational applications — voice chat across languages, live interpretation, real-time video captioning — require SimulST. Cascaded systems (ASR → MT) introduce unacceptable latency from the sequential pipeline and propagate errors between stages. End-to-end (E2E) models, which directly map source speech to target text, eliminate this error propagation but still face the fundamental READ/WRITE challenge. The paper notes in Section 1 that "conversational environments such as voice or video chat necessitate SimulST models to facilitate real-time communication across language barriers."

Scale and practicality trade-offs. Industry systems like SeamlessM4T (Section 2) use massive proprietary datasets (600k+ hours) and monotonic attention mechanisms (EMMA) to achieve strong streaming performance. But as the authors document in Appendix D, these approaches are "excessively expensive to compute at train time and suffer from both poor numerical stability and difficulty in converging." Specifically, they found that implementing EMMA required a matrix of size [batch_size × attention_heads × num_text_tokens × audio_sequence_length × audio_sequence_length] within each cross-attention layer — for a typical Whisper encoder output of 1500 frames and 8 attention heads, this consumes 2GB of VRAM per cross-attention layer at batch size 1 in fp32 precision. This makes such methods impractical for the broader research community and for deployment scenarios where smaller, cheaper models are needed.

The paper explicitly targets a middle ground: models large enough to achieve strong translation quality but small enough to be "usable in real-world chat settings" (Section 3.2). Their 445M-parameter architecture (408M at inference time) is significantly smaller than SeamlessM4T or Hibiki — deliberately chosen to demonstrate that efficient streaming policies can be trained without massive model scale.

Open-source data gap. The paper identifies "a notable gap in the literature between industry work leveraging massive proprietary datasets and less resourced research making heavy use of MTL to get the most out of smaller data scales" (Section 2). While OWSM bridged this gap for non-streaming S2TT, no equivalent existed for SimulST. The authors aim to fill this by training on 130k hours of entirely open-source or synthetically generated data — MLS, CVSS-C, Must-C, Mosel, and CCMatrix — with multi-task learning (ASR, NMT, S2TT jointly) to maximize the utility of available data.

Theoretical foundation. Beyond practical deployment concerns, the paper addresses a conceptual question: can we derive an information-theoretically principled policy for when to wait versus when to generate? The authors' core insight is that a mutual information formulation — "wait for more audio if and only if we gain information by doing so" (Section 3.1) — provides a rigorous foundation that prior heuristic or divergence-based approaches lacked. This connects the SimulST problem to fundamental concepts in information theory, offering both theoretical clarity and practical gains.

Where Prior Approaches Fall Short

The paper identifies specific limitations across several families of existing methods:

Fixed Policies (wait-k)

The simplest approach is wait-k: always wait for $k$ input words (or frames) before emitting any output, then alternate READ/WRITE thereafter. As the paper notes in Section 2, this is "usually suboptimal due to the mismatch between the sampling rate of the input audio frames and the frequency of outputted words." Speech frames arrive at a fixed rate (e.g., every 0.25 seconds), but output tokens are produced at a variable rate depending on the translation — a one-to-one alternation doesn't map naturally between the two modalities. Furthermore, different language pairs require fundamentally different amounts of input context before translation can begin, making a single $k$ inherently incapable of optimizing across directions.

Heuristic-Based Adaptive Policies

Methods like EdAtt use attention matrix weights as a heuristic signal for when to write. These are adaptive — they vary decisions based on context — but they don't learn from an explicit optimization objective tied to translation quality. They make reasonable local decisions but are not trained to globally optimize the quality-latency trade-off.

Architecture-Integrated Policies (Monotonic Attention, Transducers)

These approaches bake the streaming decision directly into the model architecture itself. Monotonic Multi-head Attention (MMA) and its enhanced variant EMMA (used in SeamlessM4T) explicitly model monotonic alignment between source audio frames and target text tokens, inherently supporting streaming. Neural Transducer models similarly support streaming through their sequence transduction formulation.

The paper's Appendix D provides a detailed critique based on attempted implementation:

"The train-time computations for EMMA requires computing a matrix of size [batch_size × attention_heads × num_text_tokens × audio_sequence_length × audio_sequence_length] within each cross-attention layer... we require 2GB of VRAM for a single cross-attention layer at batch size 1."

Beyond the computational cost, three specific problems emerge:

Numerical instability: EMMA requires computing a cumulative product across the audio sequence length dimension (typically 500–1500 frames). "A cumulative product of 500 small floating point values is numerically unstable and often results in rounding to 0." A log-sum-exp formulation might help, but isn't part of the original method.
Policy selection ambiguity: EMMA computes a separate policy for every attention head in every cross-attention layer. At inference time, "it is unclear which one to use for the final inference policy." The Seamless codebase requires selecting a layer as an argument and aggregating across heads (max, min, or mean), with the authors finding empirically that "some layers and heads learned useful policies, while the majority did not" — a brittleness that undermines reliability.
Convergence difficulty: The paper's preliminary investigations validated that monotonic attention methods "suffer from both poor numerical stability and difficulty in converging" (Section 2), and their transducer implementation was similarly "very hard to make converge."

Teacher-Guided Synthetic Alignment

Recent works like Hibiki, Fu et al. (2025), and SimulS2S-LLM take a different approach: use an existing teacher model (an NMT system or LLM) to create synthetically aligned training data that directly pairs audio segments with target text tokens, then train a SimulST model on this aligned data without learning an explicit policy. This sidesteps the need for a separate policy network entirely.

The limitation, as the paper identifies in Section 2, is that these systems are "often limited in their streaming performance based on the quality of the teacher model." The synthetic alignment quality serves as an upper bound on the trained model's streaming behavior — if the teacher produces poor alignments, the student inherits those errors without a mechanism to improve upon them.

Decoupled Policy with Reinforcement Learning

Another family of approaches trains a separate policy module using reinforcement learning to directly optimize a quality-latency trade-off metric. While this "simplifies the learning problem by decoupling the policy from the translation model" (Section 2), RL-based policy training faces its own challenges:

"RL is hard to stabilize and efficiently train, especially in cases like SimulST, with no guarantee of convergence."

The sparse reward signal (BLEU score only available at the end of complete translation sequences) combined with the sequential nature of READ/WRITE decisions creates a credit assignment problem that makes RL optimization unreliable.

Divergence-Guided Approaches (DiG-SST)

The method closest to this paper's approach is DiG-SST (Divergence-Guided Simultaneous Speech Translation). DiG-SST trains a lightweight policy module using the expected divergence between output distributions conditioned on partial versus complete input. Specifically, when the decoder's output distribution given partial audio diverges significantly from its distribution given full audio, the policy learns to READ — the intuition being that large divergence indicates the partial context is insufficient for accurate prediction.

The authors identify a critical limitation in DiG-SST's formulation (Section 2):

"DiG-SST's formulation fails to make use of valuable information from ground truth labels when computing divergence scores."

DiG-SST computes divergence between output distributions (which are probability distributions over the entire vocabulary) rather than focusing on the probability specifically assigned to the correct ground-truth token. This means the divergence signal can be high even when the model assigns high probability to the correct token under partial audio — the distributions might differ overall while still agreeing on the correct answer. Conversely, the divergence might be low even when the model is highly uncertain about the correct token — the distributions might be similar overall while both assign low probability to the correct answer. REINA addresses this by directly using the log-probability of the ground-truth token under partial versus full audio, making the signal directly relevant to translation accuracy.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning is multifaceted:

Methodological positioning. REINA belongs to the family of decoupled policy training methods (like DiG-SST and RL-based approaches), where the policy network is trained separately from the frozen translation model. This avoids the architectural complexity and training instability of integrated approaches (monotonic attention, transducers) while maintaining the flexibility to train the policy efficiently. Within this family, REINA's innovation is the use of mutual information rather than distributional divergence as the optimization signal — a shift from "how different are the output distributions?" to "how much does waiting help me predict the correct next token?"

Data and scale positioning. The paper explicitly targets the gap between industry-scale proprietary models and academic-scale open-source models. By training on 130k hours of open-source data using multi-task learning (ASR, NMT, S2TT jointly), the authors aim to demonstrate that strong streaming performance is achievable without privileged access to massive proprietary datasets. The 445M parameter count places REINAStream between small academic models (~100M parameters) and large industry models (>1B parameters).

Evaluation positioning. A significant portion of the paper's contribution is critical: it argues that the standard evaluation methodology in SimulST — plotting Average Lagging (AL) against BLEU and comparing curves — is insufficient because it "does not sufficiently disentangle a model's non-streaming translation quality from its streaming ability" (Section 4.2). The paper observes that many comparisons in the literature falsely attribute higher BLEU-at-a-given-latency to superior streaming policies when the entire difference may be explained by one model simply being a better non-streaming translator. The introduction of Normalized Streaming Efficiency (NoSE) — which divides the area under the AL/BLEU curve by the non-streaming BLEU — is a deliberate attempt to isolate and measure the quality of the streaming policy itself, independent of the underlying translation model's absolute performance.

Theoretical positioning. The paper grounds its approach in information theory, specifically mutual information. This is not merely decorative — the derivation in Section 3.1 walks through how the original policy objective ("wait if and only if we gain information") can be expressed as the difference in conditional entropies $H(s_{n+1}|a_t, S_n) - H(s_{n+1}|a_T, S_n)$ , which in turn can be estimated from the base S2TT model's log-probabilities on partial versus full audio. This derivation connects the practical training problem to a principled information-theoretic quantity, distinguishing REINA from purely heuristic approaches.

Practical positioning. The paper emphasizes that REINA is designed to be practical. Training the policy network (6M parameters) takes under 12 hours on the same hardware used for the base model. The policy is a simple binary classifier (sigmoid activation, output dimension 1) on top of a small transformer encoder applied to decoder hidden states. This stands in stark contrast to the prohibitive computational costs of EMMA and transducer training, as documented in Appendix D. The three-stage training pipeline — (1) non-streaming multi-task training, (2) truncated audio fine-tuning, (3) frozen-base REINA policy training — is modular and could be applied to other base S2TT architectures.

In summary, the paper addresses a clear gap: the need for efficient, stable, and principled methods to train streaming policies for SimulST that work with open-source data and modest compute budgets, while providing reliable evaluation metrics that isolate policy quality from underlying model quality. REINA is positioned as a theoretically grounded, computationally practical alternative to both architecture-integrated approaches (which are expensive and numerically unstable) and divergence-based decoupled approaches (which ignore ground-truth information in their optimization signal).

3. Technical Approach

3.1 Reader Orientation

This paper develops a system for converting a pretrained, non-streaming speech-to-text translation (S2TT) model into a simultaneous speech translation (SimulST) model by training a lightweight policy network that decides, at each step of translation, whether to emit the next target-language token or wait for more input audio. The core idea is deceptively simple: wait for more audio only if doing so actually helps you predict the correct next token — and the paper derives a concrete, computable training objective from this information-theoretic principle, then shows how to approximate it with a small neural network.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, organized into a three-stage training pipeline:

Non-Streaming S2TT Base Model — A Whisper Medium acoustic encoder (307M parameters) feeding into a randomly initialized 16-layer transformer decoder (101M parameters), trained jointly with a T5 text encoder (38M parameters) for auxiliary MT and ASR tasks. This is the full-context translation engine that serves as the foundation for everything that follows. Total: 445M trainable parameters, 408M at inference time when the MT encoder is discarded.
Truncated Audio Fine-Tuning (Stage 2) — The same base model architecture, fine-tuned on a mixture of 20% full audios and 80% randomly truncated audios so that the decoder learns to produce reasonable translations even when the acoustic encoder sees only partial input. This stage is critical: without it, the decoder's log-probabilities on partial audio are poorly calibrated, and the REINA signal derived from them becomes unreliable.
Policy Network (6M parameters) — A small 2-layer transformer encoder applied to the frozen decoder's last-layer hidden states, topped with a single linear layer (output dimension 1, sigmoid activation) that produces a scalar $q_\theta^n$ for each token position. This scalar is trained to correlate with the information gain from waiting for more audio, and at inference time, a threshold $\alpha$ converts it into binary READ/WRITE decisions.
REINA Loss Function — The training objective for the policy network, comprising three terms: a covariance-maximizing policy loss $\mathcal{L}_p$ that encourages $q_\theta^n$ to align with the estimated information gain, a monotonicity regularizer $\mathcal{L}_m$ that biases the policy toward consistent commitment behavior, and an L2 penalty $\mathcal{L}_r$ that prevents $q_\theta$ values from exploding.

Information flows as follows during inference: audio arrives in 0.25-second chunks → the frozen Whisper encoder processes all audio up to the current chunk → the frozen decoder produces hidden states from cross-attending to the encoder outputs → the policy network takes these hidden states and produces a scalar $q_\theta^n$ for each token position in each beam → if $q_\theta^n < \alpha$ , the system READS (waits for the next audio chunk); otherwise, it WRITES (the token with the score above threshold is selected as output, and the beam's log-probability is reset to 0 so it doesn't participate in further decoding).

3.3 Roadmap for the Deep Dive

First, the ideal information-theoretic policy (Equation 1) — what an optimal READ/WRITE decision would look like if we had access to the true mutual information. This grounds the approach and motivates everything that follows.
Second, the REINA loss derivation (Equations 2–5) — how the ideal policy is approximated using the base model's log-probabilities on partial versus full audio, why maximizing covariance with the estimated information gain is the right surrogate objective, and how batch normalization enables a simplified maximization problem.
Third, the regularization terms ( $\mathcal{L}_m$ and $\mathcal{L}_r$ ) — why the raw covariance objective alone produces degenerate policies, and how monotonicity and L2 penalties fix this.
Fourth, the three-stage training pipeline — the non-streaming multi-task pretraining, the truncated audio fine-tuning, and the frozen-base policy training, including all hyperparameters, data mixtures, and design choices.
Fifth, the streaming inference procedure — how the trained policy is used at test time with beam search, patience factors, and threshold sweeping to generate the full AL/BLEU trade-off curve.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core contribution is the REINA loss function for training a decoupled streaming policy, along with the accompanying three-stage training pipeline and the NoSE evaluation metric. The approach converts a frozen non-streaming S2TT model into a SimulST model by training only a small policy network, using the base model's own output probabilities as the supervisory signal.

The Ideal Information-Theoretic Policy

The paper begins not with a practical algorithm, but with a statement of what the optimal READ/WRITE policy should do, expressed in information-theoretic terms. This ideal formulation serves as the target that the practical training procedure aims to approximate.

Formal setup. The system is translating an input audio stream $a$ (of total length $T$ frames) into a target-language token sequence $S = (s_1, s_2, \ldots, s_N)$ , where $N$ is the number of target tokens in the ground-truth translation. At any point during streaming translation, the system has already listened to $t < T$ audio frames and has already emitted $n < N$ tokens $S_n = (s_1, \ldots, s_n)$ . The system must now decide: should it emit token $s_{n+1}$ (WRITE), or should it consume audio frame $t+1$ (READ) before making further decisions?

The core principle. The paper proposes a single, testable criterion:

"we should wait for more audio (i.e. READ) if and only if we gain information by doing so"

Information, in Shannon's sense, is measured by how much the uncertainty about the next token $s_{n+1}$ decreases when we know the full audio $a_T$ compared to when we know only the partial audio $a_t$ . This is precisely the mutual information definition.

The information gain quantity. The paper defines $\mathcal{F}(a, S, n, t)$ as the information gained about the next token by waiting for the full audio:

$\mathcal{F}(a, S, n, t) := I(s_{n+1}; a_T, S_n) - I(s_{n+1}; a_t, S_n)$

where $I(X; Y)$ is the mutual information between $X$ and $Y$ (measured in bits if using log-base-2, or nats if using natural log), $a_T$ is the complete audio of length $T$ , $a_t$ is the partial audio up to frame $t$ , $s_{n+1}$ is the next ground-truth token the model should predict, and $S_n = (s_1, \ldots, s_n)$ is the sequence of previously emitted tokens.

What it computes: $\mathcal{F}$ measures how much more information the full audio provides about the next token compared to the partial audio, given the context of previously emitted tokens. If $\mathcal{F}$ is large, waiting for the full audio substantially reduces uncertainty about $s_{n+1}$ — the partial audio alone leaves us guessing, and the extra context resolves that ambiguity. If $\mathcal{F}$ is small (near zero), the partial audio already provides essentially all the information the full audio would — waiting gains us nothing.

From information gain to a policy. Given this quantity, the ideal streaming policy $\pi_\alpha$ is trivially defined by thresholding:

$\pi_\alpha(a, S, n, t)$ returns READ when $\mathcal{F}(a, S, n, t) > \alpha$
$\pi_\alpha(a, S, n, t)$ returns WRITE when $\mathcal{F}(a, S, n, t) \leq \alpha$

where $\alpha$ is a user-chosen threshold that controls the quality-latency trade-off: higher $\alpha$ requires more information gain before the system deems waiting worthwhile, making it read more aggressively (higher quality, higher latency); lower $\alpha$ makes the system write more eagerly (lower latency, potentially lower quality). In practice, $\alpha$ is swept across multiple values at inference time to produce the full AL/BLEU trade-off curve (reported thresholds vary by dataset and model variant; e.g., for REINA on MUST-C, $\alpha$ is swept across [0.935, 0.940, 0.9425, 0.945, 0.9475, 0.950]).

Why this form: This formulation is not heuristic — it flows directly from the definition of mutual information as the reduction in uncertainty. The threshold $\alpha$ has a natural interpretation: it is the minimum number of bits of new information about the next token that the system requires before it is willing to incur the latency cost of waiting. A policy that ignores this quantity (e.g., wait-k) is implicitly assuming a constant information gain per time step, which is demonstrably false across different syntactic structures and language pairs. A policy based on distributional divergence (like DiG-SST) measures how much the model's overall output distribution changes, but not specifically whether that change actually helps predict the correct token — it conflates general model uncertainty with token-specific information gain.

The fundamental problem. The quantity $\mathcal{F}(a, S, n, t)$ requires the ground-truth next token $s_{n+1}$ to compute, and it requires access to the true probability distributions $p(s_{n+1} | a_T, S_n)$ and $p(s_{n+1} | a_t, S_n)$ — neither of which is available at test time, and the former isn't available even at training time. The remainder of the technical approach is about (1) estimating $\mathcal{F}$ from the base S2TT model's outputs, and (2) training a small neural network to predict this estimate from context that is available at test time.

From Mutual Information to a Computable Estimate

The paper's key algebraic manipulation converts the mutual information difference into a form that can be estimated from the base S2TT model's output log-probabilities.

Step 1: Expand mutual information in terms of entropy. Recall that mutual information between $X$ and $Y$ can be written as $I(X; Y) = H(X) - H(X|Y)$ , where $H$ is Shannon entropy. Applying this to both terms in $\mathcal{F}$ :

$\mathcal{F}(a, S, n, t) = I(s_{n+1}; a_T, S_n) - I(s_{n+1}; a_t, S_n)$

$= [H(s_{n+1}) - H(s_{n+1} | a_T, S_n)] - [H(s_{n+1}) - H(s_{n+1} | a_t, S_n)]$

where $H(s_{n+1})$ is the unconditional entropy of the next token (a constant that cancels out), $H(s_{n+1} | a_T, S_n)$ is the conditional entropy given full audio and previous tokens, and $H(s_{n+1} | a_t, S_n)$ is the conditional entropy given partial audio and previous tokens.

Step 2: Cancel the common term. The unconditional entropy $H(s_{n+1})$ appears in both terms and cancels:

$\mathcal{F}(a, S, n, t) = H(s_{n+1} | a_t, S_n) - H(s_{n+1} | a_T, S_n)$

This is intuitive: the information gained by waiting equals the residual uncertainty in the next token given only partial audio, minus the residual uncertainty given full audio. If the partial audio leaves you nearly as uncertain as the full audio would, $\mathcal{F}$ is small; if the partial audio leaves you much more uncertain, $\mathcal{F}$ is large.

Step 3: Express conditional entropy as an expectation over log-probabilities. By definition, $H(s_{n+1} | a_t, S_n) = -\mathbb{E}_{s_{n+1}}[\log p(s_{n+1} | a_t, S_n)]$ , where the expectation is taken over the true distribution of $s_{n+1}$ . Since we observe exactly one ground-truth token per training example, the empirical expectation collapses to the negative log-probability of the ground-truth token under the model:

$\mathcal{F}(a, S, n, t) = \mathbb{E}\left[\log p(s_{n+1} | a_T, S_n) - \log p(s_{n+1} | a_t, S_n)\right]$

In plain language: the information gained by waiting for the full audio equals, on average, how much higher the log-probability of the correct next token is when we condition on the full audio compared to when we condition on only the partial audio.

Step 4: Replace true probabilities with model estimates. We do not have access to the true $p(s_{n+1} | a_T, S_n)$ or $p(s_{n+1} | a_t, S_n)$ , but we have the frozen S2TT model, which provides estimates $\hat{p}$ of these probabilities via its output distribution. The estimated information gain is:

$\hat{\mathcal{F}}(a, S, n, t) = \log \hat{p}(s_{n+1} | a_T, S_n) - \log \hat{p}(s_{n+1} | a_t, S_n)$

where $\log \hat{p}(s_{n+1} | a_T, S_n)$ is the log-probability the frozen S2TT model assigns to the ground-truth next token $s_{n+1}$ when it sees the full audio $a_T$ and the previously emitted tokens $S_n$ , and $\log \hat{p}(s_{n+1} | a_t, S_n)$ is the log-probability it assigns to the same token when it sees only the partial audio $a_t$ (with the same $S_n$ ).

Notational shorthand in the paper. The paper introduces a compact notation: $\log \hat{p_T}^{s_{n+1}} = \log \hat{p}(s_{n+1} | a_T, S_n)$ for the full-audio log-probability, and $\log \hat{p_t}^{s_{n+1}} = \log \hat{p}(s_{n+1} | a_t, S_n)$ for the partial-audio log-probability. Then $\hat{\mathcal{F}} = \log \hat{p_t}^{s_{n+1}} - \log \hat{p_T}^{s_{n+1}}$ .

Crucial observation. These log-probabilities are exactly the negative of the cross-entropy losses the S2TT model incurs when predicting the ground-truth sequence on full and partial audio respectively. Specifically, if the S2TT decoder is trained with standard cross-entropy loss, then for token $s_{n+1}$ , the loss on full audio is $\text{CE}_{\text{full}} = -\log \hat{p}(s_{n+1} | a_T, S_n)$ , and the loss on partial audio is $\text{CE}_{\text{partial}} = -\log \hat{p}(s_{n+1} | a_t, S_n)$ . Therefore:

$\hat{\mathcal{F}} = \text{CE}_{\text{full}} - \text{CE}_{\text{partial}}$

This is why Figure 1 shows $\mathcal{L}_p$ as computed from the difference of cross-entropy terms: the information gain estimate is simply how much lower the cross-entropy loss is (on the ground-truth token) when the model has the full audio versus only the partial audio.

Why this form is valid (and why it's not trivial): The estimate $\hat{\mathcal{F}}$ relies on the base S2TT model's probabilities being reasonably well-calibrated on partial inputs — which is precisely why Stage 2 (truncated audio fine-tuning) exists. Before truncated training, a model trained only on full audios will produce essentially random log-probabilities when fed partial audios, making $\hat{\mathcal{F}}$ a noisy and misleading signal. After truncated training, the model has learned to produce coherent (if not always correct) probability distributions on partial inputs, and the difference $\log \hat{p}(s_{n+1}|a_T, S_n) - \log \hat{p}(s_{n+1}|a_t, S_n)$ becomes a meaningful measure of how much the extra audio context helps predict the correct answer.

What this estimate tells us: For each token position $n$ in each training example, we compute exactly one scalar: the difference between the log-probability of the correct token under full context and under partial context. If the partial-audio log-probability is nearly as high as the full-audio log-probability (difference near zero), waiting didn't help — the model was already confident about the correct token with the information it had. If the partial-audio log-probability is much lower (large positive difference), waiting substantially increased the model's confidence in the correct token — the model needed the extra context to figure out what to say. This scalar is the target signal that the policy network will be trained to predict.

The REINA Policy Loss: Maximizing Covariance with Information Gain

The estimate $\hat{\mathcal{F}}$ cannot be used at inference time because it requires knowledge of the ground-truth next token $s_{n+1}$ (we only know this at training time) and access to the full audio $a_T$ (we don't have it during streaming). The solution is to train a small neural network — the policy network — to produce a scalar $q_\theta$ that correlates strongly with $\hat{\mathcal{F}}$ , using only information available at test time: the partial audio $a_t$ , the previously emitted tokens $S_n$ , and the decoder's hidden states (which implicitly encode the model's current uncertainty).

The optimization objective: maximizing covariance. Rather than training $q_\theta$ to directly predict $\hat{\mathcal{F}}$ via, say, mean-squared error regression, the paper formulates the objective as maximizing the covariance between $q_\theta$ and $\hat{\mathcal{F}}$ :

$\max_\theta \left( \text{Cov}(q_\theta, \hat{\mathcal{F}}(a, S, n, t)) \right)$

$= \max_\theta \left( \mathbb{E}\left[q_\theta \cdot \hat{\mathcal{F}}(a, S, n, t)\right] - \mathbb{E}[q_\theta] \cdot \mathbb{E}[\hat{\mathcal{F}}(a, S, n, t)] \right)$

where the expectations are taken over the training distribution of $(a, S, n, t)$ tuples, $q_\theta$ is the scalar output of the policy network parameterized by $\theta$ for a given context, and $\hat{\mathcal{F}}(a, S, n, t)$ is the estimated information gain (which is a fixed scalar for each training example, computed once from the frozen S2TT model and not backpropagated through).

What this objective means, operationally: The covariance $\text{Cov}(X, Y)$ measures the degree to which $X$ and $Y$ deviate from their means together. When $X$ is high, is $Y$ also high? When $X$ is low, is $Y$ also low? Maximizing covariance means: train $q_\theta$ so that when $\hat{\mathcal{F}}$ is large (waiting for more audio substantially helped predict the correct token), $q_\theta$ tends to also be large, and when $\hat{\mathcal{F}}$ is small (waiting didn't help), $q_\theta$ tends to also be small. Critically, we don't care about the scale of $q_\theta$ — only that it covaries with the information gain. The threshold $\alpha$ will be chosen post-hoc to map $q_\theta$ values to binary decisions.

Why covariance and not regression? If we trained $q_\theta$ to directly predict $\hat{\mathcal{F}}$ (minimizing $||q_\theta - \hat{\mathcal{F}}||^2$ ), we would be imposing a specific numerical scale on $q_\theta$ , which is unnecessary — the policy only needs a monotone relationship between $q_\theta$ and $\hat{\mathcal{F}}$ , not an exact numerical match. Covariance maximization captures exactly this requirement without unnecessary constraints on the scale. Additionally, $\hat{\mathcal{F}}$ is an estimate derived from a frozen model's probabilities, which may be miscalibrated in absolute terms; requiring exact numerical prediction would force $q_\theta$ to replicate those miscalibrations. Covariance only requires that $q_\theta$ moves in the same direction as $\hat{\mathcal{F}}$ , which is more robust to miscalibration.

Simplifying with batch normalization. The covariance expression contains the term $\mathbb{E}[q_\theta] \cdot \mathbb{E}[\hat{\mathcal{F}}]$ , which is inconvenient to compute and optimize. The paper's elegant insight: if we normalize $\hat{\mathcal{F}}$ to have zero mean across each training batch, then $\mathbb{E}[\hat{\mathcal{F}}] = 0$ within that batch, and the second term vanishes regardless of $\mathbb{E}[q_\theta]$ . The paper achieves this by applying batch normalization (subtracting the batch mean and dividing by the batch standard deviation) to the $\hat{\mathcal{F}}$ values within each training batch. The normalization is not applied to the network output $q_\theta$ , only to the target signal $\hat{\mathcal{F}}$ .

After batch normalization, the optimization simplifies to:

$\max_\theta \left( \mathbb{E}\left[q_\theta \cdot \text{BN}(\hat{\mathcal{F}}(a, S, n, t))\right] \right)$

where $\text{BN}(x)$ denotes the batch-normalized version of $x$ (zero mean, unit variance across the batch). Maximizing this expectation is equivalent to minimizing its negative, yielding the policy loss:

$\mathcal{L}_p = \frac{1}{N} \sum_{n=0}^{N-1} q_\theta^n \cdot \text{BN}\left[\log \hat{p_t}^{s_{n+1}} - \log \hat{p_T}^{s_{n+1}}\right]$

where the sum runs over all $N$ target tokens in the training example (indices $n = 0, 1, \ldots, N-1$ ), $q_\theta^n = q_\theta(a, S, n, t | \theta)$ is the policy network's scalar output for token position $n$ (the network sees the decoder hidden state at position $n$ , the previously emitted tokens, and the partial audio up to $t$ ), and $\text{BN}[\cdot]$ denotes batch normalization of the information gain estimate $\log \hat{p_t}^{s_{n+1}} - \log \hat{p_T}^{s_{n+1}}$ across all tokens in the current training batch.

What $\mathcal{L}_p$ computes, token by token: For each target token position $n$ in each training example, we: (a) compute the full-audio log-probability $\log \hat{p_T}^{s_{n+1}}$ by passing the complete audio through the frozen encoder and decoder and extracting the logit for the ground-truth token $s_{n+1}$ at position $n$ ; (b) compute the partial-audio log-probability $\log \hat{p_t}^{s_{n+1}}$ by passing a truncated version of the audio (length $t < T$ ) through the same frozen model; (c) take their difference to get the estimated information gain; (d) batch-normalize this difference across all tokens in the batch; (e) compute the policy network's output $q_\theta^n$ from the decoder hidden state at position $n$ (the hidden state from the partial-audio forward pass); (f) multiply the batch-normalized information gain by $q_\theta^n$ ; (g) average this product across all $N$ tokens in the example. The loss is minimized when $q_\theta^n$ is strongly and consistently correlated with the batch-normalized information gain.

In Figure 1, this is represented as the "Policy Loss (REINA)" box that takes as input the cross-entropy losses from both the full and partial audio forward passes, computes their difference, batch-normalizes, and multiplies by $q_\theta$ .

Why this loss function works: The multiplication $q_\theta^n \cdot \text{BN}[\hat{\mathcal{F}}]$ is positive when $q_\theta^n$ and $\hat{\mathcal{F}}$ have the same sign (both positive or both negative), and negative when they have opposite signs. Minimizing $\mathcal{L}_p$ therefore pushes $q_\theta^n$ toward having the same sign as $\hat{\mathcal{F}}$ across all tokens: when $\hat{\mathcal{F}}$ is large (positive after normalization), $q_\theta^n$ is pushed upward; when $\hat{\mathcal{F}}$ is small (negative after normalization), $q_\theta^n$ is pushed downward. The batch normalization ensures that the target signal is centered at zero, meaning roughly half the tokens in each batch have positive normalized information gain and half have negative — providing a balanced training signal.

An important detail: the policy network sees partial-audio context only. When computing $q_\theta^n$ , the policy network takes as input the decoder hidden states produced during the partial-audio forward pass (the same forward pass used to compute $\log \hat{p_t}^{s_{n+1}}$ ). This is critical: at inference time, the system only has partial audio, so $q_\theta^n$ must be computable from partial-audio context alone. The full-audio log-probability $\log \hat{p_T}^{s_{n+1}}$ is used only to compute the training target $\hat{\mathcal{F}}$ , not as input to the policy network.

Regularization Terms: Monotonicity and L2 Penalties

The raw policy loss $\mathcal{L}_p$ alone yields a $q_\theta$ that correlates with information gain, but the resulting policy can behave pathologically at inference time. Two specific problems arise, and the paper addresses each with a targeted regularization term.

Problem 1: Non-monotonic commitment. At inference time, the streaming procedure works as follows: for a given beam, the policy network produces a sequence of scalars $q_\theta^1, q_\theta^2, \ldots$ for the tokens in that beam. The system compares each $q_\theta^n$ to the threshold $\alpha$ . The first time $q_\theta^n \geq \alpha$ , the system commits to READ — it stops emitting tokens and waits for more audio. Importantly, once a READ is triggered, no further tokens are emitted from that beam until after more audio arrives. Therefore, the ordering of $q_\theta$ values across token positions matters: if $q_\theta^3 < \alpha$ (WRITE at position 3) but $q_\theta^2 \geq \alpha$ (READ at position 2), the system never gets to position 3 — it already READ at position 2.

In the raw $\mathcal{L}_p$ objective, there is no constraint on the ordering of $q_\theta$ values. The network could learn to produce $q_\theta$ values that oscillate: high at position 1, low at position 2, high at position 3, etc. At training time, this doesn't matter because we evaluate the loss independently at each position. But at inference time, a READ at an early position precludes WRITEs at later positions, even if those later positions would have had high $q_\theta$ values (indicating they should have been emitted). This creates a distributional mismatch between training and inference — the network is trained assuming all positions are independently evaluated, but at test time, decisions are sequential with hard commitment.

Solution: the monotonicity loss $\mathcal{L}_m$ . The paper introduces a regularization term that encourages $q_\theta$ values to be approximately non-decreasing across token positions. The idea is that once the information gain crosses the threshold at some position, all subsequent positions should also be above the threshold — the system should not flip-flop between WRITE and READ. The loss is:

$\mathcal{L}_m = \frac{1}{N} \sum_{n=1}^{N} \left[ \max\left( \max_{m < n} \{ q_\theta^m \} - q_\theta^n - \epsilon, 0 \right) \right]$

where the outer sum runs over token positions $n = 1, 2, \ldots, N$ (starting from position 1, since position 0 has no predecessors), $\max_{m < n} \{ q_\theta^m \}$ is the maximum $q_\theta$ value among all earlier token positions $m < n$ , $q_\theta^n$ is the $q_\theta$ at the current position $n$ , and $\epsilon$ is a small slack constant (set to $0.5$ in all experiments) that allows minor violations of strict monotonicity.

What $\mathcal{L}_m$ computes, position by position: For each token position $n$ , find the highest $q_\theta$ value among all earlier positions ( $m < n$ ). If the current position's $q_\theta^n$ is lower than that maximum minus a tolerance $\epsilon$ , the loss is positive — we are penalizing the network for producing a $q_\theta^n$ value that is significantly lower than an earlier value. If $q_\theta^n$ is equal to or higher than $\max_{m<n} \{ q_\theta^m \} - \epsilon$ , the loss is zero — the monotonicity constraint is satisfied at this position.

Concrete example. Suppose for a 4-token sequence, the policy network produces $q_\theta = [0.3, 0.7, 0.4, 0.8]$ . At position 1 ( $n=1$ ): $\max_{m<1}$ is empty, so no penalty (the sum starts at $n=1$ , but in practice this position has no predecessors; the loss is computed for $n=1$ using position 0 as the predecessor, though the paper's notation starts the sum at $n=1$ ). The main effect is at position 3: $\max_{m<3}\{q_\theta^m\} = 0.7$ (from position 2), and $q_\theta^3 = 0.4$ . With $\epsilon = 0.5$ , the penalty is $\max(0.7 - 0.4 - 0.5, 0) = \max(-0.2, 0) = 0$ — the slack $\epsilon = 0.5$ allows this small dip. Without the slack, the penalty would be $\max(0.7 - 0.4, 0) = 0.3$ , pushing the network to increase $q_\theta^3$ above 0.7.

Why this form: The "soft" monotonicity (allowing violations up to $\epsilon$ ) is deliberate. The paper describes it as a "weak monotonicity constraint" — it biases the policy toward commitment behavior without rigidly enforcing that $q_\theta$ must be strictly increasing. This flexibility matters because information gain can genuinely fluctuate: a particular token might be easy to predict early in the sequence (low information gain, low $q_\theta$ ), and a later token might genuinely benefit from more context (high information gain, high $q_\theta$ ). The $\epsilon$ slack allows these fluctuations within a bounded range, while still preventing the extreme oscillation that would make the policy unusable at inference time.

The effect on policy behavior. With monotonicity regularization, the trained policy tends to produce $q_\theta$ values that generally increase across token positions, with occasional small dips. At inference time, once a beam's $q_\theta$ crosses $\alpha$ , it tends to stay above $\alpha$ — the system commits to READ and doesn't flip-flop. The paper's ablation in Figure 3 confirms this effect: the monotonicity term provides the most benefit at low latencies (aggressive streaming), where the gap between REINA and "REINA w/o monotonicity" is 19% in AL at a fixed BLEU of 35.

Problem 2: Unbounded $q_\theta$ growth. The covariance maximization objective $\mathcal{L}_p$ encourages $q_\theta$ to be large when $\hat{\mathcal{F}}$ is large and small when $\hat{\mathcal{F}}$ is small. But the objective has no mechanism to prevent $q_\theta$ from growing without bound: if doubling $q_\theta$ doubles the covariance (because $\hat{\mathcal{F}}$ is fixed), the network can trivially reduce the loss by making $q_\theta$ arbitrarily large for positive- $\hat{\mathcal{F}}$ examples and arbitrarily negative for negative- $\hat{\mathcal{F}}$ examples. In practice, this causes $q_\theta$ to explode to extreme values, making the policy hypersensitive to the threshold $\alpha$ and numerically unstable.

Solution: L2 regularization $\mathcal{L}_r$ . A simple L2 penalty on the $q_\theta$ values:

$\mathcal{L}_r = \frac{1}{N} \sum_{n=1}^{N} (q_\theta^n)^2$

where $q_\theta^n$ is the policy network output at token position $n$ , and the sum averages the squared values across all $N$ token positions. This penalty discourages $q_\theta$ from growing large in absolute value, keeping the learned scalars within a reasonable range.

The full REINA loss is the sum of all three terms:

$\mathcal{L}_{\text{REINA}} = \mathcal{L}_p + \mathcal{L}_m + \lambda \mathcal{L}_r$

where $\lambda = 0.05$ is the weight on the L2 regularization term. The paper reports that "final model performance is not very sensitive to changes in $\lambda$ " — the L2 term serves mainly as a guardrail against numerical instability rather than a finely tuned hyperparameter.

The Three-Stage Training Pipeline

The paper does not train the full system end-to-end. Instead, it proceeds in three sequential stages, each with a distinct purpose, dataset mixture, and set of frozen versus trainable parameters. This modular design is one of REINA's practical strengths: each stage can be debugged and validated independently.

Stage 1: Non-Streaming Multi-Task Training

Purpose. Train a strong non-streaming S2TT model that will later serve as the frozen base for policy training. The model must produce high-quality translations on full audio and learn internal representations that generalize well across tasks. Multi-task learning (MTL) is used to compensate for the relative scarcity of parallel S2TT data compared to ASR and MT data.

Architecture. Three components, all trained from scratch:

Acoustic encoder: Whisper Medium (307M parameters), initialized from the pretrained Whisper Medium checkpoint. The encoder processes raw audio (log-mel spectrograms) and produces a sequence of hidden states. The encoder weights are not frozen during training — they are fine-tuned on the target data.
Text decoder: A randomly initialized 16-layer transformer with model dimension 512, 8 attention heads, feedforward multiplier 4, label smoothing 0.1, and dropout rate 0.1 (101M parameters). The decoder performs cross-attention over the acoustic encoder's final layer hidden states for S2TT and ASR tasks, and over the MT encoder's hidden states for the NMT task. It uses the Mistral 7B multilingual tokenizer but learns its own embedding dictionary from scratch. Language IDs (e.g., <en>, <fr>, <de>, <es>) are added to the vocabulary and prefixed to token sequences to direct the decoder to produce output in a specific target language.
MT text encoder: A randomly initialized T5 text encoder (38M parameters). This encoder processes source-language text (not speech) for the NMT auxiliary task. During NMT training, the decoder cross-attends to this encoder's hidden states instead of the acoustic encoder's.

Learned positional encoding. The acoustic encoder outputs are augmented with a learned positional encoding "similar to Time2Vec" to provide the decoder with temporal ordering information — Whisper's encoder doesn't naturally preserve explicit positional information in a form the decoder can easily exploit.

Training tasks and loss functions. Each training batch contains a mixture of samples supporting three tasks:

ASR (Automatic Speech Recognition): For samples containing $(a_i, l_i^s, T_i^s)$ (audio, source language ID, source transcription), pass audio $a_i$ through the acoustic encoder, decode to tokens in source language $l_i^s$ , compute cross-entropy loss $\mathcal{L}_{asr}$ against the ground-truth transcription $T_i^s$ .
NMT (Neural Machine Translation): For samples containing $(T_i^s, l_i^t, T_i^t)$ (source text, target language ID, target text), pass source text $T_i^s$ through the T5 text encoder, decode to tokens in target language $l_i^t$ , compute cross-entropy loss $\mathcal{L}_{nmt}$ against the ground-truth translation $T_i^t$ .
S2TT (Speech-to-Text Translation): For samples containing $(a_i, l_i^t, T_i^t)$ (audio, target language ID, target text), pass audio $a_i$ through the acoustic encoder, decode to tokens in target language $l_i^t$ , compute cross-entropy loss $\mathcal{L}_{s2tt}$ against the ground-truth translation $T_i^t$ .

All three losses are summed: $\mathcal{L} = \mathcal{L}_{asr} + \mathcal{L}_{nmt} + \mathcal{L}_{s2tt}$ . There is no task-specific weighting — each loss contributes equally.

Training configuration. Stage 1 trains for 5 days on 24 A100-80GB GPUs with:

Optimizer: AdamW
Learning rate: fixed $10^{-4}$ (no schedule)
Weight decay: $10^{-4}$
Gradient clipping: 10.0
Effective batch size: 768

Data mixture. The dataset mixing ratios control how frequently each data source appears in a training batch:

MUST-C: ratio 1
CVSS: ratio 1
MLS (en → X): ratio 2
MLS (X → en): ratio 4
CCMatrix: ratio 4
Mosel: ratio 4

Higher ratios mean the dataset is sampled more frequently. The total training data encompasses approximately 130k hours of audio plus 60M text-to-text MT samples (10M per language pair from CCMatrix). MLS data is augmented by translating its transcripts using "an in-house NMT model" to produce S2TT training pairs, expanding the available S2TT data beyond the relatively small MUST-C and CVSS datasets.

Stage 2: Truncated Audio Fine-Tuning

Purpose. Adapt the decoder to produce reasonable log-probabilities when conditioned on partial audio. The Stage 1 model was trained only on full audios; if we were to compute $\log \hat{p}(s_{n+1} | a_t, S_n)$ directly from this model, the log-probabilities on truncated inputs would be essentially random, because the decoder has never seen the acoustic encoder produce hidden states from partial audio. The resulting $\hat{\mathcal{F}}$ estimates would be dominated by noise rather than genuine information gain.

Procedure. The same model architecture and optimizer configuration from Stage 1 is used, but the training data is modified: 20% of audios are left at their full length, and 80% are randomly truncated to some length $t < T$ . The truncation point is chosen randomly for each sample. The model is fine-tuned for 2 days using the same loss function $\mathcal{L} = \mathcal{L}_{asr} + \mathcal{L}_{nmt} + \mathcal{L}_{s2tt}$ , with the same data mixture ratios.

What this teaches the model. By seeing truncated audios during training, the encoder learns to produce meaningful representations even from incomplete acoustic input, and the decoder learns to assign calibrated probabilities given those partial representations. After Stage 2, $\log \hat{p}(s_{n+1} | a_t, S_n)$ is a meaningful quantity: it reflects the model's actual best guess of the next token given the partial audio, not an artifact of distribution shift. The 20% full-audio data prevents catastrophic forgetting — the model retains its ability to produce high-quality translations on complete inputs.

Ablation evidence. Table 4 demonstrates the importance of this stage: "REINA w/o truncated training" (trained only on MUST-C, skipping Stage 2) shows degraded NoSE scores compared to REINA (MUST-C only) with truncated training. The paper concludes that "the mutual information formulation of REINA requires a good estimate of $\log \hat{p}(s_{n+1} | a_t, S_n)$ ," and Stage 2 is what provides that good estimate.

Stage 3: REINA Policy Training (Frozen Base)

Purpose. Train the policy network using the REINA loss while keeping the entire base S2TT model frozen. Only the 6M-parameter policy network is updated.

Policy network architecture. A 2-layer transformer encoder (6M parameters) with:

Embedding dimension: 512
Attention heads: 4
Feedforward multiplier: 4
Causal attention mask (the network cannot look ahead to future tokens when producing $q_\theta^n$ for position $n$ )
Output: a single linear layer (output dimension 1, sigmoid activation) that maps the transformer's final hidden state at each position to a scalar $q_\theta^n \in [0, 1]$

The policy network takes as input the last-layer hidden states from the frozen decoder during the partial-audio forward pass. This means the policy network sees the same representations the decoder uses to predict the next token — it has access to the model's internal uncertainty and contextual information.

Training configuration. Stage 3 uses:

Optimizer: AdamW
Learning rate schedule: inverse square root with 5,000 warmup steps
Training duration: 20 epochs, completing in under 12 hours
Frozen parameters: all base model weights (Whisper encoder, text decoder, MT encoder)
Trainable parameters: policy network only (6M)

Data mixture adjustment. The mixing ratios are changed for this stage to emphasize the datasets most relevant to the streaming task:

MUST-C: ratio 2 (increased from 1 in Stage 1)
CVSS: ratio 2 (increased from 1)
MLS (en → X): ratio 6 (increased from 2)
MLS (X → en): ratio 6 (increased from 4)
CCMatrix and Mosel: not used (these are text-only or have different characteristics)

The paper trains only on S2TT data samples during Stage 3, because "our goal is to learn a policy best for streaming speech translation rather than streaming ASR or MT." The REINA loss requires the full-audio and partial-audio log-probabilities for the ground-truth target tokens, which are only available for S2TT samples where we have aligned (audio, target text) pairs.

REINA loss hyperparameters:

Monotonicity slack: $\epsilon = 0.5$
L2 regularization weight: $\lambda = 0.05$

How the training target is computed during Stage 3. For each S2TT training sample:

Partial audio forward pass: The audio is truncated to some length $t$ (the truncation point can vary across training steps; the paper doesn't specify whether a fixed $t$ or random $t$ is used during Stage 3, but the truncation mechanism from Stage 2 is presumably maintained). The truncated audio is fed through the frozen acoustic encoder → frozen decoder, producing hidden states $\mathbf{h}_n^{\text{partial}}$ and token logits at each position $n$ . The log-probability of the ground-truth token $s_{n+1}$ at position $n$ is extracted: $\log \hat{p_t}^{s_{n+1}} = \log \text{softmax}(\text{logits}_n^{\text{partial}})_{s_{n+1}}$ .
Full audio forward pass: The same audio, at full length $T$ , is fed through the same frozen model, producing $\log \hat{p_T}^{s_{n+1}}$ .
Information gain estimate: $\hat{\mathcal{F}}^n = \log \hat{p_t}^{s_{n+1}} - \log \hat{p_T}^{s_{n+1}}$ is computed for each position $n$ .
Batch normalization: The $\hat{\mathcal{F}}^n$ values across all token positions in the current batch are normalized to zero mean and unit variance.
Policy network forward pass: The decoder hidden states from the partial-audio pass, $\mathbf{h}_n^{\text{partial}}$ , are fed into the policy network, producing $q_\theta^n$ for each position.
Loss computation: $\mathcal{L}_p = \frac{1}{N} \sum_{n=0}^{N-1} q_\theta^n \cdot \text{BN}[\hat{\mathcal{F}}^n]$ , plus monotonicity and L2 terms.
Backpropagation: Gradients flow only into the policy network parameters $\theta$ ; the base model is frozen (no gradients through the decoder or encoder).

Why freezing the base model is essential. If the base model were fine-tuned during Stage 3, the information gain estimates $\hat{\mathcal{F}}^n$ would be a moving target — as the model's probabilities changed, the target signal would shift, making the optimization unstable. Freezing the base model ensures $\hat{\mathcal{F}}^n$ is a fixed, consistent target for the policy network. This is a deliberate design choice that trades off potential improvements from joint fine-tuning for training stability and modularity.

Streaming Inference with the Trained Policy

At inference time, the system performs streaming beam search over incoming audio chunks.

Audio chunking. Input audio is split into fixed-size chunks of 0.25 seconds (the standard configuration; the paper doesn't specify whether this is 250ms of audio at a specific sample rate, but Whisper typically operates on 16kHz audio, so 0.25s corresponds to 4000 samples). After each chunk arrives, the system runs the acoustic encoder on all audio received so far (cumulative encoding, not incremental — the encoder sees the full concatenation of all chunks up to the current point, not just the latest chunk). The encoder produces hidden states for the entire partial audio, and the decoder produces output distributions for all token positions.

Beam search procedure. The system maintains a beam of hypotheses (beam size 3 in all experiments). For each beam, at each token position:

The decoder produces token logits given the partial audio and the previously emitted tokens in that beam.
The policy network takes the decoder's hidden state at that position and produces $q_\theta^n$ .
If $q_\theta^n \geq \alpha$ (the policy says WRITE), the token with the highest log-probability in the beam is emitted, and the beam's log-probability is updated accordingly.
If $q_\theta^n < \alpha$ (the policy says READ), the tokens predicted so far in that beam are saved as a hypothesis, and the beam's log-probability is reset to 0 — meaning this beam stops participating in the current round of decoding. The system will wait for the next audio chunk and restart beam search from scratch on all accumulated audio, but with the previously emitted tokens from this beam as context (the exact mechanism for continuing beams across audio chunks is not fully specified in the paper, but this is the standard streaming beam search protocol).

Patience-based termination. The system doesn't immediately end the search as soon as one beam triggers a READ. Instead, it uses a patience factor (set to 3 in all experiments): search continues until either (a) the total number of beams that have encountered a READ action exceeds beam_size × patience_factor (i.e., $3 \times 3 = 9$ beams have READ), or (b) all beams hit READ simultaneously. Once either condition is met, the search ends, and the hypothesis with the highest average log-probability (total log-probability divided by sequence length, to avoid length bias) is returned.

End-of-audio handling. When the system reaches the end of the input audio (all chunks have been consumed), it stops using the policy network entirely. The remaining hypothesis continuation proceeds with standard beam search (without READ/WRITE decisions) until an EOS token is produced, using the same patience factor logic to determine when all beams have finished.

Threshold sweeping. The threshold $\alpha$ controls the quality-latency trade-off: low $\alpha$ means $q_\theta^n$ crosses the threshold quickly (more WRITE, lower latency, potentially lower quality), high $\alpha$ means the system READS more aggressively (higher latency, potentially higher quality). To evaluate the full Pareto frontier, the paper sweeps $\alpha$ across multiple values. The specific thresholds differ per model variant and dataset, determined through "trial and error" (Appendix B):

REINA (all data) on MUST-C: [0.97, 0.975, 0.976, 0.977, 0.978, 0.979]
REINA (MUST-C only): [0.935, 0.940, 0.9425, 0.945, 0.9475, 0.950]
REINA on CVSS-C: [0.97, 0.975, 0.976, 0.977, 0.978, 0.979, 0.980, 0.983, 0.985, 0.987]
REINA w/o monotonicity on CVSS-C: [0.976, 0.977, 0.978, 0.979, 0.980, 0.983, 0.985, 0.987]

The thresholds cluster near 1.0 because $q_\theta$ is bounded to $[0, 1]$ by the sigmoid activation, and the network learns to produce values near the upper end of this range for tokens where information gain is high. The tight spacing (0.0025–0.005 increments) is necessary to sample the quality-latency trade-off finely enough to construct smooth AL/BLEU curves.

Summary of Design Choices and Their Justifications

Covariance maximization over regression: Covariance only requires $q_\theta$ to co-vary with information gain, not to match its numerical scale, which is more robust to miscalibration in the frozen model's probability estimates and avoids imposing unnecessary scale constraints on the policy network output.
Batch normalization of the target, not the prediction: Normalizing $\hat{\mathcal{F}}$ to zero mean eliminates the cross-term $\mathbb{E}[q_\theta] \cdot \mathbb{E}[\hat{\mathcal{F}}]$ in the covariance without constraining $q_\theta$ , simplifying the optimization to a straightforward inner product minimization.
Ground-truth token log-probability over distributional divergence: DiG-SST uses the divergence between entire output distributions (e.g., KL divergence), which can be high even when the correct token is confidently predicted under partial audio. REINA's use of the log-probability of the specific ground-truth token makes the signal directly relevant to translation accuracy — the policy learns to READ when waiting actually helps predict the right word, not just when the model's uncertainty changes.
Frozen base model during policy training: Prevents the information gain estimates from becoming a moving target during optimization, ensuring stable and efficient training. Also keeps the base model unchanged, preserving its non-streaming translation quality.
Three-stage pipeline over end-to-end training: Separating non-streaming pretraining, truncated adaptation, and policy training allows each stage to be validated independently and reduces the risk of interference between objectives. The truncated fine-tuning stage (Stage 2) is particularly critical — without it, the partial-audio log-probabilities are meaningless and the REINA signal collapses.
Weak monotonicity ( $\epsilon = 0.5$ ) over strict monotonicity: Allows genuine fluctuations in information gain across token positions while preventing the extreme oscillation that makes inference-time commitment behavior unusable. The slack $\epsilon = 0.5$ was chosen empirically and not ablated.
Multilingual tokenizer with learned embeddings over language-specific tokenizers: The Mistral 7B tokenizer provides broad language coverage, and learning a new embedding dictionary from scratch ensures the embeddings are aligned with the speech translation task rather than with the original language model's distribution. Language ID tokens prefixed to sequences disambiguate the target language without requiring separate decoders per language pair.
Open-source data with synthetic augmentation over proprietary data: The MLS transcript translations (produced by an in-house NMT model) expand the available S2TT training data without requiring access to proprietary aligned speech-translation corpora. This enables training at scale (130k hours) while keeping the entire pipeline reproducible with public data.

4. Key Insights and Innovations

Innovation 1: Reframing the Policy Objective as Token-Specific Information Gain Rather Than Distributional Divergence

The dominant assumption in decoupled policy training for SimulST — most clearly embodied in DiG-SST — is that the system should READ when the model's output distribution given partial audio diverges significantly from its distribution given full audio. The intuition is sensible: if partial context produces a different distribution over next tokens than full context, the model is uncertain, and waiting should help. But this paper identifies a subtle flaw that fundamentally changes the optimization objective: distributional divergence is a weak proxy for what we actually care about, which is whether the probability assigned to the correct token improves with more context.

The distinction matters concretely. Two output distributions can have high KL divergence while both assign high probability to the same correct token — the model's uncertainty about other tokens increased, but the correct answer was already confidently predicted. In such cases, DiG-SST would learn to READ unnecessarily, adding latency without quality gain. Conversely, two distributions can appear similar while both assign low probability to the correct token — the model is uncertain in a consistent way, but waiting didn't resolve the uncertainty about the right answer. DiG-SST would learn to WRITE prematurely, emitting a guess rather than waiting for genuinely helpful context.

REINA's move is to replace distributional divergence with a scalar derived from the log-probability of the ground-truth token under partial versus full audio: $\log \hat{p}(s_{n+1} | a_T, S_n) - \log \hat{p}(s_{n+1} | a_t, S_n)$ . This is not merely a different loss function — it's a different quantity of interest. The policy isn't trained to detect when the model is uncertain in general; it's trained to detect when the model's uncertainty about the specific word it should say next would be resolved by waiting. This shifts the policy's attention from model-internal distributional change (which may be irrelevant to correctness) to task-relevant prediction improvement.

The theoretical grounding in mutual information — derived as $I(s_{n+1}; a_T, S_n) - I(s_{n+1}; a_t, S_n) = H(s_{n+1} | a_t, S_n) - H(s_{n+1} | a_T, S_n)$ — provides conceptual clarity that was absent from prior work. The formulation makes explicit what DiG-SST's divergence maximization implicitly approximates: the reduction in conditional entropy of the ground-truth next token. But DiG-SST approximates this by looking at the entire output distribution, conflating signal (the correct token's probability) with noise (probability mass shifting between incorrect tokens). REINA isolates the signal.

Evidence that this reframing produces genuine gains comes from the comparison in Table 3: on MUST-C en→fr, REINA (MUST-C only) achieves BLEU of 34.3 at AL 1.34 versus DiG-SST's 32.4 at AL 1.40 — better quality at lower latency, using the same base model architecture and training data scale for the policy. The NoSE comparison is starker: REINA (MUST-C only) scores 68.7 versus DiG-SST's 66.7 on the same data. These improvements come purely from changing what the policy is trained to predict, not from scaling data or model capacity.

This is a fundamental conceptual advance, not an incremental refinement. It doesn't just improve performance — it corrects a flaw in the problem formulation itself, providing a more direct and principled mapping between the training objective and the downstream goal of translation accuracy.

Innovation 2: Covariance Maximization as a Surrogate Objective That Avoids Unnecessary Scale Constraints

Training a neural network to predict a scalar target typically uses regression: minimize $\|q_\theta - \hat{\mathcal{F}}\|^2$ , penalizing the network when its output deviates from the target value in either direction. REINA takes a deliberately different approach: maximize the covariance between $q_\theta$ and $\hat{\mathcal{F}}$ , which requires only that they covary — that $q_\theta$ is high when $\hat{\mathcal{F}}$ is high and low when $\hat{\mathcal{F}}$ is low — without constraining the exact numerical scale.

This is not an arbitrary design choice; it reflects a careful analysis of what the inference procedure actually requires. At test time, $q_\theta$ is thresholded by $\alpha$ to produce binary READ/WRITE decisions. The threshold is chosen post-hoc by sweeping across values, so the absolute scale of $q_\theta$ is irrelevant — all that matters is the ordering of $q_\theta$ values relative to each other. A policy that produces $q_\theta$ values of [0.1, 0.5, 0.9] for three tokens is functionally identical (after appropriate threshold selection) to one that produces [10, 50, 90]. Regression would penalize the latter for not matching the target's numerical scale, imposing an unnecessary constraint that could distort the learned ordering.

The covariance formulation has a second advantage: it is robust to miscalibration in the frozen base model's probability estimates. $\hat{\mathcal{F}}$ is computed from a frozen S2TT model that was not trained on partial audio until Stage 2, and even after truncated fine-tuning, its absolute log-probability values may be poorly calibrated — the model might systematically over- or under-estimate its confidence. Regression would force $q_\theta$ to replicate these calibration errors, encoding them into the policy's scale. Covariance maximization is scale-invariant: it cares only about the direction of co-variation, making it resilient to systematic biases in $\hat{\mathcal{F}}$ 's magnitude.

The technical mechanism that makes this work — batch-normalizing $\hat{\mathcal{F}}$ to zero mean within each batch, causing the cross-term $\mathbb{E}[q_\theta] \cdot \mathbb{E}[\hat{\mathcal{F}}]$ in the covariance expansion to vanish — is an elegant simplification that converts a somewhat complex objective into a straightforward inner-product minimization. But the intellectual contribution isn't the batch normalization trick itself; it's the recognition that the inference procedure doesn't require scale-consistent predictions, and that designing the training objective to match this reality (ask only for covariance, not for exact values) yields a more appropriate and potentially more robust optimization problem.

This is a methodological refinement of how policy networks are trained, not a paradigm shift. It's specific to the threshold-based inference setup and wouldn't necessarily generalize to settings where the raw $q_\theta$ value is used directly without post-hoc thresholding. But within the SimulST policy training framework, it's a clean piece of objective design that avoids imposing constraints the downstream use case doesn't need.

Innovation 3: The Normalized Streaming Efficiency (NoSE) Metric as a Diagnostic for Disentangling Policy Quality from Model Quality

The paper makes a sharp methodological critique of standard SimulST evaluation: plotting Average Lagging (AL) against BLEU and comparing curves between models is confounded by the non-streaming translation quality of the underlying S2TT models. If Model A achieves higher BLEU than Model B at every latency point, is that because Model A has a better streaming policy, or because Model A's base S2TT model is simply a better translator? The standard evaluation cannot answer this question, yet the literature routinely attributes superior AL/BLEU curves to superior streaming methods without controlling for this confound.

The paper's diagnosis is precise: many comparisons "pitch models as being better at streaming than others due to having a higher BLEU vs AL curve, when in reality the difference may be accounted for entirely by one model having a superior non-streaming BLEU" (Section 4.2). This is not a hypothetical concern — SeamlessM4T's streaming performance, for instance, benefits enormously from its massive non-streaming model, making it difficult to assess whether its EMMA streaming policy is genuinely superior to alternatives or whether the same policy on a smaller base model would underperform.

NoSE addresses this by normalizing the streaming performance by the non-streaming baseline: it is the area under the AL/BLEU curve divided by the area under the non-streaming BLEU line (treated as a constant across all latencies). Formally, for a given latency range $[x, y]$ , $\text{NoSE} = \frac{\int_x^y \text{BLEU}(\text{AL}) \, d\text{AL}}{(y - x) \cdot \text{BLEU}_{\text{non-streaming}}}$ . A model that achieves 100% of its non-streaming BLEU at all latencies would have NoSE = 100%; a model that degrades significantly under streaming constraints would have a lower NoSE.

What makes this contribution distinctive is that it's not a new metric proposal in a vacuum — it emerges from a specific diagnostic question: is REINA's improvement due to better base translation quality (from multi-task training on 130k hours of data) or due to a better streaming policy? By computing NoSE for REINA (MUST-C only) — which uses the same base model and training data scale as DiG-SST — the paper can isolate the policy improvement. REINA (MUST-C only) achieves a NoSE of 68.7 versus DiG-SST's 66.7 on en→de, 70.9 versus 68.1 on en→es, and 66.2 versus 60.9 on en→fr (Table 2). These are gains of 1.9–5.3 percentage points coming purely from the change in policy training objective, cleanly separated from any model capacity or data scale confounds.

The metric has limitations that the paper acknowledges: NoSE is sensitive to the chosen latency bounds $[x, y]$ , and comparisons require all models to have reported values across the same range. The paper recommends reporting bounds explicitly. But as a diagnostic tool rather than a universal benchmark, NoSE serves a valuable function: it forces the field to distinguish between "this model translates better" and "this model streams better," a distinction that was routinely elided in prior work. This is a methodological contribution — it changes how we evaluate, not what we build — but it directly enables fairer comparisons and more honest attribution of performance gains.

Innovation 4: Identifying Truncated Audio Fine-Tuning as a Critical Prerequisite for Meaningful Policy Training Signals

The paper's ablation in Table 4 shows that skipping Stage 2 (truncated audio fine-tuning) causes a clear degradation in the trained policy's streaming performance. The finding itself — that models need to see partial inputs during training to produce calibrated probabilities on partial inputs — is not surprising. What makes this a genuine innovation is the articulation of why it matters specifically for information-gain-based policy training, and the implication that prior divergence-based approaches like DiG-SST likely suffer from the same problem without recognizing it.

The chain of reasoning: REINA's training signal $\hat{\mathcal{F}} = \log \hat{p}(s_{n+1} | a_t, S_n) - \log \hat{p}(s_{n+1} | a_T, S_n)$ requires the partial-audio log-probability to be a meaningful quantity. If the base S2TT model was trained exclusively on full audios (as is standard for non-streaming models), its decoder has never seen the acoustic encoder produce hidden states from truncated input. The encoder's representations of partial audio will be out-of-distribution, and the decoder's output log-probabilities on these representations will be essentially random — not reflecting genuine model uncertainty, but rather the arbitrary behavior of a model extrapolating beyond its training distribution. In that regime, $\hat{\mathcal{F}}$ is noise, and training a policy to covary with noise produces a garbage policy.

This is not an implementation detail — it's a structural requirement of any method that derives streaming policy signals from a frozen non-streaming model's output probabilities. DiG-SST's divergence-based loss would face the same problem: the divergence between full-audio and partial-audio output distributions is meaningless if the partial-audio distribution is an artifact of distribution shift rather than genuine model uncertainty. The fact that prior work did not identify or address this requirement (DiG-SST's paper, based on the description, does not include an equivalent truncated fine-tuning stage) suggests this is a previously unrecognized bottleneck.

The paper's solution — fine-tuning the base model on 80% randomly truncated audios (balanced with 20% full audios to prevent forgetting) before computing policy training signals — is simple but its necessity had not been previously established. The ablation confirming this (Table 4) provides the empirical grounding. This is a practical insight that changes how one would implement any decoupled policy training method on top of a non-streaming model, not just REINA. It's incremental in the sense that it's a training pipeline modification rather than a new algorithm or objective, but it clarifies a dependency that was previously obscure and that likely contributed to instability or poor performance in prior decoupled policy approaches.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on language pairs {fr, de, es} → en on CVSS-C, and on en → {fr, de, es} on MUST-C. MUST-C (Di Gangi et al., 2019) provides high-quality speech-translation parallel data derived from TED talks, while CVSS-C (Jia et al., 2022) provides translated versions of Common Voice audio. The specific splits used are MUST-C test (~4 hours per language pair for en→X, ~2.5 hours for X→en dev) and CVSS-C test (~22–23 hours for X→en depending on language). For the main results, all models are evaluated on the test splits; dev splits are used only for monitoring during training.
Base model. The non-streaming S2TT model underlying all policies is REINAStream: a Whisper Medium acoustic encoder (307M parameters) feeding a randomly initialized 16-layer transformer decoder (101M parameters), plus a T5 text encoder (38M parameters) used only during training for the MT auxiliary task (total 445M parameters at train time, 408M at inference time since the MT encoder is discarded). The model is trained on 130k hours of open-source and synthetic data across the three-stage pipeline described in Section 3.4. The same base model checkpoint (after Stage 2 truncated fine-tuning) is used for all policy training variants, ensuring that any streaming performance differences are attributable to the policy training objective, not the underlying translation quality.
Metrics. Translation quality is measured using BLEU as implemented in SacreBLEU (Post, 2018). Latency is measured using Average Lagging (AL) and Length-Adaptive Average Lagging (LAAL) as defined in Ma et al. (2020), both re-implemented by the authors based on the original SimulEval paper. The paper's novel contribution is Normalized Streaming Efficiency (NoSE), defined as the area under the piecewise-linear AL/BLEU curve over a specified latency range [x, y], divided by the area under a horizontal line at the non-streaming BLEU score: NoSE = (area under streaming curve) / ((y − x) · BLEU_non-streaming). NoSE is expressed as a percentage; 100% would mean the streaming model achieves its full non-streaming BLEU at all latencies in the range. The bounds [x, y] are chosen per language pair as "the smallest x and largest y for which our work and the works we compare to all have reported values, yielding the widest possible range for which all models have a defined AL/BLEU curve" (Section 4.2).
Baselines. The paper compares against three prior works, using self-reported results from the original papers: DiG-SST (dig_sst_2024), which trains a policy network using the KL divergence between full-audio and partial-audio output distributions as the READ/WRITE signal — this is the closest prior method and the primary baseline; DiSeg (diseg), another adaptive streaming method; and EdAtt (papi-etal-2023-attention), which uses attention matrix weights as heuristics for streaming decisions. On CVSS-C, comparisons include StreamSpeech (streamspeech) and SimulS2S-LLM (deng2025simuls2s). Additionally, the authors implement their own version of DiG-SST (denoted "Dig-SST (Our impl. MUST-C only)") on top of the REINAStream base model trained only on MUST-C, to enable a controlled comparison that isolates the policy training objective from differences in base model quality or data scale. For StreamSpeech, the paper notes an important caveat: StreamSpeech reports ASR-BLEU (BLEU against the source-language transcription used as a proxy for translation reference) rather than standard text BLEU, so comparisons against StreamSpeech are not on the same metric.
Generation budget / compute accounting. All streaming comparisons are measured in terms of the latency-quality trade-off (AL vs. BLEU curves), not in terms of FLOPs or hardware time. The controlling hyperparameter for the REINA policy is the threshold α, which is swept across 6–10 values per model-dataset combination (reported in Table B.1) to generate points on the AL/BLEU curve. All models use a beam size of 3, a streaming chunk size of 0.25 seconds, and a patience factor of 3. The policy network inference cost (6M parameters, applied once per token position) is negligible relative to the base model's encoder-decoder computation, so no separate compute accounting is performed for the policy versus the translation model.
Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense. Strategy selection (choosing the threshold α sweep range, selecting hyperparameters like ε = 0.5 and λ = 0.05) is done on the development sets (Must-C dev and CVSS-C dev), with final results reported on the corresponding test sets. The paper acknowledges that determining appropriate α sweep ranges requires "trial and error" and "several offline inference sweeps" (Appendix B), meaning the threshold values are fit to the evaluation data. This is standard practice in the SimulST literature (since the policy threshold is the control knob for the quality-latency trade-off and must be swept to construct the Pareto curve), but it means there is no held-out policy selection protocol — the reported curves represent the best threshold sweep found for each model-dataset pair.

Main Quantitative Results

Aggregate Streaming Performance on MUST-C

The headline result appears in Table 2, which reports NoSE scores for all methods across all language pairs on both MUST-C and CVSS-C. Across all six MUST-C language pairs, REINA (trained on all datasets) achieves the highest NoSE scores, with values of 81.6 (en→de), 83.7 (en→es), and 85.1 (en→fr). These represent improvements over DiG-SST's self-reported NoSE scores of 66.7, 68.1, and 60.9 — a margin of 14.9 to 24.2 percentage points (Table 2).

However, this comparison conflates two factors: REINA's larger training data (130k hours versus DiG-SST's use of only MUST-C, approximately 400–500 hours) and REINA's different policy training objective. To isolate the effect of the policy objective, the paper reports REINA (MUST-C only) — trained with the full REINA loss on only the MUST-C dataset, using the same base model as the other REINA variants but restricted to the same training data scale as DiG-SST. This variant achieves NoSE scores of 70.7 (en→de), 77.0 (en→es), and 66.2 (en→fr) — gains of 3.0 to 5.3 percentage points over DiG-SST's self-reported results (computed as the average difference across the three language pairs: (70.7−66.7) + (77.0−68.1) + (66.2−60.9) = 4.0 + 8.9 + 5.3, averaging to roughly 6.1 points, though the paper states "3.0% higher" for the MUST-C-only model versus DiG-SST in Section 4.3; the discrepancy likely reflects a specific calculation of the average difference rather than the per-pair differences listed). The paper reports that REINA (MUST-C only) yields NoSE scores 3.0% higher than Dig-SST and 8.9% higher than DiSeg (Section 4.3, Quantitative Results).

Table 3 provides selected operating points (AL, BLEU, LAAL) for REINA, REINA (MUST-C only), and competing methods. Representative numbers for en→fr: REINA achieves BLEU 34.3 at AL 1.34 (NoSE component), while DiG-SST achieves BLEU 32.4 at AL 1.40 — higher quality at lower latency. For en→es: REINA (MUST-C only) achieves BLEU 34.2 at AL 1.16 versus DiG-SST's 32.5 at AL 1.25. For en→de: the relationship reverses at higher latencies; REINA achieves BLEU 32.4 at AL 1.91, while DiG-SST achieves 33.2 at AL 2.56 — DiG-SST is slightly better in absolute BLEU but at substantially higher latency.

The full AL/BLEU curves in Figure 2 show that REINA's advantage is most pronounced at low latencies (AL < 2.0). On en→fr (Figure 2c), REINA's curve sits above DiG-SST across nearly the entire latency range, with the gap widening at low AL values. On en→de (Figure 2a), REINA outperforms at low AL but the curves converge or DiG-SST slightly exceeds at higher AL (above ~3.0). On en→es (Figure 2b), REINA maintains a consistent advantage across the full range. The diagonal relationship — REINA dominating at aggressive (low-latency) streaming settings — is consistent with the paper's claim that the REINA training objective produces policies that are particularly effective when the system must make decisions with minimal audio context.

Our implementation of DiG-SST. The paper's own re-implementation of DiG-SST, trained on MUST-C only using the same base model and data as REINA (MUST-C only), "performs far below every other model in evals" (Section 4.3). Its NoSE scores in Table 2 are not directly reported but its AL/BLEU curve in Figure 2 lies substantially below REINA (MUST-C only) across all language pairs. The paper attributes this to missing details during reproduction ("suggesting we missed details during reproduction of results"), but this also means the primary controlled comparison (same base model, same data, different policy training objective) relies on DiG-SST's self-reported results rather than the authors' own implementation. This introduces uncertainty because the self-reported DiG-SST results were obtained with a different base model architecture and training procedure, not merely a different policy objective.

Aggregate Streaming Performance on CVSS-C

On CVSS-C (Table 2), REINA achieves NoSE scores of 74.9 (fr→en), 75.4 (es→en), and 65.2 (de→en). Comparisons to other methods are limited because "CVSS-C is not commonly evaluated against in SimulST literature" (Section 4.3). StreamSpeech reports only ASR-BLEU, making direct comparison impossible, though the paper claims REINA "outperform[s] SimulS2S-LLM which outperforms StreamSpeech" (Section 4.1) as indirect evidence of state-of-the-art performance. SimulS2S-LLM does not report its offline BLEU performance, so NoSE scores cannot be computed for it (footnote to Table 2).

The full AL/BLEU curves for CVSS-C appear in Figure B.1 (Appendix B). On fr→en (Figure B.1a), REINA's curve rises from BLEU ~30 at AL ~0.5 to BLEU ~36 at AL ~2.5, consistently above the baselines. On de→en (Figure B.1c), REINA shows a more gradual improvement (BLEU ~27 to ~33 across AL 1.0 to 3.0), reflecting the greater difficulty of German-English translation due to divergent word order.

CVSS-C is notable for its short utterance length (average 4.9 seconds), making it "an important benchmark for SimulST systems" because such short audios are common in conversational use cases where streaming translation matters most (Section 4.3).

Ablation Studies and Robustness Checks

Each ablation is reported with its specific table or figure reference, along with the quantitative impact.

Monotonicity regularization (ℒ_m): Figure 3 shows the AL/BLEU curve for es→en on CVSS-C, comparing REINA (full loss) against REINA without the monotonicity term (trained with only ℒ_p + ℒ_r). The two curves are nearly identical at higher latencies (AL > 2.0), but at low latencies, REINA with monotonicity significantly outperforms. The paper quantifies this at one operating point: "at about 35 BLEU, AL decreases from 1.95 to 1.57, a 19% improvement." This represents a latency reduction of 0.38 seconds at equivalent translation quality in the aggressive streaming regime. The finding validates the paper's hypothesis that "monotonicity forces the policy to decide on a clear boundary of when to READ when the information gain waffles between timesteps" — the regularization is most valuable when decisions must be made quickly and there is little margin for flip-flopping between READ and WRITE.

Truncated training (Stage 2): Table 4 reports NoSE scores ablating Stage 2 on MUST-C. The "REINA w/o truncated training" variant (MUST-C only, skipping Stage 2) is compared against REINA (MUST-C only) with full three-stage training. The paper states that skipping truncated training "leads to a performance degradation in the trained policy," providing quantitative support for the claim that "the mutual information formulation of REINA requires a good estimate of log p̂(s_{n+1} | a_t, S_n)." Specific NoSE values for the ablation are not enumerated in the main text but appear in Table 4, which shows REINA (No Truncation) with lower scores than REINA (MUST-C only) across language pairs. The magnitude of the degradation and whether it is uniform across language pairs or larger for certain directions is not discussed — Table 4 is referenced but its numerical values are not quoted inline.

Training data scale (MUST-C only vs. all data): The comparison between REINA (all data, 130k hours) and REINA (MUST-C only, ~400–500 hours) in Table 2 and Figure 2 serves as a de facto ablation on data scale. On MUST-C, the full-data REINA achieves NoSE scores of 81.6–85.1 versus 66.2–77.0 for MUST-C only — a gap of 8.1–15.4 points. This demonstrates that scaling training data from hundreds to thousands of hours produces substantial improvements in streaming policy quality, even with the same REINA loss function and base model architecture. However, the paper cannot disentangle whether the improvement comes from the policy network seeing more diverse training examples (from MLS, CVSS, Mosel), from better base model quality due to the expanded Stage 1 training data, or from both. Since the full-data model's base translation quality is higher (visible in the non-streaming BLEU lines in Figure 2), part of the NoSE gain is attributable to better base translation rather than a better streaming policy per se — though NoSE normalizes for this to some extent.

L2 regularization weight (λ): The paper states in Section 3.1 that "final model performance is not very sensitive to changes in λ" for the L2 penalty weight (set to 0.05), but no quantitative ablation varying λ is provided. This is an informal sensitivity claim rather than a controlled experiment.

Monotonicity slack (ε): The value ε = 0.5 for the monotonicity loss slack parameter is specified but not ablated. The paper provides no evidence for how sensitive final performance is to ε or whether 0.5 represents an optimum versus an arbitrary choice.

Comparison with own DiG-SST implementation: While not a formal ablation, the paper's attempt to replicate DiG-SST on the same base model serves as a robustness check for the reproducibility of prior work. The finding that the authors' DiG-SST implementation "performs far below every other model in evals" and below DiG-SST's self-reported results (Figure 2) suggests either that DiG-SST's reported performance depends on implementation details not captured in the paper description, or that the REINAStream base model architecture differs from DiG-SST's in ways that interact poorly with divergence-based policy training. This is a negative result that the paper candidly reports but does not deeply investigate.

Inference hyperparameters (beam size, chunk size, patience): All experiments use a beam size of 3, chunk size of 0.25s, and patience factor of 3. None of these are ablated — the paper does not show how REINA's streaming performance varies with beam size, audio chunk granularity, or patience. Given the practical importance of these parameters for deployment (smaller chunks = lower latency but more frequent READ/WRITE decisions; larger beam = higher quality but more computation), their absence from the ablation study is a notable gap.

Threshold sweep range: Appendix B, Table B.1 reports the α thresholds swept to generate AL/BLEU curves. The ranges differ by model variant (e.g., REINA (MUST-C only): [0.935–0.950]; REINA (all data): [0.970–0.979]) because $q_\theta$ values shift depending on training data scale. The threshold sweep is determined by "trial and error," meaning the evaluation protocol has a degree of researcher freedom in selecting which operating points to evaluate — if a wider or differently spaced sweep had been chosen, the resulting AL/BLEU curves and NoSE scores could shift. The paper does not discuss sensitivity to the choice of sweep range.

Critical Assessment

The experimental results provide substantial evidence for three of the paper's central claims, but each carries qualifications that merit careful specification.

Claim from the executive summary: "REINA helps push the reported Pareto frontier of the latency/quality tradeoff over prior works." This claim is supported for MUST-C en→X, where REINA (all data) achieves the highest NoSE scores across all three language pairs (Table 2) and where the AL/BLEU curves in Figure 2 show REINA consistently above DiG-SST and DiSeg across most of the latency range. On CVSS-C, the evidence is weaker because few prior works report results on this dataset — the claim that REINA achieves "SOTA streaming results for models of comparable size" (executive summary) rests primarily on outperforming SimulS2S-LLM, which is a single baseline, and on indirect comparison through StreamSpeech's ASR-BLEU scores (which are not directly comparable). The SOTA claim is strongest for MUST-C and more tentative for CVSS-C.

A significant qualification: the primary baseline, DiG-SST's self-reported results, was obtained with a different base model and training setup than REINAStream. The authors' attempt to control for this by re-implementing DiG-SST on the same base model failed to reproduce the reported performance. This means the headline comparison (REINA vs. DiG-SST self-reported) is not fully controlled — we cannot be certain how much of the 3.0–8.9 percentage point NoSE gain (REINA MUST-C only vs. DiG-SST) comes from REINA's better policy objective versus differences in base model quality, training data, or implementation details not captured in DiG-SST's paper. The paper deserves credit for attempting the reproduction and for honestly reporting its failure, but this leaves the core claim of policy superiority over DiG-SST supported by imperfect evidence.

Claim from the executive summary: "REINA improves the latency/quality trade-off by as much as 21 percent compared to prior approaches." The 21% figure requires careful interpretation. It appears in the abstract as "REINA improves the latency/quality trade-off by as much as 21 percent compared to prior approaches, normalized against non-streaming baseline BLEU scores." The paper does not explicitly state where the 21% figure comes from in the results. The largest NoSE improvement in Table 2 is on CVSS-C en→fr, where REINA scores 85.1 versus DiG-SST's 60.9 — an absolute improvement of 24.2 points, which represents a 39.7% relative improvement when computed as (85.1−60.9)/60.9. On MUST-C, the largest relative improvement is on en→fr (85.1 vs. 60.9 = 39.7%) and the smallest is on en→es (83.7 vs. 68.1 = 22.9%). The 21% figure may represent a conservative lower bound across specific settings or may refer to a particular configuration not isolated in the text. Regardless, these large percentage improvements are partly driven by DiG-SST's relatively low absolute NoSE scores — a NoSE of 60.9 means DiG-SST achieves only ~61% of its non-streaming BLEU under streaming constraints, leaving substantial room for improvement. The 21% figure should be understood as "up to" rather than typical, and it reflects gains relative to a specific baseline whose own performance may be improvable.

Claim: "We are one of the first SimulST works to leverage large-scale open source data" (Section 2). This claim is well-supported by the data scale (130k hours) and the thorough documentation of data sources in Table 1. The paper does not provide an exhaustive survey of prior open-source SimulST efforts, but the data scale is substantially larger than typical academic SimulST work (which uses Must-C at ~400–500 hours) and bridges toward the scale of industry work (SeamlessM4T at 600k hours, though proprietary). The claim is more about filling a documented gap than about outperforming specific competitors on data scale.

Claim: NoSE "allows for a fairer assessment of the capabilities of the streaming policy itself." This methodological claim is supported by the logic of NoSE's construction — normalizing by non-streaming BLEU does isolate policy quality from base model quality in principle — but the paper's own results complicate the picture. The full-data REINA achieves higher NoSE than MUST-C only REINA on the same test sets (Table 2). If NoSE perfectly isolated policy quality, these two variants should score identically since they use the same REINA loss function and differ only in training data scale, which NoSE is designed to factor out through normalization. The observed gap (81.6–85.1 vs. 66.2–77.0) means either that (a) the base model's non-streaming BLEU is not a perfect normalizer because scaling training data improves the policy network in ways not captured by the non-streaming BLEU improvement, or (b) the non-streaming BLEU values used for normalization are themselves estimated with noise, introducing variance. Either way, the claim that NoSE provides "a fairer assessment" is better supported than the implicit claim that it perfectly isolates policy quality.

Genuine weaknesses in the experimental design:

No cross-validation or statistical testing. All comparisons are based on single-point estimates of BLEU on test sets of modest size (Must-C test is ~4 hours per direction, yielding a few thousand utterances). No confidence intervals, bootstrap estimates, or significance tests are reported. Given the BLEU differences between methods at similar AL values (e.g., 34.3 vs. 32.4 in Table 3 for en→fr, a difference of 1.9 BLEU), it is unclear whether these gaps are statistically reliable or within the test set variance.
Single base model family. All experiments use REINAStream with Whisper Medium as the acoustic encoder. The paper does not demonstrate that REINA improves streaming performance when applied to a different base S2TT architecture (e.g., OWSM, Canary, or a HuBERT-based encoder). The claim that REINA "can efficiently convert non-streaming Speech-to-Text Translation (S2TT) models into simultaneous S2TT models" (abstract) is supported for exactly one base architecture. Whether the information gain estimates remain well-calibrated and whether the covariance training procedure converges robustly for different encoder-decoder architectures is untested.
Limited language pair coverage. All experiments cover only four languages (English, French, Spanish, German) in six translation directions. These are all high-resource, relatively similar European languages. The paper acknowledges this as a limitation and plans to "expand to lower resourced languages" in future work (Section 3.5). The information gain formulation is language-agnostic in principle, but its practical effectiveness for language pairs with radically different word orders (e.g., English-Japanese, where the verb comes at the sentence end) or for agglutinative languages with complex morphology is unknown.
No evaluation of out-of-domain robustness. All evaluation is on Must-C and CVSS-C, which are in-domain with respect to the training data (Must-C and CVSS are both included in training, albeit in the test split). The paper does not evaluate on a fully held-out domain (e.g., news broadcasts, lecture recordings, telephone conversations), so the streaming policy's generalization to acoustic conditions and speaking styles not seen during training is uncharacterized.
No latency breakdown or system-level latency accounting. The paper measures latency in terms of AL and LAAL, which are corpus-level metrics computed from token-level READ/WRITE decisions. It does not report wall-clock latency or computational latency — the time the system actually takes to process each audio chunk through the encoder, decoder, and policy network on specific hardware. For a method that claims practical deployability ("usable in real-world chat settings," Section 3.2), this is a significant gap. A policy that achieves low AL but requires a forward pass through a 307M-parameter encoder for every 0.25-second chunk on CPU may be less "practical" than AL suggests.
The FLEURS comparison is unconvincing. Appendix E reports an attempt to compare against Seamless on FLEURS, but the evaluation is compromised by VAD-related issues: REINAStream's model "is trained entirely on audios not trimmed with VAD" and therefore "tends to expect an ending silence and over-generates" on VAD-trimmed FLEURS audio. The authors attempt to compensate by adding "2 seconds of white noise at the end" but acknowledge this "has a negative impact on BLEU" on their other datasets. The NoSE scores in Table E.1 — which show REINAStream comparable to Seamless on X→en but worse on en→X — cannot be interpreted confidently given the acknowledged domain mismatch.

Missing experiments that would strengthen the paper:

Ablation of the covariance objective versus regression. The paper motivates covariance maximization partly by arguing that regression to exact $\hat{\mathcal{F}}$ values is unnecessary and potentially harmful (Section 3.1). A direct comparison of ℒ_p (covariance) against mean-squared error regression to the same $\hat{\mathcal{F}}$ targets would directly test this motivation. Without this ablation, the benefit of the covariance formulation over standard regression remains a theoretical argument rather than an empirical finding.
Ablation of the batch normalization of $\hat{\mathcal{F}}$ . The batch normalization step is central to simplifying the covariance objective to an inner product (Equation 4 → Equation 5). Whether performance degrades without batch normalization — using the raw $\hat{\mathcal{F}}$ values and computing the full covariance including the cross-term — is not tested.
Varying the partial audio truncation point during Stage 3. The paper does not specify whether the partial audio length $t$ used during REINA policy training is fixed, randomly sampled from a distribution, or systematically varied to cover different latency regimes. The choice of $t$ determines the difficulty of the information gain estimation problem and could bias the policy toward certain operating points on the AL/BLEU curve. An ablation over truncation strategies would clarify this dependency.
Direct combination with a different base S2TT model. Training REINA on top of a publicly available S2TT checkpoint (e.g., OWSM or Whisper fine-tuned on CoVoST) would demonstrate the generality of the method and provide a more standard baseline for comparison with other work.
Evaluation with SimulEval, the standard SimulST evaluation toolkit. The paper implements its own AL and LAAL computation rather than using the community-standard SimulEval toolkit (Ma et al., 2020). While the authors state they followed the original paper's definitions, implementation discrepancies can produce small but meaningful differences in reported AL values, complicating comparisons with work that uses the standard toolkit.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Not Included in the Efficiency Gains

The REINA training procedure relies on a crucial intermediate step: computing the information gain estimate $\hat{\mathcal{F}}$ for each token position, which requires running the frozen S2TT model on both the full audio and a truncated version of the audio. This is necessary at every single training step during Stage 3, because $\hat{\mathcal{F}}$ serves as the target signal that the policy network is learning to covary with. The compute cost of this double forward pass — each requiring a full pass through the 307M-parameter Whisper encoder and the 101M-parameter decoder — is substantial and is incurred specifically for policy training, not for improving the base model's translation quality.

This is not simply a one-time cost amortized over deployment. The Stage 2 truncated fine-tuning — which the paper demonstrates is essential for producing meaningful $\hat{\mathcal{F}}$ estimates (Table 4 shows performance degradation when it is skipped) — requires additional training on randomly truncated audios for 2 days on 24 A100-80GB GPUs. Furthermore, the threshold sweep procedure for generating the AL/BLEU trade-off curve requires "several offline inference sweeps across thresholds" (Appendix B), each of which requires running the full model on the evaluation data at multiple α values, multiplying the evaluation-time compute by the number of thresholds swept (6–10 values per model-dataset combination, per Table B.1).

The paper does not account for any of these costs in its headline efficiency claims. The 4× improvement over best-of-N (or analogous efficiency metrics) is not claimed here, but the practical deployment picture is incomplete: a practitioner deciding between REINA and a simpler method like wait-k or EdAtt would need to weigh the additional training compute against the streaming quality gains, and the paper provides no such accounting.

Consequence: The reported streaming performance improvements per unit of inference-time latency do not account for the total cost of achieving that streaming performance, which includes the computationally expensive policy training stage and the trial-and-error threshold calibration. A method with lower training cost but slightly worse streaming performance might be preferable in resource-constrained settings, but the paper provides no basis for making that trade-off.

Evidence in the paper: Stage 2 takes 2 days on 24 A100 GPUs (Section 3.4). Stage 3 takes 20 epochs completing in under 12 hours (Section 3.4). Threshold sweeping is described as requiring "several offline inference sweeps" (Appendix B). The paper explicitly acknowledges that "determining appropriate α sweep ranges requires trial and error" and that this process is time-consuming, but does not factor it into any efficiency metric.

Mitigation status: Not addressed. The paper frames REINA as "efficient" and "practical" (Section 1, Section 3.2), but this refers only to the policy network's inference-time cost (6M parameters, negligible relative to the base model) and the stability of the training objective compared to EMMA or transducers, not to the total end-to-end cost of producing a deployable streaming policy. Future work on reducing the training compute (e.g., by amortizing the full-audio forward pass or using a smaller model to estimate $\hat{\mathcal{F}}$ ) is not suggested.

6.2 Streaming Performance Depends on a Single Base Model Architecture

Every result in the paper — every AL/BLEU curve, every NoSE score, every operating point in Table 3 — is obtained using exactly one base model: REINAStream, which pairs a fine-tuned Whisper Medium encoder (307M parameters) with a randomly initialized 16-layer transformer decoder (101M parameters). The paper's abstract and introduction frame REINA as a general method that "can efficiently convert non-streaming Speech-to-Text Translation (S2TT) models into simultaneous S2TT models" (Section 1, emphasis added). But this claim of generality is supported for exactly one architecture combination, one encoder family (Whisper, based on a standard Transformer encoder), and one decoder design (autoregressive transformer with cross-attention).

The REINA loss depends critically on the quality of $\log \hat{p}(s_{n+1} | a_t, S_n)$ — the log-probability the frozen model assigns to the ground-truth next token when given partial audio. This quantity is not a property of the task or the data; it is a property of this specific model's uncertainty representation. A model with different calibration characteristics — for instance, an E-Branchformer encoder (as in OWSM) with a different pattern of how partial audio is represented, or a CTC-based model where output probabilities are not conditioned autoregressively on previous tokens — would produce different $\hat{\mathcal{F}}$ estimates, potentially with different noise characteristics, scale, or correlation structure with true information gain. Whether the covariance maximization formulation remains stable and whether the learned $q_\theta$ generalizes as well for a different base architecture is entirely untested.

Consequence: A practitioner who has invested in a different S2TT architecture (e.g., OWSM, Canary, or a proprietary model) cannot assume that REINA will work as effectively on top of their model. The paper provides no guidance on what properties a base model must have for REINA to work well (Does it require the specific calibration that comes from autoregressive decoding with teacher forcing? Does it require a certain scale of acoustic encoder? Does it depend on Whisper's particular pre-training distribution?), nor any diagnostic for assessing whether a given base model is suitable.

Evidence in the paper: All experiments use REINAStream with Whisper Medium. There is no ablation varying the encoder architecture, the decoder depth, the training data distribution, or the pre-training strategy. The paper does not even evaluate whether REINA works on top of the same Whisper Medium encoder with a smaller decoder, or with the encoder frozen during Stage 1, or with a different tokenizer — all of which are smaller variations than a different architecture entirely. Section 2 mentions OWSM, Canary, and SeamlessM4T as alternative S2TT architectures but makes no attempt to apply REINA to any of them.

Mitigation status: Not addressed. The paper does not acknowledge this as a limitation, does not discuss what properties of the base model are necessary for REINA to function, and does not suggest future work on validating the method across architectures. The transition from "REINAStream achieves SOTA streaming" (a claim about a specific model) to "REINA can convert S2TT models into SimulST models" (a claim about the method) represents an untested generalization.

6.3 The NoSE Metric Does Not Fully Isolate Policy Quality from Base Model Quality

The paper introduces NoSE as a way to "disentangle a model's non-streaming translation quality from its streaming ability" (Section 4.2) and frames it as a key contribution that enables fairer comparison of streaming policies. The logic is sound in principle: by dividing the area under the AL/BLEU curve by the non-streaming BLEU, models with different absolute translation quality can be compared on how well they preserve that quality under streaming constraints. But the paper's own results reveal that NoSE does not fully achieve this disentanglement in practice.

Consider the comparison in Table 2 between REINA (all data, trained on 130k hours) and REINA (MUST-C only, trained on ~400–500 hours). These two variants use identical REINA loss functions, identical policy network architectures, identical inference procedures, and the same frozen base model checkpoint — they differ only in the scale and diversity of the training data used during Stage 3 policy training. If NoSE perfectly isolated streaming policy quality from base model quality, these two variants would have similar NoSE scores, because the base model is the same and the policy training objective is the same. Instead, the full-data REINA achieves NoSE scores that are 8.1 to 15.4 points higher (e.g., en→de: 81.6 vs. 70.7; en→fr: 85.1 vs. 66.2). This large gap means that either (a) scaling the policy training data from 400 to 130k hours genuinely improves the streaming policy itself in ways that NoSE is meant to capture (which would be a legitimate finding — better training data produces a better policy), or (b) the non-streaming BLEU normalization is not a sufficient control because the full-data model's base translation quality is itself improved by the expanded training data, and this improvement is not fully factored out by dividing by non-streaming BLEU.

The second interpretation is the more concerning one: if better base translation quality non-linearly affects the shape of the AL/BLEU curve (not just its vertical position), then NoSE still conflates policy quality with base model quality to some degree. A model that achieves 40 non-streaming BLEU might have a fundamentally different degradation curve under streaming than one that achieves 30 non-streaming BLEU, even with the same policy, because errors compound differently at different absolute quality levels.

Consequence: Comparisons using NoSE across models with substantially different non-streaming BLEU — which is precisely the use case NoSE is designed for — remain partially confounded. The paper cannot determine how much of REINA's NoSE advantage over DiG-SST (which uses a different, likely lower-quality base model) comes from the superior policy objective and how much comes from REINAStream having better base translation quality that degrades more gracefully. The MUST-C-only comparison partially controls for this by using the same base model, but the difference between REINA (MUST-C only) and DiG-SST's self-reported results is still confounded by differences in their respective base S2TT models.

Evidence in the paper: The gap between REINA (all data) and REINA (MUST-C only) in Table 2. The paper does not discuss this gap as evidence of NoSE's incomplete normalization. The non-streaming BLEU values used for normalization are not reported in the main results tables; they appear only as dotted horizontal lines in Figures 2 and B.1, making it difficult for readers to compute the numerator and denominator of NoSE independently or to assess whether the normalization is doing the intended work.

Mitigation status: Partially acknowledged. The paper recognizes that NoSE is "heavily dependent on [the latency bounds x and y]" and recommends reporting bounds explicitly. It does not acknowledge the more fundamental issue that non-streaming BLEU normalization may not fully control for base model quality, nor does it propose a more sophisticated normalization (e.g., normalizing by an estimated upper bound on streaming performance given the base model's per-token uncertainty).

6.4 Hard Problems (Long Sentences, Divergent Language Pairs) Show Diminishing Returns from REINA

The paper's results contain a pattern that is visible but not explicitly discussed: REINA's advantage over competing methods is most pronounced on easier streaming conditions (shorter sentences, language pairs with similar word order) and at low latencies (AL < 2.0), and it narrows or reverses on harder conditions. This pattern suggests a fundamental limitation of the information-gain-based approach — one that parallels a classic finding in machine learning: difficult instances are precisely those where the model's uncertainty estimates are least reliable, and the information gain signal derived from those estimates becomes correspondingly noisy.

On MUST-C en→de (Figure 2a) — the hardest of the three MUST-C language pairs due to German's divergent verb-final word order — REINA's advantage over DiG-SST is visible at low AL (below ~2.0) but the curves converge at higher AL, with DiG-SST slightly exceeding REINA above AL ~3.0. The paper acknowledges this in Section 4.3: "German, where DigSST is slightly better at higher latencies than REINA." On CVSS-C de→en (Figure B.1c) — another German-involving pair with divergent word order — the AL/BLEU curve is flatter and the absolute BLEU scores are substantially lower (~27–33) than for fr→en or es→en (~30–38 and ~32–40 respectively). The NoSE score for de→en (65.2) is the lowest of all three CVSS directions (fr→en: 74.9, es→en: 75.4), indicating that more of the non-streaming translation quality is lost under streaming constraints for German.

This pattern has an intuitive explanation rooted in the REINA formulation. The information gain estimate $\hat{\mathcal{F}} = \log \hat{p}(s_{n+1} | a_T, S_n) - \log \hat{p}(s_{n+1} | a_t, S_n)$ relies fundamentally on the frozen S2TT model's ability to assign meaningfully different probabilities to the correct token under full versus partial audio. For language pairs where the correct translation depends on words that appear much later in the source sentence (as in German, where the main verb often appears at the clause end), the partial-audio log-probability will be close to random for many token positions, and the difference $\log \hat{p_T} - \log \hat{p_t}$ will be large but with high variance — the model knows it's uncertain but can't reliably estimate how much additional context will help, because the required context hasn't appeared yet in the audio and the model has no way to anticipate its content. In this regime, the $\hat{\mathcal{F}}$ estimate becomes noisy, the covariance signal $\mathcal{L}_p$ becomes weak, and the learned $q_\theta$ becomes less reliable at distinguishing between "waiting will help a lot" and "waiting won't help at all."

Consequence: The REINA policy's advantage — and potentially any information-gain-based approach — is systematically smaller for language pairs with highly divergent word orders (German→English, English→German, and by extrapolation, English→Japanese, English→Korean, etc.) and for long sentences where critical disambiguating information appears late in the utterance. For a multilingual SimulST system intended to serve many language pairs, the method's benefits are uneven, concentrated on easier pairs.

Evidence in the paper: The convergence of REINA and DiG-SST curves on en→de at high AL (Figure 2a). The lower absolute performance and NoSE scores for de→en vs. fr→en and es→en on CVSS-C (Table 2). The paper's statement that "REINA excels at lower latency streaming, even when its non-streaming BLEU is lower than competitors" (Section 4.3) implicitly acknowledges that low-latency streaming is REINA's strength regime, with the converse implication that high-latency or high-difficulty settings are less favorable. The paper does not analyze this difficulty effect systematically (e.g., by binning test utterances by sentence length or syntactic divergence and reporting per-bin streaming performance).

Mitigation status: Not discussed. The paper does not identify divergent word order or sentence length as factors that limit REINA's effectiveness, does not analyze the relationship between linguistic properties of the language pair and streaming policy quality, and does not suggest architectural modifications (e.g., incorporating syntactic knowledge into the policy network, or using different α thresholds for different language pairs) to address the issue. Future work on "expanding to lower resourced languages" (Section 3.5) is mentioned but framed as a data availability challenge, not a fundamental methodological limitation for syntactically divergent languages.

6.5 Verification Through a Frozen Model Precludes Joint Optimization and Creates a Quality Ceiling

A foundational design choice in REINA is that the base S2TT model is frozen during policy training (Stage 3). The reasoning is sound: if the base model were fine-tuned, the information gain estimates $\hat{\mathcal{F}}$ would become a moving target, destabilizing the policy optimization. But this modularity — cleanly separating translation quality from streaming decision-making — is simultaneously a limitation: the policy network can only learn to make READ/WRITE decisions that are optimal given the frozen base model's patterns of uncertainty. It cannot improve the base model's ability to make good predictions from partial audio, nor can it encourage the base model to develop the kind of calibrated uncertainty that would make the policy's job easier.

This creates a ceiling on streaming performance that can only be raised by improving the base model independently, before policy training. The ablation in Table 4 shows that Stage 2 (truncated fine-tuning) improves policy quality by improving the base model's partial-audio calibration — but this improvement happens before policy training, not during it. If the base model, even after truncated fine-tuning, has systematic weaknesses in how it handles certain types of partial input (e.g., it overconfidently predicts the wrong word when a disambiguating cue hasn't arrived yet), the policy network has no mechanism to compensate for this. The policy can learn to READ in those situations — and indeed, the REINA loss should encourage this — but if the base model's uncertainty patterns are fundamentally unreliable, the policy's decisions will be unreliable too.

This contrasts with architecture-integrated approaches like EMMA, where the streaming decision and the translation prediction are jointly optimized within a single model, potentially allowing the model's representations to adapt to the demands of streaming. EMMA's well-documented training instability (Appendix D) makes it impractical, but in principle, joint optimization could achieve better streaming quality than any decoupled approach, because the translation model can learn to structure its internal representations in ways that make streaming decisions easier — for example, by learning to defer high-uncertainty predictions until more context arrives, or by representing partial input in a way that makes information gain more predictable.

Consequence: REINA's streaming performance is bounded above by what is achievable with a frozen, separately trained base model. No amount of policy network capacity or training data can overcome fundamental weaknesses in the base model's uncertainty calibration on partial inputs. For a base model that is particularly poor at handling partial audio (e.g., one trained with very little data augmentation or with an architecture that doesn't generalize well to truncated inputs), REINA would inherit those weaknesses without being able to improve them.

Evidence in the paper: The frozen-base design is explicitly described in Section 3.4: "we train the policy network by minimizing $\mathcal{L}_{\text{REINA}}$ and freezing all other parameters." The paper does not experiment with joint fine-tuning (policy network + base model together) during Stage 3, nor does it compare REINA's streaming performance against a hypothetical upper bound where the base model is optimized jointly with the policy. Appendix D's discussion of EMMA's instability provides indirect evidence that joint optimization is hard, but does not establish that it's impossible or that the REINA decoupling is the only viable approach.

Mitigation status: The paper does not explicitly frame the frozen-base design as a limitation, but the fact that it was chosen for stability reasons (rather than because joint optimization was tried and found to be worse) is honest. The discussion of EMMA in Appendix D implies that joint optimization introduces training difficulties that REINA deliberately avoids — trading off potential ceiling performance for trainability. Whether a hybrid approach (partial fine-tuning of the base model with a small learning rate during policy training, or alternating between base model updates and policy updates) could recover some of the joint optimization benefit without the instability is not explored. The paper's future work section (Section 5) does not mention relaxing the frozen-base constraint.

6.6 Evaluations Are Confined to a Narrow Distribution of Speech and Translation Tasks

All training and evaluation data comes from a specific cluster of domains: audiobook readings (MLS), TED talks (MUST-C), and crowdsourced voice recordings (CVSS-C, derived from Common Voice). These are all read or prepared speech in quiet, high-quality recording conditions, spoken at moderate pace by cooperative speakers. The paper does not evaluate on spontaneous conversational speech, overlapping speakers, noisy acoustic environments, accented or non-standard speech, or any of the other challenging acoustic conditions that characterize real-world cross-lingual communication.

This matters for REINA in particular because the information gain signal $\hat{\mathcal{F}}$ is derived from the frozen model's log-probabilities on partial audio. If the acoustic encoder produces degraded or noisy representations due to challenging audio conditions (background noise, reverberation, rapid speech), the partial-audio log-probabilities $\log \hat{p}(s_{n+1} | a_t, S_n)$ will be less reliable, and the trained policy may make poor READ/WRITE decisions — either being overconfident (WRITE when it should READ because the encoder's noisy representation happens to push the wrong token's probability up) or underconfident (READ when it should WRITE because the noise makes the model systematically uncertain). The paper provides no evidence about whether or how the REINA training procedure is robust to acoustic domain shift.

Furthermore, the evaluation covers only translation from speech to text. The paper's conclusion mentions extending REINAStream to simultaneous speech-to-speech translation (SimulS2ST) as future work (Section 5). This is not a minor extension: speech-to-speech translation introduces additional latency from the text-to-speech synthesizer, and the streaming policy would need to account for the fact that emitted tokens aren't immediately heard by the listener but must pass through a synthesis stage. Whether the information gain formulation (which assumes token emission is the terminal output event) transfers cleanly to this cascaded setting is untested.

Consequence: The paper's streaming performance claims — including the SOTA results and the 21% NoSE improvement — are valid only for the specific data distribution on which they were measured (read/prepared speech from professional or willing speakers in the TED, audiobook, and crowdsourced domains). A practitioner deploying REINAStream in a video conferencing application with variable microphone quality, background noise, and spontaneous disfluent speech should not expect the reported AL/BLEU trade-offs to hold. The policy may exhibit more aggressive (lower quality) or more conservative (higher latency) behavior than expected, or may show higher variance in streaming decisions, leading to a jagged user experience with inconsistent latency.

Evidence in the paper: The data sources are described in Section 3.5 and Table 1: MLS (audiobooks), MUST-C (TED talks), CVSS-C (Common Voice translations), MOSEL (a collection of existing speech translation datasets). No spontaneous conversational speech datasets are included. No evaluation is performed on out-of-domain test sets. The FLEURS evaluation in Appendix E is the closest to an out-of-domain test, but it is compromised by the VAD mismatch issue (the model "tends to expect an ending silence and over-generates" on VAD-trimmed audio) and by the ad-hoc white noise augmentation, making it impossible to assess domain robustness from these results.

Mitigation status: Not addressed. The paper does not discuss domain robustness, does not acknowledge the narrow acoustic and stylistic distribution of its training and evaluation data, and does not include domain shift experiments. The focus on open-source data, while commendable for reproducibility, means the training distribution is constrained by what speech data is publicly available — which skews heavily toward read and prepared speech. The paper suggests expanding to "lower resourced languages" as future work (Section 3.5) but does not mention expanding to more diverse acoustic conditions or speaking styles.

7. Implications and Future Directions

How This Work Changes the Landscape

REINA shifts the conceptual framing of SimulST policy learning from arbitration between distributions (divergence-based methods like DiG-SST) to token-specific information accounting (mutual-information-based methods). This is not a paradigm shift in the sense of overturning a mature consensus — SimulST policy training is a young subfield without entrenched dogma — but it is a substantive reframing with practical consequences. The paper identifies a specific, non-obvious flaw in the dominant divergence-based formulation: KL divergence between full-context and partial-context output distributions can be high even when the correct token is already confidently predicted under partial audio, and can be low even when the model is uncertain about the correct answer. By replacing this with the log-probability of the ground-truth token under partial versus full audio, REINA changes what signal the policy is trained to detect, not merely how that signal is optimized. This reframing produces measurable gains: REINA (MUST-C only) achieves NoSE scores 3.0–8.9 percentage points higher than DiG-SST across language pairs using comparable training data (Table 2), and the qualitative behavior differs — REINA's advantage is largest at low latencies (AL < 2.0, Figure 2), precisely where divergence-based methods have the weakest signal because partial-audio output distributions are least reliable.

A secondary shift concerns the role of evaluation in the field. The introduction of NoSE — however imperfect its normalization may be (see Section 6.3) — makes explicit a confound that the literature had been ignoring: comparing AL/BLEU curves without controlling for non-streaming BLEU conflates model quality with policy quality. Prior work routinely attributed higher BLEU-at-a-given-AL to superior streaming methods when the entire difference could reflect one model simply being a better non-streaming translator. Whether or not NoSE becomes the standard metric, the diagnostic question it forces — "does this model stream better, or does it just translate better?" — is likely to persist and to raise the bar for future SimulST evaluations. The paper's own results demonstrate why this matters: a nontrivial fraction of REINA's advantage over DiG-SST on MUST-C (Table 2, comparing full-data REINA at NoSE 81.6–85.1 vs. DiG-SST at 60.9–68.1) is attributable to REINAStream's base translation quality from multi-task training on 130k hours of data, not purely to the REINA policy objective. The MUST-C-only comparison (NoSE 66.2–77.0, which controls for data scale but not for base model architecture) provides a cleaner estimate of the policy gain alone. Future work that fails to disentangle these factors risks repeating the attribution errors NoSE is designed to catch.

The paper also reconciles a latent tension between two approaches to decoupled policy training that might otherwise appear to be in competition: using reinforcement learning to directly optimize the quality-latency trade-off (as in Gu et al., 2017) versus using supervised signals from a frozen teacher model (as in DiG-SST and REINA). The paper's choice to freeze the base model and train the policy with a supervised target ( $\hat{\mathcal{F}}$ ) rather than with RL is pragmatic — RL "is hard to stabilize and efficiently train, especially in cases like SimulST, with no guarantee of convergence" (Section 2) — but the REINA formulation clarifies why a supervised signal can work well: the information gain estimate, though imperfect, provides a dense, token-level training signal that directly captures what the policy needs to know, sidestepping the credit assignment problem that makes RL difficult. This doesn't settle whether RL could eventually outperform supervised approaches given enough engineering — but it establishes that a well-designed supervised objective can match or exceed RL-based methods in this domain without the training instability, making RL-based approaches less attractive for the specific problem of SimulST policy training until evidence of a clear advantage emerges.

The practical upshot for research priorities: the paper's documentation of EMMA's prohibitive training costs (Appendix D: 2GB VRAM per cross-attention layer at batch size 1, numerical instability from cumulative products of 500+ small floats, unclear layer/head selection for inference policy) combined with REINA's strong results using a 6M-parameter policy network trained in under 12 hours (Section 3.4) makes architecture-integrated streaming approaches (monotonic attention, transducers) considerably less attractive for future SimulST research unless their training difficulties can be demonstrably solved. The field now has a clear, replicable alternative that achieves SOTA streaming performance with commodity hardware and stable training, raising the burden of proof for any architecture-integrated method to justify its additional complexity.

Follow-Up Research This Work Enables

Validation of the covariance objective against direct regression to $\hat{\mathcal{F}}$ . The paper argues that maximizing covariance between $q_\theta$ and $\hat{\mathcal{F}}$ is preferable to regressing $q_\theta$ to $\hat{\mathcal{F}}$ directly because the inference procedure only cares about the ordering of $q_\theta$ values (for thresholding), not their absolute scale. This argument is theoretically clean but empirically untested. A straightforward ablation would train three policy variants on the same base model and data: (1) REINA's covariance loss $\mathcal{L}_p$ (Equation 5), (2) mean-squared error regression to batch-normalized $\hat{\mathcal{F}}$ , and (3) mean-squared error regression to raw (non-normalized) $\hat{\mathcal{F}}$ . If the covariance formulation genuinely matters, variant (1) should produce better AL/BLEU curves than (2) and (3), particularly at low latencies where calibration errors in $q_\theta$ magnitude would distort the threshold sweep. The experiment costs little beyond what the paper already reports — train three policy networks under identical conditions except the loss function — and would either validate a key theoretical motivation or reveal that the covariance formulation is incidental to REINA's gains.

Cross-architecture replication on OWSM or Canary. The paper's central claim — that REINA "can efficiently convert non-streaming Speech-to-Text Translation (S2TT) models into simultaneous S2TT models" — is supported for exactly one architecture (Whisper Medium encoder + randomly initialized transformer decoder). The most critical follow-up is to take a publicly available, differently architected S2TT model — OWSM v3.1 (E-Branchformer encoder, ~1B parameters) or Canary (FastConformer encoder, trained with CTC + autoregressive decoding) — freeze it after training, and apply the three-stage REINA pipeline (truncated fine-tuning → policy training on MUST-C). The experiment would measure: (a) whether $\hat{\mathcal{F}}$ estimates from a different encoder architecture remain sufficiently well-calibrated after truncated fine-tuning to produce a learnable policy, (b) whether the REINA loss converges with the same hyperparameters ( $\epsilon = 0.5$ , $\lambda = 0.05$ ) or requires re-tuning, (c) whether the resulting AL/BLEU curves show qualitatively similar patterns (advantage at low latency, diminishing returns on German) across architectures. A negative result — REINA fails to produce a usable policy on a non-Whisper encoder — would substantially narrow the method's claimed generality and would motivate investigation into what encoder properties (pretraining distribution? architectural inductive biases for temporal structure? output sequence length?) are necessary for information-gain-based policy training to work.

Sentence-length-stratified streaming evaluation. The paper's results show that REINA's advantage is largest at low latencies (AL < 2.0, Figure 2) and that the hardest language pair (German, with divergent word order) shows the smallest gains and lowest absolute streaming BLEU. This pattern is observed but not systematically analyzed. A targeted follow-up would evaluate REINAStream on MUST-C test utterances binned by source sentence duration (e.g., 0–3 seconds, 3–6 seconds, 6–10 seconds, 10+ seconds) and compute per-bin AL/BLEU curves and NoSE scores. The hypothesis — consistent with the information gain formulation — is that REINA's advantage over DiG-SST is concentrated on short and medium-length utterances, where the partial-audio signal is most reliable, and narrows or reverses on long utterances, where critical disambiguating information appears late. If confirmed, this would provide deployment guidance (REINA is most valuable for conversational settings with short utterances, less so for lecture or presentation translation with long sentences) and would clarify a boundary condition on the method. It would also inform whether future work should develop length-adaptive policy thresholds ( $\alpha$ varying by estimated utterance length) or hybrid policies that switch between REINA and simpler fixed-k strategies for long utterances.

Joint fine-tuning with a small learning rate during policy training. The paper freezes the base model during Stage 3 to prevent $\hat{\mathcal{F}}$ from becoming a moving target. But this creates a ceiling: the policy can only optimize streaming decisions given the frozen model's uncertainty patterns, and cannot encourage the model to develop uncertainty representations that make streaming easier. A natural experiment is to partially relax the freezing — fine-tune the base model's decoder (not the encoder, to avoid changing the acoustic representations) with a very small learning rate (e.g., $10^{-6}$ , two orders of magnitude below the Stage 1 rate of $10^{-4}$ ) during Stage 3, jointly with the policy network. The experiment would compare streaming performance (NoSE on MUST-C) and training stability (does $\mathcal{L}_p$ converge? do $q_\theta$ values remain well-behaved?) against the fully frozen baseline. A positive result — even a small NoSE improvement without training destabilization — would suggest that decoupled training is unnecessarily conservative and that modest joint optimization can push the streaming ceiling upward. A negative result — training instability or degraded policy quality despite the small learning rate — would empirically validate the frozen-base design choice and clarify the tension between modularity and performance.

Streaming robustness to acoustic domain shift. The paper evaluates on in-domain data (MUST-C and CVSS-C, both of which appear in training) and acknowledges in Appendix E that FLEURS evaluation is compromised by VAD mismatch. A focused robustness study would evaluate REINAStream on a held-out corpus that differs acoustically from training — for instance, the CHiME-5 dinner party corpus (distant microphones, overlapping speech in some segments, spontaneous conversational speech) using forced alignment to create reference translations, or the Europarl-ST corpus (European Parliament speeches, different acoustic conditions and speaking style than TED talks). The key measurement is whether the REINA policy's READ/WRITE decisions remain well-calibrated under acoustic shift (does the policy maintain similar AL for a given BLEU?). A degradation would indicate that the information gain signal $\hat{\mathcal{F}}$ is sensitive to acoustic encoder uncertainty patterns that shift across domains, and would motivate acoustic-domain augmentation during Stage 2 truncated training (e.g., adding noise, reverberation, or codec artifacts to the truncated audios). No degradation would strengthen the claim that REINA policies are robust and deployment-ready.

REINA applied to simultaneous speech-to-speech translation (SimulS2ST). The paper's conclusion mentions extending REINAStream to SimulS2ST as the "next step" (Section 5). This is not a trivial extension: a SimulS2ST system cascades the SimulST text output into a streaming text-to-speech synthesizer, introducing an additional latency source (the TTS model's own processing time and audio playback duration) that the READ/WRITE policy does not currently account for. A concrete experiment would pair the frozen REINAStream model with a streaming TTS model (e.g., FastSpeech 2 with incremental decoding), measure end-to-end latency from audio-in to audio-out (not just AL, which measures only the translation latency), and determine whether the REINA policy trained on text-emission latency needs to be recalibrated. The information-theoretic formulation would need to incorporate TTS latency into the cost of WRITE — emitting a token incurs not just the immediate delay but the downstream TTS processing and playback time before the listener hears the translation. Whether the covariance training framework can be extended to this total-latency objective without losing the simplicity that makes REINA attractive is an open question with practical urgency for voice-to-voice translation systems.

Practical Applications and Downstream Use Cases

Low-latency voice chat translation with moderate compute budgets. The paper's explicit target — "usable in real-world chat settings" (Section 3.2) — is supported by the combination of a 408M-parameter inference model (mid-range by contemporary standards) and a streaming policy that excels at low latencies. On en→fr MUST-C, REINA achieves BLEU 34.3 at AL 1.34 (Table 3) — meaning the average token is emitted about 1.3 seconds after the corresponding audio arrives, with translation quality roughly one-third of the way from zero to the non-streaming ceiling. For a two-way video call where a few seconds of latency is tolerable and translation quality needs to be "good enough to follow the conversation," this operating point is viable. The practical deployment benefit is that the entire system (acoustic encoder + decoder + policy network) fits on a single GPU with memory to spare for other components (a streaming TTS model for speech output, a VAD module, network handling), making it feasible for server-side deployment serving multiple concurrent calls without requiring the massive hardware footprint of systems like SeamlessM4T. The 0.25-second chunk size means the system can respond to new audio within one chunk duration, providing a responsive user experience without perceptible gaps in translation.

Streaming captioning for live presentations and lectures. CVSS-C's average utterance length of 4.9 seconds (Section 4.3) reflects the short-phrase structure common in conversational speech, but live presentations contain longer utterances where the REINA policy's ability to make token-level READ/WRITE decisions (rather than waiting for sentence boundaries) enables incremental caption display. On CVSS-C fr→en, REINA achieves BLEU ~30–38 across AL 0.5–3.0 (Figure B.1a), meaning captions can begin appearing within half a second of the speaker starting, with translation quality improving as more context arrives. For a presentation setting where (a) the speaker is using a known language pair (e.g., English to French or Spanish), (b) the acoustic environment is controlled (podium microphone, quiet room), and (c) slightly degraded translation quality at the start of a sentence is acceptable because it self-corrects as more context arrives, the system could be deployed without further fine-tuning. The beam search with patience factor 3 (Section 3.6) means the system won't get stuck on ambiguous early audio — it will commit to a hypothesis after a bounded number of READ cycles, preventing indefinite caption lag.

Offline batch generation of streaming training data for SimulST research. A less obvious but pragmatically valuable use case is using REINAStream as a data generation engine: run the trained SimulST model on a large corpus of speech with diverse audio lengths and acoustic conditions, record the token-level READ/WRITE decisions and the corresponding emitted translations, and use these as synthetically aligned streaming training data for other SimulST approaches. This is the inverse of the teacher-guided approach (Section 2), where existing models generate alignments for training: instead of using an NMT model or LLM as teacher, use a REINA-trained streaming policy as teacher. The synthetic alignments would reflect a principled information-gain-based policy rather than a heuristically aligned teacher, potentially producing higher-quality streaming training data. This is particularly valuable for language pairs or domains where parallel streaming-aligned data doesn't exist — the REINA system can generate it from any corpus that has audio and reference translations, and the resulting alignments can be used to train SimulST models that don't require an explicit policy network, similar to the approach of Hibiki or SimulS2S-LLM but with a more principled alignment source.

When to Prefer This Method

The paper's explicit comparisons are against DiG-SST (divergence-based policy training) and, implicitly through Appendix D, EMMA (monotonic attention-based architecture integration). The trade-offs are clear and grounded in reported results:

Prefer REINA over DiG-SST when you have a frozen S2TT model and want to train a streaming policy with the same base architecture and training data. The MUST-C-only comparison (Tables 2, 3) shows REINA achieving higher NoSE (3.0–8.9 points, Section 4.3) and better BLEU at lower AL (e.g., en→fr: BLEU 34.3 at AL 1.34 vs. DiG-SST's 32.4 at AL 1.40) using the same data scale. The advantage is robust to language pair but strongest at low latencies.
Prefer REINA over EMMA or transducer-based methods when training stability, hardware requirements, and implementation complexity are constraints. Appendix D documents that EMMA requires 2GB VRAM per cross-attention layer at batch size 1 in fp32, suffers from numerical instability (cumulative product of 500+ small floats rounding to zero), and produces policies that vary unpredictably across attention heads and layers. REINA's policy network trains in under 12 hours with a stable, well-behaved loss (the paper reports no convergence issues). The FLEURS comparison in Appendix E (Table E.1) shows REINAStream achieving comparable NoSE to Seamless on X→en directions despite a much smaller model and vastly cheaper training — making REINA the pragmatic choice unless the additional streaming quality from a large EMMA-trained model is essential and the training budget can absorb the cost.
Prefer REINA over wait-k or heuristic policies (EdAtt) when translation quality at aggressive streaming settings matters. Fixed wait-k policies "are usually suboptimal due to the mismatch between the sampling rate of the input audio frames and the frequency of outputted words" (Section 2), and heuristic policies are not trained to optimize translation quality directly. REINA's advantage is largest at low latency (Figure 3 shows a 19% AL reduction at fixed BLEU with monotonicity), precisely the regime where fixed and heuristic policies are weakest because they lack token-level adaptation to information availability.
Do not prefer REINA (over alternatives or over offline translation) when the target language pair involves extreme syntactic divergence (e.g., English→Japanese, where the main verb appears at sentence end) and utterances are long — the German results (Figure 2a, Figure B.1c) show REINA's advantage narrowing or disappearing at high AL and substantially lower absolute streaming BLEU. In such settings, the information gain signal becomes noisy (the model cannot reliably estimate how much late-arriving context will help), and neither REINA nor DiG-SST is likely to produce high-quality streaming translation. Whether architecture-integrated approaches or teacher-guided synthetic alignment handles these settings better is an open question that the paper does not address.