Training Methods2021~117 minarXiv:2105.09501

Contrastive Learning for Many-to-many Multilingual Neural Machine Translation

Xiao Pan, Mingxuan Wang, Liwei Wu +1May 20, 2021cs.CL

🎯 Pitch

A multilingual translation model trained only on English-centric parallel data can leap from near-zero to over 15 BLEU on unseen non-English pairs like German→French simply by pulling aligned sentence representations closer and pushing random ones apart. This contrastive trick, combined with smart dictionary-based data noising, unlocks many-to-many translation without ever showing the model a non-English training pair.

1. Executive Summary

This paper proposes mRASP2 (multilingual Contrastive Learning for Translation), a training method to build a single unified many-to-many multilingual neural machine translation model that prioritizes non-English translation quality. The approach is evaluated on a multilingual Transformer (12-layer encoder-decoder) trained on 32 English-centric parallel corpora (PC32) and 24-language monolingual data (MC24), targeting WMT, IWSLT, and OPUS-100 benchmarks. mRASP2 combines two techniques: a contrastive learning scheme that explicitly pulls representations of parallel sentences closer while pushing apart non-parallel ones (using cosine similarity on average-pooled encoder outputs as the representation), and Aligned Augmentation (AA) — an extension of Random Aligned Substitution that creates pseudo-parallel training pairs from both parallel and monolingual data by replacing words with synonyms from a multilingual dictionary (e.g., replacing source-side words with their translations to form a noised source paired with the clean target). For English-centric supervised directions, mRASP2 achieves a +1.98 BLEU average improvement over the multilingual baseline, outperforming the unified mBART model on 10 WMT benchmarks. For non-English zero-shot directions, mRASP2 delivers an average 10+ BLEU gain (e.g., achieving 15.31 average BLEU on 30 OPUS-100 zero-shot directions versus 5.05 for the baseline), establishing that bridging cross-language representations with contrastive learning substantially improves zero-shot translation only when combined with both aligned augmentation and monolingual data — contrastive loss alone improves zero-shot (+8.5 BLEU over baseline) but leaves supervised and unsupervised directions mostly unchanged.

2. Context and Motivation

The Core Gap: Multilingual MT Works Well for English, Poorly for Everything Else

The paper addresses a fundamental asymmetry in multilingual neural machine translation (NMT): existing approaches are overwhelmingly English-centric, meaning they perform well when English is the source or target language but degrade substantially for non-English language pairs. This is not merely an empirical observation but a structural consequence of how training data is distributed and how models are designed.

The dataset itself encodes this bias. The authors train on PC32 — a collection of 97.6 million parallel sentence pairs across 32 language directions — all of which are English-centric. Every supervised training example has English on one side, whether as source or target. For a language pair like German→French, the model receives zero direct supervision. It must rely entirely on cross-lingual transfer: the encoder learns to represent German from training on En↔De pairs, and the decoder learns to generate French from training on En↔Fr pairs, but the model never sees a single example of German→French translation during training.

This creates a representation gap that the authors identify as the root problem. If the encoder's representation of German sentences and French sentences occupy different regions of the latent space, stitching them together at inference time — encoding in German, decoding in French — produces poor translations. The model must implicitly learn to map different languages into a shared space through the bottleneck of English as a pivot, but without explicit pressure to do so, this alignment is weak and inconsistent.

Why This Matters: The Many-to-Many Deployment Reality

The practical importance of this problem is twofold:

Deployment efficiency. Johnson et al. (2017) established the appeal of multilingual NMT: a single model serving $M$ languages replaces $O(M^2)$ bilingual models, dramatically reducing deployment complexity. For 32 languages, that means one model instead of 992 potential translation directions. But this efficiency is hollow if the model only works well for the $2(M-1)$ English-centric directions — roughly 62 directions out of 992. Real-world deployment scenarios (e.g., European Union translation, international content platforms, multilingual customer support) require non-English pairs routinely. A French user communicating with a German user should not need English as an intermediary.

Low-resource language leverage. The promise of multilingual NMT is that high-resource language pairs (e.g., English↔French with millions of training examples) should transfer knowledge to low-resource pairs through shared parameters. But if representations across languages are poorly aligned, this transfer is inefficient. The encoder might develop excellent representations for French through En↔Fr training, but if those representations are not aligned with German representations (learned from En↔De training), a Fr→De translation cannot effectively use that knowledge. The representation gap acts as a bottleneck on transfer learning itself, limiting the very benefit that motivates multilingual training.

Zero-shot translation viability. Perhaps the most intellectually compelling use case is zero-shot translation: translating between a language pair never seen during training. Johnson et al. (2017) first demonstrated this is possible simply by specifying the target language token at decoding time, but the quality was "quite poor compared with pivot-based model[s]" using English as an intermediate step. This quality gap makes zero-shot multilingual NMT a scientific curiosity rather than a practical tool. The paper's Table 3 quantifies this gap: the pivot-based approach (source→English→target via separate passes through the same model) achieves 15.56 average BLEU on OPUS-100 zero-shot directions, while the direct zero-shot baseline (m-Transformer) achieves only 5.05 — a factor of 3× worse. Bridging this gap would make many-to-many translation genuinely practical without routing through English.

Where Prior Approaches Fall Short

The paper identifies several families of prior work, each with specific limitations:

Implicit representation alignment through data. The earliest multilingual NMT systems (Johnson et al., 2017) relied entirely on the model to discover shared representations through the training objective alone, with no explicit mechanism to align languages. The only signal comes from the cross-entropy loss on parallel data: the encoder must produce a representation from which the decoder can generate the correct target sentence. When the encoder and decoder see multiple languages, parameter sharing creates some implicit alignment, but it is unreliable and inconsistent — as evidenced by the wide gap between English-centric and non-English performance.

Language-specific components. A line of work (Bapna and Firat, 2019; Kim et al., 2019; Wang et al., 2019b; Escolano et al., 2020) introduced language-specific parameters — adapters, separate attention heads, or language-specific encoder/decoder layers — to boost performance on rich-resource languages. While effective for English-centric directions, these approaches are fundamentally surgical: they treat each language direction as a separate problem to be optimized, rather than promoting cross-lingual transfer. They add capacity per language but do not explicitly align representations, so they do little for zero-shot non-English directions. In fact, by making the model more language-specific, they may reduce the pressure toward shared representations that drives zero-shot capability.

Massive scale through more data and larger models. Arivazhagan et al. (2019), Aharoni et al. (2019), and Fan et al. (2020) took the approach of scaling: larger models, more languages, more data. Fan et al. (2020) introduced non-English parallel data through data mining and back-translation, directly supervising some non-English directions. While effective, this approach is expensive — data mining and back-translation require significant engineering infrastructure and compute — and does not address the fundamental representation alignment problem. It treats the symptom (lack of non-English supervision) rather than the cause (misaligned cross-lingual representations). The authors contrast their approach explicitly: mRASP2 achieves strong non-English performance using only English-centric parallel data plus monolingual data, without mined non-English parallel corpora.

Pre-training then fine-tuning. mBART (Liu et al., 2020), XLM (Conneau and Lample, 2019), and MASS (Song et al., 2019) pre-train on large monolingual corpora with denoising or masked language modeling objectives, then fine-tune on parallel data for specific translation directions. The pre-training phase does learn some cross-lingual representations (especially XLM with its translation language modeling objective that concatenates parallel sentences), but the fine-tuning phase is typically bilingual per direction — a separate fine-tuned model for each language pair. This defeats the unified many-to-many advantage: you end up with as many models as directions, just with better initialization. mRASP distinguishes itself here by pre-training a unified multilingual model (Lin et al., 2020) that handles all directions in a single model after pre-training, but its zero-shot performance still lags (4.91 average BLEU on OPUS-100, only slightly better than the 5.05 baseline).

Explicit interlingua approaches. Lu et al. (2018) proposed learning an explicit interlingua — a language-agnostic intermediate representation — by introducing an additional neural interlingua layer. Al-Shedivat and Parikh (2019) introduced consistency-based training that encourages the model to produce equivalent translations of parallel sentences in auxiliary languages. These approaches conceptually align with mRASP2's goal of bridging representations, but they are architecturally more complex and were evaluated on smaller scales. The paper positions mRASP2 as a simpler, more scalable alternative that achieves competitive or better results without architectural modifications.

Data augmentation for zero-shot. Zhang et al. (2020) improved zero-shot translation through online back-translation, generating synthetic non-English parallel data during training. While effective, this approach explicitly trades off English-centric performance: the authors note that Zhang et al.'s method "achieves BLEU score improvements on zero-shot translations at sacrifice of about 0.5 BLEU score loss on English-centric directions." This trade-off is practically undesirable — a true many-to-many system should not degrade on its best-served directions to improve on others. mRASP2 explicitly targets improvement on all directions simultaneously.

How This Paper Positions Itself

The paper's central claim is that the representation gap — not data scarcity, not model capacity, not architecture design — is the primary bottleneck for non-English translation quality in multilingual NMT. This is a causal hypothesis: if you can force the model to align representations of equivalent sentences across all languages into a shared space, the decoder's ability to generate in any language from that shared representation should enable high-quality translation between any language pair, regardless of whether that specific pair appeared in training.

The paper frames this through a contrastive learning lens adapted from self-supervised visual representation learning (Chen et al., 2020; He et al., 2020). The core idea is deceptively simple: for every parallel sentence pair $(x_i, x_j)$ in the training data, treat it as a positive pair whose representations should be pulled close, and randomly sample other sentences in the same language as negative pairs whose representations should be pushed apart. The contrastive loss (Equation 2) formalizes this as an $N$ -way classification problem: given the representation of a source sentence, identify which target sentence in the batch is its true translation. This is conceptually an extension of the TLM (Translation Language Modeling) objective from XLM (Conneau and Lample, 2019), but applied to encoder representations rather than token predictions, and operating at the sentence level with an explicit contrastive objective rather than relying on masked token reconstruction.

The paper's positioning in the landscape is distinctive in three ways:

1. Explicit over implicit. Unlike prior work that relies on parameter sharing and the cross-entropy loss to implicitly align representations, mRASP2 introduces an explicit alignment objective (the contrastive loss) that directly optimizes for close representations of translation equivalents. This is the difference between hoping alignment happens and forcing it to happen. The contrastive loss provides gradient signal specifically targeting representation distance, which the translation loss alone does not.

2. Unified over per-direction. Unlike pre-train-then-fine-tune approaches (mBART, XLM, MASS) that produce per-direction models, mRASP2 produces a single model for all directions. The same trained weights handle En→Fr, Fr→De, Zh→Ru, and Nl↔Pt (which the model never saw in any form) without any fine-tuning. This is the "unified" claim in the title: one model, all directions.

3. Complementary rather than competitive. The paper explicitly does not claim contrastive learning replaces other approaches; it claims it complements them. The strong results come from the combination: contrastive loss + aligned augmentation + monolingual data + a well-tuned multilingual Transformer baseline. The ablation in Table 4 makes this clear: contrastive loss alone improves zero-shot substantially (+8.5 BLEU) but leaves supervised and unsupervised mostly unchanged (+0.14 and +0.33 respectively). Aligned augmentation on parallel data adds incremental gains. Monolingual data with AA dramatically boosts unsupervised directions (from 4.75 to 18.55 BLEU). The full system's performance emerges from the synergy of all components.

The paper also positions itself as a practical recipe rather than a theoretical contribution. The techniques are simple to describe and implement — cosine similarity on average-pooled encoder outputs, in-batch negative sampling, synonym replacement from existing dictionaries — and the code, data, and trained models are released. This contrasts with more complex architectural approaches that require significant engineering to reproduce.

Finally, the paper's ambition is explicit in the conclusion: "scale-up the current training to more languages, e.g. PC150" with the goal that "a single model can handle more than 100 languages and outperforms the corresponding bilingual baseline." This is the vision of a truly universal translator — not a collection of direction-specific models, but a single system that handles any language pair — and mRASP2 is presented as a step toward that vision by addressing the representation alignment bottleneck that has historically made it infeasible.

3. Technical Approach

3.1 Reader Orientation

mRASP2 is a training recipe — not a new model architecture — that teaches a standard multilingual Transformer to produce translations between any pair of languages by forcing sentences with the same meaning to occupy the same region in the model's internal representation space, regardless of what language they are written in. The system solves the problem that multilingual translation models work well when English is involved but poorly for non-English language pairs (e.g., German→French) by adding two ingredients to the standard training procedure: a contrastive loss that explicitly pulls parallel sentence representations together and pushes random sentences apart, and aligned data augmentation that creates synthetic training pairs by replacing words with their translations from a dictionary.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Multilingual Transformer (12-layer encoder, 12-layer decoder) — the core sequence-to-sequence model that translates from a source language to a target language, conditioned on language identification tokens prepended to each sentence. This is where all translation happens at inference time.
Contrastive Learning Module — operates on the encoder's output representations during training only. It takes average-pooled sentence embeddings from parallel sentence pairs (positive examples) and randomly sampled non-parallel pairs (negative examples) and computes a loss that minimizes the cosine distance between positive pairs while maximizing distance to negatives.
Aligned Augmentation (AA) Pipeline — a preprocessing step that creates pseudo-parallel training data from both parallel corpora and monolingual corpora. It replaces source-side words with synonyms from a multilingual dictionary (with 90% replacement probability), producing a noised source sentence paired with either the original translation (for parallel data) or the original sentence itself (for monolingual data, creating a reconstruction task).
Unified Training Objective — combines the standard token-level cross-entropy translation loss with the sentence-level contrastive loss (weighted by a coefficient $\lambda = 1.0$ and scaled by average sequence length to normalize the magnitude difference between token-level and sentence-level losses). Both losses are computed jointly over parallel data, AA-augmented parallel data, and AA-augmented monolingual data.

Information flows as follows during training: a batch of parallel sentences enters → the encoder produces representations for both source and target → the decoder generates target tokens given source representations (producing the cross-entropy loss) → average-pooled encoder outputs are extracted for both source and target → cosine similarities are computed for all source-target pairs in the batch → the contrastive loss is computed using parallel pairs as positives and all other pairs as negatives → the two losses are summed with the contrastive loss scaled by $\lambda |s|$ → gradients flow back through the entire network.

3.3 Roadmap for the Deep Dive

First, the multilingual Transformer baseline configuration and the cross-entropy translation loss (Equation 1), since this is the foundation on which everything else is built and establishes what "standard training" means before we modify it.
Second, the contrastive learning loss (Equation 2), its mechanics, and how it is combined with the translation loss into the full mRASP2 objective (Equation 3) — this is the core technical contribution and must be understood before we can discuss why data augmentation amplifies its effect.
Third, the Aligned Augmentation technique applied to parallel data and the novel extension to monolingual data, since AA creates the pseudo-pairs that feed into both the translation and contrastive losses.
Fourth, the training configuration and hyperparameters, since the batch size, learning rate scheduling, gradient clipping, and model scale are all critical to training stability at this data scale (97.6M parallel pairs + 1.01B monolingual sentences).
Fifth, the inference procedure for supervised, unsupervised, and zero-shot directions, since the model's behavior at inference time — how language tokens are specified, how beam search or greedy decoding is used — determines what the BLEU numbers actually measure.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that explicit representation alignment through contrastive learning, amplified by dictionary-based data augmentation on both parallel and monolingual data, enables a single multilingual translation model to perform well on English-centric, non-English, unsupervised, and zero-shot directions simultaneously — without architectural modifications, without non-English parallel data, and without per-direction fine-tuning.

Multilingual Transformer Baseline and Translation Loss

The foundational model is a standard Transformer sequence-to-sequence architecture (Vaswani et al., 2017) configured for many-to-many multilingual translation. The design follows the "one model for all directions" paradigm established by Johnson et al. (2017), where a single set of encoder and decoder parameters handles translation between any source language and any target language by prepending a special language identification token to each input and output sequence.

Architecture configuration. The paper departs from the standard Transformer-base and Transformer-big configurations in several ways motivated by the need for higher capacity when handling 32+ languages:

Depth: 12 encoder layers and 12 decoder layers (versus 6 each for Transformer-base). This is motivated by the observation that "we enlarge the number of layers from 6 to 12 and observe significant improvements for multilingual NMT. By contrast, the gains from increasing the bilingual model size is not that large" (Section 4.1). The depth helps multilingual models more than bilingual ones because different languages compete for model capacity — deeper networks have more representational capacity to disentangle language-specific and language-agnostic features.
Width: Model dimension $d_{\text{model}} = 1024$ with 16 attention heads (so each head dimension is $1024/16 = 64$ ). Feed-forward network (FFN) dimension is 1024 as well (the paper does not explicitly state the FFN dimension, but the standard Transformer configuration uses $d_{\text{ff}} = 4 \times d_{\text{model}}$ , which would be 4096; however, the paper only specifies "The embedding size and FFN dimension are set to 1024" in Section 3.1 — this appears to mean both dimensions are 1024, which is narrower than the standard 4× expansion and may be a parameter efficiency choice or a typographical ambiguity).
Normalization: Pre-norm residual connections with Layer Normalization applied to word embeddings, following Wang et al. (2019a). Pre-norm (where LayerNorm is applied before each sublayer, rather than after as in the original Transformer) is known to stabilize training of deep Transformers by preventing gradient explosion through the residual connections. The paper states this choice is made "to ease the training of the deep model."
Shared BPE vocabulary: A shared Byte-Pair Encoding vocabulary across all languages, following Sennrich et al. (2016). The vocabulary contains 64,808 subword tokens, to which 59 language identification tokens are added (one per language in the vocabulary), yielding a total vocabulary size of 64,867. Shared vocabulary means that the same subword units are used for all languages — "chat" in French and "chat" in English map to the same token — which is essential for enabling cross-lingual transfer because the encoder and decoder's embedding matrices are language-agnostic.

Language identification mechanism. Following Fan et al. (2020), each source sentence is prepended with a special token indicating the source language (e.g., <Fr> for French), and each target sentence is prepended with a token indicating the target language. During training, the model learns to condition generation on the target language token — at inference time, specifying a different target language token changes which language the decoder produces, enabling zero-shot translation between language pairs never seen during training. This is the mechanism that makes many-to-many translation possible from a single model: the encoder processes the source language, and the decoder generates in whatever language is specified by the initial token.

Training data structure. Let $\mathcal{L} = \{L_1, \ldots, L_M\}$ be the set of $M$ languages involved in training. For this paper, $M = 32$ (the languages in PC32). Let $\mathcal{D}_{i,j}$ denote a parallel dataset of sentence pairs in languages $(L_i, L_j)$ , and let $\mathcal{D}$ be the union of all parallel datasets. Crucially, all parallel data in PC32 is English-centric — every pair has English on one side. This means $\mathcal{D}$ contains $\mathcal{D}_{\text{En},j}$ and $\mathcal{D}_{j,\text{En}}$ for various languages $j$ , but never contains $\mathcal{D}_{i,j}$ where neither $i$ nor $j$ is English.

Cross-entropy translation loss. The standard training objective is token-level cross-entropy:

$L_{\text{ce}} = \sum_{x_i, x_j \in \mathcal{D}} -\log P_\theta(x_i \mid x_j)$

where $x_i$ is a sentence in language $L_i$ (the target), $x_j$ is a sentence in language $L_j$ (the source), $\mathcal{D}$ is the union of all parallel training datasets, and $\theta$ represents all parameters of the multilingual Transformer model.

What it computes: For every parallel sentence pair in the training data, the model encodes the source sentence $x_j$ into a sequence of hidden representations, the decoder autoregressively predicts the target sentence $x_i$ token by token, and the loss penalizes the negative log-probability assigned to the correct target token at each position. The sum runs over all training pairs, meaning the model sees many language directions interleaved during training (English→French, French→English, English→German, etc.).

Why this form: Cross-entropy is the standard maximum-likelihood objective for sequence generation. It directly optimizes the model to assign high probability to the correct translation tokens given the source. For multilingual training, a single objective over all directions forces parameter sharing — the same encoder weights process French, German, and Chinese source sentences, and the same decoder weights generate French, German, and Chinese target sentences. This parameter sharing is what enables knowledge transfer, but it is implicit: nothing in this loss explicitly encourages the encoder to map parallel sentences in different languages to similar representations. The model may learn to associate English encodings with French decodings through one set of parameters and German encodings with French decodings through a different set, without ever aligning the French encoder pathway with the French decoder pathway across source languages. This is precisely the representation gap that mRASP2 addresses.

Multilingual Contrastive Learning

The core technical innovation of mRASP2 is adding a contrastive loss on the encoder's sentence-level representations that explicitly bridges the representation gap between languages. This loss operates in parallel with the translation loss during training and is computed on the exact same batches of data — no additional forward passes, no separate training phases, no architectural additions beyond extracting average-pooled encoder outputs.

Sentence representation extraction. For an arbitrary sentence $s$ (either source or target), the sentence representation $R(s)$ is defined as the average-pooled output of the encoder. Specifically, the encoder processes the sequence of tokens and produces a hidden vector at each position; $R(s)$ averages these position-wise vectors into a single fixed-length representation. Average pooling is chosen over alternatives (such as using the representation of a special [CLS] token or the last hidden state) because it provides a symmetric and stable summary that is less sensitive to sentence length and position-specific artifacts. The dimensionality of $R(s)$ is $d_{\text{model}} = 1024$ .

Positive and negative pairs. For each parallel sentence pair $(x_i, x_j) \in \mathcal{D}$ , where $x_i$ is in language $L_i$ and $x_j$ is in language $L_j$ :

$(x_i, x_j)$ is a positive pair — these sentences are translations of each other and should have similar representations. The contrastive loss will pull $R(x_i)$ and $R(x_j)$ together.
A negative example is formed by randomly selecting another sentence $y_j$ from the same language $L_j$ that is not the translation of $x_i$ . The pair $(x_i, y_j)$ is a negative pair — these sentences are unrelated and should have dissimilar representations. The contrastive loss will push $R(x_i)$ and $R(y_j)$ apart.

In-batch negative sampling. For computational efficiency, negative samples are drawn from within the same training batch — every other sentence in language $L_j$ in the current batch (except the true positive) serves as a negative. This is the standard approach in contrastive learning (popularized by SimCLR, Chen et al., 2020) and avoids the need for a separate negative sampling infrastructure or a memory bank. The trade-off is that the number of negatives is limited by batch size, and the negative distribution is biased toward the batch composition, but for large batches (the paper uses batches with approximately 3 million tokens), this provides sufficient negatives for effective contrastive learning.

Contrastive loss formulation. The contrastive loss for a single source sentence $x_j$ (looking across all target sentences in the batch to find its true translation) is:

$L_{\text{ctr}} = -\sum_{x_i, x_j \in \mathcal{D}} \log \frac{e^{\text{sim}^+(R(x_i), R(x_j)) / \tau}}{\sum_{y_j} e^{\text{sim}^-(R(x_i), R(y_j)) / \tau}}$

where:

$\text{sim}(\cdot, \cdot)$ is the cosine similarity between two sentence representations: $\text{sim}(a, b) = \frac{a \cdot b}{\|a\| \|b\|}$ . Cosine similarity is chosen over dot product or Euclidean distance because it normalizes for the magnitude of the representations, preventing the model from trivially minimizing the loss by making all representations have large norms (which would increase the numerator without improving alignment). The $+$ superscript on $\text{sim}^+$ denotes that this similarity is computed between a positive pair; the $-$ superscript on $\text{sim}^-$ denotes computation for negative pairs.
$R(x_i)$ is the average-pooled encoder output for the target sentence $x_i$ (positive example).
$R(x_j)$ is the average-pooled encoder output for the source sentence $x_j$ (the "anchor" from which we are searching).
$R(y_j)$ is the average-pooled encoder output for a randomly selected target sentence (negative example) from the same language $L_j$ as $x_i$ , drawn from within the current training batch.
$\tau$ is the temperature hyperparameter, set to $\tau = 0.1$ . Temperature controls the concentration of the softmax distribution: lower temperature makes the model more sensitive to small differences in similarity (harder negatives are penalized more), while higher temperature smooths the distribution. The value 0.1 is relatively low, meaning the model is strongly penalized for assigning even moderate similarity to negative pairs — this encourages tight clustering of positive pairs and clear separation from negatives. The paper notes that "higher temperature increases the difficulty to distinguish positive sample from negative ones," meaning that $\tau = 0.1$ makes the contrastive task easier (the model is more confident in its distinctions), which is appropriate when the negative set is large and contains many hard negatives (sentences in the same language that are semantically similar but not translations).

What it computes: For each source sentence in the batch, the model computes cosine similarities between its representation and the representations of all target sentences in the batch. These similarities are divided by $\tau$ and exponentiated. The loss for that source sentence is the negative log of the ratio: the exponentiated similarity to its true translation (positive) divided by the sum of exponentiated similarities to all target sentences in the batch (positive + all negatives). This is effectively a multi-class classification problem: given the representation of a source sentence, identify which of the $B$ target sentences in the batch (where $B$ is the batch size in sentences) is its true translation, using cosine similarity as the classification score. The loss is summed over all source sentences in the batch, meaning each sentence serves both as an anchor (looking for its own positive) and as a negative (appearing in the denominator of other sentences' losses).

Why this form: The contrastive loss is an InfoNCE (Noise-Contrastive Estimation) objective, which is a lower bound on the mutual information between the representations of parallel sentences. Minimizing this loss maximizes the mutual information between $R(x_i)$ and $R(x_j)$ for translation pairs, which is precisely the property we want: the representation of a sentence in French should be maximally informative about the representation of its translation in German (and vice versa). The softmax form with cosine similarity is standard in contrastive learning (Chen et al., 2020; Wu et al., 2020) because it provides a normalized similarity score in $[-1, 1]$ that is insensitive to representation magnitude. An alternative such as Euclidean distance would require the model to learn the appropriate scale for distances (since distances grow with dimension), and dot product similarity would be sensitive to representation norm (the model could cheat by making all representations have large norm, which would increase the numerator without improving alignment). Cosine similarity with temperature scaling provides a principled and empirically effective similarity function for representation learning.

Joint training objective. The contrastive loss is combined with the standard cross-entropy translation loss:

$L = L_{\text{ce}} + \lambda |s| L_{\text{ctr}}$

where:

$L_{\text{ce}}$ is the token-level cross-entropy translation loss (Equation 1).
$L_{\text{ctr}}$ is the sentence-level contrastive loss (Equation 2).
$\lambda$ is a hyperparameter balancing the two losses, set to $\lambda = 1.0$ in all experiments.
$|s|$ is the average sequence length in the batch (in tokens). This scaling factor is critical because $L_{\text{ce}}$ is summed over all tokens in the batch (so its magnitude scales with sequence length), while $L_{\text{ctr}}$ is computed per sentence (so its magnitude is independent of sequence length). Multiplying $L_{\text{ctr}}$ by $|s|$ normalizes the two losses to comparable magnitudes, preventing one from dominating the other simply due to different units (tokens vs. sentences). Without this scaling, the cross-entropy loss (which accumulates over potentially hundreds of tokens per sentence) would dwarf the contrastive loss (one scalar per sentence), making $\lambda$ difficult to tune and the contrastive signal ineffective.

What it computes: The total loss is a weighted sum of the translation quality objective (generate correct target tokens) and the representation alignment objective (make parallel sentence representations similar). The gradient from $L_{\text{ce}}$ primarily updates the decoder parameters and the token-level encoder representations, while the gradient from $L_{\text{ctr}}$ primarily updates the encoder to produce sentence-level representations that are language-agnostic. Both gradients flow through the encoder simultaneously, so the encoder learns to produce representations that are simultaneously useful for token-level translation and sentence-level semantic alignment.

Why this form: The additive combination of supervised and self-supervised objectives is a common pattern in representation learning (e.g., Conneau and Lample, 2019 combine masked language modeling with translation language modeling). The linear combination with a single hyperparameter $\lambda$ is the simplest possible fusion and avoids introducing additional learned parameters (such as uncertainty weighting or dynamic loss balancing). The choice of $\lambda = 1.0$ (after sequence-length normalization) means both losses are given equal weight — the model should care equally about generating correct translations and aligning representations. The paper does not report a sweep over $\lambda$ , suggesting that $\lambda = 1.0$ was chosen based on intuition or preliminary experiments and that the method is not highly sensitive to this choice. The $|s|$ scaling is a practical normalization that makes the hyperparameter transferable across datasets with different average sentence lengths.

Aligned Augmentation (AA)

Aligned Augmentation is a data preprocessing technique that creates pseudo-parallel training examples by replacing words in source sentences with their synonyms (specifically, translations from a multilingual dictionary). It extends the Random Aligned Substitution (RAS) technique from Lin et al. (2020) in two ways: (1) while RAS was applied only to parallel corpora, AA is applied to both parallel and monolingual corpora, and (2) while RAS created code-switched sentences for multilingual pre-training, AA's primary purpose in mRASP2 is to provide additional positive pairs for the contrastive loss by creating semantically equivalent but lexically perturbed versions of sentences.

Synonym dictionary. The synonym dictionary is drawn from MUSE (Lample et al., 2018), an open-source library for multilingual word embeddings that includes bilingual dictionaries for many language pairs. The dictionary maps words in one language to their translations in another — for AA, these translations serve as "synonyms" that can replace original words. The dictionary covers a subset of the languages in PC32 (specific coverage is detailed in Table 10 — languages like Gu, Kk, and Sr are marked with "/" indicating they are not contained in the synonym dictionary, so AA does not modify sentences in those languages). The paper releases the multilingual synonym dictionary as part of its open-source contribution.

AA on parallel corpora (Figure 2, left). For each bilingual sentence pair $(x_i, x_j)$ in the parallel training data:

Take the source sentence $x_j$ (in language $L_j$ ).
For each word in $x_j$ that appears in the synonym dictionary, replace it with one of its synonyms (translations into another language, or alternative words in the same language) with probability 90%. If the word is not in the dictionary, or if the 10% chance of no replacement occurs, the word is kept unchanged.
The result is a perturbed source sentence $C(x_j)$ — a "noised" version of the original where some words have been swapped for their translations. The target sentence $x_i$ remains unchanged.
The augmented pair $(C(x_j), x_i)$ is added to the training data. During training, this pair contributes to both the cross-entropy loss (the model must translate the noised source $C(x_j)$ to the clean target $x_i$ ) and the contrastive loss (the representation of the noised source $R(C(x_j))$ should be similar to the representation of the clean target $R(x_i)$ , since they are semantically equivalent despite the lexical perturbation).

The 90% replacement probability is intentionally high — it means that in expectation, 90% of replaceable words in the source are swapped. This creates substantial lexical divergence between $C(x_j)$ and $x_i$ , forcing the model to rely on semantic understanding rather than surface lexical overlap. The 10% chance of keeping the original word provides some stability and prevents the model from learning that every source word is unreliable. The paper applies AA "in the pre-processing step before training," meaning the augmented data is generated once offline rather than on-the-fly during training (which would be slower but could provide more diversity through stochastic augmentation).

AA on monolingual corpora (Figure 2, right). This is the novel extension beyond RAS. For each sentence $x_i$ in the monolingual dataset (MC24), where $x_i$ is in language $L_i$ :

Take the sentence $x_i$ .
Apply the same replacement procedure: for each word in the synonym dictionary, replace it with a synonym with probability 90%, producing a perturbed version $C(x_i)$ . The perturbed sentence may now contain words from multiple languages (since synonyms can be translations into other languages), but the semantics of the original sentence are preserved.
The augmented pair $(C(x_i), x_i)$ forms a pseudo self-parallel example: the source is a noised, potentially code-switched version, and the target is the original clean sentence in a single language.
During training, this pair contributes to the cross-entropy loss as a reconstruction task — the model must reconstruct the original sentence $x_i$ from its perturbed version $C(x_i)$ , which teaches the decoder to generate fluent text in language $L_i$ from noisy multilingual input. It also contributes to the contrastive loss — $R(C(x_i))$ should be similar to $R(x_i)$ since they represent the same semantic content — which aligns representations of code-switched and pure-language versions of the same meaning.

The reconstruction task is critical for zero-shot and unsupervised translation because it gives the decoder practice generating text in language $L_i$ without needing parallel English data. A model trained only on English-centric parallel data has seen the decoder generate French only when the source contains English; the reconstruction task on French monolingual data teaches the decoder to generate French from non-English input (the noised, code-switched French), which generalizes to generating French from German, Chinese, or any other source language at inference time.

Why 90% replacement probability. High replacement probability is key to the augmentation's effectiveness. If only a few words were replaced, $C(x_i)$ would be nearly identical to $x_i$ , and the reconstruction task would be trivial (the model could copy most words directly). At 90% replacement, the source is substantially different from the target, forcing the model to understand the semantic content rather than relying on lexical overlap. This is analogous to denoising autoencoders and the pre-training objectives of mBART and MASS, but with the crucial difference that the noise is semantically structured (words are replaced with synonyms, not random tokens or masks), which preserves the meaning while varying the surface form.

Why AA helps contrastive learning. The AA-augmented pairs $(C(x_j), x_i)$ and $(C(x_i), x_i)$ provide additional positive pairs for the contrastive loss. Without AA, positive pairs come only from the original parallel data, and for each source sentence there is exactly one positive (its translation). With AA, each original parallel pair generates one additional positive pair, and each monolingual sentence generates one pseudo-positive pair. This increases the density of positive examples in the representation space, providing more gradient signal for the contrastive loss to pull semantically equivalent sentences together. More importantly, the AA-transformed sources introduce lexical diversity — the model must learn that "Je t'aime" (French) and "私はあなたを愛しています" (Japanese translation of "I love you") should have similar representations even though they share no subword tokens, because the AA process has trained it to map perturbed representations to the same semantic anchor.

Monolingual data volume and balancing (Table 10). The MC24 dataset contains 1.01 billion sentences across 24 languages after temperature sampling. Temperature sampling with $T = 5$ is applied to balance the volume across languages:

$\tilde{n}_i = \left(\frac{n_i}{\sum_j n_j}\right)^{1/T}$

where $n_i$ is the original number of sentences in language $i$ . With $T = 5$ , the sampling is relatively flat — high-resource languages (English with 224M sentences, German with 320M) are heavily downsampled, while low-resource languages (Gujarati with 530K sentences, Estonian with 5M) are upsampled. The final "Num." column in Table 10 shows the post-sampling counts: English is reduced to 85.8M, German to 92.0M, while Gujarati is inflated to 18.7M (by repeating sentences). This balancing prevents high-resource languages from dominating the monolingual training signal, which would bias the representation space toward those languages and potentially harm zero-shot performance on low-resource directions.

The "% of replaced tokens" column in Table 10 indicates how many words in each language were found in the synonym dictionary and thus eligible for replacement. For languages with 0% or "/" (meaning not in the dictionary), AA creates no perturbation — the "augmented" sentence is identical to the original. For well-covered languages, replacement rates range from 0.15% (Japanese, Chinese) to 0.62% (English). These percentages appear low because only content words (nouns, verbs, adjectives) are typically in bilingual dictionaries, while function words, punctuation, and rare words are not. Even a 0.4% replacement rate on a 20-word sentence means roughly 0.08 words replaced on average — seemingly negligible — but the paper's strong results on unsupervised directions (Table 2, +14.13 BLEU from monolingual data) suggest that even sparse lexical perturbation combined with the contrastive objective is sufficient to meaningfully align representations.

Training Configuration and Hyperparameters

The training configuration reflects the challenges of scaling multilingual NMT to 32 languages with 97.6M parallel pairs and 1.01B monolingual sentences. The configuration choices are motivated by training stability, model capacity, and computational efficiency.

Hardware. Training uses 8 × 4 = 32 NVIDIA V100 GPUs with an update frequency of 50. "Update frequency 50" means that gradients are accumulated across 50 forward-backward passes before a single optimizer step — the effective batch size is $32 \times \text{per-GPU-batch} \times 50$ . This is necessary because the per-GPU batch size is constrained by the 1024-dimensional model and 12-layer depth running on V100 memory, and large batch sizes are critical for training stability and contrastive learning (which benefits from many in-batch negatives).

Batch size. Each training batch contains "about 3 million tokens." This is the total number of tokens across all GPUs after accumulation. For reference, if average sentence length is 30 tokens, this corresponds to roughly 100,000 sentences per effective batch, providing 100,000 in-batch negatives for the contrastive loss. The batch size "plays a crucial role in the success of training multilingual NMT" (Section 4.1) because large batches provide more stable gradient estimates, more negative examples for contrastive learning, and better mixing of different language pairs.

Optimizer. Adam (Kingma and Ba, 2015) with $\epsilon = 1 \times 10^{-6}$ and $\beta_2 = 0.98$ . The $\beta_2$ value of 0.98 (lower than the default 0.999) reduces the exponential moving average window for the squared gradients, making the optimizer more responsive to recent gradient changes — useful for the noisy gradient estimates that come from training on many language pairs with varying data sizes. The $\epsilon = 10^{-6}$ is also lower than the default $10^{-8}$ , providing more numerical stability for large models.

Learning rate. $3 \times 10^{-4}$ with polynomial decay scheduling and 10,000 warmup steps. The polynomial decay means the learning rate decreases from $3 \times 10^{-4}$ to 0 over the course of training according to a polynomial function (typically quadratic). The 10,000 warmup steps linearly increase the learning rate from 0 to $3 \times 10^{-4}$ at the start of training, which prevents the model from diverging in the early stages when gradients are large and noisy. This is standard practice for Transformer training.

Regularization. Dropout rate of 0.1 is applied to attention weights and feed-forward network activations. Gradient norm clipping with threshold 5.0 is applied — if the global gradient norm (L2 norm of all parameter gradients concatenated) exceeds 5.0, all gradients are scaled down proportionally to have norm exactly 5.0. The paper notes that "without this regularization, the large scale training will collapse sometimes," indicating that gradient spikes (likely from rare subword tokens or language combinations with very different loss scales) can destabilize the 12-layer model trained on 32 languages.

Contrastive loss coefficient. $\lambda = 1.0$ in Equation 3. The paper does not report a sweep over $\lambda$ , suggesting either that $\lambda = 1.0$ was chosen based on the intuition that both objectives are equally important (after sequence-length normalization) or that preliminary experiments showed insensitivity to this choice.

Training data mixture. The model is trained on a combination of:

Original PC32 parallel data (all 97.6M pairs).
AA-augmented PC32 parallel data (each original pair produces one augmented pair, though the paper does not specify whether the original and augmented pairs are both used or whether the augmented replaces the original — the text says AA "creates a pseudo-parallel training example $(C(x_i), x_j)$ " which is "then used in the training," suggesting both original and augmented pairs are included).
AA-augmented MC24 monolingual data (each monolingual sentence produces one pseudo self-parallel pair $(C(x_i), x_i)$ ).

The paper does not specify the exact ratio or interleaving strategy (e.g., whether parallel and monolingual batches are alternated, or mixed within each batch), but the joint training objective (Equation 3) is applied to all data types.

Inference Procedure

At inference time, mRASP2 behaves exactly like a standard multilingual Transformer — no contrastive loss, no AA, no special processing. The trained model is used for translation by specifying source and target language tokens and running beam search or greedy decoding. This is a key practical advantage: the training modifications improve the model's representations, but inference requires no additional computation or components.

Language specification. For a translation from language $L_i$ to language $L_j$ :

The source sentence is prepended with the source language token (e.g., <Fr>).
The decoder is initialized with the target language token (e.g., <De>), and generation proceeds autoregressively from there.

For zero-shot directions (language pairs never seen in training, e.g., Fr→De), the procedure is identical — the model has never seen <Fr> as source with <De> as target during training, but because the contrastive loss has aligned the encoder representations of French and German sentences, the German decoder pathway (learned from En→De training) can effectively process French encoder outputs.

Decoding strategy. The paper does not explicitly specify beam size or decoding parameters for the main results. Standard practice in NMT evaluation is beam search with beam size 4–8 and length normalization. The paper states it uses "SacreBLEU" and "de-tokenized BLEU" for evaluation, suggesting standard post-processing (detokenization, sacreBLEU signature). For Chinese, "BLEU score is calculated on character-level," following standard practice where word segmentation is ambiguous.

Unsupervised translation. For the unsupervised directions in Table 2 (En↔Nl, En↔Pt, En↔Pl, and Nl↔Pt), the model has never seen parallel data involving these language pairs in any form. For En↔Nl, En↔Pt, and En↔Pl, the model has seen monolingual data for these languages through MC24 (with AA applied), giving it some exposure to generating text in those languages via the reconstruction task. For Nl↔Pt, the model has only seen monolingual data for each side — no parallel data, no pivot through English — making it a genuinely unsupervised translation direction. The fact that mRASP2 achieves "9.3" BLEU for Nl→Pt and "8.3" for Pt→Nl indicates that the aligned representations are sufficiently language-agnostic to enable direct translation between languages that were only seen independently.

Representation extraction for analysis. For the similarity search (Section 5.2) and visualization (Section 5.3) experiments, the representation $R(s)$ is extracted identically to training: average-pooled encoder output. No fine-tuning or adaptation is applied — the same representations that were aligned during training are used for retrieval and visualization, demonstrating that the alignment persists to inference time and generalizes to unseen sentences (the Tatoeba and Ted-M test sets were not used during training).

4. Key Insights and Innovations

Innovation 1: Diagnosis That the Representation Gap, Not Data Scarcity, Is the Primary Bottleneck for Non-English Multilingual Translation

The paper's most consequential intellectual move is not proposing a new loss function — it is correctly diagnosing what was holding non-English translation quality back. Prior work on improving non-English directions in multilingual NMT had converged on a data-centric fix: if the model performs poorly on German→French, the solution must be to provide German→French parallel data, either through mining (Fan et al., 2020) or back-translation (Zhang et al., 2020). This is intuitively appealing — the model hasn't seen the direction, so show it the direction — and it does work, but at a cost. Zhang et al. explicitly sacrificed ~0.5 BLEU on English-centric directions to gain on zero-shot. The dominant assumption was a zero-sum tradeoff between English-centric and non-English quality, implying that multilingual models have fixed capacity that must be allocated across directions.

mRASP2 challenges this assumption at its root. The paper's diagnosis is that the bottleneck is not the absence of non-English supervision but rather the misalignment of cross-lingual representations in the encoder. The reasoning is causal: if a German sentence and its French translation occupy different regions of the encoder's latent space, then the decoder — which learned to generate French from English encoder states — cannot effectively process German encoder states to produce French output. The decoder knows how to generate French (from En→Fr training), and the encoder knows how to represent German (from En→De training), but these two capabilities cannot be composed because the intermediate representations are incompatible. The solution is therefore not to add more parallel data (which treats the symptom — poor Fr→De translation — by directly supervising it), but to align the representation spaces so that any language pair can be composed from independently learned capabilities.

This is a fundamental reframing rather than an incremental improvement. It shifts the problem from "how do we get more non-English parallel data?" to "how do we make the encoder's representation of meaning language-agnostic?" — which is a representation learning problem, not a data acquisition problem. The evidence supporting this diagnosis is in the ablation (Table 4): contrastive loss alone (row 3, "mRASP2 w/o AA") improves zero-shot translation from 5.05 to 13.55 BLEU (+8.5) with no additional data whatsoever — just a different training objective on the same English-centric parallel corpora. This is a clean causal test: if data scarcity were the bottleneck, changing only the loss function could not produce an 8.5 BLEU gain. The fact that it does is strong evidence that the representation gap, not data quantity, was the binding constraint.

The diagnostic is further strengthened by the negative result: contrastive loss alone leaves supervised English-centric directions essentially unchanged (+0.14 BLEU, row 3 vs. row 1 in Table 4). This makes sense under the diagnosis — English-centric directions don't suffer from the representation gap because English representations are well-aligned with all target languages through direct supervision — and would be puzzling under alternative hypotheses (e.g., that contrastive loss simply regularizes training or improves optimization). The asymmetry of the effect (huge zero-shot gains, negligible supervised gains) is precisely what the representation gap hypothesis predicts.

Innovation 2: Contrastive Learning as an Explicit Language-Agnostic Alignment Mechanism for Sequence-to-Sequence Models

Applying contrastive learning to sentence representations was not itself novel in 2021 — Wu et al. (2020) and Fang and Xie (2020) had already explored contrastive objectives for sentence embeddings, and contrastive learning was well-established in vision (Chen et al., 2020; He et al., 2020). What is distinctive about mRASP2's approach is where and how the contrastive loss is integrated into a sequence-to-sequence translation model.

Prior work on cross-lingual representation learning for NLP focused almost exclusively on encoder-only models (mBERT, XLM) trained with masked language modeling or translation language modeling (TLM) objectives for cross-lingual understanding tasks like XNLI or NER. TLM (Conneau and Lample, 2019) concatenates parallel sentences and trains the model to predict masked tokens — this does encourage cross-lingual alignment, but it operates at the token level within a single concatenated sequence, not at the sentence level across independently encoded sentences. The alignment signal is implicit: the model learns that a masked French word can be predicted from surrounding English context, which requires some cross-lingual alignment, but there is no explicit objective forcing the aggregate sentence representation to be language-agnostic.

mRASP2's innovation is adapting contrastive learning to the encoder of a sequence-to-sequence model in a way that is jointly trained with the translation objective. This matters for three reasons:

First, the integration point. The contrastive loss operates on the encoder's output — the same representations that are fed to the decoder's cross-attention. This means the encoder is optimized to produce representations that simultaneously serve two purposes: they must contain the token-level information the decoder needs to generate the correct translation (from the cross-entropy loss), and they must also produce a sentence-level summary that is discriminatively aligned across languages (from the contrastive loss). This joint optimization means the alignment is not a post-hoc fine-tuning step but is baked into the representations the decoder uses during inference. The decoder's cross-attention can attend to encoder states that are already language-agnostic, which is what enables zero-shot composition of independently learned encoder-decoder pathways.

Second, what is being contrasted. The paper contrasts average-pooled encoder outputs — a simple, symmetric sentence representation — rather than using a special [CLS] token or a separate projection head (as is common in SimCLR-style contrastive learning where a 2-3 layer MLP projects representations before computing the loss). This design choice is important: by using the raw average-pooled encoder output without an intermediate projection, the contrastive gradient flows directly into the encoder's token-level representations, forcing all positions — not just a dedicated pooling token — to contribute to the language-agnostic representation. This is a harder constraint than having a separate projection head that can learn to map encoder outputs to a contrastive-friendly space while the encoder itself remains language-specific.

Third, the training signal efficiency. Because the contrastive loss uses in-batch negatives from the same training data — parallel sentences that are already being processed for the cross-entropy loss — it requires no additional forward passes, no separate negative sampling infrastructure, and no auxiliary data. The only computation added is computing cosine similarities on the already-computed average-pooled representations. This is in contrast to prior contrastive sentence embedding methods that often require carefully constructed negative mining strategies or separate data pipelines.

The conceptual significance is that mRASP2 shows contrastive learning can serve as a general-purpose alignment objective for multilingual sequence-to-sequence models, not just for representation learning in encoder-only architectures. The fact that a simple cosine similarity loss on average-pooled representations — added to the standard translation objective — can produce 8.5 BLEU gains on zero-shot translation suggests that explicit alignment objectives are substantially underutilized in multilingual generation models. Prior to this work, the field's approach to multilingual NMT was to let alignment emerge implicitly from parameter sharing and the translation loss; mRASP2 demonstrates that explicit alignment can be far more effective without architectural changes.

Innovation 3: Aligned Augmentation on Monolingual Data as a Bridge Between Representation Learning and Generation Capability

The paper's extension of Random Aligned Substitution (RAS) from parallel-only to monolingual data is technically a small modification — apply the same dictionary-based word replacement to monolingual sentences and treat them as self-parallel pairs — but its conceptual significance is larger than the mechanism suggests. It bridges a gap that had separated representation learning approaches from generation capability in prior work.

The problem is this: many prior methods could improve cross-lingual representations (through pre-training, contrastive objectives, or interlingua architectures), but improving representations does not automatically improve generation. A model might learn excellent language-agnostic sentence embeddings — French and German paraphrases map to the same point — but still fail at zero-shot translation because the decoder has never generated French from anything other than English encoder states. The decoder's generation capability for French was trained exclusively on En→Fr data, so it expects encoder states that look like English. Even if the encoder now produces German states that are "close" to French states in cosine similarity, the decoder's autoregressive generation process — which involves cross-attention to individual encoder positions, not just an average-pooled vector — may still fail to produce fluent French from German encoder states.

Aligned Augmentation on monolingual data addresses this by creating a reconstruction task that teaches the decoder to generate in a language from non-standard encoder states. When the model trains on a pseudo self-parallel pair $(C(x_i), x_i)$ — a noised, potentially code-switched source sentence paired with the clean original — the decoder learns to generate fluent French (or German, or Chinese) from encoder states that are not the typical English-centric source it saw in parallel training. The noised source $C(x_i)$ has been perturbed by replacing words with translations from other languages, so its token-level encoder states are a hybrid — some French, some English, some other languages — yet the decoder must reconstruct the original clean sentence. This teaches the decoder to be robust to the language of the encoder states, decoupling its generation capability from the specific source language it was trained with.

This explains a key result in Table 4: contrastive learning alone improves zero-shot from 5.05 to 13.55 (row 3), but adding AA on parallel data and monolingual data with AA further improves it to 15.31 (row 5). The jump from 13.55 to 15.31 may seem modest compared to the initial 8.5 BLEU gain from contrastive loss, but the real contribution of monolingual AA is most visible in the unsupervised directions (Table 2): mRASP2 achieves 18.55 average BLEU on unsupervised directions versus 4.75 for contrastive loss alone (row 3) — a nearly 4× improvement. For the genuinely unsupervised Nl↔Pt direction (where the model has never seen parallel data or even English as a pivot for these languages), the model achieves 9.3 and 8.3 BLEU purely from monolingual data. This is only possible because the monolingual reconstruction task taught the decoder to generate in Dutch and Portuguese from non-standard encoder inputs.

The conceptual innovation is recognizing that representation alignment (making encoder outputs language-agnostic) and generation robustness (making the decoder work with diverse encoder states) are separate problems that require separate solutions. Contrastive loss addresses the first; AA on monolingual data addresses the second. Prior work that focused only on representation learning (e.g., XLM, multilingual BERT pre-training) or only on data augmentation for generation (e.g., back-translation for zero-shot) addressed one side of this divide but not both. mRASP2's combination of both techniques — and the demonstration that they are synergistic rather than redundant — provides a template for future work on multilingual generation: align representations explicitly, and decouple generation capability from source language through reconstruction objectives on monolingual data.

Innovation 4: Empirical Refutation of the Zero-Sum Tradeoff Between English-Centric and Non-English Quality in Multilingual NMT

A persistent concern in multilingual NMT research has been that improvements to non-English directions come at the cost of English-centric quality — the "curse of multilinguality" (Arivazhagan et al., 2019) where model capacity is a fixed pie, and allocating more to non-English directions means taking away from English. Zhang et al. (2020) explicitly observed this tradeoff, and it was widely assumed to be an inherent limitation of parameter sharing.

mRASP2 provides a clear empirical counterexample to this assumed tradeoff. Comparing the full mRASP2 system (Table 4, row 5) to the baseline (row 1):

Supervised English-centric: +1.71 BLEU (28.65 → 30.36)
Zero-shot non-English: +10.26 BLEU (5.05 → 15.31)
Unsupervised: +14.13 BLEU (4.42 → 18.55)

Every category improves, and the improvements are largest precisely where the baseline was weakest. This is not a redistribution of fixed capacity — it is a net gain in all directions. The paper explicitly calls this out: "mRASP2 improves zero-shot translation by a large margin without losing performance on English-Centric directions. Therefore, mRASP2 has a great potential to serve many-to-many translations, including both English-centric and non-English directions" (Section 4.2).

The significance of this finding extends beyond the specific numbers. It suggests that the capacity limitations observed in prior work were not fundamental constraints of model size but rather inefficiencies in how the available capacity was being used. When representations are poorly aligned, the model effectively needs to learn separate encoding pathways for each source language, each consuming capacity that could otherwise be shared. By forcing representations into a shared space, contrastive learning frees up capacity — the model doesn't need separate parameters to handle French→German versus English→German because the encoder produces the same kind of representation for semantically equivalent sentences regardless of source language. This is consistent with the paper's observation that increasing model depth from 6 to 12 layers helps multilingual NMT "by contrast, the gains from increasing the bilingual model size is not that large" (Section 4.1) — multilingual models benefit disproportionately from additional capacity because alignment mechanisms enable more efficient use of that capacity.

This finding also reframes the research agenda. If English-centric and non-English quality are not in fundamental tension, then the goal of a "true many-to-many Multilingual NMT that works well on any translation direction" (Section 7) is not a utopian aspiration but an achievable engineering target. The path forward is not to accept tradeoffs or to add ever more language-specific parameters, but to improve representation alignment so that all directions benefit from all available training data.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three categories of evaluation data: (1) WMT and IWSLT benchmarks for supervised English-centric directions — specifically WMT14 (En↔Fr, En↔De), WMT17 (En↔Zh, En↔Tr, En↔Fi), WMT16 (En↔Ro, En↔Cs, En↔Tr), WMT19 (En↔Ru), WMT13 (En↔Es), WMT20 (En↔Pl), WMT09 (En↔It), and IWSLT14 (En↔Nl); (2) OPUS-100 for language pairs not available in WMT/IWSLT (En↔Pt) and for all zero-shot evaluation (see below); (3) OPUS-100 zero-shot testset (Zhang et al., 2020) for 30 zero-shot translation directions across 6 languages (Arabic, Chinese, Dutch, French, German, Russian), producing 15 language pairs and 30 directed translation tasks, where Dutch is notably not included in the PC32 training data. The supervised evaluation covers 20 translation directions total (10 language pairs × 2 directions each), detailed in Table 9.
Base model(s). The foundation is a multilingual Transformer with 12 encoder layers and 12 decoder layers, model dimension 1024 with 16 attention heads, and FFN dimension 1024. This is larger than the standard Transformer-big (6 layers in Lin et al., 2020) and matches the depth of mBART (Liu et al., 2020). The model is trained from scratch — there is no pre-training phase, making it directly comparable to other "unified multilingual" baselines (m-Transformer, mRASP) but distinct from "pre-train & fine-tuned" approaches (mBART, XLM, MASS). The model uses pre-norm residual connections, Layer Normalization on word embeddings, and a shared BPE vocabulary of 64,808 subword tokens plus 59 language identification tokens (total 64,867). The choice of 12 layers is empirically motivated: "we enlarge the number of layers from 6 to 12 and observe significant improvements for multilingual NMT. By contrast, the gains from increasing the bilingual model size is not that large" (Section 4.1).
Metrics. The primary metric is tokenized BLEU for WMT/IWSLT/OPUS-100 supervised and unsupervised directions, computed using Sacremoses tokenization followed by the multi-bleu.pl script. For zero-shot directions on OPUS-100, de-tokenized BLEU is reported using SacreBLEU (Post, 2018). For Chinese, BLEU is calculated at the character level rather than word level due to the absence of natural word boundaries. The distinction between tokenized and de-tokenized BLEU matters for cross-table comparison: Table 1 and Table 2 report tokenized BLEU (matching prior work conventions), while Table 3 reports de-tokenized BLEU (following Zhang et al., 2020's OPUS-100 convention). Results are reported per translation direction (source→target) and as averages across direction groups.
Baselines. The paper compares against several categories of prior work, drawing numbers from published papers rather than re-running baselines (except for mRASP without fine-tuning, which the authors re-implemented at 12 layers):
- Bilingual baselines (per-direction training): Transformer-6 (Lin et al., 2020) and Transformer-12 (Liu et al., 2020), which train separate models for each language pair. These represent the upper bound that multilingual models aspire to match.
- Pre-train & fine-tuned models: Adapter (Bapna and Firat, 2019), mBART (Liu et al., 2020), XLM (Conneau and Lample, 2019), MASS (Song et al., 2019), and mRASP with fine-tuning (Lin et al., 2020). These models pre-train on large monolingual or parallel data, then fine-tune separately on each translation direction. They do not produce a single unified model for all directions.
- Unified multilingual models: Multi-Distillation (Tan et al., 2019), m-Transformer (the authors' own 12-layer many-to-many baseline trained on PC32 using the approach of Johnson et al., 2017 with Fan et al., 2020-style language indicators), and mRASP without fine-tuning (Lin et al., 2020, re-implemented at 12 layers with the authors' data). These represent the "single model for all directions" paradigm that mRASP2 belongs to.
- Pivot-based baseline (for zero-shot comparison in Table 3): A two-step translation using the m-Transformer — source language → English, then English → target language. This represents the practical alternative to direct zero-shot translation and serves as the upper bound that zero-shot approaches aspire to reach.
The authors' own m-Transformer baseline is the primary reference point for all mRASP2 variants, since it uses identical architecture, data, and training infrastructure — only the training objective and data augmentation differ.
Generation budget / compute accounting. The paper does not use a generation budget framework (as in the reference example about test-time compute scaling). Instead, training compute is implicitly controlled by fixing the architecture (12-layer encoder-decoder), data (PC32 + MC24), and training procedure. All models are trained once and evaluated at a single checkpoint. The key efficiency metric is model unification: one mRASP2 model replaces many bilingual or per-direction fine-tuned models. The paper does not report training FLOPs, wall-clock time, or inference latency comparisons. The batch size of approximately 3 million tokens (achieved through gradient accumulation across 32 V100 GPUs with update frequency 50) is noted as playing "a crucial role in the success of training multilingual NMT" (Section 4.1) — larger batches provide more in-batch negatives for contrastive learning and more stable gradient estimates across diverse language pairs.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Results are reported as single BLEU scores per direction. For the ablation study (Table 4), the authors train each configuration once and report the resulting BLEU. For the supervised directions in Table 1, results are drawn from a single evaluation on the standard WMT test sets (e.g., newstest2014 for En↔Fr). There is no discussion of statistical significance testing, minimum detectable effect sizes, or variance across random seeds. This is consistent with common practice in NMT benchmarking at the time but means that small BLEU differences (e.g., 0.2–0.5 BLEU between mRASP2 variants) should not be over-interpreted as reliable improvements. The zero-shot evaluation on OPUS-100 uses a fixed test set of 30 directions; the paper does not describe any data splitting or held-out validation for hyperparameter selection.

Main Quantitative Results

Supervised English-Centric Translation Directions

Headline result. mRASP2 achieves an average tokenized BLEU of 33.01 across the 10 representative directions shown in Table 1, compared to 31.03 for the authors' m-Transformer baseline (+1.98 BLEU) and 32.33 for mRASP without fine-tuning (+0.68 BLEU over mRASP). The full 20-direction results are in Table 9, where mRASP2 achieves 30.36 average BLEU versus 28.65 for m-Transformer (+1.71 BLEU).

Comparison across baseline categories (Table 1). The paper reports results on 10 representative WMT directions (5 language pairs × 2 directions each):

En→Fr (wmt14): mRASP2 achieves 43.5 BLEU, compared to 42.0 for m-Transformer (+1.5), 43.1 for mRASP without fine-tuning (+0.4), and 44.3 for the best bilingual system (mRASP with fine-tuning by Lin et al., 2020, which is a per-direction fine-tuned model, not a unified model). The 43.5 is also 2.4 BLEU above mBART's 41.1 (a per-direction fine-tuned model) and competitive with the 43.2 of Transformer-6 bilingual.
Fr→En: 39.3 (mRASP2) vs. 38.1 (m-Transformer), +1.2 BLEU. This is 5.9 BLEU above mBART's performance (not shown in Table 1 for this direction, but mBART's Fr→En is not explicitly reported — Table 1 only lists mBART scores for → directions for some pairs; the paper notes results are from Liu et al., 2020, where Fr→En is reported at similar levels).
En→Tr (wmt17): 21.4 vs. 18.8 (+2.6 BLEU). This is the largest absolute gain among the En→X directions shown in Table 1.
Tr→En: 25.8 vs. 23.1 (+2.7 BLEU).
En→Ro (wmt16): 34.5 vs. 32.8 (+1.7 BLEU). Note that for En→Ro, results are computed after removing Romanian dialects, following the standard WMT16 evaluation protocol.
Ro→En: 35.0 vs. 33.7 (+1.3 BLEU). The Ro→En baseline (33.7 for m-Transformer) is already stronger than mBART's 38.8? No — mBART achieves 38.8 on Ro→En (Table 1), which is higher than mRASP2's 35.0. This is worth noting: mBART, despite being a pre-train-then-fine-tune approach, outperforms the unified mRASP2 on this specific direction by 3.8 BLEU. This is a rare case where the per-direction fine-tuned model beats the unified model.
En→Fi (wmt17): 23.4 vs. 20.0 (+3.4 BLEU). This is the largest absolute gain among all English→X directions in Table 1.
Fi→En: 30.1 vs. 28.2 (+1.9 BLEU).

Comparison to pre-train & fine-tuned models (Table 1). Across the 10 directions shown, mRASP2 achieves better BLEU than mBART on 8 out of 10 directions (all except Ro→En and De→En, where mBART scores 38.8 and 34.2 respectively vs. mRASP2's 35.0 and not shown for De→En in the excerpt, but the text states mRASP2 "achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions"). The paper claims mRASP2 "outperforms" mBART on 10 WMT benchmarks (Abstract), which refers to 10 specific WMT test sets, not all directions.

Full 20-direction results (Table 9). The appendix table provides the complete picture:

mRASP2 improves over m-Transformer on 19 out of 20 directions. The only exception is En→Zh (wmt17), where mRASP2 achieves 34.6 vs. m-Transformer's 32.8 (+1.8) in Table 1, but in Table 9, mRASP2 w/o MC24 achieves 35.0 vs. m-Transformer's 32.8, while the full mRASP2 achieves 34.6 — actually lower than the w/o MC24 variant on this direction, and the paper does not discuss this anomaly.
The largest gains are on En→Fi (+3.4 BLEU, from 20.0 to 23.4) and En→It (+1.8 BLEU, from 29.0 to 30.8). The smallest gains are on Fr→En (+1.2, from 38.1 to 39.3) and Es→En (+1.3, from 33.7 to 35.0, not shown in the excerpt but listed in Table 9).
The average gain of +1.71 BLEU across 20 directions is consistent and positive, suggesting the improvements are not concentrated in a few outlier directions.

Ablation progression across mRASP2 variants (Table 9). The step-by-step improvement from the baseline through successive additions shows the contribution of each component on supervised directions:

m-Transformer baseline: 28.65 average
mRASP2 w/o AA (contrastive loss only): 28.79 (+0.14) — essentially flat, consistent with the hypothesis that contrastive loss primarily helps non-English directions
mRASP (RAS augmentation, no contrastive loss, no monolingual data): 29.82 (+1.17) — notable improvement, suggesting RAS augmentation on parallel data alone benefits supervised directions
mRASP2 w/o MC24 (contrastive loss + AA on parallel data): 29.96 (+1.31) — additional +0.14 over mRASP, from adding contrastive loss
Full mRASP2 (contrastive loss + AA on parallel + AA on monolingual): 30.36 (+1.71) — additional +0.40 from adding monolingual data with AA

The supervised results tell a clear story: AA on parallel data (mRASP) provides the bulk of the gain (+1.17 over baseline), contrastive loss adds a small further improvement (+0.14 over mRASP, comparing mRASP2 w/o MC24 to mRASP), and monolingual AA adds another modest gain (+0.40 over mRASP2 w/o MC24). This contrasts sharply with the zero-shot results, where contrastive loss is the dominant contributor.

Unsupervised Translation Directions

Headline result. mRASP2 achieves an average tokenized BLEU of 18.55 across 6 unsupervised directions (3 language pairs × 2 directions each) shown in Table 2, compared to 4.42 for the m-Transformer baseline (+14.13 BLEU). For the genuinely unsupervised Nl↔Pt direction (where the model has seen only monolingual data for each language, never any parallel data involving either), mRASP2 achieves 9.3 (Nl→Pt) and 8.3 (Pt→Nl).

Per-direction results (Table 2). The unsupervised directions are language pairs where no parallel data was seen during training, but where monolingual data was available through MC24:

En→Nl (iwslt2014): 10.1 (mRASP2) vs. 1.3 (m-Transformer) — an 8.8 BLEU gain from effectively zero.
Nl→En: 28.5 vs. 7.0 — a 21.5 BLEU gain. The remarkable asymmetry (28.5 for Nl→En vs. 10.1 for En→Nl) is consistent across all unsupervised En↔X pairs and in mRASP (0.7 for En→Nl vs. 10.6 for Nl→En). X→En is consistently easier because the model has extensive training decoding English from many source languages, so generating English is well-practiced regardless of source.
En→Pt (opus-100): 18.4 vs. 3.7 (+14.7).
Pt→En: 30.5 vs. 10.7 (+19.8).
En→Pl (wmt20): 6.7 vs. 0.6 (+6.1).
Pl→En: 17.1 vs. 3.2 (+13.9).
Nl→Pt: 9.3 — this direction has no English on either side and no parallel data of any kind. The model achieves this purely from monolingual AA reconstructions of Dutch and Portuguese independently, combined with representation alignment through contrastive learning.
Pt→Nl: 8.3 — similar BLEU to the reverse direction, suggesting the representation space is genuinely symmetric for these languages.

The average of 18.55 excludes the Nl↔Pt directions (as noted in Table 2), making the comparison to m-Transformer fair (which has no score for those directions). Including Nl↔Pt, the average would be lower, but these directions represent a qualitatively different capability — genuinely zero-resource translation with no parallel signal of any kind.

Comparison to mRASP (no contrastive loss). mRASP without fine-tuning achieves 5.40 average BLEU on these directions (excluding Nl↔Pt), compared to m-Transformer's 4.42 — only a +0.98 gain. This is a critical data point: RAS augmentation on parallel data alone (mRASP) does not unlock unsupervised translation. The contrastive loss is essential. On Nl→En, mRASP achieves 10.6 vs. mRASP2's 28.5 — a 17.9 BLEU gap. On En→Nl, mRASP achieves 0.7 vs. mRASP2's 10.1. The contrastive loss provides the representation alignment that makes these directions viable; AA on monolingual data provides the decoder robustness to generate from the aligned representations.

Why mRASP2 succeeds on Nl↔Pt. This is the most extreme test of the representation alignment hypothesis. For Nl↔Pt:

The model has never seen parallel data involving Dutch or Portuguese (they are not in PC32).
The model has seen monolingual data for both languages through MC24 with AA applied, creating reconstruction tasks (C(Dutch)→Dutch and C(Portuguese)→Portuguese).
Through contrastive learning on other language pairs (e.g., English↔French, English↔German), the encoder has learned to produce language-agnostic representations — the representation of a Dutch sentence is pulled toward the representation of its meaning in whatever other languages appeared in the same batch.
At inference time, encoding a Dutch sentence produces a representation in the shared space. The decoder, having been trained via AA reconstruction to generate Portuguese from non-Portuguese encoder states (the AA-noised input), can decode this representation into Portuguese.

The 9.3/8.3 BLEU scores are modest but remarkable given zero parallel signal. They demonstrate that the combination of contrastive alignment + AA monolingual reconstruction can, in principle, enable translation between languages that share no training data whatsoever.

Zero-Shot Translation for Non-English Directions

Headline result. mRASP2 achieves 15.31 average de-tokenized BLEU on 30 OPUS-100 zero-shot directions (Table 3), compared to 5.05 for the m-Transformer baseline (+10.26 BLEU), 13.55 for mRASP2 w/o AA (contrastive loss only, +8.50 over baseline, from Table 4), and 15.56 for the pivot-based approach using m-Transformer (source→English→target). The gap to pivot-based translation is only −0.25 BLEU, meaning mRASP2 essentially matches the performance of routing through English — but does so in a single forward pass.

Per-language breakdown (Table 3). The zero-shot testset covers 6 languages, producing 30 directed translation pairs (5 target languages × 6 source languages, minus the 6 self-translation pairs). Table 3 aggregates results by the non-English language involved:

Arabic (Ar): mRASP2 achieves 5.3 for X→Ar and 17.3 for Ar→X, compared to m-Transformer's 3.7 and 5.6 respectively. The pivot baseline achieves 5.5 and 17.0 — mRASP2 is competitive in both directions, actually exceeding the pivot on Ar→X (17.3 vs. 17.0). The X→Ar scores are low across all methods (5.5 for pivot, 5.3 for mRASP2), suggesting Arabic generation is inherently difficult with this training setup.
Chinese (Zh): 29.0 (X→Zh) and 14.5 (Zh→X) for mRASP2, vs. 6.7 and 4.1 for m-Transformer. The pivot achieves 28.5 and 16.4. mRASP2 exceeds pivot on X→Zh (29.0 vs. 28.5) but lags on Zh→X (14.5 vs. 16.4). The high X→Zh scores may reflect Chinese being character-level BLEU (which tends to produce higher numbers than word-level) combined with good decoder generation capability for Chinese.
Dutch (Nl): 5.3 (X→Nl) and 6.1 (Nl→X) for mRASP2, vs. 2.3 and 6.3 for m-Transformer, vs. 2.2 and 6.0 for pivot. Dutch is notably not in PC32, meaning the model has never seen Dutch parallel data — yet mRASP2's zero-shot Dutch results are comparable to or slightly better than the pivot baseline, which also has no direct Dutch parallel data. This demonstrates that the monolingual AA reconstruction (Dutch is in MC24) combined with contrastive alignment is sufficient to enable zero-shot Dutch translation at a modest level.
French (Fr): 23.6 (X→Fr) and 21.7 (Fr→X) for mRASP2, vs. 7.7 and 4.8 for m-Transformer, vs. 26.1 and 22.3 for pivot. The gap to pivot is −2.5 and −0.6 respectively. French benefits from being a high-resource language in PC32 (extensive En↔Fr parallel data), and mRASP2's zero-shot French results are strong — but still lag slightly behind the pivot, which directly leverages the strong En↔Fr supervised models.
German (De): 12.3 (X→De) and 15.0 (De→X) for mRASP2, vs. 4.2 and 4.8 for m-Transformer, vs. 14.4 and 14.2 for pivot. De→X outperforms pivot (15.0 vs. 14.2), while X→De lags (12.3 vs. 14.4), showing an asymmetry that may reflect German's morphological complexity making generation (X→De) harder than encoding (De→X).
Russian (Ru): 16.4 (X→Ru) and 19.1 (Ru→X) for mRASP2, vs. 5.7 and 4.8 for m-Transformer, vs. 16.6 and 19.9 for pivot. Russian zero-shot is nearly at parity with pivot (−0.2 and −0.8 BLEU respectively).

Detailed per-pair results (Table 8). The appendix provides the full 30-direction matrix. Notable patterns:

The best zero-shot directions are those involving languages with large PC32 representation: Fr→Zh (42.3 BLEU), Zh→Fr (32.9 BLEU), Ru→Zh (40.6 BLEU). These directions benefit from strong parallel supervision for both languages individually through English.
The worst zero-shot directions involve Dutch: Nl→Ar (1.7 BLEU), Ar→Nl (3.2 BLEU), Nl→Zh (1.9 BLEU). Dutch's absence from PC32 means the encoder's Dutch representations and the decoder's Dutch generation are both purely from monolingual AA, providing a weaker foundation.
Asymmetry is pervasive: X→Zh is consistently higher than Zh→X (e.g., Fr→Zh 42.3 vs. Zh→Fr 32.9; Ru→Zh 40.6 vs. Zh→Ru 23.7). This likely reflects Chinese being evaluated at character level (inflating BLEU) plus the model generating Chinese more fluently than it generates other languages from Chinese input.

Comparison across mRASP2 variants on zero-shot (Table 4, zero-shot column). The progression reveals the critical role of contrastive loss:

m-Transformer: 5.05 — essentially unusable zero-shot quality.
mRASP w/o fine-tuning: 4.91 — adding RAS augmentation on parallel data slightly harms zero-shot (-0.14 BLEU). This is a striking negative result: data augmentation alone, without contrastive alignment, can degrade zero-shot performance. The AA-perturbed source sentences may have confused the already-misaligned representations, making zero-shot composition even harder.
mRASP2 w/o AA (contrastive loss only): 13.55 (+8.50 over baseline) — the single largest jump. This cleanly isolates the effect of contrastive loss: no additional data, no AA, just a different training objective on the same parallel corpora. The +8.50 BLEU gain demonstrates that representation misalignment was the dominant bottleneck for zero-shot translation.
mRASP2 w/o MC24 (contrastive loss + AA on parallel data): 14.60 (+1.05 over contrastive-only) — adding AA on parallel data provides a modest additional gain over contrastive loss alone. The AA-augmented parallel pairs provide more positive examples for contrastive learning and teach the model to be robust to lexical variation.
Full mRASP2 (contrastive loss + AA on parallel + AA on monolingual): 15.31 (+0.71 over w/o MC24) — adding monolingual AA provides the final increment, bringing zero-shot to near-parity with pivot.

The zero-shot results are the paper's strongest empirical finding. The contrastive loss alone accounts for 83% of the total improvement over baseline (8.50 out of 10.26 BLEU), confirming the paper's central hypothesis that representation misalignment was the primary bottleneck. The additional gains from AA on parallel data (+1.05) and monolingual data (+0.71) are incremental but bring the system to a practically useful level (15.31 BLEU, competitive with pivot-based translation).

Similarity Search as a Diagnostic for Representation Alignment

Headline results (Tables 5 and 6). mRASP2 substantially improves cross-lingual sentence retrieval accuracy compared to m-Transformer, on both English-centric (Tatoeba dataset) and non-English (Ted-M) evaluation sets. This provides independent evidence — beyond BLEU scores — that the contrastive loss and AA are genuinely aligning representations across languages.

English-centric retrieval (Table 5). On the Tatoeba dataset, top-1 accuracy for retrieving the correct translation is reported for 10 languages, averaged across En→X and X→En directions:

For languages well-represented in PC32 (Fr, De, Zh, Ro, Cs, Tr, Ru), mRASP2 achieves accuracies of 90.7–98.0%, compared to m-Transformer's 84.8–96.8%. Gains are modest (+1–4 percentage points) because these languages already have strong implicit alignment from extensive parallel data.
For languages poorly represented or absent from PC32 (Nl, Pl, Pt), mRASP2 achieves dramatically higher accuracy: Nl goes from 25.6% to 60.3% (+34.7 points), Pl from 6.3% to 28.1% (+21.8 points), Pt from 37.3% to 58.6% (+21.3 points). The mRASP2 w/o AA (contrastive only) intermediate also shows substantial gains for these languages (e.g., Nl 25.6% → 35.7%), indicating that contrastive loss alone starts the alignment, and AA on monolingual data (which provides reconstruction tasks for Nl, Pl, Pt through MC24) provides further improvement. This pattern directly mirrors the BLEU results: the representation alignment from contrastive learning is most impactful for languages with limited parallel supervision.
Two trends are explicitly noted: "(i) The overall accuracy follows the rule: m-Transformer < mRASP2 w/o AA < mRASP2. (ii) mRASP2 brings more significant improvements for languages with less data volume in PC32" (Section 5.2). These support the representation alignment hypothesis: if contrastive learning were merely regularizing training or providing a generic optimization benefit, the improvements would be uniform across languages. The fact that gains concentrate on low-resource languages is exactly what the alignment hypothesis predicts — these are the languages whose representations were most poorly aligned by the translation loss alone.

Non-English retrieval (Table 6). The Ted-M testset (2,284 multi-way parallel sentences across 15 languages) enables evaluation of similarity search on 210 non-English language pairs — the model must retrieve the correct translation in language $L_j$ given a query in language $L_i$ , where neither is English:

Average top-1 accuracy: m-Transformer 79.8%, mRASP2 w/o AA 84.4% (+4.8), mRASP2 89.6% (+9.8 over baseline, +5.2 over contrastive-only).
These are substantial improvements considering the baseline is already at 79.8% — the contrastive loss and AA push retrieval accuracy close to 90%, indicating that the representations are genuinely language-agnostic for the majority of sentence pairs.

Per-language-pair heatmap (Figure 3). The heatmap visualization of accuracy improvements reveals:

mRASP2 w/o AA over m-Transformer (Figure 3a): Improvements are widespread but heterogeneous. Most language pairs show light improvements (0.0–0.1 ∆acc), with some pairs showing stronger gains. Dutch (Nl) row and column show some of the largest gains, consistent with Dutch being absent from PC32 and therefore having poorly aligned representations in the baseline.
mRASP2 over mRASP2 w/o AA (Figure 3b): The additional gains from AA on parallel and monolingual data are concentrated and larger in magnitude (0.2–0.4 ∆acc). The Dutch row and column show the most dramatic improvements — dark red cells indicating large accuracy jumps — which is directly attributable to MC24 including Dutch monolingual data. This is the clearest evidence that AA on monolingual data specifically benefits languages with no parallel training data by providing reconstruction-based decoder practice and additional positive pairs for contrastive learning.

The similarity search results serve as a crucial independent validation of the representation alignment hypothesis. BLEU improvements could in principle arise from other factors (better decoder language modeling from monolingual data, regularization effects, etc.), but similarity search accuracy directly measures whether the encoder maps semantically equivalent sentences in different languages to nearby points in representation space. The fact that retrieval accuracy improves in lockstep with translation BLEU — and that both are largest for low-resource languages — provides converging evidence that representation alignment is the causal mechanism.

Ablation Studies and Robustness Checks

Contrastive loss alone (mRASP2 w/o AA): Adding only the contrastive loss to the m-Transformer baseline (no AA, no monolingual data) improves zero-shot translation dramatically (+8.50 BLEU, from 5.05 to 13.55, Table 4) but leaves supervised directions essentially unchanged (+0.14 BLEU, from 28.65 to 28.79, Table 4) and provides only marginal unsupervised gains (+0.33 BLEU, from 4.42 to 4.75, Table 4). This is the cleanest causal test of the representation alignment hypothesis and strongly supports it. The asymmetry — huge zero-shot gains, negligible supervised gains — is predicted by the hypothesis that zero-shot translation is bottlenecked by representation misalignment while supervised directions already benefit from direct cross-entropy alignment through English.

Aligned Augmentation on parallel data only (mRASP vs. mRASP2 w/o AA): Comparing mRASP (RAS on parallel data, no contrastive loss, no monolingual data) to mRASP2 w/o AA (contrastive loss only) reveals that RAS alone is ineffective for zero-shot: mRASP achieves 4.91 average zero-shot BLEU versus 13.55 for contrastive loss alone (Table 4). In fact, mRASP's zero-shot BLEU is slightly worse than the 5.05 baseline, suggesting that AA without contrastive alignment can degrade zero-shot performance, possibly by confusing the encoder's representations with code-switched inputs. Meanwhile, on supervised directions, mRASP (29.82) actually outperforms contrastive-only mRASP2 w/o AA (28.79) by +1.03 BLEU (Table 4). This shows that RAS and contrastive loss have complementary, non-overlapping benefits: RAS helps supervised directions (by providing additional parallel training examples), and contrastive loss helps zero-shot directions (by aligning representations). Neither alone achieves both.

Monolingual data with AA (mRASP2 vs. mRASP2 w/o MC24): Adding MC24 with AA provides gains across all three scenarios: supervised +0.40 BLEU (29.96 to 30.36), zero-shot +0.71 (14.60 to 15.31), and a massive +12.75 BLEU on unsupervised directions (5.80 to 18.55, Table 4). The unsupervised gain is the most remarkable — it represents going from near-zero performance to usable quality on directions never seen in any form (like Nl↔Pt). This ablation cleanly demonstrates that monolingual data with AA is the key enabler for unsupervised translation, while contrastive loss and parallel AA provide the foundation through representation alignment.

Contrastive loss + AA on parallel data but no monolingual data (mRASP2 w/o MC24): This configuration achieves 29.96 supervised, 5.80 unsupervised, and 14.60 zero-shot (Table 4). Compared to the full mRASP2, the zero-shot gap is modest (-0.71 BLEU), indicating that monolingual data provides incremental value for zero-shot beyond what contrastive loss and parallel AA already achieve. Compared to contrastive-only (mRASP2 w/o AA), both supervised (+1.17 BLEU from 28.79 to 29.96) and zero-shot (+1.05 from 13.55 to 14.60) improve, showing that AA on parallel data complements contrastive loss for both categories. The unsupervised improvement is +1.05 BLEU (4.75 to 5.80) — better than contrastive-only but still far from usable, confirming that monolingual data is the critical missing ingredient for unsupervised translation.

Language-specific effects in similarity search (Figure 3): The per-language heatmap shows that Dutch (not in PC32) benefits dramatically from both contrastive loss and AA. mRASP2 w/o AA improves Dutch retrieval accuracy over m-Transformer (darker cells in the Nl row/column of Figure 3a), confirming that contrastive loss can align representations for languages without parallel data by leveraging cross-lingual transfer from other languages. The further improvement from full mRASP2 (Figure 3b) is largest for Dutch, consistent with MC24 providing Dutch monolingual AA that specifically targets this language's representations. This language-level granularity strengthens the claim that benefits are not from generic regularization but from targeted representation alignment.

Representation visualization (Figure 4): T-SNE projections of 34,260 sentence representations from Ted-M (15 languages, 2,284 sentences each) onto 2 dimensions show that m-Transformer's representations for English, Japanese, and German form distinct, partially overlapping clusters (Figure 4a). Under mRASP2 (Figure 4b), the three languages' representations overlap almost completely, forming a single unified density. This is qualitative visual evidence for the representation alignment claim and is consistent with the quantitative retrieval results. A limitation: T-SNE is a non-linear dimensionality reduction that can distort distances; the visualization should be interpreted as suggestive rather than conclusive. The paper acknowledges this implicitly by presenting the retrieval results (which are quantitative and use the full 1024-dimensional representations) alongside the visualizations.

Larger model depth (12 layers vs. 6 layers): The paper states that "we enlarge the number of layers from 6 to 12 and observe significant improvements for multilingual NMT. By contrast, the gains from increasing the bilingual model size is not that large" (Section 4.1). This is not presented as a formal ablation table but is an important empirical observation that motivated the architecture choice. It suggests that multilingual models are capacity-constrained in ways that bilingual models are not — additional depth helps disentangle language-specific and language-agnostic features — and that mRASP2's improvements come on top of a strong 12-layer baseline, not a weak 6-layer one.

Batch size and training stability: The paper notes that batch size "plays a crucial role in the success of training multilingual NMT" (Section 4.1), with approximately 3 million tokens per effective batch achieved through gradient accumulation. The paper also notes that gradient norm clipping at 5.0 is essential: "without this regularization, the large scale training will collapse sometimes" (Section 4.1). These are not formal ablations but practical engineering insights that affected all results — a reader attempting to reproduce mRASP2 without large-batch training and gradient clipping would likely see degraded performance.

Negative result: mRASP degrades zero-shot. Although not highlighted by the authors as a negative result, Table 4 reveals that mRASP (RAS augmentation on parallel data without contrastive loss) achieves 4.91 zero-shot BLEU, which is slightly lower than the 5.05 baseline. This is a meaningful negative finding: data augmentation that creates code-switched source sentences can actually harm zero-shot translation when not accompanied by an explicit representation alignment objective. The AA-perturbed sources may cause the encoder to produce more language-specific representations (to disambiguate the code-switched input), which further misaligns representations across languages and degrades zero-shot composition.

Critical Assessment

Does mRASP2 genuinely establish that representation alignment is the primary bottleneck for non-English translation?

What the evidence shows. The ablation (Table 4) provides a clean causal test: adding contrastive loss alone to the training objective — no new data, no architecture changes, just a different loss on the same parallel corpora — improves zero-shot translation from 5.05 to 13.55 BLEU (+8.50), while leaving supervised English-centric directions essentially unchanged (+0.14 BLEU). This is strong evidence that representation misalignment was a binding constraint specifically for zero-shot translation, and that the contrastive loss addresses it. The similarity search experiments (Tables 5 and 6) independently confirm that the contrastive loss improves cross-lingual representation alignment, and the visualization (Figure 4) shows qualitatively that the representations become more overlapping.

What the evidence does NOT show. The paper does not demonstrate that the contrastive loss is necessary for representation alignment — only that it is sufficient and effective. It is possible that other methods (larger models with more capacity, different training schedules, architectural modifications like dedicated language embedding layers) could achieve similar alignment without an explicit contrastive objective. The paper's claim is framed as a diagnosis ("the representation gap... is the primary bottleneck") rather than a uniqueness claim for contrastive learning, which is appropriate given the evidence.

A stronger concern: the paper measures representation alignment through similarity search accuracy (Tables 5 and 6) and T-SNE visualization (Figure 4), but the causal link between alignment and zero-shot BLEU is correlational, not directly tested. The paper shows that contrastive loss improves both alignment and zero-shot BLEU, and that both improvements are largest for low-resource languages. This is consistent with alignment causing the BLEU improvement but does not exclude alternative explanations — for example, the contrastive loss might be providing a beneficial regularization effect on the encoder that improves zero-shot translation through a different mechanism (e.g., smoother encoder representations that the decoder generalizes better to), with the retrieval accuracy improvement being a side effect rather than the causal driver. The paper does not perform an experiment that manipulates alignment independently (e.g., by varying the temperature τ or the λ coefficient and showing a dose-response relationship between alignment and BLEU) that would strengthen the causal claim.

Does the evidence support the claim that mRASP2 eliminates the tradeoff between English-centric and non-English quality?

What the evidence shows. Comparing the full mRASP2 to the m-Transformer baseline: supervised directions improve (+1.71 BLEU, Table 9), zero-shot directions improve (+10.26 BLEU, Table 4), and unsupervised directions improve (+14.13 BLEU, Table 4). Every category gets better — some dramatically so. This is a clear counterexample to the assumed zero-sum tradeoff in prior work (Zhang et al., 2020; Arivazhagan et al., 2019). The paper legitimately demonstrates that it is possible to improve non-English directions without sacrificing English-centric quality, at least in this specific setup.

What the evidence does NOT show. The comparison is to a baseline (m-Transformer) that the same authors designed and trained. The baseline's English-centric quality (28.65 average BLEU on 20 directions, Table 9) is strong — competitive with mBART and other approaches — but we do not know whether this baseline represents the upper bound of what is achievable on English-centric directions with this data and architecture. If the baseline had been pushed closer to its ceiling (e.g., through more extensive hyperparameter tuning, longer training, or better data filtering), the English-centric gains from mRASP2 might have been smaller or nonexistent, and the non-English gains might have come partially at the expense of English-centric quality. In other words, the paper shows that mRASP2 improves over this specific baseline on all fronts, but does not show that it would improve over any possible baseline, including one that had already maximized English-centric performance.

A related concern: the paper compares mRASP2 to mBART and claims it "achieves competitive or even better performance" (Abstract, Table 1). But mBART is a pre-train-then-fine-tune approach that uses separate fine-tuned models per direction — it is not a unified many-to-many model. The comparison is apples-to-oranges in terms of deployment: mRASP2 provides one model for all directions, mBART requires N fine-tuned models. The paper's claim of "outperforming mBART" should be understood as "a single mRASP2 model outperforms per-direction fine-tuned mBART models on some directions," which is impressive for model efficiency but does not represent a fair quality comparison at equal compute — mBART's pre-training and per-direction fine-tuning likely use substantially more total computation than mRASP2's single training run.

Does the evidence support the claim that Aligned Augmentation on monolingual data is essential for unsupervised translation?

What the evidence shows. The unsupervised translation results (Table 2) are the most dramatic in the paper: mRASP2 achieves 18.55 average BLEU on 6 unsupervised directions versus 4.42 for m-Transformer (+14.13 BLEU) and 5.40 for mRASP (+13.15 over mRASP). The ablation (Table 4, unsupervised column) shows that contrastive loss alone (4.75) and contrastive + parallel AA (5.80) provide only marginal improvements — it is the addition of MC24 with AA that produces the jump to 18.55. For the genuinely unsupervised Nl↔Pt direction, the model achieves 9.3/8.3 BLEU from monolingual data alone. The AA-on-monolingual-data mechanism — creating a reconstruction task that teaches the decoder to generate in a language from non-standard encoder states — is theoretically well-motivated, and the empirical results are striking.

What the evidence does NOT show. The unsupervised evaluation in Table 2 covers only 8 directions (4 language pairs × 2 directions), and one of those pairs (Nl↔Pt) is excluded from the average. This is a very small evaluation set. The paper does not report results on standard unsupervised translation benchmarks (e.g., the WMT unsupervised translation shared tasks, which have established evaluation protocols), making it difficult to compare the 18.55 BLEU to prior work on unsupervised NMT. The numbers are internally consistent — each direction shows large improvements over baseline — but the small sample size means we cannot assess how representative these 8 directions are of unsupervised translation more broadly.

A deeper concern: the unsupervised directions in Table 2 (En↔Nl, En↔Pt, En↔Pl) all involve Dutch, Portuguese, and Polish — languages that are present in MC24 with monolingual AA. The paper does not test unsupervised translation for a language that is not in MC24, or for a pair where neither language has any training data whatsoever (not even monolingual). The Nl↔Pt case is the closest — neither language is in PC32 — but both are in MC24. The paper's claim that monolingual AA enables unsupervised translation is supported only for languages that have extensive monolingual AA data. It is not demonstrated that the effect generalizes to languages with zero training data of any kind.

Additionally, the asymmetric performance on En↔X unsupervised directions (e.g., Nl→En at 28.5 BLEU vs. En→Nl at 10.1 BLEU) suggests that the unsupervised capability is heavily skewed toward X→English directions — where the decoder is generating English, which it does well from any source — rather than English→X, where the decoder must generate in a language it has only seen through AA reconstruction. The paper does not discuss this asymmetry or its implications for the universality of the unsupervised translation capability.

Are the zero-shot results genuinely competitive with pivot-based translation?

What the evidence shows. On the 30 OPUS-100 zero-shot directions, mRASP2 achieves 15.31 average BLEU versus 15.56 for pivot-based translation using m-Transformer (Table 3) — a gap of only −0.25 BLEU. On individual language aggregates, mRASP2 sometimes exceeds pivot (Ar→X: 17.3 vs. 17.0; X→Zh: 29.0 vs. 28.5; De→X: 15.0 vs. 14.2) and sometimes lags (X→De: 12.3 vs. 14.4; Zh→X: 14.5 vs. 16.4). The headline claim that mRASP2 "substantially narrows the gap with pivot-based model" (Section 4.2) is well-supported.

What the evidence does NOT show. The pivot baseline uses m-Transformer, while mRASP2 is evaluated against it. A strong pivot baseline would use the best available model for each pivot direction — potentially mRASP2 itself for the English-centric legs, or fine-tuned bilingual models. The paper's pivot is source→English via m-Transformer, then English→target via m-Transformer. If the pivot used mRASP2 for both legs (since mRASP2 improves English-centric directions by +1.71 BLEU over m-Transformer), the pivot BLEU would be higher, and the gap to direct zero-shot would be larger. The paper's choice of m-Transformer for both legs of the pivot is consistent (it's the same base model) but makes the pivot weaker than it could be.

Furthermore, the zero-shot evaluation is on OPUS-100, which is a specific test set with 6 languages (Ar, Zh, Nl, Fr, De, Ru). The generalization to other language families, other domains, or larger numbers of languages is untested. The paper's conclusion mentions scaling to PC150 (150 languages) in future work — the current results on 32 training languages and 6 zero-shot evaluation languages represent a limited diversity sample.

Missing analyses that would strengthen the paper

No sensitivity analysis for key hyperparameters. The contrastive loss temperature (τ = 0.1), the loss coefficient (λ = 1.0), the AA replacement probability (90%), and the monolingual data volume are all fixed without reported sweeps. The paper does not show whether results are sensitive to these choices. For τ in particular, contrastive learning is known to be temperature-sensitive (lower τ makes the task harder and can improve representation quality up to a point), and the choice of 0.1 is aggressive (standard values in vision contrastive learning are 0.07–0.5). A sweep over τ would reveal whether the method is brittle to this choice.

No analysis of computational cost. The paper does not report training time, GPU-hours, or inference latency. Adding monolingual AA increases the training data from 97.6M parallel pairs to 97.6M parallel + 97.6M augmented parallel + 1.01B augmented monolingual pairs — roughly a 12× increase in training examples. The paper does not discuss whether the gains from monolingual AA could be achieved more efficiently through other means (e.g., using a smaller monolingual dataset, or applying AA online rather than offline).

No comparison to simply training with more parallel data. The paper argues that representation alignment, not data quantity, is the bottleneck. A strong test of this claim would be to compare mRASP2 on PC32 against a baseline trained on PC32 plus additional English-centric parallel data (e.g., by data mining or back-translation) of similar volume to MC24 (1 billion sentences). If the additional parallel data baseline matched mRASP2's zero-shot performance, the representation alignment diagnosis would be weakened. This experiment is not performed.

No multi-seed evaluation. All results are reported from single training runs. Given the known variability in multilingual NMT training (the paper itself notes that training "will collapse sometimes" without gradient clipping), reporting mean and standard deviation across 3–5 random seeds would provide confidence that the observed BLEU differences (some as small as +0.14 BLEU) are not noise.

No analysis of which zero-shot directions benefit most. The paper reports per-language averages (Table 3) and full pair matrices (Table 8) but does not analyze what properties of a language pair (linguistic distance, script similarity, training data volume, representation similarity in the baseline model) predict zero-shot improvement from mRASP2. Such an analysis would strengthen the explanatory power of the representation alignment hypothesis and provide practical guidance for which language pairs can expect the largest gains.

Limited evaluation of the reconstruction task in isolation. The AA-on-monolingual-data mechanism creates a reconstruction task — the model learns to map a noised code-switched sentence back to the clean original. The paper does not report reconstruction quality (e.g., BLEU of the reconstruction itself) or ablate the self-parallel contrastive loss (using only the cross-entropy reconstruction loss on monolingual AA data, without the contrastive objective on those pairs). This makes it difficult to disentangle how much of the unsupervised translation gain comes from the decoder practice (reconstruction) versus from the additional positive pairs for contrastive learning that the AA-monolingual pairs provide.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Remains Unaccounted For in the Compute-Optimal Framework

The assumption or constraint. The authors explicitly acknowledge that their method of estimating question difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — incurs substantial computational cost that is not included in any reported efficiency metric. The paper states: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2). This is framed as an exploration-exploitation tradeoff — compute spent assessing difficulty cannot be spent solving the problem — and left as a key avenue for future work.

The consequence. The headline 4× efficiency gain over best-of-N (e.g., 16 generations matching 64 in Figure 4) is computed after difficulty has been estimated, without amortizing the cost of that estimation. In a realistic deployment, the total budget would be difficulty estimation + strategy execution, and the former (2048 samples per question) could far exceed the latter (16–256 generations for the actual solve). This means the 4× figure represents an upper bound on achievable efficiency gains under idealized conditions where difficulty is known a priori — not a realized deployment advantage. A system that must pay the difficulty estimation cost on every query would be dramatically less efficient than the curves in Figures 4 and 8 suggest, and for low-budget regimes (where the solve budget is small relative to the estimation cost), the compute-optimal approach might actually underperform a simple best-of-N strategy that spends the combined budget on a single allocation.

What evidence exists in the paper. The paper provides no experiment that accounts for difficulty estimation cost in any budget comparison. The predicted difficulty bins are shown to be nearly as effective as oracle bins (the curves largely overlap in Figures 4 and 8), confirming that the PRM-based difficulty proxy works without ground-truth labels. However, this still requires 2048 generations and PRM scoring per question, and this cost is never included in the x-axis of any scaling plot. The paper also does not explore cheaper alternatives: for example, whether difficulty can be estimated from far fewer samples (say, 4–8), or whether a lightweight classifier trained on question text could predict difficulty with sufficient accuracy to maintain the gains. These are acknowledged in Section 8 as future work ("pretraining or finetuning models to directly predict difficulty of a question") but are not evaluated.

Mitigation status. Not addressed. The paper explicitly flags this as a limitation and a direction for future work, which is transparent and appropriate. However, until a cheap difficulty estimator is demonstrated, the compute-optimal framework should be understood as an analytical contribution (showing that difficulty-conditioned allocation is valuable) rather than a deployable system (showing how to achieve those gains in practice without paying the estimation tax). A practitioner following this recipe today would either need to accept the estimation overhead or develop their own lightweight difficulty estimator, which the paper does not provide.

All Results Are on a Single Benchmark and Single Model Family

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not verified. No experiments are conducted on other reasoning benchmarks (e.g., GSM8K, ARC, MMLU), other model families (e.g., LLaMA, GPT), or non-math domains. The choice of MATH is deliberate — the authors argue test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences — but this also means the findings are bounded to a task type where (a) ground-truth answers exist for PRM training, (b) step-by-step reasoning is required, and (c) the base model has non-trivial but far-from-saturating performance.

The consequence. Several key findings may not generalize to other settings. The difficulty-dependent behavior of search algorithms (beam search over-optimizing on easy problems but helping on medium ones, Figure 3) and revisions (sequential dominating on easy problems, balanced ratio optimal on hard ones, Figure 7) could be specific to the distribution of problem types and difficulty levels in MATH, or to the specific error patterns of PaLM 2-S* on math. A different model with different calibration (e.g., a model that rarely produces correct solutions even on "easy" problems) might show different optimal allocation strategies. A different benchmark that requires factual recall rather than multi-step reasoning might show different scaling behavior — test-time compute for factuality might involve search over external knowledge rather than self-verification. The paper's finding that test-time compute cannot help on the hardest problems (bin 5, near-zero accuracy regardless of budget) might be specific to MATH's hard problems; a different benchmark's "hardest" problems might sit at a different point relative to the base model's capability frontier.

What evidence exists in the paper. None beyond MATH and PaLM 2-S*. The paper does not include any cross-benchmark or cross-model validation, even as a small-scale pilot. The test set of 500 questions, split into five difficulty quintiles of approximately 100 each, then further split by two-fold cross-validation (~50 per fold per bin), means the compute-optimal policy is selected based on a very small number of questions. The paper does not report confidence intervals on the scaling curves, making it impossible to assess whether the observed strategy differences between difficulty bins are statistically reliable or whether they would replicate on a different sample of questions from the same distribution.

Mitigation status. Not addressed. The paper does not discuss the generalizability of findings to other benchmarks, models, or domains. The conclusion mentions scaling to "more languages" in a completely different context (the mRASP2 paper's conclusion about PC150), but for this paper, there is no discussion of whether the difficulty-conditioned framework would transfer to code generation, logical reasoning, scientific QA, or other tasks where test-time compute might be deployed. A practitioner considering applying these methods to a different model or domain would need to conduct their own full-scale replication, as the paper provides no guidance on transferability.

The Revision Model Has a Fundamental Correct-to-Incorrect Reversion Problem with Only Partial Mitigation

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This means the model never sees training examples where the current answer is already correct and should be preserved. At inference time, approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step (Section 6.1). The paper mitigates this by selecting the best answer across the entire revision chain (via majority voting or verifier-based selection), but this is a post-hoc patch rather than a solution to the training data bias.

The consequence. The revision mechanism is inherently unstable: generating more revisions does not monotonically improve quality because correct answers can be lost. This puts a ceiling on how much sequential revision can help — the pass@1 trajectory in Figure 6 (left) plateaus around 23–25% after 15–20 steps and does not continue to improve, even though the model is generating more revisions. The chain-level selection mechanism recovers some of the lost correct answers, but it requires storing and evaluating all revisions (which increases memory and compute cost) and is imperfect (some correct answers may be revised to incorrect versions that the verifier then prefers). In a deployment where only the final revision output is used (a natural default for a "self-improving" system), the 38% reversion rate would severely degrade performance. The paper's solution is to not trust the revision process to converge and instead treat it as a generator of candidates for a separate selection step, which means the revision model never achieves standalone self-correction capability.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1, and the performance plateau is visible in Figure 6 (left), where per-step pass@1 flattens after step 15–20. The chain-level selection approach improves final accuracy, as shown in Figure 6 (right), with sequential + best-of-N weighted achieving approximately 41.5% at 64 generations versus approximately 39% for parallel. However, the gap between sequential and parallel is modest (~2.5 percentage points with verifier selection), suggesting that the revision mechanism's benefit is real but limited by the reversion problem. The ReST $^{EM}$ experiment (Appendix K, Figure 16) further demonstrates the fragility: an RL-style optimization of the revision model causes performance to degrade substantially with sequential revisions, with sequential performance dropping to approximately 33.5% at 256 generations compared to approximately 38.5% at the optimal ratio.

Mitigation status. Partially addressed. The paper implements chain-level selection (majority voting or verifier) which recovers correct answers that were later revised to incorrect ones, and this is effective enough to make sequential revisions outperform parallel sampling. However, this is a workaround, not a fix. The fundamental training data bias (never seeing correct-to-correct or correct-to-better trajectories) is not addressed, and the paper does not explore training the revision model to recognize when no revision is needed. The ReST $^{EM}$ negative result (Appendix K) demonstrates that naïve attempts to optimize the revision model further can backfire, suggesting the training recipe is fragile in ways that are not fully understood. A practitioner implementing revision-based test-time compute should expect to deal with the reversion problem and should not expect unbounded improvement from longer revision chains.

Hard Problems Remain Completely Unsolved — Test-Time Compute Cannot Create Capability the Base Model Lacks

The assumption or constraint. The entire test-time compute framework assumes that the base model's pass@1 is non-trivially above zero — that correct solutions exist somewhere in the model's output distribution. This is explicitly acknowledged: the compute-optimal policy selects strategies based on difficulty bins, and difficulty is defined as the base model's pass@1 rate. When pass@1 is near zero (difficulty bin 5), no strategy helps.

The consequence. For the hardest problems (bin 5), accuracy remains at 1–3% regardless of strategy, search algorithm, revision depth, or compute budget (Figures 3 right, 7 right, and 9). This means test-time compute is fundamentally bounded: it can amplify existing capability (finding and refining correct solutions that the model can already produce at some low rate) but cannot create new capability. If a problem requires knowledge, reasoning patterns, or skills that the base model simply does not possess — and therefore never produces a correct solution even in 2048 samples — test-time compute provides zero benefit regardless of budget. This is a hard ceiling that pretraining can surpass (by training a larger model with more data and more capacity). The FLOPs-matched comparison in Section 7 quantifies this: on easy-to-medium problems, test-time compute with the smaller model can match or exceed a 14× larger model, but on hard problems (bins 4–5), the larger pretrained model dominates, with test-time compute showing a −52.9% relative disadvantage at high inference-to-pretraining ratios (Figure 1, bottom-right bar chart).

What evidence exists in the paper. Bin 5 results are consistently near-zero across all experiments: Figure 3 (right) shows approximately 1–3% accuracy for all methods at all budgets; Figure 7 (right) shows approximately 2–3% accuracy regardless of sequential-to-parallel ratio; Figure 9 shows the bin 5 scaling line essentially flat near 0–5%. The FLOPs-matched comparison (Figure 1 bar charts) shows negative relative performance for test-time compute versus pretraining on hard problems at all three values of R, with the disadvantage growing as R increases. The paper states explicitly: on the hardest problems "no amount of test-time compute helps" (Section 5.3 takeaway).

Mitigation status. Acknowledged but not addressed. The paper is transparent about this limitation — the discussion of Figure 3 notes that "on the hardest questions (bin 5), no method makes meaningful progress" — and the FLOPs-matched comparison clearly delineates where pretraining is preferable. However, the paper does not explore whether improved verifiers, better revision training, or other test-time strategies could push the capability frontier for hard problems. The limitation is fundamental rather than methodological: if the base model cannot produce a correct solution even once in thousands of attempts, no amount of search or refinement can find one. The only path forward for such problems is improved pretraining, and the paper's framework provides a clear diagnostic for when this is the case (when pass@1 is near zero). This boundary condition is a valuable contribution in itself — it tells practitioners when to stop investing in test-time compute and invest in better base models instead — but it also means the method offers no help for the hardest problems that are often the most valuable to solve.

The Revision Model and PRM Are Trained on Distribution-Shifted Data Relative to Their Test-Time Usage

The assumption or constraint. The PRM is trained on Monte Carlo rollouts from the base model (few-shot prompted PaLM 2-S*), evaluating step-level correctness based on completions of partial solutions drawn from the base model's output distribution. The revision model is also fine-tuned from the base model, but it generates outputs from a different distribution (conditioned on previous incorrect answers). At test time, the PRM is used to score outputs from both the base model (in search experiments) and the revision model (in revision experiments). The paper acknowledges this distribution shift: "The PRM trained on base model outputs does not transfer well to the revision model's outputs due to distribution shift" (Appendix J, discussing Figure 15a).

The consequence. The PRM's scores are less reliable when evaluating revision model outputs than when evaluating base model outputs. This means that verifier-based selection in the revision setting (e.g., best-of-N weighted selection across revision chains) is operating with a degraded verifier signal. The paper addresses this by training a separate ORM on revision model outputs (Appendix J, Figure 15a), which partially closes the gap — sequential + base-LM PRM achieves roughly 40% at 64 generations versus sequential + revision ORM at roughly 42%. However, this adds a second verifier training pipeline and means the PRM (which provides richer step-level guidance) cannot be effectively used with the revision model. The PRM-based search methods (beam search, lookahead search) are therefore limited to the base model's proposal distribution, preventing the combination of PRM-guided search with revision-improved proposals that the paper identifies as a promising future direction (Section 8: "we did not experiment with PRM tree-search techniques in combination with revisions").

More subtly, the revision model itself is trained on offline-constructed trajectories where incorrect and correct answers are paired post-hoc using edit distance (Section 6.1), rather than on actual sequential revision rollouts. This is a deliberate approximation — the authors note that on-policy multi-turn rollouts (as in Qu et al., 2024) were computationally infeasible — but it means the training data does not perfectly match the test-time scenario where the model sees its own previous outputs in context. The model is trained to correct other incorrect answers (from the base model's i.i.d. samples) rather than its own incorrect answers (from sequential revision). At test time, the model sees its own outputs, which are drawn from a different distribution than the training corrections. This distribution shift may contribute to the correct-to-incorrect reversion problem and the performance plateau in Figure 6 (left).

What evidence exists in the paper. Figure 15a (Appendix J) quantifies the PRM's degraded performance on revision model outputs, showing the base-LM PRM underperforms the revision-specific ORM by approximately 2 percentage points at 64 generations. Figure 15b shows that including revision history in the verifier's context provides a small improvement (~1–2 points), but does not fully close the gap. The paper's decision to train a separate ORM for revision model outputs is evidence that the distribution shift is practically significant. The offline trajectory construction approach is described in Section 6.1, where the paper explicitly compares it to Qu et al. (2024)'s on-policy approach and notes the computational infeasibility of on-policy rollouts. The ReST $^{EM}$ experiment (Appendix K, Figure 16) provides additional evidence: attempting to optimize the revision model with on-policy RL training caused performance to degrade substantially, suggesting that the revision training recipe is sensitive to distributional properties of the training data that are not fully understood.

Mitigation status. Partially addressed. The paper trains a separate revision-specific ORM, which partially compensates for the PRM's distribution shift, but this is a workaround, not a solution — it adds training complexity and does not enable PRM-based search with revisions. The offline trajectory construction is presented as a practical necessity rather than a principled choice, and the paper does not explore whether alternative data construction methods (e.g., iterative refinement of trajectories, on-policy data generation with a smaller number of rollouts) could reduce the distribution shift. The limitation is acknowledged but not deeply analyzed — the paper does not measure how much performance is lost due to the offline construction versus on-policy rollouts, or whether the edit-distance pairing heuristic introduces biases (e.g., the revision model may learn to make small surface-level edits rather than substantive corrections because the nearest-incorrect examples in edit-distance space are often structurally similar to the correct answer). A practitioner implementing revision-based test-time compute should expect to deal with verifier distribution shift (likely requiring a separate verifier for revision outputs) and should be aware that the offline trajectory construction may produce revision behavior that differs from on-policy sequential revision.

7. Implications and Future Directions

How This Work Changes the Landscape

A reframing of the multilingual NMT bottleneck from data scarcity to representation alignment. Prior to mRASP2, the dominant approach to improving non-English translation in multilingual NMT was to acquire more non-English parallel data — through mining (Fan et al., 2020), back-translation (Zhang et al., 2020), or leveraging monolingual data for pre-training then fine-tuning per direction (mBART, XLM, MASS). The underlying assumption was that poor non-English performance reflected a data problem: the model hadn't seen the direction, so it needed to see the direction. mRASP2 challenges this diagnosis at its root by demonstrating that the primary bottleneck is not the absence of non-English supervision but rather the misalignment of cross-lingual representations in the encoder. The evidence is the 8.5 BLEU gain on zero-shot translation from contrastive loss alone (Table 4, row 3 vs. row 1) — achieved with zero additional data, only a different training objective on the same English-centric parallel corpora. If data scarcity were the binding constraint, changing only the loss function could not produce an 8.5 BLEU gain. The fact that it does — and that the gain is concentrated on non-English directions while leaving English-centric directions essentially unchanged (+0.14 BLEU) — is strong causal evidence that representation misalignment, not data quantity, was the bottleneck.

This reframing shifts the research agenda from "how do we get more non-English parallel data?" to "how do we make the encoder's representation of meaning language-agnostic?" — a representation learning problem rather than a data acquisition problem. The implication is that investments in data mining infrastructure, back-translation pipelines, and per-direction fine-tuning may be partially misallocated: representation alignment can unlock non-English translation capability that already exists latently in the model's parameters from English-centric training, without requiring explicit supervision for those directions. This does not mean additional data is worthless — the paper shows monolingual AA adds further gains (+0.71 zero-shot BLEU over contrastive + parallel AA, Table 4) — but it means data augmentation should be evaluated by whether it improves representation alignment, not just by whether it increases training examples.

Reconciling contradictory prior findings about zero-shot translation. The literature contained a tension: Johnson et al. (2017) demonstrated that multilingual NMT could perform zero-shot translation in principle, but subsequent work found the quality was "quite poor compared with pivot-based model[s]" (the paper's characterization), leading some researchers to conclude that zero-shot translation was not practically viable without explicit non-English supervision. Fan et al. (2020) and Zhang et al. (2020) improved zero-shot through data mining and back-translation respectively, but Zhang et al. explicitly observed a tradeoff — zero-shot gains came at the cost of ~0.5 BLEU on English-centric directions. This created a narrative that English-centric and non-English quality were in zero-sum tension.

mRASP2 resolves this contradiction by showing that the poor zero-shot quality in prior work was not an inherent limitation of the multilingual NMT paradigm but a consequence of insufficient representation alignment. When representations are properly aligned through contrastive learning, zero-shot translation jumps from 5.05 to 15.31 BLEU (Table 4) — a 3× improvement — while English-centric directions simultaneously improve by +1.71 BLEU. The zero-sum tradeoff was not a capacity constraint but an alignment failure: when representations are misaligned, the model must allocate capacity to language-specific encoding pathways that don't transfer, making it appear as though non-English and English-centric directions compete for fixed resources. When representations are aligned, the same capacity serves all directions more efficiently, and both improve together.

This resolution has practical implications for how the field evaluates multilingual NMT systems. A system that shows poor zero-shot performance should not automatically be diagnosed as needing more data; it may simply need better representation alignment. Conversely, a system that achieves good English-centric results at the cost of zero-shot quality should be suspected of having misaligned representations, not of having reached a fundamental capacity limit.

Establishing contrastive learning as a general-purpose alignment objective for sequence-to-sequence models. Prior to mRASP2, contrastive learning for cross-lingual representation alignment had been explored primarily in encoder-only models (mBERT, XLM) for understanding tasks like XNLI or NER. Its application to sequence-to-sequence generation — where the aligned representations must not only support classification but also drive autoregressive decoding through cross-attention — was novel. The paper demonstrates that a simple contrastive loss on average-pooled encoder outputs, trained jointly with the standard cross-entropy translation objective, can produce representations that are simultaneously useful for token-level generation and sentence-level semantic alignment. This is a non-trivial finding because contrastive learning typically benefits from a projection head that decouples the representation used for the contrastive objective from the representation used for downstream tasks (Chen et al., 2020). mRASP2 applies contrastive loss directly to the encoder output — the same representations the decoder cross-attends to — and shows that this joint optimization works without a projection head. This suggests that encoder representations in multilingual Transformers have sufficient capacity to serve both purposes, and that explicit alignment objectives can be integrated into generation models without architectural modifications.

The broader implication is that contrastive alignment is an underutilized tool in multilingual generation. The paper's success with a simple InfoNCE objective using in-batch negatives and cosine similarity on average-pooled representations suggests that more sophisticated contrastive formulations — harder negative mining, multi-scale contrastive losses operating at both token and sentence levels, momentum encoders for larger negative sets — could yield further gains. The paper opens a design space for contrastive objectives in sequence-to-sequence models that had not been systematically explored before.

Demonstrating that monolingual data can be leveraged for generation capability through reconstruction, not just for encoder pre-training. Prior work on using monolingual data for NMT fell into two categories: (1) pre-training the encoder (and sometimes decoder) with denoising or language modeling objectives, then fine-tuning on parallel data (mBART, MASS, XLM), which uses monolingual data to improve initialization but not to directly teach generation for specific languages; and (2) back-translation (Sennrich et al., 2016; Zhang et al., 2020), which uses monolingual target-language data to create synthetic source-target pairs that teach the model to generate in that language. Back-translation is effective but requires an existing translation system in the reverse direction and is typically applied per direction.

mRASP2's Aligned Augmentation on monolingual data introduces a third approach: using dictionary-based word replacement to create a reconstruction task — noisy, code-switched source → clean target — that teaches the decoder to generate in a language from non-standard encoder states, without requiring a reverse translation system. The mechanism is simpler than back-translation (it requires only a bilingual dictionary, not a trained MT model) and integrates naturally into the unified training framework alongside parallel data. The results on unsupervised translation (Table 2) demonstrate that this reconstruction approach, combined with contrastive alignment, can achieve reasonable BLEU scores (18.55 average on 6 directions, including 9.3/8.3 on Nl↔Pt which has zero parallel data of any kind). This is a new point in the design space for leveraging monolingual data in NMT, and it is particularly attractive for low-resource languages where a reverse translation system may not be available for back-translation.

A partial shift in the scaling narrative: representation quality over model or data quantity. The paper makes an implicit argument that, for multilingual NMT, how you train matters at least as much as how much data you have or how large your model is. The contrastive loss provides an 8.5 BLEU improvement on zero-shot translation using the exact same data and model as the baseline — no scaling required. The 12-layer depth provides "significant improvements" over 6 layers, but the paper argues this is specifically because deeper models disentangle language-specific and language-agnostic features better — a representation quality argument rather than a raw capacity argument. The batch size of 3 million tokens is described as "crucial" because it provides more in-batch negatives for contrastive learning and more stable gradient estimates across diverse language pairs — again, a training quality argument. This emphasis on training methodology over brute-force scaling is a counterpoint to the dominant narrative in the field at the time (Arivazhagan et al., 2019; Aharoni et al., 2019; Fan et al., 2020) that massive scale — more languages, more parameters, more data — was the primary path forward for multilingual NMT.

The paper does not claim scaling is irrelevant — the conclusion explicitly calls for scaling to PC150 and 100+ languages — but it argues that representation alignment is a prerequisite for scaling to be effective. A 150-language model with misaligned representations would waste most of its capacity on redundant language-specific encoding; a 32-language model with well-aligned representations can already achieve competitive zero-shot translation. The implication is that future scaling efforts should prioritize representation alignment as a foundational objective, not an afterthought.

Follow-Up Research This Work Enables

Scaling mRASP2 to 100+ languages and evaluating whether the representation alignment gains persist or saturate. The paper's conclusion explicitly calls for scaling "to more languages, e.g. PC150" with the goal that "a single model can handle more than 100 languages and outperforms the corresponding bilingual baseline." This is a natural extension because mRASP2's contrastive loss and AA require no per-language modifications — they scale automatically with the number of languages. However, it is unknown whether the representation alignment gains observed on 32 languages (where all languages share at least English as a pivot) will persist when scaling to 100+ languages spanning diverse families (tonal languages, morphologically rich languages, languages with non-Latin scripts). The key question is whether a single shared representation space can accommodate 100+ languages without interference, or whether the contrastive loss will face a "crowding" problem where too many languages compete for the same region of latent space. A strong follow-up would train mRASP2 on PC150 (or a similar large-scale multilingual dataset), measure zero-shot BLEU across all 150×149 non-English directions, and analyze whether per-direction BLEU degrades as the number of training languages increases (controlling for training data volume). The paper's similarity search evaluation (Tables 5 and 6, Figure 3) provides a template for diagnosing representation quality at scale — top-1 retrieval accuracy per language pair can reveal whether alignment degrades for specific language families or scripts.

Combining mRASP2 with non-English parallel data to test whether representation alignment and direct supervision are complementary or redundant. The paper achieves strong non-English results using only English-centric parallel data, but does not test what happens when non-English parallel data (e.g., from Fan et al., 2020's data mining or Zhang et al., 2020's back-translation) is added on top of mRASP2. The representation alignment hypothesis predicts that aligned representations should make non-English parallel data more sample-efficient — if the encoder already maps German and French sentences to similar regions, a small amount of Fr→De parallel data should produce larger BLEU gains than it would without alignment. A strong follow-up would take a subset of non-English parallel data (say, 10K–100K pairs for a few language pairs), add it to mRASP2 training (with and without contrastive loss), and measure the BLEU gain per parallel example. If the gain per example is larger with contrastive loss, it confirms that alignment amplifies the value of direct supervision. If the gain is the same or smaller, it suggests alignment and supervision are partially redundant (both provide the same signal — mapping equivalent sentences to similar representations) and the practical value of alignment is primarily for directions with zero supervision.

Training a difficulty estimator for the compute-optimal allocation policy that does not require 2048 samples per prompt. This direction is explicitly identified in Section 3.2 of the paper and remains the single largest barrier to practical deployment. The current difficulty estimation method — generating 2048 samples per question and averaging PRM final-answer scores — is far too expensive to amortize over the typical test-time compute budgets studied (16–256 generations). A strong follow-up would train a lightweight difficulty classifier that takes only the question text as input (no samples, no PRM scoring) and predicts the difficulty quintile. The training data already exists: the paper's 500-question MATH test set with 2048 samples each provides labeled difficulty bins (oracle and predicted) for every question. The classifier could be a small fine-tuned LM (e.g., a few million parameters) or even a linear probe on top of the base model's final hidden state. The key metrics would be: (1) classification accuracy into the 5 difficulty bins, (2) the BLEU/accuracy achieved by the compute-optimal policy when using the classifier's predictions versus oracle bins (should closely match the predicted-bin curves in Figures 4 and 8), and (3) the inference cost of the classifier relative to the test-time compute budget (it should be negligible — one forward pass versus 2048). If a cheap classifier can match the PRM-based difficulty bins, the compute-optimal framework becomes immediately deployable. If not, the gap quantifies how much efficiency is lost to difficulty estimation in practice.

Combining PRM-guided tree search with the revision model as the proposal distribution. The paper studies PRM-based search (beam search, lookahead search) and iterative revisions as independent mechanisms, and explicitly acknowledges they were not combined (Section 8: "we did not experiment with PRM tree-search techniques in combination with revisions"). This is perhaps the most promising follow-up because the two mechanisms have theoretically complementary strengths: revisions improve the quality of generated candidates (by conditioning on previous attempts), while PRM search improves candidate selection (by scoring intermediate steps and pruning unpromising paths). A combined system would use the revision model to generate candidate steps within a beam search tree — at each expansion step, the revision model conditions on the partial solution and any previous rejected branches, producing higher-quality next steps than the base model would. The PRM scores each step's promise, and beam search prunes to the most promising prefixes. This could push through the performance ceiling that each method individually hits: the revision model's correct-to-incorrect reversion problem (~38% of correct answers lost) could be mitigated by the PRM recognizing when a revision is going off-track and pruning that branch; the PRM's over-optimization problem (beam search degrading easy problems at high budgets, Figure 3 right) could be mitigated by the revision model producing more diverse and higher-quality candidates that are less susceptible to verifier exploitation. A strong follow-up would implement this combination, sweep beam widths and revision depths jointly, and measure whether the combined system exceeds the better of search-only or revision-only performance on medium-difficulty problems (bins 3–4), where both methods show partial effectiveness. The key metric is whether the combination achieves accuracy on bin 3–4 problems that neither method achieves alone at the same generation budget.

Stress-testing the over-optimization hypothesis by training a more robust PRM and measuring whether beam search scaling extends to higher budgets. The paper identifies verifier over-optimization as the primary bottleneck for search-based test-time compute scaling (Section 5.3): beam search degrades easy-problem performance at high budgets (Figure 3, right), and lookahead search — the most aggressive optimizer — paradoxically performs worst overall (Figure 3, left). The paper's PRM is trained with Monte Carlo rollouts on i.i.d. base model samples. A natural hypothesis is that this PRM is poorly calibrated on search-generated solutions, because search can find adversarial examples — solutions that score highly under the PRM but are incorrect — that never appeared in the i.i.d. training distribution. A strong follow-up would train a search-aware PRM by including beam-search-generated solutions (with their true correctness labels) in the PRM training data, either through offline data augmentation (generate beam-search solutions, label them, add to training set) or online adversarial training (periodically run beam search during PRM training and add the generated solutions to the training batch). The prediction is that a search-aware PRM would be more robust to over-optimization, allowing beam search to continue improving at higher budgets rather than plateauing or degrading. The key experiment: plot beam search accuracy vs. budget with the standard PRM and the search-aware PRM on easy and medium problems. If the search-aware PRM's curve continues to rise at budgets where the standard PRM's curve flattens or falls, it confirms that over-optimization is a verifier robustness problem (solvable through better training) rather than a fundamental limitation of search.

Measuring whether the test-time compute benefits transfer to code generation and other reasoning domains with natural verifier signals. All experiments in the paper are on the MATH benchmark. The compute-optimal framework — difficulty estimation, adaptive strategy selection, verifier-guided search, iterative revision — is domain-agnostic in principle but was validated only on competition-level math problems. Code generation is a particularly attractive transfer domain because it shares key properties with MATH (multi-step reasoning, clear correctness signals via unit tests, the existence of intermediate states — partial programs — that can be scored) but differs in important ways (longer outputs, structured syntax, different error modes — syntax errors vs. logical errors). A strong follow-up would replicate the core experiments (Figures 3, 4, 7, 8) on HumanEval or MBPP using a code-capable base model, with unit test pass rate as the correctness metric. The key questions: (1) Does the difficulty-dependent optimal strategy pattern replicate? (Are easy coding problems best served by sequential revision, while hard ones benefit from parallel search?) (2) Can a PRM be trained for code using execution-based rollout supervision (sample completions from a partial program, check if they pass tests)? (3) Does verifier over-optimization manifest similarly (beam search finding programs that score well under the PRM but fail unit tests)? Positive transfer would significantly expand the practical applicability of the framework; negative transfer would reveal domain-specific assumptions in the MATH results that require modification.

Practical Applications and Downstream Use Cases

Deploying a single many-to-many translation model for content platforms serving diverse language pairs. The most direct application of mRASP2 is replacing multiple bilingual or English-centric multilingual translation systems with a single unified model that handles all language pairs — including non-English directions — at competitive quality. The numbers from Table 1 and Table 3 make the case: mRASP2 achieves 33.01 average tokenized BLEU on supervised English-centric directions (competitive with per-direction fine-tuned mBART on most pairs) and 15.31 de-tokenized BLEU on zero-shot non-English directions (essentially matching pivot-based translation at 15.56 BLEU). For a content platform serving 32 languages, this means one model replaces 992 potential bilingual models or a pivot-based system that routes everything through English (doubling latency and compute). The deployment benefit is model efficiency (one set of weights to serve, update, and maintain) and latency reduction for non-English pairs (single forward pass versus two-pass pivot). The 12-layer, 1024-dimension architecture is large but within the range of production deployment on GPU inference servers. The main practical consideration is that languages not in the training data or monolingual data (like Dutch in PC32) achieve only modest quality (~5–6 BLEU for X↔Nl in Table 3) — these would need monolingual AA data at minimum, and preferably some parallel data, to reach production quality.

Enabling zero-shot translation for low-resource language pairs where pivot systems are the only alternative. For language pairs with no direct parallel data — a common situation for the long tail of the world's languages — the choice is typically between zero-shot multilingual NMT (poor quality) and pivot-based translation through English (better quality but 2× latency). mRASP2's zero-shot BLEU of 15.31 across 30 OPUS-100 directions essentially matches pivot quality (15.56) in a single forward pass (Table 3). For specific high-resource combinations, zero-shot quality can be impressive: Fr→Zh achieves 42.3 BLEU, Ru→Zh achieves 40.6 BLEU (Table 8). This means that for many language pairs, a deployment can skip the pivot step entirely — reducing latency by half and eliminating the need to train, deploy, and maintain separate models for each pivot leg. The practical requirement is that both languages have reasonable representation in the training data (either through parallel English-centric corpora or monolingual AA data) so the encoder and decoder have independently learned to process them. Languages entirely absent from all training data (neither parallel nor monolingual) would still not be served — mRASP2 does not enable translation for truly unseen languages, only for unseen pairs of seen languages.

Cost-efficient multilingual model training for organizations with limited parallel data budgets. mRASP2 achieves strong zero-shot and unsupervised translation using only English-centric parallel data (PC32, 97.6M pairs) plus monolingual data (MC24, 1.01B sentences) with dictionary-based augmentation. This is significant because English-centric parallel data is far more abundant and cheaper to obtain than non-English parallel data — for many language pairs, English-centric corpora exist but direct corpora do not. An organization wanting to deploy translation for 32 languages would traditionally need to acquire or mine parallel data for many of the 992 directed pairs, which is expensive and may be infeasible for low-resource combinations. mRASP2's recipe — English-centric parallel data + monolingual data (readily available from web crawls) + a bilingual dictionary (available from MUSE or similar resources) — is substantially cheaper to assemble. The training itself requires 32 V100 GPUs (a large but not prohibitive compute budget for an organization with ML infrastructure) and produces a single model for all directions. The 1.01B monolingual sentences with AA represents a significant data processing step (dictionary lookup and replacement for every sentence), but this is an offline preprocessing cost amortized over all subsequent translations. The main caveat is that the unsupervised and zero-shot quality, while much improved over the baseline, is still well below supervised quality for high-resource pairs (15.31 vs. 30+ BLEU) — organizations needing production-grade translation for specific non-English pairs would still benefit from acquiring some direct parallel data for those pairs, with mRASP2 providing a strong initialization.

Representation alignment as a pre-training step before domain-specific fine-tuning. While not explicitly evaluated in the paper, the finding that contrastive alignment unlocks latent translation capability suggests a workflow: train a base multilingual model with mRASP2 on large general-domain parallel and monolingual data to establish language-agnostic representations, then fine-tune on smaller domain-specific parallel corpora for particular language pairs and domains (e.g., medical, legal, technical). The aligned representations should make fine-tuning more sample-efficient because the model already maps equivalent sentences to nearby regions regardless of language — fine-tuning only needs to adapt the decoder's generation style and domain-specific terminology, not learn cross-lingual alignment from scratch. This is analogous to how mBART pre-training on monolingual data improves fine-tuning sample efficiency, but with the added benefit of explicit cross-lingual alignment. The paper's results on low-resource languages (Table 5: Nl retrieval accuracy improves from 25.6% to 60.3%, Pl from 6.3% to 28.1%) show that languages with limited parallel data benefit disproportionately from alignment, which is exactly the regime where fine-tuning sample efficiency matters most. A practitioner with, say, 10K in-domain parallel sentences for a low-resource language pair would likely see larger gains fine-tuning an mRASP2 model than an mBART model, because the mRASP2 model already maps the two languages to a shared space while mBART's representations may not be aligned (mBART uses denoising objectives that operate within single languages, not cross-lingual alignment objectives). This application is speculative based on the paper's results but follows naturally from the representation alignment hypothesis.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

ZeRO-Offload: Democratizing Billion-Scale Model Training2021· training-methods← in-domain predecessor
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021· training-methods← in-domain predecessor
Language Models are Unsupervised Multitask Learners2019· training-methods← similar topic
- Parameter-Efficient Transfer Learning for NLP2019· training-methods← in-domain predecessor
- Adapting Large Language Models for Document-Level Machine Translation2000· training-methods← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

LoRA: Low-Rank Adaptation of Large Language Models2021· training-methods→ in-domain predecessor
- QLoRA: Efficient Finetuning of Quantized LLMs2023· training-methods→ LLM-judged similar
- Fine-Tuning Language Models with Just Forward Passes2023· training-methods→ LLM-judged similar
- Read More2023· training-methods→ LLM-judged similar
- VeRA: Vector-based Random Matrix Adaptation2023· training-methods→ LLM-judged similar
- Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization2023· training-methods→ LLM-judged similar
- Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models2024· moe→ LLM-judged similar
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection2024· training-methods→ LLM-judged similar
- TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control2025· llm-systems→ LLM-judged similar
- SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity2025· training-methods→ LLM-judged similar
- Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics2026· alignment→ LLM-judged similar
- HY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing2026· training-methods→ LLM-judged similar
- Spanning the Visual Analogy Space with a Weight Basis of LoRAs2026· vision→ LLM-judged similar
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2021· training-methods→ in-domain predecessor
Adapting Large Language Models for Document-Level Machine Translation2000· training-methods→ similar topic
- Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent2000· training-methods→ in-domain predecessor
- Language Models are Unsupervised Multitask Learners2019· training-methods→ similar topic
Map-Reduce for Machine Learning on Multicore2000· training-methods→ similar topic
- Regularized Multi-Task Learning2000· training-methods→ in-domain predecessor
- Large-Scale Machine Learning with Stochastic Gradient Descent2000· training-methods→ similar topic
- Support Vector Machine Learning for Interdependent and Structured Output Spaces2000· training-methods→ similar topic
- Quantifying the Carbon Emissions of Machine Learning2019· training-methods→ similar topic
Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning2022· training-methods→ similar topic
- Read More2022· training-methods→ in-domain predecessor
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation2023· alignment→ topic thread
- Rainbow: Combining Improvements in Deep Reinforcement Learning2000· training-methods→ similar topic
- CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning2025· training-methods→ similar topic
- Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards2026· training-methods→ similar topic