Unsupervised Cross-lingual Representation Learning at Scale

ArXiv: 1911.02116

🎯 Pitch

XLM-R shows that training one massive multilingual model on 100 languages not only crushes prior cross-lingual transfer benchmarks, but for the first time matches per-language monolingual models like RoBERTa—eliminating the long-standing accuracy penalty in multilingual NLP. By identifying a 'curse of multilinguality' that degrades low-resource performance unless model capacity scales, the paper provides a practical roadmap for building single models that work across all languages without compromise.

---

1. Executive Summary

This paper studies how pretraining multilingual masked language models at scale affects cross-lingual transfer, training a Transformer-based model on over two terabytes of filtered CommonCrawl data spanning 100 languages. The core contribution is XLM-R, a large multilingual model that significantly outperforms multilingual BERT across classification, question answering, and named entity recognition benchmarks — including +14.6% average accuracy on XNLI and +13% average F1 on MLQA — while simultaneously demonstrating, for the first time, that a single multilingual model can match strong monolingual baselines like RoBERTa on per-language tasks. Through systematic ablation, the paper identifies and names the curse of multilinguality — the phenomenon where adding more languages to a fixed-capacity model eventually degrades all-language performance — establishing that cross-lingual positive transfer and capacity dilution must be traded off against each other, and that the curse can be alleviated only when model capacity is increased alongside the number of languages.

2. Context and Motivation

The Core Problem: Multilingual Models Underperform Monolingual Ones, and We Don't Know Why

The central problem this paper addresses is deceptively simple: why do multilingual language models consistently underperform their monolingual counterparts, and can we fix this by simply scaling up the data? Prior to this work, multilingual models like mBERT (Devlin et al., 2018) and XLM (Lample and Conneau, 2019) had shown remarkable cross-lingual transfer capabilities — a model fine-tuned on English training data could perform the same task in French, German, or Swahili without seeing a single labeled example in those languages. This was genuinely surprising and practically valuable. However, these models suffered from a well-documented but poorly understood limitation: for any given language, a dedicated monolingual model trained on that language's data typically outperformed the multilingual model trained on many languages simultaneously. The field lacked both a systematic understanding of why this happens and a clear recipe for closing the gap.

This gap matters for several practical reasons that the paper makes explicit:

• Low-resource language inclusion: Most of the world's 7,000+ languages lack sufficient labeled data for supervised NLP. Cross-lingual transfer from high-resource languages is one of the only viable paths to building NLP systems for these languages. If multilingual models underperform monolingual ones, the people who most need cross-lingual transfer — speakers of low-resource languages — receive worse models.

• Deployment simplicity: Maintaining separate models for each language is operationally expensive — it means N separate training pipelines, N sets of hyperparameters, N models in production. A single model that works well for all languages dramatically simplifies deployment, but only if per-language performance doesn't suffer.

• Computational efficiency in multi-language applications: Services operating across many languages (social media content moderation, international customer support, cross-border search) must currently either run many monolingual models or accept degraded performance from a multilingual one. A multilingual model competitive with monolingual performance eliminates this tradeoff.

The Undertuned Baseline Problem

The paper's investigation reveals that part of the performance gap isn't fundamental — it's an artifact of how prior multilingual models were trained. Both mBERT and XLM were pretrained exclusively on Wikipedia corpora. For high-resource languages like English or German, Wikipedia provides billions of tokens. But for low-resource languages like Swahili or Urdu, Wikipedia provides only a few tens of thousands of sentences — what the paper later shows to be "a few hundred MiB," which is "the minimal size for learning a BERT model" (Section 5.3). Below this threshold, the model cannot effectively learn the language's structure, and any cross-lingual transfer comes purely from shared vocabulary items and structural similarities to related languages seen during training.

This is visible directly in the paper's data: Figure 1 shows that for languages like Swahili (sw), Burmese (my), or Amharic (am), Wikipedia provides less than 1 GiB of text, while CommonCrawl provides 1–10 GiB — an increase of one to two orders of magnitude. The paper argues that mBERT and XLM were undertuned not in the architectural sense, but in the data sense: their pretraining corpora were too small to enable genuine representation learning in most languages beyond the top 20–30 by Wikipedia size.

This connects directly to a finding the paper emphasizes in Section 5.1: "using validation perplexity as a stopping criterion for pretraining caused the multilingual MLM in Lample and Conneau (2019) to be under-tuned." Downstream task performance continued improving even after validation perplexity plateaued, a phenomenon also observed by Liu et al. (2019) in the monolingual setting with RoBERTa. Prior multilingual models were simply not trained long enough or on enough data, and no one had systematically studied what happens when you fix this.

The Unexplored Scaling Dimension: Languages vs. Capacity

The paper identifies a deeper, structural problem that prior work had not addressed. When you pretrain a multilingual model, you face a fundamental tension: model capacity (number of parameters) is fixed for practical reasons — memory constraints, inference latency, training cost — but you are asking the model to represent an increasing number of languages. Each language competes for a share of the model's parameters. The authors frame this as a resource allocation problem:

"For a fixed sized model, the per-language capacity decreases as we increase the number of languages."

Prior work had observed that adding related languages could improve low-resource language performance through positive transfer — knowledge learned from high-resource languages helps on low-resource ones that share vocabulary, syntax, or typological features. But no one had systematically mapped the boundary where positive transfer stops helping and capacity dilution starts hurting. The paper frames this as a tradeoff to be studied empirically: at what point does adding more languages degrade performance across the board?

This is where the paper introduces its most conceptual contribution: the curse of multilinguality. The phenomenon itself — that performance degrades as languages increase — had been observed in multilingual machine translation (Arivazhagan et al., 2019), but it had never been systematically characterized for pretrained encoders used in cross-lingual transfer. The paper shows that this curse is not simply "more languages = worse performance." Instead, it follows a non-monotonic pattern: going from 7 to 15 languages initially helps low-resource languages, but beyond some threshold, further language additions hurt all languages. Figure 2 captures this precisely: low-resource language performance improves as you add related high-resource languages (positive transfer dominates), then plateaus and declines (capacity dilution takes over).

Crucially, the paper shows that the curse is capacity-dependent. When you increase the model's hidden size alongside the number of languages, performance recovers. Figure 4 demonstrates this: XLM-30 with increased capacity matches XLM-7, and XLM-100 with capacity scaling does better than XLM-100 at fixed capacity — but still doesn't fully close the gap. This means the curse isn't a fundamental ceiling but rather a manifestation of the fact that prior multilingual models were undersized relative to their language coverage. The practical implication is that scaling to more languages requires scaling model capacity, and the paper provides empirical evidence for where the inflection points lie.

The Pretraining Data Bottleneck: Wikipedia Is Too Small

The paper's third major motivation is the inadequacy of Wikipedia as a pretraining corpus for massively multilingual models. Wikipedia is the standard pretraining corpus in NLP — it's clean, multilingual, and freely available. But as Figure 1 starkly illustrates, its size distribution across languages is extraordinarily skewed. English Wikipedia provides orders of magnitude more text than Swahili Wikipedia. For the 88 languages shared between the Wiki-100 corpus (used by mBERT and XLM-100) and the CC-100 corpus (used by XLM-R), CommonCrawl provides anywhere from 10× to 1,000× more data for low-resource languages.

This matters because masked language model pretraining is fundamentally data-hungry. The model learns through exposure to diverse contexts — if you only have a few megabytes of text in a language, the model cannot learn meaningful representations for that language regardless of how much cross-lingual transfer it receives. The paper's ablation in Section 5.4 confirms this directly: models pretrained on Wikipedia for Swahili and Urdu "performed similarly to a randomly initialized model," while pretraining on CommonCrawl improved performance by up to 10 points. The prior state of the art (mBERT, XLM-100) was therefore operating in a regime where many languages were essentially untrained — their performance on those languages was purely a function of cross-lingual transfer from similar languages, not genuine representation learning.

The paper's solution — building CC-100, a cleaned CommonCrawl corpus in 100 languages — is a direct response to this bottleneck. By using CommonCrawl rather than Wikipedia, the paper increases data for low-resource languages by two orders of magnitude on average (Section 3). This isn't just "more data" — it's the difference between having enough data to learn representations and not having enough data at all.

Conflicting Signals in Prior Work

The paper enters a landscape where several high-profile results had been published but not reconciled:

mBERT works surprisingly well for cross-lingual transfer. Pires et al. (2019) showed that mBERT — trained without any cross-lingual signal — could transfer effectively across languages on NER. Wu and Dredze (2019) replicated this finding. Wu et al. (2019) showed that mBERT's internal representations for different languages align without explicit alignment training, suggesting that the multilingual pretraining objective naturally induces cross-lingual structure. But none of these studies explained why the transfer worked or what its limits were.

XLM improved on mBERT with a better training recipe. Lample and Conneau (2019) introduced XLM, which outperformed mBERT by using a better MLM objective and translation language modeling (TLM) with parallel data. But XLM still used Wikipedia-sized corpora and capped out at 100 languages with diminishing returns.

Massively multilingual NMT showed capacity dilution. In machine translation, Arivazhagan et al. (2019) trained a single model on 103 languages with 25 billion parallel sentences and found that performance degraded as languages increased — the "curse of multilinguality" in the MT context. But NMT uses supervised parallel data, while cross-lingual transfer uses only monolingual pretraining data. It was unclear whether the same dynamics would apply.

Monolingual scaling laws suggest more data and larger models help. In the monolingual setting, RoBERTa (Liu et al., 2019) had just shown that training BERT longer on more data with better hyperparameters substantially improved performance. This suggested that multilingual models might be similarly undertrained, but no one had tested this systematically across many languages.

The paper's position is that these findings are not contradictory — they're different manifestations of the same underlying dynamics (capacity dilution, data scarcity) that had never been studied in a unified framework. The paper doesn't propose a fundamentally new architecture or training objective. Instead, it argues that prior multilingual models were simultaneously undertrained (not enough data, not enough training steps) and undersized (not enough capacity for the number of languages), and that fixing both simultaneously — scaling data by two orders of magnitude and scaling model capacity — yields previously unattainable performance.

How This Paper Positions Itself

The paper explicitly positions itself at the intersection of two research threads: multilingual masked language modeling (mBERT, XLM) and large-scale pretraining (RoBERTa, GPT). The critical insight is that these threads had been pursued independently — multilingual models used small-scale Wikipedia data, and large-scale models were monolingual. The paper's contribution is merging them: applying the large-scale pretraining recipe (more data, longer training, bigger batches, better preprocessing) to the multilingual setting and studying the resulting dynamics.

The paper also positions itself as providing the first comprehensive empirical analysis of multilingual model scaling. Prior work had studied individual factors (vocabulary size in mBERT, language sampling in XLM) in isolation. This paper studies them jointly and shows how they interact: vocabulary size interacts with capacity (Figure 6), language sampling interacts with the high-resource/low-resource tradeoff (Figure 5), and data scale interacts with model size to determine whether the curse of multilinguality is observed (Figure 4).

Finally, the paper makes a deliberate choice to abandon supervision. Lample and Conneau (2019) had achieved their best results with TLM — a supervised objective that requires parallel sentences. The paper's ablations (Section 5.1) show that by improving the unsupervised MLM objective (more data, better hyperparameters, training past perplexity convergence), XLM's MLM-only performance matches its MLM+TLM performance. This is significant because parallel data is scarce and expensive compared to monolingual data. By showing that TLM is unnecessary when you scale monolingual data sufficiently, the paper makes the approach applicable to languages where parallel corpora don't exist — which is exactly the long tail of low-resource languages that most need cross-lingual transfer.

3. Technical Approach

3.1 Reader Orientation

XLM-R is a large Transformer-based masked language model pretrained on raw text in 100 languages, designed to produce token representations that can be fine-tuned for downstream NLP tasks in any of those languages. The system solves the problem that prior multilingual models underperformed monolingual ones by recognizing that the bottleneck was not fundamental — it was an artifact of training on too little data (Wikipedia) with too little model capacity, and the solution is to scale both simultaneously: train a bigger model on two orders of magnitude more data, primarily from CommonCrawl.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

1. CC-100 Corpus — a cleaned, language-identified dataset of monolingual text in 100 languages, extracted from CommonCrawl dumps and Wikipedia, totalling over 2 terabytes of data. This is the raw input to pretraining.
2. Sentence Piece Model (SPM) Tokenizer — a subword tokenizer trained jointly on raw text from all 100 languages, producing a shared vocabulary of 250K subword units. This converts raw text into token sequences without language-specific preprocessing.
3. Transformer Encoder (XLM-R) — a standard bidirectional Transformer (24 layers, 1024 hidden size, 16 attention heads, 550M parameters for the large model) trained with the masked language modeling objective on batches of concatenated text from different languages.
4. Language Sampling Schedule — a probability distribution over the 100 languages that determines which language's data appears in each training batch, controlled by a parameter $\alpha = 0.3$ that smooths the distribution to oversample low-resource languages relative to their natural frequency.
5. Downstream Fine-Tuning — after pretraining, the model is fine-tuned on task-specific labeled data (e.g., English NLI, English NER, English SQuAD) and then evaluated on non-English test sets in a zero-shot cross-lingual transfer setup, where the model has never seen labeled data in the target language.

Information flows as follows: raw web pages from CommonCrawl → language identification and quality filtering → CC-100 corpus → SPM tokenization → token sequences → batches sampled according to $\alpha = 0.3$ → masked language modeling training on the Transformer → pretrained XLM-R model → fine-tuning on English labeled data for a specific task → inference on target language test sets → cross-lingual accuracy/F1 scores.

3.3 Roadmap for the Deep Dive

• First, the Masked Language Model (MLM) objective — how the model is trained, what it predicts, and why this particular self-supervised objective induces cross-lingual representations despite no parallel data being used.
• Second, the CC-100 data pipeline — how CommonCrawl is turned from raw web dumps into a clean 100-language corpus, why Wikipedia is insufficient, and what scale increase this provides per language.
• Third, the Sentence Piece tokenizer and vocabulary design — how a single shared vocabulary across all languages is constructed, why vocabulary size matters for multilingual performance, and the tradeoffs between embedding parameters and Transformer parameters.
• Fourth, the language sampling and capacity allocation — how the $\alpha$ parameter controls the distribution of languages in training batches, how this creates the high-resource versus low-resource tradeoff, and how model capacity interacts with language count to produce or alleviate the curse of multilinguality.
• Fifth, the model architecture and training configuration — the specific Transformer sizes, hyperparameters, and training infrastructure used for XLM-R Base and XLM-R, and why certain design choices (no language embeddings, full softmax, training past perplexity convergence) matter.
• Sixth, the downstream evaluation and fine-tuning protocol — how the pretrained model is adapted to classification, sequence labeling, and question answering, and what the various training modalities (cross-lingual transfer, translate-train, translate-train-all) mean operationally.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical scaling paper whose core idea is that multilingual model performance is bottlenecked by three factors — insufficient training data for low-resource languages, model capacity too small relative to the number of languages, and inadequate training duration — and that fixing all three simultaneously through large-scale data curation, increased model size, and extended training yields a single model that matches or exceeds monolingual baselines per language.

---

The Masked Language Model (MLM) Objective

The paper uses the multilingual masked language modeling objective introduced by Devlin et al. (2018) and adapted for multilingual training by Lample and Conneau (2019). The core idea: given a sequence of tokens from a sentence in some language, randomly mask some percentage of those tokens, feed the corrupted sequence through a Transformer encoder, and train the model to predict the original (masked) tokens from the surrounding context. Because the model sees no explicit language identifiers — the input stream contains raw token sequences from all 100 languages — the only way to minimize the loss is to learn representations that capture linguistic regularities that hold across languages, effectively discovering shared syntactic and semantic structure from the distributional statistics of text.

The objective is standard masked language modeling: for a token sequence, a fraction (typically 15%) of tokens are replaced with a special [MASK] token, and the model's output at those positions must predict the original token from the shared vocabulary of subword units. The loss is cross-entropy between the predicted probability distribution and the true token identity:

$\mathcal{L}_{\text{MLM}} = -\sum_{i \in \text{masked}} \log P(x_i | \text{context})$

where the sum runs over all masked positions $i$, $x_i$ is the true token at position $i$, and $P(x_i | \text{context})$ is the model's predicted probability for that token given the unmasked context tokens.

What it computes: for each masked position, the model outputs a probability distribution over the entire shared vocabulary (250K subword units in XLM-R), and the loss penalizes low probability assigned to the correct token. The total loss per batch is the average over all masked positions in all sequences in the batch. The model learns to use both local context (surrounding words, subword patterns) and longer-range context (sentence-level and cross-sentence dependencies) to disambiguate the masked token.

Why this form: the cross-entropy objective is the standard maximum-likelihood training signal for categorical prediction. The key design choice is what to mask. Unlike autoregressive language models (GPT) that predict tokens left-to-right, MLM conditions on bidirectional context — the model sees tokens to both the left and right of the masked position. This bidirectional conditioning is what makes the resulting representations effective for downstream understanding tasks (classification, NER, QA) rather than generation tasks. The multilingual aspect matters because the model must learn a shared representational space: tokens that play similar roles across languages (e.g., subject-verb-object structure, negation markers, named entity patterns) end up with similar representations because they co-occur with similar distributional contexts, even if the surface forms are completely different.

A critical detail: the paper does not use language embeddings. Lample and Conneau (2019) included a learned vector per language added to the token embeddings, which gave the model explicit information about which language each token comes from. The paper removes this because:

"Unlike Lample and Conneau (2019), we do not use language embeddings, which allows our model to better deal with code-switching."

Removing language embeddings forces the model to infer language identity from the token sequence itself — the subword tokens, their distribution, and their contexts are the only cues. This is beneficial for two reasons: (1) in real-world code-switching scenarios (where a single sentence mixes multiple languages), a single language embedding would be ambiguous or wrong; (2) it prevents the model from using language identity as a shortcut — it must learn genuinely cross-lingual representations that are not simply "English is language-17, so activate the English-processing subnetwork."

The paper also makes a deliberate decision to use only monolingual data (the MLM objective) and not use the Translation Language Modeling (TLM) objective from Lample and Conneau (2019). TLM uses parallel sentences: concatenate a source and target sentence with special separator tokens, mask tokens in both, and train the model to predict them. This requires parallel corpora, which exist for only a fraction of the world's languages. The paper's ablation in Section 5.1 shows that by scaling monolingual data and improving training, the MLM-only model matches the MLM+TLM performance — 75% average XNLI accuracy without needing any parallel data. This is practically significant because the long tail of low-resource languages that most need cross-lingual transfer have little to no parallel data.

---

The CC-100 Data Pipeline

The paper's most impactful engineering contribution is the construction of CC-100, a cleaned CommonCrawl corpus in 100 languages. The pipeline, described in Section 3 and detailed further via reference to Wenzek et al. (2019), transforms raw web crawl data into language-specific monolingual text suitable for pretraining.

Data source. The corpus uses CommonCrawl, an open repository of web crawl data that periodically scrapes billions of web pages. The paper uses "one CommonCrawl dump for English and twelve dumps for all other languages." English is the most represented language on the web, so a single dump provides sufficient data; other languages require multiple dumps to reach adequate volumes, particularly low-resource languages.

Language identification. Each document goes through a two-stage language identification pipeline: an internal language identification model (details not specified, presumably a proprietary classifier trained on large-scale web data) combined with the fastText language identifier (Joulin et al., 2017). Both models predict the language of a document, and only documents where both agree are kept. This dual-check reduces false positives — documents misclassified as a target language, which would inject noise into the pretraining data.

Quality filtering. The paper follows the methodology of Wenzek et al. (2019): for each language, a language model is trained on the data, and documents are scored by perplexity under this model. Documents with high perplexity (poor fit to the language model) are discarded, as they are likely to be noisy, off-topic, or in a different language than identified. The language model acts as an automatic quality filter — it learns what "good" text in that language looks like from the data itself, without requiring human annotations of document quality.

Scale achieved. Figure 1 and Appendix Table 6 quantify the resulting corpus. Total size exceeds 2 terabytes. The key comparison is Wikipedia versus CommonCrawl data sizes for individual languages. For high-resource languages, Wikipedia already provides substantial data: English Wikipedia provides roughly 300 GiB, and CommonCrawl also provides roughly 300 GiB. But for low-resource languages, the difference is dramatic:

For Swahili, Urdu, Burmese, and many other languages, the increase is one to two orders of magnitude. This crosses a critical threshold that the paper identifies empirically: "a few hundred MiB of text data is usually a minimal size for learning a BERT model" (Section 5.3). Wikipedia data for many languages falls below this threshold, meaning that prior models like mBERT and XLM-100 were essentially relying entirely on cross-lingual transfer for those languages — the pretraining data was too sparse for the model to learn genuine representations of the language's grammar, vocabulary, and patterns.

Languages covered. The paper trains on 100 languages, listed in Appendix Table 6. The selection covers a wide range of language families, scripts, and resource levels: Indo-European (English, French, German, Russian, Hindi, Urdu, etc.), Sino-Tibetan (Chinese), Niger-Congo (Swahili), Afroasiatic (Arabic, Hebrew, Amharic, Somali), Turkic (Turkish, Kazakh, Uzbek), Dravidian (Tamil, Telugu), Austronesian (Indonesian, Malay, Filipino), and many smaller families and isolates. The inclusion of multiple scripts (Latin, Cyrillic, Arabic, Devanagari, Chinese characters, Thai, Korean Hangul, etc.) means the tokenizer must handle a highly diverse character set.

The Sentence Piece Tokenizer and Vocabulary Design

The tokenizer is a critical component in multilingual models because it serves as the shared interface between raw text and the model's representational space. All tokens across all languages must be mapped into a single vocabulary. The paper uses Sentence Piece (Kudo and Richardson, 2018) with a unigram language model (Kudo, 2018), applied directly to raw text without any language-specific preprocessing.

Why Sentence Piece over BPE/WordPiece. Prior multilingual models used different tokenizers: mBERT used WordPiece (Devlin et al., 2018) with language-specific tokenization; XLM-100 used byte-pair encoding (BPE) with language-specific preprocessing (e.g., separate tokenization for Chinese, which doesn't use whitespace). These approaches work well for per-language models but create complications for multilingual models: a tokenizer trained on one language's preprocessing conventions may not work well on raw text in another language.

Sentence Piece solves this by operating directly on the raw byte stream, treating the input as a sequence of Unicode characters and learning subword units that maximize the likelihood of the training data under a unigram model. There is no language-specific step — the same tokenizer processes English, Chinese, Arabic, and Swahili identically. The paper confirms this simplification doesn't hurt performance: "We did not observe any loss in performance for models trained with SPM when compared to models trained with language-specific pre-processing and byte-pair encoding (see Figure 7)."

Vocabulary size: 250K subword units. This is a deliberate, large vocabulary. For comparison, monolingual BERT uses 30K tokens; mBERT uses 110K; XLM-100 used 95K–200K. The paper's ablation (Figure 6) directly studies vocabulary size: holding the total number of model parameters constant, increasing vocabulary size from 32K to 256K yields a 2.8% increase in XNLI average accuracy. The reason is that a larger vocabulary allows the model to learn more language-specific subword units, reducing the degree to which unrelated words in different languages share token representations. For example, with a 32K vocabulary, semantically unrelated words in Swahili and English that happen to share character sequences might be forced into the same subword unit; with 250K, the model has enough capacity to learn separate units for each language's frequent patterns while still sharing subwords for genuinely cross-lingual patterns (numbers, punctuation, loanwords, cognates).

The tradeoff is in parameter allocation. The embedding layer size is $V \times H$, where $V$ is vocabulary size (250K) and $H$ is hidden size (1024 for XLM-R). This gives an embedding matrix of $250,000 \times 1,024 = 256$ million parameters — roughly half of XLM-R's 550M total parameters. The paper explicitly addresses this: "multilingual models can benefit from allocating a higher proportion of the total number of parameters to the embedding layer even though this reduces the size of the Transformer." By shifting capacity from the Transformer layers to the embedding layer, the model gets better per-language token representations at the cost of fewer parameters for cross-token attention and feed-forward processing. The ablation supports this choice: even at fixed total capacity, increasing vocabulary size helps.

The paper also reports testing vocabulary sizes of 128K, 256K, and 512K, finding that "we observe more than 3% gains in overall accuracy on XNLI by simply increasing the vocab size from 128k to 512k." The final choice of 250K (not 512K) is a pragmatic compromise: for the large model, the full softmax over 512K tokens would be computationally expensive during training and inference, and the marginal gains beyond 250K are likely diminishing.

Full softmax vs. sampled softmax. The paper specifies "a full softmax" for the output layer — meaning that at every masked position, the model computes the probability distribution over all 250K vocabulary items, not a sampled subset. This is more expensive than techniques like adaptive softmax or negative sampling but ensures that the model's predictions are calibrated across the entire vocabulary, which matters for downstream tasks that require fine-grained token distinctions.

---

Language Sampling and Capacity Allocation

The training data is imbalanced: English has 55.6 billion tokens, while Sundanese has 10 million — a ratio of over 5,000:1. If the model simply processes data in proportion to its natural frequency, it would see English tokens thousands of times more often than low-resource language tokens. This would cause the model to specialize almost entirely in English, with low-resource languages receiving negligible representation.

The paper addresses this with an exponentially smoothed sampling distribution, controlled by a single parameter $\alpha$. The sampling mechanism, inherited from Lample and Conneau (2019), works as follows.

Sampling distribution. For each language $l$, let $n_l$ be the number of sentences in the language's training corpus. A multinomial distribution is constructed where the probability of sampling a batch from language $l$ is:

$p_l = \frac{n_l^\alpha}{\sum_{k} n_k^\alpha}$

where the sum in the denominator is over all 100 languages, $\alpha \in [0, 1]$ is a smoothing parameter, and $n_l$ and $n_k$ are the corpus sizes in sentences for languages $l$ and $k$ respectively.

What it computes: the probability $p_l$ that a given training batch is sampled from language $l$. When $\alpha = 1$, $p_l$ is simply proportional to corpus size — high-resource languages dominate training. When $\alpha = 0$, all languages are sampled equally regardless of corpus size — each language gets exactly 1% of batches. When $\alpha = 0.3$, as used for XLM-R, the distribution is smoothed: high-resource languages are sampled less often than their natural frequency, and low-resource languages are sampled more often.

Why this form: exponential smoothing with $\alpha$ between 0 and 1 provides a tunable knob between the two extremes. The paper experiments with values of 0.01, 0.3, 0.7, and 1.0 (Figure 5). The results reveal a clear tradeoff: higher $\alpha$ (more high-resource sampling) improves high-resource language performance, while lower $\alpha$ (more uniform sampling) improves low-resource language performance. The paper finds that $\alpha = 0.3$ is "an optimal value for α" when considering overall (all-language average) performance. This has the practical effect of oversampling low-resource languages relative to their corpus size: Swahili, with 275M tokens, gets a larger share of training batches than its raw proportion would dictate, forcing the model to allocate representational capacity to learning Swahili patterns.

The curse of multilinguality formalized. The paper's key conceptual contribution is recognizing that this sampling distribution interacts with model capacity to produce the "curse of multilinguality." For a fixed-capacity model, as you increase the number of languages $K$:

$\text{capacity per language} \propto \frac{\text{total parameters}}{K}$

Each additional language dilutes the number of parameters available to represent each individual language's patterns. Positive transfer — where knowledge learned from related high-resource languages helps low-resource languages — can offset this dilution up to a point. But beyond some threshold of $K$, the benefits of positive transfer saturate while dilution continues, and all-language performance degrades.

Figure 2 shows this non-monotonically: going from 7 to 15 languages improves low-resource language performance (positive transfer dominates), but going from 15 to 30, 60, and 100 degrades all-language performance (dilution dominates). The degradation is steeper for low-resource languages because they receive fewer training batches even with $\alpha = 0.3$.

Figure 4 shows that increasing model capacity can alleviate the curse. Specifically:

• For XLM-7 → XLM-30 → XLM-100, the authors increase the hidden size from 768 → 960 → 1152 while keeping the other Transformer dimensions proportional. This increases the total parameter count.
• XLM-30 with increased capacity matches XLM-7 performance, overcoming the curse for the 30-language setting.
• XLM-100 with increased capacity improves over the fixed-capacity XLM-100 but still falls short of XLM-7, because "higher vocabulary dilution" — the shared vocabulary of 150K tokens gets spread thinner across 100 languages than across 7. This means that even with more Transformer parameters, the embedding layer's capacity to represent language-specific subwords is a separate bottleneck.

The implication: to scale to 100 languages, you need to scale both Transformer parameters (hidden size, layers) and vocabulary size. The paper does this for XLM-R by using 550M parameters and a 250K vocabulary, which is why XLM-R can overcome the curse sufficiently to match monolingual performance.

Model Architecture and Training Configuration

Architecture. XLM-R uses the standard Transformer encoder architecture (Vaswani et al., 2017) with two sizes:

• XLM-R Base: 12 layers, hidden size 768, feed-forward size 3072, 12 attention heads, 250K vocabulary, ~270M parameters.
• XLM-R (Large): 24 layers, hidden size 1024, feed-forward size 4096, 16 attention heads, 250K vocabulary, ~550M parameters.

The large architecture is identical in Transformer dimensions to BERT-Large and RoBERTa, making it a direct comparison point. The main difference is the larger vocabulary (250K vs. 30K–50K), which accounts for roughly half of XLM-R's parameters being in the embedding layer.

A note from Appendix B: for Transformer encoders, the number of parameters can be approximated by:

$P \approx 4LH^2 + 2L H H_{\text{ff}} + V H$

where $L$ is the number of layers, $H$ is the hidden size, $H_{\text{ff}}$ is the feed-forward dimension, and $V$ is the vocabulary size. The first term $4LH^2$ accounts for the query, key, value, and output projections in multi-head attention; the second term $2L H H_{\text{ff}}$ accounts for the two linear layers in each feed-forward block; the third term $V H$ accounts for the input embedding matrix.

Training infrastructure and hyperparameters. The large XLM-R model was trained on "five-hundred 32GB Nvidia V100 GPUs with a batch size of 8192." Training ran for 1.5 million updates. With a batch size of 8192, this means the model processed approximately $1.5\text{M} \times 8192 \approx 12.3$ billion sequences during training. Each sequence contains up to the maximum sequence length (not explicitly specified in the main paper, but typically 512 tokens for BERT-style models), so the total tokens processed is on the order of $10^{12}$.

The learning rate and optimizer are not specified in the main paper but follow the standard RoBERTa recipe — typically Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-6}$, with a peak learning rate around $10^{-4}$ and linear warmup followed by linear decay. The paper notes that these hyperparameters are "similar to Liu et al. (2019)" (RoBERTa).

Training past perplexity convergence. A critical empirical finding (Section 5.1): "using validation perplexity as a stopping criterion for pretraining caused the multilingual MLM in Lample and Conneau (2019) to be under-tuned." The authors observed that "performance on downstream tasks continues to improve even after validation perplexity has plateaued." This matches the finding from RoBERTa (Liu et al., 2019) in the monolingual setting. The practical implication: you cannot trust validation perplexity as a proxy for downstream task performance. The model must be trained longer — far past the point where perplexity stops decreasing — to maximize downstream transfer performance.

No language embeddings. As discussed in the MLM section, the model does not include learned language embeddings. This differs from XLM-100 (Lample and Conneau, 2019) and is motivated by better handling of code-switching scenarios.

Simplifying to Sentence Piece on raw text. The paper argues that using SPM directly on raw text, without language-specific tokenization preprocessing, simplifies the pipeline without hurting performance (Figure 7). This makes the model easier to apply to new languages and raw web text at inference time, since there is no need to run language-specific tokenizers or preprocessing steps.

---

Downstream Evaluation and Fine-Tuning Protocol

The paper evaluates XLM-R on four tasks, each with a specific fine-tuning protocol. The key evaluation paradigm is cross-lingual zero-shot transfer: the model is fine-tuned on labeled data in English (the source language) and then evaluated on test sets in other languages (the target languages) without having seen any labeled examples in those languages.

Cross-lingual Natural Language Inference (XNLI). The task is to determine whether a premise sentence entails, contradicts, or is neutral with respect to a hypothesis sentence (3-way classification). The benchmark provides ground-truth training data in English (from the MultiNLI corpus) and machine-translated training data in 14 other languages, plus ground-truth dev and test sets in 15 languages. The paper reports results under three training paradigms:

1. Cross-lingual Transfer (train on English, evaluate on all 15): The model is fine-tuned only on the English MultiNLI training set. At evaluation time, the model receives premise-hypothesis pairs in each of the 15 XNLI languages and must classify them — without ever having seen NLI training data in any language except English. This is the most challenging setting and the primary evaluation for cross-lingual transfer capability.

2. Translate-Train (per-language): The English training set is machine-translated to each target language, and a separate model is fine-tuned for each language on its translated training data. This uses a multilingual pretrained model but fine-tunes it on language-specific data, which is less interesting for cross-lingual transfer but establishes an upper bound.

3. Translate-Train-All (multilingual): The English training set and all 14 machine-translated versions are concatenated into a single multilingual training set, and the model is fine-tuned on this combined data. This leverages cross-lingual data augmentation — the model sees the same NLI task in 15 languages simultaneously.

For the cross-lingual transfer setting, the paper uses a single model selected on the joint dev set of all 15 languages rather than selecting the best model separately for each language. This is a practical choice: "for cross-lingual transfer, while this approach is not fully zero-shot transfer, we argue that in real applications, a small amount of supervised data is often available for validation in each language." The reported results (Table 1) show a 0.6% decrease from per-language model selection to a single shared model selection, indicating that the choice matters but not dramatically. The paper encourages the community to adopt the single-model selection protocol for fairer comparisons.

Named Entity Recognition (NER). The paper uses the CoNLL-2002 (Spanish, Dutch) and CoNLL-2003 (English, German) datasets. Three fine-tuning settings are reported (Table 2):

1. Train on English only (en): Fine-tune on the English CoNLL-2003 training set; evaluate on all four languages. This tests zero-shot cross-lingual NER transfer.

2. Train on each language (each): Fine-tune a separate model for each language on that language's training set. This tests per-language performance using the multilingual pretrained model.

3. Train on all languages (all): Fine-tune a single model on the concatenation of all four training sets. This tests multilingual learning where the model sees NER data in multiple languages.

Unlike prior work (Lample et al., 2016; Akbik et al., 2018), XLM-R does not use a linear-chain CRF on top of the Transformer representations. Instead, it uses a simple linear classification layer that predicts the NER tag for each token independently. This makes the comparison somewhat unfavorable to XLM-R (since CRFs model label dependencies, which helps for structured prediction like NER), but even without the CRF, XLM-R performs competitively.

Cross-lingual Question Answering (MLQA). The MLQA benchmark (Lewis et al., 2019) provides extractive QA data — given a passage and a question, the model must identify the span of text in the passage that answers the question. The training data is English SQuAD; evaluation is on MLQA's test sets in English, Spanish, German, Arabic, Hindi, Vietnamese, and Simplified Chinese. The model is fine-tuned on SQuAD and then evaluated on all seven MLQA languages. The paper follows the same procedure as Lewis et al. (2019) "by training on the English training data and evaluating on the 7 languages."

GLUE Benchmark. To assess whether XLM-R's English performance is competitive with state-of-the-art monolingual models, the paper fine-tunes and evaluates on the GLUE benchmark, which includes tasks like MNLI (inference), SST-2 (sentiment), QQP (paraphrase detection), MRPC (paraphrase), STS-B (semantic similarity), and QNLI (QA/inference). The comparison is to BERT-Large, XLNet-Large, and RoBERTa — all monolingual English models. This evaluation answers the question: does pretraining on 100 languages hurt English performance compared to pretraining only on English?

---

Summary of Design Choices and Their Justifications

• No language embeddings: removes a shortcut that could prevent genuine cross-lingual representation learning; better handles code-switching at inference time.
• No TLM (parallel data): the paper's ablations show that scaled monolingual MLM alone matches prior MLM+TLM results, making the approach applicable to languages without parallel corpora.
• $\alpha = 0.3$ for language sampling: empirically optimal for balancing the high-resource versus low-resource tradeoff when optimizing for all-language average performance.
• 250K vocabulary: large enough to provide language-specific subword coverage for 100 languages without making the full softmax computationally prohibitive; ablation shows it improves over smaller vocabularies.
• Training past perplexity convergence: downstream task performance continues improving after perplexity plateaus; using perplexity as a stopping criterion under-trains the model.
• Sentence Piece on raw text: simplifies preprocessing without performance loss compared to BPE with language-specific tokenization; makes the model easier to deploy on raw text.
• Two model sizes (Base and Large): enables both the scaling analysis (comparing fixed and increased capacity for different language counts in the ablation studies) and practical deployment (Base for resource-constrained settings, Large for maximum performance).
• Single model for all languages at fine-tuning: avoids the operational complexity of maintaining per-language models; a single shared model is selected on the joint dev set for practical deployment realism.

4. Key Insights and Innovations

Innovation 1: The "Curse of Multilinguality" as a Formal Diagnostic Concept

The paper's most conceptually distinctive contribution is not the existence of a performance tradeoff when adding languages — that had been observed in multilingual NMT (Arivazhagan et al., 2019) — but rather naming, systematically characterizing, and diagnosing the curse of multilinguality as a capacity-dependent, non-monotonic phenomenon for pretrained encoders. Prior work treated the underperformance of multilingual models relative to monolingual ones as a generic limitation, a vague sense that "more languages = worse per-language results." By giving it a name and an operational definition — performance degradation across all languages as language count increases beyond a threshold for a fixed-capacity model — the paper transforms an anecdotal observation into a diagnostic that can be studied, measured, and mitigated.

What makes this genuinely novel rather than an obvious observation is the non-monotonicity it uncovers. Figure 2 shows that going from 7 to 15 languages improves low-resource language performance (positive transfer from related high-resource languages dominates), but going to 30, 60, and 100 languages degrades all-language performance (capacity dilution takes over). The dominant prior assumption — implicit in the design of mBERT and XLM-100, which simply included all available Wikipedia languages — was that more languages should be monotonically better for cross-lingual transfer, or at worst, that low-resource languages would improve while high-resource languages would decline. The paper shows neither assumption holds: there is an optimal point for a given capacity, and beyond it, everyone loses.

The paper further establishes that the curse is capacity-dependent, not fundamental. Figure 4 demonstrates that increasing the Transformer's hidden size alongside the number of languages can recover the lost performance: XLM-30 with increased capacity (hidden size 960) matches XLM-7 (hidden size 768). However, XLM-100 with increased capacity (hidden size 1152) still falls short, revealing a second bottleneck — "higher vocabulary dilution." The shared vocabulary becomes a parallel capacity constraint: even with more Transformer parameters, the embedding layer cannot adequately represent 100 languages' subword inventories in 150K tokens. This two-dimensional diagnosis — Transformer capacity and vocabulary capacity as separable, interacting constraints — is more nuanced than the simple "add more parameters" prescription from the NMT literature and provides a framework for reasoning about how to scale multilingual models.

The significance of this innovation extends beyond the paper's own results. By naming and characterizing the curse, the paper gives the field a vocabulary and a set of experimental probes for studying multilingual model scaling. A researcher designing a new multilingual model can now ask: for my target language count, am I above or below the capacity-dilution threshold? Am I bottlenecked on Transformer parameters or vocabulary size? Prior to this work, these questions had no operational meaning.

Innovation 2: Demonstrating That Monolingual Data at Scale Obviates the Need for Cross-Lingual Supervision

The paper's second major conceptual contribution is the empirical demonstration that scaling monolingual data eliminates the need for explicit cross-lingual supervision in learning transferable representations. Lample and Conneau (2019) had established the state of the art using Translation Language Modeling (TLM), a supervised objective that requires parallel sentences — the model sees source and target sentences concatenated, masks tokens in both, and learns to predict them cross-lingually. The assumption in the field, natural given the results, was that parallel data provided a qualitatively different and necessary signal for cross-lingual alignment.

This paper overturns that assumption. Through systematic ablation (Section 5.1), the authors show that by scaling monolingual data (CommonCrawl instead of Wikipedia), training longer (past perplexity convergence), and using better hyperparameters (larger batches, Sentence Piece tokenization), the unsupervised MLM objective alone matches the previously reported MLM+TLM performance. Specifically, they report improving XLM's MLM-only performance from 71.3% to over 75% average accuracy on XNLI — "on par with their supervised translation language modeling (TLM) objective." This is not an incremental gain; it is a qualitative shift in what is possible with only monolingual data.

The practical implication is substantial, and the paper makes it explicit: "given our focus on unsupervised learning, we decided to not use the supervised TLM objective for training our models." This decision is justified by the reality that parallel corpora exist for only a fraction of the world's languages — precisely the high-resource languages that already have training data. The long tail of low-resource languages that most need cross-lingual transfer (Swahili, Urdu, Burmese, Amharic) typically lack any parallel data at all. By showing that TLM is unnecessary at scale, the paper makes the entire approach applicable to languages where the previous state of the art (which required parallel data) could not be replicated.

This is a fundamental conceptual shift, not a small refinement. It reframes the problem from "how do we get cross-lingual signal" (where the answer was parallel data or explicit alignment objectives) to "how do we ensure each language individually has enough monolingual data for representation learning" (where the answer is better data curation at scale). The cross-lingual transfer emerges from the shared vocabulary and the model's capacity bottleneck — the multilingual MLM objective forces the model to learn shared representations because there is no other way to fit 100 languages into a single parameter space. Parallel data becomes an optimization, not a requirement.

Innovation 3: The First Demonstration That a Single Multilingual Model Can Match Monolingual Performance Per Language

Prior to this work, the dominant assumption in the field — backed by consistent empirical evidence — was that multilingual models inherently traded per-language performance for cross-lingual transfer. You could have a single model that worked moderately well in many languages, or you could train separate monolingual models that each worked better on their target language. This tradeoff was considered fundamental, not an artifact of insufficient training.

The paper provides the first evidence that this tradeoff is not fundamental. In Table 5 (Section 5.3), the paper compares multilingual XLM-7 models against 14 monolingual BERT models trained on the same data (Wikipedia or CommonCrawl, capped at 60 GiB per language). Under cross-lingual transfer (train on English, evaluate on 7 languages), the multilingual models slightly underperform the average of monolingual models — 76.2% vs. 77.5% for CC-trained models. But under the translate-train-all setting — where the multilingual model is fine-tuned on NLI training data in all 7 languages simultaneously rather than just English — the multilingual model outperforms the average monolingual model: 80.0% vs. 77.5%. The multilingual model leverages training data from multiple languages to overcome the capacity dilution that occurs during pretraining.

This result is surprising on its face: a model that had to share its parameters across 7 languages during pretraining, and was then fine-tuned on those same 7 languages together, outperforms the average of 7 models that each had dedicated parameters and dedicated monolingual pretraining. The paper calls this "a surprising result that shows that the capacity of multilingual models to leverage training data coming from multiple languages for a particular task can overcome the capacity dilution problem."

The demonstration extends to the large-scale setting. Table 4 shows XLM-R achieving 91.8% average on GLUE, compared to 92.0% for XLNet-Large and 92.8% for RoBERTa — within 1 point of the state-of-the-art monolingual English model despite being trained on 100 languages. On MLQA (Table 3), XLM-R actually outperforms BERT-Large on English question answering (80.6 F1 vs. 80.2) while simultaneously providing cross-lingual transfer to 6 other languages. This is the paper's most striking empirical result: a multilingual model that is not merely "competitive" with monolingual models but occasionally strictly better, while also providing the cross-lingual functionality that monolingual models lack entirely.

The significance of this finding goes beyond benchmarking. It changes the default answer to the question "should I train one multilingual model or many monolingual ones?" Prior to this work, the answer was "it depends — multilingual models are worse but operationally simpler." After this work, for languages with sufficient pretraining data and a model of adequate capacity, the answer becomes "train the multilingual model — you lose little to no per-language performance and gain cross-lingual transfer for free." This has direct implications for how NLP systems are built and deployed, especially in multi-language environments where maintaining separate models was the cost of competitive performance.

Innovation 4: Vocabulary Size as a First-Class Capacity Dimension, Decoupled from Transformer Size

The paper's ablation studies establish vocabulary size as an independent, first-class capacity dimension in multilingual models, distinct from and complementary to Transformer depth and width. Prior work treated vocabulary size primarily as a computational constraint — larger vocabularies are more expensive due to the softmax, so you choose a size that balances coverage and cost, typically 30K–110K for multilingual models. The implicit assumption was that vocabulary size mattered for tokenization quality but not fundamentally for representation capacity, since the important learning happened in the Transformer layers.

Figure 6 overturns this assumption with a clean ablation: for the same XLM-100 model trained on Wikipedia, varying vocabulary size from 32K to 256K while holding the total number of parameters constant (by adjusting the Transformer width to compensate) yields a 2.8% improvement in XNLI average accuracy. Even more striking: when Transformer capacity is also increased alongside vocabulary size, the gains are larger — the paper reports "more than 3% gains in overall accuracy on XNLI by simply increasing the vocab size from 128k to 512k." This means vocabulary size is not merely about tokenization fidelity; it is about the model's ability to maintain language-specific representational capacity.

The mechanism is intuitive in retrospect: a larger vocabulary means more distinct subword units available to represent language-specific morphemes, reducing the degree to which semantically unrelated tokens across languages share the same subword representation. With 32K tokens, the Japanese character 語 and the unrelated sequence "go" in English might share a subword unit. With 250K tokens, there is enough capacity for each language to have its own frequent subword inventory, with sharing only where genuine cross-lingual overlap exists (cognates, loanwords, numbers, punctuation). The embedding layer becomes a form of language-specific preprocessing learned end-to-end during pretraining, and its capacity matters independently of the Transformer's capacity for cross-token computation.

This finding has direct implications for multilingual model design that go beyond this paper. It means that for a fixed parameter budget, the optimal allocation between embedding parameters and Transformer parameters is not the same as for monolingual models. The paper reports that XLM-R allocates roughly half its 550M parameters to the embedding layer (250K × 1024 = 256M). A monolingual model with 50K vocabulary and 1024 hidden size allocates only about 12% of its parameters to embeddings. The paper's results suggest this shift in allocation is optimal for multilingual settings, and future work on multilingual models should treat vocabulary size as a hyperparameter to be tuned alongside model depth and width, not a fixed constraint driven by computational cost.

The significance is partly theoretical — it establishes that vocabulary capacity and Transformer capacity are two separate bottlenecks that can interact — and partly practical — it gives concrete guidance (250K vocabulary for 100 languages, larger is better up to at least 512K) that subsequent work can build on.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses four evaluation benchmarks. For cross-lingual natural language inference, it uses XNLI (Conneau et al., 2018), which provides ground-truth dev and test sets in 15 languages and an English training set from MultiNLI, plus machine-translated training sets in 14 languages. For named entity recognition, it uses CoNLL-2002 (Sang, 2002) and CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003), covering English, Dutch, Spanish, and German. For cross-lingual question answering, it uses MLQA (Lewis et al., 2019), which extends SQuAD to Spanish, German, Arabic, Hindi, Vietnamese, and Simplified Chinese. For monolingual English evaluation, it uses the GLUE benchmark (Wang et al., 2018), which includes MNLI, QNLI, QQP, SST-2, MRPC, and STS-B.

• Base model(s). The paper trains two XLM-R models: XLM-R Base (12 layers, 768 hidden size, 12 attention heads, 270M parameters) and XLM-R (24 layers, 1024 hidden size, 16 attention heads, 550M parameters). For the ablation studies, the authors use a BERT-Base architecture (12 layers, 768 hidden size) with a 150K vocabulary, expanding the hidden size to 960 for 30-language models and 1152 for 100-language models in the capacity scaling experiments. The base model family is the standard Transformer encoder (Vaswani et al., 2017), chosen to maximize comparability with prior work — specifically, XLM-R Large matches BERT-Large and RoBERTa in Transformer dimensions, isolating the effect of multilingual pretraining from architectural change.

• Metrics. For XNLI, the metric is classification accuracy (percentage of premise-hypothesis pairs correctly classified as entailment, contradiction, or neutral), reported per language and averaged across all 15 languages. For NER, the metric is span-level F1 score, with results reported per language and averaged across the four CoNLL languages. For MLQA, the metrics are F1 score and exact match (EM) score on the extractive QA task. For GLUE, the metrics follow the standard per-task conventions (accuracy for MNLI, QNLI, SST-2; F1 for QQP, MRPC; Pearson/Spearman correlation for STS-B) and are reported as an average across tasks. All XNLI results for XLM-R are "averaged over five different seeds" (Table 1 caption).

• Baselines. The paper compares against several prior multilingual models: mBERT (Devlin et al., 2018), trained on Wikipedia in 104 languages with a 110K WordPiece vocabulary and 172M parameters; XLM-100 (Lample and Conneau, 2019), trained on Wikipedia in 100 languages with 570M parameters and both MLM and TLM objectives; XLM-15 (Lample and Conneau, 2019), trained on Wikipedia in 15 languages with 250M parameters; Unicoder (Huang et al., 2019), trained on Wikipedia and machine translation data in 15 languages. For monolingual baselines, it uses BERT-Large (335M parameters), XLNet-Large, and RoBERTa (355M parameters). For the multilingual vs. monolingual comparison (Table 5), it trains 14 monolingual BERT-Base models (one per language) on Wikipedia and CommonCrawl. For NER, baseline results from Lample et al. (2016) (LSTM+CRF) and Akbik et al. (2018) (Flair) are included. For MLQA, baselines are taken directly from Lewis et al. (2019). Machine translation baselines for XNLI include translate-test (translate dev and test sets to English, use a monolingual English model) and translate-train (translate the English training set to each target language and fine-tune per language).

• Generation budget / compute accounting. The paper does not use a formal "generation budget" metric since it studies encoder models evaluated through fine-tuning, not generative decoding. Instead, the primary resource axis is model capacity (number of parameters), pretraining data volume (in GiB or tokens), and number of pretraining updates. The large XLM-R model was trained for 1.5 million updates with a batch size of 8192 on 500 32GB Nvidia V100 GPUs. The ablation studies use controlled comparisons: for the vocabulary size experiment (Figure 6), the total number of model parameters is held constant while varying vocabulary size by adjusting the Transformer's hidden dimension. For the capacity scaling experiment (Figure 4), the model architecture is widened as languages increase, with details provided in- text. For data scaling comparisons (Figure 3), models are trained on Wikipedia versus CommonCrawl with the same architecture (BERT-Base).

• Cross-validation / statistical protocol. The paper does not explicitly describe a cross-validation procedure for downstream evaluations. Instead, it relies on standard train/dev/test splits provided by each benchmark. For XNLI, the paper introduces a specific protocol choice: model selection is done on the joint dev set of all 15 languages rather than selecting the best model separately for each language. This produces a single model for all languages — the paper explicitly reports the accuracy drop from per-language selection to joint selection (71.3% to 70.7% for XLM-100, a 0.6% decrease) and "encourage[s] the community to adopt this setting." The five-seed averaging for XLM-R results adds statistical rigor to the main results. However, the paper does not report standard deviations or confidence intervals for any metric — only means are given. The ablation studies (Figures 2–7) appear to report single-run results without error bars, though the number of training runs per configuration is not specified.

Main Quantitative Results

Cross-Lingual Natural Language Inference (XNLI)

The headline result appears in Table 1: XLM-R achieves 80.9% average accuracy on the 15 XNLI languages under cross-lingual transfer (train on English, evaluate on all 15), compared to 70.7% for XLM-100 (the prior best open-source multilingual model with joint model selection) and 66.3% for mBERT. This represents a 10.2% improvement over XLM-100 and a 14.6% improvement over mBERT. The gains are particularly large on low-resource languages: XLM-R outperforms XLM-100 by 15.7% on Swahili (73.9% vs. 58.2%) and 11.4% on Urdu (73.8% vs. 62.4%). Compared to mBERT, the gains on Swahili and Urdu are 23.5% and 15.8% respectively.

XLM-R also outperforms the previous state of the art (Unicoder, 75.4% average; XLM with MLM+TLM, 75.1% average) by approximately 5.5–5.8%, despite those models handling only 15 languages compared to XLM-R's 100. The XLM-R Base model (270M parameters) achieves 76.2% average accuracy, which already outperforms XLM-100 (570M parameters) by 5.5%, showing that the data scaling and training improvements yield gains beyond what raw parameter count alone would predict.

Under the translate-train-all setting (multilingual training on all 15 translated training sets), XLM-R reaches 83.6% average accuracy, a new state of the art that surpasses Unicoder's previous best of 78.5% by 5.1%. Here, the gap between XLM-R and XLM-100 widens from 10.2% (cross-lingual transfer) to 6.7% (translate-train-all: 83.6% vs. 76.9%), indicating that XLM-R's stronger pretraining benefits transfer learning even when multilingual fine-tuning data is available.

For comparison against machine translation baselines: the translate-test approach (translate everything to English, use an English-only model) yields 77.8% with RoBERTa and 76.2% with BERT-en (Table 1). XLM-R's cross-lingual transfer (80.9%) outperforms both — meaning the multilingual model doing zero-shot transfer into foreign languages outperforms a stronger monolingual English model operating on machine-translated inputs. This is a non-obvious result: it is better to let the multilingual model process the original language text than to translate everything into English and use the best available English model.

The per-language breakdown (Table 1, cross-lingual transfer row) reveals that XLM-R's performance varies by target language: English (89.1%), French (84.1%), Spanish (85.1%), German (83.9%) form the top tier; Greek (82.9%), Bulgarian (84.0%), Russian (81.2%) form a strong middle tier; Turkish (79.6%), Arabic (79.8%), Vietnamese (80.8%), Thai (78.1%), Chinese (80.2%) form a third tier; and Hindi (76.9%), Swahili (73.9%), Urdu (73.8%) trail but remain dramatically better than prior models' performance on those languages (Swahili was 50.4% for mBERT, 58.0% for XLM-100). The gap between the best and worst language for XLM-R is 15.3 percentage points (89.1% English vs. 73.8% Urdu), compared to 31.7 points for mBERT (82.1% English vs. 50.4% Swahili), demonstrating that data scaling compresses but does not eliminate the cross-lingual performance gap.

Named Entity Recognition (NER)

Table 2 reports results on CoNLL-2002 and CoNLL-2003. Under cross-lingual transfer (train on English, evaluate on all four languages), XLM-R achieves an average F1 of 80.94%, compared to 78.52% for mBERT — a 2.42 F1 improvement. The per-language breakdown: English (92.92%), Dutch (80.80%), Spanish (78.64%), German (71.40%). The large gap between English and German (21.52 F1 points) reflects the difficulty of zero-shot NER transfer, where entity boundaries and types vary across languages. XLM-R Base achieves 79.11% average under cross-lingual transfer, outperforming mBERT by 0.59 F1 despite having roughly similar parameter counts (270M vs. 172M), showing that data quality and training improvements benefit even the smaller model.

When trained on each language separately (each), XLM-R achieves an average F1 of 90.24%, outperforming mBERT's 88.28% by 1.96 F1 and the LSTM+CRF baseline from Lample et al. (2016) (84.25%) by 5.99 F1. Notably, XLM-R does not use a linear-chain CRF, while Akbik et al. (2018) (Flair) does. Despite this architectural disadvantage for structured prediction, XLM-R outperforms Flair on Dutch (92.53 vs. 90.44, a 2.09 F1 improvement) and is highly competitive on English (92.92 vs. 93.18) and German (85.81 vs. 88.27).

When trained on all languages together (all), XLM-R achieves an average F1 of 89.43%, which is higher than the cross-lingual transfer setting (80.94%) by 8.49 points but slightly lower than per-language training (90.24%). This pattern — multilingual fine-tuning improves over zero-shot transfer but slightly underperforms dedicated per-language models — is expected, as the model must now handle four languages simultaneously at fine-tuning time.

Cross-Lingual Question Answering (MLQA)

Table 3 shows that XLM-R achieves 70.7% average F1 and 52.7% average EM on zero-shot cross-lingual QA (train on English SQuAD, evaluate on seven MLQA languages). The previous state of the art, XLM-15 from Lewis et al. (2019), achieved 61.6% F1 and 43.5% EM — XLM-R improves by 9.1 F1 and 9.2 EM points. Compared to mBERT (57.7% F1, 41.6% EM), XLM-R improves by 13.0 F1 and 11.1 EM points.

On English, XLM-R achieves 80.6% F1 and 67.8% EM, which outperforms BERT-Large (80.2% F1) reported in the original MLQA paper. This means the multilingual model, pretrained on 100 languages, is better at English QA than a monolingual English model of comparable architecture. For the non-English languages, XLM-R achieves: Spanish (74.1 F1, 56.0 EM), Vietnamese (71.3 F1, 50.9 EM), Hindi (69.2 F1, 51.6 EM), German (68.5 F1, 53.6 EM), Chinese (68.0 F1, 45.4 EM), and Arabic (63.1 F1, 43.5 EM). Arabic and Chinese show the lowest transfer performance, consistent with these languages being linguistically distant from English and using different scripts. XLM-R Base achieves 63.7% F1 and 46.3% EM, which already outperforms XLM-15 (61.6 F1) — again showing that data and training improvements benefit the smaller model more than the parameter count difference would suggest.

GLUE: Monolingual English Performance

Table 4 reports that XLM-R achieves 91.8% average dev performance on the GLUE benchmark, outperforming BERT-Large (90.2%) by 1.6 points and closely approaching XLNet-Large (92.0%) — only 0.2 points behind. Compared to RoBERTa (92.8%), XLM-R trails by 1.0 point. The per-task breakdown shows XLM-R outperforming BERT-Large on MNLI (88.9/89.0 vs. 86.6/-), QNLI (93.8 vs. 92.3), QQP (92.3 vs. 91.3), SST (95.0 vs. 93.2), and MRPC (89.5 vs. 88.0), while trailing slightly on STS-B (91.2 vs. 90.0). Against RoBERTa, the gaps are largest on MNLI (90.2 vs. 88.9), SST (96.4 vs. 95.0), and STS-B (92.4 vs. 91.2), with XLM-R competitive on QNLI (93.8 vs. 94.7) and QQP (92.3 vs. 92.2).

The paper frames this as demonstrating "for the first time, the possibility of multilingual modeling without sacrificing per-language performance." The 1.0-point gap to RoBERTa is small enough that the paper argues it can be closed "by alleviating the curse of multilinguality and vocabulary dilation" — that is, further scaling model capacity and vocabulary size.

Scaling Analysis Results (Ablation Figures)

The paper's ablation experiments, presented in Figures 2–7, establish the quantitative relationships that motivate XLM-R's design:

Transfer-dilution tradeoff (Figure 2). For models trained on CommonCrawl with a BERT-Base architecture, XNLI average accuracy varies with language count: 7 languages achieves approximately 76%; 15 languages achieves approximately 77% (slight gain from positive transfer, particularly on low-resource languages); 30 languages drops to approximately 74%; 60 languages drops to approximately 72%; 100 languages drops to approximately 71%. The low-resource language curve (Swahili and Urdu average) rises from approximately 67% at 7 languages to a peak of roughly 72% at 15–30 languages, then declines to roughly 66% at 100 languages. The high-resource language curve is flatter but still declines from approximately 78% at 7 languages to roughly 74% at 100 languages. The non-monotonicity of the low-resource curve is the key empirical signature of the curse being a genuine tradeoff rather than monotonic degradation.

Wikipedia vs. CommonCrawl (Figure 3). For XLM-7 (BERT-Base, 7 languages), switching from Wikipedia to CommonCrawl improves overall XNLI accuracy from approximately 71.8% to 76.2% — a 4.4-point gain. The low-resource language improvement is larger: approximately 61.5% to 69.0% (a ~7.5-point gain). The high-resource language improvement is approximately 79% to 80.5% (a ~1.5-point gain). This confirms that the primary beneficiaries of data scaling are low-resource languages, where Wikipedia corpora were below the "few hundred MiB" threshold for effective representation learning.

Capacity scaling (Figure 4). For models trained on Wikipedia: XLM-7 (768 hidden size) achieves approximately 74% XNLI accuracy. XLM-30 with increased capacity (960 hidden size) matches this at approximately 74%. XLM-100 with increased capacity (1152 hidden size) improves over fixed-capacity XLM-100 (which achieves roughly 68%) but still falls short at approximately 71%. The authors attribute the remaining gap to vocabulary dilution — the 150K vocabulary cannot adequately cover 100 languages even with more Transformer parameters.

Language sampling α (Figure 5). For XLM-100 trained on Wikipedia, varying the α parameter from 0.01 to 1.0 reveals the high-resource vs. low-resource tradeoff. At α = 0.01 (near-uniform sampling), low-resource languages achieve approximately 72% accuracy while high-resource languages achieve roughly 68%. At α = 1.0 (proportional sampling), the pattern reverses: high-resource languages reach approximately 76%, low-resource languages drop to roughly 50%. The overall average peaks at α = 0.3 with roughly 70.7% accuracy. This 0.7-point gain over more extreme values may seem modest, but the paper argues it represents the optimal balance between the competing objectives.

Vocabulary size (Figure 6). For XLM-100 trained on Wikipedia at fixed total capacity (parameters reallocated from Transformer width to embedding size): 32K vocabulary achieves approximately 63.5% XNLI accuracy; 64K achieves approximately 65.5%; 128K achieves roughly 66.5%; 256K achieves roughly 66.3% (a 2.8-point gain over 32K). When capacity is also increased alongside vocabulary ("increased capacity" line in Figure 6), the gains are larger and the curve is steeper, with 512K vocabulary achieving notably higher accuracy than 256K. The paper reports "more than 3% gains in overall accuracy on XNLI by simply increasing the vocab size from 128k to 512k" in the text, referencing this experiment.

Batch size, preprocessing, and training scale (Figure 7). For XLM-100 on Wikipedia, increasing batch size from 2048 to 8192 improves accuracy (specific numbers not quoted but visible as an upward trend in the left panel). For preprocessing, switching from BPE with language-specific tokenization to SPM on raw text (right panel) shows essentially identical performance — the bars for "BPE" and "SPM" overlap at the 8192 batch size. This validates the simplification to Sentence Piece.

Multilingual vs. Monolingual Results

Table 5 provides the direct comparison that supports the paper's central claim. For cross-lingual transfer (train on English, evaluate on 7 languages): the average of 7 monolingual BERT models trained on CommonCrawl achieves 77.5% (per-language: English 86.7%, French 81.2%, German 81.2%, Russian 78.2%, Chinese 79.5%, Swahili 70.8%, Urdu 65.1%); XLM-7 trained on CommonCrawl achieves 76.2% (English 85.7%, French 78.6%, German 79.5%, Russian 76.4%, Chinese 74.8%, Swahili 71.2%, Urdu 66.9%). The monolingual average outperforms the multilingual model by 1.3 points. Under translate-train-all (multilingual fine-tuning on training data from all 7 languages), XLM-7 achieves 80.0% average, outperforming the monolingual BERT average (77.5%) by 2.5 points. The gains are largest on Swahili (75.7% vs. 70.8%) and Urdu (71.5% vs. 65.1%), indicating that the multilingual model benefits from seeing NLI training data in high-resource languages when learning to do the task in low-resource languages — a form of cross-lingual data augmentation that monolingual models cannot access.

The numbers reveal an interesting pattern: XLM-7 slightly underperforms the monolingual average on English (85.7% vs. 86.7%) and Chinese (74.8% vs. 79.5%), but slightly outperforms or matches on Swahili (71.2% vs. 70.8%) and Urdu (66.9% vs. 65.1%). The cross-lingual transfer deficit is concentrated in languages that already have strong monolingual models.

Ablation Studies and Robustness Checks

Training data scale: Wikipedia vs. CommonCrawl (Figure 3). The comparison of XLM-7 models trained on Wikipedia versus CommonCrawl shows a 4.4-point overall improvement from data scaling, with low-resource languages benefiting disproportionately (approximately 7.5 points vs. 1.5 points for high-resource). This directly supports the claim that prior multilingual models were undertrained due to small Wikipedia corpora for low-resource languages. The fact that the same architecture (BERT-Base) yields dramatically different results purely from data quality and quantity confirms that data scale, not model architecture, was the binding constraint.

Training duration: effect of training past perplexity convergence. The paper reports in Section 5.1 that "performance on downstream tasks continues to improve even after validation perplexity has plateaued." This finding, while not presented as a standalone figure, is credited with enabling the improvement of XLM's MLM-only performance from 71.3% to over 75% on XNLI — matching the previously reported MLM+TLM performance. The implication is that using perplexity as a stopping criterion systematically under-trains multilingual models, and this explains part of the performance gap in prior work.

Capacity scaling to alleviate the curse of multilinguality (Figure 4). The experiment comparing fixed-capacity versus increased-capacity models as language count grows demonstrates that the curse is capacity-dependent. XLM-30 with increased capacity (960 hidden size) fully recovers the lost performance and matches XLM-7. XLM-100 with increased capacity (1152 hidden size) partially recovers but still lags due to vocabulary dilution — the shared 150K vocabulary is insufficient for 100 languages. This is a non-trivial finding: it means scaling model width alone cannot fully compensate for vocabulary constraints. The interaction between Transformer capacity and vocabulary capacity as independent bottlenecks is one of the paper's key empirical insights.

Vocabulary size at fixed total capacity (Figure 6). By holding total parameters constant and only varying the allocation between embedding size and Transformer width, the experiment isolates the effect of vocabulary capacity. The 2.8% improvement from 32K to 256K vocabulary demonstrates that vocabulary size matters for representational capacity, not just computational convenience. This is a genuinely non-obvious result, since the standard intuition would predict that broader Transformer layers provide more expressive power than a larger embedding table.

Language sampling α parameter (Figure 5). The sweep of α from 0.01 to 1.0 quantifies the high-resource/low-resource tradeoff with precision. At α = 0.01, low-resource language accuracy is approximately 72% while high-resource is roughly 68%; at α = 1.0, high-resource reaches roughly 76% while low-resource drops to roughly 50%. The 26-point swing in low-resource performance across the α range demonstrates the sensitivity of the allocation mechanism. The paper's choice of α = 0.3 for XLM-R corresponds to the peak of the overall average curve, though the curve is relatively flat in the 0.01–0.3 range — suggesting the choice is not extremely brittle.

Sentence Piece vs. language-specific preprocessing (Figure 7, right panel). The comparison shows that SPM applied directly to raw text produces results indistinguishable from BPE with language-specific tokenization. This is a robustness check that validates the simplification: the multilingual model does not require language-specific preprocessing pipelines, making it easier to deploy on raw text in any of its 100 languages. The finding is non-obvious because BPE with carefully tuned language-specific preprocessing (e.g., separate Chinese tokenization, which doesn't use whitespace) would be expected to produce better tokenization. The fact that SPM is "just as good" means the model learns to handle the tokenization artifacts during pretraining.

Model selection protocol: per-language vs. joint dev set (Table 1). The paper reports that moving from per-language model selection (71.3%) to joint dev set selection (70.7%) for XLM-100 costs 0.6% average accuracy. This is a practical robustness check showing that a single model can serve all languages without catastrophic degradation — the performance drop is modest, supporting the claim that a single model is viable for multi-language deployment.

Multi-seed evaluation. The XLM-R results in Table 1 are "averaged over five different seeds," which provides a basic statistical robustness check. However, the paper does not report standard deviations or confidence intervals, making it impossible to assess whether the 0.9-point difference between XLM-R (80.9%) and the next-best model (XLM-R Base at 76.2% or XLM-100 at 70.7%) is statistically significant. Given the small test set sizes (the XNLI test set has a few thousand examples per language — the exact number varies by language), the standard error of the mean could be non-trivial.

Negative result: vocabulary dilution at 100 languages even with increased capacity. The paper's attempt to scale XLM-100 by only increasing Transformer width (Figure 4) partially fails — performance improves over the fixed-capacity baseline but does not match XLM-7 or XLM-30. This negative result is informative: it shows that the curse of multilinguality has a vocabulary component that cannot be addressed by simply making the Transformer bigger. The practical lesson is that scaling to 100 languages requires scaling both Transformer dimensions and vocabulary size, which the paper implements for XLM-R (250K vocabulary, 550M parameters).

Critical Assessment

The experiments provide strong support for the paper's central claims, but with important qualifications about the specific conditions under which the claims hold and the limitations of the experimental design.

Does XLM-R significantly outperform prior multilingual models on cross-lingual transfer? The evidence is strong and consistent across three different task types (classification, sequence labeling, QA). XLM-R achieves +10.2% on XNLI over XLM-100 (Table 1), +2.4 F1 on NER over mBERT (Table 2), and +9.1 F1 on MLQA over XLM-15 (Table 3). The gains are not concentrated in a single task or language — the per-task and per-language breakdowns show consistent improvement, with the largest gains on low-resource languages. However, the comparison to XLM-100 is somewhat confounded: XLM-100 uses a different architecture (16 layers, 1280 hidden size vs. XLM-R's 24 layers, 1024 hidden size), a different tokenizer (BPE vs. SPM), different training data (Wikipedia vs. CommonCrawl), and different training duration. The paper cannot cleanly attribute the 10.2% gain to any individual factor — it is the combined effect of all changes.

Does the curse of multilinguality exist as characterized? The evidence from Figures 2 and 4 is internally consistent and supports the non-monotonic, capacity-dependent characterization. However, the experiments are conducted entirely on XNLI. Whether the same non-monotonic pattern (7 → 15 → 30 → 60 → 100 languages) would appear on NER, QA, or other tasks is not tested. Additionally, the language sets are nested rather than randomly sampled — the 7-language set is always the same seven languages (English, French, German, Russian, Chinese, Swahili, Urdu), and larger sets add languages on top. The curse characterization might differ if different languages were chosen for the base set, since the degree of positive transfer depends on which languages are included and how related they are. A Swahili speaker benefits from other Bantu languages being in the training set; if the 15-language set added only unrelated languages, the positive transfer curve might look different.

Can a single multilingual model match monolingual per-language performance? Table 5 provides evidence for XLM-7 under the translate-train-all setting, and Table 4 shows XLM-R trailing RoBERTa by only 1.0 GLUE point. These are impressive results but the claim is conditional on the evaluation setup. The monolingual models in Table 5 were trained by the authors specifically for this comparison — they are not the best possible monolingual models (they use BERT-Base, not Large, and were trained on capped 60 GiB data). A truly optimized monolingual model per language (e.g., RoBERTa-Large for English, CamemBERT for French, etc.) would likely outperform XLM-7 by a wider margin than Table 5 shows. The paper's claim that multilingual models can match monolingual performance is best understood as: for a fixed architecture and comparable pretraining data volume, the multilingual model is competitive — not that it surpasses the best possible monolingual models optimized independently for each language.

Is scaling monolingual data sufficient to eliminate the need for parallel supervision? The paper shows that XLM's MLM-only performance improved from 71.3% to 75%+ on XNLI, matching the previously reported MLM+TLM result. However, this is an internal comparison — the paper trained both the MLM-only and the MLM+TLM baselines. There is no direct experiment showing that adding TLM to XLM-R would not provide further gains. The paper's claim that TLM is unnecessary is an empirical observation at the scale tested (CommonCrawl data, 100 languages, 550M parameters), not a proof that it would never help. It is possible that at even larger scales, TLM would again provide a signal that monolingual data alone cannot match.

Weaknesses in experimental design. The paper has several limitations that affect the strength of its conclusions. All ablation studies (Figures 2–7) appear to use a single training run per configuration — error bars or standard deviations are never reported, making it impossible to distinguish genuine trends from noise. The difficulty binning in Figure 2, which claims a non-monotonic pattern, could be sensitive to which specific 15 or 30 languages were added to the 7-language base set. The paper does not report statistical significance for any pairwise comparison. The GLUE comparison (Table 4) is on the development set — the paper does not submit to the GLUE test server, so overfitting to the dev set cannot be ruled out. The test sets for XNLI (500 test examples per language) and MLQA (varies by language) are relatively small, with the total number of test examples across all 15 XNLI languages being approximately 7,500. Differences of 1–2 percentage points may not be significant at this sample size.

Missing experiments. The paper studies the curse of multilinguality only on XNLI with BERT-Base models, but XLM-R itself is a much larger model (550M parameters). There is no experiment showing how the curse changes at the XLM-R scale — does XLM-R still exhibit performance degradation if trained on 200 languages instead of 100? The paper's inference that vocabulary size was the binding constraint comes from the BERT-Base experiments (Figure 6), but there is no equivalent analysis for the 550M model. The paper also never tests the interaction between language count and training data volume: do 30 languages on CommonCrawl outperform 100 languages on Wikipedia? This is the most natural scaling comparison — trading off language breadth against per-language data depth — and the paper does not design an experiment to measure it.

Where claims hold conditionally. The claim that XLM-R matches monolingual performance holds more strongly for languages with large CommonCrawl corpora (English, German, French) than for those with small corpora (Swahili, Urdu), where the multilingual model still outperforms prior multilingual models but trails the (hypothetical) monolingual model that would exist if enough data were available. The claim that the curse is alleviated by capacity holds for 30 languages but only partially for 100 — the vocabulary bottleneck remains. The claim that TLM is unnecessary holds at the CommonCrawl scale but might not generalize to even larger models or different tasks where cross-lingual alignment is more important than monolingual representation quality.

6. Limitations and Trade-offs

The Curse of Multilinguality Is Alleviated, Not Eliminated

The assumption or constraint. The paper demonstrates that increasing model capacity (wider Transformer layers, larger vocabulary) can offset the performance degradation that occurs when a fixed-capacity model is asked to represent too many languages. However, the experiments show that this offset is partial for 100 languages even at the largest tested scale. The authors are explicit about this residual limitation:

"XLM-100 with increased capacity (1152 hidden size) improves over the fixed-capacity XLM-100 but still falls short of XLM-7, because 'higher vocabulary dilution' — the shared vocabulary of 150K tokens gets spread thinner across 100 languages than across 7." (Section 5.1, Figure 4 discussion)

In plain terms: making the model bigger helps, but at 100 languages with a 150K vocabulary, the model still underperforms a 7-language counterpart. The final XLM-R model uses a 250K vocabulary, which the paper's ablation (Figure 6) suggests would further reduce the gap, but no experiment compares XLM-R at 100 languages against an equivalent-capacity model trained on only 7 languages with the same 250K vocabulary. The claim that the curse is "alleviated" rests on the BERT-Base-scale experiments (Figure 4, Figure 6), not on a direct test with the 550M model.

The consequence. A practitioner aiming to scale to 200, 500, or 1,000 languages cannot extrapolate from this paper's results with confidence. The paper provides qualitative guidance — scale both Transformer capacity and vocabulary size — but no quantitative scaling law that predicts how much capacity is needed for a target number of languages. Without such a law, each new language count requires a new set of expensive ablation experiments. Furthermore, the vocabulary bottleneck interacts with the Transformer bottleneck in ways the paper characterizes but does not fully model. For languages with large character inventories (e.g., Chinese, Japanese, Korean) or languages that are linguistically isolated (sharing few subwords with other languages in the training set), the effective vocabulary dilution may be more severe than for the average case the paper reports. The paper's aggregated results mask this per-language variance in how capacity is allocated.

What evidence exists in the paper. Figure 4 provides the core evidence: at the BERT-Base scale, XLM-100 with increased capacity (hidden size 1152) achieves approximately 71% XNLI accuracy, while XLM-7 (hidden size 768) achieves approximately 74%. The residual gap is ~3 percentage points and is attributed by the authors to vocabulary dilution. The vocabulary-size ablation (Figure 6) shows that increasing vocabulary from 32K to 256K yields a 2.8% gain at fixed Transformer capacity, and that combining vocabulary increases with Transformer capacity increases ("increased capacity" line in Figure 6) yields larger gains. However, the paper never reports the joint effect of maximal vocabulary and maximal Transformer capacity at 100 languages versus 7 languages — the inference that both together would close the gap is interpolation across separate experiments, not a direct measurement.

Mitigation status. The paper partially addresses this through its architectural choices for XLM-R (250K vocabulary, 550M parameters), and the strong overall results (matching monolingual performance on GLUE, outperforming mBERT substantially on all tasks) suggest the residual curse is small enough to be practically acceptable for the 100-language setting. But the paper does not provide a principled solution for generalizing to more languages. The authors acknowledge this implicitly when they state that "future work can reduce this gap even further by alleviating the curse of multilinguality and vocabulary dilution" (Section 5.3), framing it as an open problem rather than a solved one.

---

Difficulty Estimation Cost Is Not Accounted for in the Scaling Analysis

The assumption or constraint. The paper's analysis of how performance scales with language count, data volume, model capacity, and vocabulary size treats each of these dimensions independently in controlled ablation experiments. However, the paper does not account for the fixed cost of building the pretraining corpus — specifically, the cost of language identification, quality filtering, and data cleaning that produces CC-100 from raw CommonCrawl dumps. This cost varies substantially by language: identifying and filtering high-quality text in Swahili (where web data is scarce and language identification models are less accurate) is disproportionately harder than for English. The paper acknowledges using "an internal language identification model in combination with the one from fastText" (Section 3) and "train[ing] language models in each language and us[ing] it to filter documents" — both of which require compute, engineering effort, and for some languages, sufficient seed data to train the filtering models.

The consequence. A practitioner attempting to replicate XLM-R for a new set of languages — particularly languages not covered by existing language identification tools or lacking seed data for quality filtering — will encounter a hidden startup cost that the paper's narrative of "just scale up the data" does not capture. The paper shows that low-resource languages benefit most from CommonCrawl data (Figure 3, Figure 1), but these are precisely the languages where building the clean corpus is most expensive and least reliable. The language identification pipeline requires a seed classifier; the perplexity-based filtering requires a language model for each language — both require some amount of in-language text to bootstrap, creating a chicken-and-egg problem for truly low-resource languages that have almost no web presence at all. Below the paper's implicit threshold of "a few hundred MiB" (Section 5.3), even CommonCrawl may not contain enough data to train the filtering models, let alone the MLM itself.

What evidence exists in the paper. The paper does not report the computational or human effort required to build CC-100. Appendix Table 6 lists per-language corpus sizes, but these are outputs of the pipeline, not inputs. The paper does not specify how many raw CommonCrawl dumps were processed and discarded for each language, what fraction of downloaded text passed the dual language identification check, what fraction survived perplexity filtering, or how these fractions varied by language. Without this information, a practitioner cannot estimate the raw crawl volume needed to achieve a target cleaned corpus size for a new language. The paper also does not ablate the quality of the filtering pipeline — there is no experiment showing how sensitive downstream performance is to the filtering threshold or the language identification accuracy.

Mitigation status. Not addressed. The paper frames CC-100 as an engineering contribution and releases it publicly, which mitigates the replication cost for the 100 languages covered. But for extension to new languages, the paper provides no guidance on the cost-quality tradeoff in corpus construction. The filtering methodology is described primarily by reference to Wenzek et al. (2019), and the paper does not report enough detail to independently assess the pipeline's cost or failure modes. A reader wanting to replicate the approach for, say, the next 100 languages would need to independently solve the language identification and quality filtering problems, with no upper bound on the required effort from this paper.

---

Single Benchmark Family Dominates the Scaling Analysis, and Task Diversity Is Limited

The assumption or constraint. All of the paper's scaling ablations — the experiments that establish the curse of multilinguality, the effect of vocabulary size, the language sampling tradeoff, the capacity scaling interactions, and the Wikipedia-vs-CommonCrawl comparison — are conducted exclusively on XNLI. The paper states this explicitly:

"We conduct most of the analysis on XNLI, which we found to be representative of our findings on other tasks." (Section 5, opening)

The downstream evaluation covers four tasks (XNLI, NER, MLQA, GLUE), but the ablation studies that motivate the model's design choices (Figures 2–7) use only XNLI accuracy as the dependent variable. The paper's central conceptual contribution — the curse of multilinguality and its capacity-dependence — is therefore characterized for a single task type: natural language inference.

The consequence. XNLI is a sentence-pair classification task that tests whether the model can recognize entailment and contradiction relationships. This task heavily depends on the model's ability to represent predicate-argument structure, negation, and logical relationships — aspects of language that may transfer differently across languages than the capabilities tested by NER (token-level sequence labeling requiring entity boundary detection) or QA (span extraction requiring passage understanding). A scaling pattern observed on XNLI may not generalize to other task types. For example, the optimal language sampling parameter α = 0.3 was tuned for XNLI average accuracy; for a practitioner whose primary use case is NER in a specific language family, a different α might be optimal. The paper provides no evidence that the curse of multilinguality follows the same shape (non-monotonicity, capacity-dependence thresholds) for sequence labeling or question answering.

What evidence exists in the paper. The paper does provide downstream results on NER (Table 2) and MLQA (Table 3) that show XLM-R outperforming prior multilingual models, confirming that the full XLM-R model transfers well across task types. However, there is no ablation equivalent to Figure 2 for NER or MLQA — no experiment showing how NER or QA cross-lingual transfer changes as language count increases from 7 to 15 to 30 to 60 to 100. The paper's claim that XNLI is "representative" is an assertion, not a demonstrated fact. The main results on NER and MLQA evaluate only the final XLM-R model, not the intermediate configurations that motivated its design.

The paper also evaluates on only one dataset per task type (XNLI for NLI, CoNLL for NER, MLQA for QA), with no cross-task redundancy. If XNLI overestimates or underestimates cross-lingual transfer relative to other NLI benchmarks, the paper's scaling conclusions could be dataset-specific rather than task-specific.

Mitigation status. Partially addressed. The strong results on three different task families (classification, sequence labeling, QA) for the final XLM-R model suggest that the design choices informed by XNLI ablations did not catastrophically overfit to that task. But the paper does not provide the evidence needed to determine whether the scaling relationships (curse onset, optimal α, vocabulary size sensitivity) would differ for other tasks. A footnote or appendix table showing the 7/15/30/60/100-language scaling for NER accuracy would have strengthened the claim of representativeness substantially.

---

The 14× Larger Pretraining Baseline Is Weak and Not Directly Applicable

The assumption or constraint. The paper's headline claim — "XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks" — rests on comparisons against specific monolingual baselines: BERT-Large, XLNet-Large, and RoBERTa (Table 4), as well as monolingual BERT-Base models trained by the authors for the Table 5 comparison. However, these baselines have important weaknesses relative to XLM-R that make the comparison somewhat asymmetric.

For the GLUE comparison (Table 4): the monolingual models (BERT-Large, XLNet-Large, RoBERTa) were trained on different data (English Wikipedia and/or English books and web text), with different training recipes, different tokenizers, and most importantly, different total pretraining compute. XLM-R was trained for 1.5 million updates on 2+ TB of data across 100 languages, using 500 V100 GPUs. RoBERTa was trained on 160 GB of English text. While XLM-R's per-language data is less than RoBERTa's English data for English specifically, the total compute budget for XLM-R is substantially larger than for any of the monolingual baselines, since XLM-R processes data from 100 languages. The comparison is therefore between a very-large-compute multilingual model and smaller-compute monolingual models — the paper cannot distinguish whether XLM-R's competitive performance comes from multilingual transfer benefits or simply from being trained with more total FLOPs. The 1.0-point gap to RoBERTa could be larger (or reversed) if RoBERTa had been trained with an equivalent total compute budget.

For the Table 5 comparison: the monolingual BERT-Base models were trained by the authors on CommonCrawl data "capped at 60 GiB" per language. The XLM-7 model was trained on CommonCrawl from all 7 languages combined. The total data seen by XLM-7 is therefore roughly 7 × 60 GiB = 420 GiB, versus 60 GiB for each monolingual model — a 7:1 advantage in total training data. The multilingual model's slight underperformance on cross-lingual transfer (76.2% vs. 77.5% monolingual average) is measured against models that saw one-seventh as much data. A compute-matched comparison would give each monolingual model the same total compute as the multilingual model (e.g., by training each monolingual model on 420 GiB of its language's data), but the paper explicitly caps the monolingual data at 60 GiB.

The consequence. The paper's central claim — that a single multilingual model can match monolingual per-language performance — cannot be separated from the confound that the multilingual model was trained with substantially more total compute. A practitioner comparing "one XLM-R vs. N separate RoBERTa models" must account for the fact that RoBERTa's training budget produced the 92.8% GLUE score with far less total compute than XLM-R consumed. If the practitioner's goal is to maximize per-language accuracy, training separate monolingual models with the total compute budget that XLM-R used might yield per-language performance exceeding XLM-R's, undermining the "no sacrifice" claim.

What evidence exists in the paper. The paper reports no compute-matched comparison between multilingual and monolingual models. The GLUE results (Table 4) come with a footnote that the baseline numbers are "from Liu et al. (2019)" — meaning the paper did not re-train the monolingual baselines under comparable conditions. The Table 5 comparison explicitly caps monolingual data at 60 GiB, without justifying this cap relative to the multilingual model's total data budget. The paper's language about these comparisons is carefully qualified ("very competitive with strong monolingual models," "the possibility of multilingual modeling without sacrificing per-language performance"), but the experimental design does not rule out the hypothesis that the competitive performance is a consequence of total compute rather than multilingual transfer.

Mitigation status. Not addressed. The paper does not attempt a compute-matched comparison, does not report the total FLOPs for XLM-R training relative to the monolingual baselines, and does not discuss this confound. The one analysis that touches on efficiency — the observation that XLM-R outperforms XLM-100 (570M parameters) with XLM-R Base (270M parameters) — is about model size, not training compute. A reader evaluating whether to adopt XLM-R's approach must bring their own assumptions about relative training costs.

---

The Model Does Not Handle Code-Switching and Script Variation in Deployment Despite This Being a Motivating Design Choice

The assumption or constraint. The paper justifies the removal of language embeddings by arguing that it "allows our model to better deal with code-switching" (Section 3). Code-switching — the phenomenon where a single utterance mixes multiple languages — is common in multilingual communities and in social media text. The absence of language embeddings means the model must infer language identity from the token sequence, which in principle enables handling of mixed-language input. However, the paper provides no evaluation of code-switching performance. None of the evaluation benchmarks (XNLI, CoNLL, MLQA, GLUE) contain code-switched text. All test examples are monolingual.

Additionally, the paper's training data construction — language identification followed by filtering — actively removes code-switched documents. The pipeline uses dual language identification (internal model + fastText) and only keeps documents where both agree. A document containing both English and Hindi would likely be rejected or assigned to a single language, losing the code-switched portion. The pretraining data is therefore curated to be monolingual, even though the architecture is motivated by code-switching robustness.

The consequence. A practitioner deploying XLM-R in a setting where code-switching is common (e.g., social media content moderation in India, Nigeria, or Southeast Asia; multilingual customer support; conversational AI in multilingual households) has no evidence from the paper about how well the model will perform. The removal of language embeddings — presented as a feature for code-switching — could actually hurt performance if the pretraining data bias toward monolingual text causes the model to treat code-switched tokens as out-of-distribution. The paper provides no guidance on this.

A related concern applies to languages with multiple scripts or non-standard orthography. The paper includes romanized variants of some languages (Hindi Romanized, Urdu Romanized, etc. in Appendix Table 6), but it does not report whether the model successfully transfers between the native-script and romanized versions, or whether the romanized data was sufficient for the model to learn both script variants. For languages where users commonly mix scripts (e.g., Arabic-script Urdu and Roman Urdu in the same message), this is a practical deployment concern the paper does not address.

What evidence exists in the paper. None. The paper's evaluation benchmarks are exclusively monolingual and single-script. The decision to remove language embeddings is justified in a single sentence (Section 3) with no supporting experiment or ablation. Appendix A lists romanized variants as separate entries in the language table, implying they were treated as distinct languages in the training data, but no experiment reports their downstream performance or interaction with the native-script variants.

Mitigation status. Not addressed. The paper does not acknowledge the gap between the code-switching motivation and the monolingual evaluation. The release of the model weights (promised in the abstract) enables external evaluation of code-switching performance, but the paper itself provides no evidence that the design choice achieves its stated goal. A practitioner concerned about code-switching must treat this as an unvalidated feature.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the field of multilingual NLP from a regime where multilingual models were understood as inherently inferior to monolingual ones — a necessary compromise for cross-lingual transfer — to a regime where the performance gap is recognized as an artifact of insufficient data and capacity, not a fundamental tradeoff. The shift is both conceptual and methodological.

Conceptually, the paper introduces the curse of multilinguality as a diagnostic, not a fate. Prior to this work, the dominant assumption was that training one model on many languages inevitably produced worse per-language results than training dedicated monolingual models, and the only question was how large the degradation would be. The paper reframes this: the degradation is not inevitable; it is a function of three controllable variables — model capacity, vocabulary size, and per-language training data volume — and when all three are scaled appropriately, the curse is "alleviated" to the point where a single multilingual model can be competitive with strong monolingual baselines. This is not a paradigm shift in the sense of introducing a new architecture or training objective — XLM-R is a standard Transformer trained with a standard MLM objective — but it is a reframing of the problem from "how do we minimize the multilingual penalty?" to "how do we scale capacity and data to eliminate it?"

The paper also resolves a specific contradiction in the prior literature. Lample and Conneau (2019) had shown that XLM with supervised translation language modeling (TLM) substantially outperformed the unsupervised MLM-only variant, suggesting that cross-lingual supervision was necessary for strong cross-lingual transfer. This paper shows that this conclusion was an artifact of insufficient monolingual data and undertraining: by scaling the CommonCrawl data and training past perplexity convergence, the MLM-only model matches the previously reported MLM+TLM performance (~75% XNLI). The implication is that the benefit of TLM observed in prior work was partly compensating for data scarcity — when each language individually has enough data for the model to learn its structure, the multilingual bottleneck itself forces cross-lingual alignment, and explicit parallel supervision becomes an optimization rather than a requirement. This is significant because it makes the approach applicable to the long tail of languages that lack parallel corpora, which are precisely the languages that most need cross-lingual transfer.

The paper redirects research attention in several concrete ways. Vocabulary size becomes a first-class design dimension rather than a computational nuisance. Prior work treated vocabulary size as a constraint to be minimized (for softmax efficiency) or a fixed hyperparameter copied from monolingual models. The paper's ablation (Figure 6) demonstrating a 2.8% XNLI gain from 32K to 256K vocabulary at fixed total parameters, and more than 3% when capacity is jointly increased, establishes vocabulary design as an independent axis of multilingual model scaling alongside layer count and hidden dimension. This means future work on multilingual models should report and ablate vocabulary size as carefully as model depth — a practice that was not standard before this paper. Data curation for low-resource languages becomes a central research problem. The paper shows that crossing the threshold from "too little data to learn representations" (~Wikipedia scale for Swahili) to "enough data" (~CommonCrawl scale) produces dramatic gains: 10+ points on XNLI for Swahili and Urdu (Section 5.4). This reframes low-resource NLP from a problem of transfer learning to a problem of data acquisition and cleaning at scale. The question shifts from "how do we transfer from English to Swahili?" to "how do we find enough Swahili text on the web to make the model learn Swahili?" — a fundamentally different research agenda.

The paper makes certain research directions less attractive. Translation language modeling as a standard component of multilingual pretraining becomes questionable. If scaled monolingual data alone achieves TLM-level cross-lingual transfer, the marginal benefit of maintaining parallel corpora, preprocessing them, and designing the TLM training objective may not justify the engineering cost and language coverage limitations. Researchers building multilingual models for new language sets can deprioritize parallel data collection and focus on monolingual data curation. Small-vocabulary multilingual models (e.g., 30K–60K tokens) also become harder to justify. The paper's vocabulary ablation shows consistent gains from larger vocabularies up to at least 512K tokens, and the mechanism (reducing forced sharing of semantically unrelated subwords across languages) is well-motivated. A new multilingual model proposing a vocabulary under 100K would need to explain why the computational savings outweigh the demonstrated representational cost.

Follow-Up Research This Work Enables

Scaling laws for multilingual model capacity as a function of language count. The paper establishes that the curse of multilinguality exists and is capacity-dependent, but provides only point estimates at specific (language count, model size) configurations: 7 languages + 768 hidden size, 30 languages + 960 hidden size, 100 languages + 1152 hidden size. What is missing — and what this paper makes newly tractable — is a quantitative scaling law that predicts the required model capacity (parameters, vocabulary size) to achieve a target per-language performance as a function of the number of languages. A strong follow-up would train models at a grid of language counts (7, 15, 30, 60, 100, 150, 200) and model sizes (varying hidden dimension and vocabulary size independently), then fit a power-law relationship analogous to the Chinchilla scaling laws (Hoffmann et al., 2022) but for the multilingual capacity-language tradeoff. The paper's infrastructure (CC-100, the training recipe, the evaluation protocol) provides the starting point; the main additional cost is the grid of training runs. Such a law would let a practitioner answer: "I want to support 200 languages at 80% XNLI accuracy; how many parameters do I need?" without running their own grid search.

Per-task characterization of the curse of multilinguality. The paper's scaling analysis is conducted almost entirely on XNLI. The assumption that XNLI is "representative of our findings on other tasks" (Section 5) is untested. A natural follow-up would replicate the core Figure 2 experiment — varying language count from 7 to 100 at fixed BERT-Base capacity — on NER (CoNLL), QA (MLQA), and perhaps a syntactic task (e.g., universal dependencies parsing) to determine whether the non-monotonic pattern (improvement from 7 to 15 languages, degradation beyond 30) is task-specific. It is plausible that NER, which relies heavily on named entity surface forms and capitalization patterns, exhibits a different positive-transfer curve than NLI, which depends more on predicate-argument structure. If the optimal language count for cross-lingual NER transfer is 30 while the optimal for NLI is 15, a practitioner building a model for both tasks faces a genuine tradeoff that the paper does not surface. This experiment requires no new models — only fine-tuning and evaluating the existing 7/15/30/60/100-language checkpoints on additional tasks.

Does the curse re-emerge at larger model scales? The paper demonstrates the curse at BERT-Base scale but does not test whether it reappears for XLM-R-scale models if language count is pushed beyond 100. A natural stress test: take the XLM-R training recipe (550M parameters, 250K vocabulary, CC-100 data) and extend the language set to 200 or 300 languages by incorporating additional CommonCrawl dumps or other web-crawled corpora. Does XLM-R-200 maintain the same per-language performance as XLM-R-100, or does the curse re-emerge at a higher threshold? If it re-emerges, at what language count, and is it again bottlenecked by vocabulary size or by Transformer capacity? This experiment would establish whether the curse is truly "alleviated" (pushed to a language count beyond practical concern) or merely "shifted" (reappearing whenever you scale languages faster than you scale parameters). The paper's public release of code and models makes this experiment feasible for a group with access to large-scale compute.

Code-switching evaluation and training data augmentation. The paper removes language embeddings to "better deal with code-switching" (Section 3) but provides zero code-switching evaluation and acknowledges that the CC-100 filtering pipeline removes mixed-language documents. A targeted follow-up would construct a code-switching evaluation benchmark (or adapt existing ones, such as the code-switched NER and sentiment datasets for Hindi-English and Spanish-English) and evaluate XLM-R's zero-shot performance. More ambitiously, the follow-up could create a code-switched pretraining dataset by deliberately mixing sentences from different languages (with controlled mixing rates) and measure whether this improves code-switching performance without degrading monolingual performance. The experiment would test whether the no-language-embedding design choice actually delivers its stated benefit, or whether explicit code-switched pretraining data is necessary. A negative result — XLM-R performs poorly on code-switched text despite lacking language embeddings — would be valuable, as it would indicate that the pretraining data distribution (monolingual-only) matters more than the architectural inductive bias.

The interaction between language relatedness and the curse. The paper's language sets are constructed by adding languages in a fixed order, but the degree of positive transfer depends on which languages are added. Swahili (Bantu) benefits from other Bantu languages being in the training set; if the additional languages were all from unrelated families (e.g., adding only Turkic and Sino-Tibetan languages to a base set of Indo-European languages), the positive transfer might be weaker, and the curse might onset earlier. A follow-up experiment would control for language relatedness: start with a fixed base set (e.g., the paper's 7 languages) and add languages in blocks by language family (all Romance, all Germanic, all Slavic, all Bantu, all Turkic) versus by random sampling across families. Measuring the XNLI transfer curve for each block would quantify how much of the positive transfer is attributable to shared family membership versus general cross-lingual alignment. This would inform practical language selection: if adding related languages provides disproportionate benefit, a practitioner targeting a specific low-resource language should prioritize adding its relatives to the pretraining set, even at the expense of broader coverage.

Compute-matched monolingual vs. multilingual comparison. The paper's monolingual baselines are trained with less total data and compute than the multilingual models they are compared against (e.g., XLM-7 sees 420 GiB across 7 languages vs. 60 GiB per monolingual model in Table 5). A rigorous follow-up would perform a compute-matched comparison: train a multilingual model on N languages with a total compute budget of C FLOPs, and separately train N monolingual models each with a budget of C/N FLOPs (so total compute is equal across conditions). Then compare the per-language performance of the multilingual model against the average of the N monolingual models. This would answer the question the paper claims to address but doesn't cleanly test: for a fixed total FLOPs budget, is it better to train one multilingual model or many monolingual ones? The paper's results suggest the multilingual model would win at low N (where positive transfer helps) and lose at high N (where dilution dominates), but the crossover point is unknown. This experiment is feasible now that the paper has established the training recipe and evaluation protocol — it "only" requires running the grid of models.

Practical Applications and Downstream Use Cases

Single-model deployment for international products. A company operating a content moderation, customer support, or search system across many languages currently faces a choice: deploy separate monolingual models per language (high maintenance cost, each model requires its own training and serving infrastructure) or deploy a single multilingual model (simpler operations but historically worse per-language accuracy). XLM-R changes this calculus. For a service covering 15–30 languages, the paper's results show that XLM-R's per-language performance is within ~1% of the best monolingual models on GLUE (91.8% vs. 92.8% for RoBERTa, Table 4) and competitive on XNLI (80.9% cross-lingual transfer, exceeding the translate-test baseline of 77.8% with RoBERTa). The operational savings from maintaining one model instead of N — a single training pipeline, a single set of hyperparameters, a single serving container — compound with language count, while the accuracy cost is minimal. For languages where the company has no labeled data, XLM-R provides zero-shot cross-lingual transfer that outperforms the translate-test approach (Table 1: 80.9% zero-shot vs. 77.8% translate-test with RoBERTa), meaning the multilingual model is preferable even if the alternative is the best monolingual English model plus machine translation.

Low-resource language NLP through data curation rather than transfer engineering. The paper demonstrates that Swahili and Urdu performance improves by 10+ points on XNLI when pretraining data is scaled from Wikipedia to CommonCrawl (Figure 3, Section 5.4). For an organization working on NLP for under-resourced languages — a government digitizing services in regional languages, an NGO building information access tools, a research lab studying linguistic diversity — the paper suggests a concrete strategy: invest in web data collection and cleaning for the target language rather than in complex transfer learning architectures. The threshold is identifiable: "a few hundred MiB of text data" (Section 5.3) is the minimum for representation learning; above that, the model genuinely learns the language; below that, it relies on cross-lingual transfer. The CC-100 corpus construction methodology (dual language identification, perplexity-based filtering) provides a replicable recipe. A practitioner targeting a language not in the paper's 100 — say, Wolof or Quechua — can follow the same pipeline (download CommonCrawl, filter by language ID, train a language model to remove noise) and have reasonable confidence that if they can assemble a few hundred MiB of clean text, an XLM-R-style model will learn meaningful representations.

Data generation for multilingual self-training and distillation. XLM-R's strong cross-lingual transfer performance (80.9% XNLI zero-shot) means it can serve as a high-quality label generator for languages where no labeled data exists. A team building an NLI system for 50 languages could fine-tune XLM-R on English MultiNLI, then use it to pseudo-label unlabeled premise-hypothesis pairs in the other 49 languages (sourced from, e.g., news articles, Wikipedia, or translated MultiNLI). These pseudo-labels can then be used to train smaller, faster, task-specific models per language via distillation. The paper shows that XLM-R's zero-shot accuracy is already competitive with the translate-train approach (80.9% zero-shot vs. ~77.8% translate-train with BERT), meaning the pseudo-labels have quality comparable to machine-translated training data — but without requiring the N-way translation pipeline. This approach is particularly valuable for the low-resource languages where XLM-R shows the largest relative gains over prior models (e.g., 23.5% improvement over mBERT on Swahili). A downstream team can get strong NLI performance in Swahili without a single human-translated or machine-translated Swahili NLI example, purely by leveraging XLM-R's English-to-Swahili cross-lingual transfer.