ArXiv: 1910.07475

🎯 Pitch

Even state-of-the-art multilingual models suffer a nearly 40% exact match drop when transferring QA from English to languages like Arabic or Hindi. This paper introduces MLQA, a seven-language benchmark deliberately built to measure that gap by providing parallel questions and contexts through aligned Wikipedia sentences.

1. Executive Summary

This paper introduces MLQA, a multi-way aligned extractive QA evaluation benchmark spanning seven languages — English, Arabic, German, Spanish, Hindi, Vietnamese, and Simplified Chinese — constructed by mining parallel sentences from Wikipedia, crowdsourcing English questions constrained to answer spans within those aligned sentences, and then professionally translating questions and annotating answer spans into all target languages. The work defines two cross-lingual tasks: cross-lingual transfer (XLT — training on English SQuAD data and evaluating zero-shot on target-language contexts and questions) and generalized cross-lingual transfer (G-XLT — evaluating with mismatched question and context languages, e.g., Hindi questions on Arabic contexts, enabled by MLQA's highly parallel structure). On the XLT task, the best zero-shot model — XLM (a multilingual Transformer pretrained with masked language modeling and translation language modeling) — achieves a mean F1 of 59.3 across target languages, but this represents a 39.8% drop in mean Exact Match relative to the English BERT-large baseline on English data, establishing that cross-lingual transfer remains far behind training-language performance even with state-of-the-art multilingual representations.

2. Context and Motivation

The Core Problem: We Cannot Measure Cross-lingual QA Progress Without Cross-lingual QA Benchmarks

The fundamental problem this paper addresses is both simple and systemic: high-quality extractive question answering datasets exist almost exclusively in English, making it impossible to either measure progress in multilingual QA or train models that work across languages. The field has developed enormously in English extractive QA — SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2017), and Natural Questions (Kwiatkowski et al., 2019) collectively provide hundreds of thousands of annotated instances — but this explosion of resources has not crossed language boundaries. The result is a sharp asymmetry: researchers can build and benchmark increasingly sophisticated QA models for English, but cannot answer the basic question of whether those same techniques work in German, Arabic, or Hindi with any rigor.

This gap matters for two distinct reasons that the paper articulates clearly (Section 1). First, it is a measurement problem: without relevant benchmark data in multiple languages, we literally cannot tell whether we are making progress on multilingual QA. Second, it is a training problem: recent advances in extractive QA have been overwhelmingly in the fully supervised paradigm. If you want to build a QA system for Vietnamese and no training data exists, you face a chicken-and-egg situation — you cannot train a model, and you cannot evaluate whether your workaround (e.g., cross-lingual transfer) actually works.

The paper takes a clear position on which part of this problem is most urgent to address: multilingual evaluation data is the non-negotiable prerequisite. The authors argue that "while multilingual QA training data might be useful but not strictly necessary, multilingual evaluation data is a must-have" (Section 1). The reasoning is that cross-lingual representation learning — multilingual BERT, XLM, and similar approaches — has shown promising transfer results on classification tasks like NLI (Conneau et al., 2018) and document classification (Schwenk and Li, 2018), suggesting that zero-shot transfer to QA might be possible without target-language training data. But evaluating whether this actually works requires a multilingual QA benchmark to test against. The paper therefore positions itself not as providing a training resource, but as providing the measurement infrastructure that the field needs to begin making progress on cross-lingual QA at all.

The Real-World Stakes: Language Barriers Create Access Barriers

The implications of this gap extend beyond academic benchmarking. QA technology is the backbone of many practical information access systems — search engines, voice assistants, customer support automation — and the inability to extend these systems beyond English means large populations are excluded from their benefits. The paper does not belabor this point with extensive data, but the selection of seven languages spanning three scripts (Latin, Arabic, Devanagari), three language families (Indo-European, Afroasiatic, Austroasiatic), and varying levels of resource availability makes clear that the ambition is global coverage, not incremental expansion to a few typologically similar European languages.

There is also a subtler equity concern that motivates the paper's emphasis on parallel, comparable evaluation. When datasets exist in only one language, the "state of the art" is implicitly defined relative to English performance. Models optimized on English data with English-centric architectures may perform well on English benchmarks while being terrible in other languages, but without evaluation data, this failure mode is invisible. MLQA's multi-way parallel design — where the same question-concept appears in up to four languages with aligned answer spans — is explicitly intended to enable fair comparison across languages (Section 2, desideratum "Parallel"). You can ask whether a model performs equally well on a question in Spanish as it does on the same question in English, which is impossible with independently constructed monolingual datasets where differences in dataset difficulty confound any language comparison.

Prior Work: Fragmented, Small-Scale, and Not Truly Parallel

The paper's related work section (Section 3) surveys the landscape and identifies why existing resources fail to meet the need. The problems fall into several categories:

Language coverage is narrow. Several prior efforts to build cross-lingual QA data exist, but are limited to two languages: Gupta et al. (2018) release English-Hindi data, Hardalov et al. (2019) study English-Bulgarian transfer, Liu et al. (2019b) provide Chinese-English cloze QA, and Jing et al. (2019) build BiPar from parallel English-Chinese novel paragraphs. These are valuable contributions but cover only isolated language pairs, not the multi-way parallel structure that enables systematic cross-lingual comparison and the G-XLT task.

Scale is insufficient for rigorous benchmarking. Asai et al. (2018) investigate extractive QA on a manually translated set of only 327 SQuAD instances in Japanese and French. For context, MLQA provides over 5,000 instances per target language — more than an order of magnitude larger. A test set of 327 instances is too small for meaningful error analysis (stratifying by question type or difficulty produces subsets too small for reliable statistics), and risks overfitting during model development if the test set is used repeatedly.

Datasets lack parallel structure across languages. Liu et al. (2019a) construct XQA, a cross-lingual open-domain QA dataset from Wikipedia "Did you know?" questions covering nine languages. While broad in language coverage, XQA is distantly supervised — questions and answers are automatically extracted, not human-annotated — which introduces noise and makes it hard to distinguish model errors from dataset errors. Critically for cross-lingual fairness, "instances are not parallel, and answer distributions vary by language, making quantitative comparisons across languages challenging" (Section 3). If one language's subset contains systematically harder questions or longer answers, comparing F1 scores across languages is misleading.

Documents are manually translated rather than naturally occurring. Asai et al. (2018) manually translate SQuAD contexts, and contemporaneous work by Artetxe et al. (2019) introduces XQuAD — 1,190 SQuAD instances manually translated into 10 languages. Manual translation produces parallel evaluation data with guaranteed answerability, but the resulting documents are translationese — they contain artifacts of the human translation process that do not reflect how text naturally occurs in the target language. A model trained on English SQuAD patterns may perform artificially well on translated contexts because syntactic structures and information ordering are more similar to English than natural target-language text would be. MLQA's approach — mining Wikipedia articles written independently in each language and identifying naturally parallel sentences — avoids this domain shift. The paper notes that Wikipedia articles are "often translated from English to the target languages by volunteers" (Section 6), but these are still edited and adapted by native speakers rather than being professional translations of a fixed source.

Annotation protocols are inconsistent across languages. Most existing datasets in different languages were "created at different times by different authors with different annotation setups" (Section 2, desideratum "Extractive QA"). This makes cross-language analysis problematic because any observed performance difference could stem from differences in dataset difficulty, annotation guidelines, or answer selection criteria rather than genuine differences in language difficulty. The paper's desideratum of a consistent annotation protocol applied uniformly across all languages is not met by any existing resource.

Where MLQA Fits: Purpose-Built to Fill a Specific Gap

The paper positions MLQA as directly addressing the shortcomings of the prior landscape by satisfying a specific set of design criteria (Section 2, the five desiderata). Rather than being a general-purpose QA dataset, MLQA is a purpose-built evaluation benchmark optimized for the specific research question: how well do QA models transfer across languages, and how does this vary by language pair and question type?

The five desiderata reveal the paper's theory of what makes a good cross-lingual evaluation benchmark:

  • Parallel: Every instance exists in multiple languages with aligned questions and answers, enabling controlled comparison and the G-XLT task. This is the most distinctive feature relative to prior work, and the annotation pipeline is designed around this requirement — questions are generated from a sentence identified as parallel across languages, ensuring the question is answerable in all target languages without requiring the entire context document to be parallel.

  • Natural Documents: Using existing Wikipedia articles rather than manual document translation ensures the evaluation data reflects real target-language text characteristics, avoiding translationese artifacts.

  • Diverse Languages: The seven chosen languages span a deliberately broad range — three scripts, multiple language families, varying degrees of morphological complexity, and different levels of resource availability in NLP. This is not a convenience sample; the paper explicitly chooses languages that "represent a broad range of linguistic phenomena" (Section 2).

  • Extractive QA: The span-extraction paradigm (given context and question, identify the answer substring) is the de facto standard in QA research, with well-established modeling approaches, evaluation metrics, and training datasets. Adopting this paradigm maximizes compatibility with existing research infrastructure — models trained on SQuAD can be directly evaluated on MLQA without architectural changes.

  • Textual Domain: Choosing Wikipedia as the domain provides two practical advantages. First, Wikipedia is naturally multi-parallel — the same topics are covered in many languages, making parallel sentence mining possible. Second, Wikipedia is the domain of SQuAD, the most widely used extractive QA training dataset, meaning models trained on SQuAD are in-domain for MLQA evaluation, isolating the performance difference to cross-lingual transfer rather than domain shift.

A Note on What MLQA Is Not

Understanding the paper's positioning requires recognizing what the authors explicitly choose not to do. MLQA does not provide training data — it is an evaluation-only benchmark (development and test splits only). The authors adopt SQuAD v1.1 as training data, following the standard zero-shot transfer paradigm established by cross-lingual representation models like multilingual BERT and XLM. This is a deliberate choice consistent with the paper's thesis that evaluation data is the bottleneck, not training data. The paper also does not propose new modeling techniques — the baseline experiments use existing models (multilingual BERT, XLM, machine translation pipelines) to establish performance floors and demonstrate the magnitude of the remaining gap. The contribution is the benchmark itself (the dataset and the task definitions), not a method for improving cross-lingual performance.

The paper also explicitly acknowledges that MLQA answers are constrained to come from a specific aligned sentence, which "may be tempted to only read the parallel sentence and write questions which only require a single sentence of context to answer" (Section 6). This is a potential limitation — if all questions can be answered from a single sentence, the dataset does not test multi-sentence reasoning — but the authors argue this is comparable to SQuAD (where single-sentence questions are common, as documented by Sugawara et al., 2018) and that MLQA-en scores being similar to SQuAD scores supports comparable difficulty. This acknowledgment is important: the paper is transparent that its annotation pipeline trades off some context-dependence for guaranteed cross-lingual answerability.

The Meta-Scientific Motivation

Beyond the specific technical gap, there is an implicit motivation that runs through the paper: benchmarks accelerate research. The abstract explicitly states that MLQA is "intended to spur research in this area," and the introduction draws a direct parallel to SQuAD's role in catalyzing English QA progress. This theory of scientific progress — that the availability of a standardized, challenging, well-constructed evaluation dataset focuses community effort and enables quantitative comparison of approaches — is the underlying rationale for the entire project. The paper is not just filling a data gap; it is attempting to create the conditions under which cross-lingual QA can become a well-defined, measurable research problem with clear progress indicators, replicating the pattern that drove rapid advances in English QA after SQuAD's release.

3. Technical Approach

3.1 Reader Orientation

This paper presents a data construction pipeline and evaluation benchmark, not a machine learning model. The system being built is a large-scale, multi-way parallel extractive question answering dataset in seven languages, designed to enable rigorous measurement of how well QA models transfer across languages. The problem it solves is the absence of high-quality, comparable, multi-lingual evaluation data for extractive QA — it provides the measurement infrastructure that the field needs to quantify cross-lingual transfer performance, define standardized tasks, and compare methods fairly across languages. The "shape" of the solution is an annotation pipeline with three sequential stages: (1) automatic identification of parallel sentences across Wikipedia languages using multilingual sentence embeddings, (2) crowdsourced English question-answer annotation constrained to those parallel sentences to guarantee cross-lingual answerability, and (3) professional translation of questions and annotation of answer spans in each target language, producing a dataset where each question-concept exists in up to four languages with aligned answers, enabling both traditional cross-lingual transfer evaluation and a novel generalized transfer setting.

3.2 Big-Picture Architecture (Diagram in Words)

The MLQA construction system has three major components connected in a linear pipeline:

  1. Parallel Sentence Miner: Takes Wikipedia dumps in seven languages as input. Uses the LASER toolkit to embed sentences from articles on the same topic into a shared multilingual embedding space, then identifies sentences that are similar enough to be considered translations or close paraphrases. Intersects pairwise alignments with English to produce sets of sentences that are parallel across multiple languages simultaneously (4-way parallel for the final dataset). Outputs: parallel sentence sets with their surrounding paragraph contexts in each language.

  2. English QA Annotation (Crowdsourcing): Takes English parallel sentences and their paragraph contexts as input. Crowd workers on Amazon Mechanical Turk read each context, formulate a question, and highlight the shortest answer span — constrained to lie within the aligned parallel sentence to ensure the question will be answerable in target languages. Multiple workers re-annotate answers to enable quality filtering and inter-annotator agreement calculation. A representative gold answer is selected from the multiple annotations. Outputs: English questions and answer spans, filtered for quality.

  3. Target Language QA Annotation (Professional Translation): Takes English questions, English answers, and target-language contexts (containing the aligned parallel sentence) as input. Professional translators on the One Hour Translation platform translate the English question into each target language and annotate the answer span in the target-language context, using the English answer as guidance to maintain consistency. Instances where no answer exists in the target language are flagged and released separately. Outputs: the complete MLQA corpus — questions, contexts, and answer spans in all seven languages, with known parallelism relationships between instances.

Information flows strictly forward: Wikipedia articles → parallel sentences + contexts → English QA pairs → translated and answer-annotated multilingual QA pairs. The parallelism constraints are enforced at every stage: the parallel sentence defines what can be annotated, the answer-span constraint ensures cross-lingual answerability, and the translation process preserves question meaning across languages.

3.3 Roadmap for the Deep Dive

  • First, the five desiderata and how they constrain every design decision — understanding why the pipeline is built this way is essential before understanding how it works.
  • Second, the parallel sentence mining process (Stage 1) — this is the foundational step that determines which documents become part of MLQA and establishes the parallelism relationships. The LASER toolkit, the embedding-based similarity criterion, and the incremental intersection with English all need detailed explanation.
  • Third, the English QA annotation pipeline (Stage 2) — the crowdsourcing protocol, the answer-span constraint (and why it is the key to cross-lingual answerability), the quality control mechanisms, and the answer selection procedure.
  • Fourth, the target language annotation pipeline (Stage 3) — the professional translation workflow, how the English answer is used to guide target-language annotation, quality assurance for translations, and handling of unanswerable instances.
  • Fifth, the evaluation framework — how the dataset is split into development and test sets, the two task definitions (XLT and G-XLT), the evaluation metrics and their multilingual adaptations, and the baseline model configurations.
  • Sixth, the specific design choices and their justifications — why 4-way parallelism (not 7-way, not 2-way), why Wikipedia (not manual translation), why the answer-in-parallel-sentence constraint (not free-form answer selection), and why SQuAD as training data (not a custom training set).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset construction and benchmark design paper whose core idea is that by constraining English question annotation to answer spans within automatically identified parallel sentences, you can guarantee that questions are answerable in multiple target languages without requiring parallel documents, manual document translation, or target-language training data.

The Five Desiderata as Design Constraints

The entire MLQA construction pipeline is reverse-engineered from five explicit design goals stated in Section 2. Understanding these constraints is essential because every annotation decision, every filtering step, and every tradeoff in the paper traces back to one or more of these desiderata. They are not just aspirational goals — they are operational constraints that ruled out alternative approaches.

Desideratum 1: Parallel. The dataset must consist of instances that are parallel across many languages. This is not just about having the same information content — parallelism means that the same question can be asked about the same fact in multiple languages, with answers that refer to the same real-world entity or concept. The paper identifies three distinct benefits of this property. First, it makes cross-language performance comparison fairer: if model A performs at 60 F1 on Spanish and 50 F1 on Hindi, you can attribute the difference to language difficulty rather than to the Spanish subset containing easier questions. Second, it enables additional evaluation setups that would be impossible with independent datasets — specifically, the G-XLT task where questions in one language are paired with contexts in another, testing whether multilingual models can bridge languages that were never paired during training. Third, it reduces annotation cost: a single English question can be reused across multiple target languages rather than requiring independent question creation in each language. The parallelism requirement is what drove the decision to use parallel sentence mining rather than independent annotation in each language, and it is the justification for the answer-in-parallel-sentence constraint — if questions could refer to any part of the English context, there would be no guarantee that the target-language context contains the relevant information.

Desideratum 2: Natural Documents. The evaluation data should use documents written naturally in each language, not translations of English documents. This matters because machine translation or human translation produces translationese — text that exhibits lexical, syntactic, and information-structuring patterns from the source language that differ from naturally produced text in the target language. A model that performs well on translated test data might be exploiting English-like patterns that do not generalize to real target-language text, giving a misleadingly optimistic picture of cross-lingual transfer. The paper explicitly notes that "Exploiting existing naturally-parallel documents is advantageous, providing high-quality documents without requiring manual translation" (Section 2). However, the paper also acknowledges a nuance in Section 6: Wikipedia articles are "often translated from English to the target languages by volunteers," so the "natural documents" property is approximate — the pipeline acts "as an efficient mechanism of sourcing existing human translations, rather than sourcing independently-written content on the same topic." The documents are still edited and adapted by native speakers rather than being professional translations, mitigating but not eliminating translationese.

Desideratum 3: Diverse Languages. The seven chosen languages should "represent a broad range of linguistic phenomena and have sufficiently large Wikipedia" (Section 2). The selection includes:

  • Three scripts: Latin (English, Spanish, German, Vietnamese), Arabic (Arabic), and Devanagari (Hindi), and additionally Simplified Chinese characters
  • At least three language families: Indo-European (English, Spanish, German, Hindi), Afroasiatic (Arabic), and Austroasiatic (Vietnamese) — Chinese is Sino-Tibetan
  • Varying NLP resource levels: English and Chinese are extremely high-resource; Spanish, German, and Arabic are relatively high-resource; Hindi and Vietnamese are lower-resource for QA specifically
  • Different morphological types: relatively isolating (English, Vietnamese, Chinese), moderately inflectional (Spanish, German), and more inflectional (Arabic, Hindi)

This diversity is deliberate — it ensures that a model performing well on MLQA cannot succeed by exploiting properties specific to European languages or Latin-script languages, and that the benchmark reveals differential performance across typologically distinct languages.

Desideratum 4: Extractive QA. The task must follow the span-extraction paradigm established by SQuAD: given a context paragraph and a question, identify the contiguous substring of the context that answers the question. This matters for three practical reasons. First, it allows models trained on SQuAD (the largest extractive QA training resource) to be directly evaluated on MLQA without architectural modification — the input/output format is identical. Second, it enables use of established evaluation metrics (Exact Match and token-level F1) without needing to design new metrics for a new task format. Third, it "represents a different language understanding challenge" than classification-based cross-lingual benchmarks like XNLI (Conneau et al., 2018) — span extraction requires fine-grained token-level decisions rather than whole-sentence classification, testing different aspects of multilingual representation quality.

Desideratum 5: Textual Domain. The domain should be Wikipedia, specifically because (a) Wikipedia exists in many languages with overlapping topic coverage, making parallel sentence mining feasible, and (b) existing QA training datasets like SQuAD are built from Wikipedia, meaning models trained on SQuAD will be in-domain for MLQA evaluation. This isolates the performance difference to cross-lingual transfer rather than domain shift — if models performed poorly on MLQA-en relative to SQuAD, that would indicate a domain mismatch that would confound the interpretation of target-language results. The paper verifies in Section 5.3 that MLQA-en performance is comparable to adjusted SQuAD performance, confirming that the domain-shift confound is small.

With these five constraints established, the pipeline becomes a constrained optimization problem: how do you produce a large-scale, parallel, extractive QA dataset with natural documents across diverse languages, while minimizing annotation cost and ensuring cross-lingual answerability? The following subsections explain each stage of the pipeline that solves this problem.

Stage 1: Parallel Sentence Mining Across Wikipedia Languages

The first stage of the pipeline identifies sentences that express the same or very similar meaning across multiple language editions of Wikipedia. This is the foundational step — it determines which paragraphs will be used for annotation, establishes the parallelism relationships that enable the G-XLT task, and guarantees that questions answerable in English will (in most cases) also be answerable in target languages because the information is contained in the aligned sentence.

Input and Output. The input is the full Wikipedia dumps for all seven languages. The output is a set of paragraphs (each containing an aligned parallel sentence) for each language, with known multi-way parallelism relationships — for each paragraph in English, we know which paragraphs in which other languages contain the parallel sentence and can be used for target-language annotation.

Why not just align entire articles? Wikipedia articles on the same topic often differ substantially in structure, coverage, and detail across languages — an article about a historical event in English might have sections not present in the Arabic version, or might order information differently. Aligning at the article level would not guarantee that individual sentences are parallel, meaning a question answerable from the English article might not be answerable from the German one. Aligning at the sentence level provides a fine-grained guarantee of information overlap.

The LASER Toolkit. The core technology used for parallel sentence identification is LASER (Language-Agnostic SEntence Representations), described in Artetxe and Schwenk (2018, 2019) and available as an open-source toolkit. LASER produces multilingual sentence embeddings — fixed-length vector representations of sentences where semantically similar sentences from different languages map to nearby points in the same vector space, even if they share no lexical overlap. This is achieved through a sequence-to-sequence architecture trained on massively multilingual translation data: an encoder (shared across all languages) maps input sentences to a language-agnostic representation, and a decoder (language-specific) reconstructs the sentence in the target language. The key property is that the encoder learns to abstract away from surface-form differences and represent meaning in a language-independent way.

Mining procedure specifics. The procedure, described in Section 2.1 and Appendix A.6, works as follows:

  1. Encode all sentences: For each language, every sentence from every Wikipedia article (after basic preprocessing like sentence splitting) is passed through the LASER encoder, producing a vector in the shared embedding space.

  2. Restrict to same-topic articles: Rather than comparing all sentence pairs across languages (which would be computationally prohibitive at Wikipedia scale), the search is restricted to articles on the same topic. This is determined by Wikipedia's inter-language links — for a given English article, only the sentences from the corresponding article in each target language are considered as candidates.

  3. Compute similarity: For each candidate sentence pair (an English sentence and a target-language sentence from the same-topic article), a similarity score is computed in the LASER embedding space. The paper uses a "distance or margin criterion" — specifically, the margin-based scoring from Artetxe and Schwenk (2019), where a sentence pair is considered parallel if the cosine similarity between their embeddings is above a language-pair-specific threshold calibrated to maximize F1 on a held-out parallel corpus.

  4. Intersect pairwise alignments to get N-way parallel sentences: The paper first independently aligns each target language with English, producing pairwise parallel sentence sets (English-German, English-Spanish, etc.). Then these pairwise sets are intersected: a sentence is included in the N-way parallel set only if it has a parallel counterpart in all N languages. This approach is computationally efficient because it requires only N-1 pairwise alignment operations rather than a combinatorial explosion of N-way comparisons.

The scale of mined sentences. Table 1 in the main text and Table 10 in Appendix A.6 show how the number of parallel sentences decreases as more languages are intersected. Starting with millions of pairwise parallel sentences (e.g., 5.4M English-German, 1.1M English-Spanish), the intersection for 7-way parallelism yields only 1,340 sentences across the entirety of Wikipedia. This is insufficient for constructing a dataset of thousands of questions, and the authors further note that "7-way parallel sentences lack linguistic diversity, and often appear in the first sentence or paragraph of articles" — these are formulaic introductory sentences that would produce repetitive, low-diversity questions.

The 4-way compromise. As a compromise between language parallelism and data quantity/diversity, the paper selects sentences that are 4-way parallel, yielding 385,396 parallel sentences. This provides ample material for annotation while maintaining substantial multi-lingual parallelism. The 4-way parallel sentences are then sub-sampled to ensure:

  • Each language combination is equally represented (English + 3 of the 6 target languages appears equally often across combinations)
  • Parallel sentences are evenly distributed in paragraphs (not clustered at the beginning of articles)
  • Each language has many QA instances in common with every other language, enabling comprehensive G-XLT evaluation

What "4-way parallel" means in the final dataset. Except for instances rejected later in the pipeline due to quality issues or unanswerability, each QA instance is parallel between English and three target languages. For example, an instance might exist in English, German, Arabic, and Vietnamese — the same question-concept with aligned contexts and answer spans in all four languages. Table 3 shows the actual parallelism counts: the diagonal represents total instances per language, and off-diagonal cells show how many instances are shared between each target-language pair (always including English).

Paragraph extraction. For each aligned parallel sentence, the surrounding paragraph in each language is extracted to serve as the context for QA annotation. The parallel sentence is embedded within this paragraph — it is not the entire paragraph. This means the context surrounding the aligned sentence can differ across languages (different preceding and following sentences), but the core information needed to answer the question (the parallel sentence) is guaranteed to be present. This design satisfies the "Natural Documents" desideratum because the full paragraphs are naturally written, even if the aligned sentence itself may have been translated at some point.

A subtle technical point about LASER's role. LASER identifies sentences that are semantically similar enough to be considered parallel, but this does not guarantee they are exact translations. The automatically aligned sentences "occasionally differ in a named entity or information content" (Section 1, footnote 2), which is why a small percentage of instances end up unanswerable in target languages (6.6% for Vietnamese, 21.9% for German). The pipeline tolerates this imperfection because the alternative — requiring perfect parallelism — would drastically reduce the amount of usable data. The quality control steps in later stages catch and filter the worst cases.

The diversity cost of parallelism. There is a fundamental tradeoff that the 7-way vs. 4-way comparison makes concrete: stronger parallelism guarantees (more languages sharing each instance) come at the cost of reduced dataset size and diversity. The 4-way choice represents a practical optimum for this set of languages — it produces enough data for robust benchmarking (5K+ instances per language) while maintaining enough parallelism to enable G-XLT evaluation across all language pairs. For a dataset covering more languages, the optimal N-way value would be even smaller, because the probability of finding a sentence that is parallel across K languages decreases exponentially with K.

Stage 2: English Question-Answer Annotation

The second stage of the pipeline produces the English QA instances that will serve as the source for target-language translation. This stage uses Amazon Mechanical Turk crowd workers and follows a protocol broadly modeled on SQuAD (Rajpurkar et al., 2016), but with one critical difference: answers must be subspans of the aligned parallel sentence, not the full paragraph. This constraint is the key mechanism that ensures cross-lingual answerability.

Input and Output. The input to this stage is the set of English aligned sentences (the parallel sentences identified in Stage 1) along with their surrounding paragraphs. The output is a set of English questions and answer spans, quality-filtered and with a representative gold answer selected from multiple annotator judgments.

Annotation interface and workflow. Workers are presented with a paragraph where one sentence is displayed in bold — this is the aligned parallel sentence ($b_{en}$ in the notation of Figure 1). The surrounding context ($c_{en}$) is also shown but is not bold. Workers must:

  1. Read the full paragraph including the bold sentence
  2. Formulate a question ($q_{en}$) that can be answered from the paragraph
  3. Highlight the shortest answer span ($a_{en}$) within the bold sentence that answers the question

The interface enforces that answers must come from the bold sentence — the interface "will not allow answers to be selected from other sentences" (Appendix A.1, Figure 6 instructions). This enforcement is a technical feature of the annotation interface, not just a guideline.

The critical constraint: why answers come from the parallel sentence. This constraint is the linchpin of the entire cross-lingual pipeline. The reasoning is: if the answer can be found within the English parallel sentence ($b_{en}$), and that sentence has a semantically equivalent counterpart in the target language ($b_x$), then the answer should also be locatable within the target-language parallel sentence. This holds because the parallel sentences were identified as expressing the same or very similar meaning — the information needed to answer the question is guaranteed to be present in the target-language version. If workers were allowed to select answers from any part of the English paragraph, there would be no guarantee that the corresponding information exists in the target-language paragraph (since only the aligned sentence is guaranteed parallel, not the entire paragraph).

Question quality guidelines. Annotators receive detailed instructions (reproduced in Figure 6, Appendix A.1) that specify:

  • Avoid using the same words or phrases as in the paragraph (to prevent trivial string-matching questions)
  • Be specific and avoid pronouns (to ensure questions are self-contained)
  • Do not ask fill-in-the-blank style questions
  • Select the shortest answer span (not the entire bold sentence)
  • Spend approximately 6 minutes total to ask one question on each of five paragraphs

There is a "No Question Possible" button for cases where "there is no question you could possibly ask" — workers are encouraged to try hard before using this option.

Quality control: manual review and auto-approval. The first 15 questions from each worker are manually checked for quality. After this initial screening, workers "are contacted with feedback, or their work is auto-approved" — good workers are trusted, poor workers are given feedback or removed. This is a pragmatic quality control mechanism that balances annotation cost against quality assurance: full manual review of all annotations would be prohibitively expensive, but initial screening catches the worst workers early.

Answer re-annotation for quality and IAA. After initial question-answer pairs are collected, a second task is run to collect additional answer annotations. Workers are presented with the English question ($q_{en}$) and the full context ($c_{en}$) — but not the original answer — and asked to either highlight an answer span or indicate that the question is unanswerable. Two additional answer span annotations are collected per question. This produces three answer annotations total: the original from the question-writing task and two from the re-annotation task.

Why collect multiple answer annotations? Three reasons. First, it enables inter-annotator agreement (IAA) calculation: the mean token F1 score between the three answer annotations is computed, giving a measure of answer consistency. The reported IAA score is 82%, comparable to the SQuAD v1.1 development set at 84%. This validates that questions are sufficiently well-specified that annotators converge on the same answer. Second, it provides a quality filter: instances with very low IAA (< 0.3) are discarded — these are questions where annotators fundamentally disagreed about what constitutes the correct answer, indicating a poorly specified question. Third, it enables selecting a representative gold answer rather than using the original answer directly.

Gold answer selection. In 88% of cases, either two or three of the answer annotations exactly match. In these cases, the majority answer is selected as the gold reference. In the remaining 12% of cases where there is no majority, the answer with the highest mean token F1 overlap with the other two is chosen. This procedure ensures the English answers are high-quality and also makes the English evaluation fair relative to target languages, where only one answer annotation exists per question (SQuAD typically uses multiple reference answers, which inflates F1 scores relative to single-reference evaluation).

Filtering pipeline. After annotation, several filters are applied to remove low-quality instances:

  1. Unanswerable removal: Instances where re-annotators marked questions as unanswerable are discarded — if someone reading the same context cannot find the answer, the question is too ambiguous.

  2. Trivial question filter: Instances where "over 50% of the question appeared as a subsequence of the aligned sentence" are removed. These are cases where the worker essentially copied a phrase from the context and turned it into a question (e.g., context says "The Angles settled during the Early Middle Ages" and the question is "During what period did the Angles settle in the Early Middle Ages?"), making the answer trivially extractable via string matching.

  3. Low IAA filter: Instances with mean token F1 < 0.3 between the three answer annotations are removed. A manual analysis of these discarded questions revealed: 38% were poorly specified, 24% didn't make sense or had no answer, 30% had poor answers, and only 8% were high-quality challenging questions. This validates that the IAA filter is removing genuinely poor data rather than just hard questions.

Instruction quality and feedback loop. Appendix A.1 includes screenshots of both the annotation interface (Figure 5) and the expandable instructions (Figure 6). The instructions include an example paragraph, example question, and example answer to calibrate worker expectations. The emphasis on "hard questions" and "avoid using the same words" is designed to produce questions that require genuine reading comprehension rather than simple keyword matching — a known issue with crowdsourced QA data where workers can be lazy and produce trivial questions if not properly incentivized.

Scale of the English annotation. The final MLQA corpus contains 12,738 English instances (11,590 test, 1,148 development). With three answer annotations per instance, this represents approximately 38,000 individual annotation tasks, plus the initial question-writing tasks — a substantial crowdsourcing effort.

Stage 3: Target Language Question Translation and Answer Annotation

The third stage adapts the English QA instances to each target language through professional translation and answer annotation. This stage uses the One Hour Translation platform rather than crowd workers, reflecting the need for higher quality and language expertise than typical crowdsourcing can provide.

Input and Output. The input is: (1) the English question ($q_{en}$), (2) the English answer span ($a_{en}$), and (3) the target-language context paragraph ($c_x$) containing the aligned parallel sentence ($b_x$). The output is: the translated question ($q_x$) in the target language and the answer span ($a_x$) annotated in the target-language context.

Professional translator workflow. Each translator is presented with the English source materials and the target-language context. They perform two tasks:

  1. Translate the question from English into the target language, preserving meaning, tone, and specificity
  2. Annotate the answer span in the target-language context — finding the span of text in $c_x$ that answers $q_x$, corresponding to the English answer $a_{en}$

Why provide the English answer to translators? The translators are shown the English answer as guidance for annotating the target-language answer. The paper states this is done to "try to minimize cultural and personal differences in the amount of detail in the answer" (Section 2.3). Without this guidance, translators in different languages might select answers at different levels of granularity (e.g., one translator selects "Early Middle Ages" while another selects "during the Early Middle Ages" for the same question), introducing systematic annotation differences that would confound cross-language performance comparisons. By providing the English answer as a reference, the annotation is more likely to be consistent across languages in terms of what constitutes the minimal correct answer span.

Context scope for translators. Translators are "only shown the aligned sentence and the sentence on each side (where these exist)" rather than the full paragraph. The rationale is practical: in some cases the aligned sentences are not perfectly parallel, and restricting the visible context to the immediate vicinity of the parallel sentence "increases the chance of the question being answerable, as in some cases the aligned sentences are not perfectly parallel, without requiring workers to read the entire context" (Section 2.3). This is a pragmatic acknowledgment that the automatic alignment is imperfect — if the surrounding context differs enough to make the question genuinely unanswerable, showing more context would not help. The three-sentence window (aligned sentence ±1) strikes a balance between providing enough context for answer identification and avoiding confusion from non-parallel surrounding text.

Quality assurance for translations. A 2% sample of translated questions undergoes additional review by language experts. Translators whose work does not meet quality standards are removed from the translator pool, and their translations are reallocated to other translators. This is a standard professional translation quality control mechanism: spot-checking catches systematic issues (e.g., a translator who consistently misinterprets a particular construction) without requiring full double-translation of all instances.

Answer length distribution as a quality signal. The paper reports that by comparing the distribution of answer lengths (relative to context length) across target languages to the English distribution, some cases were found where "some annotators selected very long answers, especially for Chinese" (Section 2.3). This indicates that those translators were not following the instruction to select minimal answer spans. These cases were sent for re-annotation with clarified instructions. This is a clever use of cross-lingual consistency as a quality control mechanism — if one language's answer-length distribution is an outlier, it suggests an annotation style issue rather than a genuine property of the language.

Handling unanswerable instances. Translators can indicate that a question has no answer in the target language. This occurs because:

  • The aligned sentences occasionally differ in information content (the English sentence mentions a specific entity that the target-language sentence omits or replaces)
  • Some questions rely on surrounding context that differs between languages
  • Translation errors in the parallel sentence mining create sentences that are not truly parallel

The rate of "No Answer" annotations ranges from 6.6% for Vietnamese to 21.9% for German. These instances are removed from the main MLQA corpus but released separately as "an additional resource" — they are useful for evaluating QA models' ability to detect when questions are unanswerable, a task studied in SQuAD 2.0 (Rajpurkar et al., 2018) but not explored in this paper's experiments.

Why some languages have higher unanswerable rates. The paper does not analyze this in detail, but the high rate for German (21.9%) compared to Vietnamese (6.6%) is notable. Possible explanations include: the German Wikipedia has more independently written (rather than translated) content, leading to larger divergence from English articles; the LASER alignment quality varies by language pair, with English-German alignments potentially being less precise than others; or German's more flexible word order makes sentence-level alignment harder than for more fixed-order languages.

Impact on dataset parallelism. Because unanswerable instances are removed from target languages but remain in English, some instances lose their 4-way parallelism. The final dataset has: 9,019 instances that are 4-way parallel, 2,930 that are 3-way parallel, and 789 that are 2-way parallel (Section 2.4). The English source data retains all instances (12,738 total), but each target language ends up with between 5,029 (German) and 6,006 (Vietnamese) instances — this variation is driven partly by unanswerable rates and partly by the sampling of which language combinations were used.

Dataset Splits and Task Definitions

Development and test splits. MLQA is partitioned into development and test sets (Table 2). The development split is small: 1,148 English instances, and 500–517 instances per target language. The test split is roughly 10× larger: 11,590 English instances, and 4,517–5,495 instances per target language. The paper notes that since MLQA is parallel, "this means there is development data for every language" — a deliberate design choice that enables standard development-set-based hyperparameter tuning for future work, even though the current paper's baselines "only make use of the English development data and study strict zero-shot settings" (Section 6).

Task 1: Cross-lingual Transfer (XLT). This is the standard zero-shot cross-lingual evaluation: models are trained on English data only (SQuAD v1.1), development-tuned on MLQA English development data, and evaluated on MLQA test data in each target language independently. The model must take target-language context ($c_y$) and target-language question ($q_y$) and produce an answer span in the target language. This tests whether multilingual representations learned during pretraining (or machine translation) can bridge the gap between the training language and each target language.

Task 2: Generalized Cross-lingual Transfer (G-XLT). This is a novel task enabled by MLQA's parallelism: the model must answer a question in one language using a context in a different language. Formally, given a question ($q_y$) in language $y$ and a context ($c_z$) in language $z$, where $y \neq z$, extract the answer span from $c_z$. This is possible because parallel instances in MLQA have the same question meaning across languages — the G-XLT evaluation simply swaps the question language for a given context without changing the question's information content. This task tests the cross-lingual alignment of the model's internal representations: can the model match a Hindi question against a Spanish context and extract the correct answer, even though Hindi and Spanish were never explicitly aligned during training? The XLT task is a special case of G-XLT where question and context language match (the diagonal of Tables 6 and 9).

What G-XLT reveals that XLT cannot. XLT measures how well a model transfers to a target language as a whole — it conflates the model's ability to understand the question in the target language and its ability to understand the context in the target language. G-XLT decomposes these: by varying the question language independently of the context language, you can measure whether performance drops are due to poor question understanding, poor context understanding, or poor cross-lingual alignment. For example, Table 6 shows that for contexts in Arabic, English questions produce F1 = 60.0 while Arabic questions produce F1 = 54.8 — suggesting that Arabic questions are harder to understand than English questions even when the context is in Arabic.

Evaluation Metrics and Multilingual Adaptations

Standard metrics. MLQA uses the two standard extractive QA metrics: Exact Match (EM) — the fraction of predictions that exactly match the gold answer span — and mean token F1 score — the average token-level overlap between predicted and gold spans, computed as the harmonic mean of precision and recall at the token level.

Multilingual preprocessing modifications. The standard SQuAD evaluation script applies English-specific preprocessing: lowercasing, stripping ASCII punctuation, stripping English articles ("a", "an", "the"), and whitespace tokenization. These operations are not appropriate for multilingual evaluation. The paper introduces the following modifications:

  1. Punctuation stripping: Instead of stripping only ASCII punctuation characters, the script strips all Unicode characters with a punctuation General_Category — this covers non-ASCII punctuation like the Arabic question mark (؟), Chinese punctuation marks (。，), and Spanish inverted question marks (¿). The Unicode General_Category property is a standardized classification of characters — using it ensures consistent punctuation handling across all languages without language-specific rules.

  2. Article stripping: Articles are stripped only for languages that have stand-alone articles: English, Spanish, German, and Vietnamese. Arabic (where the definite article "al-" is a prefix) and Hindi (where articles are postpositions or absent) are not subjected to article stripping. Chinese does not have articles. This language-conditional preprocessing avoids incorrectly stripping substrings that happen to match article patterns in unrelated languages.

  3. Tokenization: Whitespace tokenization is used for all languages except Chinese. For Chinese, the paper uses "the mixed segmentation method from Cui et al. (2019b)" — this handles the fact that written Chinese does not use spaces between words, so a special segmenter is needed to identify word boundaries. The reference is to the CMRC 2018 dataset paper, which provides a standardized Chinese tokenization approach for QA evaluation.

Why these metric adaptations matter. Standard English QA metrics applied to other languages would produce systematically unfair comparisons. For example, a model might produce the correct Arabic answer but fail exact match because of a different punctuation character or a minor morphological variation that an Arabic-aware metric would forgive. The adapted metrics attempt to normalize language-specific surface-form variation in a principled way — the modifications are documented and reproducible rather than being ad hoc.

Baseline Model Configurations

The paper establishes several baselines to characterize current cross-lingual QA performance on MLQA. These are not the paper's contribution but define the performance landscape against which future work will be measured.

Training data and model architecture. All models use SQuAD v1.1 as training data and are trained with "the SQuAD-v1 training method from Devlin et al. (2019) and implemented in Pytext (Aly et al., 2018)." This means the standard BERT-based QA architecture: a pretrained Transformer encoder (BERT or XLM) processes the concatenated question and context, and a linear span-prediction layer on top of the final hidden states predicts start and end token probabilities for the answer span. The training objective is the sum of cross-entropy losses for the start and end positions.

Model variants tested:

  1. BERT-Large (English only): Standard BERT-large model trained and evaluated exclusively on English. This provides the upper bound for what is achievable in the training language — it establishes that the MLQA-en data is of comparable difficulty to SQuAD (Section 5.3).

  2. Multilingual BERT (M-BERT): BERT-base, cased, trained on 104 languages. This is a zero-shot cross-lingual approach: the model is fine-tuned on English SQuAD and evaluated directly on target-language MLQA data without any target-language training or development tuning. M-BERT's multilingual pretraining is expected to provide some cross-lingual generalization because the shared subword vocabulary and shared parameters across languages allow patterns learned from English QA to partially transfer.

  3. XLM (MLM + TLM): Cross-lingual Language Model pretrained with Masked Language Modeling (MLM) on 15 languages and Translation Language Modeling (TLM) on parallel data. TLM extends MLM by concatenating parallel sentences and randomly masking tokens in both languages — the model must predict masked tokens using both monolingual and cross-lingual context, explicitly training cross-lingual alignment. XLM is expected to outperform M-BERT because TLM provides a stronger cross-lingual signal than M-BERT's simpler multilingual pretraining. Same zero-shot protocol: fine-tune on English SQuAD, evaluate on target languages.

Strict zero-shot constraint. Both M-BERT and XLM baselines are "strictly zero-shot" — model selection and hyperparameter optimization use only English development data (MLQA-en dev set). There is no peeking at target-language development data. The authors acknowledge that "a single model that we test for all 7 languages" results from this constraint — the same model checkpoint is used across all target languages, even though per-language hyperparameter tuning might improve performance. This conservatism is appropriate for a benchmark paper: it establishes a clean, reproducible baseline that future work can compare against without concerns about hyperparameter hacking.

  1. Translate-Train: The SQuAD training set is machine-translated into the target language using Facebook's production translation models. Before translation, answer spans are enclosed in quotes to make them easy to extract from the translated contexts and to "encourage the translation model to map answers into single spans" (Section 4). Instances where answer extraction fails (the quoted span is not present in the translation) are discarded — approximately 5%. The translated SQuAD corpus is then used to train a target-language QA model (using M-BERT or XLM as the base architecture). This is a supervised approach in the target language, but uses synthetic training data rather than human annotations.

  2. Translate-Test: At test time, the target-language context and question are translated into English using machine translation. An English QA model (BERT-Large trained on SQuAD) processes the translated text and produces an English answer span. The English answer is then mapped back to a span in the original target-language context using attention scores from the translation model. For all languages except Hindi (where attention alignments were unavailable due to production model limitations), this mapping uses the span-alignment procedure defined by Equation 1.

The Translate-Test span mapping (Equation 1 in the paper):

RC=∑i∈Se,j∈Soaij∑i∈Seai∗,PR=∑i∈Se,j∈Soaij∑j∈Soa∗j\text{RC} = \frac{\sum_{i \in S_e, j \in S_o} a_{ij}}{\sum_{i \in S_e} a_{i*}}, \quad \text{PR} = \frac{\sum_{i \in S_e, j \in S_o} a_{ij}}{\sum_{j \in S_o} a_{*j}}

F1=2⋅RC⋅PRRC+PR\text{F1} = \frac{2 \cdot \text{RC} \cdot \text{PR}}{\text{RC} + \text{PR}}

answer=arg⁡max⁡SoF1(So)\text{answer} = \arg\max_{S_o} \text{F1}(S_o)

where $S_e$ is a candidate English span produced by the QA model (the span predicted from the translated context), $S_o$ is a candidate span in the original target-language context, $a_{ij}$ is the cross-attention weight from the translation model connecting token $i$ in the English output to token $j$ in the original input, $a_{i*} = \sum_j a_{ij}$ is the total attention from English token $i$ to all original tokens (row sum), $a_{*j} = \sum_i a_{ij}$ is the total attention from all English tokens to original token $j$ (column sum), RC is the recall-like measure (fraction of English span attention that falls within the candidate original span), and PR is the precision-like measure (fraction of candidate original span attention that comes from the English span).

What this computes: For each possible span $S_o$ in the original target-language context, the procedure computes an F1 score between the English predicted span and the candidate original span using attention weights as a soft alignment. The RC term measures: of all the attention flowing from the English predicted span back to the original context, what fraction lands within $S_o$? This captures whether $S_o$ is a good "explanation" for the English prediction in terms of attention. The PR term measures: of all the attention flowing into $S_o$ from the English output, what fraction comes from the predicted English span? This captures whether $S_o$ is specifically aligned with the predicted answer rather than with other parts of the output. The F1 score balances these two considerations. The argmax over all candidate spans selects the $S_o$ that best aligns with the English prediction.

Why this form over argmax attention mapping? A simpler approach (used by Asai et al., 2018) would be to take the original-context span with maximum attention from the English answer tokens — essentially an attention argmax. The F1-based approach is more robust because it considers whole-span alignment rather than token-level maxima. An attention argmax might select a single highly-attended token even if the surrounding tokens have low attention, producing a fragmentary span. The F1 criterion encourages selecting a coherent span where the attention mass is concentrated.

Why the paper takes this approach. Translate-Test is included because it represents a non-neural, pipeline-based alternative to zero-shot cross-lingual transfer: if machine translation works well enough, you can avoid the complexity of multilingual representation learning entirely by translating everything to English and using an English-only QA model. The results show this is competitive but not dominant — XLM zero-shot outperforms Translate-Test for most language pairs, suggesting that multilingual representations capture information that gets lost in the MT pipeline.

For Hindi, where attention alignments are unavailable: The paper notes that "instead we translate English answers using another round of translation. Back-translated answers may not map back to spans in the original context, so this Translate-Test performs poorly" (Section 4, footnote 5). The Hindi Translate-Test results in Table 5 (23.8 F1, 18.9 EM) are far below other methods, confirming that the attention-based mapping is essential for Translate-Test to work and that simple back-translation is insufficient.

Key Design Decisions and Their Justifications

Why 4-way parallelism rather than 7-way? The paper explicitly grapples with this tradeoff in Section 2.1. 7-way parallel sentences were too few (1,340 total) and lacked diversity — they were mostly formulaic first sentences of articles, which would produce a dataset of low-diversity, easy questions about article topics rather than substantive factual questions. 4-way parallelism provided 385,396 sentences to sample from, producing a dataset with over 5,000 instances per language and broad topical coverage (23% about people, 19% physical places, 13% cultural topics, 12% science/engineering, etc.). This is a classic precision-recall tradeoff in data construction: stronger constraints (more languages) increase the quality of parallelism but reduce quantity and diversity.

Why Wikipedia rather than manual document translation? Manual translation of documents at MLQA's scale (thousands of paragraphs across seven languages) would be "hugely expensive" and would produce translationese — syntactically and lexically unnatural text that would not reflect the distribution of real target-language documents. Wikipedia provides documents that are "naturally written" (with the caveat that some are volunteer translations, discussed in Section 6), are already parallel by topic, and match the domain of SQuAD — the primary training resource. The tradeoff is that the automatic alignment is imperfect, producing some unanswerable instances, but the cost and naturalness advantages outweigh this.

Why constrain answers to the parallel sentence? This is the core design choice that makes the whole pipeline work. Without this constraint, questions could reference any part of the English paragraph, and there would be no guarantee the information exists in the target-language paragraph. The constraint guarantees cross-lingual answerability while using naturally written documents (because the parallel sentence is embedded in a natural paragraph, even if that paragraph differs across languages). The tradeoff is that some questions become answerable from a single sentence rather than requiring multi-sentence reasoning — but the paper argues this is comparable to SQuAD (where single-sentence questions are common per Sugawara et al., 2018) and that MLQA-en scores similar to SQuAD support comparable difficulty.

Why SQuAD as training data rather than a custom training set? The paper's thesis is that evaluation data is the bottleneck, not training data. Using SQuAD leverages the largest existing extractive QA training resource without requiring costly multilingual training data annotation. It also ensures compatibility with the extensive ecosystem of SQuAD-trained models — any model architecture that works on SQuAD can be evaluated on MLQA without modification. The paper verifies in Section 5.3 that the domain shift from SQuAD to MLQA-en is modest (5.8% average discrepancy after adjusting for single-answer evaluation), confirming that SQuAD training generalizes reasonably to MLQA's English subset.

Why professional translation for questions rather than crowdsourcing? Crowdsourcing translation quality is highly variable, especially for languages where the available worker pool on platforms like Mechanical Turk is small. Professional translators provide higher and more consistent quality, and the One Hour Translation platform enables quality control mechanisms (spot-checking by language experts, removal of underperforming translators) that would be difficult with anonymous crowd workers. The cost is higher per instance, but the paper only needs translation for evaluation data (not training data), so the total translation budget is manageable — roughly 5,000–6,000 questions per language.

Why release "No Answer" instances separately rather than including them in the main benchmark? This separates two distinct research problems: (1) cross-lingual transfer of extractive QA (does the model find the right answer when one exists?), and (2) cross-lingual unanswerable question detection (does the model know when there is no answer?). Mixing these would confound evaluation, because a model might achieve low F1 either because it is bad at extraction or because it correctly identifies unanswerable questions but gets penalized for not producing a span. By releasing unanswerable instances separately, the paper cleanly separates these two evaluation dimensions and provides data for future work on both.

Why the specific seven languages? The paper states they were chosen to "represent a broad range of linguistic phenomena and have sufficiently large Wikipedia" (Section 2). The practical constraint of "sufficiently large Wikipedia" rules out many lower-resource languages — the parallel sentence mining approach requires substantial article overlap between languages, and languages with small Wikipedias will have few same-topic articles with English. The seven chosen languages represent a pragmatic balance of diversity and data availability, though the paper acknowledges this still leaves many language families and scripts unrepresented (no languages using the Cyrillic script, no tonal languages except Vietnamese and Chinese, no languages from sub-Saharan Africa or the Americas).

Why 2% sampling for translation quality review? This is a standard professional translation quality control practice. Reviewing 100% of translations would double the translation cost; reviewing 2% provides a statistical quality estimate and catches systematic issues. If a translator has a 50% error rate, this will be detectable in a 2% sample with high probability; if they have a 5% error rate, it may not be detected but the impact on overall dataset quality is small. The 2% figure is a cost-quality tradeoff that the paper does not defend in detail but that reflects standard industry practice.

4. Key Insights and Innovations

Innovation 1: Parallelism as a First-Class Design Principle, Not an Afterthought

The dominant approach to cross-lingual evaluation data prior to MLQA was to either (a) manually translate an existing English dataset into target languages (Asai et al., 2018; Artetxe et al., 2019) or (b) construct independent datasets in each language with varying protocols, sizes, and difficulty distributions (Liu et al., 2019a). Both approaches treat parallelism as either absent or incidental. MLQA reverses this: parallelism is the organizing principle of the entire construction pipeline, not a post-hoc alignment step. Every design decision — the answer-in-parallel-sentence constraint, the 4-way rather than 7-way compromise, the professional translation workflow that provides English answers to translators — exists to maximize the degree to which individual question instances (not just dataset-level statistics) are comparable across languages.

This is conceptually distinctive because it shifts the evaluation framework from "how well does the model transfer to Language X?" to "how well does the model transfer to Language X on this specific question, compared to Language Y on the same question?" The latter question is substantially more informative and was not answerable with prior resources. It enables precise, paired analysis — Figure 4's stratification by English question difficulty, the wh-word analysis in Figure 3, and the named-entity-type analysis in Figure 8 all depend on having the same questions in all languages in order to ask whether a question that is hard in English is proportionally hard in Vietnamese or whether the difficulty gap varies by language. Without parallelism, you cannot distinguish "this language is harder for models" from "this language's dataset happened to contain harder questions."

The significance of this move extends beyond MLQA. By demonstrating that multi-way parallel evaluation data can be constructed at scale (46,000+ annotations across seven languages) using naturally-occurring documents rather than manual translation — and by documenting the cost-quality tradeoffs explicitly (the 7-way vs. 4-way tension, the "No Answer" rates by language) — the paper provides a template for how future cross-lingual benchmarks should be constructed. The "five desiderata" framework itself is a meta-contribution: it makes explicit the evaluation design criteria that prior work had followed implicitly or not at all, providing a vocabulary for discussing what makes cross-lingual benchmarks good. This is a fundamental shift in how cross-lingual evaluation is conceived: from assembling whatever data exists to engineering data that answers specific scientific questions.

The evidence for this innovation's impact is structural rather than numerical — it is visible in the G-XLT task definition, which would be impossible without parallelism, and in the cross-lingual consistency analyses throughout Section 5. The benchmark's subsequent adoption into aggregated multilingual suites (XGLUE, XTREME) confirms that the field recognized parallelism not as a nice-to-have but as a requirement for rigorous cross-lingual evaluation.

Innovation 2: The Parallel-Sentence Constraint as a Cross-lingual Answerability Guarantee Without Parallel Documents

Prior cross-lingual QA efforts faced a fundamental tension: ensuring answerability in target languages required either (a) manually translating entire documents, which introduced translationese and was prohibitively expensive at scale, or (b) accepting that many questions would be unanswerable in target languages, which made cross-lingual comparison noisy and unfair. MLQA introduces a third path: constrain English answer spans to lie within automatically-identified parallel sentences, which are guaranteed (probabilistically) to contain the same information in all languages, while allowing the surrounding paragraph context to vary naturally. This decouples the answerability guarantee (which depends only on the parallel sentence) from the document naturalness (which depends on the full context being independently written).

This is not a trivial engineering trick — it is a conceptual reframing of what needs to be parallel in a cross-lingual dataset. The key insight is that only the information-bearing unit containing the answer needs to be cross-lingually aligned, not the full document. The paper demonstrates empirically that this constraint produces a dataset of comparable difficulty to SQuAD (Table 7: adjusted SQuAD F1 of 84.8 vs. MLQA-en F1 of 80.2 for BERT-Large, with most of the gap attributable to longer average contexts and single-answer evaluation), validating that restricting answer spans does not trivialize the task. At the same time, the target-language "No Answer" rates (6.6% for Vietnamese to 21.9% for German) demonstrate that the automatic alignment is imperfect — the guarantee is probabilistic, not absolute — and the paper's transparency about these rates (including releasing the unanswerable instances as a separate resource) provides a realistic picture of the approach's limitations.

The broader significance is methodological: this demonstrates that parallel sentence mining from Wikipedia — a technique developed for machine translation data augmentation (Schwenk et al., 2019) — can be repurposed as a building block for cross-lingual evaluation benchmarks. This opens the door for constructing extractive QA datasets in many more languages without the bottleneck of professional document translation. The 7-way → 4-way compromise analysis is particularly informative: by showing that stronger multi-way parallelism requirements exponentially reduce available data (from 385K 4-way sentences to 1,340 7-way sentences) and reduce linguistic diversity, the paper provides practical guidance for future benchmark construction about where to set the parallelism/diversity tradeoff. This is a fundamental contribution to benchmark design methodology rather than an incremental refinement of existing approaches.

Evidence: the entire annotation pipeline (Figure 1) is built around this constraint, and the paper's ability to produce 46K+ annotations across seven languages — substantially more data per language than contemporaneous XQuAD (Artetxe et al., 2019) while using natural rather than translated documents — is a direct consequence of the constraint's efficiency.

Innovation 3: G-XLT — Decomposing Cross-lingual Transfer into Question Understanding, Context Understanding, and Cross-lingual Alignment

The standard cross-lingual transfer evaluation (XLT) measures a single composite capability: given a target-language context and target-language question, can the model find the answer? This conflates three distinct sub-capabilities: (1) understanding the question in the target language, (2) understanding the context in the target language, and (3) aligning the question's information need with the context's information content across languages. By introducing G-XLT — evaluating models with mismatched question and context languages — the paper provides a diagnostic instrument that decomposes these capabilities. The diagonal of the G-XLT matrix (Tables 6 and 9) is XLT (matched languages); the off-diagonal cells reveal what happens when the question language changes while the context language remains fixed, or vice versa.

This is a diagnostic innovation, not a performance innovation. G-XLT does not claim to be a practical application (people rarely ask questions in Hindi about Arabic texts in real-world scenarios). Instead, it provides a lens for understanding why cross-lingual transfer fails. For example, Table 6 shows that for contexts in Arabic, substituting English questions (F1 = 60.0) for Arabic questions (F1 = 54.8) actually improves performance — the model understands English questions better than Arabic questions, even when answering from Arabic contexts. This suggests that XLM's Arabic question understanding is a bottleneck, not its Arabic context understanding. Conversely, for contexts in English, substituting German questions (58.5) for English questions (74.9) causes a sharp drop, confirming that target-language question understanding is a significant source of the XLT performance gap.

Without G-XLT, these distinctions would be invisible. The XLT results alone would show that Arabic (54.8 F1) performs worse than German (62.2 F1) — but this could be because Arabic contexts are harder, Arabic questions are harder, or Arabic cross-lingual alignment is worse. G-XLT reveals (at least partially) which factor dominates: for Arabic, the question language effect is small and the context language effect is large, suggesting the bottleneck is Arabic context understanding. For Spanish, the opposite pattern might hold. This is methodologically analogous to ablation studies in model architecture — by systematically varying one factor while holding others constant, you isolate its contribution — but applied to evaluation design.

The broader conceptual move is treating cross-lingual transfer not as a monolithic phenomenon but as a composition of dissociable capabilities that can be measured independently given the right evaluation design. This decomposition enables more targeted future research: if question understanding is the bottleneck, work on cross-lingual question representations is needed; if context understanding is the bottleneck, work on cross-lingual document encoding is needed. The paper does not fully exploit this diagnostic potential — it does not systematically analyze which sub-capability dominates for which language pairs — but it provides the evaluation infrastructure for others to do so.

Evidence: the full G-XLT matrices in Tables 6 and 9, and the observation that the M-BERT XLT-to-G-XLT drop (10.5 mean F1) is larger than XLM's (8.2 mean F1), suggesting XLM's explicit cross-lingual training reduces the mismatch penalty. This is a fundamental conceptual contribution — a new type of evaluation that reveals structure invisible to standard protocols.

Innovation 4: Zero-Shot Transfer Baselines Define a Large and Persistent Performance Gap — Establishing the Problem, Not Solving It

The paper's baseline results (Table 5) are notable not for their absolute performance but for the size and structure of the remaining gap. XLM achieves a mean F1 of 59.3 across target languages, compared to BERT-Large's 80.2 F1 on English — a ~21 point F1 gap (39.8% relative EM drop). This is not a solved problem being dressed up as a benchmark; it is a genuine challenge where state-of-the-art models leave substantial room for improvement. Crucially, the gap is not uniform: performance ranges from 68.0 F1 on Spanish (relatively close to English) to 48.8 F1 on Hindi (a ~26 point drop from English XLM). This structured variation — the gap is larger for languages that are typologically distant from English or have less representation in pretraining data — is itself a finding that the benchmark reveals but did not presuppose.

The innovation here is the paper's discipline in presenting its contribution as measurement infrastructure rather than method improvement. Many benchmark papers are tempted to also introduce a new model that "solves" (partially) the benchmark, blurring the line between the contribution (the evaluation framework) and the validation (the baseline). MLQA explicitly avoids this: the baselines use existing, published models (M-BERT, XLM) with standard training procedures (SQuAD fine-tuning, zero-shot transfer), and the paper makes clear that translating SQuAD or using multilingual representations are established techniques, not novel proposals. This disciplinary clarity — "here is a problem we have measured; here is how large the gap remains; go solve it" — is increasingly rare and valuable.

The paper's stratification analyses further demonstrate the benchmark's diagnostic value beyond aggregate scores. Figure 3 shows that "When" questions are consistently easier across all languages while "Where" questions are harder for most — a finding that is robust across languages but would be invisible in aggregate metrics. Figure 4's stratification by English difficulty reveals that questions the model gets wrong in English are not uniformly impossible in target languages — the model sometimes succeeds on a target-language version of a question it failed on in English, suggesting some questions genuinely are easier in some languages. These are not performance achievements; they are observations about the nature of cross-lingual transfer that the benchmark's design (parallelism, diversity, scale) makes possible to observe for the first time.

This is a fundamental intellectual contribution disguised as a baseline results table. By establishing a clear, well-measured, and large performance gap — and providing the analytical tools (difficulty stratification, question-type analysis, G-XLT decomposition) to investigate why the gap exists — the paper creates the conditions for focused research progress in a way that a benchmark with near-saturated performance could not.

Evidence: Table 5, Figures 3 and 4, and the explicit statement that "all models lag well behind training-language performance" and there is "significant room for improvement" (Section 5.1). The gap is not an incidental finding — it is the intended result that validates the benchmark's need to exist.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments evaluate on the MLQA corpus, which consists of 12,738 English QA instances and between 5,029 and 6,006 instances in each of six target languages (German, Spanish, Arabic, Chinese, Vietnamese, Hindi), partitioned into development (1,148 English, ~500 per target language) and test (11,590 English, ~4,500–5,500 per target language) splits. Since MLQA is an evaluation-only benchmark with no training partition, the authors adopt SQuAD v1.1 (Rajpurkar et al., 2016) — a large-scale English extractive QA dataset built from Wikipedia — as the training dataset for all models.

  • Base model(s). Three base architectures are evaluated: BERT-Large (English-only, 340M parameters), Multilingual BERT (M-BERT) (BERT-base cased, 104 languages, trained with masked language modeling on Wikipedia), and XLM (Cross-lingual Language Model, 15 languages, trained with both masked language modeling and translation language modeling on parallel data). BERT-Large serves as the English upper bound; M-BERT and XLM are the zero-shot cross-lingual transfer models. The choice of these architectures reflects the state of cross-lingual representation learning at the time — these are not novel models proposed by the paper but rather established techniques being benchmarked.

  • Metrics. Performance is measured using two standard extractive QA metrics: Exact Match (EM) — the fraction of predictions that exactly match the gold answer span character-for-character — and mean token F1 score — the average token-level overlap between predicted and gold spans, computed as the harmonic mean of token-level precision and recall. The standard SQuAD evaluation script is adapted for multilingual evaluation with three modifications: (1) Unicode-aware punctuation stripping replaces ASCII-only stripping, using the Unicode General_Category property to identify punctuation; (2) article stripping is applied only for languages with stand-alone articles (English, Spanish, German, Vietnamese) and not for Arabic, Hindi, or Chinese; (3) tokenization uses whitespace for all languages except Chinese, where a mixed segmentation method from Cui et al. (2019b) handles word boundary identification. These adaptations prevent language-specific surface-form variation from producing unfair cross-lingual comparisons.

  • Baselines. Five evaluation configurations are tested, spanning machine translation and multilingual representation approaches:

    • BERT-Large (English only): Standard English BERT-Large fine-tuned on SQuAD and evaluated on MLQA-en. This establishes the upper bound for training-language performance.
    • Multilingual BERT (M-BERT) zero-shot: M-BERT fine-tuned on English SQuAD and evaluated directly on target-language MLQA test data without any target-language training. All hyperparameter selection uses only English development data.
    • XLM zero-shot: XLM fine-tuned on English SQuAD and evaluated directly on target-language MLQA test data, same zero-shot protocol.
    • Translate-Train: SQuAD training instances are machine-translated into each target language using Facebook's production translation models (answers are enclosed in quotes during translation to facilitate extraction; instances where extraction fails are discarded, ~5%). A target-language QA model (M-BERT or XLM) is then trained on the translated SQuAD corpus. This is a supervised approach using synthetic target-language data.
    • Translate-Test: At test time, target-language contexts and questions are machine-translated into English. An English BERT-Large QA model processes the translated text and produces an English answer span. The English span is then mapped back to the original target-language context using attention scores from the translation model, via the F1-based span alignment procedure defined in Equation 1. For Hindi, attention alignments are unavailable; back-translation is used instead, resulting in substantially worse performance.

  • Generation budget / compute accounting. The paper does not report compute budgets, training FLOPs, or inference cost for any model. All models use the standard SQuAD fine-tuning procedure from Devlin et al. (2019) implemented in Pytext (Aly et al., 2018), but no training details (epochs, learning rates, batch sizes, hardware) are reported in the main text. The Translate-Train baselines incur additional cost from machine translation of the entire SQuAD training set (~100K instances per target language) and training separate models per language; the Translate-Test baselines incur per-query translation costs at inference time. The zero-shot M-BERT and XLM baselines incur no additional cost beyond standard fine-tuning. Since the paper's contribution is the benchmark itself, not a method requiring compute-matched comparison, the absence of cost accounting is consistent with the paper's goals — but it means the efficiency-versus-performance tradeoff between Translate-Test/Translate-Train and zero-shot transfer is not characterized.

  • Cross-validation / statistical protocol. The paper does not report statistical significance testing, confidence intervals, or cross-validation. Model selection is performed using only the English development data (MLQA-en dev set), producing a single model checkpoint that is evaluated on all seven test languages — a "strictly zero-shot" protocol (Section 4): "model selection is also constrained to be strictly zero-shot, using only English development data to pick hyper-parameters. As a result, we end up with a single model that we test for all 7 languages." This eliminates any possibility of target-language data leakage during model development but means performance is not optimized per language. The paper notes (Section 6) that future work could use the target-language development sets for hyperparameter tuning or fine-tuning, but treats this as "few-shot" experiments left for future investigation.

Main Quantitative Results

Cross-lingual Transfer (XLT): Zero-Shot and Translation-Based Baselines

The central quantitative result of the paper is in Table 5, which reports F1 and EM scores on the MLQA test set for all models on the XLT task. XLM zero-shot achieves the best overall performance with a mean target-language F1 of 59.3 (calculated as the mean of the six target-language F1 scores: 68.0 for Spanish, 62.2 for German, 54.8 for Arabic, 48.8 for Hindi, 61.4 for Vietnamese, 61.1 for Chinese). This represents a substantial cross-lingual transfer capability — the model answers questions in languages it was never trained on — but is 15.6 F1 points below XLM's own English performance (74.9 F1) and 20.9 F1 points below the BERT-Large English upper bound (80.2 F1). Expressed in Exact Match, the drop is more dramatic: XLM achieves a mean EM of 39.8% across target languages compared to BERT-Large's 67.4% EM on English — a 39.8% relative reduction.

XLM vs. M-BERT zero-shot. XLM outperforms M-BERT on five of six target languages, with the largest margins on Arabic (+9.1 F1, 54.8 vs. 45.7), Spanish (+3.7 F1, 68.0 vs. 64.3), and German (+4.3 F1, 62.2 vs. 57.9). For Chinese and Vietnamese, the two models are comparable (XLM: 61.1/61.4 vs. M-BERT: 57.5/57.1). M-BERT achieves slightly higher English F1 (77.7 vs. 74.9) but transfers less effectively. This pattern — XLM's explicit cross-lingual training (translation language modeling) improving transfer at a slight cost to English performance — is consistent with the hypothesis that TLM's cross-lingual signal strengthens multilingual alignment.

Zero-shot vs. translation-based approaches. The comparison between zero-shot transfer and machine translation pipelines reveals a nuanced pattern that varies by language:

  • Translate-Train with XLM achieves competitive or superior performance on several languages: it matches XLM zero-shot on German (61.4 F1 vs. 62.2), nearly matches on Spanish (65.2 vs. 68.0), and surpasses zero-shot on Chinese (59.8 vs. 61.1). However, Translate-Train with M-BERT significantly outperforms XLM zero-shot on Hindi (55.0 vs. 48.8 F1) and Vietnamese (62.0 vs. 61.4), suggesting that for these languages, having target-language training data — even synthetic translated data — provides benefits that zero-shot multilingual representations do not capture.

  • Translate-Test with BERT-Large is the strongest translation-based approach for Spanish (65.4 F1), comparable for German (57.9) and Vietnamese (58.2), but substantially worse for Arabic (33.6 F1) and Chinese (44.2 F1). The degradation for Arabic and Chinese suggests that the MT quality or the attention-based span mapping introduces errors for these languages that zero-shot models avoid. The Hindi Translate-Test results (23.8 F1, 18.9 EM) are far below all other methods — the paper attributes this to the unavailability of attention alignments, forcing a fallback to answer back-translation that "may not map back to spans in the original context" (footnote 5).

Language-specific performance ordering. The relative difficulty of target languages for XLM zero-shot transfer follows the ordering: Spanish (68.0 F1) > German (62.2) > Vietnamese (61.4) > Chinese (61.1) > Arabic (54.8) > Hindi (48.8). This ordering is not uniformly preserved across all methods — Translate-Train with M-BERT achieves 55.0 F1 on Hindi, making it the best Hindi model and reshuffling the ranking — but the broad pattern of Spanish and German being easiest and Arabic/Hindi being hardest is consistent. The paper does not deeply analyze the causes of this ordering (e.g., whether it correlates with pretraining data quantity, linguistic distance from English, script type, or morphological complexity), leaving such analysis to future work.

Question-type stratification (Figure 3). When XLM F1 scores are stratified by English question type (using the wh-word), "When" questions are consistently easier across all languages, with relative gains of +8.3 F1 on average — ranging from +4.1 for Arabic to +11.1 for German. This makes intuitive sense: "When" questions typically have short, formulaic answers (dates, time periods) that are likely to be similarly expressed across languages. "Where" questions show the opposite pattern, with a mean relative drop of -2.7 F1, and particularly large drops for German (-6.5), Hindi (-5.8), and Spanish (-4.6). "What" questions are also slightly harder than average (-1.6 F1 mean). The paper provides these observations without detailed linguistic explanation, but the cross-lingual consistency of the "When" advantage suggests that temporal expressions may be particularly well-aligned across languages in multilingual embedding spaces, while spatial expressions vary more.

English difficulty stratification (Figure 4). Splitting the test set by whether XLM got the English question completely wrong (0 F1) reveals that transfer performance is better when the model answers the English version correctly — expected, since questions that are intrinsically hard for the model remain hard across languages. However, the performance on questions that the model got wrong in English is "far from zero," indicating that "some questions may be easier to answer in some languages than others" (Section 5.1). This is a non-obvious finding: it means the model's difficulties are not fully invariant across languages. A question that requires reasoning about English-specific syntactic patterns might become easier in German if the relevant distinction is morphologically marked rather than relying on word order. Alternatively, differences in context paragraph length — the paper notes in Appendix A.3 that "context documents can be shorter in target languages for questions the model struggled to answer correctly in English, effectively making them easier" — could explain part of this effect.

English Performance Calibration: MLQA-en vs. SQuAD

Table 7 addresses a crucial calibration question: is MLQA-en harder than SQuAD, and if so, by how much? This matters because any gap between MLQA-en and SQuAD performance represents a domain shift confound — if the English subset is systematically harder, then some fraction of the XLT performance drop might be due to dataset difficulty rather than language transfer.

The raw comparison shows BERT-Large at 80.2 F1 on MLQA-en vs. 91.0 F1 on SQuAD v1.1 — a 10.8 F1 gap. However, the paper identifies that SQuAD scores in the literature are typically computed against multiple reference answers, which inflates F1 (a prediction that partially overlaps with any of several acceptable answers gets credit). When SQuAD scores are recomputed using a single reference answer — with the reference selected using the same method used to pick MLQA gold answers (majority or highest-F1 among available annotations) — the gap narrows substantially. The adjusted SQuAD scores (SQuAD* in Table 7) are: BERT-Large 84.8 F1, M-BERT 83.0 F1, XLM 82.1 F1. The remaining gap to MLQA-en (80.2, 77.7, 74.9 respectively) is 4.6, 5.3, and 7.2 F1 points — the paper attributes this to longer average contexts in MLQA (28% longer than SQuAD), wider article coverage (MLQA covers many more unique Wikipedia articles than SQuAD's curated set), and minor differences in preprocessing and answer length.

The critical implication is that most of the performance drop in XLT is genuinely due to cross-lingual transfer, not dataset difficulty. The English-calibrated expectation for XLM is approximately 75 F1; the mean target-language F1 of 59.3 represents a ~16 point cross-lingual gap, not a ~29 point gap that naive SQuAD-vs-MLQA comparison would suggest.

Named Entity Stratification (Appendix A.3, Figure 8)

An additional stratification analysis in the appendix categorizes answers by whether they contain named entities and what type. The key findings: (1) answer spans containing named entities are easier than those without for all languages, with the difference most pronounced for German; (2) temporal answers (DATE and TIME entities) are consistently easier across all languages, consistent with the "When" question advantage in Figure 3; (3) numeric answers (CARDINAL, ORDINAL, PERCENT, QUANTITY, MONEY) are also relatively easy in most languages; (4) Arabic shows a distinctive pattern — it performs relatively well on ORG, GPE, and LOC entities compared to other languages, which could reflect properties of Arabic Wikipedia, Arabic named entity recognition in the pretraining data, or Arabic morphological patterns that make these entity types easier to extract.

Generalized Cross-lingual Transfer (G-XLT): Mismatched Question and Context Languages

Table 6 presents the full 7×7 G-XLT matrix for XLM — F1 scores for every combination of question language (columns) and context language (rows). The diagonal represents the XLT setting (matched languages); off-diagonal cells represent mismatched settings. Table 9 provides the same for M-BERT.

The matched-language diagonal dominates. For a given context language, performance is typically highest when the question is in the same language (the diagonal). For example, with Spanish contexts, the Spanish-question F1 is 68.0, while English-question F1 is 69.5 — an exception to the rule. More typically, with German contexts: German questions achieve 62.2 F1, English questions achieve 70.6, and the remaining question languages range from 50.0 (Chinese) to 67.7 (Spanish).

English questions are the best or nearly-best question language for all context languages. Reading down any column of Table 6, the English-question row is consistently among the highest F1 scores. For Chinese contexts: English questions achieve 52.9 F1, Chinese questions achieve 61.1 F1 (the only case where English is not best or second-best). For Vietnamese contexts: English 60.2, Vietnamese 61.4. This means that G-XLT with English questions is almost as good as XLT with target-language questions — the model can answer questions asked in English nearly as well as questions asked in the native language. This is a practically useful finding: it suggests that deploying an English-language interface for a multilingual QA system may incur only a small accuracy penalty compared to translating user questions into each language.

Context language matters more than question language for some languages. For Arabic contexts, the F1 range across question languages is 40.9 (Chinese questions) to 60.0 (English questions) — a 19.1 F1 spread. For English contexts, the range is 43.6 (Hindi questions) to 74.9 (English questions) — a 31.3 F1 spread. The fact that Arabic contexts yield low F1 regardless of question language suggests that Arabic context understanding is a bottleneck, not Arabic question understanding. Conversely, English questions paired with Arabic contexts achieve 60.0 F1, while Arabic questions with English contexts achieve only 50.8 F1 — the model is much better at understanding English questions about Arabic text than Arabic questions about English text.

XLM vs. M-BERT for G-XLT. M-BERT (Table 9) shows a stronger preference for English questions and a larger XLT-to-G-XLT drop. The mean off-diagonal G-XLT F1 (averaging all non-diagonal cells) is 47.2 F1 for M-BERT and 53.4 F1 for XLM — XLM is substantially more robust to language mismatch. The mean drop from XLT to G-XLT (comparing diagonal vs. off-diagonal average) is 8.2 F1 for XLM and 10.5 F1 for M-BERT. This suggests that XLM's explicit cross-lingual training objective (TLM) produces representations that are better aligned across languages, making the model less sensitive to whether the question and context share a language.

Hindi and Arabic are the hardest context languages. For both XLM and M-BERT, the Hindi and Arabic context rows have the lowest F1 scores regardless of question language, indicating that these languages' contexts are genuinely harder for the models to extract answers from — likely due to a combination of more complex morphology, less representation in pretraining data, and greater linguistic distance from English.

Ablation Studies and Robustness Checks

The paper is a benchmark contribution, not a methods paper, so formal ablation studies (in the sense of systematically removing components and measuring performance change) are not present. However, several implicit comparisons and robustness analyses serve an equivalent function:

  • Zero-shot vs. Translate-Train (Table 5): Comparing XLM zero-shot against XLM Translate-Train (and M-BERT zero-shot against M-BERT Translate-Train) measures the value of synthetic target-language training data. Translate-Train helps most for Hindi (M-BERT: +11.2 F1, XLM: +1.9 F1) and least for Spanish (M-BERT: -10.4 F1, XLM: -2.8 F1). The fact that Translate-Train reduces performance for some languages is a notable negative result — machine translation noise can hurt more than target-language data helps when MT quality is high and zero-shot transfer is already strong.

  • Answer aggregation method (Section 5.3): The comparison between standard SQuAD scoring (multiple references) and single-reference scoring (SQuAD* in Table 7) demonstrates that reference multiplicity inflates F1 scores by 5.8 F1 points on average. This is not an ablation of a model component but rather a calibration of the evaluation protocol. For future benchmark comparisons, this analysis establishes that MLQA's single-reference scores are not directly comparable to published SQuAD scores, and researchers should use the adjusted SQuAD* numbers as the fair English baseline.

  • Translate-Test span mapping method (Equation 1 vs. argmax): The paper's F1-based attention mapping for Translate-Test is compared implicitly against the simpler attention argmax approach used by Asai et al. (2018), though no explicit quantitative comparison is presented. The Hindi Translate-Test result (23.8 F1) serves as a negative control: when attention alignments are unavailable and back-translation is used instead, performance collapses, confirming that the attention-based mapping (Equation 1) is essential for Translate-Test to work.

  • Cross-lingual model comparison (XLM vs. M-BERT): The consistent performance gap between XLM and M-BERT across most languages (Table 5) quantifies the benefit of translation language modeling over standard multilingual masked language modeling. The 5.5 F1 mean difference (comparing XLM 59.3 vs. M-BERT 52.9 mean target-language F1, calculated from Table 5) represents the effective contribution of explicit cross-lingual alignment during pretraining.

  • Question-type and difficulty stratification robustness: Figures 3, 4, and 8, and Appendix A.3 demonstrate that the relative language ordering and the performance gap are not uniform across all question types — "When" questions are robustly easier, spatial questions are robustly harder, and named-entity-containing answers are easier than non-entity answers. If these patterns did not hold across languages, it would suggest that the aggregate results were hiding fundamentally different behavior per question type; that they do hold strengthens confidence in the aggregate findings.

Negative result: XLM is weaker in English than M-BERT (Table 5). XLM achieves 74.9 F1 on MLQA-en vs. M-BERT's 77.7 — a 2.8 F1 deficit. This is a known tradeoff in multilingual models: the capacity devoted to cross-lingual alignment comes at a cost to monolingual English performance. The paper does not explore this tradeoff in detail, but it is a practically relevant finding: if English is the primary deployment language, M-BERT may be preferable to XLM.

Negative result: Translate-Test fails for Hindi due to missing attention alignments (Table 5). The 23.8 F1 for Hindi Translate-Test is the lowest score for any method-language pair, demonstrating that the attention-based mapping is non-redundant — without it, the entire MT pipeline approach collapses. This is a practical limitation for languages where translation model internals are not accessible.

Negative result: Translate-Train can underperform zero-shot transfer (Table 5). For Spanish, XLM zero-shot achieves 68.0 F1 while Translate-Train with XLM achieves only 65.2, and M-BERT zero-shot (64.3) outperforms Translate-Train (53.9). This contradicts the naive expectation that having target-language training data — even synthetic — always helps. The paper does not investigate the cause, but potential explanations include: MT errors introduce systematic noise that misleads the model, or the SQuAD answer distribution (shaped by English annotation conventions) does not match the MLQA answer distribution in the target language.

Critical Assessment

Claim: MLQA enables fairer cross-lingual comparison than prior benchmarks

Supported, but with a parallel-sentence alignment quality caveat. The core claim is that parallelism enables "fairer comparison across languages" (Section 2). The evidence is structural: Figures 3 and 4 demonstrate analyses — question-type stratification and difficulty-based stratification — that would be impossible or confounded without parallel questions across languages. The paper convincingly shows that per-language results on MLQA can be analyzed as a function of question properties that are invariant across languages (the English wh-word, the English difficulty) because the questions are the same.

However, the fairness guarantee depends on the quality of the parallel sentence alignment, and the paper's own data reveals non-trivial alignment failures: "No Answer" rates range from 6.6% (Vietnamese) to 21.9% (German). This means that for up to one-fifth of German instances, the question is not actually answerable — these instances are removed from the German test set but remain in English. This introduces a systematic selection bias: the German test set is not a random subset of the English test set; it is missing the instances where the parallel alignment failed. If alignment failures correlate with question difficulty, topic, or linguistic property (e.g., German sentences that diverge more from English tend to be more morphologically complex or on different topics), then the German subset is systematically easier or harder than the English subset in ways that confound "fair" comparison. The paper does not analyze whether unanswerable instances differ from answerable ones in difficulty-relevant properties.

The paper also acknowledges in Section 6 that Wikipedia articles are "often translated from English to the target languages by volunteers," meaning the "natural documents" desideratum is partially compromised. If German Wikipedia articles are more likely to be translations of English than Vietnamese articles are (plausible given higher English proficiency in German-speaking regions), this could explain why German performance is higher — the model is benefiting from translationese patterns that happen to align with English. This is an untested alternative explanation for the language performance ordering.

Claim: XLT reveals a large and persistent performance gap

Strongly supported, with clear quantitative evidence. Table 5 demonstrates that even the best method (XLM) achieves a mean target-language F1 of 59.3 versus 80.2 for English BERT-Large — a gap of ~21 F1 points. The gap is consistent across all methods (Translate-Train, Translate-Test, and zero-shot) and all languages, though it varies substantially by language (smallest for Spanish, largest for Hindi). The additional calibration in Table 7 confirms that this gap is not an artifact of MLQA-en being harder than SQuAD — the adjusted gap from SQuAD* to MLQA-en is modest and cannot account for the cross-lingual drop. The stratification analyses (Figures 3, 4, 8) further confirm that the gap is not concentrated in a particular question type or answer type but is pervasive.

The missing analysis: is the gap due to language or due to data? The paper attributes the gap to cross-lingual transfer difficulty, but an alternative hypothesis is that the gap is driven by differences in Wikipedia content across languages. If English Wikipedia articles are longer, better-structured, or more factually consistent than Hindi Wikipedia articles, then models might perform worse on Hindi simply because the contexts are harder to extract answers from, not because Hindi as a language is harder. The paper reports context lengths (Table 8: English mean 157.5 tokens, German 102.2, Chinese 222.9) but does not analyze whether context length or quality correlates with performance. A simple control — computing the correlation between per-language context length and per-language F1 — would partially address this, but is not reported.

Additionally, the paper does not benchmark a translate-documents baseline: machine-translate SQuAD test contexts and questions into English and evaluate the English model on them. This would isolate the contribution of MT quality to the performance gap. If English BERT-Large on translated SQuAD achieves, say, 70 F1 rather than 80, then ~10 F1 of the cross-lingual gap is attributable to MT noise, with the remainder attributable to genuine cross-lingual transfer difficulty. Without this baseline, the sources of the gap cannot be decomposed.

Claim: G-XLT enables decomposition of cross-lingual transfer

Supported as a proof of concept, but under-exploited. Tables 6 and 9 demonstrate that G-XLT is feasible and reveals interesting patterns (e.g., English questions work well across all context languages; Arabic contexts are hard regardless of question language). The paper uses these matrices primarily for qualitative observation — "English questions tend to perform best" (Section 5.2) — but does not systematically decompose the variance into question-language effects, context-language effects, and interaction effects. A simple ANOVA-style analysis: for each language pair, how much of the XLT performance is due to question understanding versus context understanding versus cross-lingual alignment? The G-XLT data contains this information but the paper leaves it implicit.

For example, the English-context column of Table 6 shows performance as a function of question language while holding context fixed. The drop from English questions (74.9) to Hindi questions (43.6) when contexts are English tells us: the model's Hindi question understanding is ~31 F1 worse than its English question understanding, even when reading English text. This is a "pure question understanding" estimate. The Hindi-context row shows performance as a function of question language while holding Hindi contexts fixed: the drop from English questions (59.6) to Hindi questions (48.8) is ~11 F1. Subtracting the pure question-understanding effect, this suggests that Hindi context understanding imposes roughly a 15 F1 penalty relative to English context understanding (74.9 - 59.6). These quantities are computable from the published matrices but are not reported or analyzed. The paper provides the data for this decomposition but does not perform it, leaving a significant analytical opportunity on the table.

Claim: Translate-Test and Translate-Train are competitive baselines

Supported with qualifications about MT quality dependence. The Translate-Test results (Table 5) show that for Spanish and German, MT pipelines are competitive with zero-shot XLM (65.4 vs. 68.0 for Spanish; 57.9 vs. 62.2 for German), while for Arabic and Chinese they are substantially worse (33.6 vs. 54.8; 44.2 vs. 61.1). This variation demonstrates that the competitiveness of MT-based approaches is highly language-dependent — they work for high-resource European languages with good MT but degrade sharply for Arabic and Chinese where MT quality is lower or the attention-based span mapping introduces errors. The paper does not report MT quality metrics (BLEU, COMET) for the translation models used, making it impossible to correlate MT quality with QA transfer performance or to predict which method will work better for a new language without running the experiment.

Translate-Train shows a similarly mixed pattern: it outperforms zero-shot XLM for Hindi (55.0 vs. 48.8 with M-BERT) and Vietnamese (62.0 vs. 61.4), but underperforms for other languages. The absence of a clear winner between Translate-Train and zero-shot is itself a finding — it means practitioners cannot simply default to "translate the training data" and must evaluate both approaches per language.

Missing experiments and analyses

Several experiments would substantially strengthen the paper's claims:

  • Controlled comparison with XQuAD. Since Artetxe et al. (2019) released XQuAD contemporaneously (1,190 SQuAD instances manually translated into 10 languages), evaluating the same models on XQuAD would calibrate MLQA results against an independently constructed benchmark. If the performance gap is similar on XQuAD, confidence in the generalizability of the findings increases. If it differs substantially, that would reveal benchmark-specific artifacts. The paper acknowledges XQuAD in the related work but provides no comparative evaluation.

  • Per-language hyperparameter tuning. The "strictly zero-shot" protocol is scientifically clean but practically pessimistic — real deployments would tune models using at least a small amount of target-language data. Reporting how much improvement comes from using the target-language development sets (which MLQA provides) would establish the benefit of "few-shot" transfer and help practitioners decide whether collecting a small validation set is worthwhile. The paper explicitly leaves this to future work (Section 6), but including even a single model with dev-set tuning would have strengthened the baseline landscape.

  • Error analysis beyond aggregate metrics. The manual analysis of XLM errors (Section 5.1) reports that 39% were completely wrong, 49% partially overlapped, 5% were annotation errors, and 7% were acceptable answers with no overlap. This breakdown is useful but shallow — it does not categorize why models fail (entity confusion, anaphora resolution failure, cross-lingual alignment error, morphological mismatch). A deeper error taxonomy would guide future model development more effectively than aggregate F1 numbers.

  • Effect of context length on performance. The paper reports mean context lengths (Table 8) and notes that MLQA contexts are 28% longer than SQuAD's on average, but does not analyze how performance varies with context length. If models disproportionately fail on long contexts, that would explain part of the SQuAD-to-MLQA gap and would suggest that context length normalization is needed for fair cross-lingual comparison.

  • Wikidata-based difficulty calibration. Using Wikidata to identify factual statements that should have the same answer across languages would provide an external difficulty control: if a question asks for a fact that Wikidata confirms is consistent across language editions, any failure to answer in a target language is unambiguously a model failure, not an alignment failure. This could calibrate the impact of imperfect parallel sentence alignment on reported scores.

What the experiments genuinely demonstrate versus what they do not

The experiments establish that (1) zero-shot cross-lingual transfer to extractive QA is possible but far from matching English performance, with a gap of ~21 F1 points for the best model, and that (2) the gap varies substantially by language, with Hindi and Arabic substantially harder than Spanish and German. The G-XLT results additionally show that (3) English questions transfer better than target-language questions, suggesting that the bottleneck is often context understanding rather than question understanding. The calibration analysis (Table 7) establishes that (4) the cross-lingual gap is not an artifact of MLQA being a harder dataset than SQuAD.

What the experiments do not establish is why the gap exists — whether it is due to linguistic distance, pretraining data quantity, morphological complexity, script differences, or dataset quality differences — or how much of the gap is irreducible (truly requiring target-language training data) versus addressable with better multilingual representations. The paper provides the measurement instrument for investigating these questions but does not itself provide the answers. This is consistent with the paper's stated contribution as a benchmark, but it means that the paper's impact depends on follow-up work that uses the benchmark to answer these causal questions.

The experiments also do not establish how well the findings generalize beyond the specific model families tested. M-BERT and XLM are the two dominant multilingual models of the era, but they share architectural assumptions (Transformer encoders, subword tokenization, Wikipedia pretraining). It remains unknown whether the same language ordering and G-XLT patterns would hold for models with different architectures (LSTMs, newer decoder-only architectures), tokenization strategies (character-level, morphological segmentation for morphologically rich languages), or pretraining data mixtures.

Concrete limitations

  • Test set size is modest. The development sets are small (~500 instances per language) and the test sets, while larger (4,500–5,500 per language), represent only a few thousand Wikipedia articles. The topical diversity analysis (Section 2.4: 500 randomly sampled articles) suggests reasonable coverage, but the per-language test sets effectively measure performance on ~2,500–2,700 unique articles — modest for a benchmark intended to support years of model development without overfitting.

  • Single training dataset (English SQuAD). All baselines use SQuAD v1.1 as training data. Using other English training datasets (NewsQA, Natural Questions) or non-English training data (CMRC for Chinese, FQuAD for French) would test whether the observed language ordering is specific to Wikipedia-domain English training or generalizes. The paper's decision to standardize on SQuAD is pragmatically justified (it is the largest and most standard dataset) but limits the generality of the baseline results.

  • No statistical significance reporting. The paper reports point estimates (F1 and EM scores) without confidence intervals or significance tests. At the per-language test set sizes (4,500–5,500 instances), differences of 1–2 F1 points may not be statistically significant. The comparison between XLM and M-BERT for Vietnamese (61.4 vs. 57.1 F1) is clearly meaningful; the comparison for Chinese (61.1 vs. 57.5) is likely also significant, but for smaller differences (e.g., Translate-Train vs. zero-shot for XLM on German: 61.4 vs. 62.2), the uncertainty matters.

  • Temporal validity. The models evaluated (M-BERT, XLM) represent the state of the art at the time of publication (2019–2020). The rapid pace of multilingual model development means that the specific performance numbers in Table 5 are already outdated, though the benchmark itself (as a measurement instrument) retains value. The paper's contribution is the benchmark, not the specific baseline scores, but readers should understand that the "large gap" described may have been substantially narrowed by subsequent models not evaluated in the paper.

6. Limitations and Trade-offs

Test Set Size Is Modest for a Long-Term Benchmark

The assumption or constraint. MLQA test sets contain 4,517–5,495 instances per target language and 11,590 in English (Table 2). The development sets are even smaller: ~500 instances per language. For a benchmark explicitly intended to "spur research" and serve as a "standardized, challenging, well-constructed evaluation dataset" analogous to SQuAD's role in English QA, the per-language instance counts are substantially smaller than other widely-used evaluation benchmarks (SQuAD v1.1 dev set: ~10,000 instances; the full SQuAD 2.0 test set exceeds 8,000).

The consequence. A test set of ~5,000 instances per language is sufficient for distinguishing large performance differences (e.g., the 13-point F1 gap between XLM and Translate-Test for Arabic in Table 5 is clearly meaningful) but provides limited statistical power for comparing methods that differ by small margins. When future models narrow the cross-lingual gap to within a few F1 points — a plausible trajectory given the rapid progress in multilingual pretraining — researchers will face uncertainty about whether small improvements reflect genuine progress or sampling noise. The problem becomes more acute when the test set is stratified for analysis: splitting 5,000 Hindi instances by five wh-word types (Figure 3) produces subsets of a few hundred instances each, where even modest F1 differences are unreliable. The two-fold cross-validation strategy described for the compute-optimal baseline paper would be even less stable here since the paper does not use it (all results are single-point estimates from the fixed test split).

What evidence exists in the paper. The paper does not compute confidence intervals, standard errors, or statistical significance for any of the reported F1 or EM scores. The per-language instance counts are reported in Table 2, but no discussion of their adequacy for discriminating between models at different effect sizes is provided. The paper does not address the tension between its goal as a long-term benchmark and the modest per-language scale — this is an unacknowledged rather than an acknowledged limitation.

Mitigation status. The paper does not acknowledge this as a limitation or suggest that the test set size may become a bottleneck as models improve. The acknowledgment in Section 6 that the development sets enable future few-shot experiments implicitly assumes the test sets are large enough to measure resulting improvements, but this assumption is not examined. No statistical framework (bootstrap confidence intervals, paired significance testing across parallel instances) is provided to help future users assess whether observed differences are reliable at the given sample sizes. This is a problem the paper's audience will need to solve themselves as they approach saturation on MLQA — and the paper's value proposition as a benchmark is precisely that it should remain useful across years of model development, which requires sufficient headroom in both performance and statistical resolution.

Parallel Sentence Alignment Is Imperfect — and the Imperfection Is Not Uniform Across Languages

The assumption or constraint. The entire MLQA pipeline depends on the assumption that the automatically identified parallel sentences express "the same or very similar meaning in multiple languages" (Section 2), which guarantees that questions answerable from the English parallel sentence are also answerable from the target-language parallel sentence. The pipeline explicitly relies on LASER's multilingual sentence embeddings to identify these parallel sentences, and the alignment is not manually verified beyond the implicit filter of translator "No Answer" annotations.

The consequence. The "No Answer" rate varies dramatically by language: 6.6% for Vietnamese vs. 21.9% for German (Section 2.3). This means that approximately one-fifth of German instances are discarded because the parallel sentence alignment failed — the English question is not actually answerable from the German context. These discarded instances are not random: they are systematically instances where the German Wikipedia content diverges from the English content in ways that LASER's similarity threshold did not detect. If unanswerable instances tend to involve more complex sentence structures, rarer entities, or less formulaic language than answerable instances, then the German test set is systematically easier than the English test set — it is missing the hardest-to-align instances. This introduces a selection bias that confounds cross-lingual comparison: higher German performance relative to Arabic (62.2 vs. 54.8 F1 for XLM zero-shot in Table 5) might partially reflect the German test set being an easier subset of the intended distribution rather than German being an inherently easier language for QA transfer.

The asymmetry of the bias matters: English retains all 12,738 instances, but each target language retains only the subset where alignment succeeded. Since alignment success is not random, per-language F1 scores are not directly comparable without adjusting for this selection effect — but no adjustment is provided.

What evidence exists in the paper. The "No Answer" rates per language are reported explicitly in Section 2.3 (6.6%–21.9%). The paper acknowledges that the "automatically aligned sentences occasionally differ in a named entity or information content" (Section 1, footnote 2) and that "the aligned sentences are not perfectly parallel" (Section 2.3). The per-language instance counts in Table 2 reflect the consequences: German has the fewest instances (5,029 including dev and test) despite having the second-largest pairwise parallel sentence count from LASER (5.4M English-German parallel sentences, Table 1) — the high discard rate offsets the larger alignment base. The paper releases the "No Answer" instances as a separate resource but does not analyze them for systematic differences from answerable instances.

Mitigation status. The paper acknowledges the imperfection but treats the release of unanswerable instances as a separate resource as a satisfactory mitigation. It does not analyze whether discarded instances differ in difficulty, length, topic, or linguistic properties from retained instances — the analysis that would be needed to bound the selection bias. It does not propose methods for adjusting cross-lingual comparisons to account for alignment-quality differences. This is a fundamental tension between the "natural documents" desideratum (which precludes manual translation or verification of all alignments) and the "parallel" desideratum (which requires genuinely comparable instances across languages) — and the paper does not fully resolve it.

Domain Generalization Is Untested Beyond Wikipedia/SQuAD

The assumption or constraint. MLQA is built entirely from Wikipedia, and all baselines are trained on SQuAD v1.1 — also Wikipedia-derived. The paper justifies this by the "Textual Domain" desideratum: Wikipedia "matches existing extractive QA training resources, in order to isolate the change in performance due to language transfer" (Section 2). The evaluation therefore measures cross-lingual transfer within the Wikipedia domain only.

The consequence. The paper's findings about cross-lingual transfer difficulty — the ~21 F1 gap for XLM zero-shot, the language ordering from Spanish to Hindi, the G-XLT patterns — are all conditioned on the Wikipedia domain. It is unknown whether these findings generalize to other domains: news (NewsQA, Trischler et al., 2017), general web search queries (Natural Questions, Kwiatkowski et al., 2019), or domain-specific corpora (biomedical, legal). Cross-lingual transfer difficulty might be substantially larger for domains that differ from Wikipedia's encyclopedic, fact-oriented, relatively formal writing style, particularly if those domains use language that is underrepresented in the Wikipedia-based pretraining data of M-BERT and XLM. Conversely, transfer might be easier for domains with more formulaic or template-driven language. The choice to standardize on Wikipedia is methodologically clean — it isolates language transfer from domain transfer — but this cleanness comes at the cost of not knowing how domain and language transfer interact.

A practitioner deploying a cross-lingual QA system on customer support tickets or medical records cannot infer from MLQA results whether their domain-specific transfer gap will be 21 F1, 40 F1, or 5 F1. The "isolate language transfer" strategy serves research goals but limits the benchmark's practical guidance value.

What evidence exists in the paper. The paper provides no multi-domain evaluation, no comparison with non-Wikipedia test data, and no analysis of whether the cross-lingual gap varies by topic or domain within Wikipedia itself. The topic analysis in Section 2.4 (500 randomly sampled articles: 23% people, 19% places, etc.) confirms topical diversity within Wikipedia but does not address cross-domain generalization. The fact that MLQA-en scores are similar to adjusted SQuAD scores (Table 7: BERT-Large 80.2 MLQA-en vs. 84.8 SQuAD*) confirms Wikipedia-Wikipedia consistency but says nothing about other domains.

Mitigation status. The paper does not claim cross-domain generalizability as a goal — the "Textual Domain" desideratum is explicitly about matching Wikipedia to SQuAD to isolate language effects. The limitation is not hidden; it is a design choice. However, the paper does not discuss what users should expect when applying models benchmarked on MLQA to non-Wikipedia domains, nor does it suggest that future work should extend the parallel-sentence mining approach to multi-domain corpora. The narrow domain scope is a known and acknowledged constraint, but its practical implications for the benchmark's utility are not fully explored.

Zero-Shot Protocol Underestimates Attainable Performance; No Few-Shot Baselines Are Provided

The assumption or constraint. All cross-lingual baselines use a "strictly zero-shot" evaluation: models are tuned on English development data only, and the same model checkpoint is evaluated on all seven target languages (Section 4). The paper explicitly notes that MLQA provides target-language development sets but that the baselines do not use them: "in our experiments, we only make use of the English development data and study strict zero-shot settings" (Section 6).

The consequence. The headline finding — "transfer results are significantly behind training-language performance" (Abstract), with a ~21 F1 gap for the best model — characterizes performance in the most pessimistic scenario: no target-language data of any kind is used for model selection or training. In practice, a team deploying a cross-lingual QA system would almost certainly use a small amount of target-language data for hyperparameter tuning, answer calibration, or lightweight fine-tuning. The paper's baseline numbers therefore represent a lower bound on what is practically achievable with existing methods, not a realistic estimate of the cross-lingual transfer difficulty facing a well-resourced practitioner.

The magnitude of this underestimation is unknown: for some languages and models, target-language dev tuning might recover 2–3 F1 points (modest); for others, it might recover 10–15 F1 points (large). The "significant room for improvement" claimed in the paper conflates two distinct sources of room for improvement: (1) the gap between zero-shot performance and what could be achieved with limited target-language resources, and (2) the gap between limited-resource performance and English-training performance. By not separating these, the paper provides an inflated estimate of how much fundamental modeling progress is needed.

What evidence exists in the paper. The paper acknowledges this explicitly in Section 6: "Other evaluation setups could be envisioned, e.g., by exploiting the target language development sets for hyper-parameter optimisation or fine-tuning, which could be fruitful for higher transfer performance, but we leave such 'few-shot' experiments as future work." The development set sizes (~500 instances per language) are reported in Table 2, confirming that few-shot experiments are feasible with the provided data. No few-shot baselines are reported.

Mitigation status. The paper frames the omission as intentional — "leave such 'few-shot' experiments as future work" — which is a reasonable scoping decision for an initial benchmark paper. However, the consequence is that the paper's most prominent quantitative claim (the size of the cross-lingual gap) is measured in a setting that the paper itself acknowledges is not the most practically relevant one. By not providing even a single few-shot reference point (e.g., "with 500 target-language dev instances, the gap closes from 21 F1 to X F1"), the paper leaves researchers without guidance on whether to invest effort in better zero-shot methods or in better sample-efficient adaptation methods. This is a missed opportunity given that the development sets are already part of MLQA and the few-shot baselines would be relatively cheap to run.

The Gap Is Measured but Not Explained — Causal Attribution Is Missing

The assumption or constraint. The paper's evaluation methodology measures whether a performance gap exists and how large it is, but does not causally attribute the gap to specific sources: linguistic distance from English, quantity of pretraining data per language, morphological complexity, script differences, Wikipedia content quality differences, or MT quality (for translation-based baselines). The paper treats language as a categorical variable and reports per-language scores (Table 5) without modeling what properties of those languages predict the scores.

The consequence. A reader who wants to know whether the cross-lingual gap will be larger or smaller for an eighth language not in MLQA — say, Turkish, Swahili, or Japanese — cannot extrapolate from the paper's results with any confidence. The language ordering (Spanish easiest, Hindi hardest) correlates plausibly with linguistic distance from English and with pretraining data quantity, but the paper does not test these correlations. Without causal attribution, the benchmark functions as a measurement instrument but not as a diagnostic instrument — it tells you that Hindi is harder than Spanish but not why, which limits its ability to guide model development. If the Hindi gap is primarily due to Devanagari script unfamiliarity, the solution is better script-agnostic tokenization; if it is due to morphological complexity, the solution is better subword modeling; if it is due to smaller Wikipedia coverage, the solution is additional pretraining data. MLQA scores alone cannot distinguish these hypotheses.

What evidence exists in the paper. The paper provides several stratification analyses that approach causal attribution without reaching it: question-type analysis (Figure 3 shows "When" questions are consistently easier), English difficulty stratification (Figure 4 shows the gap partially but not fully closes on questions the model answers correctly in English), named entity stratification (Figure 8 shows temporal entities are easier), and context length statistics (Table 8). These demonstrate that the benchmark can be used for attributional analysis, but the paper does not perform the cross-lingual attributional analysis that would explain the language ordering. There is no correlation reported between per-language F1 and any quantitative language property (typological distance, pretraining data volume, Wikipedia size or quality, MT BLEU scores for Translate-Test languages, or morphological complexity indices).

Mitigation status. This limitation is partially inherent to any benchmark paper: the purpose is to provide the measurement instrument, and causal analysis is for follow-up work using the instrument. The paper explicitly positions MLQA as "a powerful testbed for cross-lingual QA models" (Section 1) and provides the G-XLT task as a partial decomposition (separating question-language and context-language effects). But the paper could have strengthened its contribution substantially by providing an initial correlation analysis — even a simple table showing Spearman correlations between per-language F1 and readily available language properties would have transformed the benchmark from a pure measurement tool into an initial diagnostic framework, justifying the deliberate language selection (three scripts, three families) in terms of empirical outcomes rather than just a priori diversity goals.

Machine Translation Quality Is the Hidden Variable in Translation-Based Baselines

The assumption or constraint. The Translate-Train and Translate-Test baselines use Facebook's "production translation models" (Section 4) to either translate SQuAD training data into target languages or translate target-language test contexts/questions into English. The paper treats these as fixed, black-box components and does not report MT quality metrics (e.g., BLEU, COMET, or human evaluation) for any language pair. The assumption — implicit in treating the MT baselines as reference points — is that the translation quality is representative of what a practitioner could achieve with available MT systems and that conclusions drawn from the MT baselines are about the viability of the translation-based approach, not about the specific MT models used.

The consequence. The paper's findings about the competitiveness of Translate-Train vs. zero-shot transfer (Table 5) are confounded with MT quality. Translate-Train with M-BERT outperforms XLM zero-shot on Hindi (55.0 vs. 48.8 F1) but underperforms on Spanish (53.9 vs. 68.0). Is this because Translate-Train is a better approach for Hindi or because English-Hindi MT is better (relative to task difficulty) than English-Spanish MT? Without MT quality metrics, these explanations cannot be disentangled. A practitioner with access to a stronger MT system for Spanish might find Translate-Train outperforming zero-shot, reversing the paper's finding. The Translate-Test results for Arabic (33.6 F1) and Chinese (44.2 F1) are far below zero-shot XLM (54.8 and 61.1 respectively) — but if these languages had better MT (or if the attention-based span mapping were improved), the Translate-Test approach might be competitive. The paper's conclusion that "zero-shot XLM transfers best" (Section 1, summary) is accurate for the specific MT models used, but the ranking could shift with different MT systems.

The Translate-Test Hindi result (23.8 F1, 18.9 EM) is explicitly attributed to the unavailability of attention alignments, requiring a fallback to back-translation that does not produce valid answer spans. This is documented (footnote 5) but means the Hindi Translate-Test number is not comparable to other Translate-Test numbers — it measures a different (and broken) approach. Any summary statistic averaging across Translate-Test languages would be misleading because Hindi is measured with a different method.

What evidence exists in the paper. The paper does not report any MT quality metrics. The Translate-Test approach is described in Section 4, including the attention-based span mapping (Equation 1), but the quality of the underlying translation — which determines both whether the English QA model produces the correct answer and whether the attention mapping correctly localizes it — is not characterized. The "production translation models" are not identified by name, architecture, training data, or published performance. The paper acknowledges the Hindi Translate-Test quality issue explicitly (footnote 5 and the asterisk on the Hindi Translate-Test result in Table 5). The other MT baselines are presented without corresponding qualification about MT quality.

Mitigation status. The paper acknowledges that the Hindi Translate-Test result is unreliable due to missing attention alignments, but extends no analogous transparency to the other languages — the Chinese and Arabic Translate-Test results may similarly reflect MT quality limitations, but this is not discussed. The paper does not suggest that future evaluations on MLQA should report MT quality alongside Translate-Train/Translate-Test results to enable interpretation. This leaves the MT baselines as unreplicable reference points: another team cannot determine whether their Translate-Test result on German differs from the paper's because their QA model is better or because their MT system is different. The decision not to report MT metrics weakens the benchmark's value for comparing translation-based approaches, even though these approaches represent a major alternative to multilingual representation learning.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new modeling technique or advance state-of-the-art performance on an existing benchmark. Its contribution is infrastructural: it provides the measurement instrument that the cross-lingual QA field lacked, and in doing so, it changes what kind of research can be done and what questions can be asked. This is a foundational enabling contribution rather than a paradigm shift — it does not change how models work, but it changes how we evaluate whether they work across languages, and that change in evaluation practice has downstream effects on what the field prioritizes, measures, and values.

The shift from "does transfer work?" to "how does transfer vary, and why?" Prior to MLQA, cross-lingual QA research operated in a regime where the primary question was binary: can a model trained on English QA data answer questions in Language X? The evidence was fragmentary — a few hundred instances in Japanese and French (Asai et al., 2018, 327 instances), isolated two-language studies (Gupta et al., 2018 for English-Hindi; Jing et al., 2019 for English-Chinese), or distantly supervised data where noise confounded model errors with dataset errors (Liu et al., 2019a, XQA). These resources could answer whether transfer was possible at all, but not how much performance degraded relative to English, whether the degradation was uniform across languages or question types, or which sub-capabilities (question understanding, context understanding, cross-lingual alignment) were responsible. MLQA transforms the research question from a binary to a structured quantitative one: cross-lingual transfer is not a yes/no property but a continuous function of language pair, question type, and answer type, all measurable on a single, controlled benchmark. The G-XLT task in particular makes this shift concrete — by decoupling question language from context language, it enables questions about where in the processing pipeline transfer fails that were previously unanswerable. This is a reframing of the research agenda from existence proofs of cross-lingual capability to systematic characterization of cross-lingual transfer difficulty.

Reconciling contradictory evidence about multilingual representations. The paper's baseline results provide clarity on a question that, at the time of publication, was actively debated: do multilingual pretrained models (M-BERT, XLM) actually transfer to complex structured prediction tasks like extractive QA, or is their cross-lingual ability limited to classification tasks like NLI? The evidence had been mixed — M-BERT showed promising zero-shot transfer on NLI (Conneau et al., 2018, XNLI) and document classification (Schwenk and Li, 2018), but extractive QA requires fine-grained token-level alignment that is fundamentally more demanding than sentence-level classification. MLQA's baselines (Table 5) provide a clear answer: yes, zero-shot transfer to extractive QA works, with XLM achieving a mean F1 of 59.3 across six target languages, but the gap to English performance is large (15.6 F1 points below XLM's own English score, 20.9 below the English monolingual upper bound). More importantly, the G-XLT matrices (Tables 6 and 9) reveal that the transfer is asymmetric: English questions paired with target-language contexts perform nearly as well as target-language questions on the same contexts (e.g., with German contexts, English questions achieve 70.6 F1 vs. 62.2 for German questions), suggesting that the bottleneck is often context understanding in the target language, not the model's ability to align questions across languages. This reconciles the optimism from classification transfer results (representations do carry cross-lingual signal) with the caution from the large remaining gap (that signal is sufficient for classification but insufficient for high-accuracy span extraction). The paper resolves the apparent contradiction not by declaring one side correct but by showing that the answer depends on which sub-capability you measure — a nuance only accessible through the G-XLT decomposition.

Redirecting research attention from translation pipelines to multilingual representations. At the time of MLQA's publication, a practitioner building a cross-lingual QA system faced a strategic choice: should they invest in machine translation infrastructure (translating training data, translating queries at inference time) or in multilingual representation learning? Table 5 provides empirical guidance that was previously unavailable: for high-resource European languages with good MT (Spanish, German), the two approaches are competitive (Translate-Test with BERT-Large: 65.4 F1 on Spanish vs. XLM zero-shot: 68.0), but for languages with less mature MT (Arabic, Chinese) or where MT pipeline components are unavailable (Hindi attention alignments), zero-shot multilingual representations substantially outperform translation-based approaches (Arabic: 54.8 XLM vs. 33.6 Translate-Test; Chinese: 61.1 vs. 44.2). This finding — that the competitiveness of translation-based approaches is highly language-dependent and that multilingual models are more robust across languages — makes multilingual representation research more attractive as a general solution. It does not render MT-based approaches obsolete (Translate-Train with M-BERT still wins on Hindi, 55.0 vs. 48.8), but it shifts the default assumption from "translate everything to English" toward "train a multilingual model and evaluate whether MT helps on a per-language basis." This is a practical reframing of the deployment strategy that the baseline results make possible.

Establishing parallelism as a benchmark design requirement rather than a nice-to-have. The paper's five desiderata (Section 2) and the detailed description of how they were operationalized in the annotation pipeline provide a template for future cross-lingual benchmark construction that was not codified before. Prior work treated parallelism as incidental — manual translation of an English dataset (Asai et al., 2018; Artetxe et al., 2019) or independent dataset construction per language (Liu et al., 2019a) — rather than as a first-class design principle requiring explicit engineering. MLQA demonstrates that multi-way parallel evaluation data can be constructed at scale (46,000+ annotations) from naturally-occurring documents by constraining annotation to automatically-identified parallel sentences, and the paper is transparent about the cost-quality tradeoffs (the 7-way vs. 4-way tension, the per-language unanswerable rates). By making these tradeoffs explicit and providing a vocabulary for discussing them, the paper sets a new standard for what cross-lingual benchmarks should aspire to. The subsequent adoption of MLQA into aggregated multilingual suites (XGLUE, XTREME; Liang et al., 2020; Hu et al., 2020) confirms that the field recognized parallelism not as a luxury but as a requirement for rigorous cross-lingual evaluation. This is more than incremental — it changes the evaluation design conversation from "we assembled what data existed" to "we engineered data to answer specific scientific questions."

What this work does NOT change. It is equally important to be precise about the limits of this paper's impact. MLQA does not solve cross-lingual QA — the gap between English and target-language performance remains large (~21 F1 for the best zero-shot model) and the paper provides no method for closing it. It does not establish causal mechanisms — the benchmark reveals that Hindi is harder than Spanish but not why (linguistic distance? pretraining data quantity? morphological complexity? Wikipedia quality?). It does not provide a multi-domain evaluation framework — all results are on Wikipedia, and domain generalization remains untested. It does not establish that the specific baseline performance numbers are stable — the models tested (M-BERT, XLM) were current at publication but have since been superseded, and the paper provides no confidence intervals to quantify uncertainty around individual F1 scores. The contribution is measurement infrastructure, not solved science, and the paper's impact depends on the quality of research that this infrastructure enables, not on the specific baselines it reports.

Follow-Up Research This Work Enables

Systematically correlating cross-lingual transfer difficulty with measurable language properties. The paper establishes that the cross-lingual gap varies substantially by language (Spanish F1: 68.0 vs. Hindi F1: 48.8 for XLM zero-shot, a ~19-point spread) but does not attribute this variation to specific language properties. The MLQA dataset — with its seven deliberately diverse languages spanning three scripts and multiple language families, and its parallel structure that controls for question difficulty — provides the ideal testbed for such attribution. A strong follow-up would compute correlations between per-language transfer performance (the gap between English and target-language F1) and independently measurable language properties: typological distance from English (using features from WALS, the World Atlas of Language Structures), quantity of Wikipedia text available in pretraining data (e.g., number of tokens per language in M-BERT or XLM training corpora), morphological complexity metrics (e.g., type-token ratio, number of morphological forms per lemma), and script distance (measured by Unicode block overlap or tokenizer fertility). If morphological complexity explains most of the variance, the implication is that subword tokenization improvements will yield the largest gains; if pretraining data quantity dominates, the implication is that data augmentation or upsampling during pretraining is the priority. The analysis should also test whether the language ordering is stable across model architectures by evaluating newer multilingual models (XLM-R, mT5, multilingual decoders) on MLQA and checking whether the per-language ranking shifts — if Hindi-to-Spanish relative difficulty is constant across architectures, the effect is likely fundamental to the languages themselves; if it varies, the effect is architecture- or pretraining-data-dependent. This line of work would transform MLQA from a measurement instrument into a diagnostic instrument by connecting observed performance to actionable model design decisions.

Using G-XLT to diagnose whether transfer failures are in the encoder, the cross-lingual alignment, or the span prediction head. The G-XLT matrices (Tables 6 and 9) decompose cross-lingual performance into question-language and context-language effects, but the paper treats these matrices as descriptive rather than analytical. A targeted follow-up would use this decomposition to localize the source of transfer failures within the model architecture. The key manipulation: fine-tune a model on English SQuAD, then freeze different components (the multilingual encoder, the span prediction head) while fine-tuning others on a small amount of target-language data, measuring the G-XLT matrix after each intervention. If fine-tuning the encoder on target-language data substantially closes the context-language gap (the row-wise variation in Tables 6/9) but not the question-language gap, the bottleneck is in the encoder's target-language context representations. If fine-tuning the span prediction head closes both gaps equally, the bottleneck is in the task-specific layers failing to generalize across languages. A complementary approach: probe the internal representations at different encoder layers during G-XLT evaluation using canonical correlation analysis (CCA) or centered kernel alignment (CKA) to measure whether cross-lingual alignment degrades at specific depths. This would reveal whether cross-lingual transfer fails because the representations diverge early (suggesting better pretraining objectives are needed) or late (suggesting fine-tuning on English alone overwrites cross-lingual alignment, pointing to multi-task or adversarial fine-tuning strategies). MLQA's parallel structure makes this analysis possible because the same question-concept can be passed through the model in different language configurations, enabling controlled comparison of internal representations.

Quantifying the selection bias from unanswerable instances and correcting cross-lingual comparisons for alignment quality. The paper reports that "No Answer" rates range from 6.6% (Vietnamese) to 21.9% (German) and that these instances are removed from target-language test sets but remain in English (Section 2.3). This introduces an unquantified selection bias: if unanswerable instances differ systematically from answerable ones in difficulty, the per-language F1 scores are not directly comparable. A rigorous follow-up would characterize the discarded instances by (1) computing the XLM English F1 on the subset of English questions whose target-language counterparts were discarded for each language, comparing this to the F1 on questions that survived; (2) measuring linguistic properties of the aligned sentences that were discarded (sentence length, lexical diversity, syntactic complexity via parse tree depth) versus those retained; and (3) testing whether the observed language ordering (Spanish > German > Vietnamese > Chinese > Arabic > Hindi) persists after restricting all languages to only the instances that are parallel across all seven (the 1,340 7-way aligned sentences identified in Table 10, though these lack diversity). The crucial output would be a corrected per-language F1 score that estimates what performance would have been on the full English test set, had no instances been discarded. If the correction is small (1–2 F1 points), the qualitative conclusions are robust; if the correction is large and language-dependent (e.g., German F1 drops by 5 points after correction, Vietnamese by 1), the paper's headline language ordering needs revision. This work would also inform future benchmark construction by establishing how much alignment quality matters for cross-lingual fairness, providing empirical justification for the parallelism/diversity tradeoff the paper discusses qualitatively.

Extending the parallel-sentence annotation pipeline to new domains and low-resource languages. MLQA is restricted to Wikipedia — a pragmatic choice justified by the "Textual Domain" desideratum, but one that leaves domain generalization untested. A domain-extension follow-up would apply the parallel-sentence mining and annotation methodology to a non-Wikipedia domain where parallel text naturally exists: legal documents (EU legislation exists in 24 languages), biomedical abstracts (many journals require English abstracts alongside native-language ones), or software documentation (manuals translated into multiple languages). The key research question is whether the cross-lingual transfer gap measured on Wikipedia generalizes — is the Spanish-to-Hindi F1 difference similar for legal QA, or do domain-specific properties (terminology, formality, syntactic templates) change the relative difficulty ordering? A more ambitious extension would target genuinely low-resource languages (Swahili, Quechua, Amharic) where Wikipedia parallel sentences are too sparse for the MLQA pipeline. This would require developing alternative parallelism sources: parallel newswire (e.g., Global Voices, which translates news into many languages), religious texts (structurally parallel across hundreds of languages), or translated educational materials. The key methodological contribution would be adapting the answer-in-parallel-sentence constraint to domains where the "parallel sentences" are less cleanly aligned than Wikipedia — requiring new automatic alignment quality thresholds or active learning approaches where annotators flag alignment failures. Success would demonstrate that the MLQA approach generalizes beyond its original domain and language set, while failure would establish the boundary conditions on when parallelism-based annotation is viable.

Characterizing the cost-performance frontier between zero-shot transfer and supervised target-language training. The paper evaluates two extremes: strict zero-shot transfer (no target-language data used for training or tuning) and full Translate-Train (the entire SQuAD corpus machine-translated into the target language). The practically relevant space between these extremes — using 10, 100, or 1,000 target-language training instances — is unexplored. A sample-efficiency follow-up would use MLQA's target-language development sets (~500 instances per language) and test sets to plot learning curves: fine-tune a pretrained multilingual model (XLM, XLM-R, or a modern equivalent) on increasing numbers of target-language QA instances (sampled from a training dataset in that language, or from the MLQA dev set via cross-validation) and measure F1 on the MLQA test set. The key output would be a per-language estimate of how many target-language training instances are needed to match zero-shot XLM performance and how many are needed to close half the gap to English performance. For languages where the zero-shot baseline is already strong (Spanish), the number might be low — a few hundred instances suffice. For languages where zero-shot is weak (Hindi, Arabic), the number might be much larger. This analysis directly addresses the paper's acknowledged limitation (Section 6: "we leave such 'few-shot' experiments as future work") and provides actionable guidance for practitioners deciding whether to invest in target-language annotation or to rely on zero-shot transfer. It would also reveal whether the zero-shot language ordering (Spanish easiest, Hindi hardest) is preserved under few-shot fine-tuning, or whether some languages benefit disproportionately from small amounts of supervision.

Adversarial evaluation of cross-lingual alignment quality. The G-XLT results show that English questions paired with target-language contexts often outperform target-language questions on the same contexts (e.g., with German contexts: English questions 70.6 F1 vs. German questions 62.2; Table 6). This suggests that the model's target-language question understanding is a bottleneck for some languages. An adversarial follow-up would systematically degrade question quality — introducing grammatical errors, replacing content words with synonyms, scrambling word order — in both English and target languages and measuring the differential impact on G-XLT performance. The hypothesis: if cross-lingual alignment is genuinely robust, degrading English questions should affect target-language contexts and English contexts equally; if alignment is shallow (relying on keyword matching rather than semantic understanding), degrading English questions should disproportionately hurt transfer to target-language contexts where keyword overlap is absent. A complementary adversarial test would use code-switched questions (mixing English and target-language tokens) and measure whether performance interpolates between the English-question and target-language-question conditions, which would test whether the model represents questions in a language-agnostic way or processes each language independently before aligning. MLQA's parallel structure is essential for this work because it guarantees that question content is identical across the English and target-language versions, isolating the effect of language form from content — without this control, any difference could be attributed to question content differences rather than language processing differences.

Practical Applications and Downstream Use Cases

Multilingual information access without per-language training data. The paper's headline finding — that XLM zero-shot achieves reasonable extractive QA performance across six target languages without any target-language training data — directly enables deployment of QA systems for languages where annotated training data does not exist. Consider a digital library or knowledge base with documents in 50 languages but QA training resources in only English. MLQA's baselines demonstrate that a single multilingual model trained on English SQuAD can answer factoid questions from documents in Spanish (68.0 F1), German (62.2 F1), Vietnamese (61.4 F1), and Chinese (61.1 F1) at usable accuracy levels, without any per-language engineering or data collection. The G-XLT results further suggest that a practical system could accept user questions in English regardless of the document language — English questions paired with German contexts achieve 70.6 F1, higher than German questions on German contexts (62.2 F1) — meaning that English-speaking users of a multilingual search system do not need to translate their queries. For languages where zero-shot performance is weaker (Arabic: 54.8 F1, Hindi: 48.8 F1), the Translate-Train results point to a practical fallback: machine-translating SQuAD into those languages and training a dedicated model (achieving 55.0 F1 for Hindi with M-BERT Translate-Train, 54.0 F1 for Arabic with XLM Translate-Train). The specific performance numbers provide deployment teams with per-language accuracy expectations, enabling cost-benefit decisions about whether the zero-shot accuracy is sufficient for their use case or whether investment in language-specific training data is warranted.

Prioritizing language coverage investments in multilingual product development. For organizations building multilingual products (search engines, voice assistants, enterprise QA systems), a recurring resource-allocation question is: which languages should receive investment in training data collection, and which can be handled acceptably by zero-shot transfer from English? The MLQA baseline results provide an empirical basis for this decision. The language ordering by zero-shot F1 (Spanish: 68.0 > German: 62.2 > Vietnamese: 61.4 > Chinese: 61.1 > Arabic: 54.8 > Hindi: 48.8) identifies three tiers: (1) languages where zero-shot performance may be acceptable as-is (Spanish, German), (2) languages where zero-shot is borderline and moderate investment in training data or tuning could close the gap (Vietnamese, Chinese), and (3) languages where zero-shot is insufficient and significant investment is required (Arabic, Hindi). Translate-Train results further refine this: Hindi benefits more from translated training data (+11.2 F1 for M-BERT Translate-Train over M-BERT zero-shot) than Spanish does (-10.4 F1 for the same comparison, where translation noise hurts more than target-language data helps), suggesting that translation-based data augmentation is a better investment for lower-resource languages than higher-resource ones. This tiered prioritization framework, grounded in specific F1 numbers rather than intuition about linguistic distance, enables evidence-based language coverage planning that minimizes wasted investment in languages where zero-shot is already good while targeting resources to languages where the gap is largest and the return on investment is highest.

Cross-lingual fact verification and knowledge base population. Extractive QA is a building block for higher-level information extraction tasks: fact verification (does this claim match the source text?), knowledge base population (what entity holds what relation?), and multi-document summarization. The G-XLT capability demonstrated in MLQA — answering questions about a document in Language A using questions in Language B — has direct application to cross-lingual fact-checking. A fact-checking system monitoring claims in Hindi could use English-language questions (where QA models are strongest) to query Hindi-language source documents and verify whether the claim is supported. The G-XLT matrix (Table 6) provides per-language-pair accuracy estimates for this configuration: English questions on Hindi contexts achieve 59.6 F1 with XLM, compared to 48.8 F1 with Hindi questions on Hindi contexts — the cross-lingual approach is actually more accurate than the monolingual one. This counterintuitive result has practical consequences: for low-resource languages where question understanding is a bottleneck, deploying an English-language interface to a multilingual QA backend may outperform translating user questions into those languages and running monolingual QA. The MLQA results provide quantitative justification for such an architecture, with specific F1 estimates per language pair.

When to Prefer This Method

The paper does not propose a method — it provides a benchmark. MLQA is an evaluation framework, not a modeling approach, and the baseline experiments use existing methods (M-BERT, XLM, Translate-Train, Translate-Test) without claiming superiority for any of them. The question of "when to prefer this method over alternatives" therefore applies not to MLQA itself but to the evaluation strategy it enables: when should researchers or practitioners use MLQA rather than other cross-lingual QA benchmarks to evaluate their models?

The paper does not explicitly articulate such a decision rule, but the evidence and desiderata imply the following practical guidance, grounded in MLQA's specific properties:

  • Prefer MLQA when the research question requires controlled, per-instance cross-lingual comparison — e.g., "does my model perform equally well on the same question in Spanish and Hindi?" or "how does my model's question-understanding vs. context-understanding vary by language?" These questions require parallel data (MLQA's defining feature) and cannot be answered with independently-constructed per-language datasets (XQA, Liu et al., 2019a) where the question distribution differs by language, or with manually-translated English datasets (XQuAD, Artetxe et al., 2019) where the number of instances is small (~1,190) but the translation quality is high.

  • Prefer MLQA when the goal is to evaluate zero-shot transfer from English-trained models to genuinely natural target-language text, not translated text. The manual translation used in XQuAD produces translationese — text that retains English-like syntactic patterns — which may inflate transfer performance for models that exploit English-pattern recognition. MLQA's Wikipedia-sourced contexts, while imperfectly natural (some are volunteer translations), are closer to independently-produced target-language text. For research on cross-lingual robustness to natural language variation, this distinction matters.

  • Prefer XQuAD or a cross-benchmark evaluation when the goal is to test whether findings generalize beyond Wikipedia domain. MLQA's exclusive Wikipedia focus isolates language transfer from domain transfer, which is methodologically clean but narrow. If your model might overfit to Wikipedia's factual, encyclopedic style, evaluating on both MLQA (Wikipedia, 5K+ instances per language, parallel) and XQuAD (Wikipedia translated, 1,190 instances, manually verified) provides convergent evidence — consistent results across both benchmarks strengthen claims; divergent results reveal domain or construction-artifact sensitivity.

  • Prefer a cross-benchmark evaluation (MLQA + XQuAD + individual monolingual datasets) when comparing your model against published numbers from different eras. The baselines in this paper use M-BERT and XLM (circa 2019–2020), and the rapid advance of multilingual pretraining means those specific numbers are outdated. Running on multiple benchmarks, including those used in more recent papers, provides more informative historical comparison than MLQA alone, since MLQA's baseline scores capture a specific moment in model development.

The salient point is that MLQA is an evaluation instrument, not a modeling prescription. The "prefer when" logic applies to evaluation design, not to model or deployment strategy selection. The paper provides no evidence or guidance about when to prefer multilingual representations over translation pipelines, zero-shot over few-shot, or extractive QA over alternative QA paradigms — and it explicitly scopes those questions as future work (Section 6). Any decision rule about modeling approaches would be extrapolation beyond what the paper supports.