Crosslingual Generalization through Multitask Finetuning

🎯 Pitch

Multilingual models finetuned only on English tasks can suddenly perform those tasks in dozens of other languages—even ones never seen during training—because they absorb language-agnostic capabilities from incidental pretraining data.

1. Executive Summary

This paper studies how multitask prompted finetuning (MTF) on multilingual datasets enables zero-shot generalization to new tasks and languages in pretrained multilingual language models. Applying MTF to BLOOM and mT5 model families—producing BLOOMZ and mT0—the authors investigate three prompting strategies: English-only finetuning on P3, multilingual data with English prompts (xP3), and multilingual data with machine-translated prompts (xP3mt). The key finding is that English-only finetuning suffices to improve task performance on non-English languages that appeared only during pretraining, but training on multilingual data with English prompts (xP3) yields additional gains and achieves state-of-the-art zero-shot results. Finetuning on machine-translated prompts (xP3mt) significantly boosts performance on human-written non-English prompts at test time—for example, raising BLOOMZ's average accuracy on human-translated XNLI prompts from 40.4 to 43.88—while surprisingly, models also demonstrate zero-shot generalization to tasks in languages they were never intentionally exposed to during pretraining or finetuning, establishing that the models acquire higher-level capabilities that are both task- and language-agnostic only when the base pretraining corpus contains incidental traces of those languages.

2. Context and Motivation

The Core Problem: Can Multilingual Models Generalize Across Tasks and Languages Without Language-Specific Finetuning?

The fundamental question this paper tackles is whether multitask prompted finetuning (MTF)—training a language model on a diverse set of tasks expressed in natural language—can enable zero-shot generalization not just to new tasks, but to new languages as well. Prior work had established that MTF on English tasks helps English-centric models handle held-out English tasks. But for the vast multilingual landscape, the practical question is far more urgent: if you finetune a multilingual model on tasks in a few languages (or even just English), can it then perform those same types of tasks in dozens of other languages without any additional training data in those languages?

This matters because collecting supervised task data is expensive, time-consuming, and often infeasible for the roughly 7,000 languages spoken worldwide. The paper frames this explicitly in the introduction:

"Due to the difficulty of collecting supervised task data in low-resource languages, previous work typically aims to transfer capabilities learned from finetuning on English data, which can improve performance on non-English language tasks."

If MTF on English data alone can unlock task capabilities across languages, it would dramatically lower the barrier to deploying functional NLP systems for low-resource languages. You would only need to curate training tasks in English—a high-resource language with abundant labeled data—and the model would transfer those capabilities to languages where annotation is impossible. Conversely, if English-only finetuning proved insufficient, the field would need to confront the daunting challenge of collecting multilingual supervised data at scale.

The Gap: Prior MTF Work Was Monolingual

Multitask prompted finetuning emerged as a powerful technique for improving zero-shot task generalization, but the paper identifies a critical gap: essentially all prior work focused on English. The authors cite the lineage clearly:

"Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models."

The key references are Wei et al. (2021) (FLAN), Sanh et al. (2022) (T0), and related work like Min et al. (2021) and Chung et al. (2022). All these demonstrated that finetuning on collections of prompted English tasks—summarization, question answering, sentiment analysis, etc.—improves performance on held-out English tasks the model was never explicitly trained for. T0, for example, took a pretrained T5 model, finetuned it on the P3 dataset (a collection of English tasks with English prompts from PromptSource), and showed strong zero-shot generalization to unseen English tasks. But T0 was a monolingual English model trained on monolingual English data. Whether the same approach would transfer across languages using a multilingual base model was unexplored.

The paper's position is that this gap is both scientifically important—it tests whether task knowledge and language knowledge are separable in large models—and practically urgent—it determines whether the MTF paradigm can serve the multilingual world.

Where Prior Multilingual Approaches Fall Short

Before this work, the standard approach for getting multilingual models to perform tasks was task-specific or language-specific finetuning. As the authors note:

"zero-shot performance tends to be significantly lower than finetuned performance. Thus, task-specific or language-specific transfer learning via finetuning remains the predominant practice."

This meant that if you wanted a model to perform sentiment analysis in Swahili, you needed labeled sentiment data in Swahili—a classic low-resource bottleneck. Various cross-lingual transfer methods existed (e.g., training on English and evaluating on other languages, as in Wu and Dredze, 2019; Phang et al., 2020; Chalkidis et al., 2021; Vu et al., 2022), but these typically involved task-specific finetuning: you still needed to train on the exact task you wanted to transfer, just in a different language. The gap was in zero-shot task generalization across languages—performing entirely new tasks, never seen during finetuning, in languages that may have only appeared during pretraining.

On the multilingual model side, several large-scale efforts had emerged. Decoder-only models like XGLM (Lin et al., 2021) showed that cross-lingual few-shot prompting could work: using English prompts to solve tasks in other languages. In fact, XGLM found that prompting in English often outperformed human-translated prompts for non-English tasks. Encoder-decoder models like mT5 (Xue et al., 2020) and mBART (Liu et al., 2020) were pretrained on multilingual corpora but required downstream finetuning. BLOOM (Scao et al., 2022a) was pretrained on the ROOTS corpus spanning 46 natural languages and 13 programming languages, offering a decoder-only model with broad language coverage.

The paper positions itself at the intersection of these two lines of work: applying the MTF recipe (from the English T0/FLAN tradition) to these multilingual pretrained models (BLOOM and mT5), and then testing whether the resulting model—BLOOMZ or mT0—could generalize zero-shot to tasks in languages never seen during finetuning.

Unresolved Questions the Paper Sets Out to Answer

From this gap, several specific, nested questions emerge that structure the paper's investigation:

First, does English-only MTF suffice? If you take a multilingual model pretrained on 46 languages and finetune it only on English tasks with English prompts (the P3 dataset), does its performance improve on non-English held-out tasks? This tests whether task knowledge acquired in English transfers to other languages through the shared multilingual representations learned during pretraining. If the answer is yes, it has major practical implications: you can collect task data in English alone and deploy across languages.

Second, does adding multilingual task data help further? The paper created xP3—an extension of P3 that adds tasks in 46 languages (translation, simplification, program synthesis, etc.) but keeps all prompts in English. Does finetuning on this richer mixture yield better English and non-English performance than English-only finetuning? This tests whether multilingual task diversity during finetuning is beneficial even when the task instructions remain in English.

Third, does prompt language matter? The paper created xP3mt—a version of xP3 where prompts are machine-translated from English to match the language of each dataset. If you finetune on non-English prompts, does performance improve when tested on human-written non-English prompts? And does this come at a cost to English-prompt performance? This addresses a practical deployment question: should you invest in translating prompts to each language you want to support?

Fourth, and most provocatively, can models generalize to languages they've never intentionally seen? This emerged as a surprising finding during the research. The authors evaluated on languages that were not part of BLOOM's intended 46-language pretraining corpus, and found above-random performance. This tests the limits of language-agnostic capability learning and raises questions about what "unseen" really means in the era of web-scale pretraining data.

The Contamination Puzzle

A crucial nuance that the paper confronts head-on: in large-scale pretraining, "unseen" is a slippery concept. The ROOTS corpus, despite being curated from 46 languages and 13 programming languages, unintentionally contains sentences from many other languages due to code-mixing, code comments in different languages, or misclassified documents. The paper conducts a detailed contamination analysis (Appendix D, Figure 11), sampling 51 million documents from ROOTS and re-identifying the actual languages using cld3. They find that languages like Japanese (0.54%), Italian (0.28%), German (0.21%), Russian (0.03%), Thai (0.006%), and even Burmese (0.00003%) appear as "contamination" in the corpus.

This creates a fascinating experimental condition. These languages were never intentionally included—no curator said "let's collect Japanese data"—but they are present in trace amounts. When the paper reports that BLOOMZ generalizes to tasks in Japanese or Thai, it is not testing generalization to a truly novel language in the strict sense. The model has seen some Japanese text, however inadvertently. The authors are transparent about this:

"It is likely that the training data unintentionally includes small fractions of these languages (just as many tasks might appear 'implicitly' in the pretraining corpus)."

This framing is important because it shifts the question from "can the model learn a new language from scratch through task transfer?" to "can the model leverage tiny amounts of incidental exposure to a language, combined with task knowledge from other languages, to perform above random on held-out tasks?" The answer appears to be yes, which has implications for how we think about data curation and the robustness of multilingual representations.

Positioning Relative to mTk-Instruct

The paper explicitly positions itself relative to one closely related contemporary work: mTk-Instruct (Wang et al., 2022c). Both papers train multilingual models on prompted datasets. The key distinctions, as framed by the authors:

"In contrast with Wang et al. (2022c), our sole focus is crosslingual zero-shot generalization. Therefore, we consider a wider variety of prompting settings and perform a more detailed evaluation of multilingual capabilities."

mTk-Instruct used structured prompts with "Definition," "Input," and "Output" fields—a more rigid format. This paper follows the T0/PromptSource tradition of natural, human-written prompts. This difference matters: the authors observe that mTk-Instruct-13B (3.7B parameters) performs significantly worse than the same-sized mT0 on their evaluation, and attribute this to prompting style. The natural-prompt approach is hypothesized to better support generalization because it more closely resembles how humans would actually query the model.

The Scaling Dimension

Finally, the paper is motivated by a scaling question: does the gap between pretrained-only and MTF models grow with model size? Prior work in the English setting (Sanh et al., 2022; Chung et al., 2022) suggested that larger models benefit more from MTF. The paper tests this across BLOOM models from 560 million to 176 billion parameters and mT0 models from 300 million to 13 billion parameters, asking whether multilingual MTF follows the same scaling pattern and whether the benefits extend across all languages represented in the training data.

In summary, the paper addresses a gap at the intersection of two active research areas—multitask prompted finetuning and multilingual modeling—that had previously been explored independently. Its core motivation is both practical (enabling zero-shot task transfer to low-resource languages without per-language annotated data) and scientific (understanding whether task knowledge and language knowledge can be disentangled and separately transferred in large neural models). The paper's approach is systematic: rather than proposing a new technique, it thoroughly ablates the language composition of the finetuning data and the language of the prompts, providing a comprehensive empirical picture of what drives cross-lingual task generalization.

3. Technical Approach

This is primarily an empirical analysis paper whose core idea is that the language composition of both the finetuning data and the prompts determines how well a multilingual model generalizes zero-shot to new tasks in new languages — and that by carefully constructing these mixtures, you can unlock task generalization to languages never seen during finetuning, including languages that appeared only incidentally in the pretraining corpus.

3.1 Reader Orientation

The paper builds a data pipeline and finetuning procedure that converts pretrained multilingual models into models that can perform entirely new tasks in dozens of languages without any per-language or per-task annotated data. The core problem it solves is: given a pretrained multilingual model (BLOOM or mT5), what mixture of prompted tasks and what language of prompts should you finetune on to maximize zero-shot performance on held-out tasks and held-out languages? The solution is a systematic comparison of three finetuning data variants — English-only (P3), multilingual data with English prompts (xP3), and multilingual data with machine-translated prompts (xP3mt) — applied to both decoder-only and encoder-decoder architectures across a range of model sizes.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Pretrained multilingual base models (BLOOM, 560M–176B parameters; mT5, 300M–13B parameters) — the starting point, pretrained on multilingual corpora without task-specific supervision.
Prompted finetuning datasets (P3, xP3, xP3mt) — collections of supervised NLP datasets converted into natural language input–target pairs via templates. P3 is English-only; xP3 extends P3 with multilingual datasets using English prompts; xP3mt further translates prompts to match dataset languages.
Finetuning procedure — continued training of the base models on these prompted datasets for 1–13 billion additional tokens, with specific loss normalization and packing strategies, producing the finetuned variants (BLOOMZ, BLOOMZ-P3, BLOOMZ-MT; mT0, mT0-P3, mT0-MT).
Zero-shot evaluation — testing on held-out tasks (coreference resolution, sentence completion, NLI, code generation) in held-out languages using untuned prompts, with no additional finetuning or few-shot examples.

Information flows as follows: a pretrained model → gets finetuned on a prompted dataset variant (P3, xP3, or xP3mt) with loss computed only on target tokens → the resulting checkpoint is evaluated zero-shot on held-out tasks in multiple languages using English, machine-translated, or human-translated prompts → performance is compared across finetuning variants, model sizes, and language groups.

3.3 Roadmap for the Deep Dive

First, the three finetuning dataset variants (P3, xP3, xP3mt) — what they contain, how they differ, and why each variant tests a specific hypothesis about cross-lingual transfer.
Second, the base models (BLOOM and mT5) — their architectures, pretraining corpora, and how their differences enable comparison between decoder-only and encoder-decoder approaches.
Third, the finetuning procedure — the training hyperparameters, loss normalization, sequence packing, language distribution matching, and checkpoint selection strategy.
Fourth, the evaluation protocol — how zero-shot performance is measured using rank classification for multiple-choice tasks and generative metrics for open-ended tasks, why the same 5 randomly selected prompts are used across all language splits without tuning, and how held-out tasks and held-out languages are defined.
Fifth, the contamination analysis — how the paper probes the pretraining corpus to distinguish genuinely unseen languages from incidentally included ones.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 The Three Finetuning Dataset Variants: P3, xP3, xP3mt

The paper constructs three prompted finetuning datasets, each designed to test a specific hypothesis about what drives cross-lingual task generalization.

P3 (English data, English prompts): This is the existing dataset from Sanh et al. (2022), constructed using PromptSource (Bach et al., 2022). P3 consists exclusively of English-language NLP datasets paired with multiple English-language prompt templates per dataset. Each task in P3 belongs to a task taxonomy that groups datasets by the underlying capability they require (e.g., sentiment analysis, question answering, summarization). Some prompts intentionally invert the task — for example, a prompt for closed-book QA might ask the model to generate a question given an answer rather than answering a given question. This inversion forces the model to learn the relationship between inputs and outputs rather than memorizing a specific format. P3 contains no non-English data and no non-English prompts. Finetuning on P3 tests the hypothesis: does task knowledge acquired in English transfer across languages through shared multilingual representations?

xP3 (multilingual data, English prompts): This is the paper's primary contribution to the finetuning data landscape. xP3 extends P3 by adding 30 new multilingual datasets while keeping all prompts in English. The authors define four new task clusters previously absent from P3: translation, simplification, program synthesis, and miscellaneous code tasks. The code tasks are particularly motivated by BLOOM's pretraining: since 11% of BLOOM's pretraining data is code, adding code tasks to the finetuning mixture ensures alignment between the model's pretraining knowledge and its finetuning objectives. The translation tasks use data from Flores-200 (NLLB Team et al., 2022) and other sources. The simplification tasks use the BiSECT dataset (Kim et al., 2021) with splits for multiple languages.

A critical design choice in xP3 is the language distribution: xP3 aims to replicate the language distribution of ROOTS, the corpus used to pretrain BLOOM. ROOTS contains 46 natural languages in varying proportions: English makes up 30%, with Spanish, French, and other high-resource languages following. xP3 is 39% English — slightly more than ROOTS' 30%, but broadly aligned. The tail of xP3's distribution includes low-resource African languages such as Twi (tw), Bambara (bm), and Lingala (ln), which appear solely through the Flores-200 translation task. Some of these languages (Twi, Bambara) are not even part of mT5's pretraining corpus — creating an interesting experimental condition where mT0 models finetuned on xP3 encounter these languages for the first time during finetuning. Figure 2 visualizes the language distributions of ROOTS, xP3, and the mT5 corpus side by side.

The prompt construction for xP3 follows the PromptSource paradigm: each dataset has multiple human-written prompt templates in English, even for non-English datasets. For example, a Spanish sentiment analysis sample would have the Spanish input text paired with an English prompt like "Is the following review positive or negative?" and an English target like "positive." Figure 3 contrasts this with other variants: in xP3, the input is in Spanish, the prompt is in English, and the target is in English (e.g., "Yes"/"No" choice labels). The Chinese dataset C3 (Sun et al., 2020) and the XWinograd extension with CLUE (Xu et al., 2020) winograd schemas are also included, with the latter expanded from 16 to 504 Chinese samples.

Finetuning on xP3 tests the hypothesis: does adding multilingual task data — even with English prompts — improve zero-shot performance on both English and non-English tasks compared to English-only finetuning?

xP3mt (multilingual data, machine-translated prompts): This variant takes xP3 and translates the prompts for monolingual datasets into the respective dataset language using the Google Cloud Translation API. For cross-lingual datasets (e.g., WikiLingua, where the task involves producing a summary in one language from text in another), prompts remain in English. Figure 3 illustrates the effect: a PAWS-X Spanish sample in xP3 has an English prompt ("Question: Can we rewrite Sentence 1 to Sentence 2? Yes or No?"), while in xP3mt the same sample has a Spanish prompt ("Pregunta: ¿La oración 1 parafrasea la oración 2? ¿Si o no?"). The choice labels are also translated (e.g., "No, Yes" → "No, Si").

The machine translation is applied only to prompts, not to dataset inputs or targets. The rationale, as expressed in the paper, is that training on non-English prompts should better prepare models for real-world usage where users naturally prompt in their own languages. This variant tests: does finetuning on machine-translated prompts improve performance on human-written non-English prompts, and does it come at a cost to English-prompt performance?

The structural difference between the three variants is clean: P3 is English-only in both data and prompts; xP3 is multilingual in data but English-only in prompts; xP3mt is multilingual in both data and (non-crosslingual) prompts. By comparing models finetuned on these three datasets, the paper can disentangle the effects of multilingual data diversity from prompt language transfer.

3.4.2 Base Models: BLOOM and mT5

The paper finetunes two model families to test whether the findings are architecture-agnostic.

BLOOM (Scao et al., 2022a,b): BLOOM is a family of decoder-only transformer language models ranging from 560 million to 176 billion parameters. The architecture resembles GPT-3 (Brown et al., 2020): causal (autoregressive) attention, meaning each token can only attend to previous tokens. BLOOM was pretrained on the ROOTS corpus (Laurençon et al., 2022), which spans 46 natural languages and 13 programming languages, for approximately 366 billion tokens. The language distribution of ROOTS is visualized in Figure 2, with English at 30%, French at ~15%, Spanish at ~13%, and a long tail of lower-resource languages. The pretraining objective is standard next-token prediction: given a sequence of tokens, predict each subsequent token.

As a decoder-only model, BLOOM processes inputs and generates outputs in a single contiguous stream. At finetuning time, the input (prompt + dataset instance) and target (desired output) are concatenated with a space separator, and the model learns to continue generating from the input to produce the target. This means the model must learn to distinguish the end of the input from the beginning of the target based solely on the natural flow of text — there is no architectural boundary between the two. The paper notes that this "unnatural prompting format" can cause issues: for example, given "Translate to English: Je t'aime", the model might continue with additional French text before beginning the translation, because it hasn't learned a strong input-target boundary signal.

mT5 (Xue et al., 2020): mT5 is a family of encoder-decoder transformer models ranging from 300 million to 13 billion parameters. The architecture follows T5 (Raffel et al., 2020): an encoder processes the full input bidirectionally (each token attends to all other input tokens), and a decoder generates the output autoregressively, attending to both the encoder output and previously generated tokens. mT5 was pretrained on mC4, a multilingual version of the C4 corpus, sampled to cover 101 languages. The pretraining objective is masked language modeling (specifically, span corruption): spans of the input are replaced with sentinel tokens, and the decoder is trained to reconstruct the original spans. The pretraining length is 1 trillion tokens — approximately 2.7× more than BLOOM's 366 billion.

The encoder-decoder architecture provides a natural separation between input and target: the input is fed to the encoder, the target is generated by the decoder, and they never share the same token stream. This means mT5 does not suffer from the "continue the input instead of solving the task" problem that affects BLOOM. At finetuning time, the prompt and dataset instance go to the encoder, and the model generates the target text from the decoder. The loss is computed only on the decoder's output tokens.

The paper finetunes all available sizes of both model families, as listed in Table 3: BLOOM at 560M, 1.1B, 1.7B, 3B, 7.1B, and 176B parameters; mT5 at 300M, 580M, 1.2B, 3.7B, and 13B parameters. This range enables the scaling analysis in Section 4.4.

The choice to use both architectures is deliberate: it tests whether the cross-lingual generalization findings are specific to decoder-only models (which have shown strong zero-shot capabilities) or generalize to encoder-decoder models (which have shown strong transfer learning after finetuning). As the results show (Figure 4), mT0-13B substantially outperforms BLOOMZ-176B despite having 13.5× fewer parameters, a finding the paper attributes to the encoder-decoder architecture, the longer pretraining, and the masked language modeling objective.

3.4.3 Finetuning Procedure

The finetuning procedure differs slightly between BLOOM and mT5 due to their architectural differences, but follows the same high-level principles.

BLOOM finetuning: BLOOM models are finetuned for 13 billion additional tokens. The input and target are concatenated with a space separator, and loss is computed only on the target tokens (not on the input). For example, given input "Translate to English: Je t'aime." concatenated with " I love you.", the model predicts only the tokens of "I love you." and the loss on "Translate to English: Je t'aime." is masked out.

A crucial design choice is loss downscaling by target length: the loss for each token is divided by the length of the target sequence it belongs to. In notation, if a target has $L$ tokens, each token's individual cross-entropy loss is multiplied by $\frac{1}{L}$ . This ensures that short targets (e.g., a single token like "Yes" for multiple-choice QA) contribute the same total weight to the training objective as long targets (e.g., a full translated sentence of 50 tokens). Without this normalization, the loss would be dominated by long-target tasks like translation and summarization, and the model would overfit to producing long sequences at the expense of short-answer accuracy. The paper's ablations in Appendix G confirm this tradeoff: removing the loss normalization (which implicitly upweights long tasks) causes NLU validation accuracy to drop by 10 percentage points.

Samples longer than 2048 tokens are skipped entirely. This truncation is necessary because BLOOM has a finite context window, and including samples that get cut off mid-sentence would introduce noise — the model would learn to generate partial outputs with no coherent ending. The paper uses sequence packing (Kosec et al., 2021) to train on multiple samples simultaneously: rather than padding each sample to 2048 tokens with wasted computation, multiple samples are concatenated into a single sequence up to 2048 tokens, with attention masks preventing cross-sample attention. This increases training throughput.

The optimizer is AdamW, though the paper does not report the finetuning learning rate or batch size for BLOOM in the main text. The final checkpoint is selected based on validation performance on a held-out set of tasks. However, because the finetuning data distribution includes mixtures of long and short tasks, the validation performance dynamics are non-trivial. As shown in Figure 7, performance on natural language understanding (NLU) tasks continues to improve throughout training, while performance on natural language generation (NLG) tasks like translation and summarization peaks early and then declines. This means the checkpoint selection involves a tradeoff: early stopping preserves NLG capability, while longer training improves NLU. The final models use early stopping (they are trained for fewer tokens than the full 13 billion), with the exact token counts listed in Table 6: BLOOMZ-560M at 3.67B tokens, BLOOMZ-1.1B at 0.502B, BLOOMZ-3B and 7.1B at 8.39B and 4.19B respectively, and the full 176B BLOOMZ at 2.09B tokens. The -MT and -P3 variants are stopped after the same number of steps as their default counterparts, not independently validated.

mT5 finetuning: mT5 models are finetuned using the T5X framework (Roberts et al., 2022) on TPUs. The procedure follows the same principles as BLOOM but with one key difference: since mT5 is encoder-decoder, the input goes to the encoder and the target to the decoder, with no need for space separation or input-target boundary management. Loss is computed only on decoder outputs. The paper follows the same preprocessing, packing, and checkpoint selection strategies as for BLOOM. The finetuning token counts for mT0 models (Table 6) range from 1.29B for mT0-13B to 4.62B for the smaller models.

Validation and checkpoint selection: The paper uses a set of validation tasks (listed in Appendix K, Table 11) to monitor performance during training and select the final checkpoint. These validation tasks are distinct from the held-out evaluation tasks used in the results section. The validation set includes both NLU tasks (scored with accuracy) and NLG tasks (scored with BLEU). The divergence between NLU and NLG performance curves (Figure 7) means that a single checkpoint represents a compromise, with NLG performance already past its peak by the time NLU performance plateaus.

This degradation of NLG performance during continued MTF is one of the paper's important findings (Section 4.5). As the model sees more and more short-target examples (most tasks in xP3 produce single-token or single-sentence outputs), it develops a bias toward short generations. The paper quantifies this in Table 4: BLOOMZ generates solutions for HumanEval with an average of 144 characters vs. 247 for BLOOM — a 42% reduction. Solutions also contain 0.04 Python comments on average vs. 0.69 for BLOOM. The paper investigates several remedies in Appendix G: early stopping (reducing exposure to short tasks), upweighting long tasks (removing loss normalization), adding new long tasks (10% of finetuning data), and forcing a minimum generation length at inference (ignoring the end-of-sequence token probability until the generation reaches a desired length). The minimum-length strategy proves most effective, improving BLEU on translation by 9 points. The qualitative impact is visible in Figure 15: BLOOMZ without forced length answers "Rayleigh scattering." while with a 10-token minimum it produces "Rayleigh scattering. The blue color of the sky is due to Rayleigh scattering of sunlight by atmospheric molecules."

3.4.4 Language Distribution Matching and Data Sources

A key design principle for xP3 is aligning its language distribution with BLOOM's pretraining corpus, ROOTS. The rationale, though not explicitly theorized at length, is that the model's capacity for each language is shaped by its pretraining exposure: finetuning data in languages the model knows well should be more effective than finetuning in languages it barely encountered. By mirroring ROOTS, xP3 ensures that high-resource languages like English, French, and Spanish appear frequently during finetuning, while low-resource languages appear proportionally less, matching the model's relative knowledge.

The paper extends the P3 taxonomy with 30 new datasets, classified into four new task clusters:

Translation: Data from Flores-200 (NLLB Team et al., 2022) covering 46 languages in many translation directions, plus Tatoeba (Tiedemann, 2020) for additional language pairs. Translation is the only task type available for the lowest-resource languages in xP3 (Twi, Bambara, Lingala, etc.), as other supervised tasks do not exist for these languages.
Simplification: The BiSECT dataset (Kim et al., 2021), which provides sentence splitting and rephrasing data in English, Spanish, and French.
Program synthesis (text-to-code): MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), GitHub Jupyter text-code pairs, CodeContests, and XLCost (Zhu et al., 2022) for multiple programming languages (Python, C++, Java, JavaScript, PHP, C#, C).
Miscellaneous code: Tasks like estimating computational complexity from a code snippet (codeparrot/codecomplex), generating function names from code (teven/code_docstring_corpus), and a code search task (neural_code_search).

The paper also extends existing P3 task clusters with multilingual datasets: coreference resolution (XWinograd with CLUE extension), multiple-choice QA (C3 for Chinese reading comprehension, mWSC for multiple languages), extractive QA (CMRC2018 and DRCD for Chinese, MLQA and XQuAD for Vietnamese and Chinese), topic classification (CSL and TNews for Chinese), word sense disambiguation (XLWIC for multiple language pairs), paraphrase identification (PAWS-X in multiple languages), sentence completion (XCOPA, XStoryCloze), and NLI (XNLI with both human- and machine-translated prompts for evaluation).

The training prompt templates for all new datasets are documented in Appendix M. Each dataset has multiple templates (the exact number varies), following the PromptSource convention. Some templates invert the task (e.g., for translation, a template might show the translation and ask for the source language text). The choice labels are also part of the prompt templates and are specified separately from the input text. For xP3mt, these choice labels are machine-translated along with the rest of the prompt.

3.4.5 Evaluation Protocol

The paper evaluates zero-shot performance — no task-specific finetuning, no few-shot examples, and no prompt tuning — on three families of held-out tasks: coreference resolution (XWinograd across languages), sentence completion (XCOPA, XStoryCloze), and natural language inference (XNLI). These tasks are "held out" in the sense that they appear in Figure 1 in green, meaning they were excluded from the finetuning data and used only for evaluation. Additionally, the paper evaluates on HumanEval (Chen et al., 2021) for code generation, though this is technically not a "held-out task" in the zero-shot sense since xP3 includes code data — rather, it tests whether the finetuned models retain or improve upon their pretrained code capabilities.

Rank classification for multiple-choice tasks: For datasets where the model must choose a completion from several options (XNLI: entailment/neutral/contradiction; XCOPA/XStoryCloze: two possible endings; XWinograd: two referent options), the paper uses rank classification following prior work (Sanh et al., 2022; Brown et al., 2020). The process works as follows:

For each possible answer choice, construct the full prompt by inserting that choice into the template's target position.
Compute the log-likelihood of the entire target sequence under the model given the input.
Select the choice with the highest log-likelihood.

This requires that the model assign higher probability to the correct completion than to the incorrect ones. The score is the raw log-probability sum over the target tokens, not normalized by length (though the loss normalization during training indirectly encourages length-invariant scoring). The paper does not apply any calibration or normalization at inference time.

Prompt selection and aggregation: For each evaluation dataset, the authors select 5 prompts at random from PromptSource and use the same 5 prompts across all language splits of that dataset. This is a deliberate contrast with XGLM (Lin et al., 2021), which tuned prompts based on validation performance. The paper reports the median of the 5 prompt scores for each language split. The choice of median over mean is not explicitly justified but follows prior work (Sanh et al., 2022) and is robust to outlier prompts that are poorly worded or confusing.

The 5 prompts are selected randomly from the available prompts in PromptSource for each dataset, without any tuning, filtering, or selection based on validation performance. This means the results reflect expected performance under typical usage — a user might write any reasonable prompt — rather than best-case performance with an optimized prompt. The prompts used for evaluation are listed in Appendix M.

Generative task evaluation: For open-ended generation tasks (translation, summarization, code generation), the paper uses lm-evaluation-harness (Gao et al., 2021) and reports BLEU scores (Papineni et al., 2002) for translation. For code generation on HumanEval, the paper follows the standard protocol from Chen et al. (2021): generate 200 samples per problem using top-p sampling with $p = 0.95$ , evaluate at temperatures 0.2, 0.6, and 0.8, and report the best pass@k for $k \in \{1, 10, 100\}$ . A generation is considered correct if it passes the provided unit tests.

Evaluation languages: For language generalization experiments (Section 4.2, Figure 5), the paper evaluates on languages that were not in BLOOM's intended 46-language pretraining set. These include Japanese (jp), Russian (ru), and others shown in Figure 5. The evaluation uses English prompts (the same 5 randomly selected prompts) applied to these unseen-language datasets. The XWinograd, XCOPA, XStoryCloze, and XNLI datasets all have subsets in these languages.

Human-translated and machine-translated evaluation prompts: For Section 4.3 (Table 1), the paper evaluates on three prompt types:

EN: The original English prompts used in Figure 4.
MT: The English prompts machine-translated into each evaluation language using Google Cloud Translation API.
HT: For XNLI only, the prompts were human-translated into the evaluation languages by volunteers.

The human translation of XNLI prompts was done specifically for this paper to serve as a gold standard for comparing machine-translated prompt quality. The other evaluation datasets (XCOPA, XStoryCloze, XWinograd) only had machine-translated prompts available.

The paper reports results as language averages for each task: for each language, take the median across the 5 prompts, then average those medians across languages in the group. For Figure 4, the language average includes languages seen during pretraining and finetuning. For Table 1, the evaluation includes only languages where both EN and MT/HT prompts are available.

3.4.6 Contamination Analysis (Appendix D)

Given the surprising finding that models generalize to languages they were never intentionally trained on, the paper conducts a systematic contamination analysis to quantify how much of these languages actually appeared in the pretraining corpus. The methodology:

Randomly sample 1% of documents from ROOTS (51 million documents total).
For each document, use cld3 (a compact language detector used by mT5's authors) to identify the actual language of each sentence.
Compare the detected language to the document's assigned "meta language" (the language it was curated as belonging to).
Aggregate to compute the true language composition of ROOTS, including incidental contamination.

The results (Figure 11) reveal that ROOTS contains unintentional sentences in many languages beyond the intended 46. For example, Japanese appears at 0.54% (mixed primarily into English documents at 47%, Chinese documents at 5%, and Code documents at 8%), Russian at 0.03%, German at 0.21%, Italian at 0.28%, Thai at 0.006%, Turkish at 0.03%, Greek at 0.03%, Bulgarian at 0.05%, Estonian at 0.06%, Haitian Creole at 0.12%, and Burmese at 0.00003%.

The paper extrapolates: if Thai represents 0.006% of the sample and ROOTS is approximately 341 billion tokens total, BLOOM would have seen roughly 20 million tokens of Thai during pretraining. While tiny compared to English (30% of ROOTS), this is non-zero exposure. The contamination stems from code-mixing (multiple languages in one document), code comments in different languages, and misclassification during corpus curation.

This analysis is crucial for interpreting the language generalization results: when the paper reports that BLOOMZ performs above random on XNLI in Thai, it is not demonstrating generalization to a completely novel language, but rather demonstrating that the model can leverage minuscule incidental exposure combined with task knowledge from other languages. Whether the same would hold for a language with truly zero exposure remains an open question, though the paper's transparency about contamination allows readers to draw appropriately caveated conclusions.

3.4.7 Model Naming Convention

The paper produces nine distinct model variants, all publicly released:

BLOOMZ / mT0: Finetuned on xP3 (multilingual data, English prompts).
BLOOMZ-P3 / mT0-P3: Finetuned on English-only P3.
BLOOMZ-MT / mT0-MT: Finetuned on xP3mt (multilingual data, machine-translated prompts).

The base BLOOM and mT5 models (without finetuning) serve as pretrained baselines. The naming convention is: model family (BLOOM or mT5) → suffix indicating finetuning data (Z for xP3, Z-P3 for P3, Z-MT for xP3mt; for mT5, the '5' changes to '0' following the T0 convention from Sanh et al., 2022, where T5+MTF becomes T0, so mT5+MTF becomes mT0).

All models, datasets, and evaluation code are released under permissive licenses (BLOOM/BLOOMZ under RAIL, mT5/mT0 under Apache 2.0) as listed in Table 3, with direct links to Hugging Face repositories.

3.4.8 Summary of Key Design Decisions and Their Justifications

xP3 mirrors ROOTS language distribution rather than a uniform distribution over languages: aligns finetuning exposure with pretraining exposure, so the model is asked to perform tasks in languages it actually knows well.
Prompts are in English for xP3, even for non-English data: based on XGLM's finding that English prompting works well cross-lingually; also ensures the model learns to map English instructions to non-English inputs, which may be the most practical deployment scenario (English-speaking users querying multilingual content).
Loss downscaling by target length: prevents long-generation tasks from dominating the training signal; discovered through the ablation that removing it hurts NLU accuracy by 10 percentage points.
Random prompt selection without tuning: ensures results reflect expected zero-shot performance, not best-case with optimized prompts; also avoids overfitting the evaluation to specific prompt phrasings.
Two-fold architecture comparison (BLOOM + mT5) : decoder-only vs. encoder-decoder, different pretraining objectives (next-token vs. span corruption), different pretraining lengths (366B vs. 1T tokens), and different pretraining corpora (ROOTS 46 languages vs. mC4 101 languages) — this breadth means that findings that hold across both families are likely robust to these architectural and data differences.
Contamination analysis via cld3 re-identification: rather than trusting the curated language labels, which would overstate how "unseen" a language is, the paper directly measures what the model actually saw, turning a surprising result (generalization to "unseen" languages) into a more nuanced finding about leveraging incidental exposure.
Multiple size scales (560M–176B for BLOOM, 300M–13B for mT5): enables the scaling analysis that shows the gap between pretrained and MTF models widens with size, a finding that would be invisible if only the largest models were tested.

4. Key Insights and Innovations

Innovation 1: Framing Cross-Lingual Task Generalization as a Data Composition Problem Rather Than an Architecture Problem

The dominant assumption before this work was that cross-lingual transfer required either (a) explicit alignment mechanisms between languages during pretraining (Conneau et al., 2019; Lample and Conneau, 2019), (b) language-specific adapters or fine-tuning steps (Devlin et al., 2018; Aribandi et al., 2021), or (c) training on parallel or translated data that explicitly pairs languages (Liu et al., 2020). The field's mental model was that getting a model to perform Task X in Language Y required connecting X and Y through some explicit training signal — shared vocabulary, cross-lingual objectives, or task-labeled data in Y.

This paper makes a fundamentally different conceptual move: it treats cross-lingual task generalization as a property of the finetuning data mixture applied to an already-pretrained multilingual model, not a property of architecture design or explicit cross-lingual training. The core experimental maneuver — taking a model pretrained on 46 languages, finetuning it only on English tasks with English prompts (P3), and then observing improvements on non-English held-out tasks — demonstrates that task knowledge transfers across languages through the multilingual representations learned entirely during pretraining. The finetuning phase adds no new cross-lingual alignment; it merely activates capabilities that already exist latently in the pretrained model.

This is a reframing, not an incremental improvement. Prior work on cross-lingual transfer (Wu and Dredze, 2019; Phang et al., 2020; Chalkidis et al., 2021) had shown that finetuning on English task data could improve that same task in other languages — but the task was held constant. The gap this paper closes is showing that English-only multitask finetuning improves performance on entirely new, held-out tasks in non-English languages. The model learns "how to do sentiment analysis" or "how to do NLI" in English, and that capability transfers to performing sentiment analysis or NLI on Spanish or Chinese inputs without ever seeing a single Spanish or Chinese example of those tasks during finetuning.

Evidence for this is in Figure 4 and the associated text: BLOOMZ-P3 (finetuned on English-only P3) improves over the pretrained BLOOM baseline by more than 50% on multilingual sentence completion tasks, and these gains appear across all languages evaluated, not just high-resource ones. The mechanism is the shared multilingual representations in the pretrained model — the paper doesn't need to propose a new architecture or loss function to enable transfer; it simply chooses what data to finetune on.

The practical implication is significant: this finding collapses the cost structure of deploying NLP systems for low-resource languages. If English-only task data suffices to unlock task capabilities in dozens of languages, the annotation bottleneck is dramatically narrower — you only need labeled data in one high-resource language rather than per-language annotation. The paper doesn't claim this works perfectly (the gains are larger when multilingual task data is added via xP3), but establishing that English-only finetuning provides any cross-lingual task generalization is a conceptual advance that changes how to think about the problem.

Innovation 2: The Discovery That Prompt Language Transfer Behaves Asymmetrically and Can Be Trained Separately From Task Transfer

Before this work, the dominant assumption about prompting multilingual models — inherited from XGLM (Lin et al., 2021) — was that English prompts work best for cross-lingual tasks and that human-translating prompts doesn't help much. XGLM explicitly reported that using English prompts for multilingual datasets provided better performance than human-translated prompts. This had a natural interpretation: the model's strongest "task-understanding" representations are in English (its highest-resource language), so English prompts most effectively activate the right task behavior regardless of input language.

This paper complicates and ultimately overturns that picture by showing that the relationship between prompt language and task performance is trainable and asymmetric. The key finding in Table 1 is that models finetuned only on English prompts (BLOOMZ, mT0-13B) perform poorly when given non-English prompts at test time — even though those same models perform well on non-English inputs with English prompts. For example, BLOOMZ achieves 52.99% on XNLI with English prompts but drops to 37.56% on machine-translated prompts and 40.4% on human-translated prompts. The model can process non-English text — it does so successfully when the prompt is in English — but it can't reliably follow instructions in non-English languages.

The conceptual move is to show that this is a training data artifact, not a fundamental property of the model or of language. When you finetune on xP3mt (which includes machine-translated prompts), performance on non-English prompts jumps significantly: BLOOMZ-MT rises from 40.4 to 43.88 on human-translated XNLI prompts, and mT0-13B-MT rises from 44.95 to 46.87. This improvement comes at the cost of English-prompt performance (BLOOMZ-MT drops from 52.99 to 49.01 on English XNLI prompts), revealing a tradeoff between English-prompt and multilingual-prompt performance that is controlled by the language composition of the finetuning prompts.

This is a new diagnostic concept: the model's ability to follow task instructions in a given language is not the same thing as its ability to process input text in that language, and the two capabilities can be independently manipulated through training data composition. The paper is essentially demonstrating that prompt-following is a capability that must be learned per-language, even when task-solving transfers across languages automatically. This explains why XGLM found English prompts to be "best" — their model had only seen English prompts during its few-shot demonstrations, so it had only learned to follow instructions in English. It wasn't that non-English prompts are inherently worse; it was that the model had never been trained to process them.

The significance goes beyond the numerical improvements. This finding opens up a new axis of control for practitioners: you can independently decide how much to invest in prompt translation based on your deployment needs. If your users will prompt in English to analyze multilingual content, xP3 with English prompts is sufficient. If your users will prompt in their own languages, you need xP3mt-style training. And you can't get both for free — the English/non-English prompt performance tradeoff means there's a genuine design choice to make. This is a more nuanced, actionable picture than "just use English prompts."

Innovation 3: Documenting That Multitask Finetuning Can Induce Language-Generalization to Incidentally-Seen Languages — and the Contamination Analysis That Makes This Finding Scientifically Honest

Perhaps the most surprising result in the paper is that models finetuned on xP3 generalize to tasks in languages they were never intentionally trained on. Figure 5 shows BLOOMZ performing above random on XNLI, XCOPA, XStoryCloze, and XWinograd in languages like Japanese, Russian, and Thai — languages that were not part of ROOTS's curated 46-language set and not part of xP3's finetuning data. The paper doesn't claim strong performance (accuracy is modest, in the 45–62% range for most tasks, above the 33% random baseline for 3-way NLI and 50% for binary tasks), but the fact that it's systematically above random across multiple tasks and multiple "unseen" languages is striking.

The intellectual contribution here is not just the finding itself but how the paper handles it. Rather than claiming that models can generalize to truly novel languages — which would be a revolutionary claim likely to be wrong — the paper immediately confronts the contamination question. Appendix D's language identification analysis on 51 million ROOTS documents reveals that "unseen" languages are often present in trace amounts: Japanese at 0.54%, Thai at 0.006%, Burmese at 0.00003%. This transforms the finding from "the model learns languages it never saw" to "the model can leverage tiny amounts of incidental exposure — as little as 20 million tokens for Thai — combined with task knowledge from other languages, to perform above random."

This is a diagnostic contribution: the paper provides a methodology for distinguishing true language generalization from leakage through pretraining contamination, and by doing so, it establishes a more precise boundary condition for what "zero-shot" really means in the era of web-scale pretraining. The finding is that MTF acts as an amplifier: it takes the latent multilingual knowledge from pretraining (including knowledge of incidentally-seen languages) and makes it accessible for task execution. The model couldn't use its tiny Thai exposure for anything useful in the base pretrained state; after MTF, that same exposure is activated into a functional (if weak) task capability.

This is a fundamental rather than incremental finding because it changes how we should think about data curation. The standard narrative was that you need to intentionally include a language in your pretraining corpus to get any capability in that language. This paper shows that incidental exposure — the kind that naturally occurs in web-scale data through code-mixing, code comments, and misclassified documents — can be sufficient for emergent task capabilities when combined with MTF. This has implications for both data curation (don't over-filter to remove "contaminating" languages — they might be useful) and for evaluation (you can't trust curated language labels; you need to actually measure what your corpus contains).

The evidence in Figure 5 is supported by the contamination analysis in Figure 11, and the paper is appropriately cautious in its interpretation: "We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic." The mystery isn't fully resolved — the edit distance analysis in Appendix H shows that surface-level patterns in XNLI are unlikely to explain the above-random performance — but the paper's framing of the phenomenon with empirical rigor, rather than speculation, makes this a genuine insight rather than an overclaimed result.

Innovation 4: Identifying a Fundamental NLU/NLG Tradeoff in Multitask Finetuning and a Simple Inference-Time Fix

This is a negative result with practical implications. Prior work on MTF (Sanh et al., 2022; Wei et al., 2021; Chung et al., 2022) had reported steadily improving performance with more finetuning, but those works focused primarily on classification and multiple-choice tasks — what the paper calls NLU. The diagnostic contribution here is showing that this monotonic improvement is task-type-specific: when you monitor both NLU and NLG performance during finetuning, they diverge. Figure 7 reveals that NLU continues to improve while NLG (translation, summarization) peaks early and then degrades.

The mechanism is identified in Section 4.5 and Appendix G: the finetuning data (xP3) is dominated by tasks with short targets — most are single sentences or single tokens for classification. Over the course of billions of finetuning tokens, the model develops a length bias toward concise answers. The quantitative evidence is stark: BLOOMZ generates HumanEval solutions with 42% fewer characters and 94% fewer comments than BLOOM (Table 4). Qualitatively, BLOOMZ answers "Why is the sky blue?" with "Rayleigh scattering." while BLOOM (interesting, but for a different reason) generates repetitive continuations of the question (Figure 15).

This finding matters because it reveals that the standard MTF recipe — finetune on a diverse mixture, select the checkpoint with best aggregate validation performance — hides a genuine conflict between task types. You can't simultaneously optimize for both short-answer accuracy and long-form generation quality with the same training procedure. The paper explores several solutions (early stopping, upweighting long tasks, adding more long tasks to the mixture) but identifies a simple, effective inference-time fix: forcing a minimum generation length by masking the end-of-sequence token for the first N generated tokens. This raises BLEU on translation by 9 points and produces qualitatively better generations (Figure 15).

This is an incremental contribution in the sense that it's a diagnosis and patch rather than a new capability, but it's an important one because it identifies a failure mode that practitioners will encounter and provides an actionable remedy. The deeper conceptual point is that MTF mixtures are not neutral — the distribution of target lengths in the finetuning data shapes the model's output behavior in ways that persist even for unseen tasks. This is a specific instance of a broader principle: what you mix matters, not just how many tasks you include. The finding prefigures later work on data mixture optimization in instruction tuning (e.g., Longpre et al., 2023a) but makes the point through a clean, empirical demonstration with a simple fix rather than complex reweighting schemes.

Innovation 5: Demonstrating Architecture-Independent Generalization Across Decoder-Only and Encoder-Decoder Models — With a Surprising Reversal

The paper finetunes both BLOOM (decoder-only, 560M–176B parameters) and mT5 (encoder-decoder, 300M–13B parameters) on the same xP3 dataset and finds that all the core findings replicate across architectures: English-only finetuning transfers cross-lingually, multilingual data helps, machine-translated prompts improve non-English prompt performance, and language generalization to incidentally-seen languages occurs. This establishes that the phenomena are not artifacts of a specific architecture, pretraining objective, or corpus.

What makes this more than a robustness check is the surprising performance reversal. Despite having 13.5× fewer parameters (13B vs. 176B), mT0-13B substantially outperforms BLOOMZ-176B on all held-out tasks in Figure 4. The paper attributes this to three factors: the encoder-decoder architecture, the masked language modeling pretraining objective (which may produce representations more amenable to task transfer than next-token prediction), and the longer pretraining of mT5 (1 trillion tokens vs. 366 billion for BLOOM). Additionally, mT0-13B outperforms the fully English T0-11B on English tasks (Figure 9), despite T0 having the same architecture and pretraining objective but English-only pretraining — meaning the multilingual pretraining of mT5 actually helps English task performance after MTF.

This is a conceptual finding about the relationship between pretraining and finetuning: multilingual pretraining is not a tax on English performance — it can be a benefit when combined with appropriate finetuning. The paper doesn't fully explain why (it would require controlled experiments varying only language composition while holding architecture and compute constant), but the empirical result challenges the intuition that multilingual models necessarily trade off per-language quality for breadth. When followed by MTF on a diverse task mixture, the broader pretraining distribution appears to provide representations that support better generalization even within a single language.

The comparison with mTk-Instruct (Wang et al., 2022c) — which performs significantly worse than the same-sized mT0 despite also being a multilingual MTF model — adds another layer: the prompting style matters enormously. mTk-Instruct's structured "Definition / Input / Output" format produces worse generalization than mT0's natural human-written prompts, even when evaluated on the respective models' preferred formats. This suggests that how you prompt during finetuning shapes the nature of the learned task representations, not just their compatibility with specific evaluation formats. Natural prompts — which more closely resemble the language the model saw during pretraining — may produce task representations that are more flexible and transferable than structured templates that impose an artificial format. This is an architectural insight about representation learning through prompting, distinct from the usual focus on prompt engineering for evaluation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper constructs xP3 by extending the P3 dataset (Sanh et al., 2022) with 30 additional multilingual datasets spanning 46 natural languages and code. For evaluation, the paper primarily uses three families of held-out tasks: coreference resolution (XWinograd across multiple languages with the CLUE extension adding Chinese schemas), sentence completion (XCOPA and XStoryCloze), and natural language inference (XNLI). Code generation is evaluated on HumanEval (Chen et al., 2021) using the pass@k metric. Translation quality on Multi-EURLEX (Chalkidis et al., 2021) is used for the generation length ablation in Appendix G. These evaluation tasks are explicitly held out from the finetuning data — they appear in green in Figure 1 and are never used during training. The paper evaluates on a wide range of languages within each task, including both languages that appear in the finetuning data (Figure 4, e.g., Spanish, French, Chinese) and languages that were never intentionally part of pretraining or finetuning (Figure 5, e.g., Japanese, Russian, Thai).
Base model(s). The paper finetunes two model families. BLOOM (Scao et al., 2022a): decoder-only transformer models ranging from 560M to 176B parameters, pretrained on the ROOTS corpus (46 languages, 13 programming languages) for approximately 366 billion tokens with a next-token prediction objective. mT5 (Xue et al., 2020): encoder-decoder transformer models ranging from 300M to 13B parameters, pretrained on mC4 (101 languages) for 1 trillion tokens with a span corruption objective. The choice of two architectures tests whether the cross-lingual generalization findings are architecture-agnostic. The base pretrained models (without any finetuning) serve as the primary baselines, along with external comparisons to XGLM-7.5B (Lin et al., 2021), T0-11B (Sanh et al., 2022), and mTk-Instruct (Wang et al., 2022c) at 3.7B and 13B parameters.
Metrics. For multiple-choice tasks (XNLI, XCOPA, XStoryCloze, XWinograd), the paper uses rank classification accuracy: the model computes the log-likelihood of each possible completion given the prompt and input, and the highest-scoring option is selected. Accuracy is then computed as the fraction of examples where the correct answer receives the highest log-likelihood. For each evaluation dataset, 5 prompts are selected at random from PromptSource and used across all language splits. The paper reports the median of the 5 prompt scores per language split, then averages these medians across languages to produce the language-average accuracy shown in figures. For generative tasks, the paper uses BLEU (Papineni et al., 2002) for translation and pass@k for code generation on HumanEval, following the standard protocol from Chen et al. (2021): generate 200 samples per problem with top-p sampling (p=0.95), evaluate at temperatures 0.2, 0.6, and 0.8, and report the best pass@k for k ∈ {1, 10, 100}.
Baselines. The paper uses several distinct baselines. The pretrained BLOOM and mT5 models serve as the primary "no finetuning" baselines. XGLM-7.5B (Lin et al., 2021) is included as an existing multilingual decoder-only model for comparison on held-out tasks in Figure 4. T0-11B (Sanh et al., 2022) represents the state-of-the-art for English-only MTF on an encoder-decoder architecture. mTk-Instruct at 3.7B and 13B parameters (Wang et al., 2022c) is the most directly comparable prior work — a multilingual model finetuned on prompted multitask data, but using structured prompts with "Definition," "Input," and "Output" fields rather than the natural human-written prompts from PromptSource. The paper also introduces internal baselines: BLOOMZ-P3 and mT0-P3 (finetuned on English-only P3) test the cross-lingual transfer from English-only finetuning; BLOOMZ-MT and mT0-MT (finetuned on xP3mt with machine-translated prompts) test the effect of prompt language diversity. The random baseline for each task type is also noted: 50% for binary tasks (sentence completion, coreference resolution), 33% for 3-way classification (XNLI).
Generation budget / compute accounting. The paper does not measure compute in FLOPs or GPU-hours for model comparison. Instead, finetuning tokens serve as the primary compute measure, with Table 6 reporting exact token counts each final model was finetuned for (ranging from 0.502B for BLOOMZ-1.1B to 8.39B for BLOOMZ-3B, and 1.29B to 4.62B for mT0 models). For evaluation, the generation budget is implicit in the rank classification protocol — only one forward pass per answer choice is needed per example. For HumanEval, 200 samples per problem are generated for each of three temperatures. For the generation length ablation (Appendix G), the number of finetuning tokens is the controlled variable (6B vs. 13B), and inference-time minimum generation length is measured in tokens (768 tokens forced before end-of-sequence is allowed).
Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional sense. Instead, it relies on reporting the median across 5 randomly selected prompts for each evaluation, which provides robustness to individual poorly-worded prompts without requiring per-task prompt tuning. For checkpoint selection during finetuning, a separate set of validation tasks (listed in Appendix K, Table 11) is used — these are distinct from the held-out evaluation tasks. The -MT and -P3 variants are stopped after the same number of training steps as their default counterparts, meaning no independent validation is performed for these variants. The contamination analysis in Appendix D samples 1% of documents from ROOTS (51M documents) and re-identifies languages using cld3, providing a quantitative estimate of incidental language exposure. No confidence intervals, standard errors, or statistical significance tests are reported for any result — all numbers are point estimates from single evaluation runs.

Main Quantitative Results

English-Only Finetuning Transfers Cross-Lingually (Figure 4, Appendix B Figure 10)

The most fundamental result is that English-only multitask finetuning improves zero-shot performance on non-English held-out tasks. In Figure 4, BLOOMZ-P3 (176B parameters, finetuned only on English P3) improves over the pretrained BLOOM baseline by more than 50% on multilingual sentence completion. This finding is visible across all three evaluation tasks (coreference resolution, sentence completion, NLI) and across all languages evaluated. The pretrained BLOOM baseline hovers near random for coreference resolution (~50%) and NLI (~33%), consistent with the paper's note that "all of our experiments being zero-shot and using untuned prompts" explains the low pretrained baselines. The improvement from English-only MTF is substantial relative to this low starting point: BLOOMZ-P3 lifts performance well above random on all tasks, demonstrating that task knowledge acquired from English finetuning data transfers to non-English inputs through the shared multilingual representations learned during pretraining.

Figure 9 provides the English-only breakdown of the same finding: on English held-out tasks, BLOOMZ (finetuned on xP3) outperforms BLOOMZ-P3 (finetuned on P3), establishing that adding multilingual data helps even for English-only evaluation. Each dot in Figure 9 represents performance on one of the 5 evaluation prompts, and the spread across prompts is substantial, "highlighting that prompt engineering may still be necessary after MTF."

Multilingual Data with English Prompts (xP3) Improves Over English-Only (P3) for Both English and Non-English Tasks (Figure 4)

The comparison between BLOOMZ (xP3-finetuned) and BLOOMZ-P3 (P3-finetuned) in Figure 4 shows that adding multilingual task data — even when all prompts remain in English — yields consistent improvements on all held-out tasks. On multilingual sentence completion, BLOOMZ substantially outperforms BLOOMZ-P3. On coreference resolution and NLI, BLOOMZ maintains a clear advantage. The gains are present in both the English-only evaluation (Figure 9) and the multilingual evaluation (Figure 4 and Appendix B, Figure 10), meaning the benefit of multilingual finetuning data is not simply from better non-English language coverage — it also improves the model's ability to perform tasks in English, likely because xP3 adds more tasks, more datasets, and more diverse prompts than P3, which has been shown to help generalization (Chung et al., 2022; Iyer et al., 2022).

A striking finding is that mT0-13B (finetuned on xP3) outperforms the fully English T0-11B on English tasks (Figure 9). The paper notes: "Ignoring embedding parameters T0-11B and mT0-13B have about the same size." Since T0-11B was pretrained on English-only data and finetuned on English-only P3, while mT0-13B was pretrained on 101 languages and finetuned on the 39% English xP3, this means that multilingual pretraining plus partially-multilingual finetuning beats fully-English pretraining plus English-only finetuning on English tasks. The paper attributes this to xP3 adding additional tasks and prompts, concluding that "the benefit of scaling tasks is larger than the benefit of pretraining and finetuning on relatively more English tokens."

Appendix B, Figure 10 breaks down the multilingual task generalization by language and prompt. The figure shows "Performance by prompt varies substantially," with individual dots (one per prompt per language) spread across a wide range. For some languages, prompt choice can swing accuracy by 10–20 percentage points. The paper also notes that "mT0 consistently outperforms BLOOMZ on Swahili (SW), possibly due to it being a larger part of its pretraining corpus" — referring to Figure 2, which shows that Swahili is a larger fraction of mC4 (mT5's pretraining corpus) than of ROOTS (BLOOM's pretraining corpus), consistent with the language proportion analysis in Section 4.6.

mT0-13B Outperforms BLOOMZ-176B Despite 13.5× Fewer Parameters (Figure 4)

Despite BLOOMZ having 176 billion parameters vs. mT0-13B's 13 billion (a 13.5× difference), mT0-13B achieves substantially higher accuracy on all three held-out tasks in Figure 4. The paper attributes this to three factors mentioned in Section 4.1: the encoder-decoder architecture paired with the masked language modeling pretraining objective (Wang et al., 2022a; Tay et al., 2022a), and mT5's longer pretraining — 1 trillion tokens vs. BLOOM's 366 billion. The gap between BLOOMZ and mT0-13B is particularly large on sentence completion and coreference resolution.

The comparison with mTk-Instruct adds another dimension: mTk-Instruct-13B (3.7B parameters) performs significantly worse than the same-sized mT0-3.7B, which the paper attributes to prompting style. mTk-Instruct uses structured prompts with specific "Definition," "Input," and "Output" fields, while mT0 uses natural human-written prompts from PromptSource. The paper notes that "Wang et al. (2022c) find that T0 performs worse than Tk-Instruct on their prompts" — meaning the prompt format advantage is evaluation-format-dependent, with each model performing better on the prompt style it was trained with. However, the paper's evaluation uses PromptSource-style natural prompts, which naturally favors mT0.

Language Generalization to Incidentally-Seen Languages (Figure 5)

Figure 5 reports performance on tasks in languages that were not intentionally included in BLOOM's pretraining or in the xP3 finetuning data. The specific tasks and languages are: XWinograd in Japanese (jp), Russian (ru), and other languages; XCOPA in Japanese, Turkish (tr), and others; XStoryCloze in additional languages; and XNLI in Thai (th), Turkish, Greek (el), and others. The key finding is that BLOOMZ performs systematically above random on all these tasks, but the base BLOOM model does not, meaning the MTF process unlocked the ability to use incidentally-acquired multilingual knowledge for task execution.

The performance levels are modest but clearly above random: for BLOOMZ, the language-average scores in Figure 5 range from approximately 55–62% on binary tasks (XCOPA, XStoryCloze, XWinograd, where random is 50%) and approximately 45–50% on 3-way XNLI (where random is 33%). The base BLOOM model, in contrast, performs near the random baseline on all tasks. The paper conjectures that "the models are learning higher-level capabilities that are both task- and language-agnostic."

The contamination analysis in Appendix D, Figure 11 provides the crucial context: the ROOTS corpus unintentionally contains sentences in these "unseen" languages. For example, Thai constitutes 0.006% of the sampled ROOTS documents, Japanese 0.54%, Russian 0.03%, German 0.21%. If extrapolated to the full ROOTS corpus of approximately 341 billion tokens, BLOOM would have seen roughly 20 million tokens of Thai — a tiny fraction compared to English's ~102 billion tokens (30% of ROOTS), but non-zero. The paper explicitly frames this as "leveraging tiny amounts of incidental exposure combined with task knowledge from other languages" rather than generalization to genuinely novel languages.

Multilingual Prompting: Machine-Translated Prompts During Finetuning Improve Performance on Non-English Test Prompts (Table 1)

Table 1 reports the core prompt language transfer results for the largest models (BLOOMZ-176B and BLOOMZ-MT-176B; mT0-13B and mT0-13B-MT). The table compares three prompt types at evaluation time: EN (English prompts, the same ones used throughout Figure 4), MT (machine-translated evaluation prompts), and HT (human-translated evaluation prompts, available only for XNLI). The results are broken out by task (XNLI, XCOPA, XStoryCloze, XWinograd).

For BLOOMZ on XNLI, the drop from English to non-English prompts is dramatic: 52.99 (EN) → 37.56 (MT) → 40.40 (HT). This means that BLOOMZ, which was finetuned only on English prompts, loses approximately 12–15 percentage points when the prompt is translated to other languages — despite performing well on non-English inputs with English prompts. The same pattern holds for mT0-13B on XNLI, though the drop is less severe: 48.24 (EN) → 39.31 (MT) → 44.95 (HT).

For BLOOMZ-MT, which was finetuned on xP3mt with machine-translated prompts, the pattern reverses: performance on non-English prompts improves substantially while English prompt performance declines. On XNLI, BLOOMZ-MT achieves 41.16 (MT) and 43.88 (HT) — improvements of 3.6 and 3.48 points respectively over BLOOMZ — but drops to 49.01 on English prompts, a loss of approximately 4 points. The paper quantifies this on XNLI human-translated prompts: "BLOOMZ-MT raises the average performance on human-translated prompts from 41.13 to 45.55. This comes at the cost of a reduction in its performance on English prompts, from 53.58 to 49.74." The numbers cited here (41.13, 45.55, 53.58, 49.74) differ slightly from Table 1 (40.4, 43.88, 52.99, 49.01) because these are averages computed over additional XNLI language subsets beyond those in Table 1, as noted in the text.

For mT0, the MT variant provides similar performance gains on XNLI and XWinograd non-English prompts (mT0-13B-MT achieves 41.66 MT and 46.87 HT on XNLI vs. mT0-13B's 39.31 MT and 44.95 HT), while results on XCOPA and XStoryCloze are mixed — mT0-13B-MT shows slight gains on MT prompts for XStoryCloze (82.86 vs. 83.37 for EN) but slight declines on XCOPA (79.64 vs. 81.16 for EN). The paper also notes that models perform better on human-translated prompts than machine-translated ones for XNLI, consistent with Lin et al. (2021): "Similar to Lin et al. (2021), we also find that models perform better on human-translated prompts than machine-translated ones for XNLI."

For tasks like XCOPA and XStoryCloze, the differences between EN and MT prompts are much smaller than for XNLI — typically 1–3 percentage points for BLOOMZ, and sometimes even favoring MT prompts for mT0-MT variants. The paper does not analyze why XNLI shows much larger prompt language sensitivity than the other tasks, but it may relate to NLI requiring more nuanced understanding of the logical relationship expressed in the prompt.

Figure 13 provides qualitative examples of the prompt language effect on sentiment analysis, a task that was trained on. With an English prompt ("Would you rate the previous review as positive, neutral or negative?"), both BLOOMZ and BLOOMZ-MT output "positive" in English. With a Chinese prompt, BLOOMZ still outputs "positive" (in English) while BLOOMZ-MT correctly outputs the Chinese word for "praise" (赞扬), showing that finetuning on machine-translated prompts enables the model to respond in the prompt language even when the target language was not explicitly trained for that specific task.

Appendix I, Table 9 extends the prompt language analysis to unseen languages — those that BLOOMZ was never intentionally exposed to during pretraining or finetuning. For BLOOMZ, performance drops significantly when translating prompts to these unseen languages, similar to the pattern for seen languages. Unlike on seen languages, BLOOMZ-MT performs worse than BLOOMZ for machine-translated prompts in unseen languages, presumably because BLOOMZ-MT has not been finetuned on prompts in those specific languages (the xP3mt data covers only the 46 languages of ROOTS, not the incidentally-seen languages).

Scaling: The Gap Between Pretrained and MTF Models Widens with Model Size (Figure 6)

Figure 6 plots aggregate performance vs. model size for BLOOM and BLOOMZ models from 560M to 176B parameters, and mT5 and mT0 models from 300M to 13B parameters. The transparent lines correspond to individual languages, while the thick lines are average accuracy scores across languages. The key finding is that "even at 560M parameters, multitask finetuning improves zero-shot generalization," and that "the gap between pretrained and multitask finetuned models grows significantly as parameters increase."

For BLOOM, the pretrained baseline is near random (0.50 for binary tasks) at all scales, slightly improving at the largest scale but never escaping the random baseline by more than a few points. BLOOMZ, in contrast, shows a steep scaling curve: the 560M model achieves modest gains over BLOOM-560M, but by 176B parameters, BLOOMZ is substantially above the pretrained baseline. The paper states that "scaling up parameters benefits all languages evaluated," though the transparent lines in Figure 6 show that the magnitude of benefit varies by language — some languages improve more with scale than others.

The mT5/mT0 comparison shows a similar pattern but from a different starting point. Since mT5 is an encoder-decoder model pretrained with a masked language modeling objective, its zero-shot performance without finetuning is not meaningful (the paper states there is "no data point at 0 tokens, as its base model, mT5, is not suitable for evaluation without finetuning"). The mT0 models show improvement with scale, with mT0-13B significantly outperforming mT0-3.7B, which in turn outperforms smaller variants.

Generation Tasks: NLU Improves While NLG Degrades During Finetuning (Figure 7, Table 2)

Figure 7 plots validation performance during training for natural language understanding (NLU) and natural language generation (NLG) tasks, measured using accuracy and BLEU respectively. For BLOOMZ-7.1B, NLU performance (accuracy) steadily increases throughout the 13 billion tokens of finetuning. NLG performance (BLEU on translation and summarization), however, jumps initially — rising from BLOOM-7.1B's zero-shot baseline to a peak early in finetuning — and then decreases as training continues. The paper quantifies this: "generative performance jumps initially and then decreases." The validation NLG tasks measured are translation and summarization, and the BLEU score at the end of finetuning is lower than at the peak. This divergence means that checkpoint selection involves a tradeoff between NLU and NLG quality.

Table 2 reports pass@k results on HumanEval for BLOOM, BLOOMZ, and BLOOMZ-P3 at all model sizes, alongside external baselines from prior work (GPT-Neo, GPT-J, GPT-NeoX, Codex). At the largest scale (176B), BLOOM achieves pass@1 = 15.52%, pass@10 = 32.20%, pass@100 = 55.45%. BLOOMZ achieves pass@1 = 12.06%, pass@10 = 26.53%, pass@100 = 48.44% — a decline from BLOOM across all k values. BLOOMZ-P3, which had no code data in finetuning, drops further to pass@1 = 6.13%. For small models, however, the pattern reverses: BLOOMZ-560M outperforms BLOOM-560M (pass@1: 2.18% vs. 0.82%), suggesting that code data in xP3 partially counteracts the NLG degradation at smaller scales, but at larger scales the length bias dominates.

The qualitative impact of the length bias is documented in Figure 12, which shows example HumanEval solutions from BLOOM and BLOOMZ. BLOOM generates a solution with a loop, explicit comments, and verbose variable names (approximately 247 characters on average). BLOOMZ generates a concise solution using any() with a generator expression and no comments (approximately 144 characters on average). Table 4 quantifies this across all HumanEval generations: BLOOMZ averages 144 characters per solution vs. BLOOM's 247 (a 42% reduction) and produces 0.04 Python comments per solution vs. BLOOM's 0.69 (a 94% reduction). The finetuning data statistics in the same table show that xP3 code targets average 531 characters and 0.85 comments — meaning the bias toward brevity is learned from the non-code tasks in the mixture, not from the code data itself.

Effect of Language Proportions: Higher-Resource Languages Benefit More (Figure 8)

Figure 8 plots performance on XCOPA, XNLI, and XWinograd against the proportion of each language in BLOOM's pretraining corpus (ROOTS). The x-axis shows individual languages ordered by their ROOTS proportion, from high-resource (English, Spanish, French at >10% each) to low-resource. For XCOPA and XNLI, there is a clear positive correlation: languages that appear more frequently in pretraining (and, since xP3 mirrors ROOTS, also appear more frequently in finetuning) show substantially higher accuracy. For example, English and Spanish XNLI accuracy is substantially above the performance on lower-resource languages like Swahili or Urdu, which appear at <1% in ROOTS.

For XWinograd, the trend is "less consistent." The paper attributes this to the fact that "XWinograd language subsets are not translations of each other and have a significantly different number of samples. Thus, some language subsets of XWinograd may be inherently more difficult than others." This is an important caveat: when evaluation sets are not parallel across languages, differences in performance may reflect differences in dataset difficulty rather than (or in addition to) differences in the model's language capability. The XCOPA and XNLI results are more interpretable because their language subsets are translations of the same underlying examples, so differences in accuracy can be more confidently attributed to language factors.

Full Per-Language and Per-Prompt Results (Appendix K, Table 10)

Table 10 in Appendix K provides the complete raw evaluation results for all models, tasks, datasets, language configurations, and prompts. This table spans multiple pages and shows results at the granularity of individual dataset-language pairs. Some notable details visible only in this table: for XWinograd, performance varies dramatically by language for the same model — BLOOMZ achieves 67.87% on English, 77.11% on French, 52.03% on Japanese, 63.50% on Portuguese, 57.78% on Russian, and 62.90% on Chinese (all median accuracies with EN prompts). The French performance being higher than English is notable and may reflect the specific composition of the XWinograd French subset or the model's relative strength in French. For XNLI, language-level performance can be extracted from the full table for detailed analysis.

Ablation Studies and Robustness Checks

Prompt language at evaluation time (Table 1): The comparison of EN, MT, and HT evaluation prompts demonstrates that prompt language matters significantly for models finetuned only on English prompts, with drops of 12–15 points on XNLI when switching from English to non-English prompts. Finetuning on xP3mt substantially closes this gap, with BLOOMZ-MT gaining 3.5 points on human-translated XNLI prompts compared to BLOOMZ, at the cost of approximately 4 points on English prompts. For mT0, the tradeoff is less pronounced, with mixed results on XCOPA and XStoryCloze.

Prompt language in unseen languages (Appendix I, Table 9): When evaluating on languages BLOOMZ never intentionally saw, translating prompts to those languages causes performance drops similar to those for seen languages. Unlike the seen-language case, BLOOMZ-MT performs worse than BLOOMZ on machine-translated prompts for these languages, since xP3mt did not include machine-translated prompts in these specific unseen languages.

Loss normalization by target length (Appendix G, Table 5): Removing the per-target length normalization (described in Section 3.2) implicitly upweights long tasks, since they contribute more tokens. The paper reports that this "worsens accuracy on our NLU validation tasks by 10%" — a substantial degradation, justifying the normalization as necessary to balance short-answer and long-answer task performance during finetuning.

Early stopping, adding long tasks, and upweighting long tasks (Appendix G, Table 5): Three strategies for mitigating the NLG degradation are tested on BLOOMZ-7.1B: early stopping at 6B tokens (vs. 13B), upweighting longer tasks (removing loss normalization), and adding new long tasks (10% of finetuning data). All three "result in longer average generations as shown in Table 5 and slightly better BLEU scores, albeit effects are still small" — BLEU increases from 0.00 to 0.00–0.06 depending on the method. None of these training-side interventions fully solve the problem.

Forcing minimum generation length at inference (Appendix G, Table 5): The most effective intervention is at inference time: forcing the model to generate at least 768 tokens before allowing the end-of-sequence token raises the BLEU score from 0.00 to 9.05 on Multi-EURLEX translation. This is a dramatic improvement and is recommended as the primary remedy: "we recommend forcing a minimum generation length at inference for long generations." Figure 15 shows the qualitative effect on question answering: "Why is the sky blue?" — BLOOMZ answers "Rayleigh scattering." but with a 10-token minimum length it produces "Rayleigh scattering. The blue color of the sky is due to Rayleigh scattering of sunlight by atmospheric molecules."

Causal vs. non-causal finetuning (Appendix J): The paper experimented with non-causal (prefix) finetuning where the model attends bidirectionally over input tokens and causally only over target tokens, motivated by prior work showing benefits (Wang et al., 2022a; Tay et al., 2022c). "However, in our experiments, non-causal finetuning did not improve over causal finetuning." This is a negative result worth noting: the architectural modification that helped in monolingual settings did not transfer to the multilingual MTF setup.

Special token separators (Appendix J): Instead of space-separating inputs and targets for decoder-only models, the paper tried using the end-of-sequence token or a new learned token as a separator. Both "significantly worsened results." The paper hypothesizes that "the models may need to train on more tokens, possibly even during pretraining, to learn these new special tokens," citing Zeng et al. (2022) in support.

Prompt rewriting (Appendix J): The paper experimented with rewriting prompts to be more suitable for decoder-only models (PromptSource was designed for encoder-decoder models where input and target are architecturally separated). This also did not improve over the simple space-separation approach.

BitFit (bias-only finetuning) (Appendix J): Finetuning only the bias terms of the model (Zaken et al., 2021), which has been shown to be effective for downstream task performance in other settings, performed "15 absolute percentage points worse on the average of held-out tasks for BLOOMZ-7.1B" compared to full finetuning. This suggests that the cross-lingual task generalization capabilities require updating more than just the bias parameters.

Replication across architectures: All core findings (cross-lingual transfer from English-only finetuning, improvement from multilingual data, prompt language tradeoffs, language generalization) are replicated across both BLOOM (decoder-only) and mT5 (encoder-decoder) families. This is a robustness check embedded in the main results rather than a separate ablation.

Edit distance analysis for XNLI language generalization (Appendix H, Table 8): To test whether above-random XNLI performance on unseen languages could be achieved through surface-level patterns alone (e.g., entailment pairs having smaller edit distances than contradiction pairs), the paper computes Levenshtein distances between premises and hypotheses across Thai, Turkish, and Greek XNLI samples. The results confirm that "distances are generally largest for neutral samples and smallest for entailment samples," but the differences between labels are small: for Thai, entailment pairs average 79.08 edits vs. 81.52 for contradiction — only 2.44 edits difference. The paper concludes that "comparing characters based on edit distance alone is likely not sufficient to fully explain the language generalization of models," though it doesn't completely rule out surface-level heuristics as a contributing factor.

ReST $^{EM}$ revision model training: Not applicable — this paper does not use reinforcement learning or iterative self-improvement for finetuning. The closest thing to a negative training result is the finding that continued finetuning past the NLG peak degrades generation quality (Figure 7), which is a form of "overtraining" on the short-target-dominated mixture.

Critical Assessment

Claim 1: English-only finetuning suffices for cross-lingual task generalization.

The experiments in Figure 4 do demonstrate that BLOOMZ-P3 substantially outperforms BLOOM on non-English held-out tasks, and that BLOOMZ-P3 improves over BLOOM by more than 50% on multilingual sentence completion. The evidence supports the claim that English-only finetuning improves non-English task performance. However, there are important caveats. First, the absolute performance is still low — BLOOMZ-P3 is not competitive with task-specific finetuned models, nor with BLOOMZ (xP3-finetuned). The claim is about existence of transfer, not about its practical sufficiency. Second, the evaluation is on held-out tasks that are structurally similar to trained tasks (all involve natural language understanding with rank classification), and the transfer may not extend to tasks requiring different output structures. Third, the languages where transfer is demonstrated are those well-represented in BLOOM's pretraining — English-only finetuning transfers to languages the model already knows well from pretraining. The paper doesn't isolate whether transfer would work for a language with minimal pretraining exposure but some finetuning exposure, or vice versa. The claim holds, but its practical boundaries are not fully mapped.

Claim 2: Multilingual data with English prompts (xP3) further improves performance.

Figure 4 shows BLOOMZ > BLOOMZ-P3 for all tasks, and Figure 9 confirms this for English-only evaluation as well. The claim is well-supported for the specific mixture used (xP3). However, the paper cannot disentangle two confounded variables: xP3 adds both multilingual data and more tasks/datasets/prompts. The improvement over P3 could be entirely due to having more tasks and more diverse prompts, not specifically because those tasks happen to be in multiple languages. The paper acknowledges this confound when discussing why mT0-13B beats T0-11B: "This is likely due to xP3 adding additional tasks and prompts, which has been shown to help generalization." The paper never runs a control where the same number of additional tasks are added but all in English — such a baseline would be needed to attribute the gains specifically to multilingual task data. What the experiments actually show is that "xP3 > P3," not necessarily that "multilingual > English-only" when controlling for task count and prompt diversity.

Claim 3: Finetuning on machine-translated prompts improves performance on human-written non-English prompts.

Table 1 demonstrates this clearly for BLOOMZ-MT vs. BLOOMZ on XNLI human-translated prompts (43.88 vs. 40.4) and machine-translated prompts (41.16 vs. 37.56). For mT0, the pattern holds for XNLI and XWinograd but is mixed for XCOPA and XStoryCloze. The tradeoff — English prompt performance declines — is also clearly shown. The claim is well-supported with the caveat that the improvements are modest (a few percentage points) and come at a cost to English-prompt performance. A limitation: the machine translation of prompts was done with Google Cloud API, and the quality of these translations is not evaluated. Errors in machine-translated prompts could actually hurt training — if a prompt is mistranslated, the model learns to follow a garbled instruction, which may not transfer to well-formed human-written prompts. Human-translated evaluation prompts are only available for XNLI, so the generalization from machine-translated training prompts to human-translated test prompts is only tested on one task. The paper also doesn't compare training on machine-translated prompts vs. human-translated prompts — it only compares English vs. machine-translated, so the ceiling (what human-translated training prompts would achieve) is unknown.

Claim 4: Models generalize to tasks in languages never intentionally seen.

The evidence in Figure 5 shows BLOOMZ performing above random on XNLI, XCOPA, XStoryCloze, and XWinograd in languages like Japanese, Thai, and Russian. However, the contamination analysis in Appendix D fundamentally reframes this claim: these languages were incidentally present in the pretraining corpus. What the paper actually demonstrates is not generalization to truly unseen languages, but rather that MTF can activate latent knowledge of languages that appeared only as trace contamination during pretraining. This is a genuinely interesting finding, but it is narrower than the headline "generalization to languages never intentionally seen" might suggest. The paper is transparent about this distinction: "We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic." The edit distance analysis in Appendix H partially rules out surface-level heuristics but doesn't fully explain the mechanism. A missing experiment: test on a language with genuinely zero presence in the pretraining corpus (if one could be identified with certainty — the contamination analysis shows how hard this is). The paper doesn't attempt this, and given the difficulty of guaranteeing zero contamination in web-scale data, it may be infeasible. The more precise version of the claim is: MTF can surface task capabilities from extremely small amounts of incidental language exposure (as little as ~20 million tokens for Thai), even when that language was never a curated part of the training data. This is what the experiments actually show, and it's a robust finding across multiple tasks and languages in Figure 5.

Claim 5: The performance gap between pretrained and MTF models widens with scale.

Figure 6 supports this for BLOOM/BLOOMZ. The gap at 560M is small; at 176B it is large. For mT5/mT0, the trend is harder to assess because mT5 has no meaningful zero-shot baseline. The claim holds for the decoder-only family. A weakness: only one model at the largest scale for each family (BLOOMZ-176B, mT0-13B), and the scaling curve for BLOOMZ between 7.1B and 176B is interpolated from only two points on the far right of Figure 6 — the paper doesn't have a 30B or 65B intermediate checkpoint to verify the trend is smooth. Additionally, the scaling analysis confounds model size with pretraining data quantity (all BLOOM models were trained on the same 366B tokens, which may be suboptimal for the largest model) — a Chinchilla-optimal scaling law might show different patterns.

Claim 6: Multitask finetuning on short targets biases models toward short answers, hurting generative tasks.

Figure 7, Table 2, Table 4, and the qualitative examples in Figures 12 and 15 provide strong converging evidence for this claim. The mechanism (length bias from target distribution) is clearly identified, and the inference-time fix (minimum generation length) is effective, improving BLEU by 9 points. This is one of the paper's most robust and practically actionable findings. The limitation is that the training-side interventions (early stopping, adding long tasks, upweighting) are only tested at the 7.1B scale and show modest effects — it's unclear whether a more thorough data mixture rebalancing would solve the problem at larger scales without the inference-time hack.

Missing Experiments That Would Have Strengthened the Paper

A task-count-controlled comparison: xP3 has more tasks/datasets/prompts than P3. To isolate the effect of multilingual data specifically, the paper should have created a version of P3 with the same number of tasks and prompts as xP3 but all in English. The fact that mT0-13B (multilingual finetuning) beats T0-11B (English-only finetuning on English-only pretraining) is attributed to xP3 having more tasks, not to multilingual data per se. Without this control, the claim that multilingual task data specifically helps remains confounded.
Prompt translation quality evaluation: The paper machine-translates prompts for xP3mt using Google Cloud API but never evaluates the quality of these translations. A human evaluation or back-translation check would quantify how much noise is introduced by imperfect machine translation of prompts, which could explain why gains from xP3mt are modest.
Performance on languages with no contamination: The language generalization finding is qualified by the contamination analysis. Finding even one language with verifiably zero presence in ROOTS and testing whether MTF transfers to it would test the strong claim of language-agnostic capability learning. This may be infeasible given web-scale data realities, but the paper could at least acknowledge the impossibility of proving zero exposure.
Statistical significance / confidence intervals: All reported numbers are point estimates. For a 500-question test set split into 5 language groups and 5 prompts, the variance could be substantial. Figure 10 suggests prompt-level variance is high (dots are widely spread). Without confidence intervals, it's difficult to determine whether the 2-3 point differences between model variants in Table 1 are reliable or noise, especially for the smaller per-language sample sizes.
Evaluation on more generation tasks: The NLG degradation is evaluated primarily on translation and summarization (Figure 7) and code generation (Table 2). Testing on additional generative tasks (dialogue, story generation, long-form QA) would establish whether the length bias is a general phenomenon or specific to certain task formats.
Full finetuning comparison with parameter-efficient methods: The paper shows BitFit performs 15 points worse than full finetuning, but doesn't compare with other parameter-efficient methods like LoRA or prompt tuning, which have shown strong performance in other settings. This would help establish whether full finetuning is necessary or if the cross-lingual transfer can be achieved more efficiently.
Evaluation in languages finetuned on vs. only pretrained on: The paper doesn't disaggregate results by whether a language appeared in the finetuning data or only in pretraining. For example, mT0 is finetuned on 46 languages but pretrained on 101 — do the 55 languages that were only in pretraining show different generalization patterns than the 46 that were also in finetuning? This analysis would help distinguish the contributions of pretraining exposure vs. finetuning exposure to cross-lingual task transfer.

6. Limitations and Trade-offs

Limitation 1: xP3's Language Coverage Is Tied to ROOTS, Which Excludes Most of mT5's Pretraining Languages

The assumption or constraint. The paper constructs xP3 to deliberately mirror the language distribution of ROOTS, BLOOM's pretraining corpus: 46 natural languages plus code. However, mT5 was pretrained on mC4, which covers 101 languages — more than double the language count of xP3. The paper explicitly acknowledges this asymmetry:

"The pretraining corpus of mT0 contains more than 101 languages (Xue et al., 2020), however, we finetune on only 46 languages. Likely, finetuning on the full 101 languages mT0 has seen during pretraining would lead to better performance."

This means the mT0 models are finetuned on only a subset of the languages they actually know from pretraining — roughly 55 languages that mT5 encountered during its 1-trillion-token pretraining receive no task-specific signal during MTF. The paper's stated rationale for this decision is methodological: "we decided to use only the languages of BLOOM in order to study language generalization (§4.2)." This choice prioritizes a clean cross-model comparison over maximizing mT0's absolute performance.

The consequence. The results for mT0 almost certainly understate what multilingual MTF can achieve on an encoder-decoder architecture. The paper never measures performance on the 55 languages that mT5 saw during pretraining but that were excluded from xP3. For those languages, the mT0 models are in the same situation that BLOOMZ is in for incidentally-seen languages — the pretraining exposure exists, but no finetuning data activates it for task execution. The paper also cannot distinguish between two competing explanations for mT0-13B outperforming BLOOMZ-176B: is it the encoder-decoder architecture, the longer pretraining (1T vs. 366B tokens), or the broader language coverage (101 vs. 46 languages)? By finetuning mT5 on only 46 of its 101 languages, the paper introduces a confound: mT0's advantage might be partially architectural, but it is also operating with a finetuning distribution that covers less than half of its pretrained language knowledge. A practitioner deploying mT0 would want to know whether finetuning on all 101 languages would yield substantially better results, but the paper provides no evidence either way.

What evidence exists in the paper. Section 6 explicitly acknowledges this limitation in the third limitation paragraph. Figure 2 visualizes the language distributions of ROOTS, xP3, and mC4 side-by-side, making the coverage gap visually apparent — mC4 contains numerous languages (particularly in the long tail) that are absent from both ROOTS and xP3. The paper notes that "out of those [new languages only seen during finetuning for mT0], we only evaluated on code (HumanEval), where mT0 performed at the random baseline (0.00 in Table 10)." This is a single data point suggesting that finetuning on a new language without corresponding pretraining exposure is insufficient, but it says nothing about the converse case: pretraining exposure without finetuning data.

Mitigation status. The paper takes no steps to mitigate this limitation — it does not create an extended xP3 variant covering all 101 mT5 languages, nor does it evaluate mT0 on the 55 excluded languages to measure what is lost. The authors mention an extended dataset (xP3x, covering 277 languages) as future work but explicitly state they "are yet to finetune models on it." A practitioner wanting to deploy mT0 for languages outside the 46-language xP3 set has no guidance from this paper on expected performance.

Limitation 2: Prompt Translation Quality Is Unevaluated, and the Cost of Non-English Prompt Support Is Not Quantified

The assumption or constraint. The paper constructs xP3mt by machine-translating English prompts into the respective dataset languages using the Google Cloud Translation API. The quality of these translations — whether they are grammatical, idiomatic, or correctly preserve the task semantics — is never evaluated. The paper simply states: "We use the Google Cloud API for machine translation." There is no human evaluation, no back-translation check, and no measurement of how often the machine translation garbles the instruction (e.g., by mistranslating a key term, changing the meaning of answer choices, or producing unnatural phrasing).

Furthermore, the paper evaluates non-English prompt performance primarily through machine-translated evaluation prompts (MT) and, for XNLI only, human-translated prompts (HT). The gap between MT and HT in Table 1 suggests that machine translation quality matters: for BLOOMZ on XNLI, MT prompts score 37.56 while HT prompts score 40.40 (a ~2.8 point gap). For BLOOMZ-MT, the gap narrows to 41.16 vs. 43.88 (~2.7 points). This means the measured performance on machine-translated evaluation prompts understates what users would experience with well-formed human-written prompts. But the complementary question — whether training on imperfect machine-translated prompts actually hurts the model's ability to understand well-formed human prompts — is unexamined.

The consequence. A practitioner reading the paper might conclude that finetuning on machine-translated prompts is beneficial (the gains on non-English prompts in Table 1 are real), but they cannot estimate the ceiling: would human-translated training prompts be substantially better than machine-translated ones? If machine translation errors introduce noise into the training signal (the model learns to follow slightly garbled instructions), the model might develop spurious associations between mistranslated words and task behaviors that don't transfer to natural human prompt phrasing. The ~2.7-point gap between HT and MT evaluation prompts for BLOOMZ-MT suggests this is a real concern — even after training on machine-translated prompts, human-written prompts are still easier. The paper cannot tell us whether this residual gap is due to (a) inherent differences between human and machine phrasing, (b) training noise from imperfect machine translations, or (c) the fact that xP3mt still contains English prompts for cross-lingual tasks, diluting the non-English prompt signal.

Additionally, the observation that BLOOMZ-MT's English-prompt performance drops (52.99 → 49.01 on XNLI, per Table 1) while its non-English performance rises establishes a tradeoff, but the paper provides no quantitative framework for deciding where on this tradeoff curve to operate. If you serve users in 46 languages, do you train on machine-translated prompts for all of them and accept the English degradation? Should you use a mixture of English and machine-translated prompts? At what ratio? The paper offers no guidance.

What evidence exists in the paper. Table 1 reports the EN/MT/HT performance for both base and MT model variants. The consistent gap between MT and HT evaluation prompts for XNLI (visible for both BLOOMZ and mT0, with and without -MT finetuning) is the primary evidence that prompt translation quality matters. Section 6 does not mention prompt translation quality as a limitation. Appendix M provides examples of human-translated and machine-translated prompts side-by-side for XNLI Spanish, allowing inspection of their differences, but no systematic quality analysis.

Mitigation status. Not addressed. The paper treats prompt translation as a black-box preprocessing step. There is no ablation studying the effect of translation quality (e.g., by using a different translation API, by comparing high-resource-language translations to low-resource ones, or by having humans verify a sample). The limitation paragraph in Section 6 does not mention prompt translation fidelity.

Limitation 3: The Task Diversity vs. Multilingual Data Confound Prevents Attributing Gains to Multilinguality Specifically

The assumption or constraint. When the paper compares BLOOMZ (xP3-finetuned) against BLOOMZ-P3 (P3-finetuned), it attributes the improvement to the multilingual nature of xP3. However, xP3 differs from P3 along two confounded dimensions: it adds data in multiple languages and it adds more tasks, more datasets, and more prompts. The paper itself acknowledges this confound when discussing a related finding — that mT0-13B outperforms the fully-English T0-11B on English tasks:

"This is likely due to xP3 adding additional tasks and prompts, which has been shown to help generalization (Chung et al., 2022; Iyer et al., 2022)."

The same logic applies to the P3 vs. xP3 comparison: the improvement could come entirely from having more diverse task types (translation, simplification, program synthesis) and more prompt templates, not specifically from those tasks being in non-English languages. The paper never runs a control where an English-only dataset with the same number of tasks and prompts as xP3 is constructed and compared against, which would be necessary to isolate the effect of multilingual data per se.

The consequence. The paper's central claim — that "finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks" — is true of xP3 as a whole but may not be true of multilinguality specifically. A practitioner might conclude from this paper that adding non-English task data is important for improving their multilingual model, but the evidence could equally support the conclusion that adding any diverse task data, regardless of language, would achieve similar gains. The practical decision these results inform — should I invest in collecting multilingual task data, or should I invest in expanding the diversity of my English task data? — cannot be confidently answered from the experiments in this paper.

The confound is particularly acute for the code tasks. xP3 adds program synthesis and miscellaneous code tasks that are absent from P3. Since BLOOM was pretrained on 11% code, adding code-related finetuning tasks may unlock latent capabilities from pretraining regardless of whether those tasks involve multiple programming languages. The improvement from xP3 over P3 could be primarily driven by better utilization of the code pretraining, not by multilingual NLP data.

What evidence exists in the paper. The confound is visible in Figure 1, which shows the task taxonomy tree: xP3 adds entire new task clusters (translation, simplification, program synthesis, miscellaneous code) that did not exist in P3. The paper acknowledges the task diversity explanation in Section 4.1 when discussing the mT0 vs. T0 result, and again in the BLOOMZ vs. BLOOMZ-P3 comparison: the gains "likely due to xP3 adding additional tasks and prompts." Yet the paper's abstract and introduction frame the finding as demonstrating the value of multilingual finetuning specifically. Ablation experiments that add the same new task types but in English only are not reported — Appendix J ("Ideas that did not work") contains no mention of such an experiment, and none of the model variants in Table 3 represent this control condition.

Mitigation status. Not directly addressed. The paper acknowledges the task diversity confound in passing in two places (Section 4.1 discussion of T0 vs. mT0, and implicitly in citing Chung et al., 2022 for the benefits of scaling tasks), but does not treat it as a limitation that affects the interpretation of the central xP3 vs. P3 comparison. The limitation is absent from Section 6.

Limitation 4: The Generation Length Bias Has Only a Partial Fix, and the Training-Side Tradeoff Between NLU and NLG Remains Unresolved

The assumption or constraint. The paper documents that MTF on the short-target-dominated xP3 mixture causes a systematic bias toward short generations, degrading performance on generative tasks like translation, summarization, and code generation. The evidence is strong: BLOOMZ generates HumanEval solutions with 42% fewer characters and 94% fewer comments than BLOOM (Table 4); validation NLG performance (BLEU) peaks early in finetuning and then declines while NLU (accuracy) continues to improve (Figure 7); and BLOOMZ underperforms BLOOM on HumanEval pass@k for the largest model (Table 2, 176B: pass@1 12.06% vs. 15.52%).

The paper's primary remedy is a forced minimum generation length at inference — masking the end-of-sequence token probability for the first N generated tokens, preventing the model from stopping too early. This is shown to be effective: forcing 768 tokens raises BLEU on Multi-EURLEX translation from 0.00 to 9.05 (Table 5). However, this is an inference-time hack, not a training solution. The model still "wants" to stop early; the forced minimum length overrides this preference, which can produce verbose or repetitive outputs when the model has finished saying what it knows (the model pads or rambles to meet the minimum). The training-side interventions explored (early stopping, upweighting long tasks, adding more long tasks to the mixture) produce only marginal improvements: BLEU scores rise from 0.00 to at most 0.06 (Table 5). Upweighting long tasks, which would be the most principled training fix, "worsens accuracy on our NLU validation tasks by 10%."

The consequence. There is a fundamental tradeoff in the current MTF formulation that the paper does not resolve: you cannot simultaneously optimize for short-answer NLU tasks and long-form NLG tasks with a single finetuning procedure and a single model checkpoint. The validation curves in Figure 7 diverge — NLU keeps improving, NLG peaks and declines. Any chosen checkpoint represents a compromise. The inference-time fix (forced minimum length) patches the symptom but does not address the underlying model bias, and it requires the practitioner to know in advance how long the output should be — feasible for translation (you can estimate target length from source length) but problematic for open-ended generation where the ideal output length is unknown.

For practitioners, this means the BLOOMZ and mT0 models released by the paper are not suitable for generative tasks out of the box. Using them for translation, summarization, or code generation requires either the forced-length trick (with manually chosen thresholds per task) or accepting degraded performance. The paper's recommendation — "forcing a minimum generation length at inference for long generations" — offloads the problem to the user without providing a principled way to choose the minimum length. The 9-point BLEU improvement on Multi-EURLEX is achieved with a 768-token minimum, but this was tuned to match the average target length of that specific dataset (1965 characters ≈ 491 tokens, so 768 tokens ensures generation is at least as long as the target). A user with a different task would need to determine the appropriate threshold through trial and error.

What evidence exists in the paper. Figure 7 shows the divergent NLU/NLG validation curves for BLOOMZ-7.1B. Table 4 quantifies the length bias on HumanEval generations. Table 5 reports the BLEU scores and average generation lengths for various interventions on a 7.1B model. The 10-point NLU degradation from upweighting long tasks is mentioned in Appendix G: "We do not upweight longer tasks, as it worsens accuracy on our NLU validation tasks by 10%." Figure 15 shows qualitative examples of the forced-length trick working for question answering. The full BLOOMZ and mT0 model cards (Table 3) do not mention the length bias or the recommended inference-time fix.

Mitigation status. Partially addressed. The paper identifies the problem clearly, explores multiple training-side interventions (early stopping, upweighting, adding long tasks), and provides a practical inference-time workaround. But the workaround has clear limitations (requires per-task tuning, produces verbose outputs, does not fix the model's underlying preference for brevity) and the training-side solutions all involve unacceptable tradeoffs. The paper does not explore more sophisticated solutions such as: (a) training separate models for NLU and NLG tasks, (b) conditioning the model on desired output length during training so it can be controlled at inference, (c) a two-stage training procedure where NLG tasks are trained first (to lock in generation quality) followed by NLU tasks, or (d) dynamic loss weighting that adjusts the NLU/NLG balance throughout training. The limitation is not mentioned in Section 6.

Limitation 5: The Difficulty Estimation and Task Routing Analogue Is Absent — There Is No Mechanism for Adapting Prompt Language or Finetuning Strategy Per-User or Per-Query

The assumption or constraint. The paper treats prompt language as a static property of the finetuning data: you either finetune on English prompts only (BLOOMZ, mT0) or on a mixture of English and machine-translated prompts (BLOOMZ-MT, mT0-MT). At inference time, you use whatever prompt language matches your deployment scenario and accept the corresponding performance level. Table 1 reveals that this is a forced tradeoff: BLOOMZ-MT gains 3.5 points on human-translated XNLI prompts but loses 4 points on English prompts relative to BLOOMZ. There is no mechanism in the current approach for the model to adapt its prompt-following behavior dynamically based on the language of the incoming prompt — you pick one model variant and accept its per-language performance profile.

This matters because real-world deployments rarely serve prompts in a single language. A multilingual assistant might receive queries in English, Spanish, and Swahili. Serving all three with BLOOMZ means Spanish and Swahili prompt performance suffers; serving all three with BLOOMZ-MT means English prompt performance degrades. The paper provides no answer to the question: which model should I deploy for a mixed-language prompt distribution? The results show that the tradeoff exists but not how to navigate it.

The consequence. A practitioner with a multi-language deployment must either (a) deploy separate models for different languages (doubling or tripling serving costs), (b) deploy BLOOMZ-MT and accept the English degradation (hoping that non-English users benefit more than English users lose), or (c) deploy BLOOMZ and accept that non-English prompters get substantially worse performance (Table 1: BLOOMZ on XNLI human-translated prompts drops to 40.4 vs. 52.99 for English). None of these options is clearly superior without knowing the query language distribution and the relative importance of performance in each language. The paper provides no guidance on making this decision and no analysis of how performance varies as a function of the proportion of non-English prompts in the finetuning mixture (the only two points are 0% non-English prompts in xP3 and a fixed proportion in xP3mt).

Furthermore, the paper does not explore whether prompt language capability can be decoupled from task capability in a more granular way. For example, could the model handle English prompts for translation tasks but Spanish prompts for sentiment analysis? Could a single model learn to follow instructions in any of its training languages depending on the prompt language, without the English/non-English tradeoff? These questions are not addressed because the paper's experimental design treats prompt language as a dataset-level property rather than a per-example variable.

What evidence exists in the paper. Table 1 is the central evidence for the prompt language tradeoff. The gap between EN and HT/MT performance for BLOOMZ and the reverse gap for BLOOMZ-MT quantify the tradeoff's magnitude. Appendix I, Table 9 shows the same pattern for unseen languages. The paper's discussion of the tradeoff is brief: "BLOOMZ-MT raises the average performance on human-translated prompts from 41.13 to 45.55. This comes at the cost of a reduction in its performance on English prompts, from 53.58 to 49.74" (Section 4.3). No analysis explores whether the tradeoff is linear, whether an intermediate mixture proportion would find a better balance point, or whether the degradation in English is uniform across tasks.

Mitigation status. Not addressed. The paper acknowledges the tradeoff as an empirical observation but does not treat it as a deployment limitation. There are no experiments with intermediate prompt language mixtures (e.g., 25% non-English prompts, 50%, 75%) that would map out the tradeoff curve. No inference-time strategy is proposed for handling mixed-language deployments. Section 6 does not mention this as a limitation. The paper also doesn't discuss whether prompt language transfer could be achieved more cheaply through parameter-efficient finetuning (e.g., LoRA adapters for each prompt language) rather than a full-model retraining that forces the tradeoff.

Limitation 6: The Absence of a Unified Model Combining Search, Revisions, and Prompt-Language Adaptation Leaves Practical Performance Gaps

The assumption or constraint. The paper studies three independent axes of variation — finetuning data language (P3 vs. xP3), prompt language (English vs. machine-translated), and base model architecture (decoder-only vs. encoder-decoder) — but examines each axis in isolation, producing separate model variants optimized for different conditions. There is no investigation of whether these axes interact. For example, does finetuning on machine-translated prompts (xP3mt) provide larger relative benefits for encoder-decoder models (mT0) than for decoder-only models (BLOOMZ)? The results in Table 1 hint at such interactions — mT0-13B-MT shows larger gains on XWinograd MT prompts (+5.44 points over mT0-13B) than BLOOMZ-MT does (+1.66 points over BLOOMZ) — but the paper does not analyze them systematically.

More importantly, the paper never combines the best-performing elements of its different variants. The optimal model for English-prompt users is BLOOMZ/mT0 (trained on xP3); for non-English-prompt users, it's BLOOMZ-MT/mT0-MT. But a model that performs well on both prompt types — perhaps through a more sophisticated training mixture, multi-task learning with prompt-language conditioning, or dynamic prompt translation at inference — is never constructed or evaluated. The paper's models represent two points on a tradeoff curve without exploring whether the curve itself can be shifted upward (i.e., whether a model can be good at both simultaneously).

The consequence. The paper leaves on the table the natural next step: a model that combines all the insights — multilingual finetuning data, non-English prompt training, and appropriate architecture — into a single deployment artifact that handles prompts in any language without per-language performance cliffs. The current state of the art, as represented by the paper's released models, forces users to choose between English-prompt quality and non-English-prompt quality. This is not a fundamental limitation — it's an engineering gap in the experimental design — but it means that the paper's models do not represent the ceiling of what the techniques it introduces could achieve. A practitioner deploying today must accept the tradeoff even though the paper's own results suggest that better solutions are possible (e.g., through mixture optimization of prompt languages during finetuning).

Furthermore, the paper does not study whether the prompt language tradeoff interacts with language generalization to unseen languages. If you deploy BLOOMZ-MT (optimized for non-English prompts), does its already-modest performance on incidentally-seen languages like Thai get better or worse compared to BLOOMZ? Appendix I, Table 9 shows that BLOOMZ-MT performs worse than BLOOMZ on machine-translated prompts for unseen languages, suggesting that xP3mt's machine-translated prompt training does not transfer to languages outside its 46-language coverage. But this interaction is not discussed, analyzed, or framed as a limitation.

What evidence exists in the paper. The architectural comparison (Figure 4) and the prompt language comparison (Table 1) are presented in separate sections with no cross-analysis. The interaction between model variant and prompt language is visible by comparing corresponding cells across rows in Table 1, but the paper does not compute or discuss relative improvements. The Appendix I results (Table 9) show the BLOOMZ-MT vs. BLOOMZ comparison on unseen-language prompts, but the paper does not connect this to the seen-language prompt tradeoff.

Mitigation status. Not addressed. The paper's conclusion mentions no plans to study combined variants, and the released model artifacts (Table 3) include only the base variants for each dataset configuration, not any mixtures. The limitation is absent from Section 6. The paper treats its three core model variants (xP3-finetuned, P3-finetuned, xP3mt-finetuned) as the complete set of contributions rather than as ablations whose best features could be combined.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the framing of cross-lingual transfer from an architecture-centric problem to a data-composition problem. Before this work, the dominant assumption was that getting a multilingual model to perform Task X in Language Y required explicit cross-lingual alignment — shared vocabularies, parallel data, language-specific adapters, or translation-based data augmentation. The conceptual model was that language and task knowledge were entangled, and transferring one required transferring the other through deliberately constructed bridges.

What this paper demonstrates — with systematic, architecture-spanning evidence — is that task knowledge and language knowledge are largely separable in pretrained multilingual models, and can be activated independently through careful finetuning data design. The cleanest demonstration is the BLOOMZ-P3 result: a model finetuned on only English tasks with only English prompts improves by more than 50% on non-English held-out tasks in languages it encountered only during pretraining. No cross-lingual alignment was added during finetuning — no parallel data, no translation, no language-specific components. The task capability transferred because the multilingual representations from pretraining already contained the necessary cross-lingual mappings. The finetuning merely activated them.

This is a reframing rather than a paradigm shift. The paradigm — multitask prompted finetuning — was established by Sanh et al. (2022) and Wei et al. (2021). This paper extends that paradigm into the multilingual setting and shows that it works with essentially the same recipe. The contribution is conceptual: it changes what we think we need to add during finetuning to achieve cross-lingual generalization. The answer turns out to be: nothing special — the pretraining already did the hard work. This reframing has practical consequences. It collapses the perceived cost structure for multilingual NLP deployment. If English-only task data suffices to unlock task capabilities across dozens of languages (with multilingual data providing further incremental gains), the annotation bottleneck is far narrower than previously assumed.

The paper also resolves the apparent contradiction between XGLM (Lin et al., 2021) — which found English prompts work best cross-lingually — and the intuition that users should be able to prompt in their own languages. The resolution, laid out in Table 1, is that prompt-following is a capability that must be learned per-language, distinct from task-solving capability. XGLM's models had only seen English prompts, so they could only follow English prompts. This paper shows you can train prompt-following in other languages via machine-translated prompts (xP3mt), and it works — but at a measurable cost to English-prompt performance. The contradiction wasn't about inherent properties of languages; it was about training data composition. This shifts the research question from "which prompt language is best?" to "how do we balance the prompt-language tradeoff for our deployment distribution?"

The identification of incidental language exposure as a sufficient substrate for task generalization (Figure 5, Appendix D) introduces a new diagnostic concept: in the era of web-scale pretraining, "unseen" is a continuous variable, not a binary one. Languages bleed into training corpora through code-mixing, code comments, and curation errors. This paper shows that even trace amounts — as little as 0.006% of the corpus, or ~20 million tokens for Thai — can, when combined with MTF, produce above-random task performance. This doesn't mean models learn new languages from zero exposure (the paper is careful not to claim this). It means the threshold for "enough exposure to be useful after MTF" is orders of magnitude lower than previously assumed. This reframes data curation: aggressively filtering to remove non-target languages may be counterproductive, since those "contaminants" could later be activated into functional capabilities.

The NLU/NLG tradeoff finding (Figure 7, Table 4) is an important diagnostic, not a conceptual shift, but one with immediate practical implications. It reveals that the standard MTF recipe — diverse task mixture, train until validation performance plateaus — hides a genuine conflict between task types. The finding that forcing minimum generation length recovers most of the lost NLG quality is a pragmatic patch, but the deeper message is that the distribution of output lengths in the finetuning mixture shapes model behavior in durable, task-transferable ways. This should prompt more careful thinking about data mixture design in future MTF work, rather than the "throw everything in and hope" approach that had become common.

Finally, the paper's architecture-spanning replication (BLOOM decoder-only, mT5 encoder-decoder) provides confidence that these phenomena are not artifacts of a specific model design. The finding that mT0-13B substantially outperforms BLOOMZ-176B — and that mT0-13B beats the English-only T0-11B on English tasks — is a concrete empirical contribution to the ongoing debate about encoder-decoder vs. decoder-only architectures for transfer learning. It suggests that at current scales, the encoder-decoder + span corruption combination may be more parameter-efficient for zero-shot task generalization than decoder-only + next-token prediction, at least when evaluated on NLU tasks with rank classification.

Follow-Up Research This Work Enables

Prompt-language mixture optimization. The paper treats prompt language as binary: English-only (xP3) or English + machine-translated (xP3mt). The results in Table 1 reveal a clear tradeoff — BLOOMZ-MT gains ~3.5 points on human-translated XNLI prompts but loses ~4 points on English prompts relative to BLOOMZ — but map only two points on what is presumably a continuous curve. A natural follow-up would train models on prompt mixtures ranging from 0% to 100% non-English prompts in increments (10%, 25%, 50%, 75%) and measure English and non-English test performance at each point. This would reveal whether the tradeoff is linear, convex, or has a Pareto-optimal intermediate mixture where most non-English gains are achieved with minimal English degradation. The paper's released xP3mt dataset and finetuning code make this experiment straightforward to run — it is essentially a data sampling experiment with no new data collection required. A strong follow-up would also test whether the optimal mixture depends on the language: does adding Spanish prompts help English performance more or less than adding Swahili prompts? If the benefit of multilingual prompts comes primarily from the high-resource languages in the mixture, practitioners could target prompt translation efforts more efficiently.

Decoupling prompt-following from task-solving through parameter-efficient finetuning. The paper establishes that prompt-following is a per-language capability that can be trained separately from task-solving, but the -MT models achieve this through full-model finetuning, which forces the English/non-English tradeoff. A compelling follow-up would test whether prompt-language adaptation can be achieved through parameter-efficient methods (LoRA adapters, prompt tuning, or bias-only finetuning) that leave the base model's task-solving capabilities intact while adding language-specific prompt-following modules. The experimental design: take BLOOMZ (xP3-finetuned, strong on English prompts), freeze its weights, and train a lightweight adapter on xP3mt data. If this produces near-BLOOMZ-MT performance on non-English prompts without degrading English-prompt performance, it would solve the tradeoff identified in Table 1. The paper's negative BitFit result (15 points worse than full finetuning, Appendix J) suggests this may be non-trivial, but more sophisticated parameter-efficient approaches (LoRA, adapters at different layers) might succeed where bias-only finetuning failed. This experiment would also test whether the prompt-language tradeoff is caused by catastrophic interference during full-model training — if parameter-efficient adaptation avoids the tradeoff, interference is the mechanism and future work can focus on mitigation strategies.

Stress-testing the language generalization threshold. The paper's most surprising finding — that BLOOMZ generalizes to tasks in languages never intentionally included (Figure 5) — is also its most caveat-bound. The contamination analysis (Appendix D, Figure 11) shows these languages are present in ROOTS at trace levels, but cannot determine the minimum exposure needed for this effect. A systematic follow-up would identify languages at the extreme low end of ROOTS contamination (Burmese at 0.00003%, or languages below the detection threshold of the 1% sample), construct MTF models with controlled amounts of additional exposure to these languages during pretraining or finetuning, and measure whether task generalization emerges above some threshold. This is an ambitious experiment requiring pretraining control, but a cheaper version is possible: take BLOOMZ, which already has incidental exposure to ~20M tokens of Thai, and evaluate on a Thai task carefully designed to be unsolvable through surface heuristics (Appendix H's edit distance analysis suggests XNLI may have exploitable surface patterns). If BLOOMZ's Thai performance holds up on a task requiring genuine language understanding (e.g., Thai reading comprehension from XQuAD, or a constructed task that controls for surface cues), the language-agnostic capability hypothesis is strengthened. If performance drops to random, the finding is largely an artifact of evaluation task structure.

Task-count-controlled comparison of English-only vs. multilingual finetuning data. The paper's central comparison — xP3 vs. P3 — confounds multilingual task data with additional tasks, datasets, and prompts. A clean follow-up would construct an English-only dataset that matches xP3 in task diversity, dataset count, and prompt template count — adding English translation tasks (e.g., monolingual paraphrasing instead of cross-lingual translation), English simplification tasks, English code tasks, etc. — and compare models finetuned on this enriched English-only dataset against models finetuned on xP3. If the enriched English-only model matches or exceeds xP3 performance, then the paper's gains come from task diversity, not multilinguality. If xP3 still outperforms, then multilingual data provides a benefit beyond what can be achieved with English data alone. The paper acknowledges this confound in Section 4.1 but does not resolve it; this follow-up would directly address it and clarify what practitioners should invest in — broader English task coverage, or multilingual task collection.

Multilingual MTF on Chinchilla-optimal pretrained models. The paper's base models (BLOOM at 366B tokens, mT5 at 1T tokens) predate the Chinchilla scaling laws (Hoffmann et al., 2022), and the paper notes that "the pre-trained models we use, BLOOM and mT5, are suboptimal in many aspects such as compute allocation" (Section 6). A natural follow-up would apply the xP3 finetuning recipe to more recently released, compute-optimally-pretrained multilingual models (e.g., LLaMA-family multilingual variants, or newer encoder-decoder models trained under Chinchilla-optimal budgets) and measure whether the cross-lingual generalization patterns replicate and whether the NLU/NLG tradeoff persists at different pretraining quality levels. This would test whether the paper's findings are robust to improvements in base model quality or are partially artifacts of the specific (under-trained for their size) models used.

Training a model to predict difficulty or prompt-language preference from the input. Although this paper does not involve test-time compute allocation or difficulty estimation, the finding that prompt-language transfer involves a tradeoff (Table 1) suggests a related adaptive mechanism: can a single model learn to handle prompts in any language without the static tradeoff? A follow-up would add prompt language as an explicit conditioning signal during finetuning — for example, prepending a language tag to the input — and training on a mixture of English and non-English prompts where the tag matches the prompt language. At inference, the user sets the tag to their prompt language. If this matches the per-language performance of dedicated BLOOMZ (for English) and BLOOMZ-MT (for non-English) without degradation, it would demonstrate that the tradeoff in the paper is a training data limitation, not a fundamental capacity limitation of the model.

Practical Applications and Downstream Use Cases

Low-resource language task deployment without per-language annotation. The paper's most directly actionable finding for practitioners is that English-only MTF (P3) on a multilingual pretrained model yields non-trivial zero-shot task performance across languages. For an organization wanting to deploy, say, a sentiment analysis system in 20 languages where labeled data exists only in English, the recipe is: take BLOOM or mT5, finetune on English P3 (or the richer xP3 if multilingual data is available), and evaluate zero-shot on the target languages. The paper quantifies the baseline: BLOOMZ-P3 (English-only finetuning) improves over pretrained BLOOM by more than 50% on multilingual sentence completion. While absolute performance is not at production levels for many applications, it provides a starting point that requires no labeled data in the target languages — a dramatic cost reduction over per-language annotation. For applications where moderate accuracy is acceptable (content filtering, triage, pre-labeling for human review), this may be sufficient as-is. For higher-stakes applications, it provides a warm-start for active learning or lightweight per-language adaptation.

Prompt localization for multilingual user-facing assistants. The finding that finetuning on machine-translated prompts (xP3mt) improves human-written non-English prompt performance by 3-4 percentage points on XNLI (Table 1) provides a concrete deployment strategy for multilingual chatbots and assistants. If serving users who prompt in their native languages, deploy BLOOMZ-MT or mT0-MT rather than the English-prompt-optimized variants. The cost is a ~4-point degradation on English prompts, so the decision hinges on the query language distribution: if >50% of queries are non-English, the -MT variant likely provides net benefit. The paper's numbers allow this tradeoff to be estimated quantitatively: average the expected per-language gains and losses weighted by your query distribution. For organizations with the resources to translate prompts via API (Google Cloud Translation, as used in the paper), xP3mt provides a template for constructing the finetuning data without human translation costs.

Data mixture design for multitask finetuning to avoid length bias. The NLU/NLG tradeoff (Figure 7) is a negative finding with a practical prescription: if your downstream deployment includes generative tasks (translation, summarization, code generation, long-form QA), do not rely on the standard MTF checkpoint selection. Either (a) early-stop based on NLG validation performance and accept lower NLU accuracy, (b) add more long-generation tasks to your finetuning mixture (the paper shows adding 10% long tasks helps marginally, Table 5), or (c) use the inference-time minimum-generation-length trick, which recovers 9 BLEU points on translation. The forced-length approach requires per-task tuning of the minimum length, but for tasks with predictable output length (translation, where target length correlates with source length; code generation, where docstrings suggest function complexity), this is feasible. The paper's quantitative evidence — BLOOMZ's 42% shorter HumanEval solutions (Table 4), 94% fewer comments — provides a diagnostic that practitioners can check on their own models by comparing average generation lengths against their pretrained base model.

Self-improvement and data generation pipelines in multiple languages. The paper demonstrates that cross-lingual task generalization works, meaning a model finetuned on xP3 can be used to generate pseudo-labels or training data in languages where no supervised data exists. A practical pipeline: use BLOOMZ or mT0 to generate task outputs for unlabeled data in a low-resource language, filter by confidence (using rank classification score or majority voting across prompts), and use the high-confidence generations to fine-tune a smaller, specialized model for that language. The paper's per-language performance numbers (Appendix K, Table 10) allow estimation of pseudo-label quality for each language-task combination, informing which languages this pipeline is viable for and which require human annotation.

When to Prefer This Method

The paper does not explicitly position its finetuning variants against each other as a decision rule (the variants test different hypotheses, and the paper's purpose is to understand cross-lingual generalization, not to prescribe deployment choices). However, the results in Table 1 implicitly define a tradeoff that practitioners must navigate. As the paper does not frame "prefer X when" as a contribution, no such matrix is included here. The practical decision space between BLOOMZ, BLOOMZ-MT, BLOOMZ-P3, and their mT0 equivalents is discussed under Practical Applications and Downstream Use Cases above.