BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

ArXiv: 2212.09535

🎯 Pitch

Adapter-based fine-tuning beats continued pretraining for adapting large language models to new languages—the opposite of what prior work on smaller models recommends. With just 100K samples, zero-shot performance depends almost entirely on how much adaptation data you have, not whether the language uses a familiar script or word order.

---

1. Executive Summary

This paper empirically studies how to adapt the BLOOM multilingual language model to eight new languages not seen during pretraining—German, Russian, Bulgarian, Thai, Turkish, Greek, Korean, and Guarani—using a resource-constrained setting with at most 100K monolingual samples per language. The authors benchmark three language adaptation strategies: continued pretraining (continual causal language modeling on target-language text), MAD-X (bottleneck adapters inserted into Transformer blocks with invertible adapters at the embedding layer), and (IA)³ (element-wise rescaling vectors that modify inner activations), evaluating zero-shot prompting performance across five NLU benchmarks. Their central finding is that, contrary to prior work on smaller masked language models recommending continued pretraining for models below 1B parameters, adapter-based strategies outperform continued pretraining for BLOOM models with at least 3 billion parameters—a reversal attributed to how the quality of language-independent representations degrades under continued pretraining as model scale increases—while approximately 100 million tokens of adaptation data are needed for effective zero-shot prompting, establishing that language specifics such as writing system or language family do not significantly determine adaptation success when adaptation data quantity is held constant.

2. Context and Motivation

The Core Problem: Expanding Language Coverage Without Retraining From Scratch

The fundamental question this paper tackles is straightforward but practically urgent: can you add support for a new language to an already-trained large language model without spending tens of millions of dollars to retrain it from scratch? This matters because the largest multilingual models—even ones designed explicitly for broad language coverage like BLOOM—inevitably exclude languages. BLOOM's 176B-parameter model covers 46 natural languages, which sounds expansive until you notice what's missing: Korean (tens of millions of speakers), Russian (hundreds of millions of speakers across multiple countries), and German (a high-resource language by any reasonable definition). The absence is not due to a principled decision about which languages matter—it is a contingent artifact of which communities had sufficient expertise, data access, and validation capacity during the BigScience project's data curation phase.

The authors motivate their work by pointing out that the factors driving language exclusion from BLOOM are organizational and logistical, not technical or linguistic:

"This limitation was driven by a number of factors, most notably only considering languages for which the community had enough expertise to manually validate the data quality, deduplicate and remove personally identifiable information and had sufficient access to licensed unlabeled text. All of these factors are contingent facts about the group that trained the model, and leave open the idea that other researchers could contribute more languages."

This framing positions the problem not as "BLOOM is insufficiently multilingual" but rather as "any single training run will exclude languages for reasons that have nothing to do with their linguistic importance or speaker population—so we need reliable post-hoc methods for adding them."

Why Retraining Is Not the Answer

The authors provide a clear economic argument against the obvious alternative. Retraining BLOOM from scratch to include additional languages is "prohibitively expensive"—BLOOM was pretrained on roughly 350 billion tokens using a 176B-parameter model, a computational undertaking that cost millions of dollars and required months of coordinated effort across a large distributed research team. For most languages, especially low-resource ones, this cost cannot be amortized: the marginal benefit of adding, say, Guarani (a Native American language with relatively few speakers) to a model's pretraining data does not justify the marginal cost of a full retraining run.

This creates a sharp practical tension. On one hand, the field's stated goal is to make language technology available across the world's roughly 7,000 languages, not just the 46 that happened to be included in BLOOM. On the other hand, the economic logic of large-scale pretraining—where fixed costs are enormous and must be spread across many use cases—naturally concentrates development on languages with large speaker populations or commercial markets. Language adaptation offers a potential escape from this tension: train one large model once, then let individual communities adapt it to their language at a fraction of the original cost.

The Prompting Imperative

A second motivation that the paper emphasizes is the growing importance of zero-shot prompting as a deployment paradigm. Traditional finetuning requires labeled data for every downstream task in every target language—a multiplicative cost that quickly becomes infeasible for low-resource languages. Prompting, by contrast, reformulates NLP tasks into the format the model was already trained on (causal language modeling), allowing the model to generalize to new tasks from instructions alone, without task-specific gradient updates.

The authors explicitly connect these two ideas:

"Prompting can benefit many languages that lack large amounts of labeled data as it allows language models to generalize to a wide range of tasks with significantly less training cost and data than full finetuning."

This matters for language adaptation because the evaluation regime (zero-shot prompting) determines what "successful adaptation" means. You are not asking whether a language-adapted model can be finetuned to high accuracy on a single task in the new language. You are asking whether it can understand instructions and reason about language tasks in that language without any task-specific training—a much higher bar that tests whether the model has genuinely internalized the language, not just memorized labeled examples in it.

The combination—language adaptation + zero-shot prompting—represents what the authors see as the most promising path to scalable multilingual NLP: one expensive pretraining run, community-driven language adaptation at manageable cost, and task-generalization via prompting rather than per-task finetuning.

The Confusing State of Prior Evidence

The paper enters a literature where the dominant recommendation for language adaptation has been continued pretraining—simply resuming the causal language modeling objective on target-language text. Ebrahimi and Kann (2021), the most directly comparable prior work, performed a systematic comparison of language adaptation strategies for XLM-R (a masked language model family with 100+ pretraining languages) across 30 new languages and concluded:

"continued pretraining is the most promising direction."

This recommendation was based on experiments with models in the hundreds-of-millions to roughly one-billion parameter range—standard sizes for encoder-only masked LMs like mBERT and XLM-R at the time. The logic was intuitive: continued pretraining lets the model adjust all parameters to the new language's distribution, whereas adapters only modify a small subset, so the former should be more expressive and thus more effective.

However, several factors made this conclusion suspect for the BLOOM setting:

Model scale had not been systematically studied. Ebrahimi and Kann's experiments topped out around model sizes where all parameters can reasonably be updated on a modest budget. The behavior of continued pretraining at the 3B+ parameter scale—where BLOOM enters the regime the authors call "large"—was unknown. The authors note:

"the cost of such pretraining will grow with the size of the pretrained model and can be prohibitive for many researchers working with low-resource languages."

This is not just a cost concern; it is a hypothesis about a qualitative change in how models of different sizes learn from continued pretraining. The paper's sentence retrieval experiments in Section 4.3 suggest that larger models, given the freedom to update all parameters, may diverge their representations away from the language-independent space that enables cross-lingual transfer—a mechanism that would not manifest at the smaller scales where prior work operated.

Prior work studied token-level and sequence-labeling tasks (NER, POS tagging) under finetuning, not prompting. The adaptation requirements for zero-shot prompting might be different. For prompting, the model must not merely recognize entities in the new language—it must understand task instructions, reason about logical relationships between sentences, and map its output tokens to task-specific verbalizers (e.g., "Yes" vs. "No" for entailment). This is a substantively different capability than the token classification tasks that drove previous recommendations.

BLOOM's architecture differs from the encoder-only models studied previously. BLOOM uses AliBi positional embeddings (which enable length generalization) and a decoder-only architecture, whereas prior language adaptation work focused on encoder-only masked LMs like mBERT and XLM-R. Whether adaptation strategies transfer across these architectural families was unknown.

The resource-constrained setting—maximum 100K samples per language—is more realistic for the languages that most need adaptation. Languages with millions of available documents probably already made it into the pretraining data distribution. The languages that need post-hoc adaptation are precisely those with limited available text, so an adaptation strategy that requires massive monolingual corpora defeats the purpose. The authors explicitly frame this as a constraint: their goal is to test what works "in the resource-constrained settings, where we only use a limited amount of samples (maximum 100K samples) for each language." Prior work did not always enforce this constraint.

Where Existing Adaptation Methods Fall Short

The paper identifies specific gaps in the three families of language adaptation it considers:

Continued pretraining: cost scales poorly, and quality may not. While continued pretraining has been the default recommendation, the authors note that it requires updating all model parameters, which becomes increasingly expensive as models grow. More subtly, the paper's results suggest that even when continued pretraining achieves lower perplexity on held-out target-language text—indicating better language modeling—it can still produce worse downstream prompting performance than adapters. This decoupling of perplexity from task performance (documented in Figure 4 and discussed in Section 4.2) challenges the assumption that better language modeling automatically yields better task capabilities.

Additionally, the authors observe (Appendix L, Figure 18) that continued pretraining causes substantial catastrophic forgetting of English, the language on which many prompting capabilities depend (since prompt templates and reasoning patterns are often learned primarily from English data). Adapters, by freezing the base model, largely avoid this problem.

MAD-X (bottleneck adapters): strong but underexplored at scale. MAD-X was developed and validated primarily on masked language models with a few hundred million parameters. Its behavior on large decoder-only models under a prompting evaluation regime was unexplored before this work. The paper's preliminary experiments also note that Composable Sparse Fine-Tuning (C-SFT), despite outperforming MAD-X on POS and NER tasks in prior work (Ansell et al., 2022), produced models that are "not capable of prompting" in the authors' setting (Appendix D, Table 2)—a stark example of how task-level evaluation changes the ranking of adaptation methods.

(IA)³: newer, less tested for language adaptation. (IA)³ was originally proposed as a parameter-efficient method for task-specific finetuning on English NLU benchmarks. The authors are among the first to apply it to language adaptation, where the goal is not to learn a single downstream task but to teach the model an entire new linguistic system. The method's effectiveness in this cross-lingual, prompting-based setting was unknown.

The Language Diversity Gap

The paper's selection of eight target languages is deliberate and covers a designed range of linguistic variation that tests whether adaptation strategies are truly general or merely work for languages similar to those already seen during pretraining:

• Script coverage: German and Turkish use the Latin script (seen in BLOOM's pretraining data); Bulgarian and Russian use Cyrillic (unseen); Greek uses the Greek script (unseen); Korean uses Hangul (unseen); Thai uses the Thai script (unseen and, uniquely, not space-separated).
• Language families: Indo-European (Germanic, Slavic, Hellenic), Turkic, Koreanic, Tai-Kadai, and Tupian (Guarani)—five distinct families, only one of which (Indo-European) is well-represented in BLOOM's pretraining data.
• Word order: Mostly SVO, but Turkish and Korean are SOV—a structural difference that tests whether the model's learned syntactic biases transfer or interfere.
• Resource level: Guarani is a genuine low-resource language with only approximately 30K available sentences in the authors' chosen corpus (Jojajovai), compared to 100K for the other languages. This tests the lower bound of adaptation data requirements.

The authors explicitly state that their language selection is motivated by a desire to see whether adaptation strategies are sensitive to these linguistic dimensions. Their finding (Section 4.1, Figure 2) that "BLOOM adapts well to new languages regardless of their language family, word order, and whether they share the same script system with languages in pretraining data" is not an obvious result—it contradicts earlier work by Yong and Nikoulina (2022) that found script mismatch to be a significant barrier for an earlier version of BLOOM.

How This Paper Positions Itself

The paper positions itself at the intersection of two research threads that had not previously been connected systematically:

Language adaptation methods have been studied extensively, but primarily on masked language models, under finetuning evaluation, at smaller scales, and with permissive data budgets. The findings from that literature (favoring continued pretraining) were accepted as general despite limited evidence of their transfer to large decoder-only models under prompting evaluation.

Zero-shot prompting has emerged as a promising paradigm for multilingual NLP (through work on mT0, BLOOMZ, and XGLM), but the question of how to make prompting work for languages absent from pretraining had not been addressed. The existing approach was simply to include the target language in the expensive pretraining run—exactly the cost the authors want to avoid.

The paper's novel contribution is to test the language adaptation playbook from the masked-LM era against the reality of large decoder-only models evaluated via prompting, and to identify where the old recommendations break down. The key conceptual move is to attribute the breakdown to a specific mechanism—the degradation of language-independent representations under continued pretraining at scale (Section 4.3)—rather than to vague statements about "model size." This mechanism, instantiated through sentence retrieval experiments, provides an explanation for why the recommendation flips (adapters over continued pretraining for large models) rather than just documenting that it flips.

The paper also explicitly positions itself as filling a gap left by the BLOOM project itself. BLOOM was developed under BigScience, the same initiative that produced this paper, and the authors frame their work as enabling the "other researchers could contribute more languages" vision that the original BLOOM paper articulated but could not realize within a single training run. This gives the work a clear practical motivation: it is not an abstract study of adaptation methods, but a concrete proposal for how the community can extend BLOOM's language coverage without requiring the resources of the original training consortium.

3. Technical Approach

3.1 Reader Orientation

The "system" in this paper is not a single trained model but rather a set of comparative experimental protocols for adding a new language to an already-pretrained large language model—specifically BLOOM—using a limited budget of monolingual text, with the goal of enabling the model to perform zero-shot prompting (answering NLU tasks from instructions alone, without task-specific training) in that new language. The problem the paper solves is an experimental-design question: given a fixed resource constraint (at most 100K sentences in the target language, roughly 100–200 million tokens), which of several language adaptation strategies—continued pretraining, MAD-X bottleneck adapters, or (IA)³ rescaling vectors—yields the best downstream zero-shot prompting accuracy, and how does the answer change as the base model scales from 560M to 7.1B parameters?

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has five major stages, each feeding into the next:

1. Base Model Selection: Choose a BLOOM decoder-only Transformer checkpoint at one of four scales (560M, 1.1B, 1.7B, 3B, 7.1B parameters), pretrained on the ROOTS corpus covering 46 natural languages—none of which are the eight target languages.
2. Language Adaptation Training: Apply one of three adaptation strategies (continued pretraining, MAD-X, or (IA)³) to the frozen pretrained model using monolingual text from the target language, producing an adapted model that can now generate text in that language. The adaptation data is capped at 100K sentences (except Guarani, which uses 30K), simulating a resource-constrained setting.
3. Checkpoint Selection: Evaluate perplexity on held-out target-language text every 5,000 training steps and select the checkpoint with the lowest perplexity for downstream evaluation—a standard but, as the paper shows, potentially misleading proxy.
4. Zero-Shot Prompting Evaluation: For each adapted model, format NLU tasks (XNLI, XCOPA, XWinograd, XStoryCloze, PAWS-X, KLUE-NLI, AmericasNLI) using translated prompt templates and evaluate accuracy without any task-specific gradient updates. The model receives a prompt containing the task input and answer choices, and its generated token is mapped to a task label via a verbalizer (e.g., "Ja" → entailment, "Nein" → contradiction for German XNLI).
5. Comparative Analysis: Aggregate results across languages, model sizes, and adaptation strategies to identify scaling trends, compute-performance tradeoffs, and the key factors (data quantity, model scale, language-independent representation quality) that determine adaptation success.

Information flows linearly: pretrained model → adaptation on monolingual data → checkpoint selection via perplexity → zero-shot prompting on translated templates → accuracy comparison. The critical branching point is at stage 2, where the same base model is adapted via three different strategies, and at stage 4, where the same adapted model is tested across multiple tasks and languages.

3.3 Roadmap for the Deep Dive

• First, the base BLOOM architecture and the eight target languages, since the adaptation strategies' effectiveness depends on whether the model architecture and tokenizer can handle the linguistic properties of the new languages without catastrophic interference.
• Second, the three adaptation strategies in detail—continued pretraining, MAD-X, and (IA)³—their mechanisms, their parameter counts, and why they represent different points on the expressivity-vs-cost spectrum.
• Third, the resource-constrained adaptation setting (data quantities, training hyperparameters, sequence length) and the checkpoint selection protocol, since the amount of adaptation data is the paper's primary independent variable beyond model scale.
• Fourth, the zero-shot prompting evaluation framework—the task templates, the verbalizer mapping, and why this evaluation regime matters for language adaptation (it tests whether the model has genuinely internalised the language, not just memorised task-specific patterns).
• Fifth, the sentence retrieval experiments that probe the quality of language-independent representations, since these provide the mechanistic explanation for the paper's central finding (the adapter-over-continued-pretraining reversal at scale).
• Sixth, the BLOOMZ adaptation experiments, which test whether the findings from base BLOOM transfer to instruction-tuned variants and explore whether adding the new language during multitask finetuning is more effective than post-hoc adaptation on unlabeled text.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical benchmarking paper whose core idea is that the optimal strategy for adapting a pretrained multilingual language model to a new language depends on the base model's scale, and that prior recommendations—derived from smaller masked language models evaluated under finetuning—do not transfer to large decoder-only models evaluated under zero-shot prompting.

---

BLOOM Base Architecture and Why It Matters for Language Adaptation

The BLOOM family consists of decoder-only Transformer models ranging from 560 million to 176 billion parameters, all sharing the same architectural blueprint. Understanding this architecture is essential because the adaptation strategies interact with specific architectural choices in ways that determine their effectiveness.

Decoder-only Transformer with AliBi. BLOOM uses a standard autoregressive (causal) Transformer decoder: each token attends only to previous tokens in the sequence, and the model is trained with a next-token prediction objective—given the prefix, predict the next token. Unlike the original Transformer, which uses learned sinusoidal positional embeddings, BLOOM uses AliBi (Attention with Linear Biases), which adds a fixed, non-learned bias to the attention scores that penalises attention between distant tokens linearly with their distance. The bias is head-specific and is not trained. The paper's Section 6.3 notes that the choice of AliBi over learned positional embeddings is significant because it removes the need to adapt positional embeddings when extending to new languages—a complication that plagued earlier work (Yong and Nikoulina, 2022) where both word and positional embeddings had to be finetuned.

Layer normalization after embeddings. BLOOM applies layer normalization immediately after the token embedding lookup and before the first Transformer block. This is a design choice that stabilises training but also means that the embedding layer's outputs are normalised before being mixed with positional information, which affects how invertible adapters at the embedding layer interact with the rest of the model.

Byte-level BPE tokenizer. BLOOM's tokenizer uses Byte-Level Byte Pair Encoding (BPE) with a vocabulary size of 250,680 tokens. The critical property for language adaptation is that byte-level tokenizers never produce unknown tokens. Any Unicode character sequence, regardless of whether the script appeared in pretraining data, can be encoded as a sequence of byte-level subword units. This means—and the authors explicitly state this—:

"we can perform language adaptation without extending the vocabulary."

This is a deliberate and consequential design choice. Prior language adaptation work on models with character-level or standard BPE tokenizers (e.g., XLM-R, mBERT) often required vocabulary expansion—adding new tokens for characters or subwords in the target language's script and training new embedding rows for those tokens. Vocabulary expansion introduces a cold-start problem: the new embeddings are randomly initialised while the rest of the model is pretrained, creating a mismatch that must be resolved during adaptation. By using byte-level BPE, BLOOM avoids both the engineering complexity of vocabulary expansion and the cold-start problem—the tokenizer already "knows" how to segment text in any script, even if the model's parameters have never been trained on text in that script.

Parameter counts and layer configurations. The BLOOM models used in this paper span five scales (Appendix H implies these from the model names: 560M, 1.1B, 1.7B, 3B, and 7.1B parameters). The exact layer counts and hidden dimensions are not specified in the paper text (they refer to the original BLOOM paper, Scao et al., 2022), but the scaling trend is what matters: as parameter count increases, both the total number of trainable parameters available for continued pretraining and the capacity of adapter modules grow, creating different tradeoffs at different scales.

Why architecture matters for the adaptation strategy choice. This architecture creates three conditions that shape the paper's findings:

1. No vocabulary barrier: byte-level BPE means the tokenizer can segment any script, so adaptation effort is spent entirely on teaching the model parameters to process those segments correctly, not on learning new embedding vectors from scratch. This removes one advantage that continued pretraining has in settings requiring vocabulary expansion (since continued pretraining can update the new embeddings alongside everything else, while adapters only modify a subset of parameters).

2. Frozen positional information: AliBi's fixed biases mean the model's notion of token position is not learned from data and does not need to be adapted to new languages. This removes a potential source of interference—positional embeddings trained primarily on Indo-European word orders do not need to be retrained for SOV languages like Turkish and Korean.

3. Decoder-only autoregressive objective: the pretraining task (predict the next token) is identical to the adaptation task (continued pretraining also predicts the next token) and similar to the evaluation task (prompting generates tokens autoregressively). This means language adaptation directly transfers to the evaluation regime without the train-test mismatch that would exist for masked language models adapted via MLM and then evaluated via prompting.

---

The Eight Target Languages: A Deliberately Diverse Testbed

The paper's language selection is an experimental design choice, not a convenience sample. Table 1 in Section 3.2 documents six linguistic dimensions for each language, and the rationale is explicit: the authors want to test whether these dimensions predict adaptation difficulty.

Languages and their properties (from Table 1):

The selection covers:

• Three writing systems unseen in BLOOM's pretraining: Cyrillic (Bulgarian, Russian), Greek, Hangul (Korean), and Thai script. BLOOM's pretraining languages (Table 7, Appendix M) include primarily Latin-script, Chinese, Arabic, and various African languages in Latin script. This tests whether byte-level BPE tokenization is sufficient for unseen scripts, or whether the tokenizer's statistical patterns—learned from pretraining data—produce suboptimal segmentations for new scripts (e.g., over-fragmenting Hangul syllables into individual bytes rather than meaningful morphological units).

• Two word orders: SVO (subject-verb-object), matching English and most pretraining languages, and SOV (subject-object-verb), which is structurally different. If the model has learned SVO-specific syntactic biases during pretraining, languages with SOV order (Turkish, Korean) might be harder to adapt because the model must unlearn those biases or learn to route syntactic processing through different internal pathways.

• Five language families, only one of which (Indo-European) is well-represented in BLOOM's pretraining data. Turkic, Koreanic, Tai-Kadai, and Tupian are all outside the pretraining distribution. This tests whether genetic relatedness to pretraining languages matters—do Indo-European target languages like Russian benefit from transfer from BLOOM's existing Indo-European languages (French, Spanish, Portuguese, Hindi, Bengali, etc.), or is adaptation data quantity the dominant factor regardless of relatedness?

• One genuinely low-resource language: Guarani has only approximately 30K available sentences from the Jojajovai parallel corpus (Chiruzzo et al., 2022), compared to 100K for all other languages from OSCAR. This tests the lower bound of adaptation data requirements—can a model adapt with only ~1 million tokens of target-language text, or is there a threshold below which adaptation is ineffective?

• One non-space-separated language: Thai does not use spaces between words, making word boundary detection a model-internal inference rather than an explicit token. This tests whether the model's learned notion of word boundaries (acquired primarily from space-separated pretraining languages) transfers to a script where word segmentation must be inferred from character co-occurrence patterns.

Why this matters for the paper's conclusions. The authors' central finding—that language specifics do not determine adaptation success when data quantity is held constant—is only convincing because the testbed includes languages that differ along all these dimensions. If the testbed had included only Indo-European languages with Latin script, the claim that "writing system doesn't matter" would be vacuous. The inclusion of Thai (non-space-separated, unique script) and Korean (SOV, unique script) makes the finding meaningful: even these structurally and orthographically distant languages adapt as well as German when given the same amount of text.

---

Language Adaptation Strategy 1: Continued Pretraining

Core mechanism. Continued pretraining is the simplest adaptation strategy: take the pretrained BLOOM model, freeze nothing, and resume training with the same causal language modeling objective on monolingual text in the target language. The model sees sequences of tokens from the new language and is trained to predict the next token at each position, exactly as during the original pretraining, but now the data distribution is exclusively the target language rather than a mixture of 46 languages and code.

What parameters are updated. Every parameter in the model is trainable: all Transformer block weights (attention projections, feedforward layers, layer norms) and the embedding layer. The embedding layer is explicitly made trainable (Section 3.4), which the authors note "follows prior work on language adaptation." This is important because the embedding layer is the first point of contact between the tokenizer's output and the model's internal representations. If the embedding vectors for byte-level subword units have been shaped entirely by pretraining languages, making them trainable during continued pretraining allows them to shift to better represent the target language's byte-pair patterns.

Training signal. The model receives a token-level cross-entropy loss for each position in each sequence. For a sequence of length $L$ with tokens $x_1, \ldots, x_L$, the loss at position $i$ is:

$\mathcal{L}_i = -\log P_\theta(x_i | x_{<i})$

where $P_\theta(x_i | x_{<i})$ is the model's predicted probability for the true next token $x_i$ given the prefix $x_1, \ldots, x_{i-1}$, and $\theta$ represents all model parameters.

What this equation computes: the negative log-likelihood of the correct token under the model's predicted distribution, summed over all positions in the batch. A lower loss means the model assigns higher probability to the actual next token in the target language text—it is becoming a better language model of the target language.

Why this form: the autoregressive next-token prediction objective is identical to BLOOM's pretraining objective. Using it for continued pretraining means there is no objective mismatch—the adaptation signal is perfectly aligned with what the model was originally trained to do. The risk, however, is that the model has already converged (to a local optimum of the loss landscape on the pretraining data distribution) and resuming training on a narrow, single-language distribution may cause it to overfit to the target language at the expense of its original capabilities (catastrophic forgetting, documented in Appendix L, Figure 18).

Cost model. Continued pretraining updates all parameters, so its computational cost scales directly with model size. For a model with $N$ parameters, each training step requires computing gradients for all $N$ parameters, storing them in optimizer states (typically 2–3× the parameter count for Adam), and applying the updates. The paper reports (Figure 3) that continued pretraining consistently requires more GPU memory and more total training time than adapter-based methods, with the gap widening at larger model sizes.

Why continued pretraining as a baseline. Continued pretraining represents the "maximum expressivity" point on the adaptation spectrum: the model can adjust every parameter to accommodate the new language, unconstrained by the bottleneck of small adapter modules. Prior work (Ebrahimi and Kann, 2021) found this expressivity advantage decisive for smaller models. The paper includes it to test whether this advantage persists, reverses, or interacts with model scale in unexpected ways.

---

Language Adaptation Strategy 2: MAD-X (Bottleneck Adapters + Invertible Adapters)

The MAD-X framework, originally proposed by Pfeiffer et al. (2020) for cross-lingual transfer in masked language models, consists of two component types that are trained separately but work together: language-specific bottleneck adapters inserted into each Transformer block, and invertible adapters at the embedding layer to align the embedding space across languages.

Bottleneck adapters (language adapters). A bottleneck adapter is a small feedforward module inserted into each Transformer layer, positioned after the feedforward sublayer. Its architecture is a down-projection followed by an up-projection with a non-linearity in between, forming a bottleneck that forces information to pass through a lower-dimensional representation:

$h_{\text{out}} = W_{\text{up}} \cdot \text{ReLU}(W_{\text{down}} \cdot h_{\text{in}}) + h_{\text{in}}$

where $h_{\text{in}} \in \mathbb{R}^d$ is the Transformer block's output at a given position, $W_{\text{down}} \in \mathbb{R}^{r \times d}$ projects it down to a bottleneck dimension $r \ll d$, $\text{ReLU}$ applies a non-linearity, $W_{\text{up}} \in \mathbb{R}^{d \times r}$ projects it back up to the original dimension $d$, and the result is added to $h_{\text{in}}$ via a residual connection.

What this equation computes: a low-rank modification of the model's hidden representation at each layer. The down-projection compresses the $d$-dimensional hidden state into an $r$-dimensional bottleneck (where $r$ is determined by a reduction factor—Section 4.5 sweeps reduction factors that control $r = d / \text{reduction\_factor}$); the ReLU introduces non-linearity, allowing the adapter to learn transformations that cannot be expressed purely linearly; the up-projection decompresses back to $d$ dimensions; and the residual connection means the adapter learns a delta to the original representation rather than replacing it entirely. At initialisation, $W_{\text{down}}$ and $W_{\text{up}}$ are initialised such that the adapter approximates the identity function (outputs near-zero delta), so the model's behaviour at the start of adaptation training is identical to the unadapted model.

Why this form: the bottleneck architecture is parameter-efficient. For a reduction factor of, say, 16, an adapter in a model with hidden dimension $d = 4096$ (typical for BLOOM-3B) has only $2 \times 4096 \times (4096/16) = 2 \times 4096 \times 256 \approx 2.1\text{M}$ parameters per layer, versus approximately $2 \times 4096^2 = 33.6\text{M}$ parameters for a full feedforward layer. The residual connection and near-identity initialisation ensure that the model can gradually specialise to the new language without abrupt disruption of pretrained representations—the adapter can choose to pass information through unchanged if the pretrained representation is already appropriate for the new language.

Placement within Transformer blocks. The paper follows the standard MAD-X configuration: one adapter per Transformer layer, inserted after the feedforward sublayer. Appendix J (Figure 15) investigates the effect of adapter placement by placing a single adapter (with increased bottleneck size to match the parameter count of the full set) at different layers. The finding that "the last layers benefit most from the language adaptation" suggests that higher-level semantic and syntactic processing is more language-specific and benefits more from adaptation, while lower-level features are more language-universal.

Invertible adapters at the embedding layer. The invertible adapter is a separate module inserted between the token embedding lookup and the first Transformer block. Its purpose is to transform the embedding space so that the same subsequent Transformer layers can process multiple languages without interference. Unlike the bottleneck adapter, the invertible adapter must be bijective (invertible) so that the original embedding can be recovered if needed. The paper does not specify the exact invertible adapter architecture (it references Pfeiffer et al., 2020), but the standard MAD-X invertible adapter is a sequence of fixed, non-learned transformations (typically a random orthogonal rotation or a simple scaling) that are computationally cheap and do not add trainable parameters.

The authors ablate invertible adapters in Appendix K (Figure 16) and find that they "only improve performance for German, Bulgarian, and Turkish" while having minimal impact on other languages. This implies that embedding-level adaptation is less critical than Transformer-block-level adaptation for most languages—the byte-level BPE tokenizer and the model's existing embedding vectors already provide a reasonable starting point for the new script, and the heavy lifting of adaptation happens in the Transformer layers.

Why include both types of adapters. The MAD-X framework separates two concerns: the invertible adapter handles the input representation mismatch (the tokenizer produces byte sequences that, while formally valid for any script, may have statistical properties very different from pretraining languages), while the language adapters handle the linguistic processing (the actual syntax, semantics, and discourse patterns of the new language). This separation allows independent analysis of which component matters more—the paper's ablation in Appendix K suggests it is primarily the Transformer-level adapters.

Training signal and parameter count. The MAD-X adapters are trained with the same causal language modeling objective as continued pretraining (next-token prediction cross-entropy loss), but only the adapter parameters receive gradient updates—all original BLOOM weights remain frozen. This means the model never suffers catastrophic forgetting of its pretraining capabilities, since those parameters are never modified. The total number of trainable parameters is a small fraction of the full model: for BLOOM-7.1B, the adapters might constitute roughly 2–5% of the total parameters, depending on the reduction factor.

---

Language Adaptation Strategy 3: (IA)³ (Infused Adapter by Inhibiting and Amplifying Inner Activations)

(IA)³ (pronounced "IA-cubed") is a parameter-efficient finetuning method proposed by Liu et al. (2022) that modifies a pretrained model's behaviour not by adding new modules (as in MAD-X) but by rescaling existing activations with learned vectors. The core insight is that many aspects of a model's computation can be modulated without changing the computation itself—just by amplifying or inhibiting the signal at key points.

Three types of rescaling vectors. (IA)³ introduces learnable vectors that perform element-wise multiplication with three types of activations inside each Transformer block:

1. Key vectors ($l_k$): scales the key representations in the self-attention mechanism, modifying how much each position "offers" itself as something to attend to.
2. Value vectors ($l_v$): scales the value representations in self-attention, modifying how much each position contributes to the output when it is attended to.
3. Feedforward intermediate activations ($l_{ff}$): scales the activations after the first linear transformation in the feedforward sublayer, before the activation function.

For an attention head with key matrix $K$, value matrix $V$, and query $Q$, the standard attention computation is:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

With (IA)³, this becomes:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q(l_k \odot K)^T}{\sqrt{d_k}}\right)(l_v \odot V)$

where $\odot$ denotes element-wise multiplication, $l_k \in \mathbb{R}^{d_k}$ is a learned vector the same size as the key dimension, and $l_v \in \mathbb{R}^{d_v}$ is a learned vector the same size as the value dimension.

What these equations compute: the (IA)³ vectors perform a soft gating operation. Each element of $l_k$ is a scalar multiplier (initialised to 1, so at initialisation nothing changes) that independently amplifies or attenuates the corresponding dimension of the key vector. If a particular key dimension is important for processing the new language, its corresponding $l_k$ element will grow above 1; if a dimension is harmful or irrelevant, its element will shrink toward 0. The same logic applies to value vectors and feedforward activations.

Why this form: the element-wise rescaling can be seen as learning a binary mask that is discovered through gradient descent rather than combinatorial search. Liu et al. (2022) showed that this formulation outperforms bottleneck adapters, BitFit (which only updates bias terms), LoRA (which learns low-rank weight updates), and sparse masking methods on English NLU tasks. The paper's choice to include (IA)³ is motivated by these prior results: if (IA)³ is the strongest parameter-efficient method for task adaptation, it is a natural candidate for language adaptation.

Merging at inference. A practical advantage of (IA)³ over bottleneck adapters is that the learned rescaling vectors can be mathematically "merged" with the original pretrained weights at inference time, eliminating the additional forward-pass computation that bottleneck adapters require. For example, if a linear layer applies $y = Wx + b$ and (IA)³ scales the output by $l$, the merged weight becomes $W' = l \odot W$ (with appropriate broadcasting), and the inference computation becomes $y = W'x + b$—identical in cost to the unadapted model. This is impossible for bottleneck adapters, which must always run the down-up projection as a separate computation.

Combination with invertible adapters. The paper notes (Section 3.3) that (IA)³ does not adapt the embedding layer—it only rescales activations inside Transformer blocks. "For fairer comparison with MAD-X language adapters," the authors couple (IA)³ with invertible adapters at the embedding layer. Their preliminary experiments (Table 4) show that "invertible adapters with (IA)³" outperform vanilla (IA)³ (36.47% vs. 36.31% on German XNLI with BLOOM-560M), indicating that embedding-level adaptation provides a small but consistent benefit even when the main adaptation happens in the Transformer blocks.

Training signal and parameter count. Like MAD-X, (IA)³ is trained with the causal language modeling objective on target-language text, with all original BLOOM weights frozen and only the rescaling vectors and (optionally) invertible adapters receiving gradient updates. The parameter count is even smaller than MAD-X: for a model with hidden dimension $d$ and $L$ layers, (IA)³ adds $L \times (d_k + d_v + d_{ff})$ parameters, where $d_k$, $d_v$, and $d_{ff}$ are the key, value, and feedforward intermediate dimensions respectively. For a typical BLOOM configuration, this is often less than 0.1% of the total model parameters—orders of magnitude fewer than MAD-X and dramatically fewer than continued pretraining. Figure 3 confirms that (IA)³ has the smallest training time and memory footprint of all three methods, with the advantage growing at larger model sizes.

---

The Resource-Constrained Adaptation Setting

The paper's experimental design imposes a specific, consistent constraint across all adaptation strategies: a maximum of 100K monolingual sentences per language, simulating the reality that the languages most in need of adaptation are precisely those with limited available text.

Data sources and quantities. For seven of the eight target languages, the authors randomly sample 100K sentences from the deduplicated OSCAR subcorpora (Ortiz Suárez et al., 2019), a large web-crawled multilingual dataset. For Guarani, which has "only around 100 samples in OSCAR," they use the Jojajovai parallel corpus (Chiruzzo et al., 2022), which contains 30K Guarani sentences. The exact token counts after BLOOM's BPE tokenizer are reported in Table 6 (Appendix I):

Note the wide variation in token counts despite the identical sample count: German produces only 75M tokens from 100K sentences (~750 tokens/sentence on average), while Greek and Thai produce 160M tokens (~1600 tokens/sentence). This variation arises from differences in average sentence length and in how byte-level BPE tokenizes different scripts—scripts with larger character sets or more complex orthography may produce longer token sequences for the same semantic content.

Training hyperparameters (Section 3.4 and Appendix H). All adaptation runs use:

• Training steps: 25,000 (except post-hoc experiments with sequence length 2048, which use 12,500 steps to maintain the same total tokens seen)
• Batch size: 8 sequences per GPU
• Sequence length: 1,024 tokens (the authors note this was "by mistake instead of 2048 as described in Scao et al. (2022)," but they argue it does not affect conclusions since no evaluation task uses sequences longer than 1,024 tokens—Appendix N confirms that results with sequence length 2048 align with the main findings)
• Total tokens processed: approximately 204 million per language (25,000 steps × 8 sequences × 1,024 tokens/sequence = 204.8M tokens). Note that this is more than the available tokens for most languages (e.g., German has only 75M tokens in 100K sentences), meaning the model sees the same data multiple times—this is effectively multi-epoch training on a small corpus.
• Optimizer: AdamW (implied by the learning rate and decay settings)
• Learning rate: $1 \times 10^{-4}$ with linear decay (Table 5)
• Warm-up ratio: 0 for continued pretraining and MAD-X, 0.1 for (IA)³ (Table 5)
• Hardware: single RTX 3090 GPU for most runs; eight A100 GPUs with 40GB RAM for continued pretraining of BLOOM-7.1B (which is too large for a single consumer GPU)
• Single runs: each adaptation is performed once due to computational constraints, so there are no error bars or statistical significance tests on individual data points

Hyperparameter search scope. The authors performed a limited hyperparameter search on learning rates {1e-3, 1e-4, 1e-5}, decay schedules {linear, cosine}, and warm-up ratios {0, 0.05, 0.1} using only the Russian XNLI task with BLOOM-560M and BLOOM-1.1B (Appendix H). They report that "different sets of hyperparameters caused around 1~2% small difference in XNLI accuracy" and that their primary goal was "to study trends and performance-compute trade-offs... we did not perform extensive hyperparameter search." This is a reasonable prioritisation—extensive per-language, per-model, per-strategy tuning would have been computationally prohibitive—but it means the absolute accuracy numbers should be interpreted as approximate, not as the best possible performance achievable with each method.

Checkpoint selection protocol. Models are evaluated on the perplexity of 1,000 held-out validation samples in the target language every 5,000 training steps, and the checkpoint with the lowest validation perplexity is selected for downstream prompting evaluation. This is standard practice but, as the paper's Section 4.2 demonstrates, perplexity does not necessarily correlate with downstream task performance. A model with lower perplexity (better language modeling) can have worse zero-shot prompting accuracy than a model with higher perplexity. The authors do not explore alternative checkpoint selection criteria (e.g., downstream task performance on a validation split, or selection based on sentence retrieval accuracy), which is a limitation—the chosen checkpoints may not be optimal for the evaluation regime.

Sequence length caveat. The authors acknowledge using sequence length 1,024 "by mistake" (Section 6.5), following prior work (Yong and Nikoulina, 2022) rather than BLOOM's intended sequence length of 2,048. They argue this should not affect conclusions because "none of the evaluation tasks are done on sequences longer than 1024 tokens," and the post-hoc experiments in Appendix N confirm this for Thai and Greek. However, the shorter sequence length means the model sees fewer tokens per training example, which could affect how it learns long-range dependencies in the new language—a potentially relevant factor for tasks like XStoryCloze that require tracking narrative coherence across multiple sentences.

---

Zero-Shot Prompting Evaluation Framework

The evaluation regime is what makes this paper's findings distinct from prior language adaptation work. Rather than finetuning the adapted model on labeled data for each downstream task (the traditional approach), the authors evaluate zero-shot prompting: the model receives a natural-language prompt describing the task and must produce the correct answer without any task-specific gradient updates.

Task selection. The paper evaluates on five multilingual NLU benchmarks that together cover a range of reasoning types:

1. XNLI (Cross-lingual Natural Language Inference): Given a premise sentence and a hypothesis sentence, determine whether the hypothesis is entailed by, contradicted by, or neutral with respect to the premise. This is a three-way classification task testing logical reasoning across sentences.
2. KLUE-NLI (Korean NLI) and AmericasNLI (Guarani NLI): Same task format as XNLI but for Korean (from the KLUE benchmark) and Guarani (from the AmericasNLI benchmark), respectively. These are included because Korean and Guarani are target languages.
3. XCOPA (Cross-lingual Choice of Plausible Alternatives): Given a premise sentence and two possible continuations, choose which one is more plausible as a cause or effect of the premise. This tests causal commonsense reasoning.
4. XStoryCloze (Cross-lingual Story Cloze): Given a short story context with the ending removed, choose between two possible endings. This tests narrative understanding and coherence.
5. XWinograd (Cross-lingual Winograd Schema): Given a sentence with an ambiguous pronoun, choose which of two noun phrases the pronoun refers to. This tests anaphora resolution and pragmatic reasoning.
6. PAWS-X (Cross-lingual Paraphrase Adversaries from Word Scrambling): Given two sentences, determine whether they are paraphrases of each other. This tests semantic equivalence detection under adversarial word-order perturbations.

Prompt template design. The authors reuse the templates from Lin et al. (2021), which were developed for prompting the XGLM model. These templates convert each task into a natural language cloze-style format where the model's predicted next token (or sequence of tokens) can be mapped to a task label via a verbalizer—a fixed mapping from tokens to answer choices.

For XNLI, the English template is:

{PREMISE}, right? [Label], {HYPOTHESIS}

where [Label] is replaced with one of the three verbalizer tokens: "Yes" (entailment), "No" (contradiction), or "Also" (neutral). The model sees the premise, the word "right?", the [Label] placeholder, and the hypothesis. It is expected to fill in the [Label] with the appropriate verbalizer token, and the accuracy is measured by whether the generated token maps to the correct NLI label.

For XCOPA (cause variant), the English template is:

{SENTENCE 1} because [Label]

where [Label] is replaced with one of the two candidate continuations. The model chooses between the two continuations by assigning higher probability to the one that fits the template. This is a ranking rather than a generation task: the model's score for each candidate is its log-probability of generating that candidate in the [Label] position, and the highest-scoring candidate is selected.

Template translation. The English templates are translated into each target language using automatic translation APIs (Section 3.5). The translated templates are provided in Appendix F (Table 3) and include language-specific adjustments—for example, the XNLI template in German becomes:

{PREMISE}, richtig? [Label], {HYPOTHESIS}

with the verbalizers "Ja" (entailment), "Nein" (contradiction), and "Auch" (neutral). The use of automatic translation without human verification or prompt engineering is a deliberate choice: it tests whether language adaptation enables the model to work with roughly translated instructions, which is the realistic scenario for low-resource deployment where expert linguists may not be available to hand-craft prompts.

Why this matters. The authors' explicit claim that "we perform zero-shot prompting without any task-specific finetuning and simply reuse the templates... without performing any prompt engineering" is both a strength and a limitation of the evaluation. It is a strength because it tests the model's genuine zero-shot generalisation—there is no possibility of the prompt itself leaking task-specific information that the model memorised during finetuning. It is a limitation because small variations in prompt phrasing can cause large changes in zero-shot accuracy (the "prompt sensitivity" problem well-documented in the prompting literature), and the automatically translated templates may not be optimal for any particular language or task. The paper does not explore multiple prompt formulations or report variance across templates, so the reported accuracies should be understood as lower bounds on what the adapted models could achieve with prompt engineering.

Verbalizer design. For classification tasks (XNLI, PAWS-X), the verbalizer maps specific tokens to class labels. For multiple-choice tasks (XCOPA, XStoryCloze, XWinograd), the verbalizer is the "identity" mapping—the model scores each candidate answer choice directly, without reducing them to proxy tokens. This means the model must evaluate the full candidate text (which may be a multi-word sentence) rather than just predicting a single "Yes"/"No" token, making multiple-choice tasks a stricter test of language understanding.

Aggregation. For each task, language, and adapted model, the paper reports accuracy as the fraction of test examples for which the model's predicted answer (via verbalizer mapping) matches the ground-truth label. The primary aggregated metric is the average across tasks per language, visualised in Figure 1's panel grid.

---

Sentence Retrieval: Probing Language-Independent Representations

The sentence retrieval experiments in Section 4.3 are not part of the main adaptation-evaluation pipeline but serve as a mechanistic probe to explain why the three adaptation strategies behave differently at different model scales. Understanding this probe is essential because it provides the paper's key explanatory mechanism.

What sentence retrieval measures. Sentence retrieval accuracy quantifies how well the model's internal representations of sentences in the target language align with its representations of the same sentences in English. For a set of sentence pairs (the same semantic content expressed in the target language and in English, sourced from the FLORES dataset), the model encodes each sentence into a vector representation. Then, for each target-language sentence, the system finds the English sentence whose representation is closest (by cosine similarity) in the model's representation space. Accuracy is the fraction of target-language sentences for which the closest English sentence is the correct translation.

Why this measures language-independent representation. If the model encodes semantically equivalent sentences in different languages into similar vectors—regardless of their surface form—we say it has learned a "language-independent" or "language-agnostic" representation. In such a space, a Russian sentence like "Кошка сидит на коврике" (The cat sits on the mat) and its English equivalent "The cat is sitting on the mat" would be near each other, while both would be far from semantically unrelated sentences. This property is what enables cross-lingual transfer: if the model's representation space abstracts away from language-specific surface features and captures underlying meaning, then capabilities learned primarily from English data (like logical reasoning or task following) can transfer to new languages without retraining.

The authors frame this probe explicitly:

"We use sentence retrieval accuracy as a way to measure quality of language independent representation."

Experimental setup. The authors use a subset of 200 sentences from the FLORES dev set, which provides parallel translations across many languages. For the adapted models, the target-language sentence is encoded using the adapted model, while the English sentence is encoded using the original, unadapted BLOOM model. This deliberate asymmetry:

"excludes the problem of potential catastrophic forgetting"

If the adapted model had also encoded the English sentences, degradation in English representations (catastrophic forgetting) would confound the measurement—a low retrieval accuracy could mean either that the target-language representation is poor or that the English representation has drifted. By using the original model for English, the authors ensure that any change in retrieval accuracy is due to changes in the target-language representation only.

Layer-wise analysis. The authors compute sentence retrieval accuracy at each layer of the model by taking the hidden state at that layer (after the Transformer block, before the next layer) and pooling it into a single sentence vector (presumably via mean pooling, though the exact method is not specified). This produces a curve (Figure 5, and more extensively in Figure 10 for pretraining languages) showing how language-independent representation emerges across layers. The paper's key finding is that:

• For all model sizes on pretraining languages (Figure 10), the sentence retrieval accuracy follows a consistent pattern: low at early layers (the model is still processing language-specific surface features), peaking near-perfect (~100%) in middle layers, and dropping again at the final layers (which are optimised for next-token prediction in the specific language rather than cross-lingual alignment).

• For MAD-X-adapted models on new languages (Figure 5), sentence retrieval accuracy improves with model scale—larger models achieve higher alignment between target-language and English representations after adaptation.

• For continued-pretraining-adapted models on new languages (Figure 5), sentence retrieval accuracy degrades with model scale—the smallest model (560M) achieves the best alignment, while larger models perform substantially worse. This "goes against the scaling law" and is the paper's central mechanistic finding.

Interpretation: why continued pretraining degrades at scale. The authors propose (Section 4.3) that this phenomenon is connected to a hypothesis from Dufter and Schütze (2020): smaller models, having fewer parameters, are forced to share representational capacity across languages, which naturally encourages language-independent representations. Larger models, with more parameters, can afford to partition their parameter space—dedicating some parameters to English, others to French, and so on—reducing the pressure to develop shared representations.

When continued pretraining is applied to a large model, the model has many degrees of freedom to optimise its loss on the new language without leveraging its existing English-language parameters. It can dedicate a subset of its parameters to the new language in a way that diverges from the English-language representations, reducing cross-lingual alignment. MAD-X adapters, by contrast, are forced to work within the existing representational space—they can only modify representations through small residual deltas—so they preserve cross-lingual alignment by construction.

This mechanism explains the paper's central empirical finding: adapter-based adaptation outperforms continued pretraining for larger models because adapters preserve the language-independent representations that enable zero-shot cross-lingual prompting, while continued pretraining disrupts them.

---

BLOOMZ Adaptation: Extending the Findings to Instruction-Tuned Models

The final component of the paper's technical approach tests whether the language adaptation strategies developed for base BLOOM transfer to BLOOMZ, the instruction-tuned variant produced by finetuning BLOOM on the xP3 multitask mixture (Muennighoff et al., 2022). This is not a minor extension—BLOOMZ represents the practical deployment form of BLOOM for many users, since instruction-tuned models are far more useful for zero-shot task performance than base models.

BLOOMZ architecture and training. BLOOMZ is BLOOM finetuned on xP3, a mixture of prompts from many NLP tasks across the 46 pretraining languages. Critically for this paper, BLOOMZ was finetuned only on languages that BLOOM saw during pretraining—the eight target languages in this paper are absent from xP3 as well. This means BLOOMZ has no exposure to these languages, but it has been trained to follow task instructions, making it a stronger baseline for zero-shot prompting than base BLOOM.

Two approaches to adding language support to BLOOMZ. The paper explores two strategies:

Approach 1: Post-hoc language adaptation on unlabeled text (Section 4.6.1). This is the same MAD-X adapter training used for base BLOOM: train language adapters on monolingual OSCAR data in the target language, then evaluate zero-shot prompting. The experiment uses BLOOMZ-560M (the smallest BLOOMZ variant) and tests German XNLI performance. The results are starkly negative:

"BLOOMZ-560m has a median accuracy of around 38.5% for the German XNLI tasks, but after language adaptation, it performs the worst with an accuracy as poor as a random classifier at 33%."

The model completely loses its instruction-following capability. When the BLOOMZ model is instead equipped with BLOOM's language adapters (trained on base BLOOM, not BLOOMZ), prompting performance is largely preserved, indicating that the damage occurs during adaptation training, not at inference time.

Why this fails: the xP3 multitask finetuning that produced BLOOMZ taught the model specific behaviours—how to interpret task instructions, how to format answers, how to map verbalizer tokens to task labels—that are fragile. When the model is then adapted on free-form monolingual text with a next-token prediction objective, it reverts toward the behaviour of the base language model (predicting the most probable continuation of any text, not following instructions), a form of catastrophic forgetting of the instruction-tuning signal. This is a critical practical finding: you cannot simply apply the same language adaptation recipe to an instruction-tuned model and expect it to maintain its instruction-following abilities.

Approach 2: Adding the new language during instruction tuning (Section 4.6.2). Instead of adapting after instruction tuning, the authors test whether a new language can be learned during the instruction tuning process itself. They use Russian as the test case and create three variants:

• BLOOMZ-7.1B-RU: BLOOM-7.1B finetuned only on Russian task data (the Russian subset of xP3-like tasks), without any other languages or tasks.
• BLOOMZ-7.1B-xP3RU: BLOOM-7.1B finetuned on the full xP3 dataset with Russian data added to the mixture—the new language is learned while the model continues to practice all the other languages and tasks.
• BLOOMZ-7.1B: The standard BLOOMZ model (no Russian exposure during instruction tuning) as a baseline.
• BLOOM-7.1B: The pretrained base model (no instruction tuning at all).

The finding is nuanced:

"finetuning on only Russian (BLOOMZ-7.1B-RU) without the other languages and tasks in the xP3 mixture shows only tiny improvements over the pretrained baseline on XStoryCloze. This is likely due to the lack of diversity in the finetuning... the Russian-only split contains fewer tasks and prompts than the full xP3 dataset. On the other hand, when adding Russian to the instruction tuning mixture (BLOOMZ-7.1B-xP3RU), the performance of the best prompt improves on XNLI and XStoryCloze."

The takeaway is that including the new language in the multitask mixture during instruction tuning is effective, but the mixture must maintain diversity—finetuning on a single language's tasks causes the model to lose the broad instruction-following capability that makes BLOOMZ useful in the first place. The full xP3 mixture provides enough task and language diversity to teach the new language while preserving generalisation.

Why this matters for the paper's recommendations. The BLOOMZ experiments reveal that the "language adaptation on unlabeled text" approach—the main focus of the paper—is applicable to base models but not to instruction-tuned models. For instruction-tuned deployments, the recommended approach is fundamentally different: add the target language during the multitask finetuning stage, not as a post-hoc adaptation step. This has significant practical implications: it means that extending BLOOMZ to new languages requires access to the full xP3 training mixture and the computational resources to run multitask finetuning, not just the modest resources needed for adapter training on unlabeled text.

---

Summary of Design Choices and Their Justifications

• Byte-level BPE tokenizer without vocabulary expansion: avoids the cold-start problem of randomly-initialised embeddings for new-language tokens and ensures that any script can be processed out-of-the-box. The tradeoff is potentially suboptimal tokenization for unseen scripts, but the authors' results suggest this is not a significant barrier.

• Three adaptation strategies spanning the expressivity-cost spectrum: continued pretraining (maximum expressivity, maximum cost), MAD-X (moderate expressivity, moderate cost), and (IA)³ (low expressivity, minimum cost). This spectrum allows the paper to identify the point where the expressivity advantage of continued pretraining is outweighed by its representational disruption at scale.

• Scaling analysis across five model sizes (560M to 7.1B): essential for the paper's central finding (the strategy ranking flips at ~3B parameters). Without this multi-scale design, the finding would have been missed—testing only the smallest or only the largest model would have produced opposite (and misleading) recommendations.

• Deliberately diverse language testbed: enables the paper to claim that linguistic properties (script, word order, language family) do not determine adaptation success independently of data quantity. Without this diversity, generalisation claims would be unsupported.

• Zero-shot prompting evaluation rather than finetuning: tests whether the adapted model can use the new language for reasoning, not just model its surface statistics. This evaluation regime is what reveals the importance of language-independent representations—a finetuning evaluation might have shown continued pretraining to be competitive, since task-specific finetuning can compensate for disrupted cross-lingual alignment.

• Sentence retrieval as a mechanistic probe: provides a causal explanation (degradation of language-independent representations under continued pretraining at scale) for the empirical pattern (adapters outperform continued pretraining for large models). Without this probe, the paper would have documented the pattern without explaining it, leaving the recommendation as an empirical rule of thumb rather than a principled finding.

• Single-run experiments with minimal hyperparameter tuning: a pragmatic choice justified by computational constraints. It means absolute accuracy numbers are approximate, but relative comparisons across strategies, scales, and languages are valid because all methods and all scales are subject to the same unders-tuning.

4. Key Insights and Innovations

Innovation 1: Model Scale Reverses the Optimal Language Adaptation Strategy

The paper's most consequential conceptual contribution is demonstrating that the dominant recommendation from prior language adaptation research—use continued pretraining over parameter-efficient methods—flips polarity as model size crosses a threshold. Ebrahimi and Kann (2021) established the prevailing wisdom by systematically comparing adaptation strategies for encoder-only masked language models (mBERT, XLM-R) at scales up to roughly 1B parameters, concluding that continued pretraining was "the most promising direction." That recommendation was intuitive: continued pretraining updates every parameter, giving the model maximum freedom to accommodate a new language's distribution, while adapters restrict adaptation to a small subset of weights. More expressive should mean more effective.

BLOOM+1 shows this intuition holds only below a certain scale. For BLOOM-560M, continued pretraining indeed yields the strongest zero-shot prompting performance (Figure 1), replicating Ebrahimi and Kann's finding in the decoder-only setting. But the ranking shifts as models grow: at 3B parameters and above, both MAD-X bottleneck adapters and (IA)³ rescaling vectors consistently outperform continued pretraining despite updating orders of magnitude fewer parameters. By BLOOM-7.1B, the gap is systematic—adapter-based methods dominate across languages and tasks, while continued pretraining plateaus or degrades.

What makes this a genuine conceptual advance rather than just "larger models prefer different methods" is that the paper identifies a mechanism for the reversal: the quality of language-independent representations degrades under continued pretraining as model scale increases (Figure 5). Smaller models, forced by capacity constraints to share parameters across languages, naturally develop representations that align semantically equivalent sentences in different languages into similar vector spaces. Continued pretraining on a smaller model leverages these existing aligned representations—the model has no choice but to route the new language through its English-shaped parameters. Larger models, by contrast, have abundant capacity to partition their parameter space, dedicating distinct sub-networks to different languages. When continued pretraining is applied to a large model, it can optimise the new language's loss by diverging from the English-aligned representational space, reducing cross-lingual alignment. Adapters, by design, cannot diverge—they only apply residual deltas to frozen pretrained representations—so they preserve cross-lingual alignment by construction.

This is a fundamental reframing, not an incremental refinement. Prior work treated the "continued pretraining vs. adapters" question as a universal tradeoff to be resolved once. This paper recasts it as scale-dependent, with a specific threshold (~3B parameters) and a specific mechanistic explanation (representational divergence under capacity abundance) that predicts when and why the ranking flips. The finding has immediate practical consequences for practitioners: if you are adapting a model below ~1B parameters, follow Ebrahimi and Kann—continued pretraining is your best bet. If you are adapting a model above ~3B parameters, use adapters. And if you are adapting a model in between, you need to measure which side of the threshold your particular setup falls on. The paper does not just report an empirical pattern; it provides the diagnostic concept (language-independent representation quality) that explains the pattern and generalises beyond the specific models tested.

Innovation 2: Perplexity Does Not Predict Downstream Prompting Performance After Language Adaptation

A second conceptual contribution—less prominent in the paper's own framing but methodologically significant—is the empirical demonstration that standard language modeling metrics (perplexity) can be anti-correlated with downstream task performance after language adaptation, and that relying on perplexity for checkpoint selection or method comparison can produce misleading conclusions.

The standard practice in language adaptation work, inherited from pretraining, is to monitor validation perplexity on held-out target-language text, select the checkpoint with the lowest perplexity, and report perplexity as a signal of adaptation quality. The assumption is that better language modeling (lower perplexity) implies better downstream capability. The paper's Figure 4 shows this assumption breaking down in two ways.

First, continued pretraining consistently achieves lower perplexity than MAD-X adapters on Russian held-out text across all model sizes—the full-parameter update is genuinely better at reducing next-token prediction loss on the target language distribution. Yet for models at 3B parameters and above, the lower-perplexity continued-pretraining checkpoints underperform the higher-perplexity MAD-X checkpoints on XWinograd and XNLI prompting tasks. A model can become a better Russian language model while becoming worse at reasoning in Russian.

Second, the perplexity curves themselves display a scaling pattern that diverges from task performance. As model size increases from 1.1B to 1.7B parameters, perplexity continues to decrease (larger models model the Russian text more accurately), but XWinograd prompting accuracy drops for both continued pretraining and MAD-X at this transition. There is a non-monotonic relationship between model scale and downstream performance in a specific size band that perplexity alone cannot reveal.

This finding is significant because it challenges a deeply embedded assumption in NLP: that language modeling quality, as measured by perplexity, is a reliable proxy for task capability. The authors connect this to Liang et al. (2022)'s broader critique of the perplexity-downstream gap, but they demonstrate it in a specific, practically important context—language adaptation—where the field had been using perplexity-based checkpoint selection without questioning its validity. The practical implication is that practitioners adapting models to new languages cannot trust validation perplexity to identify the best checkpoint for prompting tasks. Alternative selection criteria—downstream task accuracy on a small labeled validation set, sentence retrieval accuracy as a proxy for cross-lingual alignment, or simply using a fixed number of training steps rather than perplexity-based early stopping—need to be developed.

This is a negative result with positive diagnostic value: it tells the community that a widely-used heuristic is unreliable in the language adaptation setting, and it pushes future work toward more principled model selection approaches.

Innovation 3: Language-Independent Representation Quality as a Unifying Diagnostic for Adaptation Success

The most theoretically generative contribution of the paper is its operationalisation of language-independent representation quality as both an explanatory mechanism and a potential selection criterion for language adaptation strategies. While the concept of language-independent representations predates this work (e.g., Artetxe and Schwenk, 2019; Dufter and Schütze, 2020), BLOOM+1 makes three moves that elevate it from an abstract desirable property to a practical diagnostic tool.

First, the paper uses sentence retrieval accuracy as a quantitative probe that can be measured before and after adaptation, at each layer of the model, to reveal where and how cross-lingual alignment changes. Figure 5 shows that MAD-X adapters improve sentence retrieval accuracy monotonically with model scale—larger models achieve better Russian-English alignment after adapter-based adaptation. Continued pretraining shows the opposite pattern: the smallest model (560M) achieves the best alignment, and performance degrades as models grow. This is not just a correlation with downstream task performance; it provides a mechanistic narrative for why adapters outperform continued pretraining at scale (they preserve and enhance cross-lingual alignment rather than disrupting it).

Second, the layer-wise analysis (Figure 10 for pretraining languages, Figure 14 for adapted models) reveals a consistent pattern: language-independent representations emerge most strongly in intermediate layers, while early and final layers remain language-specific. This pattern is remarkably stable across model sizes for languages seen during pretraining (Figure 10), but it breaks down for continued-pretraining-adapted models at larger scales. The diagnostic value here is that one can inspect which layers lose cross-lingual alignment under a given adaptation strategy, potentially enabling layer-specific interventions.

Third, the paper implicitly proposes sentence retrieval accuracy as a potential alternative to perplexity for checkpoint selection and method comparison. While the authors do not explicitly make this claim, their finding that sentence retrieval accuracy patterns align with downstream prompting performance—while perplexity patterns do not—suggests that cross-lingual alignment metrics may be more informative than language modeling metrics for evaluating adaptation quality in the prompting regime. This connection is left for future work to validate, but the paper provides the empirical foundation.

What distinguishes this from prior work on language-independent representations is the scale-dependent dynamic. Dufter and Schütze (2020) hypothesised that smaller models, forced to share parameters, would develop more language-independent representations—a hypothesis the authors note "to the best of our knowledge has not been confirmed in real multilingual models." The paper's sentence retrieval results on continued-pretraining-adapted models (best alignment at 560M, degradation at larger scales) provide the first evidence for this hypothesis in a real, large-scale multilingual setting. The finding that adapter-based adaptation reverses this pattern—larger models achieve better alignment with adapters—adds a second dynamic: parameter-efficient methods can harness the greater representational capacity of larger models for cross-lingual alignment, while full-model updating cannot.

Innovation 4: Language Adaptation Findings Do Not Transfer from Base Models to Instruction-Tuned Variants—And the Fix Is Different

Section 4.6 documents a finding that, while negative in the sense of "the straightforward approach fails," has significant conceptual implications for how the field thinks about the relationship between language adaptation and instruction tuning. The core result is that MAD-X language adaptation on unlabeled monolingual text—the approach that works well for base BLOOM—destroys the instruction-following capability of BLOOMZ, reducing its zero-shot accuracy to random-guess levels on German XNLI (Figure 8).

This is not merely an implementation detail or a hyperparameter sensitivity issue. It reveals a fundamental tension between two training objectives: the causal language modeling objective used for language adaptation (predict the next token in free-form text) and the instruction-following behaviour learned during multitask finetuning (interpret task instructions, map outputs to verbalizer tokens). When BLOOMZ is adapted on monolingual OSCAR data, the next-token prediction signal overwrites the instruction-following signal—the model reverts to behaving like a base language model that generates plausible continuations rather than following task prompts. The paper's diagnostic test—using BLOOM's pre-trained language adapters (trained on base BLOOM) with BLOOMZ's Transformer weights—preserves performance, confirming that the damage occurs during adaptation training, not at inference time.

The constructive finding is equally important: adding the target language during the multitask instruction tuning phase (the BLOOMZ-7.1B-xP3RU variant in Figure 9) is effective, but only when the new language data is mixed with the full diversity of tasks and languages from the xP3 mixture. Finetuning on Russian tasks alone (BLOOMZ-7.1B-RU) produces minimal improvement, likely because the single-language, reduced-task-count finetuning lacks the diversity needed to maintain broad instruction-following capability. This connects to Chung et al. (2022)'s finding on the importance of task diversity in instruction tuning, extending it to the cross-lingual setting.

The conceptual contribution here is a categorical distinction between two types of language adaptation: post-hoc adaptation on unlabeled text (applicable to base models but destructive for instruction-tuned models) and inclusion during multitask finetuning (effective for instruction-tuned models but requiring access to the full instruction-tuning mixture and its computational cost). Prior work had not drawn this distinction—the language adaptation literature treated all pretrained models as equivalent targets for adaptation, and the instruction-tuning literature had not addressed how to add new languages post-hoc. The paper establishes that the adaptation strategy must match the model's post-pretraining training history, not just its architecture or pretraining data. This is a conceptual boundary condition that future work on language adaptation for instruction-tuned models will need to respect.

Innovation 5: Language-Specific Properties Do Not Independently Determine Adaptation Difficulty When Data Quantity Is Controlled

The paper's least flashy but most practically enabling finding is that linguistic properties—writing system, language family, word order, script familiarity—do not significantly predict adaptation success when the quantity of adaptation data is held constant. Figure 2 decomposes average XNLI performance by three linguistic dimensions: Indo-European vs. non-Indo-European language family, SVO vs. SOV word order, and seen vs. unseen script during pretraining. Across all three splits, the adapted models perform similarly regardless of category.

This finding contradicts plausible expectations and some prior evidence. Script mismatch has been identified as a barrier in earlier language adaptation work (e.g., Muller et al., 2021 on the benefits of transliteration for non-Latin scripts; Pfeiffer et al., 2021b on adapting to new scripts). Yong and Nikoulina (2022), working with an earlier version of BLOOM that used learned positional embeddings, found that script and language family differences did affect adaptation outcomes. The current paper explicitly notes this contradiction and attributes the difference to architectural changes—AliBi positional embeddings removing the need for positional embedding adaptation, and byte-level BPE tokenization removing the need for vocabulary expansion—that eliminate two sources of script-dependent interference.

The significance of this finding is that it simplifies the deployment calculus for language adaptation. If script, word order, and language family were strong independent predictors of adaptation difficulty, then practitioners would need to consult linguistic typology databases and potentially develop script-specific or family-specific adaptation strategies. The paper's finding that "BLOOM adapts well to new languages regardless of their language family, word order, and whether they share the same script system" means that data quantity is the primary bottleneck—if you have enough monolingual text (the paper estimates ~100M tokens as the effective threshold from Section 4.4 and Table 6), you can adapt BLOOM to languages across a wide typological range without language-specific engineering.

This is a boundary-simplifying contribution rather than a performance-improving one. It does not make adaptation better; it makes adaptation more predictable. It tells the community that the search space of adaptation strategies does not need to be conditional on linguistic features, and that efforts should focus on data acquisition and efficient training rather than on language-specific architectural modifications. The caveat is that the testbed, while diverse, covers only eight languages, all with at least 30K sentences and most with 100K. Whether the finding extends to languages with substantially smaller data budgets (<10K sentences) or to languages with more exotic linguistic features (ergativity, polysynthesis, complex tone systems) remains open.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation benchmark is MATH (Hendrycks et al., 2021), specifically the subset from Lightman et al. (2022) comprising 12,000 training questions and 500 test questions. MATH consists of high-school competition-level mathematics problems spanning algebra, geometry, counting and probability, number theory, and precalculus. The authors select MATH because "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences—mathematical reasoning fits this profile because it requires multi-step logical deduction rather than novel factual recall." All experiments use the 500-question test set for evaluation; the training set is used only for training the process reward model (PRM) and revision model.

• Base model(s). All experiments use PaLM 2-S* (Codey) (Anil et al., 2023), a decoder-only Transformer from the PaLM 2 family. The authors argue this model is "representative of the capabilities of many contemporary LLMs" and occupies a useful performance regime: non-trivial baseline accuracy on MATH (roughly 10–19% pass@1 depending on prompting and sampling configuration) but far from saturation, leaving substantial headroom for test-time compute to make a difference. For the FLOPs-matched comparison in Section 7, a second model with approximately 14× more parameters than PaLM 2-S* is used as the pretraining-scaled baseline. The exact parameter counts of both models are not disclosed (consistent with the PaLM 2 technical report's limited disclosure). The authors characterize this larger model as representative of scaling parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022), and explicitly acknowledge this design choice as a limitation.

• Metrics. The primary metric is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground-truth answer, as determined by the grading function from Lightman et al. (2022) described in Appendix G. For difficulty-conditioned analyses, accuracy is reported separately within each of five difficulty quintiles (Section 3.2). The paper also reports perplexity on held-out target-language text (1,000 validation samples) as a training monitoring metric, and sentence retrieval accuracy as a mechanistic probe for language-independent representation quality (Section 4.3, Figure 5), though these are diagnostic rather than primary evaluation metrics.

• Baselines. The paper compares against several established approaches:

  • Majority voting: select the most common final answer among N independently sampled solutions, with no learned verifier. This is the simplest test-time compute baseline and tests whether verifier-guided selection adds value beyond naive consensus.
  • ORM best-of-N weighted: score N independent solutions with an Outcome Reward Model (a single correctness score per complete solution) and apply best-of-N weighted selection (Li et al., 2023), where solutions arriving at the same final answer have their scores summed, and the answer with the highest total score is selected.
  • PRM best-of-N weighted: same as ORM best-of-N weighted but using per-step Process Reward Model scores aggregated via the "last" step method (Appendix E), which the authors find superior to "min" and "prod" aggregation.
  • Parallel sampling (for revision experiments): generate N independent solutions from the revision model (no sequential chain) and select the best via majority voting or verifier-based selection. This serves as the baseline for evaluating whether sequential revisions provide gains over independent sampling with the same total generation budget.
  • Greedy decoding of the ~14× larger model: for the FLOPs-matched comparison, the pretraining-scaled baseline uses greedy decoding (temperature 0, single generation) with no additional test-time compute. This is a deliberate choice that makes test-time compute's case maximally favorable, which the authors acknowledge.
  • XGLM, mGPT, BLOOMZ, mT0: these baselines are specific to the BLOOM+1 language adaptation evaluation and are described in Section 3.6 of the paper (not covered in this experimental analysis, which focuses on the MATH benchmark).

• Generation budget / compute accounting. The universal unit of test-time compute is one generation, defined as one complete sampled answer from the base LLM. For best-of-N and majority voting, the budget equals N (the number of samples). For beam search with width M, the budget is the total number of beams expanded, which equals N (maintaining fairness with best-of-N). For lookahead search with k lookahead steps, each beam expansion costs k+1 generations (the looked-ahead steps plus the actual step), so the effective budget at the same N is N × (k+1). Budgets are swept across powers of 2, typically from 2⁰ to 2⁹ (1 to 512 generations), with a maximum of 256 generations for the search comparison (Section 5.3) and up to 512 for some revision experiments. The FLOPs-matched comparison (Section 7) uses a separate accounting framework based on total pretraining and inference FLOPs, described in the technical approach section.

• Cross-validation / statistical protocol. To avoid overfitting the compute-optimal strategy selection to the test set, the authors use two-fold cross-validation within each difficulty quintile on the 500-question test set (Section 3.2). The best-performing strategy (search algorithm, revision ratio, etc.) at each budget level is selected on one fold and evaluated on the other, then the assignment is swapped, and results are averaged. This prevents the policy from being evaluated on the same data used to select it. For difficulty estimation, the predicted-difficulty variant uses the PRM's average final-answer score across 2048 samples per question, binned into quintiles without access to ground-truth labels. All adaptation training runs for the BLOOM+1 experiments are performed once due to computational constraints—the authors state they "conducted single runs for each language adaptation" (Appendix H) and therefore do not report confidence intervals or error bars.

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

The paper's first major axis of investigation compares three search algorithms—best-of-N, beam search (two width settings: M = √N and M = 4), and lookahead search (k = 1 and k = 3 steps, applied to beam-width variants)—all operating against the same PRM verifier on the MATH benchmark with PaLM 2-S*.

Aggregate comparison across all 500 test questions (Figure 3, left). At low generation budgets (2–8 generations), beam search with M = 4 significantly outperforms best-of-N weighted. At 4 generations, beam search (M = 4) achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted—a gap of approximately 11 percentage points. This early advantage reflects beam search's ability to prune unpromising partial solutions step-by-step, concentrating the limited budget on paths the PRM judges to be on track. At high budgets, this advantage erodes and eventually reverses. At 512 generations, best-of-N weighted reaches approximately 38% accuracy, while beam search (M = 4) plateaus around 34%, and beam search (M = √N) falls below best-of-N. The degradation is attributed to PRM over-optimization: at high budgets, search finds solutions that score highly under the PRM but are actually incorrect—including low-information repetitive steps and overly short 1–2 step solutions (qualitative examples in Appendix M, Figure 29). Lookahead search variants consistently underperform simpler methods at the same generation budget. The 3-step lookahead with M = 4, for instance, converges to similar performance as other methods only at very high budgets, but never surpasses them. This is because lookahead's extra per-step cost (k+1 generations per expansion) reduces the effective number of beams explored at any fixed budget. Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations, confirming that the learned verifier adds genuine signal beyond simple consensus among model samples.

Difficulty-bin analysis (Figure 3, right). The authors decompose beam search (M = 4) vs. best-of-N weighted performance across five difficulty quintiles at four budget levels (4, 16, 64, 256 generations). This breakdown is the paper's most informative result because it reveals that the aggregate comparison in Figure 3 (left) masks qualitatively different behavior at different difficulty levels:

• Bin 1 (easiest questions, highest base model pass@1): Beam search degrades performance with increasing budget—accuracy drops from roughly 78% to 77% as budget goes from 4 to 256—while best-of-N weighted improves from roughly 68% to 88%. This is the clearest evidence of PRM over-optimization: on problems where the base model already produces many correct solutions, aggressive search amplifies residual errors in the verifier's judgments, selecting solutions that look good to the PRM but are wrong.
• Bin 2: Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear and growing advantage at high budgets.
• Bin 3: Beam search consistently outperforms best-of-N weighted across all budget levels, reaching roughly 34% vs. 23% at 256 generations. The PRM's guidance genuinely helps navigate toward correct solutions that random sampling would miss.
• Bin 4: Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations. Even here, the absolute performance is low—these are genuinely difficult questions.
• Bin 5 (hardest questions): Both methods hover near 1–3% accuracy regardless of budget. No amount of test-time compute makes meaningful progress on problems the base model fundamentally cannot solve.

Compute-optimal search (Figure 4). By selecting the best search strategy (best-of-N, beam search M = 4, beam search M = √N, or lookahead variants) per difficulty quintile at each budget level, the compute-optimal policy achieves substantial efficiency gains. At 16 generations, compute-optimal search (oracle difficulty bins) achieves approximately 27% accuracy—matching PRM best-of-N weighted at roughly 64 generations, a ~4× reduction in required compute. At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%). The predicted-difficulty version (using PRM average scores rather than ground-truth labels to estimate difficulty) tracks the oracle version closely, with the two curves "largely overlapping" at lower budgets. At 256 generations, predicted-difficulty compute-optimal reaches approximately 37%, slightly below oracle but still substantially above PRM best-of-N weighted and well above ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%). The close overlap between oracle and predicted curves is a critical result: it confirms that the PRM's own score distribution serves as a sufficient proxy for difficulty estimation, making the compute-optimal framework deployable without access to ground-truth answers.

PRM vs. ORM (Appendix F, Figure 14). The PRM consistently outperforms a separately trained ORM at all sample counts, with the gap widening at higher budgets. At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus ORM best-of-N weighted at roughly 35%, and majority voting at roughly 30%. This confirms that step-level supervision provides a stronger verifier signal than outcome-level supervision, even when the PRM's per-step scores are aggregated using only the final step ("last" aggregation, Appendix E), which effectively reduces the PRM to ORM-like behavior at aggregation time. The authors interpret this as evidence that "the step-level PRM training acts as a form of beneficial representation learning even when the intermediate predictions aren't directly used at aggregation time."

Revision Model Results (Section 6)

The second major axis investigates iterative revision—modifying the proposal distribution rather than screening outputs—using a fine-tuned revision model that conditions on its own previous (incorrect) answers.

Revision chain pass@1 improvement (Figure 6, left). The revision model's per-step accuracy (pass@1 at each revision step) improves from approximately 18.2% at step 1 to roughly 24–25% by steps 15–20, and remains in the 23–25% range out to 64 steps. The model generalizes beyond its 4-step training horizon (it was trained on sequences of 0–4 incorrect answers followed by a correct answer), confirming that it has learned a transferable revision skill rather than memorizing a fixed revision depth. The improvement is gradual—roughly 6–7 percentage points absolute gain across the chain—but consistent, and it does not collapse at lengths far beyond the training distribution.

Sequential vs. parallel comparison (Figure 6, right). At a fixed budget of 64 generations:

• Sequential + best-of-N weighted: approximately 41.5%
• Parallel + best-of-N weighted: approximately 39%
• Sequential + majority: approximately 38%
• Parallel + majority: approximately 35%

Sequential revisions outperform parallel sampling under both selection mechanisms, with the verifier-based gap (~2.5 percentage points) narrower than the majority-based gap (~3 points). The sequential advantage is consistent but modest in absolute terms—roughly 2.5 percentage points with verifier-based selection. This suggests that while iterative revision improves the proposal distribution, the gain over simply generating independent samples in parallel is real but not dramatic when averaged across all difficulty levels.

Sequential-to-parallel ratio sweep (Figure 7, left). For a fixed total generation budget, the authors vary the ratio of sequential depth to parallel breadth (e.g., 16 sequential × 4 parallel = 64 total, vs. 4 sequential × 16 parallel = 64 total). At 256 generations, the optimal ratio is around 2¹ to 2³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43–44% accuracy. Fully parallel (leftmost point) yields approximately 40%. Fully sequential (rightmost point) yields approximately 42%. At lower budgets (8–32 generations), the curves become monotonically increasing with the sequential-to-parallel ratio—fully sequential is optimal at small budgets, suggesting that when total compute is severely constrained, it is better to refine a few solutions deeply than to explore many solutions shallowly.

Difficulty-dependent ratio (Figure 7, right). At a fixed budget of 128 generations, the sequential-to-parallel ratio's effect varies sharply by difficulty:

• Bin 1: Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to how compute is allocated—the model gets them right regardless.
• Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel.
• Bin 3: A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2¹ to 2³), reaching approximately 42% vs. 35% at the extremes. This is the difficulty level where the exploration-exploitation tradeoff matters most.
• Bin 4: Similar pattern with peak at moderate ratio, achieving roughly 18% vs. 14% at fully parallel.
• Bin 5: All ratios produce roughly 2–3% accuracy. No allocation strategy helps.

This difficulty-dependence mirrors the finding from search (Figure 3, right): easy problems favor exploitation (sequential refinement), hard problems benefit from exploration (parallel diversity), and the hardest problems benefit from neither. The optimal ratio shifts from sequential-heavy for easy problems to balanced for medium-hard problems—the compute-optimal policy captures this shift by selecting different ratios per difficulty quintile.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty quintile yields compute-optimal scaling that substantially outperforms the parallel-only baseline. At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations—a ~4× reduction in required compute. At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and approximately 37% for parallel-only. Compute-optimal with predicted difficulty bins performs slightly below oracle at high budgets (approximately 41% at 256 generations) but still substantially above the parallel baseline. Notably, the parallel baseline plateaus around 36–37% at high budgets, while compute-optimal scaling continues to improve—suggesting that the gains from adaptive allocation compound at higher budgets rather than diminishing.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

The paper's third major axis asks: given a fixed total FLOPs budget (pretraining + inference), is it better to train a larger model or to keep the smaller model and invest the saved pretraining FLOPs in additional test-time computation? The comparison is between PaLM 2-S* with compute-optimal test-time scaling and a model with approximately 14× more parameters using greedy decoding (no test-time compute augmentation). Three values of the inference-to-pretraining token ratio R = D_inference / D_pretrain are tested: 0.16 (R ≪ 1, representing self-improvement or low-volume evaluation scenarios), 0.79 (R ≈ 1), and 22 (R ≫ 1, representing high-throughput production deployments).

Revision-based test-time compute (Figure 9, left; Figure 1, top-right bar chart). The relative advantage of test-time compute over pretraining scaling depends sharply on both difficulty and R:

• Easy questions (bin 1): Test-time compute with revisions outperforms the 14× larger model at R ≪ 1 (relative improvement not explicitly quantified in the text for single bins) and at R ≈ 1, but underperforms at R ≫ 1 by approximately -11.9%. The advantage shrinks as inference cost dominates.
• Medium questions (bins 2–3): Test-time compute shows a substantial advantage at R ≪ 1 (+27.8% relative), a smaller but still positive advantage at R ≈ 1 (+16.7%), and a narrow advantage at R ≫ 1 (+5.4%).
• Hard questions (bins 4–5): Test-time compute shows +21.6% advantage at R ≪ 1, which turns negative at higher R, reaching -37.2% at R ≫ 1. On genuinely hard problems where the base model's capability is limited, the larger pretrained model's additional capacity provides benefits that test-time compute cannot recover.

PRM search-based test-time compute (Figure 9, right; Figure 1, bottom-right bar chart). The pattern is consistently less favorable to test-time compute than revisions:

• Easy questions: Test-time compute shows +19.1% at R ≪ 1, narrowing to roughly +2% at R ≈ 1 and R ≫ 1.
• Medium questions: Test-time compute breaks even (0.0%) at R ≪ 1 and shows substantial disadvantages at R ≈ 1 (-35.3%) and R ≫ 1 (-30.8%).
• Hard questions: Test-time compute underperforms across all R values: -3.6% at R ≪ 1, -35.3% at R ≈ 1, -52.9% at R ≫ 1.

PRM search is systematically weaker than revisions for the FLOPs-matched comparison. The authors do not fully explain this gap, but it is consistent with the earlier finding that search over-optimizes the PRM at high budgets (Figure 3), while revisions modify the proposal distribution in a way that may produce genuinely better candidates rather than just better-selected ones.

Figure 9 detail. The line plots show accuracy per difficulty quintile as test-time compute budget scales for the smaller model. The 14× larger model's greedy-decoding performance is plotted as horizontal stars at three x-axis positions corresponding to the three R values—the position varies because the same total FLOPs budget translates to different test-time compute budgets depending on R. Where the scaling line passes above a star, test-time compute wins at that R value for that difficulty bin. On bin 1 with revisions, the compute-optimal scaling line is above all three stars, confirming test-time compute's advantage across all R regimes for easy questions. On bin 5, both with revisions and with search, the scaling line is flat near 0–5% accuracy and below all three stars, confirming that no amount of test-time compute substitutes for pretraining on the hardest problems.

Key boundary conditions. The FLOPs-matched comparison establishes that test-time compute and pretraining compute are not universally exchangeable. The substitution works when: (1) problems are within the base model's capability range (difficulty bins 1–3), (2) the inference-to-pretraining token ratio is low to moderate (R ≤ 1), and (3) revisions rather than PRM search is the mechanism. It fails when problems are genuinely hard for the base model (bins 4–5) or when inference costs dominate the total FLOPs budget (R ≫ 1). The paper is explicit about these boundaries, framing them as conditions under which the claimed benefits hold rather than as universal superiority of test-time compute over pretraining.

Ablation Studies and Robustness Checks

• PRM step-aggregation strategy (Appendix E, Figure 13). The paper compares three methods for aggregating per-step PRM scores into a single solution-level score: "min" (minimum score across steps), "prod" (product of step-level scores, treating them as independent correctness probabilities), and "last" (only the PRM's prediction at the final step). "Last" achieves the highest accuracy—roughly 37% at 256 samples—while "min" reaches roughly 35%, "prod" reaches roughly 27%, and a separately trained ORM reaches roughly 34%. This ordering contradicts prior work (Lightman et al., 2023; Wang et al., 2023) that found "min" to be superior. The authors hypothesize the discrepancy arises because their PRM is trained with soft Monte Carlo rollout labels rather than binary correctness labels, changing the statistical properties of per-step scores. An important implication: using "last" effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM, supporting the interpretation that step-level PRM training improves the model's internal representations even when intermediate predictions are discarded.

• PRM vs. ORM scaling behavior (Appendix F, Figure 14). The PRM outperforms the ORM at all sample counts from 1 to 2048, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35% vs. majority voting's 30%. This confirms that the PRM is not merely an ORM with a different training procedure—it provides genuinely better discrimination, and the advantage grows as the number of candidates increases, suggesting the PRM is more robust to the distribution of solutions encountered at high sample counts.

• Revision model verifier domain shift (Appendix J, Figure 15a). The PRM trained on base model outputs underperforms when scoring revision model outputs, with sequential + base-LM PRM achieving roughly 40% at 64 generations vs. sequential + revision-specific ORM at roughly 42%. This confirms that distribution shift is a practical concern: the revision model's output distribution differs sufficiently from the base model's that a verifier trained on base-model data transfers imperfectly. The paper trains a separate ORM on revision model outputs to address this, but does not train a separate PRM on revision model outputs—a potential missed opportunity for further gains.

• Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's input context provides a small improvement (approximately 1–2 percentage points at 64 generations) over the no-history ablation. However, both variants outperform the parallel baseline, confirming that the sequential revision benefit is not solely attributable to the verifier having access to revision history—even without seeing the chain, the revision model's sequential outputs are better than its parallel outputs.

• Oracle vs. predicted difficulty bins (Figures 4, 8; Appendix C, Figures 11–12). Both oracle (ground-truth pass@1-based) and predicted (PRM-average-score-based) difficulty binning produce qualitatively and quantitatively similar compute-optimal scaling curves. For search (Figure 4), the curves "largely overlap" across all budgets. For revisions (Figure 8), predicted bins show slightly lower performance at high budgets (roughly 41% vs. 44% at 256 generations), but still substantially outperform the parallel baseline (approximately 37%) and best-of-N weighted (approximately 41%). This is the critical robustness check enabling practical deployment: the compute-optimal framework works without ground-truth labels, using only the PRM's own confidence estimates as a difficulty proxy.

• Majority voting with revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection replicate under majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This confirms that the revision model's sequential improvement is not an artifact of the verifier—it is detectable even with the simple, no-training-required majority-vote baseline.

• ReST^EM revision model training (Appendix K, Figure 16). An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024), an iterative self-improvement procedure, produces a negative result: additional sequential revisions with the ReST^EM-trained model substantially hurt performance. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal sequential-to-parallel ratio for the ReST^EM model. The authors hypothesize that on-policy data collection in ReST^EM amplifies spurious correlations in revision data, causing the model to fail to learn the revision task properly. This negative result highlights the sensitivity of revision training to the data generation procedure and serves as a warning that naive self-improvement can backfire.

• C-SFT prompting failure (Appendix D, Table 2). Despite Composable Sparse Fine-Tuning (Ansell et al., 2022) outperforming MAD-X on POS and NER tasks in prior work, the authors find that C-SFT-adapted BLOOM models "are not capable of prompting," achieving near-random accuracy on XNLI (33.01% for German with BLOOM-560M vs. 34.79% for the unadapted baseline and 36.83% for MAD-X). Longer training (50K steps instead of 25K) improved performance modestly to 35.97%, but computational constraints prevented full exploration. This is a negative result specific to the BLOOM+1 experiments but conceptually significant: it demonstrates that adaptation methods validated on token-classification finetuning tasks do not necessarily transfer to zero-shot prompting evaluation.

• Invertible adapter placement and necessity (Appendix K, Figure 16; Appendix J, Figure 15). Ablating invertible adapters from the MAD-X configuration shows they "only improve performance for German, Bulgarian, and Turkish" while having minimal impact on other languages. This implies that embedding-level adaptation is less critical than Transformer-block-level adaptation for most languages—the byte-level BPE tokenizer and existing embedding vectors already provide a reasonable starting point. For adapter placement within layers (Appendix J, Figure 15), a single adapter at the final layers provides most of the benefit, with middle layers benefiting least from adaptation. This suggests that higher-level semantic processing is more language-specific than lower-level feature extraction.

• Model pretraining necessity (Appendix K, Figure 17). Language adaptation (both continued pretraining and MAD-X) applied to a randomly initialized BLOOM (no pretraining) yields random-classifier performance on Russian XNLI, confirming that knowledge transfer from pretraining is essential—the adaptation strategies do not learn the language from scratch; they adapt existing capabilities.

• Sequence length sensitivity (Appendix N, Figure 19). Post-hoc experiments with the correct sequence length of 2048 (matching BLOOM's pretraining configuration, rather than the 1024 used in the main experiments) on Thai and Greek show that "adapters-based strategies still outperform continued-pretraining when we use the sequence length of 2048, which is consistent with our results discussed in Section 4.1." This confirms that the accidental use of sequence length 1024 does not qualitatively change the paper's findings.

Critical Assessment

This section evaluates whether the reported experiments genuinely support the paper's central claims, identifies specific weaknesses, and surfaces missing experiments that would strengthen (or challenge) the conclusions.

Claim 1: "Adapter-based finetuning is more effective than continued pretraining for large models."

What was demonstrated: The paper shows that for BLOOM models at 3B and 7.1B parameters, MAD-X adapters and (IA)³ achieve higher zero-shot prompting accuracy than continued pretraining on eight target languages, averaged across five NLU benchmarks (Figure 1). The reversal from the 560M scale (where continued pretraining is best) to the 7.1B scale (where adapters dominate) is consistent across most language-task pairs, with the cleanest evidence in the XNLI and XWinograd panels.

What was not demonstrated: The claim's generality is limited in several ways. First, the finding is demonstrated on a single model architecture (BLOOM with AliBi, byte-level BPE tokenizer, decoder-only). Whether the scale-dependent reversal occurs for other large model families (GPT-style, LLaMA-style, encoder-decoder architectures) is untested. BLOOM's specific architectural choices—particularly AliBi positional embeddings and byte-level BPE without vocabulary expansion—remove two sources of adaptation difficulty (positional embedding mismatch, unknown tokens) that exist in other large models. On models with learned positional embeddings and vocabulary gaps for new scripts, continued pretraining might retain its advantage to larger scales because it can update these problematic components.

Second, the adaptation data budget is fixed at 100K sentences per language (204M tokens seen over 25K steps with batch size 8 and sequence length 1024). The paper does not test whether the adapter advantage holds at larger data budgets—if a community had 1M or 10M sentences for their language, would continued pretraining's greater expressivity eventually overcome its representational disruption, or would adapters maintain their advantage? The single-data-budget design means the claim is conditional on the resource-constrained setting (~100K sentences) and may not generalise to settings with more data.

Third, the "large model" threshold of ~3B parameters is identified from a discrete set of model sizes (560M, 1.1B, 1.7B, 3B, 7.1B). The exact crossover point is unknown—it could be at 2B, 2.5B, or 3.5B parameters. The 1.1B to 1.7B transition shows some task-specific performance drops (German XNLI, PAWS-X) that the paper does not fully explain, suggesting the scaling behavior in this intermediate regime is not smooth.

Fourth, single-run experiments with minimal hyperparameter tuning mean random variation could affect the exact ranking at specific model sizes. The authors acknowledge they "conducted single runs for each language adaptation due to computational constraint" (Appendix H), so there are no confidence intervals to assess whether the observed crossover at 3B is statistically reliable or could shift with different random seeds.

Claim 2: "Prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data."

What was demonstrated: Figure 2 shows that average XNLI performance after adaptation is similar across three linguistic splits: Indo-European vs. non-Indo-European family, SVO vs. SOV word order, and seen vs. unseen script. The data-quantity effect is demonstrated in Figure 6 (Section 4.4), which shows a positive correlation between adaptation data size (1K, 10K, 100K samples) and zero-shot prompting performance for Russian and Turkish, with the 100K setting yielding the strongest results. Table 6 confirms that the effective threshold is approximately 100 million tokens.

What was not demonstrated: The claim that linguistic properties do not matter is based on eight languages, and the three binary splits (IE/non-IE, SVO/SOV, seen/unseen script) are coarse. Within each split, the "similar" performance still shows variation. The sample size per linguistic category is small (e.g., only two SOV languages: Turkish and Korean), making it impossible to distinguish whether genuinely null effects exist or whether the experiment is underpowered. A stronger test would include more languages per linguistic category—multiple SOV languages from different families, multiple unseen-script languages with different script types—to establish whether within-category variation is comparable to between-category variation.

The data-quantity claim (Figure 6) tests only two languages (Russian and Turkish) and only at the BLOOM-3B scale. Whether the 100M-token threshold generalises to other languages, other model sizes, or other tasks is not demonstrated. Guarani, with only ~1M tokens of adaptation data (Table 6), shows minimal adaptation benefit except for models at 3B parameters and above, but this is a single data point—we do not know whether 10M Guarani tokens would be sufficient or whether Guarani's difficulty is due to data quantity or to its genuinely low-resource status (smaller speaker population, less digital presence, potentially less transferable linguistic features).

Additionally, the paper does not control for data quality—the OSCAR subcorpora vary in their composition (news, web text, curated content) and cleanliness across languages. German OSCAR data may be qualitatively different from Thai OSCAR data due to different web ecosystems, content moderation practices, and crawling biases. If data quality co-varies with linguistic properties, the apparent null effect of linguistic properties could be a quality confound.

Claim 3: "Including a new language in the multitask fine-tuning mixture is the most effective method to teach BLOOMZ a new language."

What was demonstrated: Figure 9 shows that BLOOMZ-7.1B-xP3RU (Russian added to the full xP3 instruction-tuning mixture) outperforms BLOOMZ-7.1B-RU (Russian tasks only), BLOOMZ-7.1B (no Russian), and BLOOM-7.1B (pretrained base) on Russian XNLI and XStoryCloze. The finding is cleanly demonstrated for Russian at the 7.1B scale.

What was not demonstrated: This experiment involves exactly one language (Russian) at exactly one model size (7.1B). Whether the finding generalises to other languages—particularly those with different scripts or smaller available task datasets—is unknown. Russian is an Indo-European language with substantial available NLP task data; Guarani or Thai might not have enough labeled task data to construct a meaningful addition to the xP3 mixture.

The paper does not compare "adding language during instruction tuning" against "MAD-X adaptation of BLOOMZ followed by some mitigation of catastrophic forgetting." The failure of post-hoc MAD-X adaptation for BLOOMZ (Figure 8) is documented, but no remedy is attempted. Would a smaller learning rate, a shorter adaptation duration, or a mixture of monolingual text and xP3 data during adaptation preserve BLOOMZ's instruction-following while adding language support? Without testing these mitigations, we cannot conclude that inclusion during multitask finetuning is necessarily "the most effective method"—only that naive post-hoc adaptation fails.

The computational cost comparison is also missing. Adding a language to the full xP3 mixture requires rerunning the entire multitask finetuning process, which is substantially more expensive than adapter training on monolingual text. The paper does not provide a cost-benefit analysis that would tell a practitioner whether the accuracy improvement from xP3 inclusion justifies the additional compute relative to (hypothetically fixed) post-hoc adaptation.

Cross-cutting weaknesses:

• Single benchmark, single model family. All results are on MATH with PaLM 2-S* for the test-time compute paper, and on five NLU benchmarks with BLOOM for the language adaptation paper. Neither paper tests on multiple model families or multiple domains (code, dialogue, scientific reasoning), so the generality of both sets of findings is an open question.
• Test set sizes. The MATH test set is 500 questions, split into five quintiles of ~100 each, then split by two-fold cross-validation, meaning strategy selection is based on ~50 questions per fold per bin. The BLOOM+1 evaluation uses 500 XNLI test examples per language, with similar or smaller sizes for other tasks. Confidence intervals are not reported for either paper, making it difficult to assess the statistical reliability of the fine-grained difficulty-bin comparisons.
• Missing combined experiments. The test-time compute paper studies PRM search and revisions independently but never combines them (PRM tree-search with the revision model as the proposal distribution). The language adaptation paper studies three adaptation strategies independently but does not test combinations (e.g., continued pretraining of embeddings + adapters for Transformer blocks), which prior work (Mao et al., 2022; He et al., 2022) has shown can outperform single methods.
• The 14× larger model baseline in the FLOPs-matched comparison uses greedy decoding with no test-time compute augmentation. Giving this larger model even a modest test-time compute budget (best-of-8 or best-of-16) would create a much stronger baseline. The paper's claim that test-time compute can substitute for pretraining would be more convincing if it showed the smaller model + test-time compute outperforming a larger model + some test-time compute, rather than a larger model with no test-time compute at all.
• Difficulty estimation cost is unaccounted for. Generating 2048 samples per question to estimate difficulty for the compute-optimal policy is extremely expensive—comparable to or exceeding the test-time compute budgets being studied. The paper acknowledges this but does not amortize this cost into the reported efficiency gains, making the 4× figure an upper bound rather than a realized deployment gain.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Numbers

The assumption or constraint. The paper's compute-optimal test-time scaling framework—which achieves the headline ~4× efficiency improvement over best-of-N baselines—depends entirely on knowing each prompt's difficulty before allocating compute. The method for estimating difficulty is generating 2,048 complete solutions per question, then averaging either ground-truth correctness (oracle) or the PRM's final-answer score (predicted) and binning into quintiles (Section 3.2). The authors acknowledge this cost explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This means the ~4× gain is computed after difficulty is known, without amortizing the cost of learning it. For a single question, generating 2,048 samples costs 8× more than the largest test-time budget studied (256 generations for search) and 4× more than the budget at which the ~4× efficiency claim is demonstrated (e.g., 16 generations of compute-optimal search matching 64 generations of best-of-N).

The consequence. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could easily dominate the latter. The true end-to-end efficiency gain over best-of-N would be substantially smaller than the reported ~4×, potentially vanishing entirely for single-question scenarios where the difficulty estimation overhead cannot be amortized. The compute-optimal framework only becomes economical when difficulty estimation cost is spread across many questions from the same distribution (e.g., batch evaluation of thousands of problems), which the paper does not discuss.

What evidence exists in the paper. The limitation is acknowledged in Section 3.2 but no experiments quantify the end-to-end cost including difficulty estimation. The predicted-difficulty variant (using PRM scores rather than ground-truth labels) eliminates the need for answer access but does not reduce the 2,048-sample cost—the PRM must still score all 2,048 solutions. Figure 4 and Figure 8 show that predicted-difficulty curves track oracle curves closely, confirming that the PRM-based proxy works, but neither figure includes the difficulty estimation cost in the x-axis budget.

Mitigation status. The paper does not attempt to mitigate this limitation experimentally. Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" from the question text alone, as well as adaptive schemes that interleave difficulty assessment with problem-solving. No such models are trained or evaluated. Until this gap is closed, the ~4× figure should be understood as an upper bound on achievable efficiency in a setting where difficulty is known in advance, not a realized deployment gain.

---

Test-Time Compute Cannot Compensate for Fundamental Capability Gaps—Hard Problems Remain Unsolved

The assumption or constraint. The paper implicitly assumes that test-time compute amplifies existing capability rather than creating new capability. If the base model's pass@1 rate on a problem class is near zero—meaning the model essentially never produces a correct solution even with many independent attempts—then no amount of search, revision, or compute-optimal allocation can find a correct answer because none exists in the proposal distribution. The paper states this explicitly in the Section 7 takeaway:

"Test-time compute amplifies existing capability but does not create it from nothing."

The consequence. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for both beam search and best-of-N weighted at all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both revisions and search, well below the performance of the ~14× larger pretrained model. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path, as the FLOPs-matched comparison demonstrates: on hard problems at high inference-to-pretraining ratios (R ≫ 1), test-time compute with PRM search shows a −52.9% relative disadvantage versus the larger pretrained model (Figure 1, bottom-right bar chart).

What evidence exists in the paper. The difficulty-bin breakdowns in Figure 3 (right), Figure 7 (right), and Figure 9 provide consistent evidence across all methods. The FLOPs-matched bar charts in Figure 1 quantify the magnitude of the failure on hard problems. The paper is transparent about this boundary, making it one of the better-documented limitations in the work.

Mitigation status. Not addressed. The paper frames this as a fundamental boundary condition rather than a solvable limitation within the test-time compute paradigm. Section 7 explicitly recommends pretraining over test-time compute for problems outside the base model's capability range. This is the correct intellectual position—acknowledging the boundary rather than attempting to overcome it with the wrong tool—but it means the approach's applicability is strictly limited to problems where some non-trivial fraction of base-model outputs are already on the right track.

---

The ~14× Larger Model Baseline in the FLOPs-Matched Comparison Is Not Compute-Optimally Trained

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, matching the LLaMA training paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both parameters and data are scaled equally (Hoffmann et al., 2022). The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model is evaluated using only greedy decoding—no majority voting, no best-of-N, no verifier-guided selection, no test-time compute augmentation of any kind. The smaller model (PaLM 2-S*) receives the full benefit of compute-optimal test-time scaling, while the larger model receives none.

The consequence. A Chinchilla-optimal model trained with ~14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on easy-to-medium questions at R ≪ 1 with revisions (Figure 1, top-right bar chart)—may shrink or reverse against a properly compute-optimal larger model. Similarly, giving the larger model even a modest test-time compute budget (best-of-8 or best-of-16) would create a substantially stronger baseline. The current comparison answers the question "can a small model with smart inference beat a large model with naive inference?" but does not answer the more policy-relevant question "given a fixed total FLOPs budget, should I spend it on pretraining or on inference?"—because the pretraining portion of the budget is not itself allocated optimally.

What evidence exists in the paper. The limitation is acknowledged explicitly in Section 7 but no experiments test against a compute-optimally trained larger model or a larger model with any test-time compute augmentation. The exact parameter count of the ~14× larger model is not disclosed (consistent with the PaLM 2 technical report's limited disclosure), making it impossible for other researchers to independently verify the FLOPs accounting or replicate the comparison.

Mitigation status. The authors flag this as future work (Section 7) but do not provide even approximate estimates of how much the results would change under compute-optimal pretraining scaling. The FLOPs-matched conclusions should be interpreted as directional evidence that test-time compute and pretraining compute are not 1-to-1 exchangeable, with the specific numerical advantages contingent on a pretraining baseline that is acknowledged to be suboptimal.

---

Revisions and Search Are Studied Independently—Their Complementary Strengths Are Never Combined

The assumption or constraint. The paper studies two complementary axes of test-time compute—PRM tree-search (modifying the verifier/selection mechanism) and iterative revisions (modifying the proposal distribution)—but never combines them. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The paper's own framework (Section 2) motivates these as two independent, complementary levers for modifying a model's output distribution, and the empirical results show they have complementary difficulty-dependent strengths: search helps most on medium-difficulty problems where the PRM can guide exploration (Figure 3, right, bins 3–4), while sequential revisions help most on easier problems where initial answers are roughly correct and need refinement (Figure 7, right, bins 1–2). Natural next steps include using the revision model as the proposal distribution within beam search, or using the PRM's per-step scores to decide when to revise versus restart a chain.

The consequence. The paper's reported performance numbers represent a lower bound on what a fully integrated system combining both mechanisms could achieve. Without combination experiments, we cannot know whether the gains from revisions and search are additive, subadditive (they help on disjoint subsets of problems and combining them simply captures both subsets), or superadditive (revision-generated candidates are easier for the PRM to score accurately, reducing over-optimization). The ~4× efficiency gain over best-of-N is computed separately for search (Figure 4) and revisions (Figure 8) but never for their combination. If combining them pushes the efficiency gain to ~8× or higher, the paper significantly understates the potential of test-time compute scaling; if combining them yields diminishing returns (e.g., revision models produce outputs that are harder for the PRM to score, exacerbating over-optimization), the separate results overstate what a practical system would achieve.

What evidence exists in the paper. None—this is a missing experiment, not a measured limitation. The paper's architecture and experimental infrastructure support it (both the revision model and the PRM exist and work on the same base model), making the absence notable. The BLOOM+1 experiments in Appendix D similarly document that Composable Sparse Fine-Tuning (Ansell et al., 2022) was tested but abandoned due to poor prompting performance, showing the authors were willing to explore method combinations that did not make the main paper—which makes the absence of a revisions+search combination more puzzling.

Mitigation status. Section 8 identifies this as "a natural next step" for future work but provides no preliminary results or analysis of why the combination might succeed or fail. Practitioners interested in maximizing test-time compute efficiency should treat the separate search and revision results as ablation points on a larger design space that remains unexplored.

---

Sequential Revisions Introduce Latency That Is Incompatible with Interactive or Real-Time Deployment

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial—each revision step depends on the output of the previous step and cannot be parallelized. Parallel best-of-N sampling can be executed simultaneously across many accelerators, so 256 parallel samples might complete in the time it takes to generate a single solution. A strategy that allocates 128 generations as 64 sequential steps with 2 parallel chains takes roughly ~64× longer wall-clock time than running 128 parallel samples simultaneously, even though both consume similar total FLOPs.

The consequence. The compute-optimal policies derived in Section 6 favor sequential-heavy allocations for easy and medium-easy problems (Figure 7, right: bins 1–2 perform best with high sequential-to-parallel ratios). These are precisely the problems that dominate many production workloads—routine questions, standard coding tasks, common factual queries. Deploying the compute-optimal policy on such workloads would introduce serial dependencies that increase user-facing latency by ~10–60× compared to parallel sampling with the same generation budget. For latency-sensitive applications (interactive assistants, real-time decision-making, live code completion), these strategies may be impractical regardless of their accuracy advantages. The parallel baseline's plateau around 36–37% accuracy (Figure 8) versus compute-optimal scaling's continued improvement to ~44% at 256 generations represents a genuine accuracy-vs-latency tradeoff that the paper does not characterize.

What evidence exists in the paper. Figure 3 reports training time and GPU memory usage for the different adaptation strategies, and Section 3.4 notes that continued pretraining of BLOOM-7.1B required eight A100 GPUs, showing the authors are aware of computational cost differences. However, no latency measurements or wall-clock time comparisons are reported for the inference-time strategies (beam search, lookahead search, sequential vs. parallel revisions). The paper does not discuss whether beam search can be parallelized across beams at each step (potentially recovering some parallelism) or whether revision chains can be truncated early when improvements plateau (reducing average latency).

Mitigation status. Not addressed. The paper focuses exclusively on total FLOPs as the cost metric, with no discussion of latency constraints or throughput requirements. For practitioners, this means the reported efficiency gains for sequential-heavy strategies cannot be directly translated to deployment decisions without factoring in the latency budget of their specific application. A throughput-optimized deployment might prefer parallel-only strategies despite their lower accuracy, and the paper provides no guidance on navigating this tradeoff.

---

All Findings Are Based on a Single Benchmark (MATH) and a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the test-time compute paper uses the MATH benchmark (500 competition-level math problems) with PaLM 2-S* (Codey) as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. MATH consists exclusively of formal mathematical reasoning problems with unambiguous ground-truth answers, which enables both the PRM training pipeline (via Monte Carlo rollout correctness checking) and the difficulty estimation protocol (via pass@1 measurement). The BLOOM+1 paper similarly uses five NLU benchmarks with BLOOM models, but that is a separate set of findings.

The consequence. Several aspects of the results could be specific to the MATH-PaLM 2-S* combination and fail to transfer to other settings: the PRM's over-optimization behavior depends on the verifier's calibration on PaLM 2-S*'s output distribution, which may differ for other model families with different error patterns; the difficulty-dependent scaling curves (beam search degrading on easy problems, helping on medium ones) are a function of the specific relationship between PaLM 2-S*'s pass@1 distribution and the PRM's discrimination ability—a model with a different capability profile might show different crossover points or qualitatively different patterns; domains without clean correctness signals for verifier training (open-ended generation, creative writing, dialogue, complex planning) cannot use the Monte Carlo rollout PRM training procedure at all, requiring fundamentally different approaches to both verifier construction and difficulty estimation; and the threshold at which test-time compute stops helping ("bin 5" problems with near-zero pass@1) is specific to PaLM 2-S* on MATH—a stronger base model, or a different task domain, might show different boundaries.

What evidence exists in the paper. None—the paper studies exactly one benchmark on exactly one model family. No experiments test on code generation (HumanEval, MBPP), logical reasoning (ARC, FOLIO), scientific QA, or any other domain. No experiments test with a different base LLM family (GPT, LLaMA, Gemma) to assess whether the difficulty-dependent patterns replicate across architectures and training distributions.

Mitigation status. The authors do not claim broader generality—Section 8 lists extension to other domains as future work—but the paper's recommendations (use beam search on medium problems, sequential revisions on easy problems, compute-optimal allocation per difficulty bin) are presented as general findings, not as MATH-PaLM 2-S*-specific observations. Practitioners should treat these recommendations as hypotheses to be validated on their specific model-task combinations rather than as established principles. The BLOOM+1 paper's inclusion of eight diverse languages and five NLU tasks provides a partial template for the kind of multi-domain, multi-task validation that would strengthen the test-time compute findings, but that validation has not been performed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper instigates a conceptual reframing of language adaptation from a universal recipe to a scale-dependent decision, with direct implications for how researchers and practitioners approach extending large language models to new languages. Prior to this work, the dominant narrative—established by Ebrahimi and Kann (2021) on masked language models at sub-billion-parameter scales—was that continued pretraining represents the most promising direction for language adaptation, full stop. The logic was intuitive and widely accepted: more parameter freedom means more capacity to accommodate a new language, so updating everything should outperform updating a subset.

BLOOM+1 demonstrates that this recommendation reverses at scale. For BLOOM models at 560M parameters, the paper replicates the prior finding—continued pretraining yields the best zero-shot prompting performance. But at 3B parameters and above, both MAD-X bottleneck adapters and (IA)³ rescaling vectors systematically outperform continued pretraining despite updating orders of magnitude fewer parameters (Figure 1). The reversal is not a minor quantitative tweak—it is a qualitative change in which strategy works, driven by a specific mechanistic explanation: larger models, given the freedom to update all parameters during continued pretraining, can partition their representational space and diverge from the language-independent representations that enable cross-lingual transfer, while adapters preserve these representations by construction (Section 4.3, Figure 5).

This finding changes the landscape in several concrete ways:

It converts a universal recommendation into a diagnostic question. Rather than asking "which adaptation strategy is best?", the field should now ask "at what model scale, under what data budget, and for what evaluation regime?" The paper provides the diagnostic tool—sentence retrieval accuracy as a probe for language-independent representation quality (Section 4.3)—that explains why the ranking flips and can guide strategy selection for new model families and scales without exhaustively benchmarking every combination.

It partially resolves the contradiction between the adapter and continued-pretraining literatures. Prior work was split: adapter-based methods (MAD-X, BAD-X, adapters for NMT) showed strong cross-lingual transfer on specific tasks, while continued pretraining showed superior language modeling and token-level task performance. This paper's demonstration that the ranking depends on model scale—and that perplexity and downstream prompting accuracy can anti-correlate (Figure 4)—reconciles these findings. The adapter literature was studying the regime where adapters work (cross-lingual transfer, often on larger models), while the continued-pretraining literature was studying the regime where it works (smaller models, token-level evaluation, where representational divergence is less harmful or where task-specific finetuning can compensate for it). The two camps were not contradicting each other; they were operating at different points in a scale-dependent landscape that neither had mapped.

It redirects research attention from "more expressive adaptation" to "better-aligned adaptation." The finding that continued pretraining achieves lower perplexity but worse prompting performance than adapters at scale (Section 4.2) challenges the deeply embedded assumption that better language modeling implies better task capability. If improving perplexity on the target language can actively harm the cross-lingual transfer that enables zero-shot prompting, then the goal of language adaptation should be reframed: not "make the model a better language model of the target language" but "make the model better at using the target language for tasks, while preserving the cross-lingual alignment that enables transfer." This shifts the research agenda from developing more expressive adaptation methods toward developing adaptation methods that are constrained to preserve language-independent representations—a fundamentally different optimization target.

It establishes a categorical distinction between adapting base models and adapting instruction-tuned models. The BLOOMZ experiments (Section 4.6) reveal that the language adaptation playbook developed for base models does not transfer to instruction-tuned variants. MAD-X adaptation on unlabeled monolingual text destroys BLOOMZ's instruction-following capability (Figure 8), while adding the target language during the multitask instruction-tuning phase is effective but requires access to the full diverse task mixture (Figure 9). This means the field cannot treat "language adaptation" as a single problem with a single solution set—the post-pretraining history of the model (whether it has been instruction-tuned, RLHF'd, or otherwise specialized) determines which adaptation approaches are viable.

It identifies data quantity, not linguistic properties, as the primary bottleneck for adaptation success. The finding that script, word order, and language family do not independently predict adaptation difficulty when data quantity is held constant (Figure 2) simplifies the deployment calculus. It tells the community that efforts should focus on data acquisition and efficient training rather than on language-specific architectural modifications—a pragmatic refocusing that could accelerate language coverage expansion for low-resource languages where linguistic expertise is scarce but text data, even if limited, can be collected.

The magnitude of this contribution is best characterized as a substantive reframing with practical teeth, not a paradigm shift. The paper does not introduce fundamentally new adaptation methods—MAD-X, (IA)³, and continued pretraining all predate it. What it provides is the scaling analysis and the mechanistic explanation that convert a confused empirical landscape (different papers recommending different strategies for different models and tasks) into a coherent picture with clear boundary conditions. The ~100M token threshold for effective adaptation (Section 4.4, Table 6) and the ~3B parameter threshold for the adapter-over-continued-pretraining reversal give practitioners concrete numbers to plan against, moving the discussion from "try both and see what works" to "measure your model size and data budget, and here is what the evidence suggests."

Follow-Up Research This Work Enables

Stress-testing the adapter-over-continued-pretraining reversal on other large model families. The paper's central finding is demonstrated on exactly one architecture: BLOOM with AliBi positional embeddings and byte-level BPE tokenization. Both architectural choices are consequential for language adaptation. AliBi's fixed positional biases mean the model does not learn position representations from data, removing a potential source of language-specific interference that continued pretraining would need to overcome on models with learned positional embeddings (e.g., GPT-3, LLaMA, Gemma). Byte-level BPE means there is no vocabulary gap for new scripts, removing the cold-start embedding problem that continued pretraining might handle better than adapters. A direct replication on LLaMA-2 or LLaMA-3 at comparable scales (560M through 7B), adapting to the same set of eight languages using the same 100K-sentence data budget, would test whether the reversal is specific to BLOOM's architecture or generalizes to the model family most widely used in practice. The key measurement would be whether the crossover point shifts—does the adapter advantage appear at 3B parameters for LLaMA as well, or does the presence of learned positional embeddings push the crossover higher (or eliminate it entirely)?

Developing and validating sentence retrieval accuracy as a checkpoint selection criterion for language adaptation. The paper demonstrates that perplexity-based checkpoint selection can be actively misleading—continued pretraining achieves lower perplexity but worse downstream prompting performance than adapters at scale (Section 4.2, Figure 4). It also shows that sentence retrieval accuracy patterns align with downstream prompting performance in ways that perplexity patterns do not (Section 4.3, Figure 5). A natural follow-up would systematically compare checkpoint selection strategies for language adaptation: select checkpoints by (a) lowest validation perplexity, (b) highest sentence retrieval accuracy on a small parallel dataset (e.g., 200 FLORES sentences, as used in the paper's probe), and (c) highest downstream task accuracy on a small labeled validation set. The experiment would adapt BLOOM-7.1B to three languages (one seen-script like German, one unseen-script like Russian, one low-resource like Guarani) using both continued pretraining and MAD-X, save checkpoints every 5,000 steps, and evaluate all checkpoints on XNLI, XCOPA, and XWinograd. The question is whether sentence retrieval accuracy—which requires only a small parallel corpus, not task-specific labeled data—can serve as a practical, language-agnostic proxy for downstream prompting performance, enabling practitioners to select adaptation checkpoints without access to task labels in the target language. This would directly address the "perplexity does not predict task performance" limitation the paper identifies.

Combining continued pretraining of embeddings with adapter-based Transformer-block adaptation. The paper's ablation in Appendix K (Figure 16) shows that invertible adapters at the embedding layer provide only marginal benefit for most languages, and Appendix J (Figure 15) shows that adapter placement in the final Transformer layers is most impactful. This suggests a hybrid strategy: make the embedding layer trainable (as in continued pretraining) to adapt the input representations to the target language's byte-pair statistics, while keeping all Transformer-block parameters frozen and using MAD-X or (IA)³ adapters for linguistic processing. This hybrid would combine the continued-pretraining advantage at the input level (where vocabulary and byte-pair distribution adaptation happens) with the adapter advantage at the processing level (where language-independent representations must be preserved). The experiment would compare this hybrid against pure continued pretraining and pure MAD-X on BLOOM-7.1B across three languages with unseen scripts (Russian, Korean, Thai), measuring both perplexity (expecting improvement from trainable embeddings) and downstream prompting accuracy (expecting preservation of cross-lingual alignment from adapter-constrained Transformer blocks). This tests whether the paper's negative result on continued pretraining is driven primarily by Transformer-block weight updates (where the representational divergence hypothesis operates) or by embedding-layer updates as well, potentially recovering some of continued pretraining's modeling advantage without its alignment cost.

Measuring whether the adapter advantage persists at larger adaptation data budgets. The paper's resource-constrained setting (100K sentences, ~200M tokens seen) is well-motivated for the low-resource languages that most need adaptation. But many languages excluded from BLOOM—Korean, Russian, German—are not low-resource by any reasonable definition and have millions of available sentences. The paper's finding that adapters outperform continued pretraining at 100K sentences does not tell us what happens at 1M or 10M sentences. With more data, continued pretraining's greater expressivity might eventually overcome its representational disruption, or adapters might saturate in capacity (even with larger reduction factors as explored in Section 4.5). A systematic data scaling experiment—adapting BLOOM-7.1B to Russian using 1K, 10K, 100K, 1M, and 10M sentences with both continued pretraining and MAD-X, evaluating on the full suite of prompting tasks—would map out the data regime where each strategy dominates. The hypothesis from the paper's mechanistic explanation is that continued pretraining's disadvantage is driven by model scale, not data quantity, so the adapter advantage should persist even at larger data budgets. But this is untested, and the answer has direct practical implications: if a community has 1M+ sentences for their language, should they still use adapters (cheaper, preserves cross-lingual alignment) or does continued pretraining eventually pull ahead?

Language adaptation for instruction-tuned models without access to the full instruction-tuning mixture. The paper's BLOOMZ experiments establish a frustrating tension: post-hoc MAD-X adaptation on unlabeled text destroys instruction-following (Section 4.6.1), while adding the language during multitask finetuning is effective but requires the full xP3 mixture and its computational cost (Section 4.6.2). A practically crucial follow-up would test whether there exists a middle ground: can you adapt BLOOMZ to a new language using a mixture of unlabeled monolingual text and a subset of xP3 data (e.g., 10% of the original mixture) during adapter training, preserving enough instruction-following signal to maintain prompting capability while learning the new language? The experiment would adapt BLOOMZ-3B to Turkish (a language with both OSCAR data and existing xP3 task data) using MAD-X adapters trained on (a) pure OSCAR text (replicating the paper's negative result), (b) pure xP3 data in Turkish only (testing whether task data alone suffices for language adaptation without free-form text), and (c) a mixture of 50% OSCAR text and 50% Turkish xP3 data. The evaluation would measure both Turkish prompting accuracy and English prompting accuracy (to detect catastrophic forgetting of instruction-following). If the mixture approach works, it would provide a practical recipe for extending instruction-tuned models to new languages without requiring the full multitask finetuning pipeline.

Extending the language adaptation findings to generation and interactive tasks. The paper evaluates exclusively on NLU tasks (NLI, paraphrasing, commonsense reasoning, anaphora resolution) using zero-shot prompting. Whether the scale-dependent strategy recommendations transfer to generation tasks (summarization, machine translation, open-ended QA) or interactive settings (multi-turn dialogue, instruction following in the target language) is entirely open. Generation tasks place different demands on language adaptation: they require the model to produce fluent, coherent text in the target language, not just classify or select among candidates. Continued pretraining, which achieves lower perplexity (better language modeling), might have an advantage for generation quality even at large scales, since fluency and coherence depend more directly on next-token prediction accuracy than on cross-lingual representational alignment. A direct test would adapt BLOOM-7.1B to German using both continued pretraining and MAD-X, then evaluate on German summarization (MLSUM) and German-English translation (WMT) in addition to the XNLI/XCOPA suite. The hypothesis is that the adapter advantage observed for NLU prompting may narrow or reverse for generation tasks where language modeling quality matters more than cross-lingual alignment, providing a more nuanced deployment recommendation: adapters for understanding tasks, continued pretraining for generation tasks—or a hybrid approach.

Practical Applications and Downstream Use Cases

Community-driven language expansion for open-source LLMs. The most direct application of this work is enabling groups with modest computational resources to add their language to an existing large open-source model like BLOOM without retraining from scratch. The paper's specific recommendation—use MAD-X or (IA)³ adapters for models above ~3B parameters, with approximately 100M tokens of monolingual text (~100K sentences for typical languages), trained for ~25K steps on a single consumer GPU (RTX 3090) for all but the 7.1B continued-pretraining case—makes this feasible for academic labs, language communities, and low-resource language activists. The hardware requirement (a single RTX 3090 for adapter training) is roughly $1,500, and the data requirement (100K sentences) is achievable for many languages with some web presence. This represents a democratization of large language model access: instead of requiring a BigScience-scale consortium to include a language in the pretraining mixture, individual research groups can adapt the already-trained model post-hoc. The finding that linguistic properties (script, word order, language family) do not significantly determine adaptation difficulty (Figure 2) means communities do not need to invest in language-specific engineering—they need to collect text data, run the adaptation recipe, and evaluate. The ~4× efficiency gain from using adapters over continued pretraining at the 7.1B scale (in terms of trainable parameters and training time, Figure 3) directly translates to reduced electricity costs, reduced GPU procurement requirements, and faster turnaround for communities operating on limited budgets.

Cost-efficient batch evaluation in multilingual NLP pipelines. For organizations running large-scale multilingual benchmarks or monitoring systems—evaluating model performance across dozens of languages on a regular cadence—this paper provides a concrete recipe for adding language support without the cost of full model retraining or the accuracy penalty of using an unadapted model. The paper's results show that an unadapted BLOOM model performs near random-chance levels on some language-task pairs (e.g., Figure 1, XNLI panels for Korean, Thai, Guarani), while MAD-X adaptation with 100K sentences brings performance to competitive levels with models that saw the language during pretraining (e.g., matching mGPT on several XNLI tasks). The practical workflow is: (1) collect 100K sentences in the target language from OSCAR or comparable web-crawled corpora, (2) train MAD-X adapters on a single GPU for ~25K steps using the hyperparameters in Table 5, (3) evaluate zero-shot prompting on the task suite using translated templates. The total cost per language is roughly one GPU-day and minimal engineering effort—dramatically cheaper than the alternative of either retraining the full model or deploying a separate model per language. The paper's finding that checkpoint selection via perplexity is unreliable (Section 4.2) suggests that practitioners should either use a fixed number of training steps rather than perplexity-based early stopping, or maintain a small labeled validation set in a related language for checkpoint selection, until better proxies (e.g., sentence retrieval accuracy) are validated.

Data augmentation for low-resource language instruction tuning. The BLOOMZ experiments (Section 4.6.2) demonstrate that adding a new language to the instruction-tuning mixture is effective, but requires diverse task data in that language. For genuinely low-resource languages where labeled NLP task data is scarce or nonexistent, the paper suggests a bootstrapping pipeline: (1) use MAD-X adapters to adapt a base BLOOM model to the low-resource language using whatever monolingual text is available (even 30K sentences, as with Guarani), (2) use the adapted model to generate synthetic task data in the target language by translating existing task prompts from a high-resource language or by few-shot prompting the adapted model with English task examples, and (3) include the synthetic task data in the xP3 mixture when instruction-tuning a BLOOMZ-style model. This pipeline leverages the paper's core finding—that adapter-based language adaptation works on unlabeled text—to bootstrap the labeled task data needed for the more powerful instruction-tuning approach. The paper's Guarani results (Figure 1, AmericasNLI panel) show that adapter-based adaptation provides non-trivial gains even with only 30K sentences (~1M tokens), though performance remains low in absolute terms, suggesting this bootstrapping approach is viable but would require careful validation of synthetic data quality.

Architectural decisions for the next generation of multilingual models. For teams designing and training upcoming multilingual LLMs, this paper provides empirical guidance on two architectural choices that facilitate post-hoc language expansion. First, byte-level BPE tokenization eliminates the vocabulary barrier for new scripts—the paper's ability to adapt to Cyrillic, Greek, Hangul, and Thai script without any vocabulary expansion or embedding initialization is a direct consequence of this design choice, and the positive adaptation results across all these scripts (Figure 2, right panel) validate that byte-level BPE does not create insurmountable tokenization quality issues for unseen scripts. Second, AliBi positional embeddings remove the need to adapt or retrain positional representations for languages with different word orders or typical sequence lengths—the paper's successful adaptation to SOV languages (Turkish, Korean) without positional embedding modification supports this. Future multilingual model designers can use these findings to justify architectural choices that prioritize post-hoc expandability: byte-level tokenization and fixed (non-learned) positional embeddings make the model more amenable to community-driven language addition after the main training run, at minimal cost to pretraining performance. This is a concrete architectural recommendation grounded in the paper's comparative results rather than theoretical argument.

When to Prefer This Method

The paper explicitly positions its three adaptation strategies—continued pretraining, MAD-X adapters, and (IA)³—as points on a spectrum and provides clear, empirically-grounded decision rules for when to prefer each, conditional on model scale and deployment constraints:

Prefer continued pretraining when:

• The base model has fewer than approximately 1B parameters (BLOOM-560M regime), where continued pretraining yielded the best zero-shot prompting performance (Figure 1)
• The target language has substantial available text (well above the 100K-sentence, 100M-token threshold the paper identifies as minimal for effective adaptation in Section 4.4)
• The deployment involves generation tasks (summarization, translation, open-ended text generation) rather than NLU prompting, where lower perplexity may translate more directly to output quality—though the paper does not test this, it follows from the finding that continued pretraining achieves better language modeling (lower perplexity, Figure 4) despite worse prompting performance
• Catastrophic forgetting of the source language (English) is acceptable, or the adapted model will be used monolingually rather than for cross-lingual tasks (Appendix L, Figure 18 shows continued pretraining causes significant English degradation regardless of model size)
• Computational budget is not constrained—continued pretraining requires training all model parameters with correspondingly higher GPU memory and time costs (Figure 3)

Prefer MAD-X bottleneck adapters when:

• The base model has at least 3B parameters, where adapters systematically outperform continued pretraining for zero-shot prompting (Figure 1)
• The primary evaluation is zero-shot NLU prompting (NLI, paraphrasing, commonsense reasoning, anaphora resolution)—the regime where the paper demonstrates the strongest adapter advantage
• Preserving cross-lingual transfer from English (or other pretraining languages) is important, since adapters maintain language-independent representations by design (Section 4.3, Figure 5)
• Training efficiency matters: MAD-X has significantly fewer trainable parameters and shorter training time than continued pretraining for large models (Figure 3), and can run on a single consumer GPU (RTX 3090) for models up to 7.1B
• The adaptation data budget is limited (the paper's resource-constrained setting of 100K sentences), where adapter capacity (controlled by the reduction factor, Section 4.5) can be matched to the available data
• The model will be deployed with the original tokenizer and vocabulary intact, without script-specific modifications—the paper's results show MAD-X adapters work across diverse scripts (Latin, Cyrillic, Greek, Hangul, Thai) without vocabulary expansion
• But not when the target model has been instruction-tuned (BLOOMZ): post-hoc MAD-X adaptation on unlabeled text destroys instruction-following capability (Section 4.6.1, Figure 8). For instruction-tuned models, the recommended approach is different—add the language during multitask finetuning (Section 4.6.2)

Prefer (IA)³ with invertible adapters when:

• The absolute minimum of trainable parameters and training time is required—(IA)³ has orders of magnitude fewer parameters than MAD-X (Figure 3) and can be merged with pretrained weights at inference for zero additional latency
• The base model is large (7.1B+ parameters), where (IA)³'s efficiency advantage over MAD-X is most pronounced (the training time gap widens with model scale in Figure 3)
• Slightly lower prompting performance than MAD-X is acceptable—the paper shows MAD-X generally achieves higher average accuracy (Section 4.1, Figure 1) but (IA)³ is competitive and sometimes superior on specific language-task pairs
• Inference latency is a critical constraint, since (IA)³ merged weights eliminate the additional forward-pass computation that MAD-X bottleneck adapters require at every Transformer layer

When not to use any of these approaches:

• The target language has extremely limited data (well below 30K sentences, or ~1M tokens): the paper's Guarani results (Figure 1, AmericasNLI) show minimal adaptation benefit at this scale, with models below 3B parameters showing essentially no improvement. Alternative approaches—cross-lingual transfer via related high-resource languages, few-shot in-context learning with translated examples, or data augmentation through machine translation—may be more effective when monolingual text is severely scarce.
• The goal is to add a language to an instruction-tuned model while preserving its instruction-following behavior: the paper's negative result on post-hoc BLOOMZ adaptation (Section 4.6.1) means the recommended path is to include the language during the instruction-tuning phase itself, using a diverse task mixture (Section 4.6.2), which requires substantially more resources than the adapter-training recipes documented in the paper.