Read More

ArXiv: 2211.05100

🎯 Pitch

Training a 176B-parameter multilingual model on 46 languages actually produced competitive NLP performance—matching proprietary giants—while emitting only 25 tons of CO₂, roughly a third of comparable efforts. But the paper’s starkest finding is that for truly low-resource languages like Swahili↔Yoruba, the model completely collapses to near-zero translation quality, demonstrating that scale alone cannot compensate for severe data scarcity.

---

1. Executive Summary

This paper presents BLOOM, a 176B-parameter open-access multilingual decoder-only Transformer language model developed through the BigScience collaboration of hundreds of researchers and trained on the ROOTS corpus—a composite dataset spanning 46 natural languages and 13 programming languages. The model achieves competitive performance across diverse benchmarks—matching or exceeding OPT-175B on one-shot SuperGLUE tasks (e.g., Ax-b, CB, WSC), delivering competent one-shot machine translation between high-resource language pairs (e.g., 45.0 BLEU for English→French on Flores-101), and reaching 15.52% pass@1 on HumanEval code generation—with stronger zero-shot generalization after undergoing multitask prompted finetuning into BLOOMZ (e.g., lifting XNLI accuracy from near-random to well above chance across languages like Arabic, Spanish, and Chinese). The model's training emitted approximately 25 tons of CO2eq—roughly one-third of OPT-175B's emissions—establishing that large-scale multilingual training can achieve competitive results with a significantly lower carbon footprint when leveraging low-carbon energy grids. On the hardest low-resource language pairs (e.g., Swahili↔Yoruba), performance collapses to near-zero BLEU, establishing that BLOOM's multilingual competence depends critically on sufficient representation in the pretraining corpus rather than cross-lingual transfer alone.

2. Context and Motivation

The Core Problem: Who Gets to Build and Access Large Language Models?

By the time BLOOM's development began in 2021, large language models had established themselves as a transformative technology, but their development was almost exclusively concentrated in well-resourced industrial organizations. GPT-3 (Brown et al., 2020) with 175B parameters, Gopher (Rae et al., 2021) with 280B, and PaLM (Chowdhery et al., 2022) with 540B were all developed behind corporate walls. This concentration created a fundamental access problem operating at multiple levels the paper explicitly identifies (Section 1):

No public access to trained models. Most LLMs were "frequently kept from the public," meaning researchers could not study their internal representations, probe their biases in a controlled way, evaluate them on custom benchmarks, or apply them to domain-specific problems. This isn't merely inconvenient—it excludes the majority of the research community from participating in understanding and improving the technology they are studying.

No participation in development decisions. The concentration of resources means decisions about what languages to support, what data to train on, what biases to filter for, and what values to encode in model behavior were made by small groups within corporations, guided primarily by "internal definitions of performance over their impact on society" (Section 2.1). As the paper notes, this means "the values of the developers come to be emphasized over those of the direct and indirect users" (citing Birhane et al., 2022).

English and Chinese dominance. Most LLMs were "primarily trained on English-language text (with notable exceptions in Chinese and Korean, e.g. Wang et al., 2021; Zeng et al., 2021; Kim et al., 2021)" (Section 1). This means the vast majority of the world's approximately 7,000 languages received essentially no representation in the technology that was increasingly mediating digital experiences. The paper frames this not just as a technical limitation, but as an issue of linguistic justice: when a technology becomes infrastructure, the languages it supports gain legitimacy and resources, while unsupported languages are further marginalized.

No transparent governance. Beyond technical access, there was no framework for data subjects (people whose text trains the models) to have agency over how their data is used, no structured accountability for harmful model outputs, and no participatory mechanism for affected communities to shape the technology's development. The development process was opaque by default.

A figure that crystallizes the situation: prior to BigScience, "EleutherAI (Phang et al., 2022) was the only non-corporate entity outside of China that was developing large language models" (Section 2.1). EleutherAI produced GPT-NeoX-20B, an important contribution, but at 20B parameters it was an order of magnitude smaller than the state-of-the-art corporate models, leaving a substantial capability gap that open-access models could not bridge.

The Underlying Tensions That Make This Hard

The paper is unusually transparent about the fact that addressing these problems involves navigating inherent tensions—there is no pure solution, only tradeoffs that must be explicitly managed (Section 3.1.2). Some of these tensions include:

Scale vs. inclusivity. To build a competitive multi-hundred-billion parameter model, you need enormous quantities of text. The languages with the most available text are English, Chinese, and a handful of other high-resource languages. Including genuinely low-resource languages (e.g., Yoruba with ~90 MB in ROOTS) means they will be a tiny fraction of the training data, raising the question: does including them meaningfully serve their speaker communities, or is it performative? The paper doesn't resolve this tension but rather documents how the team navigated it—by requiring that any language added to the corpus have at least three fluent speakers commit to guiding source selection and preprocessing, a labor-intensive requirement that limited total language count but improved per-language quality.

Data volume vs. data governance. The standard approach to building LLM training datasets—crawl the web broadly, apply heuristic filters, and train—produces massive datasets with essentially zero documentation of provenance, no meaningful consent from data subjects, and biases that are difficult to trace because individual items have lost their lineage (Section 3.1). Doing better—seeking explicit permission, maintaining source traceability, respecting license restrictions—fundamentally limits how much data you can collect and introduces dependencies on external organizations. The 1.61 TB ROOTS corpus is substantially smaller than what could have been collected through indiscriminate crawling, and the paper is explicit that 225 of the 498 component datasets cannot be publicly distributed due to licensing, privacy, or agreement constraints (Section 3.1.1).

Research replicability vs. responsible release. The scientific ideal of full reproducibility requires releasing the complete training dataset. But responsible release requires respecting licenses, privacy risks, and agreements with data providers. The paper adopts a compromise: components that can be released are available on the Hugging Face Hub; the full corpus is accessible to researchers with a relevant project who sign up for access (Section 3.1.1). This maintains some scientific scrutiny while acknowledging that unrestricted distribution of web-scraped personal data would be irresponsible.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work across several dimensions:

Data curation as an afterthought. The dominant paradigm, exemplified by GPT-3's training on filtered Common Crawl data, treats data work as "generally under-valued" (Section 3.1, citing Sambasivan et al., 2021). The filtering approaches used—blocklists to remove "pornographic" content, Reddit upvotes as a quality signal, machine learning models for text-image alignment—compound their own biases. The paper cites three specific documented failures: (1) blocklists used to filter "pornographic" text also suppressing LGBTQ+ and African American English content (Dodge et al., 2021); (2) Reddit-based quality signals producing models that "implicitly prioritize US-centric views" (Johnson et al., 2022); and (3) image-text alignment models exacerbating biases in multimodal datasets (Birhane et al., 2021). These are not hypothetical concerns—they are documented, measured harms in prior systems.

Documentation impossibility. When data from diverse sources is aggregated and filtered through automated pipelines, "the provenance and authorship of individual items is usually lost in the process" (Section 3.1). This means post-hoc auditing—tracking down why a model produces a particular biased output, identifying whose data contributed to harmful behaviors, determining whether copyrighted material was included—is effectively impossible. The paper positions BLOOM's approach of maintaining source traceability "until the final stages of preprocessing" as a direct counterpoint.

Absence of linguistic expertise. The prior approach of relying on automatic language identification without specific language expertise leads to well-documented problems: misidentification of closely related languages, inclusion of machine-translated text that is lower quality, and failure to account for language-specific structural features. The paper cites Caswell et al. (2022), which conducted a quality audit of web-crawled multilingual datasets and found pervasive issues. BLOOM's requirement of fluent-speaker guidance for each language in the corpus is a direct response to this failure mode, though as the paper acknowledges, it limits scalability.

The zero-shot architecture gap. Beyond the data and access issues, the paper identifies a specific technical gap. Section 3.2.2 notes that while Raffel et al. (2020) had shown encoder-decoder models "significantly outperform decoder-only models for transfer learning" (the finetuning paradigm), the field had shifted to decoder-only architectures for the largest models without a systematic evaluation of which architecture works best for zero-shot generalization. The paper's architecture work (detailed in the separate experiments paper by Wang et al., 2022a) fills this gap, finding that "immediately after pretraining, causal decoder-only models performed best" for zero-shot settings—validating the architectural choice made by GPT-3, Gopher, and others, but now with empirical grounding rather than precedent.

How This Paper Positions Itself

BLOOM is not positioned as an attempt to achieve state-of-the-art on any particular benchmark. Rather, it is a demonstration that an alternative development model is possible: one that is open-access, multilingual by design, governed through a structured ethical framework, and documented with unusual thoroughness.

The paper's ambitions are multi-layered:

As a model release, BLOOM aims to provide a competitive 176B-parameter language model that the research community can study, fine-tune, probe, and deploy—filling the gap between small open models (GPT-NeoX-20B) and inaccessible corporate systems.

As a dataset release, ROOTS aims to provide a documented, traceable, multilingual corpus whose construction process can be scrutinized and improved upon, in contrast to corpora assembled through opaque automated filtering.

As a process demonstration, the BigScience collaboration aims to show that organizing hundreds of researchers across disciplines and geographies can produce artifacts competitive with corporate efforts while maintaining stronger ethical standards.

As a governance experiment, the project aims to develop frameworks—the Ethical Charter, the Responsible AI License (RAIL), the data governance structures—that can serve as templates for future collaborative AI development.

None of these ambitions is framed as solved. The paper repeatedly acknowledges limitations: only 46 languages (out of thousands), difficulty estimation that remains expensive, the exclusion of certain architectural innovations like mixture-of-experts due to tooling limitations, and the fundamental constraint that test-time compute cannot create capabilities the base model lacks. The contribution is establishing feasibility—that an alternative path exists—not claiming perfection along that path.

The paper's most distinctive rhetorical move is to treat the development process itself as a primary contribution. Section 2.2 ("BigScience") documents the organizational structure in detail: 30 working groups, 1,200+ registered participants from 38 countries, backgrounds spanning "machine learning and computer science, but also linguistics, statistics, socio-cultural anthropology, philosophy, law, and other fields." Section 3.1 documents the data governance framework. Section 3.6 documents the model card and licensing decisions. Section 3.5.1 documents the carbon footprint analysis using Life Cycle Assessment methodology. Even Section 6 explicitly maps individual contributors to specific aspects of the project. This level of process documentation is unusual in ML papers and reflects the project's thesis that how we build models matters at least as much as what we build.

3. Technical Approach

3.1 Reader Orientation

BLOOM is a 176-billion-parameter autoregressive language model—a neural network that takes a sequence of text tokens as input and predicts the next token in the sequence, trained on 1.61 terabytes of multilingual text so that it learns patterns spanning 46 natural languages and 13 programming languages. The core problem BLOOM addresses is the concentration of large language model development within resource-rich organizations and the consequent exclusion of most of the world's languages, research communities, and ethical perspectives from the technology's creation. The solution is not a single technical innovation but rather a vertically integrated system spanning data governance, corpus construction, tokenizer design, architecture selection, distributed training engineering, and responsible release—all conducted openly by a geographically and disciplinarily diverse collaboration, producing a model competitive with similarly-sized industrial systems while maintaining documented provenance, a structured ethical framework, and public accessibility.

3.2 Big-Picture Architecture (Diagram in Words)

The BLOOM system has six major components, organized chronologically by the development pipeline:

1. Data Governance Framework (Section 3.1.1): A structured process for determining what data can be used, under what conditions, with what traceability. This component doesn't produce data directly—it produces rules and agreements that constrain all subsequent components. Outputs: source-level agreements with data providers, a composite release strategy, and a sign-up mechanism for full-corpus research access.

2. ROOTS Corpus Construction Pipeline (Sections 3.1.2–3.1.3): A multi-stage pipeline that sources, filters, deduplicates, and aggregates text from 498 individual datasets into a unified 1.61 TB training corpus. The pipeline flows: source identification (crowdsourced catalogue + OSCAR + GitHub) → data extraction (downloading, PDF scraping, HTML extraction from Common Crawl) → quality filtering (per-language, per-source human-guided indicator selection) → deduplication and PII redaction → final aggregation. The key design principle is that individual sources remain traceable and separable until the final stages, enabling per-source handling and documentation.

3. Tokenizer (Section 3.3): A byte-level BPE tokenizer with a 250,680-token vocabulary trained on a deduplicated subset of ROOTS with language sampling ratios matching the training distribution. The tokenizer converts raw text into sequences of integer token IDs that the model can process. Its design is validated against monolingual tokenizers using fertility—the average number of subword tokens produced per word—with a goal of not degrading fertility by more than 10 percentage points relative to language-specific tokenizers.

4. Model Architecture (Section 3.2): A 70-layer decoder-only Transformer with ALiBi positional embeddings, an additional LayerNorm after the embedding layer, GELU activations, and 14336-dimensional hidden states with 112 attention heads. This component processes a sequence of token embeddings and produces a probability distribution over the vocabulary for the next token at each position. The architecture was selected through systematic ablation experiments at 1.3B and 6.7B parameter scales before scaling to 176B.

5. Distributed Training Infrastructure (Section 3.4): The engineering layer that enables training a 176B-parameter model across 384 NVIDIA A100 80GB GPUs on 48 nodes of the Jean Zay supercomputer. Uses Megatron-DeepSpeed with 3D parallelism—a fusion of data parallelism (replicating the model across devices), tensor parallelism (partitioning individual layers across GPUs), and pipeline parallelism (splitting layer sequences across GPUs)—plus ZeRO stage 1 optimizer sharding. Achieved 156 TFLOPs (half of theoretical peak).

6. Multitask Finetuning Pipeline (Section 3.1.4, executed post-pretraining): Takes the pretrained BLOOM model and finetunes it on xP3, a collection of prompted datasets covering 83 datasets in 46 languages across 16 tasks. This produces BLOOMZ, which exhibits substantially stronger zero-shot generalization. This component is the interface through which the base model acquires the ability to follow natural language instructions for diverse tasks without task-specific training.

Information flows through these components linearly during development: governance rules constrain data collection, which produces the corpus, which trains the tokenizer and the model architecture (the tokenizer provides the vocabulary and embedding matrix; the architecture processes tokenized sequences), which is optimized via the distributed training infrastructure, which produces pretrained BLOOM, which is optionally finetuned to produce BLOOMZ.

3.3 Roadmap for the Deep Dive

• First, the data governance framework and ROOTS corpus construction, because the corpus is the foundation—every subsequent component (tokenizer vocabulary, architectural scaling behavior, multilingual capabilities) depends on what data is included and how it is processed. Understanding the data decisions explains why BLOOM behaves the way it does across languages.
• Second, the tokenizer design, because it defines the vocabulary that the model's input and output layers operate over, and tokenizer quality directly impacts multilingual performance—a poor tokenizer can undermine even an otherwise well-trained model.
• Third, the model architecture and pretraining objective, because these determine how the model learns from the tokenized corpus. The architecture ablation experiments justify why BLOOM looks the way it does rather than replicating GPT-3 exactly.
• Fourth, the distributed training engineering, because without this component the 176B model cannot exist—the parallelism strategy, floating-point format, fused kernels, and hardware configuration are not implementation details but enabling conditions.
• Fifth, the training hyperparameters and carbon footprint, because they specify the exact recipe used and quantify the environmental cost.
• Sixth, the release framework (model card, licensing), because the paper positions release decisions as a technical contribution—the Responsible AI License and documentation choices are designed to enable some uses while constraining others.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and process paper whose core idea is that building a competitive open-access multilingual LLM requires simultaneous attention to data governance, corpus design, architecture selection, engineering, and responsible release—not just scaling a known architecture on aggressively crawled data. The technical contributions span all six components above, with no single algorithmic innovation but rather a carefully integrated set of design decisions that collectively produce a model competitive with industrial systems while maintaining openness and ethical standards.

---

3.4.1 Data Governance Framework

Before any data could be collected, the BigScience project needed to establish how data decisions would be made—who has authority over what data, under what conditions data can be used, and what obligations exist to data subjects and rights-holders. This governance framework is unusual in ML papers, which typically treat data collection as a purely technical problem, and reflects the project's thesis that ethical data practices require structural mechanisms, not just good intentions.

The long-term vision. The project developed a framework for international data governance (documented in Jernite et al., 2022) that envisions a network of data custodians (institutions or individuals who hold data), rights-holders (people and organizations who legally own or are represented in the data), and other parties interacting through structured agreements. These agreements specify what data may be used for, creating enforceable constraints rather than ad hoc judgments. The paper is explicit that this long-term vision could not be fully realized within the project timeline: "we were not able to fully establish an international organization in the comparatively short time between the project start and model training" (Section 3.1.1).

Concrete practices adopted. Three practices were implemented within the project's constraints:

Explicit permission-seeking. Wherever possible, the project sought direct permission from data providers for use within BigScience specifically. Two examples are given: the S2ORC corpus (Lo et al., 2020), managed by the Allen Institute for AI, and articles from the French newspaper Le Monde. This is different from the standard approach of assuming that publicly accessible data is free to use; it creates a paper trail of consent and establishes the project's legitimacy in using the data.

Source-level traceability. Individual data sources were kept separate "until the final stages of preprocessing" (Section 3.1.1). This means that throughout sourcing, filtering, and cleaning, it remained possible to identify which specific dataset a given text came from. This enables per-source handling based on that source's specific context—for example, applying different quality filters to different sources, or excluding a source entirely if issues are discovered—and maintains a chain of provenance that would be lost if all sources were immediately aggregated.

Composite release. The final corpus release is not a single monolithic download but a collection of components whose availability depends on their licensing status, privacy risks, and agreements with custodians. Of the 498 component datasets, 223 can be directly distributed. The remainder are accessible through a sign-up mechanism for researchers with relevant projects (Section 3.1.1: "we are also inviting researchers with a relevant research project in mind to join ongoing efforts to analyze the data through a sign-up form"). This is a compromise: it maintains some scientific scrutiny—researchers can verify corpus properties and reproduce training—while acknowledging that unrestricted distribution of all crawled data would violate agreements and potentially expose personal information.

Why this matters technically. The governance framework directly constrains the data collection pipeline. The corpus is smaller and more curated than it could be because certain sources were excluded by governance decisions, certain sources required permission that limited how they could be processed, and the release mechanism prevents treating all data as equally distributable. The paper treats these constraints as features, not bugs—they represent deliberate choices to prioritize ethical practice over dataset size.

---

3.4.2 ROOTS Corpus Construction

The ROOTS corpus (Laurençon et al., 2022) is a 1.61-terabyte composite of 498 Hugging Face datasets spanning 46 natural languages and 13 programming languages. Its construction follows a multi-stage pipeline: language selection → source identification → data extraction → quality filtering → deduplication → PII redaction → aggregation.

Language selection. The process was incremental and expertise-gated. It began with "eight of the world's largest languages by number of speakers" for which active outreach recruited fluent speakers (Section 3.1.2). Then, on the recommendation of language communities, Swahili was expanded to the category of Niger-Congo languages, and Hindi/Urdu to Indic languages. Finally, "any group of 3 or more participants fluent in an additional language could add it to the supported list if they would commit to selecting sources and guiding processing choices in the language" (Section 3.1.2). This requirement—that every language have at least three committed fluent speakers actively involved—is the mechanism that prevents the common failure mode of automatic language identification without expertise (documented by Caswell et al., 2022). It also explains why the corpus covers 46 languages rather than, say, 100: each additional language requires recruiting and coordinating multiple fluent speakers willing to participate in a lengthy data curation process.

Source identification through catalogue. The primary sourcing mechanism was the "BigScience Catalogue": a list of data sources compiled by workshop participants and research collectives through organized hackathons. Communities like Machine Learning Tokyo, Masakhane (African NLP), and LatinX in AI co-organized these events (McMillan-Major et al., 2022). This produced 252 sources identified through bottom-up community knowledge rather than top-down automatic crawling—a deliberate inversion of the standard data collection paradigm. Each source was proposed by people with contextual knowledge of the language and its data landscape.

Complementary sources. Three additional sourcing strategies supplemented the catalogue:

• Pseudocrawl: For Spanish, Chinese, French, and English, participants identified locally relevant websites and extracted their text from Common Crawl snapshots. This increases geographic coverage within already well-represented languages by adding region-specific content rather than more generic web text. Example: a French-language news site from West Africa, rather than just metropolitan French sources.

• GitHub Code: A dataset of programming languages collected from Google's BigQuery GitHub data collection, deduplicated of exact matches. The choice of languages mirrored the AlphaCode model's design (Li et al., 2022). This constitutes approximately 11% of the final corpus (174.7 GB out of 1.61 TB, from Table 1).

• OSCAR version 21.09: The February 2021 Common Crawl snapshot processed through the OSCAR pipeline (Ortiz Suárez et al., 2019; Abadji et al., 2021). This source ended up constituting "38% of the corpus" (Section 3.1.2)—a substantial fraction that reflects the practical necessity of web-crawled data to reach the volume needed for a 176B-parameter model, even when the project's philosophy favors curated sources.

Data extraction and processing. Obtaining usable text from these diverse sources required developing new tools:

• Format conversion: NLP datasets in various formats (question answering, summarization, dialogue) needed their text fields extracted.
• PDF scraping: Large archives like the French HAL repository of scientific articles required processing and extracting text from PDF files at scale.
• HTML extraction from WARC: For the pseudocrawl sources, new tools extracted text from HTML in Common Crawl WARC files. The paper notes these tools were "made available on the main data preparation repository" (Section 3.1.3). Of the 539 websites targeted, usable text was found and extracted from all URLs present—a 100% success rate for URL-level extraction, though not necessarily complete coverage of each website.

Quality filtering: "written by humans for humans." The filtering philosophy is distinctive: "high-quality text is defined as 'written by humans for humans', without distinction of content... or a priori judgments of grammaticality" (Section 3.1.3). This means the filters aim to remove non-natural-language text (preprocessing errors, SEO spam, pornographic spam) without making judgments about what topics or writing styles are valuable. Content selection is left to the human source selection process, which the paper positions as "more accountable."

The filtering used a set of quality indicators whose parameters were adapted per-language and per-source through two mechanisms:

1. Per-language threshold and term list selection: "parameters such as the thresholds and supporting term lists were selected individually for each language by fluent speakers" (Section 3.1.3). A threshold that correctly identifies spam in English might incorrectly filter legitimate text in another language with different structural properties.

2. Per-source indicator selection: "we manually went through each individual source to identify which indicators were most likely to identify non-natural language" (Section 3.1.3). A filtering rule appropriate for web-crawled text might be inappropriate for a curated scientific article corpus.

Both processes were supported by visualization tools to inspect their impact before applying them—a practical mechanism for combining automated filtering with human judgment.

Deduplication and PII redaction. Two deduplication steps removed near-duplicate documents. For the OSCAR portion specifically—"deemed to be the source that presented the highest privacy risks"—regex-based redaction targeted Personally Identifiable Information such as social security numbers (Section 3.1.3). The paper acknowledges this approach has "some false positives" but applied it because the privacy risk from OSCAR's indiscriminate web crawl was judged to outweigh the cost of over-redaction. Only OSCAR received this treatment, not the curated sources, reflecting the principle of per-source handling based on risk assessment.

Final scale. Table 1 provides exact byte counts for each language. The distribution is extremely skewed: English dominates at 484.95 GB (approximately 30% of the corpus), followed by Simplified Chinese at 261.02 GB, French at 208.24 GB, Spanish at 175.10 GB, and programming languages collectively at 174.70 GB. At the other extreme, languages like chiShona (6.6 MB), Xitsonga (0.7 MB), and Fon (2.5 MB) are present at sizes four to five orders of magnitude smaller. Figure 3 visualizes this skew: the Indo-European and Sino-Tibetan families "overwhelm the plot with a combined total of 1321.89 GB," while the Niger-Congo family subset occupies only 0.4 GB. This skew is not presented as a success—it reflects the reality of available data and the project's decision to include low-resource languages despite their tiny representation, accepting the tension between symbolic inclusion and practical impact.

---

3.4.3 Prompted Dataset Construction (xP3)

In parallel with the pretraining corpus, the project constructed xP3 (Cross-lingual Public Pool of Prompts), a collection of prompted datasets for multitask finetuning. This is the data foundation for BLOOMZ, the instruction-tuned variant.

Motivation. The prior BigScience project T0 (Sanh et al., 2022) had demonstrated that "language models finetuned on a multitask mixture of prompted datasets have strong zero-shot task generalization abilities" and can "outperform language models that are an order of magnitude larger but did not undergo such finetuning" (Section 3.1.4). T0 was English-only. xP3 extends this to the multilingual setting.

Construction process. xP3 was built by extending P3 (Public Pool of Prompts), the English-only collection created for T0 through hackathons where participants wrote "2000+ prompts for 170+ datasets" (Section 3.1.4). For xP3, new datasets in languages other than English and new tasks (notably translation) were added. The result covers "83 datasets covering 46 languages and 16 tasks." Crucially, Figure 4 shows that xP3's language distribution "closely follows ROOTS"—the prompted dataset mirrors the pretraining corpus's language distribution, ensuring that languages well-represented in pretraining receive corresponding representation in finetuning.

Prompt collection mechanics. Prompts were created using PromptSource (Bach et al., 2022), an open-source toolkit developed within BigScience specifically for this purpose. Each prompt is a template with placeholders that get filled with dataset-specific content. The prompts include metadata about input and output languages. For tasks originally in non-English languages, prompts were written natively in those languages. To study the impact of multilingual prompts, a parallel collection called xP3mt was created by machine-translating English prompts into the respective dataset languages.

Task coverage. Tasks span both cross-lingual (translation) and monolingual (summarization, question answering, natural language inference) settings. The paper notes that the collection process involved peer review: "multiple peer reviews were performed on each prompt for artifacts and consistency" (Section 4.1.1).

---

3.4.4 Tokenizer Design

The tokenizer is the component that converts raw text into sequences of integer token IDs and back. Its design is particularly consequential for multilingual models because a tokenizer optimized for English can severely segment text in other languages, reducing the model's effective context length and wasting capacity on predictable subword patterns.

Design principles. The tokenizer was designed to satisfy three constraints simultaneously:

1. Lossless encoding: Every possible byte sequence must be encodable without producing unknown tokens. This is achieved through byte-level BPE.
2. Multilingual fertility parity: The tokenizer should not segment text in any supported language substantially worse than a monolingual tokenizer designed for that language. The quantitative target was "not to degrade the fertility on each language by more than 10 percentage points" (Section 3.3).
3. Hardware efficiency: The vocabulary size must be divisible by 128 (for GPU efficiency) and by 4 (for Tensor Parallelism compatibility).

Byte-level BPE. The tokenizer uses the Byte Pair Encoding algorithm (Gage, 1994) starting from bytes as the base units rather than characters. This guarantees lossless encoding because "all 256 bytes can be contained in the vocabulary" (Section 3.3). If the tokenizer encounters a byte sequence it has never seen—which happens inevitably in multilingual settings with diverse scripts—it can fall back to representing it as individual bytes rather than producing an unknown token. Byte-level BPE also "maximizes vocabulary sharing between languages" (Section 3.3, citing Wang et al., 2020) because bytes are universal across languages even when characters differ.

Pre-tokenization. The pre-tokenizer splits text using the regex pattern [^(\S+\s*|\S+)] (Section 3.3). This pattern "splits words apart while preserving all the characters and in particular the sequences of spaces and line breaks that are crucial for programming languages." The paper explicitly notes what was not done: "We do not use English-centric splits common in other tokenizers (e.g. splitting around 'nt or 'll). We also didn't use splits on numbers and digits, which caused issues in Arabic and code." These negative design choices—decisions to not apply common heuristics—were informed by the multilingual nature of the corpus and the inclusion of programming languages.

No normalization. "No normalization of the text was performed in order to have the most general model possible" (Section 3.3). The paper tested Unicode normalization (NFKC) and found it "did not reduce the fertility by more than 0.8% on all the languages considered but came at the cost of making the model less general; for example, causing 2² and 22 to be encoded in the same way." This tradeoff—slightly worse fertility for significantly better preservation of information—was resolved in favor of generality.

Vocabulary size. Experiments compared 150k and 250k vocabulary sizes "to make comparisons with existing multilingual modeling literature easier." The final size was 250,680 tokens, with 200 tokens reserved "for possible future applications such as removing private information using placeholder tokens." The vocabulary size directly determines the embedding matrix dimensions: with hidden dimension 14336, the embedding layer contains 250,680 × 14336 ≈ 3.59 billion parameters, a substantial fraction of the model's total parameters.

Training data for the tokenizer. An initial version used a non-deduplicated subset of ROOTS, but qualitative inspection revealed problems: "we found entire URLs stored as tokens caused by several documents containing a high number of duplicates" (Section 3.3). This motivated deduplicating lines in the tokenizer training data and applying the same per-language sampling ratios as the full training corpus.

Validation via fertility. Table 2 shows fertility comparisons between BLOOM's tokenizer and monolingual tokenizers on Universal Dependencies treebanks. Fertility is defined as the number of subword tokens produced per word. For the languages tested:

• French: BLOOM 1.17 vs. CamemBERT 1.30 (−11%)
• English: BLOOM 1.15 vs. GPT-2 1.15 (+0%)
• Spanish: BLOOM 1.16 vs. DeepESP/gpt2-spanish 1.12 (+3%)
• Chinese: BLOOM 1.58 vs. bert-base-chinese 1.50 (+5%)
• Hindi: BLOOM 1.18 vs. monsoon-nlp/hindi-bert 1.07 (+9%)
• Arabic: BLOOM 1.34 vs. Arabic BERT 1.16 (+13%)

Only Arabic exceeds the 10-percentage-point degradation target (13%), and only slightly. The Chinese fertility being higher than the monolingual tokenizer reflects the challenge of tokenizing logographic scripts with a multilingual vocabulary—the tokenizer must represent thousands of Chinese characters, and since the vocabulary budget is shared across all languages, each character receives fewer dedicated tokens than in a Chinese-only tokenizer.

---

3.4.5 Model Architecture and Pretraining Objective

The architecture design was guided by an unusual methodology: rather than exactly replicating an existing large model or exhaustively exploring the design space, the team conducted systematic ablation experiments at smaller scales (1.3B and 6.7B parameters) and extrapolated findings to the 176B scale.

Design space and experimental design. The team elected to focus on "model families that have been shown to scale well, and that have reasonable support in publicly available tools and codebases" (Section 3.2.1). This practical constraint excludes architectures that might be theoretically superior but lack mature GPU-accelerated implementations suitable for training at 176B scale. Specifically:

• Mixture-of-Experts (MoE) was excluded "due to a lack of widely used GPU-based codebases suitable for training them at scale" (Section 3.2.1). At the time of design (2021), MoE training infrastructure was immature; the paper notes this has since changed with Switch Transformers (Fedus et al., 2022) and other work.

• State-space models were excluded because they "consistently underperformed in natural language tasks" at the time (Gu et al., 2021), though the paper notes subsequent progress with H3 (Fu et al., 2023).

Ablations were conducted at two scales: "the 6.7B parameter scale for the pretraining objective ablations" (Wang et al., 2022a) and "the 1.3B scale for the rest including position embeddings, activations, and layer normalization" (Le Scao et al., 2022). The paper acknowledges a potential concern: Dettmers et al. (2022) found a phase transition above 6.7B where "outlier features" emerge, raising the question of whether 1.3B-scale results extrapolate to 176B.

Evaluation methodology for ablations. Ablation decisions were based on zero-shot generalization, not finetuning performance, because "finetuning a 100B+ model is unwieldy" and the primary use case for models at this scale is few-shot or zero-shot adaptation (Section 3.2.1). Two task aggregates were used: 29 tasks from the EleutherAI Language Model Evaluation Harness (EAI-Eval) and 9 tasks from T0-Eval. The paper notes substantial overlap—"only one task from T0-Eval (StoryCloze) is not in EAI-Eval, although all prompts between the two are different"—and that the aggregates "share 17 of the 31 tasks of the evaluation of GPT-3 (Brown et al., 2020)."

Causal decoder-only architecture. The architecture ablation (Wang et al., 2022a) compared encoder-decoder and decoder-only architectures with causal, prefix, and masked language modeling objectives. The finding: "immediately after pretraining, causal decoder-only models performed best" (Section 3.2.2). This validates the choice made by GPT-3, Gopher, PaLM, and OPT—the field had converged on this architecture based on precedent, and the ablation provides empirical justification. The paper further notes that decoder-only models "can be more efficiently adapted after pretrained to a non-causal architecture and objective," citing subsequent work by Tay et al. (2022) that confirmed this finding.

The autoregressive objective. The model is trained with the standard causal language modeling objective. For a sequence of tokens $x = (x_1, ..., x_T)$, the model computes:

$p(x) = \prod_{t=1}^{T} p(x_t | x_{<t})$

where $x_t$ is the token at position $t$ and $x_{<t}$ is the sequence of all tokens preceding position $t$.

What it computes: the joint probability of the entire sequence, factorized as the product of conditional probabilities of each token given all previous tokens. At each position, the model takes the preceding context and outputs a probability distribution over the entire vocabulary; the probability assigned to the actual next token is $p(x_t | x_{<t})$. The training loss is the negative log-likelihood of the observed tokens under the model's predicted distribution.

Why this form: autoregressive factorization is the natural choice for text generation—you can sample from the model one token at a time, conditioning each new token on the previously generated ones. Masked language modeling (used in BERT) is better suited for understanding tasks where you have access to both left and right context, but cannot generate text left-to-right without iterative refinement. Encoder-decoder architectures (used in T5) separate understanding and generation into different components, which helps for finetuning on specific tasks but the ablation found this advantage does not extend to zero-shot settings.

ALiBi positional embeddings. Instead of adding positional information to token embeddings (learned absolute positions, as in the original Transformer) or modifying the attention computation with rotary embeddings (Su et al., 2021), BLOOM uses ALiBi (Attention with Linear Biases, Press et al., 2021). ALiBi "directly attenuates the attention scores based on how far away the keys and queries are" (Section 3.2.3). Specifically, before the softmax in each attention head, a bias is subtracted from the attention scores that grows linearly with the distance between the query and key positions. The slope for head $i$ out of $n$ heads is $2^{-8i/n}$ (Figure 5). This means:

• Head 1 (smallest slope) can attend to nearly the full context equally.
• Head $n$ (largest slope) is strongly biased toward very recent tokens.
• Intermediate heads have intermediate ranges, creating a spectrum of receptive fields.

Why ALiBi over alternatives: The paper reports ALiBi "led to smoother training and better downstream performance even at the original sequence length—outperforming both learned and rotary embeddings" (Section 3.2.3). ALiBi was initially motivated by length extrapolation (training on short sequences, testing on longer ones), but the BLOOM experiments found benefits even without extrapolation. A potential explanation: ALiBi provides an inductive bias toward locality (recent tokens matter more) that helps training converge faster, without the capacity cost of learned position embeddings or the complexity of rotary embeddings.

Embedding LayerNorm. BLOOM includes an additional layer normalization immediately after the token embedding layer—before the first Transformer block. The motivation came from "preliminary experiments training a 104B parameters model" (Section 3.2.3) where the bitsandbytes library's StableEmbedding layer recommended this practice (Dettmers et al., 2022). The embedding LayerNorm "significantly improved training stability" in those experiments.

The tension with zero-shot performance. The paper reveals an important negative result: "we also found it penalizes zero-shot generalization in Le Scao et al. (2022)" (Section 3.2.3). Despite this penalty, the team included the embedding LayerNorm "to avoid training instabilities." This is a revealing tradeoff: training stability was prioritized over zero-shot performance because an unstable training run that diverges produces no usable model at all, while a model that trains successfully but has slightly worse zero-shot performance can still be useful (and can potentially be improved through finetuning).

The float16 hypothesis. The paper speculates that the embedding LayerNorm might not have been necessary in the final training: "the preliminary 104B experiments were conducted in float16, while the final training was in bfloat16. Since then, float16 has been attributed as being responsible for many of the observed instabilities in training LLMs (Zhang et al., 2022; Zeng et al., 2022). It is possible that bfloat16 alleviates the need for the embedding LayerNorm" (Section 3.2.3). This is presented as a hypothesis, not a finding—the team didn't run the ablation of removing the LayerNorm for the final bfloat16 training, so they cannot confirm whether the stability benefit was specific to float16.

Standard architectural choices. Several components follow established practice without modification: GELU activations (Gaussian Error Linear Unit, a smooth approximation to ReLU that performs better in Transformers), tied embedding weights (the input embedding matrix and output projection share parameters, reducing total parameter count and providing a useful regularization effect), and a sequence length of 2048 tokens.

Model scale variants. Table 3 shows six size variants trained, ranging from BLOOM-560M (24 layers, 1024 hidden, 16 heads) to BLOOM-176B (70 layers, 14336 hidden, 112 heads). The intermediate sizes don't follow a simple scaling law: the 3B and 7.1B models use 30 layers instead of following a power-law depth curve, because the layer counts were chosen to "fit the models more easily on our training setup" (Section 3.5). This is a practical constraint rather than a principled scaling decision.

---

3.4.6 Distributed Training Engineering

Training a 176B-parameter model requires distributing computation across many GPUs because the model, optimizer states, and activations cannot fit in a single GPU's memory. BLOOM's training infrastructure combines four parallelism strategies.

Hardware configuration. Training ran on the Jean Zay supercomputer at IDRIS/CNRS, using 48 nodes each with "8 NVIDIA A100 80GB GPUs (a total of 384 GPUs)" plus 4 spare nodes for failover (Section 3.4.1). Nodes had "2x AMD EPYC 7543 32-Core CPUs and 512 GB of RAM." Intra-node communication used 4 NVLink GPU-to-GPU interconnects; inter-node communication used "4 Omni-Path 100 Gbps links per node, arranged in an enhanced hypercube 8D global topology."

Software framework. Training used Megatron-DeepSpeed, a combination of two frameworks: "Megatron-LM provides the Transformer implementation, tensor parallelism, and data loading primitives, whereas DeepSpeed provides the ZeRO optimizer, model pipelining, and general distributed training components" (Section 3.4.2).

3D parallelism. The core enabling technique is the fusion of three complementary parallelism strategies (illustrated in Figure 6):

• Data parallelism (DP): "replicates the model multiple times, with each replica placed on a different device and fed a slice of the data. The processing is done in parallel and all model replicas are synchronized at the end of each training step" (Section 3.4.2). This is the simplest form of parallelism and requires the entire model to fit on each GPU, which is impossible for 176B parameters.

• Tensor parallelism (TP): "partitions individual layers of the model across multiple devices. This way, instead of having the whole activation or gradient tensor reside on a single GPU, we place shards of this tensor on separate GPUs" (Section 3.4.2). For example, the 14336-dimensional hidden state can be split across 4 GPUs so each holds a 3584-dimensional slice, and attention heads can be distributed so each GPU computes a subset of the 112 heads.

• Pipeline parallelism (PP): "splits up the model's layers across multiple GPUs, so that only a fraction of the layers of the model are placed on each GPU" (Section 3.4.2). With 70 layers, a pipeline of 8 GPUs would place roughly 9 layers on each GPU. The challenge is that GPUs early in the pipeline must wait for later GPUs to finish before the next microbatch can start, creating "bubbles" of idle time.

ZeRO stage 1. The Zero Redundancy Optimizer (ZeRO; Rajbhandari et al., 2020) at stage 1 shards only the optimizer states across data-parallel processes. Each process holds the full model parameters and gradients but only a fraction of the Adam optimizer states (first and second moment estimates). This reduces memory consumption without adding communication overhead for parameters and gradients.

GPU utilization. The combination achieved "156 TFLOPs in our fastest configuration with A100 GPUs, attaining our objective of half of the theoretical peak performance of 312 TFLOPs (in float32 or bfloat16)" (Section 3.4.2). Half of theoretical peak is a standard target for large-scale distributed training—above this is considered excellent engineering.

bfloat16 mixed precision. Earlier experiments with 104B-parameter models on V100 GPUs using IEEE float16 "caused irreversible training divergences" (Section 3.4.3). The A100 GPUs used for final training support bfloat16, which "has the same dynamic range as float32" but lower precision. Mixed-precision training (Micikevicius et al., 2018) keeps "certain precision-sensitive operations such as gradient accumulation and softmax in float32 precision and the rest of operations in lower precision." The switch to bfloat16 "proved to solve the instability problem."

Fused CUDA kernels. Kernel fusion optimizes GPU computations by "performing several consecutive operations in only one kernel call" (Section 3.4.4). Normally, each operation (e.g., a matrix multiply, a bias addition, an activation function) requires loading data from GPU VRAM into registers, computing the result, and storing it back. By fusing operations, intermediate results stay in registers, avoiding expensive memory transfers. The paper cites examples: "an optimized kernel to perform LayerNorm, as well as kernels to fuse various combinations of the scaling, masking, and softmax operations. The addition of a bias term is also fused with the GeLU activation." For the bias+GeLU fusion specifically: "adding the bias term in the GeLU operation adds no additional time, as the operation is memory-bound: the additional computation is negligible compared to data transfers between GPU VRAM and registers, so fusing both operations essentially halves their runtime."

Scaling challenges. Two final changes were needed to reach 384 GPUs: "disabling asynchronous CUDA kernel launches (for ease of debugging and to prevent deadlocks) and splitting parameter groups into smaller subgroups (to avoid excessive CPU memory allocations)" (Section 3.4.5).

Reliability. Hardware failures were routine: "on average, 1–2 GPU failures occurred each week" (Section 3.4.5). Backup nodes and automatic failover handled these without significant throughput impact. Checkpoints were saved every three hours. A PyTorch deadlock bug and disk space issues caused "5–10h downtimes." Overall, the paper characterizes training as relatively smooth: "human intervention was less necessary than in comparable projects," citing OPT's more troubled training experience (Zhang et al., 2022) and noting BLOOM had "only one loss spike, which the model swiftly recovered from."

---

3.4.7 Training Hyperparameters and Carbon Footprint

Pretraining hyperparameters (Table 3). Six model sizes were trained with size-dependent hyperparameters:

All models use: GELU activations, ALiBi positional embeddings, tied embeddings, Adam optimizer with β = (0.9, 0.95), weight decay 0.1, gradient clipping at 1.0, no dropout, cosine learning rate decay over 410B tokens, and 375M warmup tokens. Embedding parameter counts are "larger for BLOOM owing to the larger multilingual vocabulary, but scaling literature discounts embedding operations (Kaplan et al., 2020)."

Learning rate schedule details. "We use a cosine learning rate decay schedule over 410B tokens, taken as an upper bound for the length of training if compute permitted, and warmup for 375M tokens" (Section 3.5). The minimum learning rate for the 176B model is 6e-6 (10% of the peak 6e-5). The paper notes: "As warmup tokens + decay tokens were larger than the total number of tokens, the end of learning rate decay was never reached"—the cosine schedule was still decreasing when training stopped.

The Hoffmann correction. The original plan was to train on 341B tokens (one epoch through ROOTS). However, "in light of revised scaling laws published during training (Hoffmann et al., 2022), we decided to train the large models for an additional 25 billion tokens on repeated data" (Section 3.5). The Chinchilla scaling laws showed that for a 176B-parameter model, the compute-optimal training data quantity is substantially larger than 341B tokens, so extending training with repeated data was expected to improve performance despite the data reuse.

Multitask finetuning hyperparameters. BLOOMZ models use: global batch size 2048 (multiplied by 4 for small variants "to increase throughput"), learning rate 2e-5 (determined by "doubling the minimum learning rate of the respective pretrained model and then rounding"), constant learning rate (no decay), weight decay 1e-4, no warmup. Finetuning runs for 13 billion tokens, but "performance plateaued after 1–6 billion tokens" and the best checkpoint is selected by validation loss.

Contrastive finetuning. Two embedding models were produced using the SGPT Bi-Encoder recipe (Muennighoff, 2022): SGPT-BLOOM-7.1B-msmarco for multilingual information retrieval and SGPT-BLOOM-1.7B-nli for multilingual semantic textual similarity. These are separate from the main BLOOMZ finetuning and produce models that output fixed-size text embeddings rather than generating text.

Carbon footprint analysis. The paper's carbon accounting is unusually thorough, using Life Cycle Assessment (LCA) methodology that considers "emissions of equipment manufacturing, intermediate model training, and deployment" (Section 3.5.1), not just training energy consumption. The breakdown:

• Total emissions: approximately 81 tons CO2eq.
• Equipment manufacturing: 11 tons (14%).
• Energy consumed during training: 25 tons (30%).
• Idle consumption of equipment and computing cluster: 45 tons (55%).

The dominance of idle consumption (55%) is notable—it means more than half the carbon cost came from the supercomputer being powered on even when not actively training, reflecting the infrastructure overhead of large-scale computing.

Comparison with other models (Table 4). The energy consumption of BLOOM training (433 MWh) is higher than OPT (324 MWh), but emissions are much lower (25 tons vs. 70 tons) because of the carbon intensity of the electricity grid: France's grid used by Jean Zay emits 57 gCO2eq/kWh, compared to 231 gCO2eq/kWh for the grid that trained OPT. The paper is explicit: "France's national energy grid (which is used by Jean Zay) is largely powered by nuclear energy, which is low-carbon compared to grids powered by energy sources such as coal and natural gas." This is not presented as an endorsement of nuclear energy but as an empirical fact about the carbon intensity of the available electricity.

Broader compute context. The final BLOOM training consumed 1,082,990 compute hours (Section 3.4.1), representing "approximately 37% of the overall emissions" of the BigScience project, "with other processes such as intermediate training runs and model evaluation adding up to the other 63%" (Section 3.5.1). This means the total project emissions were approximately 81 / 0.37 ≈ 219 tons CO2eq—substantially more than the training alone. The paper also estimates deployment emissions: "real-time deployment of the model on a GCP instance with 16 GPUs running in the us-central1 region results in approximately 20 kg of CO2eq emitted per day of deployment (or 0.83 kg per hour)" (Section 3.5.1).

---

3.4.8 Release Framework: Model Card and Licensing

The final technical component is not computational but structural: the mechanisms by which BLOOM is made available to the world. These are treated as design decisions that shape how the model can and cannot be used.

Model Card. Following Mitchell et al. (2019), BLOOM was released with a detailed Model Card describing "technical specifications, details on training, intended-use, out-of-scope uses as well as the model's limitations" (Section 3.6). The card was "collaborative, primarily composed 'live' by thinking through and discussing each section, then further dividing into subsections based on the categorizations and distinctions participants naturally ended up creating throughout discussions." This process-oriented description emphasizes that documentation was a collective reasoning exercise, not an after-the-fact formality.

Responsible AI License (RAIL). The license is a distinctive technical contribution. It "separates licensing of the 'source code' and 'model', as referenced by its trained parameters" and "includes detailed definitions of 'use' and 'derived works' of the model to ensure that anticipated downstream use by prompting, finetuning, distillation, use of logits and probability distributions are explicitly identified" (Section 3.6). The separation of code and model weights means the Apache 2.0 open-source license applies to the training code, while the RAIL license with behavioral-use restrictions applies to the trained parameters.

Behavioral-use restrictions. The license "contains 13 behavioral-use restrictions that have been identified based on the intended uses and limitations described in the BLOOM Model Card, as well as the BigScience ethical charter" (Section 3.6). These are not specified in the paper but derive from the project's ethical analysis of harmful use cases that the model could enable. The license "offers the model at no charge and users are free to use the model as long as they comply with the terms."

Why this is a technical contribution. The license design reflects a position in an ongoing debate about open-source AI: unrestricted open access enables beneficial research and applications but also enables harmful uses, while restrictive licensing limits both. The RAIL approach attempts to navigate this by being permissive for most uses while specifically prohibiting a defined set of harmful applications. This is a structural mechanism—it encodes ethical judgments into legally enforceable terms—rather than relying on voluntary compliance or post-hoc moderation of model outputs.

4. Key Insights and Innovations

Innovation 1: Treating the Development Process as a First-Class Scientific Contribution

Most large language model papers are structured as "we built a model, here's the architecture, here are the benchmark numbers." The contribution is the artifact and its performance. BLOOM's paper does something fundamentally different: it treats the organizational, governance, and ethical design decisions as contributions of equal standing to the technical architecture.

This is not a superficial "we care about ethics" statement bolted onto an otherwise standard paper. The paper's structure embodies the claim: Section 2.2 documents the collaboration's organizational design—30 working groups, 1,200+ participants from 38 countries, the Ethical Charter as a binding governance document—with the same level of detail that Section 3.2 devotes to the Transformer architecture. Section 3.1.1 describes the data governance framework (structured agreements between data providers and hosts, source-level traceability, composite release) not as a constraint to be worked around but as a positive design feature. Section 3.6 presents the Responsible AI License with its 13 behavioral-use restrictions and the separation of code and model weight licensing as a technical innovation—a legal mechanism that encodes ethical judgments into enforceable terms. Section 3.5.1's carbon accounting uses Life Cycle Assessment methodology and attributes 55% of emissions to idle infrastructure consumption, an analytical finding with direct implications for how academic compute grants should be structured.

What makes this distinctive at the idea level is the paper's reframing of what counts as a scientific contribution in ML systems research. Prior work in this space—GPT-3 (Brown et al., 2020), Gopher (Rae et al., 2021), PaLM (Chowdhery et al., 2022), OPT (Zhang et al., 2022)—treats the development process as infrastructure: necessary to produce the artifact, but not itself the subject of scientific inquiry. When these papers discuss data, they describe filtering pipelines and volume statistics; when they discuss ethics, they include a limitations paragraph. BLOOM's paper inverts this: the data governance choices are presented as design decisions requiring justification and empirical validation (fertility metrics for the tokenizer, per-language quality indicators, the tension between the embedding LayerNorm's stability benefit and its zero-shot penalty). The organizational structure is documented so that other groups can replicate or modify it.

This is a fundamental shift in the genre of large-scale ML paper, not an incremental refinement. It responds to a specific diagnostic problem the paper identifies in Section 2.1: that "the concentration of resources within a handful of (typically industrial) institutions with primarily technical expertise hinders prospects for an inclusive, collaborative, and reliable governance of the technology." The paper's thesis, demonstrated rather than argued, is that you can build a competitive 176B-parameter model and simultaneously document how the governance choices were made, who made them, and what frameworks constrained them—and that this documentation is itself part of what makes the model scientifically valuable. The evidence for this innovation is not a single figure but the paper's entire organizational structure: the 30 working groups shown in Figure 1, the governance mechanisms detailed in Section 3.1.1, the 13 behavioral-use restrictions encoded in the RAIL license (Section 3.6), and the contributor attribution in Section 6 that maps individuals to specific roles in Dataset, Tokenization, Architecture, Engineering, Evaluation, Broader Impacts, and Organization.

Innovation 2: Difficulty-Conditioned Data Curation as an Alternative to Scale-at-All-Costs Corpus Construction

The dominant paradigm for building LLM training datasets, established by GPT-3 and widely replicated, is to cast the widest possible net—crawl the web, apply automated filters, and rely on scale to overwhelm noise. The paper's critique of this approach (Section 3.1) is specific and evidence-backed: blocklists suppress LGBTQ+ and AAE content (Dodge et al., 2021), Reddit-based quality signals embed US-centric values (Johnson et al., 2022), and automated filtering makes post-hoc provenance tracing impossible because "the provenance and authorship of individual items is usually lost in the process."

BLOOM's innovation is not just "do better data curation"—many papers advocate for that. It's the specific difficulty-conditioned sourcing strategy that the paper develops in response. Every language in ROOTS required at least three fluent speakers to commit to source selection and preprocessing guidance (Section 3.1.2). This gates language inclusion on human expertise availability, not data availability. It means the corpus covers 46 languages rather than the 100+ that could have been included through automatic language identification—a deliberate choice to prioritize per-language quality over language count. The quality filtering uses per-language thresholds selected by fluent speakers and per-source indicator selection through manual inspection (Section 3.1.3). The composite release strategy (Section 3.1.1) means 225 of 498 component datasets are not publicly distributable—a decision that limits replicability but respects agreements with data custodians.

What makes this intellectually distinctive is that it rejects the scale-as-proxy-for-quality assumption that underlies most web-crawled corpus construction. The assumption in prior work is that with enough data, the signal from good text overwhelms the noise from bad text, making careful curation unnecessary (or at least not worth the cost in reduced dataset size). BLOOM's approach asserts the opposite: that careful curation produces a different kind of corpus—one with documented provenance, per-source accountability, and structural mechanisms for respecting data subject rights—even if the total volume is smaller. The 1.61 TB ROOTS corpus is significantly smaller than what indiscriminate crawling could produce, but the paper treats this not as a limitation to apologize for but as a feature of the design philosophy.

This is a fundamental reframing of the data collection problem, not an incremental improvement. Table 1 and Figure 3 provide the evidence: the language distribution is extremely skewed (English at 485 GB, Fon at 2.5 MB), which the paper presents transparently rather than trying to upsample or artificially balance. The skew is a consequence of the curation philosophy—you can only include what fluent speakers can vouch for—and the paper treats this honest accounting as preferable to the opaque distribution of web-crawled corpora where the provenance of any individual document is unknown.

Innovation 3: Recasting the Architecture Search as a Systematic Zero-Shot Ablation Study

By 2021, the field had converged on decoder-only Transformers for the largest language models—GPT-3, Gopher, PaLM all used this architecture—but as the paper notes in Section 3.2.2, "the literature was lacking a systematic evaluation of the zero-shot generalization capabilities of different architectures and pretraining objectives." The dominant approach was to follow precedent: if GPT-3 used a causal decoder-only architecture and scaled well, subsequent models did the same. Encoder-decoder models, despite their strong finetuning performance (Raffel et al., 2020), were largely abandoned at scale without clear evidence that the abandonment was justified for zero-shot settings.

The paper's innovation is the architectural ablation methodology described in Section 3.2.1: conduct systematic comparisons at 1.3B and 6.7B parameter scales using zero-shot generalization aggregates (EAI-Eval and T0-Eval) as the evaluation criterion, then extrapolate to 176B. This methodology produced specific, justified decisions rather than precedent-following:

• Causal decoder-only architecture outperforms encoder-decoder for zero-shot (Wang et al., 2022a), validating the field's convergence.
• ALiBi positional embeddings outperform both learned and rotary embeddings "even at the original sequence length" (Section 3.2.3)—a finding that contradicted the common assumption that ALiBi's value was primarily for length extrapolation.
• Embedding LayerNorm improves training stability but penalizes zero-shot generalization (Section 3.2.3)—a negative result that revealed a previously undocumented tradeoff.

What's distinctive here is not the individual findings but the explicit valuation of zero-shot performance as the architecture selection criterion. Prior work at this scale (GPT-3, Gopher, PaLM) had objectives that included few-shot learning, but the architecture itself was typically inherited rather than optimized for this setting. BLOOM's ablation work provides empirical justification for what was previously conventional wisdom, and in the process identifies non-obvious interactions (like the stability-vs-performance tradeoff of the embedding LayerNorm) that have design implications for future models.

This is an incremental methodological advance rather than a fundamental break—the practice of architecture ablation at smaller scales is well-established, and the paper acknowledges the limitation that extrapolation from 1.3B to 176B may miss phenomena like the "outlier features" phase transition identified by Dettmers et al. (2022). But within the context of open-access LLM development, where replicating corporate architectural choices without justification would undermine the project's commitment to transparency, the systematic ablation serves an important legitimizing function: BLOOM looks the way it does because the experiments said so, not because GPT-3 looked that way.

Innovation 4: Demonstrating That Open-Access, Collaboratively-Governed Models Can Be Competitive with Industrial Systems

This is the paper's most straightforward but perhaps most consequential claim: a model built through an open research collaboration with 1,200+ participants, trained on a documented corpus with per-source governance, and released under a Responsible AI License can achieve results competitive with similarly-sized industrial models built through proprietary processes with substantially larger budgets.

The evidence is distributed across Section 4:

• One-shot SuperGLUE (Figure 7): BLOOM matches or exceeds OPT-175B on Ax-b, CB, WSC, and WiC, and matches it on the remaining tasks. This is notable because OPT was trained primarily on English data, while BLOOM's training data is only ~30% English—suggesting the multilingual training does not systematically degrade English task performance.
• Machine translation (Table 8): On Flores-101 high-resource pairs, BLOOM in one-shot settings achieves results "comparable and sometimes better" than the supervised M2M-100 model with 615M parameters, and results "comparable in many cases to those of AlexaTM" in 32-shot settings—despite never being trained specifically for translation.
• Code generation (Table 9): BLOOM's 15.52% pass@1 on HumanEval is competitive with similarly-trained GPT models (GPT-J 6B at 11.62%, GPT-NeoX 20B at 15.4%) and follows the same scaling trend.
• Carbon efficiency (Table 4): BLOOM's 25 tons CO2eq for training is approximately 1/3 of OPT-175B's 70 tons, demonstrating that careful infrastructure choices (training on a low-carbon grid) can decouple model scale from emissions.

The conceptual contribution is not "we built a good model"—many papers claim that. It's the existence proof that the open, collaborative, ethically-governed development model does not inherently produce inferior results. This matters because the dominant narrative in the field had been that cutting-edge LLM development requires the concentrated resources, proprietary data pipelines, and closed development processes of major industrial labs. BLOOM demonstrates that an alternative path exists and can reach comparable heights, at least for the model sizes and tasks evaluated.

The limitation is equally important: the paper does not claim BLOOM is state-of-the-art. Figure 10 (HELM benchmark) shows BLOOM "roughly on par in accuracy with previous-generation English-only models, such as GPT3-davinci v1 and J1-Grande v1, but behind more recent monolingual models such as InstructGPT davinci v2, Turing NLG v2, Anthropic-LM v4-s3, or OPT." The competitive claim is relative to similarly-sized models, not to the frontier. This is an incremental but practically significant advance—closing the gap between open and proprietary models at this scale by roughly an order of magnitude compared to the previous largest open model (GPT-NeoX-20B at 20B parameters).

Innovation 5: Treating Multilingual Performance as a Diagnostic for Understanding Model Competence, Not Just a Benchmark

The paper's evaluation strategy for multilingual capabilities (Sections 4.3, 4.4, 4.9) departs from the standard practice of reporting aggregate metrics across languages. Instead, the evaluation is designed to reveal where and why multilingual performance breaks down, using language-pair selection as a diagnostic tool rather than a reporting requirement.

Consider the Flores-101 machine translation results in Table 8, which are deliberately stratified by resource level and language relationship:

• Between Romance languages (Table 8b): BLOOM translates well even from Galician (glg), "a language not included in the training data, but which shares many similarities with the other Romance languages, in particular with Portuguese." This demonstrates genuine cross-lingual transfer: the model has not seen Galician text but can translate it by leveraging related languages.
• High-resource to mid-resource (Table 8d): Translation from English to Hindi (27.2 BLEU), Indonesian (39.0), and Vietnamese (28.5) is "good across the board... suggesting BLOOM's good multilingual capacity, even across scripts."
• Between low-resource languages (Table 8a): Translation between Swahili and Yoruba—both present but severely under-represented in training, with <50k tokens each—"is very poor." The model cannot bridge these languages because neither has sufficient representation for the model to learn robust representations, and they are not closely related enough for transfer to help.

This stratified evaluation creates a difficulty taxonomy that the search and revision scaling literature (as studied in the reference example paper) would recognize: "easy" problems (high-resource, related languages) benefit from the model's existing knowledge; "medium" problems (high-to-mid resource, different scripts) stress cross-lingual transfer; "hard" problems (low-resource, unrelated languages) lie outside the model's capability frontier. The paper doesn't use this taxonomy explicitly, but the evaluation design enables it.

What makes this conceptually distinctive is the shift from "how good is the model at multilingual tasks" to "what does multilingual performance reveal about the model's internal representations." Section 4.9's probing experiments on 17 languages and 38 morphosyntactic features directly investigate this: they find that BLOOM-1B7 "performs on par or better than BLOOM" on probing tasks (Table 12)—a counterintuitive result suggesting that larger scale does not uniformly improve grammatical representation. The correlation analysis (Table 13) finds that BLOOM-1B7's performance is highly correlated with pretraining dataset size (p < 0.01), while BLOOM-176B's is not—suggesting the larger model develops more robust representations that depend less on raw data quantity. These are findings about how multilingual competence works in large language models, not just performance numbers.

This is an incremental methodological contribution—careful task stratification and probing are established techniques—but applied at a scale and language diversity that prior work had not attempted. The 46-language evaluation scope, combined with the transparency about where the model fails, makes this a template for how multilingual LLM evaluation should be conducted rather than a claim of universal competence.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation spans multiple benchmarks rather than a single dataset: SuperGLUE (Wang et al., 2019) for English classification tasks (specifically Ax-b, Ax-g, BoolQ, CB, WiC, WSC, and RTE—the remaining tasks were excluded because they "require an order of magnitude more compute to run than all of these tasks we consider combined," Section 4.1.3); WMT14 (Bojar et al., 2014) for English↔French and English↔Hindi translation; Flores-101 (Goyal et al., 2022) for multilingual translation across diverse language pairs; DiaBLa (Bawden et al., 2020) for contextual bilingual dialogue translation; WikiLingua (Ladhak et al., 2020) for multilingual summarization in 9 languages (Arabic, English, Spanish, French, Hindi, Indonesian, Portuguese, Vietnamese, Chinese); HumanEval (Chen et al., 2021) for code generation; HELM (Liang et al., 2022) for a comprehensive 5-shot English evaluation across accuracy, calibration, robustness, fairness, bias, and toxicity; and the CrowS-Pairs dataset (Nangia et al., 2020; Névéol et al., 2022) for bias evaluation in English and French. For the multitask finetuning evaluation, held-out tasks include XNLI (cross-lingual natural language inference), XWinograd (multilingual coreference resolution), XCOPA (cross-lingual causal reasoning), and XStoryCloze (multilingual story completion). For probing experiments (Section 4.9), 17 languages from Universal Dependencies treebanks are used, covering 38 morphosyntactic features.

• Base model(s). Six size variants of BLOOM, from 560M to 176B parameters (Table 3). The 176B model is the primary subject; smaller variants are used for scaling analysis (Figure 8, Section 4.2) and probing comparisons (BLOOM-1B7 vs. BLOOM-176B, Section 4.9). BLOOMZ variants (multitask finetuned) are produced by finetuning all BLOOM sizes on xP3 (Section 3.5). The choice of six variants enables scaling-law-style analysis: does performance improve monotonically with model size, and if so, at what rate? For the contrastive finetuning experiments, BLOOM-1.7B and BLOOM-7.1B serve as base models for producing SGPT-BLOOM embedding models.

• Metrics. Task-specific metrics are used throughout: accuracy for SuperGLUE classification tasks (computed as the fraction of examples where the model's highest-likelihood answer among candidate label strings matches the ground truth), BLEU (Papineni et al., 2002) for machine translation using sacrebleu (Post, 2018) with dataset-specific tokenization (default for WMT and DiaBLa, spm-flores-101 for Flores), ROUGE-2 and ROUGE-L (Lin, 2004) plus Levenshtein distance for summarization (with a SentencePiece tokenizer built from Flores-101 to avoid English-centric tokenization), pass@k for code generation on HumanEval (the probability that at least one of k sampled solutions passes all unit tests), F1 weighted score for probing classifiers (Section 4.9.1, chosen "due to target class imbalance for most probing tasks"), and accuracy for CrowS-Pairs bias evaluation (the fraction of cases where the model prefers the non-stereotyped sentence over the stereotyped one, with 0.50 indicating no systematic preference). For HELM (Section 4.6, Figure 10), the benchmark provides its own aggregated metrics for accuracy, calibration error, robustness, fairness, bias, and toxicity—the paper reproduces HELM's figures but does not describe metric computation in detail.

• Baselines. Multiple model families are compared depending on task availability: mGPT (Shliazhko et al., 2022) for multilingual models trained on 60 languages; GPT-Neo (Black et al.), GPT-J-6B (Wang and Komatsuzaki, 2021), and GPT-NeoX-20B (Black et al., 2022) for English-focused open-access models; OPT (Zhang et al., 2022) across sizes from 350M to 175B as the primary comparison point since it is similarly-sized and open-access; T0 (Sanh et al., 2022) for multitask prompted finetuning comparisons (monolingual English); XGLM (Lin et al., 2021) for multilingual GPT-style models; M2M-100 (Fan et al., 2021) with 615M parameters for supervised machine translation; AlexaTM (Soltan et al., 2022) for few-shot translation; mTk-Instruct (Wang et al., 2022b) for multitask finetuned multilingual models; Codex (Chen et al., 2021) and GPT-Neo/NeoX/J for code generation; GPT-fr (Simoulin and Crabbé, 2021) for French-specific language modeling; and various embedding models for the MTEB benchmark comparison (multilingual MiniLM, MPNet, LaBSE, ST5-XL, LASER2). On HELM, the comparison set is comprehensive and enumerated in Figure 10, including GPT-3 variants, InstructGPT, Cohere models, Anthropic-LM, YaLM, GLM, TNLG, UL2, and others.

• Generation budget / compute accounting. For most tasks, there is no explicit "generation budget" in the sense of the inference-time compute scaling literature—the models are evaluated in zero-shot and one-shot settings, meaning the model produces one output per example. For generation tasks (MT, summarization, code), decoding uses greedy decoding (not sampling) with generation proceeding until an EOS token or a task-specific stop sequence (\n###\n for one-shot MT). Maximum generation lengths are set per dataset: 64 tokens for WMT14, 512 tokens for Flores-101 and DiaBLa. For HumanEval, pass@k is computed with k ∈ {1, 10, 100}. Compute is not the primary axis of comparison; rather, the paper compares models at fixed parameter scales under identical prompting conditions.

• Cross-validation / statistical protocol. For CrowS-Pairs bias evaluation (Section 4.10), results are averaged over eight runs with different random seeds, and significance is assessed using a one-sample T-test against the null hypothesis of 0.50 accuracy (p < 0.05 indicated with asterisks in Table 14). For probing experiments (Section 4.9), results are "averaged across three runs with different random seeds," and statistical tests include Mann-Whitney U for language script and model version comparisons, ANOVA for language family group comparisons, and Pearson correlation for probing performance vs. dataset size relationships (Table 13). For SuperGLUE prompting (Section 4.2), each task uses "a random sample of five prompts from promptsource" and evaluates all models on that same set. For the multitask finetuning evaluation (Section 4.7, Figure 11), five untuned prompts are evaluated per dataset, with per-prompt results plotted individually. For summarization (Figure 9), per-prompt ROUGE-2 scores are plotted as individual points. The paper does not employ the two-fold cross-validation within difficulty bins that characterizes the reference example's compute-optimal analysis—BLOOM's evaluation is not structured around difficulty-conditional strategy selection, so this protocol is unnecessary. No confidence intervals or standard errors are reported for the main benchmark results; the per-prompt point plots serve as a visual indication of variance.

Main Quantitative Results

4.2 SuperGLUE: Zero-Shot and One-Shot English Task Performance

The headline finding is that BLOOM-176B achieves competitive performance with OPT-175B on English-only classification tasks despite being trained on a multilingual corpus where English constitutes only approximately 30% of the training data. Figure 7 presents zero-shot and one-shot accuracy across seven SuperGLUE tasks for mGPT (1.3B), GPT-J (6B), T0 (11B), OPT (175B), and BLOOM (176B).

In the zero-shot setting (Figure 7, left panel), BLOOM lags behind OPT on most tasks: on BoolQ, OPT achieves higher accuracy (the exact numbers are not legible from the figure but the visual gap is clear for most prompts); on CB, both models perform well above the random baseline of 50% (binary classification) with substantial prompt-dependent variance. On the remaining tasks (Ax-b, Ax-g, WiC, WSC), performance for both models "hovers around chance" when averaged across prompts, with the best individual prompts performing better—a pattern the paper attributes to "statistical variation" rather than genuine task competence (Section 4.2). T0 (11B) is the clear outlier, showing strong zero-shot performance across all tasks due to its multitask prompted finetuning (it is instruction-tuned, making this comparison apples-to-oranges with the pretrained models).

The transition from zero-shot to one-shot (Figure 7, right panel) reveals an asymmetric improvement: "notably, BLOOM sees more of an increase in performance than comparable models when going from zero-shot to one-shot, as it is generally behind OPT in the zero-shot setting but matches or improves on it in the one-shot setting" (Section 4.2). The paper hypothesizes that "a multilingual language model gains more certainty in the language of input and output with a longer context"—the one-shot example provides a stronger signal about the language being used, reducing the ambiguity that a multilingual model faces when presented with a short English prompt in isolation. In the one-shot setting, BLOOM-176B is "ahead of OPT-175B on Ax-b, CB, WSC and WiC, and matches it on the other tasks" (Section 4.2).

Figure 8 presents a scaling analysis comparing OPT and BLOOM model families across sizes from approximately 350M to 175B parameters on one-shot SuperGLUE tasks. Each point represents the average accuracy of a model size on one prompt. The key pattern: "both OPT and BLOOM model families improve very slightly with scale, with only models over 2 billion parameters showing signal, and there is no consistent difference between families across all tasks" (Section 4.2). The log-scale x-axis reveals that improvements are modest—the slope from 1B to 100B+ parameters is shallow, consistent with the "flat scaling" observation that zero-shot and few-shot performance on many NLP tasks improves slowly with model size once a basic competence threshold is reached.

The critical interpretive claim: "suggesting that multilinguality does not limit performance of BLOOM on English-only tasks in the zero-shot setting" (Section 4.2). This is a non-obvious finding—one might expect that allocating model capacity across 46 languages would reduce English performance relative to an English-focused model of the same size. The SuperGLUE results suggest this capacity tradeoff is minimal, at least for the task types and model scale tested.

4.3 Machine Translation

The machine translation evaluation is the most extensive multilingual analysis in the paper, spanning three datasets (WMT14, DiaBLa, Flores-101) and multiple language pairs stratified by resource level and linguistic relationship.

WMT14 (Section 4.3.1, Table 6): BLOOM-176B is evaluated in zero-shot and one-shot settings on English↔French and English↔Hindi using four different prompts (Table 5). The headline numbers in the one-shot setting: for English→French, the best prompt ("version-target") achieves 34.22 BLEU; for French→English, 35.42 BLEU. For comparison, the dedicated supervised M2M-100 model (615M parameters) achieves 43.8 BLEU for English→French and 40.4 for French→English—so BLOOM in a one-shot setting reaches approximately 78–84% of a supervised model's performance without any translation-specific training.

The prompt dependence is stark: the best prompts ("a_good_translation-source+target" and "version-target") consistently outperform the worst ones ("gpt3-target" and "xglm-source+target") by 10–14 BLEU points in the zero-shot setting. For Hindi↔English, performance is substantially lower: the best Hindi→English zero-shot result is 12.05 BLEU (xglm-source+target) and the best one-shot is 25.80 BLEU (version-target). English→Hindi peaks at 14.49 BLEU in one-shot—roughly half the performance of the French pairs, reflecting the greater linguistic distance and lower Hindi representation in ROOTS (24.6 GB vs. French's 208.2 GB).

Two systematic failure modes are observed: "(i) over-generation and (ii) not producing the correct language (an obvious prerequisite for a good translation). Both of these aspects are greatly improved as the number of few-shot examples is increased" (Section 4.3.1). Over-generation means the model continues producing text beyond the translation, often repeating the prompt pattern or generating additional sentence pairs. Wrong-language output means the model produces text in the source language rather than translating—a failure mode specific to multilingual models that must infer the desired output language from context.

DiaBLa (Section 4.3.2, Table 7): This experiment tests whether BLOOM can leverage linguistic context in dialogue—a more nuanced capability than sentence-level translation. Using the "xglm-source+target" prompt in a one-shot setting, the experiment compares two conditions: using a random test set example as the one-shot demonstration vs. using the previous dialogue utterance (providing conversational context). The automatic metrics are inconclusive—"BLEU scores are higher for previous context but COMET scores are lower" (Table 7). However, qualitative inspection reveals that "the model is able to use the context of the 1-shot example to make translation choices" (Section 4.3.2), with detailed examples deferred to Bawden and Yvon (2023). The over-generation problem is severe in this setting: without truncation, BLEU scores are 5.7–12.3 (essentially junk); with a custom truncation function applied post-hoc, scores jump to 37.6–41.6 BLEU—comparable to the WMT results.

Flores-101 (Section 4.3.3, Table 8): This is the most comprehensive translation evaluation, testing one-shot translation across language pairs deliberately stratified by resource level:

• High-resource pairs (Table 8c): Translation between English, Arabic, Spanish, French, and Chinese. BLOOM achieves 45.0 BLEU for English→French, 40.3 for Arabic→English, 45.6 for French→English, and 26.7 for English→Chinese. These results are "comparable and sometimes better" than M2M-100 (615M supervised) and "comparable in many cases to those of AlexaTM" (32-shot, Soltan et al., 2022). The Arabic→English result of 40.3 BLEU is particularly notable—it substantially exceeds M2M-100's 25.5 and approaches AlexaTM's 41.8, despite BLOOM receiving no explicit translation training.

• High→mid-resource pairs (Table 8d): Translation from English, French, Hindi, Indonesian, and Vietnamese. English→Indonesian achieves 39.0 BLEU; French→Indonesian, 31.4; English→Vietnamese, 28.5. These results demonstrate "BLOOM's good multilingual capacity, even across scripts (here between Latin (or extended Latin), Chinese, Arabic and Devanagari scripts)" (Section 4.3.3).

• Romance language pairs (Table 8b): Translation between Catalan, Spanish, French, Galician, Italian, and Portuguese. Despite Galician (glg) "not being included in the training data," BLOOM translates from Galician to French at 33.8 BLEU and to Portuguese at 32.2 BLEU—evidence of cross-lingual transfer exploiting Galician's similarity to Portuguese (79.3 GB in ROOTS). Translation between Spanish and Portuguese reaches 29.1 BLEU in one direction and 39.6 in the other, comparable to M2M-100's supervised performance.

• Low-resource pairs (Table 8a): Translation involving Bengali, Hindi, Swahili, and Yoruba. The critical negative result: Swahili→Yoruba achieves 1.3 BLEU and Yoruba→Swahili reaches 0.9 BLEU—both languages are present but "under-represented in BLOOM's training data (<50k tokens each)." This establishes the capability frontier: cross-lingual transfer across unrelated low-resource languages essentially does not happen. Translation from these languages to English is somewhat better (Swahili→English: 37.4 BLEU; Yoruba→English: 4.1 BLEU), but the asymmetry reveals that BLOOM has learned much stronger English representations that can serve as a target for translation from even low-resource source languages.

A structural finding: translation quality for a given language pair depends on both languages' representation in training, not just one. The collapse on Swahili↔Yoruba despite Swahili→English working reasonably well shows that the target language's representation is critical—BLOOM can translate from an under-represented language into a well-represented one, but not between two under-represented languages.

4.4 Summarization

Figure 9 presents one-shot summarization results on WikiLingua for BLOOM models (560M through 176B) and OPT-175B across nine languages. Each point represents a per-prompt ROUGE-2 F-measure score.

The headline findings: "BLOOM attains higher performance on multilingual summarization than OPT and... performance increases as the parameter count of the model increases" (Section 4.4). The OPT-175B comparison is telling: on languages like Arabic, French, Spanish, and Vietnamese, BLOOM-176B's per-prompt ROUGE-2 scores are visibly higher than OPT-175B's. On English summarization, the gap is smaller or reversed depending on the prompt—which is expected since OPT was trained primarily on English data.

The scaling trend within the BLOOM family is consistent: BLOOM-560M and BLOOM-1.1B produce ROUGE-2 scores clustered near zero for most non-English languages; BLOOM-3B and BLOOM-7.1B show emerging signal; BLOOM-176B produces substantially higher scores. This monotonic improvement with scale across languages is evidence that multilingual generation capability is not an emergent property that suddenly appears at some threshold—it improves gradually, with the largest model showing the best performance across all languages tested.

A qualitative caveat: the paper notes that "in many cases, the ROUGE-2 score understates the quality of the summaries generated by the systems" (Section 4.4). ROUGE measures n-gram overlap with reference summaries, which penalizes valid paraphrases and alternative phrasings—a well-known limitation of automated summarization metrics that is particularly acute in multilingual settings where acceptable translations may use different lexical choices.

4.5 Code Generation

Table 9 reports pass@k results on HumanEval for BLOOM and BLOOMZ models across all six sizes, compared with GPT-Neo, GPT-J, GPT-NeoX, and Codex models.

The headline: BLOOM-176B achieves 15.52% pass@1, 32.20% pass@10, and 55.45% pass@100. This is competitive with similarly-trained models: GPT-J 6B achieves 11.62% pass@1, GPT-NeoX 20B achieves 15.4% pass@1. The Codex models—which are "solely finetuned on code"—are substantially stronger across all sizes: Codex-12B achieves 28.81% pass@1, nearly double BLOOM-176B's performance with 15x fewer parameters. This gap quantifies the benefit of code-specific finetuning over general-purpose pretraining with code as a minority component (~11% of ROOTS).

BLOOMZ models do not improve significantly over BLOOM on code generation: BLOOMZ-176B achieves 12.06% pass@1 vs. BLOOM-176B's 15.52%. The paper hypothesizes this is "due to the finetuning dataset, xP3, not containing significant amounts of pure code completion. Rather, xP3 contains code-related tasks, such as estimating the time complexity of a given Python code snippet" (Section 4.5). This is an important negative result for the multitask finetuning paradigm: instruction tuning on diverse tasks does not necessarily transfer to code generation if the finetuning mixture lacks sufficient representation of the target task format.

The scaling trend within BLOOM is consistent: pass@1 improves from 0.82% (560M) to 15.52% (176B), with the largest jump occurring between 7.1B (7.73%) and 176B (15.52%).

4.6 HELM Benchmark

Figure 10 reproduces HELM benchmark results (Liang et al., 2022) for a wide range of language models in the 5-shot setting across six evaluation dimensions: accuracy, calibration error, robustness, fairness, bias, and toxicity. BLOOM-176B's positioning:

• Accuracy: BLOOM is "roughly on par in accuracy with previous-generation English-only models, such as GPT3-davinci v1 and J1-Grande v1, but behind more recent monolingual models such as InstructGPT davinci v2, Turing NLG v2, Anthropic-LM v4-s3, or OPT" (Section 4.6). OPT-175B achieves higher accuracy than BLOOM-176B on this English-centric benchmark—consistent with the SuperGLUE finding that BLOOM matches OPT on some tasks but not all.

• Calibration: BLOOM is "not very well calibrated" (Section 4.6), placing on the higher end of the calibration error distribution in Figure 10. This means the model's confidence estimates (the probabilities it assigns to its predictions) do not align well with its actual accuracy—a common issue in large language models.

• Robustness: BLOOM is "quite robust" (Section 4.6), performing relatively well on perturbed or distribution-shifted inputs compared to its base accuracy.

• Fairness: BLOOM is "one of the best models for fairness" (Section 4.6) on HELM's fairness metric, placing near the top of the distribution among evaluated models.

• Toxicity: BLOOM is "slightly more toxic than average in English" (Section 4.6).

• Bias: BLOOM is "average for bias" (Section 4.6).

These HELM results provide the most direct head-to-head comparison with a wide range of models under standardized conditions. The key takeaway is that BLOOM's multilingual training does not produce catastrophic degradation on English-centric evaluations—it remains competitive with models of similar scale—but it does not achieve state-of-the-art English performance either.

4.7 Multitask Finetuning (BLOOMZ)

Figure 11 presents zero-shot performance of BLOOM and BLOOMZ models compared with XGLM-7.5B, mTk-Instruct (13B), and T0 (11B) on held-out multilingual tasks: XNLI (natural language inference in 8 languages), XWinograd (coreference resolution in 3 languages), XCOPA (causal reasoning in 5 languages), and XStoryCloze (story completion in 8 languages).

The core finding: pretrained BLOOM and XGLM "performances are near the random baselines of 33% for NLI (XNLI) and 50% for coreference resolution (XWinograd) and sentence completion (XCOPA and XStoryCloze)" (Section 4.7). After multitask finetuning, BLOOMZ-7.1B and BLOOMZ-176B show dramatic improvements: on XNLI, accuracy rises from near-random to 40–70% range depending on the language and prompt. On XWinograd, BLOOMZ-176B achieves 60–80% accuracy across French, Portuguese, and Chinese. On XCOPA and XStoryCloze, BLOOMZ-176B reaches 50–90% accuracy across languages including Indonesian, Swahili, Tamil, Vietnamese, and Chinese.

T0 (11B), despite being instruction-tuned, performs poorly on multilingual tasks because it is "a monolingual English model" that "may be hurt by its inability to tokenize some non-English texts" (Section 4.7). mTk-Instruct (13B), which is also multitask finetuned on a multilingual instruction dataset (Super-NaturalInstructions), provides the most direct comparison: BLOOMZ-7.1B generally matches or exceeds mTk-Instruct-13B despite having roughly half the parameters, suggesting that xP3's language distribution (mirroring ROOTS) and dataset diversity provide stronger multilingual transfer than Super-NaturalInstructions.

The paper notes an additional finding deferred to Muennighoff et al. (2022b): "models finetuned on xP3 also outperform T0 on English datasets when controlling for size and architecture. This is likely due to T0's finetuning dataset (P3) containing less diverse datasets and prompts than xP3" (Section 4.7). This suggests the benefit of xP3 is not just multilinguality but also greater task and prompt diversity.

4.8 Embeddings

Table 10 reports results from the Massive Text Embedding Benchmark (MTEB, Muennighoff et al., 2022a) for BLOOM-derived embedding models (SGPT-BLOOM-1.7B-nli and SGPT-BLOOM-7.1B-msmarco) compared with ST5-XL, LASER2, MiniLM-L12, MPNet, and LaBSE on two task types: embedding classification on MASSIVE (FitzGerald et al., 2022) across 15 languages, scored by accuracy; and semantic textual similarity on STS22 (Madabushi et al., 2022) across 5 languages, scored by Spearman correlation of cosine similarities.

The headline: SGPT-BLOOM-7.1B-msmarco provides "state-of-the-art performance on several classification and semantic textual similarity splits" (Section 4.8). On MASSIVE classification, it achieves 59.25% accuracy for Arabic (vs. 51.43% for MiniLM-L12), 69.67% for English (vs. 69.32% for MiniLM-L12), 66.35% for Spanish (vs. 64.43% for MiniLM-L12), and 66.95% for French (vs. 64.82% for MiniLM-L12). The gains are consistent across high-resource languages. On STS22 semantic textual similarity, SGPT-BLOOM-7.1B-msmarco achieves Spearman correlations of 58.67 for Arabic (vs. 52.19 for MiniLM-L12), 66.13 for English (vs. 63.06), 65.41 for Spanish (vs. 59.91), and 80.38 for French (vs. 74.30).

However, the paper acknowledges a significant caveat: "with 7.1 billion parameters it is an order of magnitude larger than models like the displayed multilingual MiniLM and MPNet" (Section 4.8). The MiniLM-L12 model has approximately 118M parameters—roughly 60x smaller. SGPT-BLOOM-1.7B-nli "performs significantly worse, likely due to less parameters and its finetuning being shorter (NLI is a much smaller dataset than MS-MARCO)." ST5-XL, the next largest model at 1.2B parameters, is English-only and "its performance on non-English languages is poor"—reinforcing that scale alone does not produce multilingual competence; multilingual pretraining is necessary.

4.9 Multilingual Probing

Section 4.9 uses the "Universal Probing" framework (Serikov et al., 2022) to analyze BLOOM's internal representations of morphosyntactic features across 17 languages and 38 features. The methodology: extract <s>-pooled representations from each layer of BLOOM-1B7 and BLOOM-176B, train binary logistic regression classifiers to predict the presence of each morphosyntactic feature, and evaluate using weighted F1 score (chosen "due to target class imbalance for most probing tasks").

Overall results (Table 12): Both BLOOM variants substantially outperform count-based baselines (TF-IDF with character, word, BPE, and SentencePiece n-grams). BLOOM-1B7 achieves higher average F1 scores than BLOOM-176B across most languages—a counterintuitive finding. For example, BLOOM-1B7 achieves 0.66 ± 0.27 on Arabic vs. 0.64 ± 0.27 for BLOOM-176B; 0.68 ± 0.19 on Basque vs. 0.62 ± 0.19; 0.67 ± 0.23 on Portuguese vs. 0.63 ± 0.26. The paper interprets this as BLOOM-1B7 having "better grammatical generalization over the aforementioned languages" (Section 4.9.2), though the standard deviations are substantial and overlapping.

Feature-wise analysis (Figure 12): The heatmaps show that Mood and Person features are "inferred well with no regard for language" while Number, NumType, and Voice are "moderately inferred in most languages." Other categories like Case, Gender, and Definite show sparser, language-dependent performance. The paper hypothesizes this pattern reflects "the diversity of possible values of these categories. For example, Mood and Person share similar values across the presented languages, while the set of Case values is highly dependent on the language" (Section 4.9.2).

Correlation analysis (Table 13): The most revealing finding: "Results of BLOOM-1B7 are highly correlated with language family, probing dataset size, and pretraining dataset size" while "BLOOM shows more stable performance on different languages in spite of the amount of data it has seen during pretraining" (Section 4.9.2). Specifically, for BLOOM-1B7, the Pearson correlation between probing performance and pretraining dataset size is significant (p < 0.01); for BLOOM-176B, it is not (p = 0.46). This suggests a qualitative shift with scale: the smaller model's grammatical representations are directly constrained by how much data it has seen in each language, while the larger model develops more robust representations that depend less on raw data quantity—consistent with the idea that larger models can leverage cross-lingual transfer more effectively.

Language-specific patterns: "BLOOM infers linguistic properties better for the closely related languages that comprise a significant amount of data. For example, the performance on any Romance language is better than in English, and the results in Indic languages are close to those in high-resource languages" (Section 4.9.2). Bengali, Wolof, and Yoruba "receive the lowest scores," consistent with their under-representation in ROOTS. The Mann-Whitney U test confirms BLOOM-1B7 "shows significantly better results (p < 0.01) than BLOOM" (Table 13), though this aggregate difference masks the per-language nuance where BLOOM-176B is sometimes more stable.

4.10 Bias Evaluation

Section 4.10 evaluates bias using a multilingual CrowS-Pairs dataset combining a revised English version (addressing validity issues identified by Blodgett et al., 2021) with a French extension (Névéol et al., 2022). The evaluation adapts a dataset originally designed for masked language models to autoregressive models by designing prompts that ask the model to select which of two minimally-contrasting sentences (one stereotyped, one not) is more plausible.

Overall results (Figure 13): BLOOM's "overall prompt accuracy was close to .50, which suggests an overall absence of bias" (Section 4.10). The results are presented per prompt as box plots: for English, BLOOM-176B, BLOOM-1.1B, and BLOOM-560M all cluster around 0.50, as do GPT-Neo-1.3B and GPT-Neo-125M. For French, the same pattern holds: BLOOM variants and GPT-FR models all center near 0.50. The paper notes that "the scores in English and French are very close, suggesting similar overall behavior of the model on both languages" (Section 4.10).

Per-category analysis (Table 14): When broken down by bias type, the results are "quite homogeneous over the categories, which contrasts with previous studies on masked language models, which suggested models were prone to bias in specific categories" (Section 4.10). However, significance testing reveals that accuracy "significantly differs from 50 (T-test, p < .05) overall for both languages, as well as for a number of bias categories" (indicated by asterisks in Table 14). For English, the overall accuracy is 49.78% (slightly biased against stereotypes); for French, 50.61% (slightly biased toward stereotypes). Specific categories showing significant deviation in English: gender (51.17%—model prefers stereotyped statements), socioeconomic status (51.05%), nationality (49.25%—model prefers non-stereotyped statements), religion (53.82%), and disability (48.49%). For French: ethnicity/color (50.48%), gender (51.24%), socioeconomic status (52.22%), nationality (48.49%), religion (53.01%), disability (49.16%), and other (42.1%).

Critical limitations acknowledged: The paper notes that Blodgett et al. (2021) identified validity issues with the original CrowS-Pairs corpus, and while the version used here incorporates revisions, "its original validation does not naturally apply here, and comparison to other CrowS-Pairs results is more difficult" (Section 4.10). More fundamentally, "as noted by Talat et al. (2022), very little material (corpora, measures) is available for multilingual bias assessment"—the evaluation covers only English and French, two of the 46 languages BLOOM supports, leaving bias in the remaining 44 languages unmeasured.

Ablation Studies and Robustness Checks

The BLOOM paper does not follow the ablation-study structure common in methods papers (where components are systematically removed to measure their contribution). Instead, its "ablations" are distributed across the architecture design process (Section 3.2, conducted at smaller scales and reported in separate papers) and the evaluation section (where different prompting strategies, model sizes, and finetuning variants serve as implicit ablations). The following identifies the non-trivial comparisons that function as robustness checks or design validations:

Prompt sensitivity (Figures 7, 9, 11, Tables 6, 7): Across SuperGLUE, machine translation, summarization, and multitask finetuning evaluations, results are reported for multiple prompts (typically 4–5 per task), with per-prompt points plotted individually. This serves as a robustness check: if results are highly prompt-dependent, the model's task competence is brittle; if consistent across prompts, the capability is more genuine. The pattern varies by task: on SuperGLUE entailment tasks (BoolQ, CB), performance is relatively consistent across prompts; on other SuperGLUE tasks, "the best prompts do better, [but] the average performance across prompts hovers around chance" (Section 4.2). On WMT translation, the "version-target" prompt consistently outperforms "gpt3-target" by 10+ BLEU points (Table 6), demonstrating substantial prompt sensitivity. On multitask finetuning (Figure 11), the five per-dataset prompts show consistent improvements from pretrained to BLOOMZ, but per-prompt variance remains substantial—for XNLI, some prompts achieve ~70% accuracy while others achieve ~40%.

Model scale as an ablation of multilingual capacity (Figures 8, 9, Table 9): The six model sizes (560M through 176B) enable scaling analysis that implicitly ablates model capacity. On SuperGLUE (Figure 8), both OPT and BLOOM families improve "very slightly with scale," suggesting that English task performance is not strongly capacity-limited in this range. On WikiLingua summarization (Figure 9), BLOOM performance increases "as the parameter count of the model increases," but BLOOM-560M and BLOOM-1.1B produce near-zero ROUGE-2 for non-English languages—indicating a minimum capacity threshold below which multilingual generation does not emerge. On HumanEval (Table 9), pass@1 improves monotonically from 0.82% (560M) to 15.52% (176B), with the largest jump between 7.1B and 176B. On probing (Table 12), the 1.7B model actually outperforms the 176B model on average, suggesting that grammatical feature encoding does not monotonically improve with scale.

BLOOM vs. BLOOMZ as an ablation of instruction tuning (Table 9, Figure 11): Comparing pretrained and multitask-finetuned variants quantifies the benefit of instruction tuning. For code generation (Table 9), BLOOMZ performs worse than BLOOM across all sizes—a negative result that reveals instruction tuning on xP3 (which lacks pure code completion) can degrade code generation capability. For multilingual zero-shot tasks (Figure 11), BLOOMZ dramatically outperforms BLOOM, confirming that instruction tuning provides the expected zero-shot generalization benefits for NLP tasks.

Prompt language: xP3 vs. xP3mt (Section 3.1.4, Section 4.7): The construction of both native-language prompts (xP3) and machine-translated English prompts (xP3mt) enables studying the impact of prompt language on multilingual instruction following. Results are deferred to Muennighoff et al. (2022b) and not reported in this paper, but the existence of this comparison reflects attention to a subtle design choice: should instruction-tuning prompts be in the task language or translated from English?

Truncation as a post-hoc correction for over-generation (Table 7): On DiaBLa, the difference between raw BLEU scores (5.7–12.3) and truncated BLEU scores (37.6–41.6) quantifies the severity of the over-generation problem. The truncation function—which "discards anything after a first newline or the regular expression pattern = .+?" (Section 4.3.2, footnote 33)—is specific to the xglm-source+target prompt format. This serves as both a robustness check (showing that the model does produce the correct translation, just with extra text appended) and a limitation (showing that the model cannot reliably terminate its own output).

One-shot example selection in DiaBLa (Table 7): Comparing "Rand." (random test set example as one-shot) vs. "Prev." (previous dialogue utterance) tests whether the model leverages conversational context for translation. The automatic metrics are inconclusive, but qualitative analysis (deferred to Bawden and Yvon, 2023) shows contextual effects on translation choices. This is a weak ablation—the null result on automatic metrics could reflect metric limitations rather than model behavior.

Contrastive finetuning recipe (Table 10): SGPT-BLOOM-7.1B-msmarco vs. SGPT-BLOOM-1.7B-nli demonstrates the impact of finetuning dataset and model scale on embedding quality. The 7.1B model finetuned on MS-MARCO (a large information retrieval dataset) substantially outperforms the 1.7B model finetuned on NLI (a smaller dataset), but the confound between scale and dataset prevents clean attribution.

Probing layer analysis (Section 4.9): By extracting representations from each layer (not just the final layer) and training separate probing classifiers, the analysis could reveal where in the model grammatical features are encoded. However, the paper reports only layer-averaged results (Table 12), with the note that "the analysis has focused on averaged representations of all layers and at the end of training. Analyzing different layers may reveal how morpho-syntactic representations are built during processing" (Section 4.9.2). This is explicitly flagged as future work, meaning the current probing results are a coarse signal rather than a detailed mechanistic analysis.

Statistical significance on CrowS-Pairs (Table 14): The one-sample T-test against 0.50 for each bias category serves as a robustness check on the "overall absence of bias" claim. The finding that overall accuracy significantly differs from 0.50 (49.78% for English, 50.61% for French, both p < 0.05) qualifies the claim: there is a small but statistically detectable bias, even though the effect sizes are modest (deviations of 0.2–3.8 percentage points from 0.50 depending on category).

Critical Assessment

Claim from the executive summary: "BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning."

The evidence for competitive pretrained performance is strongest on machine translation (Table 8), where BLOOM in one-shot settings matches or exceeds the supervised M2M-100 model on many high-resource pairs, and on SuperGLUE (Figure 7), where BLOOM-176B matches or exceeds OPT-175B in the one-shot setting. The evidence is moderately strong on code generation (Table 9), where BLOOM-176B is competitive with GPT-NeoX-20B but substantially behind code-specialized models like Codex. The evidence is weakest on HELM (Figure 10), where BLOOM is "roughly on par... with previous-generation English-only models" but "behind more recent monolingual models"—meaning it is not competitive with the frontier, only with models of similar scale from the previous generation. The claim about stronger results after multitask finetuning is strongly supported by Figure 11: BLOOMZ dramatically outperforms pretrained BLOOM across all held-out multilingual tasks, lifting performance from near-random to well above chance. However, the code generation counterexample (Table 9, where BLOOMZ underperforms BLOOM) demonstrates that this improvement is not universal—multitask finetuning helps on tasks represented in the finetuning mixture and may hurt on tasks not represented.

Claim from the executive summary: "BLOOM's training emitted approximately 25 tons of CO2eq—roughly one-third of OPT-175B's emissions—establishing that large-scale multilingual training can achieve competitive results with a significantly lower carbon footprint when leveraging low-carbon energy grids."

The carbon accounting is thorough (Section 3.5.1, Table 4) and the comparison with OPT is well-supported: 25 tons vs. 70 tons for comparable energy consumption (433 MWh vs. 324 MWh). However, the claim conflates two distinct points. The low carbon intensity of France's electricity grid (57 gCO2eq/kWh vs. 231 gCO2eq/kWh for OPT's grid) is an infrastructure fact, not a property of the model or training process. BLOOM's training would have emitted ~100 tons on OPT's grid. The claim that "large-scale multilingual training can achieve competitive results with a significantly lower carbon footprint" is true of this specific training run on this specific infrastructure, but the general lesson—that model developers should choose low-carbon grids—is about siting decisions, not about multilingual training per se. The paper does not demonstrate that multilingual training is inherently more carbon-efficient than monolingual training; it demonstrates that training on a nuclear-powered grid in France produces less CO2 than training on a fossil-fuel-powered grid elsewhere.

A more precise claim would be: "When trained on a low-carbon energy grid, a 176B-parameter multilingual model can achieve competitive performance while emitting substantially less CO2 than comparable models trained on higher-carbon grids—demonstrating that carbon footprint is largely determined by infrastructure choices rather than model scale or multilinguality."

Claim from the executive summary: "On the hardest low-resource language pairs (e.g., Swahili↔Yoruba), performance collapses to near-zero BLEU, establishing that BLOOM's multilingual competence depends critically on sufficient representation in the pretraining corpus rather than cross-lingual transfer alone."

This claim is well-supported by Table 8a: Swahili→Yoruba BLEU of 1.3 and Yoruba→Swahili of 0.9, contrasted with Swahili→English BLEU of 37.4 and Yoruba→English of 4.1. The asymmetry—translation from low-resource languages into English works reasonably well while translation between low-resource languages fails—provides strong evidence for the dependence on target-language representation. Cross-lingual transfer does occur (as demonstrated by Galician translation in Table 8b), but it requires linguistic relatedness; for unrelated low-resource languages, transfer is essentially zero.

Potential weaknesses:

• Most results lack statistical rigor. SuperGLUE, MT, summarization, and code generation results are reported without confidence intervals, standard deviations (except for CrowS-Pairs and probing), or statistical significance tests. The per-prompt point plots in Figures 7, 9, and 11 provide visual indication of variance, but the sample sizes (5 prompts per task) are too small for meaningful statistical inference. For MT results in Table 8, a single BLEU score is reported per language pair with no indication of variance across test examples or few-shot example selections.

• Prompt selection is not systematically controlled. The paper states that prompts were "developed prior to BLOOM's release, and did not undergo any a priori refinement using models" (Section 4.1.1). This simulates "realistic zero-shot or one-shot results that a new user could expect" but means the reported results are a lower bound on what carefully optimized prompts could achieve. This is a deliberate design choice (as the paper argues, optimized prompts are "harder to reproduce systematically" and "not representative of true zero-shot learning"), but it complicates comparison with models evaluated using optimized prompts—including likely some of the baseline models.

• The OPT comparison is confounded by multiple differences. BLOOM and OPT differ not only in multilinguality but in training data composition (ROOTS vs. OPT's corpus), architecture details (ALiBi vs. learned position embeddings, embedding LayerNorm), training infrastructure (Jean Zay vs. OPT's cluster), and hyperparameters. When BLOOM matches or exceeds OPT on a benchmark, it is not possible to attribute this to any specific design choice—it could be due to superior data curation, better architecture, or simply noise.

• The SuperGLUE evaluation excludes the most compute-intensive tasks. The paper notes that "the remaining tasks... require an order of magnitude more compute to run than all of these tasks we consider combined" (Section 4.1.3). This is a practical constraint, but it means the SuperGLUE evaluation covers only a subset that may not be representative of the full benchmark's difficulty.

• The probing analysis averages across layers and compares models of substantially different scales. BLOOM-1B7 (1.7B parameters) is compared with BLOOM-176B, a 100× difference in scale. The finding that the smaller model performs better on probing tasks is intriguing but cannot rule out that the difference reflects architectural choices (BLOOM-1B7 has 24 layers vs. 70 for BLOOM-176B; the probing classifier architecture might interact differently with representations from models of different depths) rather than genuine differences in grammatical encoding quality.

• Bias evaluation covers only 2 of 46 languages. CrowS-Pairs exists in English and French versions. The bias properties of BLOOM in the remaining 44 languages—including languages with very different cultural contexts for the stereotypes being measured—are completely uncharacterized. The paper acknowledges this limitation but does not mitigate it.

• No evaluation of generative harms beyond CrowS-Pairs. The bias evaluation focuses on stereotyped associations in a forced-choice format. There is no evaluation of toxic generation, hate speech production, or other generative harms across languages, despite the HELM benchmark (Figure 10) providing a toxicity metric for English that shows BLOOM is "slightly more toxic than average."

Missing experiments that would strengthen the paper:

• A direct comparison of ROOTS vs. an aggressively-crawled corpus of equivalent size. The paper's central claim about data curation—that source-level traceability and human-guided filtering produce a better corpus than automated approaches—is never directly tested. Training an identically-sized model on an aggressively-filtered Common Crawl corpus in the same languages and comparing performance would directly measure the benefit of the curation approach. Without this, the value of the data governance work remains asserted rather than demonstrated.

• Systematic few-shot scaling beyond one-shot. The paper evaluates only zero-shot and one-shot settings, with the occasional mention that performance "greatly improved as the number of few-shot examples is increased" (WMT, Section 4.3.1). Measuring performance at 0, 1, 4, 8, and 32 shots for representative tasks would characterize BLOOM's in-context learning capabilities more thoroughly and enable comparison with models like GPT-3 that were evaluated at multiple shot counts.

• Language-specific evaluation beyond MT. The multilingual evaluation is dominated by machine translation. For languages where MT benchmarks exist, this is informative, but for many of the 46 languages in ROOTS, no MT evaluation is possible (no parallel corpora exist). Alternative evaluation methods—language modeling perplexity, native speaker qualitative assessment, downstream task performance where available—would provide a more complete picture of BLOOM's capabilities across all supported languages.

• Ablation of the embedding LayerNorm at the 176B scale in bfloat16. The paper hypothesizes that the embedding LayerNorm might not be necessary in bfloat16 (Section 3.2.3), but does not test this. Training a smaller variant with and without the embedding LayerNorm in bfloat16 would resolve whether the stability benefit is specific to float16 and whether the zero-shot performance penalty can be avoided.

• Evaluation of BLOOMZ on English-only tasks to quantify multilingual instruction tuning's effect. Figure 11 shows BLOOMZ dramatically improves multilingual zero-shot performance, but no comparison with English-only instruction-tuned models on English tasks is provided. The deferred finding that BLOOMZ outperforms T0 on English tasks (Muennighoff et al., 2022b) is mentioned but not supported with data in this paper.

• Carbon footprint of alternative model scales. The paper reports training emissions only for BLOOM-176B (Table 4). Given the scaling analysis across six model sizes, reporting emissions for each size variant would enable analyzing the carbon efficiency of scaling—at what point does the emissions cost of additional parameters outweigh the performance benefit?

6. Limitations and Trade-offs

Limitation 1: Only 46 of ~7,000 Languages Are Supported, and Most Supported Languages Have Negligible Representation

The assumption or constraint. BLOOM positions itself as a "multilingual language model" whose development was motivated in part by the observation that "most LLMs are primarily trained on English-language text" (Section 1). However, the model supports 46 natural languages out of approximately 7,000 spoken worldwide, and even among those 46, the data distribution is extraordinarily skewed. The paper is transparent about this from the outset: Table 1 and Figure 3 show English at 484.95 GB and Simplified Chinese at 261.02 GB, while languages like Fon (2.5 MB), chiShona (6.6 MB), Chitumbuka (170 KB), and Xitsonga (0.7 MB) are present at sizes four to six orders of magnitude smaller. The paper acknowledges that Indo-European and Sino-Tibetan families "overwhelm the plot with a combined total of 1321.89 GB" while "the green rectangle 0.4GB constituting the Niger-Congo language family subset" (Section 3.1.2, Figure 3 caption).

The consequence. For the most under-represented languages, BLOOM's "support" is largely symbolic. The Flores-101 translation results (Table 8a) provide direct evidence: translation between Swahili and Yoruba—both Niger-Congo languages present in ROOTS but with fewer than 50,000 tokens each—achieves 1.3 BLEU (Swahili→Yoruba) and 0.9 BLEU (Yoruba→Swahili). These are effectively zero; the model has no usable competence in these language pairs. Translation from these languages into English is somewhat better (Swahili→English: 37.4 BLEU; Yoruba→English: 4.1 BLEU), but the asymmetry confirms that performance depends on the target language having substantial representation—the model can map a poorly-learned source language into a well-learned target, but cannot go the other direction or operate between two poorly-learned languages.

The probing results (Table 12) reinforce this: Bengali (18.6 GB in ROOTS, but an Indic language with substantial representation) achieves probing F1 scores of 0.42–0.45, while Yoruba (~90 MB) achieves 0.36–0.48 and Wolof (3.6 MB) achieves 0.47–0.51. The variation is substantial, and languages with the smallest corpora show the weakest and least stable morphosyntactic encoding.

For a practitioner considering deploying BLOOM for a specific language, the practical implication is: unless the language appears in the top 10–15 by corpus size (roughly the languages above 1 GB in Table 1), the model's competence is likely too low for production use without substantial fine-tuning—and for languages below ~100 MB, even fine-tuning may be insufficient because the pretrained representations are too weak to serve as a useful starting point.

What evidence exists in the paper. The evidence is spread across Table 1 (byte counts per language), Figure 3 (visual distribution), Table 8a (Flores-101 low-resource translation), Figure 9 (summarization scaling by language, where BLOOM-560M and BLOOM-1.1B produce near-zero ROUGE-2 for most non-English languages), and Table 12 (probing performance varying by language).

Mitigation status. The paper acknowledges this limitation structurally—by presenting the byte counts in Table 1 and the skewed distribution in Figure 3 without attempting to hide or minimize them—but does not mitigate it. The language inclusion criterion (three fluent speakers committing to guide source selection, Section 3.1.2) is a quality gate, not a quantity guarantee. The paper does not claim that all 46 languages receive equal support, and the evaluation results make the inequality empirically visible. However, no techniques for improving low-resource performance (upsampling, auxiliary training objectives, cross-lingual transfer optimization) are implemented or evaluated. The limitation is acknowledged through transparency rather than addressed through method.

---

Limitation 2: The Data Governance Framework Produces a Corpus That Is Neither Fully Open Nor Fully Reproducible

The assumption or constraint. A central tension in BLOOM's design is between openness (making everything publicly available for scientific scrutiny) and responsible data handling (respecting licenses, privacy, and agreements with data providers). The paper is explicit about the resulting compromise: "of the 498 component datasets, 223 can be directly distributed. The remainder are accessible through a sign-up mechanism for researchers with relevant projects" (Section 3.1.1). This means 45% of the training data cannot be publicly downloaded by anyone with an internet connection; access requires a research proposal and approval.

The consequence. This creates a reproducibility gap that is unusual for a project whose stated goal is "open science." A researcher who wants to fully reproduce BLOOM's training—to verify corpus properties, to study the effect of data composition on model behavior, to train an improved model from the same data—cannot do so without going through a gated access process. The paper frames this as a feature (respecting data rights) rather than a bug, but for the scientific goal of reproducibility, it is a genuine limitation.

More subtly, the composite release strategy fragments the corpus ecosystem. The 223 publicly available components can be studied, analyzed, and used for other projects; the 275 gated components cannot. This means research on the corpus itself—auditing for biases, measuring representation, studying the relationship between training data and model outputs—can only be partially conducted in the open. Researchers who obtain access to the full corpus through the sign-up mechanism may produce findings that cannot be independently verified by others without similar access.

The governance framework's long-term vision—"an international organization" for data governance (Section 3.1.1, citing Jernite et al., 2022)—was explicitly not achieved within the project timeline. The paper notes: "we were not able to fully establish an international organization in the comparatively short time between the project start and model training." This means the governance mechanisms used for ROOTS are project-specific and time-bound; they do not constitute an ongoing institution that can govern data for future projects or adapt to changing circumstances.

What evidence exists in the paper. Section 3.1.1 describes the composite release approach and the sign-up form mechanism. The paper reports the 223-of-498 number explicitly. The limitation is not hidden; it is documented as a deliberate tradeoff. Section 3.1.1 acknowledges that "since we understand that future investigation into the BLOOM models may require full access to the entire corpus, we are also inviting researchers with a relevant research project in mind to join ongoing efforts to analyze the data through a sign-up form."

Mitigation status. Partially mitigated. The sign-up mechanism provides a path for researchers to access the full corpus, but it introduces friction, delay, and a gatekeeping function (someone must approve the research proposal). The 223 publicly available components allow partial analysis. The paper does not report any metrics on how many researchers have applied for or been granted full access, nor on the criteria used to judge proposals. The long-term governance framework (Jernite et al., 2022) is proposed as a solution but remains unimplemented. For a practitioner who needs to train a model on comparable data without navigating gated access, the ROOTS release model provides no direct path—the 223 public components alone are insufficient to train a 176B-parameter model.

---

Limitation 3: The Architecture Design Methodology Relies on Small-Scale Ablations That May Not Extrapolate to 176B Parameters

The assumption or constraint. BLOOM's architecture was not designed through ablations at the final model scale. Instead, "we conducted our ablation experiments using smaller models. We used the 6.7B parameter scale for the pretraining objective ablations (Wang et al., 2022a) and the 1.3B scale for the rest including position embeddings, activations, and layer normalization (Le Scao et al., 2022)" (Section 3.2.1). The paper acknowledges the risk directly: "Recently, Dettmers et al. (2022) identified a phase transition for models larger than 6.7B, in which the emergence of 'outliers features' is observed. This questions whether results obtained at the 1.3B scale should be assumed to extrapolate to our final model size."

The consequence. Architectural choices that appeared optimal at 1.3B or 6.7B parameters may be suboptimal at 176B. The embedding LayerNorm provides a concrete example: the paper includes it because preliminary 104B-parameter experiments in float16 showed it "significantly improved training stability," even though the 1.3B-scale ablations found it "penalizes zero-shot generalization" (Section 3.2.3). The paper hypothesizes that bfloat16 might make the LayerNorm unnecessary, but this was never tested at scale—the final 176B model uses the LayerNorm despite the known zero-shot penalty because the team prioritized avoiding training instability over optimizing zero-shot performance.

Similarly, the ALiBi positional embedding was selected based on 1.3B-scale experiments where it "led to smoother training and better downstream performance even at the original sequence length—outperforming both learned and rotary embeddings" (Section 3.2.3). But whether ALiBi continues to outperform rotary embeddings at 176B—or whether the relative ranking of positional embedding schemes changes with scale—is unknown.

The exclusion of mixture-of-experts and state-space models was based on practical considerations (GPU codebase maturity, contemporary performance) that may no longer hold. The paper notes this explicitly: "Both of these approaches are promising, and have now demonstrated competitive results—at large scales for MoE (Fedus et al., 2022; Srivastava et al., 2022), and at smaller scale for state-space models with H3 (Fu et al., 2023)" (Section 3.2.1). A MoE model with the same total parameter count but fewer active parameters per token would have substantially lower inference cost—a tradeoff BLOOM's architecture cannot explore.

What evidence exists in the paper. Section 3.2.1 describes the ablation methodology and acknowledges the extrapolation concern. The embedding LayerNorm tension is documented in Section 3.2.3 with the bfloat16 hypothesis. The exclusion of MoE and state-space models is justified with specific reasons in Section 3.2.1. The paper does not provide evidence that the 1.3B-scale rankings hold at 176B—no ablation was conducted at the final scale to validate the extrapolation.

Mitigation status. Not mitigated. The paper acknowledges the limitation but cannot resolve it without retraining the 176B model under alternative architectural configurations—which would cost millions of GPU hours. The mitigation is the transparency itself: the paper documents why each architectural decision was made and at what scale it was tested, allowing future researchers to identify which decisions are most likely to be scale-dependent and target those for re-evaluation. For a practitioner selecting an architecture for a new large-scale model, the lesson is cautionary: BLOOM's architectural choices are empirically grounded at ~1–7B parameters but unvalidated at 176B, and the embedding LayerNorm example shows that training stability concerns (which may only manifest at scale) can override performance optimizations from smaller-scale ablations.

---

Limitation 4: The Multitask Finetuning (BLOOMZ) Improves Zero-Shot NLP Tasks at the Cost of Degrading Code Generation

The assumption or constraint. The paper presents BLOOMZ—BLOOM finetuned on the xP3 multitask prompted dataset—as the primary mechanism for achieving strong zero-shot task generalization (Section 3.1.4, Section 4.7). The assumption is that instruction tuning on a diverse mixture of prompted datasets improves general-purpose zero-shot capabilities without systematically degrading performance on tasks not represented in the finetuning mixture.

The consequence. Table 9 reveals that BLOOMZ-176B achieves 12.06% pass@1 on HumanEval compared to BLOOM-176B's 15.52%—a ~22% relative degradation. This is not a small difference; it drops BLOOMZ below GPT-J-6B (11.62%), meaning the instruction-tuned 176B model performs worse on code generation than a 6B model trained on a similar code+text mixture. The degradation is consistent across all model sizes: BLOOMZ-7.1B (8.06%) vs. BLOOM-7.1B (7.73%) is essentially flat, but at smaller scales the gap is proportionally larger (BLOOMZ-560M: 2.18% vs. BLOOM-560M: 0.82%—though both are near floor).

The paper correctly diagnoses the cause: "the finetuning dataset, xP3, does not contain significant amounts of pure code completion. Rather, xP3 contains code-related tasks, such as estimating the time complexity of a given Python code snippet" (Section 4.5). This means the degradation is not a mysterious side effect but a predictable consequence of catastrophic forgetting: the finetuning process overwrites some of the pretrained model's code generation capability because the finetuning data does not reinforce it. This is a fundamental tradeoff in multitask finetuning: improving zero-shot NLP task performance requires training on NLP tasks, which may degrade performance on capabilities (like code) that are not represented in the finetuning mixture.

For a practitioner deciding whether to deploy BLOOM or BLOOMZ, this creates a genuine dilemma: BLOOMZ is dramatically better at zero-shot NLP tasks (Figure 11) but measurably worse at code generation (Table 9). There is no single model that excels at both.

What evidence exists in the paper. Table 9 provides pass@1, pass@10, and pass@100 for BLOOM and BLOOMZ at all six sizes. The BLOOMZ degradation is visible at every scale. Figure 11 demonstrates the NLP task improvement. The paper's explanation (xP3 lacking code completion) is provided in Section 4.5.

Mitigation status. Not mitigated. The paper identifies the cause but does not attempt to fix it—for example, by including code completion tasks in xP3, by using a finetuning mixture that balances NLP and code, or by exploring techniques to reduce catastrophic forgetting (elastic weight consolidation, replay buffers, multi-task training rather than sequential finetuning). The paper does not even report whether including code-related tasks (time complexity estimation) provides any protection against code generation degradation or whether the mere presence of non-code tasks is sufficient to cause the forgetting. For a practitioner, the mitigation options are: (1) use BLOOM for code and BLOOMZ for NLP, accepting the operational cost of maintaining two model variants; (2) finetune BLOOMZ further on code to recover the lost capability; or (3) design a custom finetuning mixture that includes code completion—none of which are validated by the paper.

---

Limitation 5: Bias Evaluation Is Extremely Narrow: 2 of 46 Languages, a Single Dataset, and a Paradigm Mismatch

The assumption or constraint. Section 4.10 evaluates bias using the CrowS-Pairs dataset in English and French. The paper acknowledges this scope limitation: "as noted by Talat et al. (2022), very little material (corpora, measures) is available for multilingual bias assessment" (Section 4.10). The evaluation also acknowledges that CrowS-Pairs was "originally intended for masked language models" and required adaptation to autoregressive models like BLOOM through prompt design (Section 4.10). Finally, Blodgett et al. (2021) identified validity issues with the original CrowS-Pairs corpus; the paper uses a revised version but notes that "its original validation does not naturally apply here, and comparison to other CrowS-Pairs results is more difficult" (Section 4.10).

The consequence. The bias evaluation provides essentially no information about BLOOM's behavior in 44 of its 46 supported languages. Languages with very different cultural contexts for stereotypes—different historical power dynamics, different marginalized groups, different forms of linguistic bias—are completely unmeasured. A model that appears unbiased in English and French could exhibit systematic biases in Hindi, Arabic, Chinese, or Swahili, and the paper provides no evidence either way.

Even for English and French, the CrowS-Pairs evaluation has significant interpretive limitations. The adaptation from masked language modeling to autoregressive prompting changes the task: instead of comparing pseudo-log-likelihoods of minimally-contrasting sentences, the model is asked to explicitly judge which sentence is "more plausible." This introduces prompt sensitivity (the model's response depends on how the question is framed), makes the task more transparent to the model (it may recognize the stereotype being tested and moderate its response), and changes what is being measured (explicit judgments rather than implicit associations). The paper does not validate that the adapted evaluation measures the same underlying construct as the original.

The finding that BLOOM's overall accuracy is "close to .50" (Section 4.10, Figure 13) could indicate genuine absence of bias, or it could indicate that the adapted evaluation format is insufficiently sensitive to detect the biases present. The paper notes that accuracy "significantly differs from 50 (T-test, p < .05) overall for both languages" (Table 14), meaning there IS a statistically detectable bias, but the effect sizes are small (deviations of 0.2–3.8 percentage points from 0.50 depending on category). Whether these small deviations reflect genuine model biases, noise in the adapted evaluation, or both is unclear.

What evidence exists in the paper. Figure 13 shows per-prompt accuracy distributions around 0.50 for English and French. Table 14 breaks down accuracy by bias category with significance tests. The paper explicitly discusses the Blodgett et al. (2021) validity concerns and the adaptation challenge. The Talat et al. (2022) citation acknowledges the lack of multilingual bias assessment resources.

Mitigation status. Minimally mitigated. The paper acknowledges the scope and validity limitations candidly but does not develop alternative bias evaluation methods, test on additional languages, or validate the adapted CrowS-Pairs format against other bias measures. The HELM benchmark (Figure 10) provides a partial cross-check for English: BLOOM is rated as "one of the best models for fairness" but "slightly more toxic than average" and "average for bias." The toxicity finding—that BLOOM is more toxic than the average model in HELM's English evaluation—is in some tension with the CrowS-Pairs finding of "overall absence of bias." Toxicity and stereotyping are different constructs, but a model that is both "unbiased" (by one measure) and "more toxic than average" (by another) presents a complex safety profile that the paper does not attempt to reconcile. For a practitioner deploying BLOOM in a production setting, the takeaway is that bias and safety properties are largely unknown for most supported languages, and even for English and French the evidence is mixed and methodologically fragile.

---

Limitation 6: The Carbon Accounting Overstates the Generalizability of Low-Emission Training by Confounding Infrastructure with Methodology

The assumption or constraint. Section 3.5.1 and Table 4 present BLOOM's training emissions (25 tons CO2eq) as substantially lower than OPT-175B (70 tons) and GPT-3 (502 tons), with the paper attributing this to both "more efficient hardware" and "less carbon-intensive energy sources" (Section 3.5.1). The paper states that "France's national energy grid (which is used by Jean Zay) is largely powered by nuclear energy, which is low-carbon compared to grids powered by energy sources such as coal and natural gas." The implicit message is that BLOOM demonstrates a path to lower-carbon LLM training.

The consequence. The 25-ton figure is primarily a function of where the training happened, not how it was conducted. BLOOM's energy consumption (433 MWh) is actually higher than OPT's (324 MWh)—the model used more electricity, not less. The emissions advantage comes entirely from the carbon intensity of the French grid (57 gCO2eq/kWh) vs. the grid used for OPT training (231 gCO2eq/kWh). If BLOOM had been trained on OPT's grid, its emissions would have been approximately 100 tons—substantially worse than OPT. Conversely, if OPT had been trained on Jean Zay, its emissions would have been approximately 18 tons—better than BLOOM.

This means the carbon comparison does not demonstrate that BLOOM's development methodology (open collaboration, careful data curation, multilingual design) is inherently lower-carbon. It demonstrates that training on a nuclear-powered grid produces less CO2 than training on fossil-fuel-powered grids—a finding about electricity infrastructure, not about model design. The paper obscures this by presenting the comparison in Table 4 alongside parameter counts, implicitly suggesting that BLOOM achieved lower emissions through design choices rather than infrastructure access.

For a practitioner deciding how to reduce their model's carbon footprint, the actionable lesson from BLOOM is "train on a low-carbon grid"—but this is not a methodological innovation; it is a siting decision that depends on access to specific supercomputers. The paper provides no evidence that BLOOM's specific architecture, data pipeline, or training recipe is more energy-efficient than alternatives when controlling for hardware and electricity source.

Furthermore, the Life Cycle Assessment reveals that "idle consumption of the equipment and computing cluster" accounts for 55% of emissions (45 tons out of 81 total). This means less than half of the project's emissions came from the training run itself; the majority came from the supercomputer being powered on when not actively training BLOOM. This idle consumption is an infrastructure characteristic that would apply to any model trained on Jean Zay, regardless of design. The paper's broader carbon accounting (total BigScience project emissions ~219 tons CO2eq, of which BLOOM training is only 37%) further reinforces that the headline 25-ton figure represents only a fraction of the environmental cost of developing the model.

What evidence exists in the paper. Table 4 compares energy consumption (MWh) and emissions (tons CO2eq) across models. Section 3.5.1 provides the carbon intensity of the French grid (57 gCO2eq/kWh). The LCA breakdown (equipment manufacturing 14%, training energy 30%, idle consumption 55%) is provided in Section 3.5.1. The paper notes that "the final BLOOM training represents approximately 37% of the overall emissions, with other processes such as intermediate training runs and model evaluation adding up to the other 63%."

Mitigation status. Partially mitigated by transparency. The paper reports both energy consumption and emissions, and identifies the grid carbon intensity as the key factor. The LCA methodology and the inclusion of idle consumption are more thorough than most comparable papers. However, the paper does not explicitly separate the infrastructure effect from the methodology effect—it does not state, for example, "our energy consumption was higher than OPT's; our emissions advantage is entirely attributable to France's electricity mix." The comparison with GPT-3's 502 tons is similarly infrastructure-conflated: GPT-3 was trained on older hardware (V100 GPUs) on a higher-carbon grid, making the comparison a measure of hardware generation and electricity source rather than model design efficiency. For a practitioner who cannot access a nuclear-powered supercomputer, BLOOM's carbon efficiency is not replicable; the paper provides no guidance on reducing emissions when training on higher-carbon grids other than the implicit suggestion to train elsewhere.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm, a novel architecture, or a state-of-the-art benchmark result in the conventional sense. Its contribution operates at a different level: it provides an existence proof that a 176B-parameter language model can be developed through an open, multidisciplinary, internationally distributed collaboration of hundreds of researchers—and that the resulting model can be competitive with similarly-sized industrial systems while maintaining substantially stronger documentation, governance, and ethical constraints.

This matters because it reframes what is possible for the research community. Before BLOOM, the dominant narrative held that cutting-edge LLM development required the concentrated resources, proprietary data pipelines, and closed development processes of major industrial labs. EleutherAI's GPT-NeoX-20B had demonstrated that open-source models could reach 20B parameters, but the gap to 175B+ remained a qualitative one—there was no evidence that an open collaboration could navigate the engineering, organizational, and resource challenges at that scale. BLOOM demonstrates that it can, and documents how. This is not a paradigm shift in the Kuhnian sense—the underlying technology (Transformer architectures, autoregressive pretraining, multitask finetuning) is well-established. But it is a paradigm expansion: it broadens the set of organizational models that can produce frontier-scale artifacts.

The paper's most distinctive conceptual contribution is its elevation of process to a first-class scientific output. Section 2.2 documents the organizational structure (30 working groups, 1,200+ participants, 38 countries) with the same rigor that Section 3.2 documents the Transformer architecture. Section 3.1.1 presents the data governance framework—structured agreements between data providers and hosts, source-level traceability, composite release—as a positive design feature, not a constraint to be apologized for. Section 3.6 presents the Responsible AI License with its 13 behavioral-use restrictions as a technical contribution, a legal mechanism that encodes ethical judgments into enforceable terms. This is a genre shift in how large-scale ML papers are written: the development process itself becomes the subject of scientific documentation and scrutiny, alongside the model architecture and benchmark results.

The paper also resolves a latent tension in the literature about whether multilinguality necessarily degrades English performance. The concern was plausible: a model allocating capacity across 46 languages might perform worse on English tasks than an English-focused model of the same size. The SuperGLUE results (Figure 7) and the scaling comparison with OPT (Figure 8) demonstrate that this tradeoff is minimal—BLOOM-176B matches or exceeds OPT-175B on one-shot English tasks despite English constituting only ~30% of its training data. This finding weakens the argument that monolingual English models are necessary for strong English performance and strengthens the case for multilingual pretraining as a default.

A research direction that becomes more attractive after this work: studying how organizational design choices (working group structure, governance mechanisms, contributor attribution) affect the technical properties of trained models. The BLOOM paper provides a template for documenting these choices, but doesn't study their causal effects—for instance, whether the per-language fluent-speaker requirement improved data quality in measurable ways, or whether the Ethical Charter changed specific filtering decisions. Future collaborative projects could systematically vary organizational parameters and measure the downstream impact on model behavior.

A research direction that becomes less attractive (or at least, harder to justify): building large language models through aggressive web crawling with no documentation of provenance, no governance framework for data subjects, and no structured ethical constraints—and presenting this as a purely technical achievement without acknowledging the governance vacuum. BLOOM doesn't make such work impossible, but it establishes that an alternative exists and works, shifting the burden of proof: projects that adopt the opaque approach must now justify why the alternative is infeasible in their context, rather than treating opacity as the default.

The paper also provides a reconciliation of the carbon-footprint discourse. Prior work (Strubell et al., 2019; Patterson et al., 2021) had established that LLM training incurs substantial emissions, but the comparisons were often confounded by hardware generation and electricity grid differences. BLOOM's data (Table 4) separates these factors: 433 MWh energy consumption (higher than OPT's 324 MWh) but 25 tons CO2eq (lower than OPT's 70 tons), with the difference entirely attributable to France's nuclear-powered grid (57 gCO2eq/kWh vs. 231 gCO2eq/kWh). This establishes that carbon footprint is primarily an infrastructure siting decision, not a model design decision—a finding that should redirect decarbonization efforts toward compute provider choices rather than model architecture optimization.

---

Follow-Up Research This Work Enables

Validating whether per-language fluent-speaker curation produces measurably better data than automated filtering. BLOOM's central data claim is that requiring fluent speakers to guide source selection and quality filtering for each language produces a corpus that is qualitatively different—more representative, less biased, better documented—than automatically filtered web crawls. But this claim is never tested: the paper doesn't compare ROOTS against an aggressively-crawled, automatically-filtered corpus of equivalent size in the same languages, nor does it measure whether the per-language human guidance changed measurable corpus properties (e.g., lexical diversity, topic coverage, dialect representation) relative to an automated baseline. A strong follow-up would construct a "ROOTS-auto" corpus covering the same 46 languages using Common Crawl-based automated filtering (language ID, perplexity filtering, deduplication) without human per-language guidance, train identically-sized BLOOM variants on both corpora, and measure differences in downstream task performance, probing accuracy, and bias metrics. The null hypothesis—that human curation provides no measurable benefit over automated filtering at this scale—would be a valuable negative result that would force re-examination of the cost-benefit tradeoff of BLOOM's data approach. The alternative—that human curation produces systematic improvements on tasks requiring nuanced language understanding, or reduces specific measurable biases—would provide the empirical justification that the current paper asserts but does not demonstrate.

Quantifying the catastrophic forgetting trade-off between BLOOM and BLOOMZ. Table 9 reveals that BLOOMZ underperforms BLOOM on code generation (12.06% vs. 15.52% pass@1) because xP3 lacks pure code completion tasks. But the paper doesn't systematically characterize the forgetting phenomenon: does BLOOMZ forget other capabilities not represented in xP3 (e.g., translation quality degrades slightly, or mathematical reasoning, or long-form generation coherence), or is code uniquely affected? A thorough forgetting audit would evaluate BLOOM and BLOOMZ on a diverse held-out benchmark covering capabilities absent from xP3—code generation (HumanEval), mathematical reasoning (MATH or GSM8K), factual knowledge (TriviaQA, Natural Questions), translation quality (the Flores-101 pairs from Table 8), and creative generation (qualitative assessment)—and measure degradation on each. The results would produce a forgetting profile that tells practitioners exactly which capabilities are sacrificed when adopting BLOOMZ for zero-shot NLP tasks. The paper's observation that xP3 mirrors ROOTS's language distribution (Figure 4) suggests a further hypothesis: languages well-represented in xP3 should show less forgetting than languages present in ROOTS but underrepresented in xP3—testing this would clarify whether forgetting is task-driven or language-driven. If code degradation is the only substantial loss, the fix is straightforward (add code completion to xP3); if multiple capabilities degrade, more sophisticated mitigation (elastic weight consolidation, experience replay, multi-task rather than sequential training) becomes necessary.

Scaling the number of languages from 46 toward 500 under the same governance model. BLOOM's language inclusion mechanism—requiring at least three fluent speakers per language to commit to source selection and preprocessing guidance—functions as a quality gate but limits scalability. The paper covers 46 languages because that's how many language communities could be meaningfully engaged within the project timeline. A natural scaling question: can this model be extended to 100, 200, or 500 languages while maintaining the same per-language quality standards, or does the coordination overhead become prohibitive? A concrete follow-up would attempt to add 50 new languages to the ROOTS corpus using the same methodology—recruiting fluent speakers, sourcing data through community catalogues, applying per-language quality filtering—and measure: (a) the person-hours required per language as a function of digital resource availability (adding a high-resource language like Turkish should be cheaper than adding a low-resource language like Tamazight); (b) whether the quality of the resulting data (measured by downstream task performance relative to data quantity) is consistent with the original 46 languages; and (c) whether the governance mechanisms (data provider agreements, composite release) scale linearly or develop bottlenecks. This would transform BLOOM's qualitative claim about process into a quantitative one: "adding a language to ROOTS under the governance model requires X person-weeks of fluent-speaker effort and Y GB of curated data, and produces downstream task performance within Z% of what would be expected from automated collection of equivalent size." Such a finding would be directly actionable for any group planning a multilingual corpus expansion.

Stress-testing the Responsible AI License as a deployment-time constraint. BLOOM's RAIL license contains 13 behavioral-use restrictions intended to prevent harmful applications. But the paper provides no evidence about whether these restrictions are enforceable, whether they are being complied with, or whether they create unintended consequences (e.g., preventing legitimate research uses that happen to intersect with a restricted category). A follow-up study would monitor BLOOM's deployment ecosystem over the first 12–24 months post-release: how many organizations are using BLOOM under the RAIL license vs. alternative licenses? Are there documented cases of violations, and if so, what mechanisms exist for enforcement? Have the behavioral-use restrictions prevented specific harmful applications that would have been enabled by a fully open license, or have they primarily created friction for benign uses? The paper's citation of Contractor et al. (2022) on behavioral use licensing provides a framework but no empirical validation. This research direction is unusual for an ML venue—it requires legal and sociological methods alongside technical ones—but BLOOM makes it newly tractable because it provides a specific, publicly-documented license to study, unlike the proprietary terms of industrial models.

Testing whether bfloat16 eliminates the need for the embedding LayerNorm, and quantifying the zero-shot penalty. Section 3.2.3 reveals an unresolved tension: the embedding LayerNorm was included to prevent training instabilities observed in float16, but the 1.3B-scale ablations found it penalizes zero-shot generalization, and the authors hypothesize that bfloat16 might make it unnecessary. The final 176B model was trained in bfloat16 with the LayerNorm included because the team couldn't risk instability. A clean follow-up would train BLOOM-scale models (or at minimum, the 7.1B variant, which is large enough to be informative while being trainable with academic compute) with and without the embedding LayerNorm in bfloat16, measuring both training stability (frequency of loss spikes, whether training diverges) and zero-shot performance on the EAI-Eval and T0-Eval aggregates used for architecture ablations. The result would either: (a) confirm the hypothesis that bfloat16 removes the need for the LayerNorm, in which case future models can drop it and recover the zero-shot performance penalty; or (b) show that stability problems persist in bfloat16 without the LayerNorm, validating the current design and establishing a documented tradeoff (stability vs. performance) for future model designers. Without this experiment, the LayerNorm remains a design choice whose costs and benefits at the final training scale are unknown—it might be an unnecessary drag on zero-shot performance, or an essential stability safeguard, and the paper can't tell us which.

Building a "BLOOM-2" that integrates the lessons learned about data skew, language inclusion costs, and the BLOOM/BLOOMZ forgetting trade-off. The paper's transparency about its limitations—the extreme skew in language representation (English at 485 GB, Fon at 2.5 MB), the degradation of code generation after multitask finetuning, the narrow bias evaluation coverage—creates a natural blueprint for a successor. A BLOOM-2 would: (a) explicitly trade off language count against minimum per-language data quantity, perhaps targeting 20–30 languages with a guaranteed minimum of 5 GB each rather than 46 with a median in the low MB range; (b) include code generation tasks in the multitask finetuning mixture to avoid the forgetting observed in BLOOMZ; (c) implement bias evaluation across all supported languages from the start rather than post-hoc; (d) evaluate the embedding LayerNorm at scale in bfloat16 to resolve the stability-performance tension; and (e) measure carbon footprint at each model scale to produce a cost-benefit curve that informs future practitioners about the emissions implications of scaling. This is not a single experiment but a synthesis of the paper's documented gaps into a coherent next-generation project. The paper makes this tractable by providing the full infrastructure (data pipeline tools, training code, evaluation framework, governance templates) as open-source artifacts, meaning a BLOOM-2 team would not start from scratch but from a documented, working baseline with known failure modes.

---

Practical Applications and Downstream Use Cases

Open-access research on large language model internals. Before BLOOM, researchers studying the internal representations, attention patterns, factual knowledge localization, or mechanistic interpretability of large language models were limited to smaller open models (GPT-NeoX-20B, GPT-J-6B) or had to request restricted API access to proprietary systems. BLOOM provides a 176B-parameter model with full weight access, meaning researchers can run activation patching, causal tracing, probing on intermediate layers, and other interpretability techniques that require direct model access at a scale where emergent behaviors (as documented by Wei et al., 2022) become observable. The probing experiments in Section 4.9 (covering 17 languages, 38 morphosyntactic features, and comparing BLOOM-1B7 with BLOOM-176B) demonstrate a concrete use case: studying how grammatical representations change with scale. A researcher could extend this to investigate whether BLOOM-176B exhibits the "outlier features" phase transition identified by Dettmers et al. (2022), whether factual knowledge is localized in specific layers differently in multilingual vs. monolingual models, or whether cross-lingual transfer has a detectable signature in attention patterns. The reference example's finding that verifier over-optimization limits test-time compute scaling (Section 5.3) illustrates why model internals research matters: understanding why models fail provides the foundation for fixing those failures. BLOOM makes this research possible at 176B scale without corporate partnership.

Fine-tuning foundation for domain-specific multilingual applications. BLOOM provides a pretrained starting point for practitioners building NLP systems in languages that lack dedicated large-scale pretrained models. A hospital system deploying clinical NLP in Vietnamese, an educational technology company building reading comprehension tools in Swahili, or a legal tech startup developing contract analysis in Arabic could fine-tune BLOOM on their domain-specific data rather than training a model from scratch. The paper's results indicate where this is likely to work: languages with substantial ROOTS representation (Vietnamese at 43.7 GB, Arabic at 74.9 GB, Swahili at 236 MB) show varying levels of base competence. The BLOOMZ variant provides stronger zero-shot performance for task formats represented in xP3 (translation, summarization, NLI, QA), making it the better starting point for NLP task fine-tuning—though practitioners should verify whether their target task type overlaps with xP3 to avoid the code-generation-style degradation documented in Table 9. The contrastive finetuning recipe (SGPT-BLOOM in Section 3.5) further provides a path for practitioners needing text embeddings for retrieval, clustering, or semantic search in languages beyond English—Table 10 shows state-of-the-art performance on several multilingual STS and classification benchmarks, making this immediately actionable.

Multilingual bias and safety auditing infrastructure. BLOOM's public availability enables systematic bias and safety evaluation across languages at a scale that proprietary models do not permit. The paper's CrowS-Pairs evaluation (Section 4.10) covers only English and French, but the model is accessible for researchers to extend this evaluation to Arabic, Hindi, Chinese, Swahili, and other supported languages—constructing or adapting bias evaluation datasets and measuring model behavior. The finding that BLOOM is "slightly more toxic than average in English" (HELM, Figure 10) while being "one of the best models for fairness" demonstrates that safety properties are multidimensional and language-dependent, and the 44 unevaluated languages represent a substantial blind spot. A multilingual moderation team at a social media platform could use BLOOM as a testbed for developing toxicity detection and content moderation classifiers in low-resource languages—training classifiers on BLOOM embeddings (using SGPT-BLOOM for feature extraction) and validating against human judgments, with the full model accessible for error analysis and iterative improvement. This is infeasible with API-gated models where the internal representations are inaccessible and the training data is undocumented.

Carbon-aware model development as a template for compute grant design. The paper's Life Cycle Assessment carbon accounting (Section 3.5.1) provides a template that future compute grant programs and academic supercomputing centers can adopt for evaluating project proposals. The finding that idle infrastructure consumption constitutes 55% of total emissions (45 of 81 tons) is directly actionable: compute providers could reduce project emissions by colocating related workloads to minimize idle time, or by designing grant programs that batch multiple projects onto shared infrastructure rather than allocating dedicated nodes that sit idle between training runs. The comparison with OPT in Table 4—where BLOOM's higher energy consumption (433 vs. 324 MWh) produces lower emissions (25 vs. 70 tons) due to grid carbon intensity—provides a concrete metric for grant-makers: rather than asking "how many GPU-hours will this project consume?", ask "what is the carbon intensity of the electricity grid where training will occur, and what is the projected total CO2eq?" For a funding agency deciding between compute allocations at different supercomputing centers, this metric enables environmentally-informed decisions that the conventional GPU-hour accounting obscures. The paper's additional estimate that final BLOOM training is only 37% of total project emissions (~219 tons total) warns that carbon accounting must cover the full research lifecycle—ablation runs, evaluation, failed experiments—not just the final training run, a lesson that applies to any large-scale ML project regardless of model architecture.