Scaling Pre-training to One Hundred Billion Data for Vision Language Models

🎯 Pitch

Scaling vision-language models to a staggering 100 billion image-text pairs hardly budges standard Western benchmarks, but it dramatically boosts recognition of long-tail cultural concepts and low-resource languages. The study also reveals that common quality filters, while improving Western metrics, systematically erase this cultural diversity, making raw scale essential for inclusive AI.

1. Executive Summary

This paper empirically investigates the impact of scaling vision-language model pre-training data by one order of magnitude — from 10 billion to 100 billion image-text pairs — using the newly introduced WebLI-100B dataset and SigLIP contrastive models across ViT-B/16, ViT-L/16, and ViT-H/14 architectures. The central finding is that performance saturates on traditional Western-centric benchmarks (zero-shot classification on ImageNet, retrieval on COCO Captions) at this extreme scale, while tasks measuring cultural diversity (geo-localization on Dollar Street, zero-shot classification on GeoDE and GLDv2) and multilinguality (Crossmodal-3600 retrieval in low-resource languages) show statistically significant gains — for instance, ViT-L/16 Dollar Street 10-shot accuracy improves by 5.8 absolute percentage points from 10B to 100B examples versus less than 1% on Western-centric 10-shot tasks. The paper further demonstrates that quality filtering via CLIP-based methods, while beneficial for Western benchmarks, inadvertently reduces cultural representation even in 100-billion-scale datasets, establishing that raw data scale is essential for inclusive multimodal systems precisely when filter-based curation would otherwise discard long-tail cultural concepts.

2. Context and Motivation

The Core Question: What Happens When We 10× the Largest VLM Datasets?

The paper tackles a deceptively straightforward question with enormous practical implications: if the largest reported vision-language datasets plateau around 10 billion image-text pairs, what further benefits — if any — unlock when we push to 100 billion? This is not an idle curiosity. The relationship between data scale and model performance in machine learning has historically followed power-law scaling (the error rate $E(N) \propto N^{-c} + \varepsilon$ , where $N$ is dataset size), as documented across domains from neural language models (Kaplan et al., 2020) to vision transformers (Zhai et al., 2022) and contrastive language-image models (Cherti et al., 2023). Power laws imply that each additional order of magnitude of data continues to yield improvements, albeit diminishing ones. If this pattern holds, going from 10B to 100B image-text pairs should produce meaningful — perhaps practically worthwhile — accuracy gains across the board.

The paper exists because no one knows whether this extrapolation actually holds, and the answer has direct consequences for how organizations allocate billion-dollar training budgets. Should a team invest in scaling web crawling infrastructure to collect 100 billion examples? Or have we already reached a regime where the cost of collecting, storing, and training on that data exceeds the marginal benefit? The question matters because the world's web data is already sufficiently vast — the Common Crawl repository (Common Crawl, 2021) contains vastly more data than has ever been assembled into a single vision-language dataset — meaning the constraint is not data availability but the decision to invest in harnessing it.

Why This Problem Matters Beyond Benchmark Scores

The paper's motivation extends beyond the narrow question of whether ImageNet zero-shot accuracy improves by another half a percent. It addresses a structural tension in the field of vision-language modeling that has emerged as datasets have grown:

The filtering dilemma. As web-scale datasets expanded from millions (Conceptual Captions; Sharma et al., 2018) to billions (LAION-5B; Schuhmann et al., 2022; WebLI-10B; Chen et al., 2022) of examples, practitioners discovered that raw web data is noisy — many image-text pairs are misaligned, low-quality, or irrelevant. The standard remedy is filtering: using models like CLIP to score and retain only high-quality aligned pairs. This filtering demonstrably improves performance on canonical benchmarks like ImageNet classification and COCO retrieval (Fang et al., 2023; Abbas et al., 2023). But filtering is not a neutral operation — it is a selection mechanism that prioritizes certain kinds of content over others. If the filter is trained predominantly on English-language, Western-centric data (as CLIP was), it may inadvertently discard image-text pairs that represent non-Western cultural contexts, low-resource languages, or long-tail concepts. As Nguyen et al. (2024) and Pouget et al. (2024) have shown, popular filtering techniques can diminish cultural understanding in VLMs.

The paper positions 100-billion-scale raw, minimally filtered data as a potential resolution to this dilemma. The intuition is that when you have 100 billion examples, you can afford to retain data that would be filtered out in a 5-billion-sample quality-filtered dataset — the long-tail concepts survive because the sheer volume of data gives them statistical presence even after inevitable noise. This is a fundamentally different argument for data scaling than "more data = better benchmarks." It says: scaling raw data preserves diversity that filtering destroys, even if it doesn't help (and might slightly hurt) on canonical Western benchmarks.

The inclusivity crisis in vision-language models. A growing body of work has documented that VLMs exhibit systematic biases in their visual understanding. Goyal et al. (2022) showed that vision models pretrained on uncurated images without supervision are more robust and fair than those trained on curated datasets. Ananthram et al. (2024) diagnosed "Western cultural bias" in large VLMs, finding that models systematically fail to recognize non-Western objects, scenes, and cultural practices. Birhane et al. (2021) revealed that multimodal datasets contain misogyny, pornography, and malignant stereotypes that propagate into trained models. Alabdulmohsin et al. (2024) demonstrated that CLIP exhibits strong gender-occupation association biases that require explicit mitigation strategies like data balancing.

The root cause is not mysterious: when datasets are predominantly sourced from English-language Western web content (English represents approximately 35% of web pages in the WebLI-100B distribution, per Appendix F), and when quality filters are trained on that same content distribution, the resulting models learn a vision of the world that is skewed toward the cultural contexts and visual concepts prevalent in Western industrialized nations. Dollar Street (Rojas et al., 2022) — a dataset of household items photographed across 63 countries — makes this starkly visible: models struggle to recognize objects from lower-income regions because their training data underrepresents these contexts.

The paper connects data scale directly to this inclusivity problem. Its central claim is that scaling to 100 billion raw examples meaningfully improves performance on cultural diversity benchmarks, and that this is not achievable with smaller filtered datasets because filtering removes the very examples that provide cultural coverage. This is an important reframing: data scale becomes not just a performance lever but an inclusivity lever.

Where Prior Work Falls Short

The paper identifies specific gaps in the existing literature that motivate its investigation:

1. Dataset scale has plateaued around 10B, with no empirical study of the next order of magnitude. The largest reported vision-language datasets prior to this work are LAION-5B (Schuhmann et al., 2022) — 5 billion publicly released image-text pairs — and WebLI (Chen et al., 2022) — 10 billion image-text pairs used internally at Google DeepMind but never described at larger scale. These datasets have been instrumental in training state-of-the-art models including PaLI (Chen et al., 2022), PaLI-3 (Chen et al., 2023), SigLIP (Zhai et al., 2023), and PaliGemma (Beyer et al., 2024). But the Scaling Laws literature — most prominently Kaplan et al. (2020) and Hoffmann et al. (2022) in language, and Cherti et al. (2023) in vision-language — rigorously characterizes the relationship between data size and performance, and these characterizations suggest that there should be some benefit to going past 10 billion. The question is: what kind of benefit, on what tasks, and is it worth it?

2. No systematic investigation of what scaling helps (and what it doesn't). Prior scaling studies in vision-language models (Cherti et al., 2023; Pham et al., 2023) have measured aggregate performance across Western-centric benchmarks — ImageNet zero-shot, COCO retrieval, standard VQA tasks. The paper points out that these benchmarks are overwhelmingly English-language and culturally homogeneous, representing objects, scenes, and concepts most common in Western industrialized societies. If data scaling yields diminishing returns on these benchmarks (which the paper empirically demonstrates), the natural conclusion would be "scaling past 10B isn't worth it." But this conclusion may be an artifact of which benchmarks we evaluate on — the benchmarks themselves are saturated or approaching saturation for concepts well-represented in the training data, while failing to measure the long-tail concepts where additional data would make the biggest difference.

The paper is the first to ask: if we evaluate data scaling on benchmarks explicitly designed to measure cultural diversity and multilinguality, do the scaling curves look different? The answer, per the results, is a clear yes — scaling from 10B to 100B shows statistically significant gains on Dollar Street, GeoDE, and Crossmodal-3600 low-resource languages, while Western benchmarks are essentially flat (p-value of 0.9 for Western-centric tasks, vs. 0.002 for cultural diversity tasks).

3. Quality filtering is known to help Western benchmarks, but its effect on diversity at web-scale is uncharacterized. Multiple data-centric works have shown that CLIP-based filtering, deduplication, and dataset pruning improve downstream accuracy on standard benchmarks (Fang et al., 2023; Abbas et al., 2023; Maini et al., 2023). Pouget et al. (2024) recently demonstrated that filtering reduces cultural diversity in contrastive VLMs trained on WebLI-10B. But Pouget et al. (2024) studied filtering at the 10-billion scale using a fixed training budget (a single epoch over 10B examples). The present paper extends this analysis by asking a different counterfactual: what if we take a 100-billion-scale raw dataset and filter it down to the same size as a high-quality 5-billion-sample subset? Specifically, does the filter create an unfair tradeoff — improving Western performance at the cost of cultural diversity — even when the unfiltered source contains 100 billion examples and thus should have ample cultural content? The finding (Section 5.1, Figure 4) that CLIP-filtered 5B datasets underperform a random English 5B subset on cultural diversity benchmarks, while outperforming it on Western metrics, directly demonstrates that the filter itself is the source of lost diversity, not the absence of diverse content in the source data.

4. The relationship between data scale and model fairness is unexplored. The fairness literature in vision-language models has focused predominantly on mitigation strategies — data balancing (Alabdulmohsin et al., 2024), algorithmic interventions, or architectural modifications. The paper asks a simpler, more fundamental question: does raw data scale, absent any explicit bias mitigation, improve fairness metrics? The results (Section 4.4, Table 5) show that while representation bias (the preference to associate random images with "Male" over "Female") and association bias (stereotypical occupation-gender mappings) are not reduced by scaling — aligning with Alabdulmohsin et al. (2024)'s finding that explicit balancing is necessary — performance disparity across geographic regions and income levels does improve with data scaling. This is a nuanced finding that complicates the simple narrative that "uncurated web data always contains harmful biases that scale with dataset size."

How This Paper Positions Itself

The paper explicitly frames itself as an empirical investigation rather than a method proposal. The contribution statement (Section 1) is precise about this:

"Our goal in this paper is to answer the following question: should one invest in scaling up the size of the pretraining dataset to 100 billion examples?"

The answer, refined through the paper's results, is effectively: it depends on what you care about. If your evaluation suite consists exclusively of Western-centric benchmarks (ImageNet, COCO, Flickr, standard VQA), scaling from 10B to 100B raw web data probably isn't justified — the gains are minimal and may not survive statistical significance testing. But if your objective is to build inclusive multimodal systems that perform equitably across cultures, languages, and geographic regions, 100-billion scale is essential — and, critically, quality filtering cannot substitute for it.

This positions the paper at the intersection of three research threads:

Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022; Cherti et al., 2023): the paper extends scaling analysis beyond aggregate Western-centric metrics to disaggregated cultural and linguistic dimensions, showing that scaling laws are task-dependent — the exponent and asymptotic limit differ systematically between Western benchmarks and cultural diversity benchmarks (see Table 2 vs. Table 3).
Data-centric AI / dataset curation (Fang et al., 2023; Abbas et al., 2023; Gadre et al., 2023): the paper provides a cautionary result: quality filters optimized for aggregate performance may damage cultural representation at scale, making raw data scale the only viable path to inclusivity until new filtering techniques are developed that explicitly preserve diversity.
Fairness and inclusivity in VLMs (Goyal et al., 2022; Pouget et al., 2024; Alabdulmohsin et al., 2024; Nguyen et al., 2024): the paper adds data scale as a new axis in the inclusivity toolkit — not a replacement for bias mitigation strategies (which remain necessary, as the representation and association bias results show), but a complementary lever for improving performance on underrepresented cultural and linguistic groups.

The paper also explicitly connects to a broader trend in the field: the shift from model scaling (bigger architectures, more parameters) to data scaling (more training examples, better coverage). This mirrors the argument of Chinchilla (Hoffmann et al., 2022) in language modeling — that for a fixed compute budget, it's better to train a smaller model on more data than a larger model on less data. The present paper extends this intuition to the diversity axis: for building inclusive VLMs, it's better to train on 100 billion raw examples (sacrificing some per-sample quality) than to filter 100 billion down to a cleaner 5 billion subset (sacrificing cultural coverage). The compute budget is the same in both cases (both datasets are trained to completion), but the outcomes differ qualitatively.

The Scale Gap Being Addressed

To appreciate what a 10× jump represents, consider the magnitudes involved. The largest prior datasets — LAION-5B and WebLI-10B — represent years of web crawling and filtering effort. WebLI-100B is a full order of magnitude larger. Per the paper's description (Section 3.1), the 1B and 10B subsets are created by randomly sampling 1% and 10% from the full 100B dataset, ensuring comparability by controlling for data distribution — the subsets are proper subsets with identical collection methodology, not independently curated datasets of different sizes. This is a deliberate experimental design choice: it isolates the effect of dataset size from confounding factors like crawling methodology, language distribution, or annotation quality.

The paper trains all models to a fixed budget of 100 billion seen examples, meaning models train for 100 epochs on 1B data, 10 epochs on 10B data, and 1 epoch on 100B data. This is a compute-matched comparison — all models see the same number of training examples and consume the same number of FLOPs, but they differ in the diversity (number of unique examples) versus repetition (number of epochs) balance. The fact that the 100B (1 epoch) models match or outperform the 10B (10 epoch) models on cultural diversity, while the 10B models match the 100B models on Western benchmarks (Table 2 vs. Table 3), tells us something specific: for Western-centric concepts, 10 epochs over 10B unique examples provides sufficient coverage — the model has already seen enough "cat" and "car" images to learn those concepts — but for culturally diverse concepts (traditional Igorot dance, Bengali New Year celebrations, Telugu text), repetition of the limited examples in the 10B subset cannot substitute for genuine exposure to unique examples from these long-tail distributions. This is the key mechanism the paper identifies: data diversity, not just data quantity, drives the gains on cultural diversity metrics, and achieving that diversity at the long tail requires web-scale data.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical scaling study, not a method proposal. The "system" is a series of SigLIP contrastive vision-language models (ViT-B/16, ViT-L/16, ViT-H/14) trained on three different scales of the WebLI dataset — 1 billion, 10 billion, and 100 billion image-text pairs — to isolate the effect of dataset size on model performance. The problem it solves is a measurement problem: we lack empirical evidence about whether scaling vision-language pre-training data by an order of magnitude past the current ~10 billion frontier yields meaningful benefits, and — critically — whether those benefits manifest uniformly across task types or concentrate in culturally and linguistically diverse evaluations. The solution shape is a carefully controlled compute-matched comparison where all models process the same total number of training examples (100 billion seen examples), but differ in unique data diversity (100 epochs over 1B vs. 10 epochs over 10B vs. 1 epoch over 100B), evaluated across a deliberately broad spectrum of benchmarks spanning Western-centric, cultural diversity, multilingual, and fairness metrics.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has five major components:

WebLI-100B Dataset — A raw web-crawled corpus of 100 billion image-text pairs with only essential safety filters applied (harmful content removal, PII removal, near-duplicate removal against evaluation sets). Serves as the unified source from which 1B and 10B subsets are randomly sampled (1% and 10% respectively), ensuring distributional comparability across scales.
Optional Quality Filters — Two filtering pipelines (CLIP-L/14 scoring, and a custom web-trained classifier) that produce 5-billion-sample English-only subsets from the raw data, used only in the filtering analysis (Section 5.1), not in the primary scaling experiments.
SigLIP Contrastive Training Pipeline — The core training apparatus: a dual-encoder architecture (separate Vision Transformer and text Transformer) trained with sigmoid loss on image-text pairs. Configurations span three ViT sizes (B/16, L/16, H/14) and three data scales (1B, 10B, 100B). All models train to a fixed budget of 100 billion seen examples using a batch size of 32K, an inverse square root learning rate schedule with 200M warmup/cooldown examples, learning rate 0.001, weight decay 0.0001.
Evaluation Suite — Four categories of benchmarks: (a) Western-centric (ImageNet, CIFAR-100, Pet zero-shot; 10-shot classification on 8 datasets; COCO/Flickr retrieval), (b) Cultural diversity (Dollar Street zero-shot and 10-shot geo-localization; GeoDE classification at country and region level; GLDv2 landmark recognition), (c) Multilingual retrieval (Crossmodal-3600 across 36 languages, reporting separately for 7 low-resource and 28 high-resource languages), (d) Fairness (representation bias, association bias on FairFace, performance disparity by income on Dollar Street and by region on GeoDE).
Transfer Assessment (PaliGemma) — The contrastively trained vision encoders are frozen or unfrozen and plugged into the PaliGemma generative VLM, then fine-tuned on downstream tasks (captioning, VQA, OCR, multilingual, remote sensing) to assess whether pre-training data scale benefits transfer to generative settings.

Information flows as follows: raw web crawl → minimal filtering → WebLI-100B → random sampling to create 1B/10B subsets → SigLIP contrastive pre-training at each scale (all models see 100B total examples) → evaluation on the full benchmark suite → optional filtering analysis on 5B subsets → optional PaliGemma transfer.

3.3 Roadmap for the Deep Dive

First, the WebLI-100B dataset construction and the critical design choice to apply minimal filtering — since this decision is the linchpin of the paper's inclusivity argument and determines what cultural content survives into training.
Second, the SigLIP contrastive training configuration, including the sigmoid loss function (since it differs from the more common InfoNCE/softmax contrastive loss), the training hyperparameters, and the compute-matched comparison framework — because these define what is being held constant across scales.
Third, the evaluation protocol, particularly the cultural diversity and multilingual benchmarks (Dollar Street, GeoDE, GLDv2, Crossmodal-3600), since these are less familiar to many readers and are central to the paper's claims.
Fourth, the scaling law fitting methodology, since the paper uses fitted exponents and asymptotic limits (reported in Tables 2-3) to quantify whether observed performance gaps are expected to grow or shrink with additional compute, and the statistical testing framework (Wilcoxon signed rank) that validates the significance of differences.
Fifth, the data quality filtering analysis configuration (CLIP filter, web-trained classifier, random English baseline), since Section 5.1 tests the interaction between filtering and cultural diversity.
Sixth, the language rebalancing experiment (Section 5.2), where low-resource languages are upsampled to 1% each, since it probes whether the observed low-resource language gains from scaling can be further amplified.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical scaling analysis paper whose core idea is that the benefits of scaling vision-language pre-training data from 10 billion to 100 billion examples are task-dependent: negligible on Western-centric benchmarks but statistically and practically significant on cultural diversity, multilingual, and fairness metrics, driven by the long-tail coverage that raw web scale provides and that quality filtering destroys.

WebLI-100B Dataset Construction and the Minimal Filtering Philosophy

The foundation of all experiments is WebLI-100B, a dataset of 100 billion image-text pairs sourced from the public web. The construction methodology follows the approach of Chen et al. (2022), Jia et al. (2021), Schuhmann et al. (2022), and Zhai et al. (2022): crawl publicly available web pages, extract images and their associated text (both alt-text — the HTML alt attribute describing the image — and page title), and pair them.

What makes WebLI-100B distinct is not the crawling methodology but the filtering philosophy. The paper explicitly states (Section 3.1):

"In this work, we apply only essential data filters, such as removing harmful images and personally identifiable information (PII). This approach ensures the dataset remains as multilingual and diverse as possible."

This is a deliberate design choice that trades per-sample quality for cultural coverage. In most prior work — LAION-5B (Schuhmann et al., 2022), DataComp (Gadre et al., 2023), Conceptual Captions (Sharma et al., 2018) — datasets undergo extensive filtering: CLIP-based alignment scoring, resolution thresholds, language detection and English-only retention, deduplication, and various heuristic quality rules. These filters improve downstream benchmark performance (as the paper confirms in Section 5.1) but systematically remove content from non-English languages, non-Western cultures, and long-tail visual concepts that happen to have noisy or misaligned text. By stripping filtering down to safety essentials only, WebLI-100B retains whatever cultural and linguistic diversity the web contains, at the cost of including many low-quality or misaligned pairs.

The paper does not detail the exact harmful content or PII removal procedures, which is a practical limitation for reproducibility, but the design intent is clear: maximize cultural coverage by minimizing curation.

Subset construction for scaling experiments. The 1 billion and 10 billion subsets are created by randomly sampling 1% and 10% respectively from the full 100 billion dataset. This is a critical experimental control: it ensures that the data distribution is identical across scales (same language mix, same cultural content proportions, same noise characteristics), isolating the effect of unique example count (diversity) from confounding factors like crawling source bias or filtering intensity. An alternative approach — constructing independently curated datasets of different sizes — would conflate scale with curation differences. By random sampling from a common source, the paper cleanly measures what happens when you simply see more unique web content.

Near-duplicate removal against evaluation sets. The paper states:

"To ensure fair evaluations, we remove near-duplicate images across more than 90 common vision-language tasks from our dataset."

This is standard practice to prevent training-set contamination of evaluation benchmarks, but the paper does not specify the deduplication threshold (e.g., perceptual hash distance, CLIP embedding cosine similarity) or the list of the "more than 90" tasks. The intent is to ensure that reported improvements on, say, Dollar Street reflect genuine generalization rather than memorization of evaluation images seen during training. This is particularly important for the paper's claim that 100B scale improves cultural diversity metrics — if the 100B dataset contained near-duplicates of Dollar Street images while the 10B subset did not (due to random sampling), the comparison would be unfair.

Language distribution (Appendix F). The dataset's language composition is documented in Table 12. English dominates at 35.4% of pages, followed by Japanese (8.8%), Spanish (8.2%), Russian (7.0%), German (3.9%), and so on. The seven languages designated as "low-resource" — Maori (0.001%), Telugu (0.036%), Swahili (0.046%), Filipino (0.111%), Bengali (0.113%), Hebrew (0.240%), and Hindi (0.267%) — collectively represent only 0.814% of the data. This extreme skew is important context for the multilingual results in Section 4.3 and the rebalancing experiment in Section 5.2: the gains observed for low-resource languages from scaling to 100B occur despite these languages remaining a tiny fraction of the training data, suggesting that even a 0.814% share of 100 billion examples provides enough absolute examples (roughly 814 million) for meaningful learning gains compared to a 0.814% share of 10 billion (81.4 million).

The "alt-text and page title" text pairing. The paper uses both alt-text (the HTML attribute providing a textual description of an image, intended for accessibility and SEO) and page title (the content of the HTML <title> tag) as the paired text for each image. This is specified in Section 3.1: "We utilize both the alt-text and page title associated with each image as the paired text." This is a common practice in web-scale vision-language datasets, but it has implications: alt-text and page titles are often noisy, only loosely related to image content, or in a language different from the visual content. By retaining both rather than selecting one, the dataset maximizes the text signal available per image, accepting that some fraction will be low-quality.

SigLIP Contrastive Training: The Sigmoid Loss Formulation

The paper trains all models using SigLIP (Sigmoid Loss for Language Image Pre-training), introduced by Zhai et al. (2023), rather than the more common CLIP-style softmax contrastive loss. Understanding why SigLIP is used — and what its loss function computes — is important because the training objective shapes what the model learns from noisy web data.

The CLIP softmax loss (what SigLIP replaces). In standard CLIP training (Radford et al., 2021), a batch of $N$ image-text pairs produces an $N \times N$ similarity matrix where each row represents one image's cosine similarity to all $N$ texts, and each column represents one text's similarity to all $N$ images. The loss encourages each image to be maximally similar to its paired text and minimally similar to the $N-1$ other texts (and vice versa for texts). This is implemented as a symmetric cross-entropy loss over the softmax-normalized similarity matrix:

$\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^N \left[ \log \frac{\exp(\tau \cdot \text{sim}(I_i, T_i))}{\sum_{j=1}^N \exp(\tau \cdot \text{sim}(I_i, T_j))} + \log \frac{\exp(\tau \cdot \text{sim}(I_i, T_i))}{\sum_{j=1}^N \exp(\tau \cdot \text{sim}(I_j, T_i))} \right]$

where $\text{sim}(I, T)$ is the cosine similarity between image and text embeddings, and $\tau$ is a learned temperature parameter. The key property of this loss is that it is batch-dependent: the denominator sum is over all $N$ negative examples in the batch, making the loss sensitive to batch composition. Larger batches provide more negatives, which generally improves learning, but this coupling between batch size and negative count introduces practical constraints — scaling batch size requires scaling hardware, and the loss behavior changes with batch size.

The SigLIP loss (what this paper uses). SigLIP reformulates the contrastive learning objective as a set of independent binary classification problems. For a batch of $N$ image-text pairs, it constructs:

$N$ positive pairs — the correctly matched image-text pairs $(I_i, T_i)$ , which should receive label $y = 1$ .
$N^2 - N$ negative pairs — all mismatched combinations $(I_i, T_j)$ for $i \neq j$ , which should receive label $y = 0$ .

The loss for a single pair with true match label $y \in \{0, 1\}$ is:

$\mathcal{L}_{\text{SigLIP}}(i, j) = -\left[ y_{ij} \cdot \log \sigma(z_{ij}) + (1 - y_{ij}) \cdot \log (1 - \sigma(z_{ij})) \right]$

where $z_{ij} = \tau \cdot \text{sim}(I_i, T_j) + b$ is the logit — the temperature-scaled cosine similarity plus a learned bias term $b$ — $\sigma(\cdot)$ is the sigmoid function mapping logits to $[0, 1]$ , and $y_{ij} = 1$ if $i = j$ (positive pair) and $0$ otherwise.

The total loss sums over all pairs in the batch:

$\mathcal{L}_{\text{total}} = -\frac{1}{N^2} \sum_{i=1}^N \sum_{j=1}^N \left[ y_{ij} \cdot \log \sigma(z_{ij}) + (1 - y_{ij}) \cdot \log (1 - \sigma(z_{ij})) \right]$

What it computes: For each image-text pair in the batch, the model produces an embedding for the image (via the ViT encoder) and an embedding for the text (via the text Transformer encoder). Their cosine similarity is computed, scaled by a learned temperature $\tau$ , and shifted by a learned bias $b$ to produce a logit. The sigmoid function converts this logit to a predicted probability that the pair is matched. The binary cross-entropy loss then compares this prediction against the ground truth (1 for matched pairs, 0 for mismatched pairs). The model is penalized both for assigning low probability to correctly matched pairs and for assigning high probability to mismatched pairs.

Why this form: The SigLIP loss has a crucial advantage over the softmax loss for large-scale training: it is batch-independent in its per-pair computation. Each pair's contribution to the loss depends only on that pair's own similarity and label, not on the other pairs in the batch. This means:

Batch size can be scaled up without altering the loss semantics — doubling the batch size doubles the number of negative examples in the loss summation (each new image pairs with all existing texts and vice versa), but each individual pair's loss term is unchanged. In the softmax loss, doubling the batch changes each row's normalization constant (the denominator changes from a sum over N to a sum over 2N), which changes the per-pair gradient.
Training is more robust to noisy negative pairs — if a "negative" pair (image $i$ with text $j$ , $i \neq j$ ) happens to be semantically related (e.g., two different images of cats with "cat" captions), the softmax loss forces the model to push their similarity to zero (since only one text can win the softmax competition per image). The sigmoid loss treats each negative independently, so the penalty for a false negative is contained to that single pair rather than distorting the entire row's normalization.
The learned bias $b$ provides an adaptive prior — it can learn that most random image-text pairs have low similarity (shifting the decision boundary to be skeptical of matches), which is appropriate for web data where mismatches vastly outnumber matches. This is a trainable parameter that the model adjusts to the noise level of the data.

The paper uses SigLIP specifically because it is the training objective used by the "state-of-the-art models like PaliGemma and SigLIP" (Section 1), and the authors' prior work (Zhai et al., 2023) showed it performs competitively with or better than softmax contrastive loss while being simpler to scale. The choice is consistent with the paper's goal of a controlled scaling study: use a well-understood, production-proven training recipe and vary only the data scale.

Training hyperparameters (Section 3.2). The paper specifies the following concrete configuration, which is held constant across all model sizes and data scales:

Hyperparameter	Value
Batch size	32,000
Learning rate schedule	Inverse square root
Warmup examples	200 million
Cooldown examples	200 million
Learning rate	0.001
Weight decay	0.0001
Image resolution	224 × 224 pixels
Text tokenizer	Multilingual mT5 (Xue, 2020)
Maximum text sequence length	64 tokens
Maximum training examples seen	100 billion

The inverse square root learning rate schedule (or "inverse sqrt" schedule) means the learning rate at training step $t$ scales proportionally to $1 / \sqrt{t}$ after warmup, with a linear cooldown to zero at the end of training. This is the same schedule used by Zhai et al. (2022) for scaling vision transformers. The warmup phase linearly increases the learning rate from 0 to the peak value over the first 200 million examples; the cooldown phase linearly decreases it from the schedule's value to 0 over the final 200 million examples.

The mT5 tokenizer (Xue, 2020) is a multilingual text tokenizer supporting 101 languages, which is essential for the paper's multilingual experiments — using an English-only tokenizer (like BPE trained on English Wikipedia) would either fragment non-English text into subwords or produce out-of-vocabulary tokens, preventing the model from learning meaningful representations for low-resource languages.

The maximum sequence length of 64 tokens constrains how much alt-text and page title content the model can process. Web alt-text and page titles vary widely in length, and truncating to 64 tokens introduces a trade-off: very long, descriptive alt-texts get cut off, potentially losing detail important for fine-grained visual concepts. The paper does not discuss this truncation's effect on cultural diversity (longer captions might describe culturally specific details that get truncated).

The Compute-Matched Comparison Framework

A central design principle of the paper's experiments is that all models are compared under a fixed total training budget of 100 billion seen examples. This means:

1B data, 100 epochs: The model sees each of 1 billion unique image-text pairs 100 times over the course of training, for a total of 100 billion examples processed.
10B data, 10 epochs: The model sees each of 10 billion unique pairs 10 times, also totaling 100 billion examples.
100B data, 1 epoch: The model sees each of 100 billion unique pairs once, totaling 100 billion examples.

All three regimes consume the same amount of training FLOPs (same total examples seen × same model architecture × same per-example FLOP cost), making the comparison compute-matched. Any performance differences between, say, the 10B/10-epoch model and the 100B/1-epoch model cannot be attributed to one model having received more total training computation — they must arise from the tradeoff between unique data diversity (more distinct examples) and per-example repetition (more epochs over fewer examples).

This design choice is explicit and important because prior scaling studies have sometimes compared models trained to different total budgets, making it impossible to separate the effect of data scale from the effect of more training steps. By holding total compute constant, the paper isolates the marginal benefit of data diversity at the expense of repetition.

The paper further evaluates models at intermediate checkpoints — specifically at 3, 7, 10, 17, 26, 33, 49, 66, and 100 billion seen examples — and reports results after a cooldown phase at each checkpoint. This enables the scaling law fitting (discussed below) by providing datapoints along the training curve, and it also allows readers to see whether models at smaller data scales (with many repetitions) eventually catch up to models at larger data scales (with one pass each) — which would suggest that repetition can substitute for diversity for some concepts.

The implicit tradeoff being measured. The experimental design probes a fundamental tension in machine learning: is it better to see more unique data once, or to see less unique data multiple times? For frequent concepts — common objects like "cat," "car," "person" — repetition likely suffices because the 10B subset already contains millions of examples of these concepts; seeing each cat image 10 times versus 1 time may not change the learned representation much. For rare concepts — an "Igorot dance," a "Bengali New Year celebration," a specific traditional garment — the 10B subset may contain only a handful of examples (or zero), while the 100B dataset contains many more. In this case, no amount of repetition over the scarce examples in the 10B set can substitute for genuine exposure to diverse instances of the rare concept. The results in Tables 2 and 3 confirm this hypothesis: Western-centric benchmarks (dominated by frequent concepts) show negligible differences between 10B and 100B, while cultural diversity benchmarks (which depend on long-tail concepts) show significant gains.

The paper describes checkpoint selection as follows (Section 3.2):

"We cool down the models at various training steps where they have seen 3, 7, 10, 17, 26, 33, 49, 66, and 100 billion examples, and evaluate them after the cool-downs. Unless otherwise specified, we report results using the checkpoints where models have been trained on 100 billion examples."

The cooldown phase is a linear decay of the learning rate to zero, which stabilizes model parameters before evaluation. This is standard practice in large-scale training where the learning rate schedule matters for final performance.

Evaluation Protocol: Benchmarks Spanning Western, Cultural, Multilingual, and Fairness Dimensions

The paper's evaluation suite is deliberately comprehensive, spanning four domains. Here we detail each benchmark, what it measures, and why it was chosen.

Western-Centric Benchmarks

Zero-shot classification is evaluated on three datasets: ImageNet (Deng et al., 2009) — 1,000 object categories spanning everyday objects, animals, and scenes; CIFAR-100 (Krizhevsky et al., 2009) — 100 object categories at 32×32 resolution; and Oxford-IIIT Pet (Parkhi et al., 2012) — 37 cat and dog breeds. In zero-shot classification, the model is not fine-tuned on any task-specific data; instead, it receives the class names as text prompts, encodes them with the text encoder, encodes the test images with the image encoder, and assigns each image to the class whose text embedding has the highest cosine similarity. This directly tests whether the contrastive pre-training has aligned image and language representations in a way that generalizes to unseen categories.

10-shot classification uses 8 datasets: ImageNet, CIFAR-100, Pet, Caltech-UCSD Birds (Wah et al., 2011), Caltech 101 (Li et al., 2022), Cars196 (Krause et al., 2013), Colorectal Histology (Kather et al., 2016), and Describable Textures Dataset (Cimpoi et al., 2014). In the 10-shot setting, the model is given 10 labeled examples per class (not the full training set) and must learn to classify from this limited supervision. This evaluates the quality of the learned visual representations for fine-grained discrimination tasks — a good pre-trained encoder should produce features that are linearly separable given only a few labeled examples.

Zero-shot retrieval uses COCO Captions (Chen et al., 2015) and Flickr30k (Young et al., 2014) in both directions: image-to-text (given an image, retrieve the matching caption from a pool) and text-to-image (given a caption, retrieve the matching image). Performance is measured by Recall@1 — the fraction of queries for which the correct match is ranked first among all candidates. These benchmarks test fine-grained alignment between image regions and textual descriptions, since captions describe specific objects, attributes, and relationships in each image.

The paper reports error rates (100% - accuracy) for all these metrics, which is standard in scaling law analysis because error rates are better modeled by power laws than accuracies (since accuracy is bounded above at 100%, making it harder to fit asymptotic behavior).

Cultural Diversity Benchmarks

These benchmarks are central to the paper's claims and deserve detailed explanation, as they evaluate dimensions that most vision-language papers ignore.

Dollar Street (Rojas et al., 2022) is a dataset of 38,000 images of household items — stoves, toilets, beds, front doors, etc. — photographed across 63 countries spanning a wide range of income levels. The dataset is organized by "topics" (96 specific item categories like "stove," "toilet," "bed") and each image is labeled with the country and approximate monthly income of the household. For zero-shot classification, the paper follows Rojas et al. (2022) by mapping the 96 Dollar Street topics to corresponding ImageNet classes (e.g., "stove" maps to ImageNet classes for kitchen appliances), resulting in a curated subset of 21,000 images. This mapping is necessary because Dollar Street and ImageNet use different class taxonomies; the mapping allows zero-shot evaluation using ImageNet class names as text queries.

However, this mapping introduces a subtle limitation: Dollar Street topics like "toilet" may map to ImageNet classes that poorly represent the visual diversity of toilets worldwide (a squat toilet in rural India looks very different from a Western flush toilet, and the ImageNet class may be biased toward the latter). The zero-shot evaluation therefore indirectly measures whether the model can recognize diverse cultural instantiations of the same high-level concept. Better performance on Dollar Street zero-shot means the model has learned visual representations of "stove" or "toilet" that generalize across cultural and economic contexts, not just the Western-centric examples that dominate ImageNet.

Dollar Street 10-shot geo-localization is a different task: given an image from Dollar Street, predict which country or region it was photographed in, using only 10 labeled examples per country for training. This evaluates whether the model can learn to associate specific visual patterns — architectural styles, clothing, vegetation, household item designs — with geographic regions, which is a proxy for cultural understanding. Geo-localization is a hard task because it requires recognizing subtle cultural markers (e.g., the design of electrical outlets, the style of window frames, the type of cooking vessel) that are rarely explicitly captioned in web data but are implicitly present.

GeoDE (Ramaswamy et al., 2024) is a dataset of 62,000 manually annotated images collected from diverse geographic locations, specifically designed to be geographically balanced (unlike ImageNet, which overrepresents Western countries). The paper evaluates both zero-shot classification (predicting the object category) and 10-shot geo-localization at two granularities: country level (predicting which country the image is from) and region level (predicting which of 6 geographic regions: Africa, Americas, East Asia, Europe, Southeast Asia, West Asia). The region-level task is coarser but provides insight into whether the model has learned macro-level cultural visual signatures.

Google Landmarks Dataset v2 (GLDv2; Weyand et al., 2020) contains 1,542 images representing 884 landmarks across 84 countries. The zero-shot classification task is to recognize which landmark (or which country's landmark) an image depicts. This is a cultural diversity task because landmarks are inherently cultural — recognizing Angkor Wat (Cambodia), the Taj Mahal (India), or the Sydney Opera House (Australia) requires exposure to images of these culturally specific structures during training. Landmark recognition is particularly relevant to the paper's thesis because many landmarks from non-Western countries are long-tail concepts in English-dominated web crawls; a 10-billion-sample dataset might contain thousands of Eiffel Tower images but only a handful of, say, Borobudur Temple images.

Why these benchmarks collectively test cultural diversity. The three benchmarks (Dollar Street, GeoDE, GLDv2) evaluate different but complementary aspects of cultural understanding:

Dollar Street tests recognition of everyday objects in diverse cultural contexts — can the model recognize a "stove" whether it's a gas range in a Canadian kitchen or a clay chulha in a rural Indian household?
GeoDE tests recognition of geographic visual signatures — can the model distinguish visual patterns characteristic of West African markets from those characteristic of Southeast Asian markets?
GLDv2 tests recognition of culturally significant landmarks — does the model know what Angkor Wat looks like, and can it distinguish it from visually similar but culturally distinct structures?

Together, they provide a multi-faceted view of cultural competence that no single benchmark could capture.

Multilingual Benchmark: Crossmodal-3600

Crossmodal-3600 (Thapliyal et al., 2022) is a geographically diverse set of 3,600 images, each accompanied by human-generated captions in 36 languages. The languages span a wide range of resource levels, from English and Spanish (high-resource, abundant web data) to Maori and Telugu (very low-resource, tiny fractions of the web). The paper evaluates zero-shot retrieval in both image-to-text and text-to-image directions for each language. For image-to-text retrieval, the model must find the correct caption in, say, Telugu from a pool of Telugu captions; for text-to-image, it must find the correct image given a Telugu caption.

The paper aggregates results into low-resource languages (7 languages: Bengali, Filipino, Hindi, Hebrew, Maori, Swahili, Telugu) and high-resource languages (the remaining 28), reporting averages for each group. The per-language breakdown is available in Appendix B (Table 8, under XM3600 entries).

The language distribution in the training data (Appendix F, Table 12) provides crucial context: Maori represents only 0.001% of web pages, meaning in the 10-billion subset, there are roughly 100,000 Maori-language pages, while in the 100-billion dataset, there are roughly 1 million. This 10× increase in absolute examples — from barely any signal to a modest but usable amount — is what drives the observed improvements in low-resource language retrieval.

Fairness Benchmarks

The paper evaluates fairness across three dimensions (Section 3.3, Section 4.4):

Representation bias (RB) measures the model's preference to associate randomly chosen images with binary gender labels. The metric, from Alabdulmohsin et al. (2024), computes how often the model assigns a higher similarity to "Male" than to "Female" when shown images from ImageNet. A value of 50% would indicate no systematic preference; values near 85% (as observed in Table 4) indicate a strong bias toward associating generic images with the male label. This is computed using 1st-order statistics of the model's embedding similarities.

Association bias (AB) measures stereotypical associations between gender and occupation. Using the FairFace dataset (Karkkainen and Joo, 2021) — a face dataset with balanced race, gender, and age annotations — the paper computes the probability that the model prefers stereotypical occupation labels (e.g., "secretary" over "manager") when the image depicts a particular gender. This is visualized in Figure 2 as a heatmap for five occupation pairs: librarian vs. scientist, nurse vs. doctor, housekeeper vs. homeowner, receptionist vs. executive, and secretary vs. manager. The metric uses 2nd-order statistics (covariance) between the gender attribute and occupation preference, as defined in Alabdulmohsin et al. (2024).

Performance disparity measures the gap in accuracy between different subgroups. For Dollar Street, the paper reports accuracy broken down by four income levels (0-200 USD/month, 200-685, 685-1998, >1998) and computes the maximum gap across these groups (Table 5). For GeoDE, it reports accuracy broken down by six geographic regions (Africa, Americas, East Asia, Europe, Southeast Asia, West Asia) and similarly computes the maximum gap. Lower disparity means more equitable performance across groups.

The distinction between these three fairness metrics is important. Representation bias and association bias are intrinsic model biases — the model's internal representations encode gendered or stereotyped associations regardless of the task. Performance disparity is an extrinsic fairness metric — it measures whether the model works equally well for different groups on actual downstream tasks. The paper finds (Section 4.4) that data scaling reduces performance disparity but does not reduce representation or association bias, which is a nuanced result: scaling helps the model serve diverse users more equitably, but does not fix the stereotypes encoded in the model's representations (which likely require explicit interventions like the data balancing studied in Alabdulmohsin et al., 2024).

Scaling Law Fitting Methodology

The paper fits power-law scaling curves to the empirical data to quantify how performance changes with dataset size and to estimate asymptotic limits. While the fitting methodology is not described in exhaustive detail in the paper, the approach follows Alabdulmohsin et al. (2022) and is referenced as such:

"we also fit data scaling laws for every combination of model and dataset following the recipe proposed in Alabdulmohsin et al. [2]"

The fitting procedure models the relationship between dataset size $N$ (number of seen examples) and error rate $E(N)$ using a power law:

$E(N) = \alpha \cdot N^{-c} + \varepsilon$

where $\alpha$ is a scale parameter, $c$ is the scaling exponent (determining how quickly error decreases with more data; higher $c$ means faster improvement), and $\varepsilon$ is the asymptotic error limit — the irreducible minimum error the model approaches as $N \to \infty$ , representing the Bayes error rate or fundamental limitation of the model architecture on that task.

What it computes: For each combination of model size (B, L, H), data scale (1B, 10B, 100B subsets), and evaluation metric, the fitting procedure uses the error rates measured at multiple points along the training curve (the checkpoints at 3, 7, 10, 17, 26, 33, 49, 66, and 100 billion seen examples) to estimate the three parameters ( $\alpha$ , $c$ , $\varepsilon$ ) that best fit the observed data. This produces the exponents and limits reported in Tables 2 and 3.

Why this form: The power-law form $E(N) = \alpha N^{-c} + \varepsilon$ is the standard parameterization in neural scaling laws (Hestness et al., 2017; Kaplan et al., 2020; Alabdulmohsin et al., 2022). It captures the empirical observation that error decreases proportionally to some negative power of data size — a straight line on a log-log plot — plus an irreducible asymptotic floor. The exponent $c$ is the key quantity for understanding scaling behavior: large $c$ means the model exploits additional data efficiently, small $c$ means diminishing returns set in quickly. The asymptotic limit $\varepsilon$ reveals whether the model is fundamentally performance-limited (high $\varepsilon$ ) or has room to improve further with even more data (low $\varepsilon$ ).

Interpreting the reported exponents and limits (Tables 2 and 3). For example, ViT-L/16 on ImageNet zero-shot classification shows exponents of -0.92 (1B), -0.91 (10B), and -0.82 (100B), with asymptotic limits around 27-31% error. The negative exponents mean error decreases with more data, but the relatively consistent values across scales suggest the scaling behavior is stable — adding more data continues to help at a similar rate. By contrast, ViT-L/16 on Dollar Street 10-shot shows exponents of -1.09 (1B), -0.38 (10B), and -0.94 (100B), with asymptotic limits around 57-63% error — much higher than ImageNet, indicating that Dollar Street geo-localization is fundamentally harder and the model has more room to improve with additional data.

The paper uses these fitted parameters to predict whether performance gaps between scales are expected to widen or narrow in the infinite-compute regime, and reports:

"we do not observe significant differences at the 95% confidence level (𝑝-value of 0.09)" for Western-centric tasks.

This weak significance (p = 0.09, above the standard 0.05 threshold) supports the paper's central claim that scaling from 10B to 100B does not produce statistically meaningful gains on Western benchmarks.

Wilcoxon Signed-Rank Testing for Statistical Significance

To formally assess whether the observed differences between 10B and 100B model performance are statistically significant, the paper applies Wilcoxon's signed-rank test (Wilcoxon, 1992). This is a non-parametric statistical test that does not assume normally distributed errors, making it appropriate for the bounded, non-Gaussian distribution of accuracy scores across benchmarks.

The test works as follows: for each evaluation metric, the paper computes the difference in performance (10B - 100B) across the three model sizes (B, L, H). It ranks these differences by absolute magnitude, sums the ranks of positive and negative differences separately, and computes a test statistic that measures whether the central tendency of the distribution differs from zero. The $p$ -value is the probability of observing a test statistic at least as extreme under the null hypothesis that there is no systematic difference between the two conditions.

The paper reports:

For Western-centric benchmarks: $p = 0.9$ , indicating no statistically significant difference — the observed small variations (some metrics slightly favoring 10B, some slightly favoring 100B) are consistent with random noise.
For cultural diversity benchmarks: $p = 0.002$ , indicating statistically significant evidence at the 99% confidence level — the observed gains from 100B over 10B are too consistent across metrics and model sizes to be explained by chance.

This statistical framework is important because it acknowledges that individual benchmark comparisons can be noisy (a 0.5% improvement on one task could be random), and instead asks: is the pattern of improvements across the entire suite of cultural diversity metrics systematically favoring the 100B scale? The answer, per the p-value, is a clear yes.

Data Quality Filtering Setup (Section 5.1)

The filtering analysis in Section 5.1 uses three datasets, all containing 5 billion English image-text pairs, allowing a direct comparison of data composition at a fixed size:

1. CLIP-filtered 5B. The raw WebLI-100B is scored by a CLIP-L/14 model (Radford et al., 2021), which computes the cosine similarity between the image embedding and the English alt-text embedding for each pair. Pairs with alignment scores below a threshold (not explicitly stated in the paper) are discarded, and the top 5 billion highest-scoring pairs are retained. This is the standard quality filtering approach used in many prior works (e.g., LAION filtering, DataComp best-pool filtering). The CLIP model was trained predominantly on English-language data, so its alignment scores implicitly favor Western visual concepts and English descriptions.

2. Classifier-filtered 5B. The paper trains a separate binary classifier on the raw web data to distinguish aligned from misaligned image-text pairs. The details of this classifier (architecture, training data construction, threshold tuning) are sparse: the paper states "we train a VLM on the web data to classify image-text pairs as aligned or misaligned, and tune its threshold to retain another filtered dataset of the same size." This alternative filter is intended to test whether the CLIP-specific biases are responsible for the diversity loss, or whether any quality filter — even one trained on the same web data distribution — would have similar effects. The results (Figure 4) show similar patterns for both filters, suggesting the issue is not CLIP-specific but inherent to the act of quality filtering itself.

3. Random English 5B baseline. From the raw WebLI-100B, the paper samples 5 billion English-language pairs at random, without any quality scoring. This serves as a baseline distinguishing the effect of language restriction (English-only) from the effect of quality filtering (CLIP/scoring-based selection). Any difference between the random English baseline and the CLIP-filtered 5B can be attributed to the filter's selection bias rather than the language restriction.

All three datasets are used to train ViT-L/16 SigLIP models (same training recipe as the main experiments) and evaluated on Western-centric, cultural diversity, and fairness benchmarks. The key comparison is visualized in Figure 4, which plots average error rates across seen examples for each dataset.

The implicit hypothesis being tested. If CLIP filtering simply removed noisy or misaligned pairs uniformly across all cultural contexts, we would expect the CLIP-filtered 5B to outperform the random English 5B on all metrics — better data quality should help everywhere. The paper's finding that CLIP-filtered data underperforms random English data on cultural diversity (Figure 4, middle panel) implies that filtering is non-uniform: it disproportionately removes valid image-text pairs from non-Western cultural contexts because those pairs tend to have lower CLIP alignment scores. This could be because CLIP itself was trained on Western-centric data and thus has a poorer understanding of non-Western visual-textual relationships, creating a feedback loop where existing model biases are amplified through data filtering.

Language Rebalancing Setup (Section 5.2)

The language rebalancing experiment addresses a different question from the main scaling study: if low-resource languages benefit from scaling because they gain more absolute examples in a 100B dataset, can we achieve similar or better gains by upsampling those languages within a fixed dataset size?

The rebalancing procedure. From the 35 languages in Crossmodal-3600, the paper identifies 7 as "low-resource" based on their representation in the web data (Appendix F): Bengali (0.113%), Filipino (0.111%), Hindi (0.267%), Hebrew (0.240%), Maori (0.001%), Swahili (0.046%), and Telugu (0.036%). Collectively, these represent only 0.814% of the dataset. The rebalancing intervention is:

Identify all image-text pairs where the web page's content-language meta tag matches one of the 7 low-resource languages.
Upsample each low-resource language to 1% of the training data — for example, Telugu goes from 0.036% to 1% (a roughly 28× increase in representation), Maori goes from 0.001% to 1% (a 1000× increase).
The remaining 93% of the data retains the original distribution of all other languages (including English, high-resource languages, and unidentified languages).

This upsampling is applied during training — the dataset is constructed with the modified mixing ratios, and the model samples from this rebalanced distribution. The total dataset size remains unchanged (100 billion seen examples); only the proportion of different languages shifts.

Why this rebalancing matters. If the gains from scaling to 100B on low-resource languages (Figure 3) are driven purely by the increased absolute number of examples for those languages, then upsampling should produce similar gains without needing the full 100B scale. If upsampling doesn't help (or helps less than scaling), it suggests that the benefits of scale come not just from more examples of low-resource languages, but from the diversity of concepts within those language communities that only emerges when the crawled data is sufficiently large — rare cultural concepts in low-resource languages may simply not exist in the smaller 10B crawl, and upsampling the few examples that do exist cannot create diversity that wasn't captured.

The rebalancing experiment is run at all three data scales (1B, 10B, 100B) to see whether the effect interacts with scale. The results (Figure 5 in the paper, described in Section 5.2) show that rebalancing significantly improves low-resource language retrieval (as expected, since the model sees proportionally more of these languages) while slightly decreasing high-resource language and Western-centric task performance (since those examples are now proportionally fewer). The cultural diversity tasks show a mild improvement, suggesting some coupling between language representation and cultural visual understanding — when the model sees more text in Hindi or Swahili, it also learns to better recognize the visual concepts associated with those language communities.

Transfer to Generative Models: PaliGemma Setup (Section 4.5)

To assess whether pre-training data scale benefits transfer beyond contrastive tasks to generative vision-language applications, the paper integrates the trained SigLIP vision encoders into PaliGemma (Beyer et al., 2024), a "compact and versatile 3B VLM for transfer."

PaliGemma architecture. PaliGemma combines a vision encoder (typically a SigLIP-pretrained ViT) with a language model decoder (Gemma-2B) via a linear projection layer. The vision encoder produces patch-level visual features; these are projected into the language model's embedding space and concatenated with text token embeddings as a prefix to the text sequence. The language model then autoregressively generates text conditioned on the visual prefix.

Transfer setup. The paper explores two common transfer configurations:

Frozen vision encoder: The SigLIP-pretrained ViT weights are kept completely fixed during PaliGemma stage-1 pre-training. Only the linear projection layer and the language model are updated. This tests how well the contrastively learned visual representations transfer to a generative setting without any adaptation — essentially, whether the visual features are sufficiently general-purpose to serve as a drop-in component for a different task family.
Unfrozen vision encoder: The ViT weights are allowed to update during PaliGemma stage-1 pre-training, jointly optimized with the projection layer and language model. This tests whether the pre-training provides a good initialization that can be further adapted to generative tasks — a stronger integration that typically yields better downstream performance but at the cost of more training computation and potential forgetting of contrastive pre-training knowledge.

Stage-1 pre-training recipe. The PaliGemma models are pre-trained for 50 million seen examples at 224×224 resolution, following the "stage-1 recipe" from Beyer et al. (2024). This stage-1 pre-training is on a mixture of multimodal tasks (captioning, visual question answering, OCR) designed to teach the model to attend to visual tokens and produce coherent text conditioned on images.

Downstream evaluation. After stage-1, models are fine-tuned on a diverse set of downstream tasks spanning four categories (Appendix C):

Semantics: Image captioning (COCO, NoCaps, COCO-35L, XM3600 English), visual question answering (OKVQA, AOKVQA, GQA, NLVR2, MARVL, VizWizVQA), counting (TallyQA, CountBenchQA), and referring expression comprehension (RefCOCO variants) — 19 tasks total.
OCR: Document VQA, OCR-VQA, ChartQA, SciCap, AI2D, ScienceQA, InfoVQA, TextCaps, TextVQA, ST-VQA, Screen2Words, WidgetCap — 14 tasks requiring reading and understanding text in images.
Multilinguality: Crossmodal-3600 (average over all 36 languages and over 35 non-English languages), COCO-35L (average over all languages, and English-only), xGQA (multilingual visual question answering) — 5 aggregated metrics.
Remote Sensing: RSVQA-lr, RSVQA-hr (two test splits) — 3 tasks testing domain transfer to satellite and aerial imagery.

The aggregated results (Table 6) average within each category, producing four category-level scores and an overall average. The paper notes "when taking the noise level into consideration, we do not observe consistent performance gains across downstream tasks as we scale the pre-training dataset," which aligns with the main finding that Western-centric and standard VLM benchmarks (which dominate this transfer suite) do not benefit from the 10B→100B scale.

Why this transfer experiment is included. The paper's primary findings are based on contrastive zero-shot and few-shot evaluations, which directly probe the vision-text alignment learned during pre-training. But in practice, many production VLMs use contrastively pre-trained vision encoders as components within generative architectures (Flamingo, PaLI, PaliGemma, LLaVA). The transfer experiment addresses a potential objection: even if data scale doesn't help contrastive zero-shot metrics, maybe it helps the visual representations transfer better to generative tasks where richer visual semantics are needed. The null result — no consistent improvement from 10B to 100B in transfer tasks — reinforces the paper's thesis that traditional VLM benchmarks have saturated with respect to data scale, and the gains are concentrated in the cultural and linguistic dimensions that these benchmarks do not measure.

Summary of Key Design Choices and Their Justifications

Minimal filtering on raw web data rather than quality-filtered curation: preserves cultural and linguistic diversity at the cost of per-sample quality, enabling measurement of diversity-driven gains from scale. The paper's entire argument about inclusivity depends on this choice — if the dataset were heavily filtered, the diversity gains from scale would be attenuated (as Section 5.1 demonstrates).
Random subset construction (1B, 10B from 100B) rather than independent dataset construction: isolates the effect of unique example count by keeping distribution identical across scales. Without this control, any observed differences could be attributed to different crawling methodologies, time periods, or filtering rules.
Compute-matched comparison (100B seen examples for all models) rather than fixed-epoch training: isolates the tradeoff between data diversity (unique examples) and per-example repetition (epochs). This is the cleanest way to answer "should I spend my compute on more unique data or more epochs over existing data?"
SigLIP loss rather than softmax contrastive loss: enables stable large-batch training (32K batch size) with batch-independent per-pair loss computation, and is the production-proven choice from prior work. The paper does not compare SigLIP vs. softmax training — it uses SigLIP throughout to hold the training objective constant.
Diverse evaluation suite spanning four dimensions (Western, cultural, multilingual, fairness) rather than only standard benchmarks: the entire contribution of the paper hinges on evaluating dimensions beyond Western-centric metrics. Without Dollar Street, GeoDE, GLDv2, and Crossmodal-3600, the paper would have found no meaningful gains from 100B scale and would have concluded "scaling past 10B isn't worthwhile" — which would be the wrong conclusion for inclusive system builders.
Wilcoxon signed-rank test for statistical significance rather than per-benchmark t-tests: accounts for the fact that individual benchmark noise can obscure systematic patterns, and provides a single summary statistic for whether the suite of metrics collectively favors one condition over another.
Two-stage transfer evaluation (frozen and unfrozen PaliGemma) rather than only direct contrastive evaluation: tests whether pre-training scale benefits manifest in the generative setting that dominates production VLM deployment, addressing the concern that contrastive metrics might not capture all practically relevant benefits.

4. Key Insights and Innovations

Innovation 1: Data Scaling as an Inclusivity Lever, Not a Performance Lever

The paper's most intellectually distinctive contribution is a fundamental reframing of why we scale datasets. The dominant narrative in the scaling laws literature — from Kaplan et al. (2020) through Hoffmann et al. (2022) to Cherti et al. (2023) — treats data scaling as a mechanism for improving aggregate model performance on canonical benchmarks. More data means lower error rates on ImageNet, better recall on COCO, higher accuracy on VQA. Under this framing, the value of scaling from 10B to 100B examples would be assessed by asking: "How much does ImageNet zero-shot accuracy improve?" The paper's answer — essentially nothing, p = 0.9 — would, under the conventional framing, be a negative result: scaling past 10B isn't worth the investment.

The paper reveals this conclusion as an artifact of which benchmarks we use to evaluate scaling. By including a deliberately broad evaluation suite that spans cultural diversity (Dollar Street, GeoDE, GLDv2) and multilinguality (Crossmodal-3600 in 36 languages), the paper demonstrates that the scaling curve looks entirely different depending on what you measure. On Western-centric benchmarks, the 10B → 100B gain is negligible (Table 2: ViT-L ImageNet 0-shot improves from 29.7% to 28.5% error — a 1.2 percentage point drop that fails statistical significance). On cultural diversity benchmarks, the same data scale jump produces gains of 5.8 percentage points for ViT-L on Dollar Street 10-shot (from 64.1% to 58.3% error) and 4.5 points for ViT-H (from 59.1% to 53.7%), with p = 0.002 confirming the pattern is systematic.

This is not an incremental refinement of scaling methodology. It is a diagnostic insight that redefines what the scaling curve measures: the benefits of scale are concentrated in the long tail of the data distribution — rare concepts, low-resource languages, culturally specific visual patterns — precisely the content that canonical Western benchmarks systematically exclude. The "diminishing returns" observed on ImageNet are not evidence that additional data has no value; they are evidence that ImageNet itself has become a saturated metric for the kinds of concepts that large-scale web datasets are rich in. A dataset of 1 billion examples already contains millions of cat and car images, sufficient to saturate performance on those concepts. The marginal billion examples contribute disproportionately rare cultural content — a Telugu-language caption for a traditional festival image, a Swahili description of a local market scene — that is invisible to Western-centric evaluation.

This insight has direct practical consequences that the field had not articulated before: the decision to scale data is a decision about who your model serves. Organizations optimizing for ImageNet leaderboard position should not invest in 100B-scale raw data — quality-filtered subsets are cheaper and equally effective. Organizations building systems intended for global deployment across diverse cultural and linguistic contexts must invest in this scale, because no amount of filtering or algorithmic intervention can create cultural diversity that was never crawled.

The paper's comparison of 10B (10-epoch) vs. 100B (1-epoch) training under a compute-matched budget — both seeing 100 billion total examples — adds conceptual depth to this finding. The fact that 10 epochs over 10B examples does not match 1 epoch over 100B examples on cultural diversity metrics (while it does match on Western metrics) demonstrates that repetition cannot substitute for unique data diversity on long-tail concepts. For "cat," seeing the same 1,000 cat images 10 times is roughly equivalent to seeing 10,000 distinct cat images once — the concept is broad enough that limited unique examples still provide good generalization. For "Igorot dance" or "Telugu household item," the 10B subset may contain only a single example or none at all, and no amount of repeated viewing of that single example teaches the model what these concepts look like across their natural variation. This is a specific, testable hypothesis about where data diversity matters — in the long tail — that goes beyond the generic claim "more data is better."

Innovation 2: The Filtering-Diversity Tradeoff at Web Scale

The paper's second major conceptual contribution is empirically demonstrating — and quantifying — a tradeoff that the data-centric ML community has been circling but had not systematically measured at this scale: quality filtering for benchmark performance versus cultural diversity are opposing objectives, and the tension does not resolve with more source data.

Prior work had established elements of this picture separately. Pouget et al. (2024) showed that CLIP-based filtering reduces cultural diversity in contrastive VLMs trained on 10B-scale data. Fang et al. (2023) and Abbas et al. (2023) showed that quality filtering improves standard benchmark performance. These findings, taken together, imply a tradeoff — but were obtained in different experimental contexts (different datasets, different training recipes, different filtering methods). The present paper provides the first controlled, side-by-side demonstration using a common source dataset (WebLI-100B), a common training recipe (SigLIP ViT-L), and a common evaluation protocol (Western, cultural, and fairness metrics), with filtering as the only experimental variable.

The design of the filtering comparison (Section 5.1, Figure 4) is what makes the insight land. Three 5-billion-sample English datasets are created from the same 100B source: (1) a random English-language subset (the baseline — controls for language restriction only), (2) a CLIP-filtered subset retaining the top-scoring 5B pairs, and (3) a classifier-filtered subset using a custom web-trained alignment model. The CLIP filter — the standard tool used by LAION, DataComp, and many other large-scale dataset efforts — produces the best Western-centric performance, consistent with a large body of prior work. But it produces the worst cultural diversity performance, and the classifier filter shows the same pattern, ruling out the hypothesis that CLIP-specific biases are the culprit.

The finding that matters conceptually is not just that filtering hurts diversity — that could be explained by filtering removing all noisy content, including culturally diverse but poorly aligned pairs. The more specific finding is that the CLIP-filtered 5B dataset underperforms the random English 5B baseline on cultural diversity (Figure 4, middle panel), despite containing the same number of examples and being drawn from the same source. This means the filtering process does not simply remove noise uniformly; it actively selects against culturally diverse content, assigning lower alignment scores to valid image-text pairs from non-Western contexts because the CLIP model itself was trained on Western-centric data and has a poorer grasp of non-Western visual-textual correspondences.

This is a negative result with significant implications for how the field approaches data curation. The default pipeline for creating large-scale vision-language datasets — crawl the web, filter with CLIP, train — embeds a feedback loop: a model trained on Western-centric data is used to curate the next generation's training data, which amplifies the Western-centrism of the resulting models, which are then used to curate future datasets. The paper's demonstration that this loop persists even when the source data contains 100 billion examples — a scale that should, in principle, include abundant cultural diversity — shows that the filtering stage is the bottleneck, not the crawling stage. This redirects the research agenda: rather than solving cultural bias by crawling ever-larger datasets (which will continue to be filtered by the same biased tools), the priority should be developing diversity-preserving filters or training algorithms that are robust to noisy but culturally significant data.

The practical implication is stark and counterintuitive: for cultural diversity, a random subset of 5 billion English examples outperforms a CLIP-curated 5 billion subset, even though the random subset contains more noisy and misaligned pairs. The model is better off seeing some noisy cultural content than missing that cultural content entirely. This challenges the dominant data-centric AI narrative that better curation always yields better models — it depends entirely on what "better" means and who defines the evaluation.

Innovation 3: Disaggregating Fairness — Representation Bias, Association Bias, and Performance Disparity Respond Differently to Scale

The paper's third conceptual contribution is demonstrating that "fairness" is not a monolith with respect to data scaling, and that different fairness dimensions respond to scale in fundamentally different ways. This is a diagnostic insight that complicates simple narratives — both the optimistic "bigger datasets will solve bias" and the pessimistic "web data is irredeemably biased" positions.

The fairness evaluation framework (Section 4.4) distinguishes three dimensions that prior work had often conflated or studied in isolation:

Representation bias — the model's intrinsic preference for one demographic label over another when shown generic images — remains unchanged by scale. Table 4 shows that models trained on 1B, 10B, and 100B examples consistently associate random ImageNet images with "Male" over "Female" approximately 85% of the time, with no systematic trend. This aligns with Alabdulmohsin et al. (2024)'s finding that explicit data balancing is necessary to mitigate this type of bias — raw data scale alone, without intervention in the training distribution, does not fix representational skews in the model's embedding space.

Association bias — stereotypical mappings between gender and occupation (e.g., "nurse" vs. "doctor," "secretary" vs. "manager") — similarly does not diminish with scale. Figure 2 visualizes this across model sizes and data scales: the heatmaps show that models continue to exhibit strong stereotypical preferences regardless of data volume. A ViT-H trained on 100B examples still assigns high probability to female images being labeled "housekeeper" and male images being labeled "homeowner." The persistence of these associations at 100B scale suggests they are deeply embedded in the statistical structure of web data — not artifacts of insufficient data coverage — and that mitigation requires explicit algorithmic or data-engineering interventions beyond mere scaling.

Performance disparity — the gap in downstream task accuracy between geographic regions or income levels — does improve with data scale. Table 5 shows that the maximum accuracy gap across income groups on Dollar Street zero-shot classification decreases from 32.5% (ViT-B, 1B) to 29.0% (ViT-B, 100B), and the maximum gap across GeoDE regions decreases from 4.7 to 4.4 percentage points for ViT-B, and from 4.0 to 2.8 for ViT-L. ViT-H shows a reduction from 3.6 to 2.7 percentage points. These improvements are modest in absolute terms — a few percentage points — but they are consistent across model sizes and occur without any explicit fairness intervention, suggesting that scaling raw, unfiltered web data genuinely improves the model's ability to serve underrepresented groups on actual tasks.

What makes this disaggregation intellectually significant is that it provides a mechanistic hypothesis for why different fairness dimensions respond differently. Representation and association biases reflect statistical regularities in language — the co-occurrence of gender words with certain concepts in web text — that are pervasive across the entire data distribution. Adding more data doesn't change these regularities; if anything, it reinforces them (the web consistently associates "nurse" with female pronouns, "CEO" with male pronouns, across virtually all subdomains). Performance disparity, in contrast, reflects coverage — the presence or absence of training examples that represent specific visual contexts. When the model has seen more images from low-income countries or non-Western regions (as the 100B dataset provides), it can better recognize objects in those contexts. The improvement in disparity with scale is driven by the same mechanism as the improvement in cultural diversity: long-tail coverage.

This insight has practical consequences for fairness research and practice. It suggests that different fairness metrics require different mitigation strategies: coverage-based disparities (like geographic or income performance gaps) can be addressed through data scale and diversity, while statistical association biases require explicit interventions like data rebalancing or algorithmic debiasing. The paper is careful not to overclaim — it does not present scaling as a universal fairness solution — but by disaggregating the fairness construct, it provides a more actionable picture of which problems scale can address and which it cannot.

Innovation 4: The Insufficiency of Quality Filtering at Any Scale for Cultural Inclusivity

A corollary of the filtering-diversity tradeoff (Innovation 2) that deserves separate recognition is the paper's finding that increasing source data scale does not rescue quality filtering from its diversity cost. This is a non-obvious result with direct implications for dataset construction methodology.

The intuition one might have is: "CLIP filtering on a 10B dataset reduces diversity because the 10B source doesn't have enough cultural content to survive aggressive filtering. If we filter 100B down to a 5B high-quality subset, the source is so much larger that diverse content should still make it through." The paper tests exactly this hypothesis by comparing the CLIP-filtered 5B subset (drawn from the 100B source) against a random English 5B subset (also drawn from the 100B source). If the intuition were correct, the CLIP-filtered subset — now drawing from a vastly larger pool — should capture enough cultural diversity to match or exceed the random subset on cultural benchmarks. It does not (Figure 4). The CLIP-filtered 5B performs worse on cultural diversity than the random English 5B baseline, replicating the same pattern observed when filtering at smaller scales.

This tells us something specific about the filtering mechanism: the CLIP model's alignment scores are systematically biased against non-Western content, not just noisier for it. If the scores were merely noisier but unbiased, then filtering the top 5B from a 100B pool should still capture the most culturally diverse pairs among those 100B — noise would affect which specific examples survive, but the aggregate cultural coverage should improve relative to a random 5B sample. The fact that it degrades implies that CLIP assigns lower scores on average to valid non-Western image-text pairs than to comparable-quality Western pairs. This is a form of annotation bias — the filtering model itself contains the cultural biases it is being used to "clean" — and it means that no dataset size, however large, can compensate for a biased filter. The ranking produced by the filter is fundamentally broken with respect to cultural diversity; taking the top-scoring pairs, whether from a pool of 10B or 100B, will always disproportionately discard non-Western content.

This finding reframes the data quality problem. The field's response to noise in web-scale datasets has been to develop increasingly sophisticated filtering pipelines — CLIP scoring, text quality heuristics, deduplication, resolution thresholds, OCR-based removal. These pipelines are evaluated by their impact on standard benchmark performance, which is dominated by Western-centric tasks. The paper's results imply that standard benchmarks provide exactly the wrong signal for optimizing filters for inclusivity — a filter that improves ImageNet accuracy is likely harming Dollar Street performance, and we would never know because the evaluation protocol doesn't include Dollar Street. The implication is not that filtering should be abandoned, but that filter development must incorporate diversity-aware evaluation, and that current state-of-the-art filters should be understood as optimizing for a specific (Western-centric) definition of quality that may conflict with other valid definitions.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the WebLI-100B dataset — 100 billion image-text pairs crawled from the public web with only essential safety filters applied (harmful content removal, PII removal, near-duplicate removal against more than 90 common vision-language evaluation sets). The 1B and 10B subsets are created by randomly sampling 1% and 10% respectively from the full 100B corpus, ensuring identical distribution across scales. For the data quality filtering analysis (Section 5.1), 5-billion-sample English subsets are constructed via CLIP-L/14 scoring, a custom web-trained classifier, and random sampling. For language rebalancing (Section 5.2), low-resource languages are identified via the content-language meta tag and upsampled.
Base model(s). The paper trains SigLIP (Zhai et al., 2023) contrastive vision-language models across three Vision Transformer architectures: ViT-B/16, ViT-L/16, and ViT-H/14, paired with text Transformers of matching scale. SigLIP uses a sigmoid-based binary classification loss rather than the standard softmax contrastive loss, computing independent match/mismatch predictions for each image-text pair in a batch. Models are trained from scratch at each data scale (1B, 10B, 100B) with fixed hyperparameters: batch size 32K, inverse square root learning rate schedule with 200M warmup and cooldown examples, learning rate 0.001, weight decay 0.0001, image resolution 224×224, and mT5 tokenizer with 64-token max sequence length.
Metrics. All results are reported as error rates (100% − accuracy) rather than accuracy, following standard scaling law practice where error rates are better modeled by power-law functions. For zero-shot classification (ImageNet, CIFAR-100, Pet, Dollar Street, GeoDE, GLDv2), the metric is the fraction of test images assigned to the incorrect class based on maximum cosine similarity between image embedding and class name text embeddings. For retrieval (COCO, Flickr, Crossmodal-3600), the metric is 1 − Recall@1 — the fraction of queries for which the correct match is not ranked first. For 10-shot tasks, the metric is classification error after fine-tuning on 10 labeled examples per class. For fairness, representation bias is the fraction of ImageNet images preferred as "Male" over "Female" (Alabdulmohsin et al., 2024), association bias is the gendered occupation preference probability on FairFace, and performance disparity is the maximum accuracy gap across income groups (Dollar Street) or geographic regions (GeoDE). Scaling law parameters — exponent $c$ and asymptotic limit $\varepsilon$ — are fitted following Alabdulmohsin et al. (2022) using checkpoints at 3, 7, 10, 17, 26, 33, 49, 66, and 100 billion seen examples.
Baselines. The primary comparison is across data scales: 1B, 10B, and 100B, all trained to a fixed budget of 100 billion seen examples (100 epochs for 1B, 10 epochs for 10B, 1 epoch for 100B), making this a compute-matched comparison. For the quality filtering analysis (Section 5.1), the baselines are: (1) a random English-language 5B subset sampled from WebLI-100B without quality scoring, (2) a CLIP-L/14-filtered 5B subset retaining the highest alignment scores, and (3) a classifier-filtered 5B subset using a web-trained alignment model. For language rebalancing (Section 5.2), the baseline is the unmodified data distribution at each scale. For transfer experiments (Section 4.5), PaliGemma models with SigLIP vision encoders pre-trained at different scales are compared in both frozen and unfrozen configurations against the same downstream fine-tuning protocol.
Generation budget / compute accounting. The compute budget is measured in total training examples seen — all models process exactly 100 billion image-text pairs over the course of training, regardless of dataset size. This means models trained on 1B data complete 100 epochs, 10B data complete 10 epochs, and 100B data complete 1 epoch. All models use identical architecture, batch size (32K), image resolution (224×224), and training hyperparameters, so total FLOPs are matched. Models are evaluated at cooldown checkpoints after seeing 3, 7, 10, 17, 26, 33, 49, 66, and 100 billion examples. Unless otherwise specified, the main results report performance at the final 100B checkpoint.
Cross-validation / statistical protocol. The paper uses Wilcoxon's signed-rank test (Wilcoxon, 1992) to assess whether observed performance differences between 10B and 100B models are statistically significant across the suite of evaluation metrics. This non-parametric test ranks the performance differences across all metrics and model sizes, testing whether the central tendency of the distribution differs from zero. Two separate tests are conducted: one aggregating all Western-centric metrics, and one aggregating all cultural diversity metrics. The paper reports p-values of 0.9 for Western-centric tasks and 0.002 for cultural diversity tasks. For scaling law fits, 95% confidence intervals are used to assess whether fitted parameters differ between scales.

Main Quantitative Results

Western-Centric Benchmarks: Saturation at 100B Scale

The paper's headline finding for Western-centric benchmarks is that scaling from 10B to 100B examples produces negligible improvements that fail statistical significance testing.

Aggregate Western-centric performance (Table 2). Across all three model sizes (B, L, H) and four Western-centric task categories (zero-shot classification, 10-shot classification, zero-shot retrieval), the differences between 10B and 100B models are small and inconsistent in direction:

Zero-shot classification, average across ImageNet, CIFAR-100, Pet: ViT-B shows 32.98% → 32.70% error (0.28 point improvement); ViT-L shows 21.97% → 20.44% (1.53 point improvement); ViT-H shows 17.62% → 17.83% (0.21 point degradation). The pattern is not monotonic with model size.
10-shot classification, average across 8 datasets: ViT-B shows 30.77% → 30.43% (0.34 point improvement); ViT-L shows 23.59% → 23.10% (0.49 point improvement); ViT-H shows 21.30% → 21.21% (0.09 point improvement). These differences are well under 1 percentage point.
Zero-shot retrieval, average across COCO and Flickr in both directions: ViT-B shows 45.48% → 46.24% (0.76 point degradation); ViT-L shows 39.83% → 39.23% (0.60 point improvement); ViT-H shows 36.08% → 35.92% (0.16 point improvement).

The Wilcoxon signed-rank test across all Western-centric metrics yields p = 0.9, confirming that the distribution of differences is consistent with random variation. The scaling law analysis reinforces this: exponents and asymptotic limits for 10B and 100B data (Table 2, columns 5–7) are not significantly different at the 95% confidence level (p = 0.09), indicating that the gap between 10B and 100B is not expected to widen even with infinite compute.

Individual benchmark highlights (Table 2). Some Western-centric tasks show slightly larger absolute changes — ViT-L Pet 0-shot improves from 12.46% to 9.46% error (3.0 point improvement), ViT-H COCO I2T@1 degrades from 42.04% to 42.48% error (0.44 point degradation) — but these are isolated cases within a broader pattern of flat or noisy trajectories. No Western-centric task shows a consistent, large-magnitude improvement across all three model sizes. The contrast with cultural diversity benchmarks, where improvements of 5–6 points are observed in the same model and training regime, makes the Western saturation result particularly stark.

Interpretation. The implication is that for concepts well-represented in web data — common objects, animals, everyday scenes, Western landmarks — a dataset of 10 billion unique examples already provides sufficient coverage to approach the irreducible error floor as determined by the SigLIP architecture, training objective, and inherent difficulty of these benchmarks. Adding 90 billion more examples contributes mostly redundant instances of already-saturated concepts, producing negligible marginal benefit. This finding holds under compute-matched comparison: spending the same training FLOPs on 10 epochs over 10B unique examples versus 1 epoch over 100B unique examples yields equivalent Western-centric performance.

Cultural Diversity Benchmarks: Significant Gains from 100B Scale

In contrast to Western-centric metrics, cultural diversity benchmarks show statistically significant and practically meaningful improvements when scaling from 10B to 100B examples (Table 3).

Dollar Street 10-shot geo-localization. This task — predicting the country of origin for household-item images using only 10 labeled examples per country — shows the largest absolute gains:

ViT-B: 75.81% → 72.12% error (3.69 point improvement)
ViT-L: 64.09% → 58.29% error (5.80 point improvement)
ViT-H: 59.10% → 53.69% error (5.41 point improvement)

These improvements of 3.7–5.8 percentage points on Dollar Street 10-shot stand in sharp contrast to the sub-1-point changes observed on Western-centric 10-shot tasks. The gains are consistent across model sizes and represent meaningful progress on a task that directly measures cultural visual understanding — the ability to recognize that a stove in a rural Bolivian kitchen and a stove in a Swedish kitchen are both stoves, but belong to geographically identifiable contexts.

GeoDE 10-shot geo-localization. Similar patterns emerge for geographic region and country prediction on the GeoDE dataset:

Country level: ViT-B improves from 71.47% to 71.36% error (minimal); ViT-L improves from 62.31% to 57.85% (4.46 points); ViT-H improves from 50.22% to 47.55% (2.67 points).
Region level: ViT-B improves from 60.80% to 59.18% (1.62 points); ViT-L improves from 53.59% to 48.29% (5.30 points); ViT-H improves from 47.63% to 44.68% (2.95 points).

The ViT-L model shows particularly strong gains on GeoDE, improving by roughly 4–5 points on both granularities. The ViT-H model, being larger, may already capture more geographic signal from the 10B subset, leaving less room for improvement at 100B — the gains for ViT-H (2.7–3.0 points) are smaller than for ViT-L but still substantially larger than any Western-centric 10-shot task.

Dollar Street zero-shot classification. Gains are more modest but still present:

ViT-B: 51.88% → 51.60% error (0.28 point)
ViT-L: 48.10% → 49.03% error (0.93 point degradation)
ViT-H: 48.58% → 47.35% error (1.23 point improvement)

The zero-shot Dollar Street task is inherently easier than the 10-shot setting because it maps Dollar Street topics to ImageNet classes — the model can leverage its strong ImageNet-aligned representations. The 10-shot geo-localization task is harder (requiring fine-grained geographic discrimination) and benefits more from the diverse cultural signal in the 100B dataset.

GLDv2 zero-shot landmark recognition. Improvements are consistent:

ViT-B: 60.96% → 59.40% error (1.56 points)
ViT-L: 46.37% → 45.72% error (0.65 points)
ViT-H: 40.08% → 38.78% error (1.30 points)

Landmark recognition benefits from 100B scale because many culturally significant landmarks — Angkor Wat, Borobudur, the Great Mosque of Djenné — are long-tail concepts in English-dominated web crawls but appear more frequently in the expanded 100B dataset. However, the gains are smaller than for Dollar Street 10-shot, likely because landmarks are inherently distinctive visual entities that can be learned from relatively few examples.

Statistical significance and scaling law analysis. The Wilcoxon signed-rank test across all cultural diversity metrics yields p = 0.002, confirming that the improvements are systematic rather than attributable to chance. The scaling law exponents (Table 3, columns 5–7) show more negative values for cultural diversity tasks than for Western-centric tasks at the same data scale — for example, ViT-L Dollar Street 10-shot has an exponent of −0.94 at 100B vs. ViT-L ImageNet 10-shot at −0.63 — indicating that cultural diversity metrics are further from saturation and have more room to improve with additional data. The asymptotic limits for cultural diversity tasks are also substantially higher than for Western tasks: ViT-L Dollar Street 10-shot is predicted to plateau around 57.5% error at infinite data, versus ViT-L ImageNet 10-shot at around 32.5% error. This quantifies the intuition that geo-localization is fundamentally harder and requires vastly more diverse data than object recognition.

The compute-matched comparison reveals a specific mechanism. The fact that 1 epoch over 100B unique examples outperforms 10 epochs over 10B unique examples on cultural diversity, while the two regimes match on Western metrics, demonstrates that repetition of limited cultural examples cannot substitute for unique cultural diversity. The 10B subset contains some culturally diverse content — roughly 10% of what the 100B dataset contains — but that 10% sample of long-tail concepts provides insufficient coverage for the model to generalize across the full diversity of cultural visual expression. Seeing the same few Dollar Street-like images 10 times does not teach the model what kitchens look like in countries it hasn't seen examples from, whereas seeing 10× more unique cultural images (from the 100B dataset) does provide that generalization signal.

Multilingual Retrieval: Disproportionate Gains for Low-Resource Languages

The Crossmodal-3600 zero-shot retrieval results (Section 4.3, Figure 3, and Appendix B Table 8) reveal a striking asymmetry: low-resource languages benefit substantially more from the 10B → 100B scaling than high-resource languages, and this gap widens with model size.

Aggregated low-resource vs. high-resource performance (Figure 3, computed from Table 8). The paper defines 7 languages as low-resource: Bengali, Filipino, Hindi, Hebrew, Maori, Swahili, and Telugu, collectively representing 0.814% of the training data. The remaining 28 languages are classified as high-resource. Averaging image-to-text and text-to-image retrieval error rates:

ViT-B low-resource: 84.27% → 83.16% error (1.11 point improvement from 10B to 100B)
ViT-B high-resource: 55.42% → 55.53% error (0.11 point degradation)
ViT-L low-resource: 77.14% → 75.01% error (2.13 point improvement)
ViT-L high-resource: 46.75% → 45.43% error (1.32 point improvement)
ViT-H low-resource: 73.69% → 70.93% error (2.76 point improvement)
ViT-H high-resource: 43.11% → 41.81% error (1.30 point improvement)

The pattern is consistent: low-resource languages improve by 1.1 to 2.8 points when scaling from 10B to 100B, while high-resource languages improve by −0.1 to 1.3 points. The gap in improvement between low- and high-resource languages is 1.2 points for ViT-B, 0.8 points for ViT-L, and 1.5 points for ViT-H — widening with model capacity, suggesting that larger models are better able to absorb and utilize the additional low-resource language signal in the 100B dataset.

Per-language results (Table 8, XM3600 entries). The most dramatic gains occur for the lowest-resource languages:

Telugu image-to-text: ViT-L improves from 76.67% to 69.69% error (7.0 point improvement); ViT-H improves from 73.08% to 65.31% (7.8 points).
Bengali image-to-text: ViT-L improves from 66.36% to 63.75% (2.6 points); ViT-H improves from 61.22% to 56.69% (4.5 points).
Swahili image-to-text: ViT-B improves from 88.17% to 88.72% (slight degradation); ViT-L improves from 81.33% to 79.47% (1.9 points); ViT-H improves from 76.86% to 74.14% (2.7 points).

Languages at the extreme low end of representation, like Maori (0.001% of web pages), show essentially no improvement — ViT-L Maori image-to-text remains at ~99.2% error across all scales — because even the 100B dataset contains too few Maori examples (roughly 1 million) to provide a meaningful learning signal. The languages that benefit are those in the "moderately low-resource" range — Telugu (0.036%, ~36 million examples in 100B), Bengali (0.113%, ~113 million), Hebrew (0.240%, ~240 million) — where the 10B dataset provides some signal but the 100B dataset provides substantially more.

High-resource language performance is largely saturated. English image-to-text retrieval shows ViT-L at 52.42% → 51.67% error (0.75 point improvement from 10B to 100B), while German shows 24.28% → 24.03% (0.25 point). These languages are well-represented in both 10B and 100B datasets, so additional scale provides minimal benefit — the model has already seen enough German text to saturate its retrieval capability.

Interpretation. The mechanism driving these gains parallels the cultural diversity findings: low-resource languages are long-tail concepts in the data distribution. At 10B scale, a language like Telugu appears in only ~3.6 million examples (0.036% of 10B), which is barely sufficient for learning meaningful text representations. At 100B scale, the same language appears in ~36 million examples — a 10× increase in absolute signal — pushing it past a threshold where the model can learn useful Telugu-text-to-image alignments. The widening gap with model size suggests a complementarity between data scale and model capacity: larger models can exploit the additional low-resource language signal more effectively than smaller models, producing a super-linear benefit from scaling both data and model size simultaneously for underrepresented languages.

Fairness: Disaggregated Results by Metric Type

The fairness evaluation (Section 4.4) reveals that scaling from 10B to 100B affects different fairness dimensions in fundamentally different ways, with no single trend that characterizes "fairness" as a whole.

Representation bias (Table 4). The model's preference to associate generic ImageNet images with "Male" over "Female" remains stubbornly high across all scales:

Model	1B	10B	100B
ViT-B	83.2%	84.5%	85.2%
ViT-L	88.2%	86.4%	85.5%
ViT-H	86.8%	85.0%	86.6%

There is no systematic trend toward improvement — ViT-B gets slightly worse at 100B, ViT-L improves slightly, ViT-H fluctuates. The values remain clustered around 85%, indicating a strong and persistent gender skew in the model's embedding space that data scale alone does not mitigate. This aligns with Alabdulmohsin et al. (2024)'s finding that explicit data balancing is required to address representation bias — the web's language statistics (where "male" is the default or more frequent referent) are ubiquitous across all scales.

Association bias (Figure 2). The gendered occupation preference visualizations show that stereotypical associations persist across all data scales and model sizes. The heatmaps display the probability that the model prefers a stereotypical label given the gender of the person in the image. For example, for the librarian-vs.-scientist comparison: a ViT-H model trained on 100B data, when shown female images, assigns roughly 88% probability to "librarian" over "scientist." The patterns are qualitatively similar across 1B, 10B, and 100B — data scale neither consistently amplifies nor attenuates these stereotypical associations. This suggests that gendered occupation stereotypes are deeply embedded in the co-occurrence statistics of web text, present in roughly equal proportion regardless of how much of the web is crawled, and that scaling alone cannot "dilute" them out of the model's representations.

Performance disparity (Table 5). In contrast to representation and association bias, performance disparities across geographic and income groups show consistent improvement with data scale:

Dollar Street by income: The maximum accuracy gap across four income groups (0–200, 200–685, 685–1998, >1998 USD/month):
- ViT-B: 32.5% (1B) → 29.9% (10B) → 29.0% (100B) — a 3.5 point reduction in disparity.
- ViT-H: 32.2% (1B) → 33.0% (10B) → 32.1% (100B) — a smaller 1.1 point reduction from 1B to 100B, with a fluctuation at 10B.
GeoDE by region: The maximum accuracy gap across six geographic regions (Africa, Americas, East Asia, Europe, Southeast Asia, West Asia):
- ViT-B: 4.7 (1B) → 5.5 (10B) → 4.4 (100B) — a slight reduction with fluctuation.
- ViT-L: 3.2 (1B) → 4.0 (10B) → 2.8 (100B) — a clear trend toward lower disparity.
- ViT-H: 3.6 (1B) → 3.0 (10B) → 2.7 (100B) — 0.9 point reduction in maximum gap.

The absolute disparity reductions are modest (2–4 points for Dollar Street, 0.5–1.2 points for GeoDE), but the trend is consistent: in 8 of 9 model-size/metric combinations (3 models × 3 comparisons — Dollar Street income, GeoDE region, GeoDE country), the 100B model shows lower disparity than the 1B model. The one exception (ViT-L Dollar Street) shows a slight increase from 29.7 (1B) to 30.4 (100B). The mechanism is the same as for cultural diversity improvements: the 100B dataset provides better coverage of images from underrepresented regions and income levels, enabling the model to learn visual representations that generalize more equitably.

Interpretation. The disaggregation of fairness metrics into representation bias, association bias, and performance disparity — and the finding that only performance disparity improves with scale — is one of the paper's more nuanced contributions. It demonstrates that "fairness" is not a unitary property that can be addressed by a single intervention. Equity in downstream task performance (the model works equally well for users in Nigeria as in Germany) can be improved through data diversity and scale, because the limiting factor is coverage — the model simply hasn't seen enough examples from certain regions. Stereotypical associations in the model's internal representations, in contrast, reflect language statistics that are pervasive across the web and cannot be overcome by simply adding more web data — they require explicit intervention in the training distribution or objective.

Transfer to Generative Models: No Consistent Benefit

The PaliGemma transfer experiments (Section 4.5, Table 6) test whether pre-training data scale benefits transfer to generative vision-language tasks when the contrastively trained vision encoder is used as a component within an autoregressive VLM.

Aggregated results (Table 6). Averaging across 42 downstream tasks grouped into four categories:

Frozen ViT-L: 1B pre-training achieves 75.1 average; 10B achieves 73.7 (degradation); 100B achieves 75.3 (0.2 point gain over 1B, 1.6 points over 10B). The 10B checkpoint performs anomalously worse than both 1B and 100B, suggesting the differences may be dominated by training noise rather than genuine scaling effects.
Unfrozen ViT-L: 1B achieves 73.6; 10B achieves 72.7; 100B achieves 73.9 (0.3 point gain over 1B).

Category-level patterns (computed from Table 9, reported in Table 6). Semantics tasks (captioning, VQA, referring expressions) show similarly flat trajectories: frozen ViT-L achieves 77.1 (1B), 76.4 (10B), 77.2 (100B); unfrozen achieves 76.0 (1B), 75.4 (10B), 76.4 (100B). OCR tasks show ViT-L frozen at 69.5 (1B), 66.9 (10B), 70.0 (100B) — the 10B degradation is larger than the 100B gain. Multilingual transfer shows frozen ViT-L at 66.9 (1B), 66.0 (10B), 67.0 (100B). Remote sensing is essentially flat: frozen ViT-L at 92.0 (1B), 91.8 (10B), 92.1 (100B).

The paper acknowledges: "when taking the noise level into consideration, we do not observe consistent performance gains across downstream tasks as we scale the pre-training dataset." This null result aligns with the main findings — the PaliGemma downstream suite is dominated by Western-centric and English-language tasks (COCO captions, VQAv2, OKVQA, RefCOCO, TextVQA), exactly the category that shows saturation at 10B scale in the contrastive evaluations. The one category where we might expect gains — multilingual transfer — contains tasks like XM3600 captioning (averaged across 36 languages), where the low-resource language signal might be diluted by the many high-resource languages in the average.

Interpretation. The transfer results serve as a robustness check confirming that the saturation on Western-centric contrastive benchmarks is not an artifact of the contrastive evaluation protocol — it persists when the same vision encoders are deployed in generative settings. They also highlight that current generative VLM evaluation suites are similarly Western-centric, potentially masking the diversity benefits that contrastive evaluations on Dollar Street and Crossmodal-3600 reveal.

Ablation Studies and Robustness Checks

Data quality filtering vs. cultural diversity (Figure 4, Table 10): The filtering experiment compares three 5B-sample English datasets derived from WebLI-100B — a random English subset, a CLIP-L/14-filtered subset, and a classifier-filtered subset — to test whether quality filtering trades off cultural diversity for Western benchmark performance. The CLIP filter produces the best Western-centric results (average error at 30B seen examples: ~23.9% for CLIP vs. ~24.3% for baseline and ~26.5% for classifier), confirming the standard data-centric finding that filtering helps canonical benchmarks. However, on cultural diversity tasks, CLIP filtering produces the worst performance (average error at 30B: ~62.4% for CLIP vs. ~60.1% for baseline and ~61.8% for classifier). The random English baseline — which contains more noise but also more cultural content — matches or exceeds the CLIP-filtered data on cultural diversity throughout training. The classifier filter, which was trained on the same web data distribution rather than a Western-centric pre-training, shows a similar but slightly less severe diversity penalty. The fairness metrics show a similar pattern: random English data achieves better (lower) average fairness error than either filtered variant. This ablation demonstrates that quality filtering is not a scale-independent improvement — it imposes a specific cost on cultural representation that is not recovered by drawing from a larger source pool.

Language rebalancing (Figure 5, Table 11): Upsampling the seven low-resource languages (Bengali, Filipino, Hindi, Hebrew, Maori, Swahili, Telugu) from their natural representation (~0.814% combined) to 1% each (7% combined) produces substantial gains on low-resource language retrieval across all data scales. For ViT-L at 100B, low-resource language retrieval error drops from 75.01% to 70.10% (4.9 point improvement after rebalancing), while high-resource language error increases slightly from 45.43% to 45.75% (0.32 point degradation), and Western-centric tasks show a small decline (26.87% to 27.55% average error). Cultural diversity tasks improve mildly (44.01% to 43.29% at 100B), suggesting some coupling between language representation and visual cultural understanding — seeing more text in low-resource languages helps the model recognize visual concepts associated with those language communities. The key insight is that rebalancing amplifies the gains already observed from scaling: the combination of 100B data and language rebalancing produces the best low-resource language performance in the study. This confirms that language coverage and data scale are complementary — rebalancing helps even at 100B scale, implying that even 100B examples are insufficient to saturate learning for languages at ~0.04% natural representation.

Model size scaling interaction with data scale (Tables 2, 3, 8): The paper's use of three ViT sizes (B/16, L/16, H/14) across all three data scales provides an implicit ablation on model capacity effects. For Western-centric tasks, the improvement from B → L → H is substantial at all data scales (e.g., ImageNet 0-shot at 100B: ViT-B 39.04% error, ViT-L 28.49%, ViT-H 24.90%), but the interaction with data scale is weak — ViT-H gains roughly the same relative improvement over ViT-B at 1B and 100B. For cultural diversity, the interaction is more interesting: ViT-L Dollar Street 10-shot gains 5.80 points from 10B to 100B, while ViT-H gains 5.41 points, and ViT-B gains 3.69 points. The larger models benefit more from data scaling on cultural diversity tasks (in absolute terms), suggesting that model capacity and data diversity are complementary for long-tail concept learning — larger models can absorb more of the diverse signal in the 100B dataset. For multilingual retrieval, the low-resource improvement from 10B to 100B widens with model size (ViT-B: 1.11 points; ViT-L: 2.13 points; ViT-H: 2.76 points), consistent with the interpretation that capacity enables better utilization of the additional language signal.

Epochs vs. unique data tradeoff (implicit in Tables 2–3): The compute-matched comparison (1B/100 epochs vs. 10B/10 epochs vs. 100B/1 epoch) serves as an ablation on whether repetition can substitute for unique data diversity. On Western-centric tasks, 10B/10-epochs roughly matches 100B/1-epoch (Table 2), indicating that for frequent concepts, seeing fewer unique examples multiple times is equivalent to seeing more unique examples once. On cultural diversity tasks (Table 3), 100B/1-epoch consistently outperforms 10B/10-epochs — repetition of a limited set of culturally diverse examples does not achieve the coverage of fresh unique examples. This isolates the mechanism: the benefit of 100B scale is specifically about unique example diversity, not about seeing more total training examples, since total examples are held constant.

Transfer setting (frozen vs. unfrozen): Table 6 (and detailed Table 9) compares two PaliGemma configurations — frozen and unfrozen vision encoder — both initialized from the same SigLIP pre-training. The unfrozen setup generally achieves slightly higher performance (averaging ~0.5–1.5 points better within each data scale), but the pattern across data scales is similar in both configurations: no consistent trend favoring larger pre-training data. This serves as a robustness check that the null transfer result is not an artifact of freezing the vision encoder — even when the encoder is allowed to adapt during PaliGemma training, the 100B pre-training provides no transfer advantage over 1B or 10B on this suite of predominantly Western-centric generative tasks.

Critical Assessment

Claim: "Scaling from 10B to 100B examples produces limited benefits on Western-centric benchmarks."

Assessment: This claim is well-supported by the reported data. The aggregate Western-centric results show sub-1-percentage-point changes between 10B and 100B across most metrics (Table 2), and the Wilcoxon test confirms no statistically significant pattern (p = 0.9). The scaling law analysis further supports this by finding no significant difference in asymptotic limits or exponents between the 10B and 100B conditions (p = 0.09, above the 0.05 threshold). Caveats: The test set is a single aggregation of benchmarks — the claim might not generalize to every individual Western-centric task, some of which (ViT-L Pet 0-shot, 12.46% → 9.46%) show non-trivial improvements. The 0.09 p-value for scaling law differences, while above 0.05, is suggestive of a very weak effect that a larger model or more checkpoints might eventually detect. The claim is also conditional on the SigLIP training recipe — other contrastive objectives (CLIP softmax, CoCa) might show different saturation points, but this specific recipe saturates by ~10B.

Claim: "Cultural diversity metrics show statistically significant gains from 100B scale."

Assessment: Supported with qualifications. The gains on Dollar Street 10-shot (3.7–5.8 points across model sizes) and GeoDE 10-shot (2.7–5.3 points) are genuine and statistically significant (p = 0.002). However, the absolute level of performance remains poor. Even at 100B scale, ViT-H Dollar Street 10-shot error is 53.69% — the model fails to correctly geo-localize nearly half the images. ViT-L GeoDE country classification error is 57.85% — worse than random for a task with many country classes. The paper frames these as improvements, which they are, but a reader interested in whether 100B scale makes cultural diversity "good enough" for deployment would find the answer is no. The gains are meaningful in a relative sense (5.8 points is large compared to typical benchmark improvements) but leave the model far from human-level or practically reliable cultural understanding. The Dollar Street zero-shot results are especially weak — ViT-L actually degrades from 10B to 100B (48.10% → 49.03%), suggesting that the zero-shot protocol (mapping Dollar Street topics to ImageNet classes) may be a poor measure of cultural competence, or that the additional data introduces noise that hurts on this specific task while helping on 10-shot geo-localization.

Claim: "Low-resource languages benefit more from 100B scale than high-resource languages."

Assessment: Well-supported by Figure 3 and Table 8. The 2.1–2.8 point average improvement for low-resource languages versus 0.1–1.3 points for high-resource languages is a clear and consistent finding. Caveats: The absolute performance on the lowest-resource languages remains abysmal — Maori retrieval error exceeds 99% at all scales, Swahili remains above 74% image-to-text error even at 100B, and Telugu at 65–70% error. The paper's claim that 100B scale "is vital for building truly inclusive multimodal systems" needs to be understood as "vital but not sufficient" — 100 billion examples is still not enough data for languages below ~0.04% representation to achieve usable retrieval performance. The Crossmodal-3600 benchmark has only 3,600 images — the per-language test set is roughly 100 images × 36 languages, meaning the reported error rates have wide confidence intervals. A single misranked image in a low-resource language changes the error rate by ~1 percentage point.

Claim: "Data quality filtering reduces cultural diversity even at 100B scale."

Assessment: The evidence in Figure 4 is clear but the experimental design has limitations. The strong result — CLIP-filtered 5B underperforms random English 5B on cultural diversity — directly supports the claim. Weakness: The paper uses only two specific filter configurations (CLIP-L/14 and a custom web-trained classifier), both retaining 5 billion examples. It does not explore whether a different filtering threshold (e.g., retaining 20B or 50B instead of 5B from the 100B source) could preserve cultural diversity while still providing quality benefits. The claim that "filtering reduces diversity" is supported, but the stronger implication — "filtering cannot be made diversity-preserving" — is not tested with alternative filter designs, ensemble filters, or diversity-aware filtering objectives. Additionally, the random English baseline controls for language restriction but not for quality — it contains more noisy pairs than the filtered datasets, and the paper does not quantify how much of the cultural diversity advantage is attributable to genuine diverse signal versus noise that happens to correlate with cultural categories.

Claim: "Performance disparity across subgroups improves with data scale."

Assessment: Supported but the effect sizes are modest. The maximum improvement in disparity is about 3.5 points for ViT-B Dollar Street (32.5% → 29.0% gap) and about 1.2 points for ViT-L GeoDE region (4.0 → 2.8). Weakness: The paper does not report confidence intervals or statistical significance for the disparity metrics specifically. The disparity metric is a max-gap — the difference between the best-performing and worst-performing subgroup — which is sensitive to outliers and may not capture the full distribution of performance across groups. A subgroup with very few test examples (e.g., the lowest-income bin in Dollar Street) could drive the disparity metric even though its accuracy estimate is noisy. The paper also reports only 100B checkpoint results for disparity; it does not show whether disparity improves monotonically with data scale or fluctuates, as would be visible from the intermediate checkpoint evaluations.

Claim: "Representation bias and association bias are not reduced by data scale."

Assessment: Well-supported by Table 4 and Figure 2. The ~85% male preference on ImageNet is remarkably stable across all scales, and the gendered occupation association heatmaps show qualitatively identical patterns. Weakness: The fairness evaluation is limited to binary gender and a handful of occupations on the FairFace dataset. The paper does not test whether scaling affects biases along other demographic dimensions (race, age, perceived socioeconomic status) or other types of harmful associations (violence, criminality, dehumanization). The claim that scaling "does not reduce bias" is limited to the specific biases measured.

Overall Design Weaknesses

Single training recipe. All experiments use the same SigLIP loss, the same hyperparameters, the same image resolution (224×224), and the same training schedule. This is appropriate for a controlled scaling study but means we cannot know whether the saturation point on Western benchmarks or the diversity gains at 100B are specific to this training configuration. A different contrastive loss (e.g., standard CLIP softmax), a higher resolution, or a longer training schedule might shift where saturation occurs or change the relative benefit of unique data vs. repetition.

Difficulty estimation for benchmark tasks is absent. Unlike the reference example paper which carefully estimates prompt difficulty to enable adaptive allocation, this paper treats all evaluation examples within a benchmark as equally informative. It does not analyze whether the cultural diversity gains are concentrated in "hard" examples (very rare concepts, extremely low-resource languages) versus distributed across the benchmark, which would provide additional mechanistic insight into where scale matters.

The 100B dataset is not publicly released. The paper describes WebLI-100B and its construction philosophy but does not make the dataset available, citing only that prior WebLI datasets have been used internally. This fundamentally limits reproducibility — no external researcher can verify the scaling curves, test alternative training recipes, or evaluate on additional cultural diversity benchmarks. The 1B and 10B subsets are also not released, meaning even relative comparisons across scales cannot be replicated.

No analysis of compute cost for data collection. The paper frames its central question as "should one invest in scaling up the size of the pretraining dataset to 100 billion examples?" but provides no information about the cost (in dollars, FLOPs, or engineering effort) of crawling, storing, and preprocessing 100 billion image-text pairs versus 10 billion. The training compute cost is held constant (all models see 100B examples), but the data acquisition cost — which is the primary practical barrier — is unquantified. A practitioner reading the paper would not know whether the 100B dataset costs 2×, 10×, or 100× more to assemble than the 10B dataset.

Crossmodal-3600 has limited language coverage. The multilingual evaluation covers 36 languages, which the paper itself acknowledges as a limitation. The 7 "low-resource" languages are those with the lowest representation among these 36, but this set excludes many of the world's most underserved languages (e.g., languages of sub-Saharan Africa beyond Swahili, indigenous languages of the Americas, Austronesian languages beyond Filipino and Maori). The paper's claim about inclusivity is bounded by the languages it evaluates on — we cannot extrapolate from Maori (0.001% representation, no improvement) and Telugu (0.036%, large improvement) to predict where the threshold lies for languages at 0.005% or 0.0001% representation.

Missing Experiments

Ablation on the number of epochs at fixed data scale. The compute-matched comparison holds total seen examples constant at 100B, but a natural ablation would be: for the 10B dataset, train for 1 epoch (10B seen examples), 10 epochs (100B), and 100 epochs (1 trillion seen examples). This would reveal whether the Western-centric saturation at 10B is due to data diversity limits or training budget limits — if 100 epochs on 10B matches 1 epoch on 100B for Western tasks, diversity is the bottleneck; if 100 epochs outperforms, training budget (not unique data) is the bottleneck. The paper's design cannot distinguish these.

Ablation on language-specific data volume. The paper attributes low-resource language gains to increased absolute data volume (e.g., Telugu images increase from ~3.6M to ~36M), but does not directly test this hypothesis by creating a 10B dataset with Telugu artificially upsampled to match its 100B absolute count (~36M) and comparing against the true 100B dataset. If the upsampled 10B dataset matches the 100B dataset on Telugu retrieval, the mechanism is purely about absolute examples; if not, there is an additional diversity benefit from the 100B crawl (e.g., more varied Telugu captions, more diverse visual contexts) that upsampling within a smaller crawl cannot replicate.

Ablation on text source (alt-text vs. page title). The paper uses both alt-text and page title as text pairs but does not report whether one source is more important for cultural diversity or multilingual gains. Page titles are more likely to be in the page's language but less directly descriptive of the image; alt-text is more descriptive but often missing or in English regardless of page language. An ablation training with only alt-text, only page titles, or both would reveal which text source drives the multilingual and cultural diversity improvements.

Continuous scaling curve. The paper evaluates at three data scales (1B, 10B, 100B) with logarithmic spacing, which provides three points for fitting scaling laws. Additional intermediate scales (e.g., 3B, 30B) would allow more precise estimation of where saturation begins for different task categories and whether the cultural diversity benefits show a smooth power-law trend or a threshold effect (e.g., the Telugu retrieval improvement might jump between 10B and 100B rather than improving smoothly, if there is a critical mass of examples needed for the language to "emerge" in the model's representations).

Broader cultural diversity benchmarks. The paper uses Dollar Street, GeoDE, and GLDv2, which are standard but limited. Dollar Street focuses on household items — it does not measure recognition of culturally specific activities (festivals, rituals, occupations), social contexts (markets, places of worship), or abstract cultural concepts. Additional benchmarks like the Cultural Diversity Dataset (CDD), the GeoChat benchmark for cultural VQA, or custom retrieval tasks with culturally specific queries would strengthen the claim that 100B scale broadly improves cultural competence rather than specifically improving geo-localization and landmark recognition.

6. Limitations and Trade-offs

6.1 The 100-Billion-Scale Dataset Is Not Publicly Released, Making All Results Non-Reproducible

The assumption or constraint. The paper's central empirical contribution — that scaling vision-language pre-training to 100 billion image-text pairs benefits cultural diversity, multilinguality, and fairness while saturating on Western benchmarks — depends entirely on WebLI-100B, a proprietary dataset that is not released. The paper describes the dataset's construction philosophy ("only essential data filters, such as removing harmful images and personally identifiable information") and reports its language distribution (Appendix F, Table 12), but provides no mechanism for external verification, no subset release, and no access terms. The 1B and 10B subsets — created by random sampling 1% and 10% from the full 100B corpus — are similarly unavailable.

The consequence. No researcher outside Google DeepMind can verify any of the paper's empirical claims. The scaling curves in Tables 2–3, the filtering analysis in Figure 4, the language rebalancing results in Figure 5 — all are assertions about a dataset whose composition, noise characteristics, cultural diversity, and evaluation-set contamination status cannot be independently assessed. This matters concretely because many of the paper's findings are benchmark-specific magnitudes: the claim that ViT-L Dollar Street 10-shot improves by 5.80 points from 10B to 100B (Table 3) could reflect either a genuine diversity benefit or an idiosyncrasy of WebLI-100B's crawling methodology (e.g., overrepresentation of certain countries, specific alt-text conventions, temporal shifts in web content). Without access to the dataset, it is impossible to distinguish these explanations or to determine whether the findings generalize to other web-scale datasets crawled with different methodologies (e.g., LAION-5B's Common Crawl pipeline, DataComp's filtering approach).

The non-reproducibility also limits the paper's practical influence on the broader community. The paper frames itself as guidance for whether "one should invest in scaling up the size of the pretraining dataset to 100 billion examples," but this guidance is actionable only for organizations capable of independently crawling 100 billion image-text pairs — a capability restricted to a handful of large industrial labs. The research community cannot build on this work by testing alternative training recipes, different contrastive objectives, or additional cultural diversity benchmarks at the 100B scale, because the essential resource does not exist outside Google.

What evidence exists in the paper. The paper does not address data release or reproducibility as a limitation. The statement of contribution frames WebLI-100B as a novel dataset introduction ("we introduce WebLI-100B, a novel dataset containing 100 billion image-text pairs"), but the dataset's public status is never clarified. The related work section notes that LAION-5B is "the largest publicly . . . available image-text dataset" and WebLI-10B is "the largest . . . privately available," implying — but not stating — that WebLI-100B continues this private status. The acknowledgments thank "Google DeepMind at large for providing a supportive research environment," consistent with internal-only research.

Mitigation status. The paper offers no mitigation. It does not propose a public subset, a standardized crawling recipe that others could follow, or a mechanism for independent replication. The limitation is structural — the paper's findings are anchored to a proprietary artifact — and no amount of methodological detail in the paper itself can substitute for access to the underlying data.

6.2 The Difficulty Estimation Cost Is Unaccounted For in the Scaling Analysis

The assumption or constraint. The paper compares models trained on 1B, 10B, and 100B unique examples under a compute-matched regime where all models process 100 billion total training examples (100 epochs for 1B, 10 epochs for 10B, 1 epoch for 100B). This comparison assumes that the only cost metric that matters is training FLOPs — the computation consumed by the SigLIP training loop. But the decision to scale from 10B to 100B unique examples carries a massive data acquisition cost that the paper does not quantify: crawling, storing, preprocessing, deduplicating, and safety-filtering 100 billion image-text pairs from the public web. This cost is not amortized or analyzed anywhere in the paper. The paper's central framing — "should one invest in scaling up the size of the pretraining dataset to 100 billion examples?" (Section 1) — is explicitly an investment question, yet the paper provides none of the economic data (dollar cost, engineering effort, storage requirements, crawling duration) needed to answer it.

The consequence. The headline finding that 100B scale "is vital for building truly inclusive multimodal systems" conflates two separate questions: (1) Does 100B diverse data improve inclusivity metrics? (2) Is the improvement worth the cost of acquiring 100B diverse data? The paper answers (1) affirmatively but provides zero information about (2). A practitioner evaluating the paper's recommendation would need to know: What is the marginal cost of crawling 90 billion additional examples beyond the 10 billion already crawled? Is this cost dominated by storage (100B image-text pairs at typical resolutions require petabytes of storage), network bandwidth, compute for safety filtering and deduplication, or engineering time? Does the cost scale linearly with dataset size, or are there economies of scale in crawling infrastructure? Without these estimates, the paper's policy recommendation — invest in 100B scale — is qualitatively correct but quantitatively ungrounded, and could lead practitioners to underestimate the resource commitment required.

The omission of data acquisition cost also creates an asymmetry in the paper's comparison between scaling data and other approaches. The paper critiques quality filtering (Section 5.1) for reducing cultural diversity while improving Western benchmarks — but filtering a 100B dataset down to a 5B high-quality subset also reduces storage and training costs (the 5B subset can be stored more cheaply and trained on more epochs within the same budget). If the data acquisition cost of 100B is 20× higher than that of 10B, a practitioner might reasonably choose to invest that budget in better filtering techniques, language rebalancing, or targeted data collection for underrepresented cultures rather than in raw scale. The paper's compute-matched framework systematically ignores this tradeoff by holding training compute constant while varying data diversity, without acknowledging that diversity itself has an unmeasured cost.

What evidence exists in the paper. The paper does not mention data acquisition cost anywhere. Section 3.1 describes the dataset construction process qualitatively ("100 billion image-text pairs from the web") but provides no metrics on crawling cost, storage size, preprocessing FLOPs, or engineering effort. The compute-matched comparison framework (Section 3.2) is explicit about holding training compute constant but silent on the data acquisition budget. The discussion (Section 6) identifies data filtering and benchmark limitations as open issues but does not flag cost analysis as a missing component.

Mitigation status. The paper does not address this limitation. It neither provides cost estimates nor acknowledges their absence as a gap. A reasonable mitigation would be to report the total storage footprint of WebLI-100B (in petabytes), the number of compute hours required for crawling and preprocessing, or a rough dollar cost per billion examples — any of which would allow practitioners to extrapolate the investment required. The paper's framing as an investment guide is incomplete without this information.

6.3 Hard Problems in Cultural Diversity and Multilinguality Remain Fundamentally Unsolved Despite Scale

The assumption or constraint. The paper's central claim is that scaling to 100B examples produces statistically significant gains on cultural diversity and multilinguality benchmarks. However, the absolute performance levels achieved at 100B scale remain poor on the hardest tasks, particularly for the lowest-resource languages and the most challenging cultural discrimination tasks. The paper does not claim that 100B scale solves these problems, but its framing — "this data scale is vital for building truly inclusive multimodal systems" — could be misinterpreted as suggesting that 100B scale is sufficient for inclusivity, when the results show it is necessary but far from sufficient.

The consequence. Several specific capability boundaries are clearly visible in the reported results:

Crossmodal-3600 retrieval for the lowest-resource languages is near chance. Maori, representing 0.001% of web pages, shows image-to-text retrieval error rates of 99.25% (ViT-L, 100B) — essentially no better than random guessing among 100 captions per image. Swahili (0.046%) shows ViT-L image-to-text error of 79.47% at 100B, and text-to-image error of 89.57%. Telugu (0.036%) — the language the paper highlights in Figure 1 as benefiting from scale — shows image-to-text error of 69.69% (ViT-L, 100B), meaning the model fails to retrieve the correct caption for nearly 70% of Telugu queries. These are languages with tens of millions of speakers. For the vast majority of the world's ~7,000 languages, which have web representation below 0.001%, the paper's results imply that even 100B scale provides essentially no usable signal — the model has not learned these languages in any functional sense.

Dollar Street 10-shot geo-localization remains poor. The best result in the paper — ViT-H at 100B — achieves 53.69% error on Dollar Street 10-shot. This means that given 10 labeled examples per country, the model geo-localizes images incorrectly more than half the time. A practitioner deploying a culturally inclusive VLM would find this error rate unacceptable for applications requiring reliable geographic reasoning (e.g., disaster response, agricultural monitoring, cultural heritage documentation). The paper's 5.8-point improvement for ViT-L (64.09% → 58.29% error) represents genuine progress but leaves the model far from practical utility.

GeoDE country-level geo-localization shows similarly poor absolute performance. ViT-L achieves 57.85% error at 100B (Table 3), meaning fewer than half of GeoDE images are correctly assigned to their country of origin. The region-level task (6 classes) fares better — ViT-H achieves 44.68% error at 100B — but this coarser granularity (continent-scale) is less useful for applications requiring fine-grained cultural understanding.

The implication is that 100B scale is an intermediate milestone, not a destination. The scaling law exponents in Table 3 — e.g., ViT-L Dollar Street 10-shot at 100B has an exponent of −0.94 — indicate that performance is still improving with data, meaning the model is far from saturation on these tasks. However, the asymptotic limits — 57.5% error for ViT-L Dollar Street 10-shot — suggest that even infinite data may not bring the error rate below ~50% with the current SigLIP architecture and training recipe. The paper does not discuss whether alternative approaches (higher resolution, different contrastive objectives, explicit geographic supervision) might achieve lower asymptotic limits, or whether the high error floor is inherent to the task difficulty or the model class.

What evidence exists in the paper. The evidence is in the reported numbers themselves. Table 3 shows Dollar Street 10-shot error at 100B ranging from 53.69% (ViT-H, best) to 72.12% (ViT-B, worst). Table 8 shows XM3600 per-language error rates where Maori remains above 99% image-to-text and text-to-image retrieval error across all model sizes and data scales. The paper's Figure 3 highlights improvements in low-resource languages but does not comment on the fact that the absolute error rates for these languages remain 70–80% despite scaling.

Mitigation status. The paper does not explicitly frame the poor absolute performance as a limitation, but it does acknowledge in Section 6 (Limitations) that "the benchmarks used in this paper to evaluate VLM inclusivity are necessarily limited, since inclusivity is a broad societal concept that should be reduced to a handful of metrics." This acknowledges scope limitation but not the capability limitation — the problem is not just that the benchmarks are narrow, but that on the benchmarks they do use, performance is unacceptably low for deployment. The paper does not discuss whether 1 trillion examples (the next order of magnitude) might close this gap, or whether fundamental architectural innovations are needed.

6.4 Representation and Association Biases Are Not Reduced by Scaling, Requiring Separate Mitigation

The assumption or constraint. The paper evaluates three fairness dimensions — representation bias, association bias, and performance disparity — and finds that only performance disparity improves with data scale (Section 4.4). Representation bias (the model's preference to associate generic images with "Male" over "Female") remains at ~85% across all data scales (Table 4), and association bias (gendered occupation stereotypes) shows qualitatively identical patterns at 1B, 10B, and 100B (Figure 2). The paper does not claim that scaling fixes these biases; it explicitly notes that "this finding aligns with previous research highlighting the necessity of bias mitigation strategies, such as data balancing" (Section 4.4). However, the paper's title and abstract frame 100B scale as essential for "building truly inclusive multimodal systems" — a framing that must be qualified by the fact that some of the most harmful forms of bias (stereotypical gender associations, representational skews) are entirely unaffected by the proposed intervention.

The consequence. The practical implication is that scaling raw data is complementary to, not a substitute for, bias mitigation. An organization that scales its dataset to 100B examples without also implementing explicit bias mitigation (data balancing, algorithmic interventions, post-hoc calibration) will produce a model that serves diverse geographic and linguistic groups more equitably (better Dollar Street and Crossmodal-3600 performance) but that continues to exhibit strong gender stereotypes in its representations and predictions. For applications where these stereotypes could cause harm — hiring tools, content moderation, medical diagnosis — scaling alone provides no protection.

The paper's findings also highlight a fundamental limitation of the "scale cures all" narrative: some statistical regularities in web data (e.g., the association of female-gendered images with domestic occupations) are not diluted by additional data because they are pervasive across essentially all subdomains of the web. Adding 90 billion more image-text pairs does not change the fact that on the web, nurses are disproportionately described as female and doctors as male. The distribution itself is biased, and scaling the distribution amplifies the bias signal, not the diversity signal, for these associations. This distinguishes representation/association bias from performance disparity, where the limiting factor is coverage (absence of examples from certain groups) rather than skew (asymmetric statistical associations) — coverage improves with more data, skew does not.

What evidence exists in the paper. Table 4 shows representation bias values of 83.2–88.2% across all data scales for ViT-B/L/H, with no systematic trend. Figure 2 visualizes association bias as heatmaps across 3 model sizes × 3 data scales × 5 occupation pairs × 2 genders — a 90-cell grid where the patterns are qualitatively identical regardless of scale. The paper explicitly states that "we do not see a reduction in association bias by simply increasing the size of the training data" (Section 4.4). The performance disparity results in Table 5, in contrast, show modest but consistent improvement — confirming that the two fairness dimensions respond to scale differently.

Mitigation status. The paper acknowledges this limitation implicitly by citing Alabdulmohsin et al. (2024) on the necessity of data balancing for bias mitigation, but it does not test whether combining 100B scale with explicit bias interventions would outperform either approach alone. This is a notable gap, because the natural follow-up question is: does scaling raw data help bias mitigation strategies work better (by providing more diverse positive examples of counter-stereotypical associations), or does it simply provide more data for biased regularities to dominate? The paper's framing as an "inclusivity" investigation could be strengthened by exploring this interaction, but it does not.

6.5 Single Training Recipe, Single Model Family, Single Image Resolution — Generalization to Other Configurations Is Unknown

The assumption or constraint. All experiments use a single contrastive training recipe: SigLIP loss with ViT-B/16, ViT-L/16, and ViT-H/14 architectures at 224×224 resolution, batch size 32K, inverse square root learning rate schedule, and the mT5 tokenizer with 64-token max sequence length (Section 3.2). The paper explicitly states these choices are fixed across all data scales to isolate the effect of data size, which is methodologically correct for the intended comparison. However, the paper implicitly generalizes its findings — saturation on Western benchmarks, gains on cultural diversity — to vision-language pre-training broadly, framing the conclusions as guidance for "the development of truly inclusive multimodal systems" generally, not for SigLIP-trained ViT encoders specifically.

The consequence. Several aspects of the paper's findings could be specific to the SigLIP recipe:

The sigmoid loss may interact with noisy data differently than softmax contrastive loss. The paper argues that SigLIP's batch-independent per-pair computation provides robustness to noisy negative pairs (Section 3.4 of the prior sections). This means SigLIP may be more tolerant of the noisy, minimally filtered raw web data than a standard CLIP softmax loss, where each misaligned pair distorts the entire row's normalization. If so, the finding that raw 100B data matches or slightly exceeds filtered data on Western benchmarks (and substantially exceeds it on cultural diversity) might not hold for softmax-contrastive training, where noise tolerance is lower. Softmax-trained models might require more aggressive filtering, which would reintroduce the diversity penalty documented in Section 5.1.

The 224×224 resolution may bottleneck fine-grained cultural visual recognition. Many culturally distinctive features — textile patterns, architectural ornamentation, facial features, writing systems — require higher resolution to distinguish. The paper's SigLIP training at 224×224 may place a ceiling on cultural diversity performance that is independent of data scale: even with 100B examples, the model may lack the spatial resolution to discriminate between visually similar but culturally distinct patterns. The paper does not test whether higher resolution (e.g., 384×384 or 512×512) would change the scaling behavior, either by accelerating saturation on Western benchmarks or by amplifying cultural diversity gains.

The ViT architecture and model size range (B to H) may not represent the scaling behavior of other architectures. The paper observes that larger models (ViT-H) benefit more from data scaling on cultural diversity tasks in absolute terms — ViT-H Dollar Street 10-shot improves by 5.41 points versus ViT-B's 3.69 points (Table 3) — and that the multilingual low-resource improvement widens with model size (Figure 3). This suggests a complementarity between model capacity and data diversity that the paper does not systematically explore. If this trend continues, an even larger model (e.g., ViT-G/14 or ViT-e) might extract substantially more benefit from 100B scale than ViT-H does, potentially changing the cost-benefit calculus. Conversely, a smaller model (ViT-S or ViT-Ti) might saturate on cultural diversity earlier, making 100B scale unnecessary for small-model deployments. The paper's three-point model size sweep (B, L, H) is suggestive but insufficient to establish the full model-size × data-size interaction.

The mT5 tokenizer's multilingual capabilities may not generalize to other tokenizers. The paper's multilingual gains depend partly on the tokenizer's ability to represent low-resource language text as meaningful subword units. The mT5 tokenizer was trained on 101 languages with balanced sampling (Xue, 2020), which may overrepresent low-resource languages relative to more commonly used tokenizers (e.g., GPT-2 BPE, CLIP's byte-pair encoding) that are English-dominated. A model using an English-centric tokenizer might fragment Telugu or Swahili text into byte-level tokens, preventing the model from learning meaningful text representations for these languages regardless of data scale. The paper's multilingual results may therefore be specific to the mT5 tokenizer choice, and practitioners using other tokenizers would not necessarily replicate the low-resource language gains.

What evidence exists in the paper. The paper does not perform any ablation on training recipe components (loss function, resolution, tokenizer). It does not train a CLIP-softmax baseline or a higher-resolution variant to test whether the saturation point shifts. The transfer experiments with PaliGemma (Section 4.5) use the same 224×224 SigLIP-pretrained encoders, so they do not test alternative pre-training configurations. The paper's related work discussion (Section 2) notes that CLIP-style softmax contrastive learning is "the dominant approach" but does not justify why SigLIP is used or discuss whether findings might transfer.

Mitigation status. The paper does not address this generalization limitation. It does not claim that findings are recipe-agnostic, but it also does not acknowledge that they might be recipe-specific. In the limitations section (Section 6), the paper notes only that "the benchmarks used in this paper to evaluate VLM inclusivity are necessarily limited" — a different limitation. A reasonable mitigation would be a brief experiment with an alternative loss (e.g., softmax contrastive at a smaller scale) showing that the qualitative pattern (saturation on Western tasks, gains on cultural diversity) is not SigLIP-specific, or an explicit caveat that findings are conditional on the SigLIP recipe.

6.6 Sequential Dependency and Latency in Training Are Not Analyzed — Only Throughput Matters in the Compute-Matched Framework

The assumption or constraint. The paper compares 1B/100-epoch, 10B/10-epoch, and 100B/1-epoch training regimes under a compute-matched framework where all models process 100 billion total training examples. This framework treats all FLOPs as equivalent regardless of when they occur, implicitly assuming that training latency (wall-clock time to completion) is not a constraint and that the data loading pipeline can feed 32K-example batches at the same throughput for all dataset sizes. For the 100B/1-epoch case, this means loading, preprocessing, and feeding 100 billion unique image-text pairs — a data pipeline challenge fundamentally different from loading 1 billion unique pairs 100 times, where caching and prefetching can dramatically reduce I/O overhead.

The consequence. The compute-matched comparison ignores two practical constraints that could make the 100B/1-epoch regime more expensive or slower than the FLOPs equivalence suggests:

Data loading I/O overhead. Training for 1 epoch over 100 billion unique examples requires reading and preprocessing every image-text pair exactly once, without the benefit of caching or data repetition. For 10B/10-epoch training, the dataset is 10× smaller — it may fit entirely in distributed storage caches after the first epoch, making subsequent epochs I/O-free. The 100B dataset, at roughly 10× the storage footprint, may exceed cache capacity, requiring continuous network reads from slower storage. The paper does not report whether WebLI-100B training required different data pipeline infrastructure (e.g., more storage nodes, higher network bandwidth) than WebLI-10B training, or whether the per-example I/O cost differed between scales.

Shuffling overhead at scale. Training for a single epoch requires the model to see a uniformly random sample of the 100 billion examples — a global shuffle across a dataset far larger than any single machine's memory. Implementing this requires distributed shuffle algorithms (e.g., shuffle buffers spanning multiple workers) whose overhead scales with dataset size. The paper does not discuss whether the 100B training required larger shuffle buffers, more worker coordination, or longer preprocessing time relative to the 10B training, all of which add to the wall-clock time but not the training FLOPs.

The practical impact of these overheads is that "compute-matched" does not mean "latency-matched" or "cost-matched." A team with a fixed training time budget (e.g., "we can train for 3 weeks on our cluster") might find that the 100B/1-epoch regime is infeasible due to I/O bottlenecks, while the 10B/10-epoch regime completes within the time budget. The paper's headline comparison — equal FLOPs, equal cost — could mislead practitioners into thinking the resource requirements for 100B scale are comparable to those for 10B scale, when the infrastructure demands may be substantially different.

What evidence exists in the paper. The paper provides no information about training infrastructure, data pipeline design, I/O characteristics, or wall-clock training time for any of the three regimes. The training section (Section 3.2) specifies the hyperparameters (batch size 32K, learning rate schedule, cooldown checkpoints) but not the hardware (TPU/GPU count, interconnect, storage system) or the per-training-run time. This is standard for scaling law papers — Kaplan et al. (2020) and Hoffmann et al. (2022) similarly report FLOPs without infrastructure detail — but it leaves a practitioner unable to estimate the total resource commitment for replicating the 100B training run.

Mitigation status. The paper does not address this limitation. In the discussion (Section 6), the authors state "we focus on the impact of scaling raw, unfiltered data, and leave the improvement of data quality at the 100 billion scale for future work" — focusing on data scope rather than infrastructure scope. A reasonable mitigation would be to report the hardware configuration (e.g., TPU v4 pod slice size, total TPU-hours for each training run) or to note that the 100B training required specific data pipeline optimizations relative to the 10B training, giving a sense of the practical engineering barrier.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture, a new loss function, or a new training algorithm. Its contribution is diagnostic, not methodological — and precisely because of that, it changes the landscape by revealing a blind spot in how the field evaluates progress in vision-language pre-training. The shift it triggers is a reframing of what scaling curves measure: not universal model capability, but capability on the specific benchmarks we choose to evaluate on.

The dominant narrative in the scaling laws literature — from Kaplan et al. (2020) through Hoffmann et al. (2022) to Cherti et al. (2023) — has been that data scaling produces predictable, smoothly diminishing returns on aggregate benchmark performance. Under this narrative, the key question is "What is the scaling exponent?" — how quickly does error decrease with dataset size? The paper demonstrates that this question, while mathematically precise, is incomplete in a way that matters for real-world deployment. The scaling curve for "average benchmark performance" is a weighted mixture of many underlying curves: one for frequent Western concepts (cat, car, Eiffel Tower), which saturate early because 10 billion examples already contains millions of instances of these concepts; and one for long-tail cultural concepts (Igorot dance, Telugu household items, Bengali New Year celebrations), which continue to improve well past 10 billion examples because their representation in the data is sparse and each additional order of magnitude provides a meaningful increase in unique diversity.

The paper's evidence for this reframing is the stark asymmetry between Tables 2 and 3, summarized by the Wilcoxon test: p = 0.9 for Western-centric metrics (no significant difference between 10B and 100B), p = 0.002 for cultural diversity metrics (highly significant). This is not a small effect hidden in noise — it is a qualitative difference in scaling behavior that was invisible to the field because the evaluation suites used to track progress systematically excluded cultural diversity benchmarks. The paper's reframing is: scaling data does not help "VLMs" — it helps VLMs on the specific concepts represented in the marginal data, and those concepts are disproportionately from the long tail of cultural and linguistic diversity. The practical consequence is that an organization optimizing for ImageNet and COCO leaderboard position should not invest in 100B-scale raw data (quality-filtered 5B subsets are equally effective and cheaper). An organization building globally deployed systems intended to serve users across cultures, languages, and geographies must invest in this scale, because no amount of filtering, algorithmic intervention, or model scaling can create cultural diversity that was never crawled.

Resolving prior contradictions. The paper's filtering analysis (Section 5.1, Figure 4) provides a definitive resolution to a tension that has been building in the data-centric AI community. On one side, works like Fang et al. (2023), Abbas et al. (2023), and Gadre et al. (2023) demonstrated that CLIP-based filtering, deduplication, and pruning substantially improve standard benchmark performance — more data is not always better if the data is noisy. On the other side, Pouget et al. (2024) and Nguyen et al. (2024) demonstrated that these same filtering techniques reduce cultural diversity in trained models. The tension was: is filtering good (because it improves quality) or bad (because it reduces diversity)? The paper's resolution — both are true simultaneously, because the evaluation determines the answer — reframes the filtering question from "should we filter?" to "what are we optimizing for, and who defines quality?" The CLIP-filtered 5B dataset achieves the best Western-centric performance in the paper (Figure 4, left panel) and the worst cultural diversity performance (Figure 4, middle panel). There is no single dataset that dominates on all metrics. This means the field's standard practice — optimizing datasets for aggregate benchmark performance and assuming this produces "better" models — is optimizing for a specific (Western-centric) definition of quality that may conflict with other valid definitions.

The paper also reconciles an apparent contradiction in the fairness literature. Prior work had shown that web-scale training data contains pervasive gender and racial biases (Birhane et al., 2021; Alabdulmohsin et al., 2024), and that scaling data can both amplify and mitigate different forms of unfairness depending on the metric (Dehghani et al., 2023). The paper's disaggregation of fairness into representation bias, association bias, and performance disparity — and the finding that only performance disparity improves with scale — provides a mechanistic framework for understanding when data scaling helps and when it does not. Coverage-based disparities (the model performs worse on certain geographic regions because it has never seen examples from those regions) improve with scale because more data fills the coverage gap. Statistical skews (the model associates "nurse" with women because web text consistently makes this association) do not improve because the skew is pervasive across the entire data distribution — adding more data reinforces the skew, it does not dilute it. This distinction between coverage problems and skew problems gives fairness researchers a concrete diagnostic tool: if a fairness metric measures a coverage gap, data collection targeted at the underrepresented group can help; if it measures a statistical skew, explicit bias mitigation (data balancing, algorithmic debiasing) is necessary regardless of data scale.

Which research directions become more attractive. The paper makes several directions newly compelling:

Diversity-preserving data filtering becomes an urgent priority. The demonstration that CLIP filtering systematically removes cultural content, and that a random baseline outperforms it on diversity metrics, means the field needs filters that optimize for both quality and cultural coverage — not filters optimized for Western benchmark performance and assumed to generalize.
Multi-objective dataset curation — explicitly trading off Western benchmark accuracy against cultural diversity and multilinguality — becomes a well-defined optimization problem with measurable outcomes, rather than an unquantified worry about "bias."
Scaling studies with disaggregated evaluation become the expected standard. A scaling paper that reports only aggregate ImageNet and COCO numbers is now demonstrably incomplete — it may be reporting saturation on one set of concepts while missing continued gains on another.
Pretraining data scale as a fairness intervention for coverage-based disparities is now empirically supported, making it a concrete engineering option alongside algorithmic fairness methods.

Which research directions become less attractive. The paper undermines several lines of argument:

"More data always helps." The paper shows this is false for Western benchmarks past ~10B examples. Additional resources spent on raw data crawling for these benchmarks are wasted.
"Better filtering solves the noise problem." The paper shows that current filtering techniques introduce a diversity cost that does not diminish with scale — filtering 100B down to 5B produces the same diversity penalty as filtering 10B down to 5B. Engineering effort spent on incrementally improving CLIP-based filtering without addressing its cultural biases will not produce more inclusive models.
"Scaling will solve bias." The paper shows it solves some bias (coverage-based performance disparities) but not others (stereotypical associations), meaning scaling advocates must specify which bias they are addressing.

Follow-Up Research This Work Enables

1. Diversity-preserving quality filters that explicitly balance benchmark performance and cultural coverage. The paper's finding that CLIP filtering reduces cultural diversity even when drawing from a 100B source pool (Section 5.1) demonstrates that current filters are biased in ways that no amount of source data can compensate for. The natural follow-up is to develop and evaluate filters that explicitly optimize for cultural coverage alongside alignment quality. A concrete experiment: take the raw WebLI-100B (or, for reproducibility, LAION-5B expanded to a larger crawl), train a filtering model that receives both alignment scores (from CLIP or a similar model) and cultural diversity signals (e.g., geographic distribution of image locations, language distribution of text, topic coverage across Dollar Street categories), and produce a filtered 5B-subset that matches the random English baseline on Dollar Street 10-shot and Crossmodal-3600 low-resource retrieval while maintaining the Western benchmark advantage of current CLIP-filtered data. The key metric is whether the Pareto frontier between Western accuracy and cultural diversity can be shifted outward — i.e., can filtering improve both simultaneously, or is there an inherent tradeoff? The paper's evidence suggests the current frontier is suboptimal (the random baseline achieves better diversity with similar Western performance to the classifier filter in Figure 4), meaning there is room for improvement.

2. Scaling curves for alternative contrastive objectives at the 10B–100B transition. The paper uses SigLIP loss exclusively, arguing it is well-suited to noisy data, but does not test whether the saturation point on Western benchmarks or the cultural diversity gains are loss-specific. A direct experiment would train CLIP-softmax and CoCa (Yu et al., 2022) models on the same 1B, 10B, and 100B data splits, matched for total FLOPs, and measure whether the qualitative pattern (Western saturation at 10B, cultural diversity gains at 100B) replicates. The hypothesis is that SigLIP's batch-independent per-pair computation makes it more tolerant of the noisy raw data than softmax contrastive loss, where misaligned pairs distort entire row normalizations. If so, softmax-trained models might show greater degradation from raw 100B data on Western benchmarks (because the noise penalty is higher), potentially making the case for filtering stronger for softmax recipes. Conversely, softmax models might show larger cultural diversity gains if the softmax normalization makes them more sensitive to rare positive pairs. Running this experiment at even one model size (ViT-L) would clarify whether the paper's central finding is a property of data scale or an interaction between data scale and training objective. The experiment is straightforward — it requires only a different loss implementation in the training loop, with the same data and infrastructure.

3. Language-specific data volume thresholds for emergent multilingual capability. The paper's Crossmodal-3600 results show that Telugu (0.036% of data, ~36M examples at 100B scale) benefits substantially from 10B→100B scaling while Maori (0.001%, ~1M examples) does not improve at all, even at 100B. This suggests there is a critical data volume threshold — some minimum number of examples below which a language is essentially invisible to the model, regardless of total dataset size. A systematic follow-up would: (a) identify or construct image-text datasets for languages spanning the representation range from 0.0001% to 0.1% of web data, (b) train a series of ViT-L SigLIP models on data splits where specific low-resource languages are artificially upsampled to target absolute example counts (100K, 300K, 1M, 3M, 10M, 30M, 100M), and (c) measure the relationship between absolute example count and Crossmodal-3600 retrieval performance for each language. The output would be a threshold curve — a function mapping language representation to expected retrieval quality — that could guide data collection efforts. If the threshold is sharp (e.g., performance jumps from ~random to ~50% recall around 3M examples), it would provide a concrete target for data crawling and language rebalancing strategies. If the threshold is smooth (gradual improvement with log data volume), it would imply that 100B scale is merely one point on a longer trajectory, and that substantially larger datasets (trillion-scale) are needed to serve languages below ~0.01% representation.

4. Interaction between model capacity and data diversity — does the ViT-H advantage on cultural diversity widen further with larger models? The paper observes that ViT-H gains 5.41 points on Dollar Street 10-shot from 10B to 100B, while ViT-L gains 5.80 and ViT-B gains 3.69 (Table 3). For multilingual low-resource retrieval, the improvement from 10B to 100B widens from 1.11 (ViT-B) to 2.76 (ViT-H) points (Figure 3). This suggests a complementarity: larger models extract more benefit from diverse data. The natural extension is to test this trend at larger model scales — ViT-G/14 (1.8B parameters) or ViT-e/14 (4B parameters) — using the same 10B and 100B data splits to see whether the cultural diversity gains continue to grow with model size, and whether the Western-centric saturation point shifts (does ViT-e still saturate on ImageNet at 10B, or does the increased capacity allow it to extract marginal benefit from 100B on Western tasks too?). If the complementarity holds, the cost-benefit calculus shifts: training a ViT-e on 100B data may be substantially more valuable (in absolute diversity gains) than training a ViT-H on 100B data, making data scale more attractive for large-model regimes. If the complementarity saturates (ViT-G gains no more than ViT-H), model capacity is not the bottleneck for extracting diversity signal.

5. Are Dollar Street gains driven by unique visual diversity or by correlated text signal? The paper attributes Dollar Street improvements to the 100B dataset's broader coverage of culturally diverse visual content. But an alternative explanation is that the gains are driven by correlated text signal — the 100B dataset contains more alt-text and page titles in non-English languages, and the improved text representations for low-resource languages (documented in Crossmodal-3600 results) indirectly help the model better categorize images from those regions, even when the images themselves are not culturally distinctive. A clean ablation: train two ViT-L models, both on the same 100B images but with (a) the original multilingual alt-text and (b) machine-translated English alt-text for all non-English text. If the Dollar Street gains persist in (b), the mechanism is visual diversity; if they disappear, the mechanism is text-driven and might be achievable through text-only interventions (e.g., multilingual text augmentation) without needing 100B images. The converse experiment — keeping the alt-text distribution at 10B scale but adding 100B-scale images with synthetic English captions — would test the complementary hypothesis that visual diversity alone is sufficient. Together, these experiments would decompose the diversity gain into visual and textual components, informing whether future data collection should prioritize image crawling (to capture diverse visual contexts) or text crawling (to capture diverse linguistic descriptions).

6. Stress-testing the filtering-diversity tradeoff: does the diversity penalty persist when filtering to larger subsets? The paper's filtering analysis (Section 5.1) compares 5B-example English subsets — CLIP-filtered, classifier-filtered, and random. But the filtering-to-5B design conflates quality selection with dataset size reduction. A natural stress test: apply the same CLIP filter to retain 20B, 40B, and 80B subsets from the 100B source, and measure whether the cultural diversity penalty shrinks as the filtered subset grows. If CLIP is systematically biased against non-Western content (assigning lower scores on average), then even an 80B filtered subset would underrepresent cultural content relative to an 80B random subset — the penalty might shrink but not disappear. If instead the penalty is driven by the filter's threshold (discarding the lowest-scoring pairs, which happen to be disproportionately non-Western), then relaxing the threshold to retain 80B instead of 5B might recover most of the cultural diversity while still providing some quality benefit over raw data. This experiment would produce a filtering Pareto curve — accuracy on Western benchmarks vs. accuracy on cultural diversity benchmarks as a function of the retained dataset size — which would be directly actionable for practitioners deciding how aggressively to filter their data. The paper's current results (comparing only 5B subsets) provide a single point on this curve, not the full tradeoff surface.

Practical Applications and Downstream Use Cases

1. Global-scale visual search and product recognition. A deployment scenario where the paper's findings have immediate relevance is visual search engines intended for global audiences — e.g., a user in rural India photographs a household item (a traditional cooking vessel, a specific style of textile) and wants to find similar products or information about the object. The paper's Dollar Street 10-shot results quantify the challenge: a ViT-L trained on 10B examples achieves 64.09% error on geo-localizing household items by country, while the 100B-trained model achieves 58.29% error (Table 3) — a 5.8-point improvement that translates to correctly identifying the cultural context of roughly 5–6 additional images per 100 queries. In a visual search system serving millions of users, this improvement means thousands fewer misrouted queries per day, directly affecting whether users in underrepresented regions find relevant results or encounter a broken experience. The practical recommendation from the paper is that such systems should invest in 100B-scale raw web crawling rather than relying on quality-filtered smaller datasets — the Dollar Street gains are driven by unique cultural diversity that filtering removes (Figure 4). The absolute error rate of 58.29% remains high for production, but the paper demonstrates a trajectory: each additional order of magnitude of raw data reduces the error rate by ~5–6 points on this task. A system architect can use this to estimate the data investment needed to reach a target performance level for their user base.

2. Cost-efficient data collection for low-resource language support. The paper's Crossmodal-3600 results (Figure 3) show that Telugu retrieval error drops from 76.67% (ViT-L, 10B) to 69.69% (ViT-L, 100B) — a 7-point improvement — and the language rebalancing experiment (Figure 5) shows that upsampling low-resource languages from 0.814% to 7% of the training data further reduces error to 70.10% (averaged across all 7 low-resource languages for ViT-L at 100B). This provides a concrete cost model for teams building multilingual VLMs. Telugu represents 0.036% of the web, meaning roughly 36 million examples in the 100B dataset. The rebalancing experiment shows that artificially inflating this to ~1B Telugu examples (1% of 100B) produces the best Telugu performance observed in the paper. A team launching a VLM product in Telugu-speaking markets can use these numbers to decide between: (a) crawling a general 100B dataset and hoping the natural 36M Telugu examples suffice, (b) crawling a 100B dataset with explicit Telugu upsampling to ~1B examples (the rebalancing approach), or (c) commissioning targeted data collection of ~500M–1B Telugu image-text pairs at a known cost per example. The paper's rebalancing results (Table 11) also show that upsampling low-resource languages causes only a small degradation in high-resource language performance (ViT-L high-resource retrieval degrades from 45.43% to 45.75% error) and Western benchmarks (from 26.87% to 27.55% average error), meaning the rebalancing intervention can be applied with modest cost to the majority-language experience.

3. Culturally aware content moderation and accessibility tools. Automated alt-text generation for images — where a model describes an image in text to make it accessible to visually impaired users — is a deployment context where cultural awareness directly impacts user experience. If a model trained predominantly on Western data encounters an image of a traditional Igorot dance and generates "people dancing at a festival," it provides a technically correct but culturally impoverished description. If the 100B-trained model recognizes the specific cultural context (as the attention map visualization in Table 1 suggests — the 100B model focuses on intricate patterns on traditional decorations), it can generate a more informative description. The paper's GLDv2 landmark recognition results (Table 3) provide a quantifiable analog: ViT-H improves from 40.08% error (10B) to 38.78% error (100B) on recognizing culturally significant landmarks — a 1.3-point improvement that represents correctly identifying several dozen additional landmarks from non-Western countries in the test set. For a content moderation system flagging culturally insensitive or mislabeled images, or an accessibility tool generating descriptions of culturally specific artifacts, the 100B scale provides measurably better cultural recognition that directly improves the quality of the text output for users from underrepresented cultures. The practical tradeoff is between the cost of the 100B data pipeline and the user experience improvement for the fraction of users who encounter culturally specific content — the paper's numbers allow a product team to estimate both sides.

4. Geographic fairness in medical and agricultural image analysis. While the paper's evaluations are on general vision-language tasks, the finding that performance disparity across geographic regions decreases with data scale (Table 5) has direct implications for domain-specific applications. The GeoDE region accuracy gap for ViT-L decreases from 4.0 percentage points (10B) to 2.8 points (100B) — a 1.2-point reduction in the maximum accuracy difference between the best and worst geographic region. If this pattern generalizes to medical imaging (where diagnostic models may perform worse on skin conditions in darker skin tones due to training data imbalance) or agricultural monitoring (where crop disease detection may fail on crop varieties common in sub-Saharan Africa but rare in North America), the paper's recommendation would be: invest in raw data scale rather than targeted collection for each domain separately, because the long-tail coverage of the general web crawl incidentally captures diverse medical and agricultural imagery. The caveat is that the paper does not evaluate on domain-specific tasks, so this is an extrapolation — but the mechanism (coverage of long-tail visual concepts improving with scale) should transfer to any visual task where the bottleneck is exposure to diverse examples of the target concept.

When to Prefer This Method

The paper explicitly frames its central question as an investment decision: "should one invest in scaling up the size of the pretraining dataset to 100 billion examples?" (Section 1). It provides clear empirical evidence that the answer depends on which evaluation metrics matter most for the deployment context. The guidance below is drawn directly from the paper's results and framing, not imposed as generic boilerplate.

Prefer scaling raw data to 100 billion examples (over quality-filtered smaller datasets) when:

Cultural diversity metrics are a primary evaluation criterion — the paper shows statistically significant gains on Dollar Street 10-shot, GeoDE 10-shot, and GLDv2 that quality-filtered data of any size cannot match (Figure 4, middle panel).
Serving low-resource languages is a product requirement — Crossmodal-3600 results show 1–8 point improvements for languages like Telugu, Bengali, and Swahili at 100B scale (Table 8), and language rebalancing amplifies these gains (Figure 5).
Performance parity across geographic regions or income levels is a deployment goal — Table 5 shows that scaling from 10B to 100B reduces the maximum accuracy gap across regions and income groups by 0.5–5.5 points, even without explicit fairness interventions.
The training compute budget is allocated primarily to training FLOPs and the data acquisition cost (crawling, storage, preprocessing 100B pairs) is acceptable — the paper does not quantify data acquisition cost, but the compute-matched comparison (all models process 100B total examples) shows that per-example training FLOPs are identical across scales.

Prefer quality-filtered smaller datasets (or accept 10B scale as sufficient) when:

Western-centric benchmarks (ImageNet, COCO, Flickr, standard VQA) are the sole evaluation criteria — Table 2 shows p = 0.9, meaning 10B and 100B are statistically indistinguishable on these metrics.
Only high-resource languages are targeted — Figure 3 shows high-resource language retrieval improves by only 0.1–1.3 points from 10B to 100B, versus 2.1–2.8 points for low-resource languages.
Representation bias (gendered label preferences) or association bias (stereotypical occupation-gender mappings) are the primary fairness concern — Table 4 and Figure 2 show these biases are unaffected by data scale, requiring explicit mitigation strategies (e.g., data balancing per Alabdulmohsin et al., 2024) regardless of dataset size.
Data acquisition cost dominates the total budget and the organization cannot amortize it across many model training runs — the paper provides no cost estimates for crawling 100B examples, but the infrastructure investment for a 10× larger dataset is presumably substantial, and if only a single model will be trained, the marginal benefit may not justify the fixed cost.