PaLM 2 Technical Report

🎯 Pitch

PaLM 2 matches or exceeds its 540B-parameter predecessor with a much smaller model, proving that smarter data mixtures and compute‑optimal scaling beat brute‑force parameter scaling alone. At the same time, it gains inference‑time controllability over toxicity without harming other capabilities, a critical advance for safer deployment.

1. Executive Summary

This technical report introduces PaLM 2, a new state-of-the-art Transformer-based language model that unifies compute-optimal scaling (scaling model parameters and training tokens in roughly 1:1 proportion, independently validating Hoffmann et al. 2022), an improved dataset mixture (a more multilingual and diverse pre-training corpus spanning hundreds of languages, code, mathematics, and parallel multilingual documents), and architectural and objective improvements (a tuned mixture of different pre-training objectives based on UL2). The largest model, PaLM 2-L, is significantly smaller than the largest PaLM model yet outperforms it across a diverse set of tasks—achieving passing grades on every evaluated professional language proficiency exam, improving BIG-Bench Hard performance by over 26% with direct prompting, and surpassing the dedicated Minerva model on mathematical reasoning—while simultaneously offering faster and more efficient inference. The report establishes that model scaling is not the only path to improved performance, demonstrating that meticulous data selection and efficient architecture can unlock capabilities that previously required a ~14× larger model, though memorization analysis reveals that under-represented languages can exhibit elevated verbatim memorization when training sequences are highly repeated.

2. Context and Motivation

The Core Gap: Scaling Alone Is Not the Only (or Best) Path Forward

The paper enters a landscape where the dominant narrative in large language model (LLM) development has been straightforward: increase model size, and performance improves predictably. This narrative was crystallized by the scaling laws of Kaplan et al. (2020), which suggested that model size should grow faster than training data size—in other words, to get a better model, you should primarily invest in more parameters. The field largely embraced this approach, producing progressively larger models: GPT-3 (175B), Gopher (280B), PaLM (540B), and eventually GPT-4 (parameter count undisclosed but widely believed to be enormous).

PaLM 2 explicitly challenges this orthodoxy on multiple fronts simultaneously:

First, the scaling ratio itself was wrong. Hoffmann et al. (2022) demonstrated that Kaplan et al.'s conclusion was an artifact of suboptimal hyperparameter tuning at smaller scales. When smaller models are tuned properly—specifically, ensuring the learning rate fully decays by the end of training—the optimal allocation of a fixed compute budget reverses: data size and model size should grow in roughly 1:1 proportion, not with models growing faster than data. The PaLM 2 report independently validates this finding at much larger scales (up to $1 \times 10^{22}$ FLOPs, Section 2.1), an important replication because scaling law conclusions are notoriously sensitive to experimental methodology and can change at different scale regimes.

The practical implication is stark: if you have a fixed training compute budget, the optimal model under the Kaplan et al. framework would be substantially over-parameterized and under-trained on data relative to the Chinchilla-optimal model. PaLM 2-L—significantly smaller than PaLM 540B yet trained on more data—is the living proof that this reallocation works at production scale.

Second, and perhaps more importantly, the paper argues that the quality and composition of training data matter just as much as pure quantity. Prior LLMs were trained on datasets dominated by English text. Chowdhery et al. (2022) report that approximately 78% of non-code data in PaLM was English. The implicit assumption was that adding more diverse, non-English data might dilute English performance—a zero-sum tradeoff between multilinguality and core English capability.

PaLM 2 directly contradicts this zero-sum assumption. By designing a more multilingual pre-training mixture spanning hundreds of languages and domains (including programming languages, mathematics, and parallel multilingual documents), the paper demonstrates that larger models can handle more disparate non-English data without degrading English language understanding. In fact, English performance improves substantially (Table 2: PaLM 2-L achieves 76.9% average accuracy on English QA and classification tasks vs. 70.4% for PaLM). This finding reframes multilinguality from a necessary compromise into a potential synergy—exposure to diverse linguistic structures may actually strengthen the model's language understanding capabilities across the board.

Third, the paper argues that architectural and objective innovations can contribute meaningfully beyond raw scale. Prior LLMs almost exclusively used a single causal language modeling objective (predicting the next token). PaLM 2 adopts a tuned mixture of different pre-training objectives based on UL2 (Tay et al., 2023), which combines causal language modeling with masked language modeling and other span-denoising objectives. The intuition is that different objectives teach the model different aspects of language understanding—causal modeling excels at generation fluency, while span-denoising may improve bidirectional reasoning and comprehension. By training on a mixture, the model learns a richer representation. The paper treats this as an important, though secondary, contributor to the overall gains, alongside data and compute-optimal scaling.

Why This Problem Matters: The Economics of Inference

The motivation for developing a smaller-but-better model extends beyond academic curiosity into hard economic realities. The paper makes explicit that these three innovations—compute-optimal scaling, improved data mixtures, and architectural improvements—collectively enable a model that is not just better, but also significantly cheaper and faster to serve.

This matters for several practical reasons articulated throughout the report:

Broader deployment: A smaller model can run on less powerful hardware, enabling deployment in resource-constrained environments (edge devices, consumer hardware, low-latency production systems). The report notes that "a smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model's downstream application for more applications and users" (Section 1).
Faster interaction: Lower inference latency enables more natural conversational pacing in applications like Bard. When a model takes seconds rather than minutes to respond, the user experience transforms from "query and wait" to genuine dialogue.
Environmental and operational costs: Inference compute dominates the lifetime cost of deployed LLMs. A model that achieves superior performance with fewer parameters reduces the carbon footprint and operational expense of every API call, every user query, every generated response.

The report frames this elegantly: "We thus find that it is more beneficial to invest more compute in training a smaller model compared to modifying a model's architecture to be more inference-efficient" (Section 6). In other words, the most inference-efficient model is simply a model with fewer parameters that was trained optimally, rather than a cleverly compressed version of a larger one.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in the prior landscape:

1. The Kaplan et al. scaling regime was suboptimal. Though Kaplan et al. (2020) provided the foundational framework for understanding LLM scaling, their empirical finding that model size should grow faster than data was based on undertrained smaller models. The paper's replication of Hoffmann et al.'s finding at much larger scales—showing that the 1:1 ratio holds up to $1 \times 10^{22}$ FLOPs—is a critical validation that the prior conventional wisdom was wrong, not just a quirk of small-scale experiments. Figure 5 in the report directly shows this, with the optimal parameter and token counts growing in lockstep across four orders of magnitude of compute.

2. Prior multilingual models faced an apparent English-multilingual tradeoff. Models like PaLM that were largely English-dominated (78% non-code English) achieved strong English benchmarks but relatively weaker multilingual performance. The intuition that adding more languages would necessarily dilute English capability had not been systematically challenged at scale. PaLM 2's experiment—dramatically increasing the multilingual data proportion while observing improved English performance—refutes this assumed tradeoff. The finding is important because it changes the calculus for pre-training data design: rather than carefully balancing English vs. non-English proportions, one can aggressively include diverse language data and rely on model capacity to absorb the benefit without penalty.

3. The UL2 objective mixture had not been validated at very large scale. Tay et al. (2023) demonstrated the benefits of a mixture of pre-training objectives at moderate scales (up to 20B parameters). Whether these benefits would persist or diminish at larger scales—where the raw signal from more data might overwhelm the benefits of a more sophisticated objective—was an open question. PaLM 2's strong results suggest that this architectural innovation does scale, though the report is careful to present it as one factor among several rather than the dominant driver of improvement.

4. No single prior model combined all three innovations. Chinchilla (Hoffmann et al., 2022) demonstrated compute-optimal scaling but used a relatively standard English-dominated data mixture and a single causal LM objective. GLaM (Du et al., 2022) explored mixture-of-experts architectures but trained on a conventional data mixture. UL2 (Tay et al., 2023) introduced objective mixtures but at smaller scale and without the data diversity innovations. PaLM 2 is the first publicly documented model to unify all three dimensions—compute-optimal scaling, a radically more diverse data mixture, and a tuned mixture of pre-training objectives—at production scale. The report can thus be read as an ablation: given all three, how much better can a model be than its predecessor? The answer, across the board, is "substantially."

How PaLM 2 Positions Itself

The paper positions PaLM 2 not as a single-technique breakthrough but as a synthesis of modeling advances, data improvements, and scaling insights. There is no single "key trick" that accounts for the improvements. Instead, the report argues that the field's prior focus on raw parameter count as the primary driver of capability was misplaced, and that roughly equal attention to data design and training methodology yields outsized returns.

Importantly, the paper is careful to distinguish between the pre-trained models (which are the subject of this report) and the fine-tuned variants and user-facing products that build on them. Section 1 includes an explicit disclaimer: "user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report." This distinction matters because it anticipates the natural question: "Does Bard achieve these numbers?" The answer is a clear "not necessarily"—and the report's evaluations are of the base model's capabilities, not of any deployed product's end-to-end performance.

The report also positions PaLM 2 within a broader responsible AI framework. Unlike many technical reports that treat safety evaluations as an afterthought, PaLM 2 dedicates substantial space to evaluating potential harms and biases across dialog, classification, translation, and question answering uses (Section 5 and Appendix D). This includes inference-time control via control tokens (Section 5.1), where conditioning on a special token during generation can significantly reduce toxic outputs without retraining or architectural modification. The paper explicitly frames these evaluations as tools for downstream developers: "These sections help downstream developers assess potential harms in their specific application contexts, so that they can prioritize additional procedural and technical safeguards earlier in development" (Section 1).

Finally, the paper positions itself in a specific temporal context—May 2023—and is explicit that PaLM 2 development continues: "Our research and development of PaLM 2 continues, including version updates for PaLM 2 as it is implemented into products." This forward-looking statement signals that the report's results represent a snapshot of an evolving model, not a final destination.

3. Technical Approach

3.1 Reader Orientation

PaLM 2 is a family of Transformer-based language models that generate text given a prompt, trained to maximize the probability of correctly predicting tokens in a sequence. The core problem is how to build a language model that achieves better performance than its larger predecessor while being smaller, faster, and more multilingual—solved by simultaneously optimizing three interdependent design dimensions: the ratio of model parameters to training tokens (compute-optimal scaling), the composition and quality of the pre-training dataset (improved data mixtures spanning hundreds of languages and domains), and the choice of pre-training objectives (a tuned mixture of language modeling tasks rather than a single objective).

3.2 Big-Picture Architecture (Diagram in Words)

The PaLM 2 system has five major components that interact in a sequential pipeline design:

Scaling Law Engine — a set of experiments on smaller models that determines the optimal relationship between model parameters $N$ and training tokens $D$ for a given FLOPs budget. This produces a scaling law (the "recipe") that specifies how large the model should be and how many tokens it should train on to minimize training loss. This component is used once, before the main model is built, to inform architecture decisions.
Pre-training Data Pipeline — ingests raw web documents, books, code repositories, mathematics content, conversational data, and parallel translation pairs across hundreds of languages; applies deduplication, quality filtering, and PII removal; injects special control tokens marking toxicity levels on a fraction of documents; and inserts canary sequences for memorization evaluation. The output is a cleaned, deduplicated, multilingual corpus significantly larger and more diverse than PaLM's.
Pre-training Objective Mixer — takes the cleaned corpus and applies a mixture of pre-training objectives based on UL2, rather than a single causal language modeling objective. Different spans of text are randomly assigned different training objectives (causal next-token prediction, span denoising, etc.), so the model learns to understand language through multiple complementary lenses during a single training run.
Transformer Model — a stack of Transformer layers, smaller in parameter count than PaLM 540B for the largest variant, trained on the mixed-objective data pipeline using the compute-optimal scaling recipe. The model ingests tokenized text, processes it through self-attention and feed-forward layers, and outputs a probability distribution over the vocabulary at each position. It is trained to minimize a weighted combination of losses from the different objectives.
Inference-Time Control Mechanism — a small fraction of pre-training data was tagged with special control tokens (e.g., <toxicity_level=low>) that mark the toxicity of the surrounding text. At inference time, a developer can condition the model's generation on a control token to steer output toxicity without retraining, changing architecture, or incurring additional computational overhead. This component is embedded within the pre-trained model's learned representations and activated by token-level conditioning.

Information flows as follows: the Scaling Law Engine determines the model size and token budget → the Data Pipeline prepares the training corpus → the Objective Mixer assigns a training objective to each training example → the Transformer Model trains on these examples, learning to predict tokens under the assigned objective → at deployment, the Inference-Time Control Mechanism allows external conditioning on behavior via special tokens, without modifying the underlying model.

3.3 Roadmap for the Deep Dive

First, the compute-optimal scaling methodology (Section 2 of the paper): the empirical procedure that determines the "recipe" for how many parameters and tokens to use. This is foundational because it governs the size of everything else.
Second, the scaling law results and their interpretation: what the optimal ratio is and why it contradicts prior work. This establishes the key architectural decision (smaller model, more data).
Third, the pre-training dataset design: what data is included, in what proportions, with what cleaning and filtering steps. This is the "fuel" for the model and explains the multilingual improvements.
Fourth, the architectural and objective improvements based on UL2: how the training objective differs from standard causal language modeling, and why a mixture of objectives helps. This is the "engine" that processes the fuel differently.
Fifth, the inference-time control mechanism using control tokens: how a small fraction of pre-training data is tagged with toxicity signals to enable steerable generation. This is a responsible AI innovation that operates orthogonally to the core language modeling pipeline.
Sixth, the context length extension: how the model was trained to handle longer sequences than PaLM without hurting benchmark performance. This is a structural capability improvement that enables new downstream uses.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical methodology paper whose core idea is that simultaneous optimization of the scaling ratio, data composition, and training objectives produces a model that is better than its larger predecessor across nearly all dimensions—multilingual capability, reasoning, coding, and responsible AI control—while being smaller, faster, and cheaper to serve.

Compute-Optimal Scaling: The Experimental Methodology

The paper's scaling law experiments are designed to answer a specific question: given a fixed training compute budget measured in FLOPs, what number of model parameters $N$ and what number of training tokens $D$ minimizes the model's final training loss? This is not a theoretical derivation but an empirical measurement, following the procedure established by Hoffmann et al. (2022).

The experimental design proceeds in four steps:

Step 1: Select compute budgets and train multiple model sizes at each budget. The authors choose four compute budgets spanning three orders of magnitude: $1 \times 10^{19}$ , $1 \times 10^{20}$ , $1 \times 10^{21}$ , and $1 \times 10^{22}$ FLOPs. For each budget, they train several models of different sizes, where the number of training tokens for each model is determined by the relationship:

$\text{FLOPs} \approx 6ND$

where $N$ is the number of model parameters and $D$ is the number of training tokens.

What this means operationally: if you have a compute budget of $1 \times 10^{22}$ FLOPs and a model with $1 \times 10^{9}$ parameters, you train it on approximately $1 \times 10^{22} / (6 \times 1 \times 10^{9}) \approx 1.67 \times 10^{12}$ tokens. If you have a larger model with $1 \times 10^{10}$ parameters, you train it on only $1.67 \times 10^{11}$ tokens—one-tenth as many—because each training step costs more. This creates a family of models at each compute budget that trade off parameter count against training duration.

Why this relationship: the factor of 6 comes from the approximate FLOPs per token for a Transformer forward-backward pass: roughly $2N$ for the forward pass and $4N$ for the backward pass. The relationship is approximate because it ignores embedding operations, attention masking, and other overhead, but it is standard in scaling laws literature (Kaplan et al., 2020) and sufficient for relative comparisons. Using an exact rather than approximate count would shift the absolute numbers slightly but not the shape of the scaling curves.

Step 2: Ensure consistent hyperparameter tuning. A critical methodological choice—and one that distinguishes this work from Kaplan et al. (2020)—is that the authors use cosine learning rate decay and ensure each model's learning rate fully decays to zero at its final training token. This matters because undertrained models (where the learning rate hasn't fully decayed) have artificially high loss, which would bias the scaling law toward recommending larger models and fewer tokens. Kaplan et al. (2020)'s original finding that models should grow faster than data was later attributed to precisely this issue: their smaller models were stopped before convergence, making them look worse than they should, and the apparent benefit of scale was partly an artifact of better convergence at larger sizes. By ensuring full learning rate decay for every model at every compute budget, the authors eliminate this confound.

Step 3: Construct isoFLOP curves and find the minimum. For each compute budget, the authors plot final validation loss against model size (specifically, non-embedding parameter count, which excludes the output layer parameters since they are shared with the embedding layer in these experiments). They fit a quadratic curve to these points—this is the "isoFLOP curve" shown in Figure 4—and find the minimum of that quadratic. The minimum indicates the model size that achieves the lowest possible loss for that FLOPs budget; any smaller model is undertrained (could have used more parameters), and any larger model is overtrained (ran out of tokens before converging).

Concretely, for the $1 \times 10^{22}$ FLOPs budget (Table 1), the authors train four models with parameter counts of 3.86B, 7.08B, 9.50B, and 16.1B. The corresponding losses are 2.488, 2.404, 2.400, and 2.406 respectively. The minimum is at approximately 9.50B parameters, which the quadratic fit confirms as the compute-optimal size. Models smaller than this (3.86B) have higher loss because they lack capacity; models larger than this (16.1B) have higher loss because they couldn't be trained on enough tokens within the budget.

Why quadratic fitting: the loss surface as a function of model size for fixed FLOPs is empirically well-approximated by a parabola near the minimum. Higher-order polynomials could overfit the small number of points (typically 4-5 per budget), and a linear fit would miss the optimum entirely. The quadratic fit is a pragmatic choice that has been validated in prior work.

Step 4: Fit the scaling law across budgets. With four estimates of the optimal model size (one per compute budget), the authors fit a power-law curve that predicts the optimal parameter count $N_{\text{opt}}$ and optimal token count $D_{\text{opt}}$ as functions of FLOPs. Figure 5 shows the resulting scaling law: both $N_{\text{opt}}$ and $D_{\text{opt}}$ grow roughly proportionally to the square root of the FLOPs budget, meaning they grow in equal proportion to each other. This is the key finding: data and model size should be scaled 1:1, not with model size growing faster as Kaplan et al. (2020) concluded.

The paper also notes a methodological refinement: they predict error bars on extrapolated predictions using a "leave-one-out" estimator, fitting the scaling coefficients with only 3 of the 4 points and measuring the prediction error on the held-out point. This provides uncertainty quantification on the scaling law's extrapolation to larger budgets, though specific error bar values are not reported in the main text.

Scaling Law Results and Interpretation

The results in Table 1 and Figure 5 establish that the optimal ratio of parameters to tokens is approximately 1:1 across all four compute scales tested. Specifically, at $1 \times 10^{22}$ FLOPs, the PaLM 2-optimal model size is approximately 10.7B parameters (trained on roughly $1.04 \times 10^{11}$ tokens), compared to the Chinchilla-optimal estimate of approximately 10B parameters. The close agreement with Hoffmann et al. (2022) is notable because that study was conducted at smaller scales and with a different training mixture—the fact that both studies converge on the same 1:1 ratio suggests this is a robust property of Transformer language models rather than an artifact of specific data or architecture choices.

What the ratio means operationally: if a project previously allocated a $1 \times 10^{22}$ FLOPs budget to train a model under the Kaplan et al. assumption (favoring larger models), the resulting model would have been substantially over-parameterized and under-trained on data—perhaps a 50B parameter model trained on only $3.3 \times 10^{10}$ tokens. Under the PaLM 2 scaling law, that same budget optimally buys a 10.7B parameter model trained on $1.04 \times 10^{11}$ tokens—roughly one-fifth the parameters but over three times the data. The smaller model with more data achieves lower loss, as confirmed by the empirical results in Table 1 where the 9.50B model achieves loss 2.400 versus 2.406 for the 16.1B model.

A critical nuance: training loss is not downstream performance. The paper explicitly cautions in Section 2.2 that the 9.50B model, which achieves the lowest training loss and is closest to the compute-optimal configuration, slightly underperforms the 16.1B model on downstream tasks (Table 15 in the appendix: average accuracy 57.68% for 9.50B vs. 58.26% for 16.1B, a difference of roughly 0.6 percentage points). This is a subtle but important finding: the scaling law optimizes for training loss (next-token prediction accuracy), but downstream task performance can diverge from training loss at the margin. The larger model, despite having slightly worse perplexity, generalizes marginally better to some downstream tasks. The paper attributes this to the difference between optimizing for token-level prediction (which the scaling law does) and optimizing for task-level accuracy (which involves different inductive biases).

The practical decision space. Because of this divergence, the choice of model size involves tradeoffs beyond pure loss minimization. A smaller model offers faster inference, lower serving cost, and lower latency—benefits that may outweigh a sub-percentage-point difference in downstream accuracy for many production deployments. The paper notes that "there are several other considerations besides the optimal training loss, such as training throughput and serving latency, which affect the decision regarding the optimal model size." The PaLM 2 family, with its Small, Medium, and Large variants, represents a practical exploration of this tradeoff space, with different model sizes optimized for different deployment scenarios.

Pre-Training Dataset Design

The PaLM 2 pre-training corpus is the most significant architectural departure from prior LLMs. Rather than treating data as a raw quantity to be maximized (as in the scaling laws literature) or as a mostly monolingual English resource (as in PaLM), the PaLM 2 corpus is deliberately designed across multiple dimensions: language diversity, domain coverage, data quality, and responsible AI considerations.

Component 1: Document sources. The corpus is composed of a diverse set of sources listed in Section 3: "web documents, books, code, mathematics, and conversational data." Unlike PaLM, which was heavily skewed toward English web text (approximately 78% of non-code data), PaLM 2 includes a substantially higher percentage of non-English data. The paper does not disclose the exact English percentage, but Table 21 lists the top 50 languages by percentage in the multilingual web documents, showing a distribution that is heavily multilingual: Spanish at 11.51%, Chinese at 10.19%, Russian at 8.73%, Japanese at 7.61%, French at 6.55%, and so on, with the tail extending to Telugu at 0.12% and Marathi at 0.13%. The paper notes that "we did not apply any filtering to explicitly keep or remove any languages"—the distribution reflects the natural language distribution of the web corpus used.

Why include so many languages: the explicit goal is to improve multilingual capabilities (translation, multilingual QA, language understanding across diverse languages) without degrading English performance. The paper's central claim is that "larger models can handle more disparate non-English datasets without causing a drop in English language understanding performance." This is an empirical hypothesis, not an obvious truth—prior intuition suggested a zero-sum tradeoff where adding non-English data would dilute the model's English ability because the total training budget is fixed. The results (Table 2: PaLM 2-L at 76.9% average English accuracy vs. PaLM at 70.4%) refute this, suggesting instead that multilingual data provides complementary linguistic signal that strengthens the model's general language understanding, not just its non-English capabilities.

Component 2: Parallel multilingual data. In addition to monolingual non-English data, PaLM 2 is trained on parallel data —source-target text pairs where one side is in English and the other is in a different language. The paper states this covers "hundreds of languages" (Section 3). Parallel data is particularly valuable because it teaches the model an explicit mapping between languages, which "ingrains an inherent ability to translate into the model" without requiring a separate translation objective or fine-tuning stage. The translation results (Table 9: PaLM 2 achieving MQM scores of 3.0 for Chinese→English and 0.9 for English→German, competitive with or better than Google Translate) are a direct consequence of this design choice. The paper notes that parallel data was "a minor part of the mixture" yet produced translation quality "on par with production translation services," suggesting that even small amounts of parallel data at scale can yield strong translation capabilities.

Component 3: Data cleaning and quality filtering. The paper describes "several data cleaning and quality filtering methods, including de-duplication, removal of sensitive-PII and filtering" (Section 3). De-duplication is specifically linked to memorization reduction: "apply deduplication to reduce memorization (Lee et al., 2021)." The memorization results (Figure 8) confirm that PaLM 2 on average memorizes significantly less training data than PaLM, with the medium-sized PaLM 2 model memorizing fewer sequences than even the least-memorizing model from the PaLM family. The removal of personally identifiable information (PII) is a responsible AI measure intended to reduce privacy risks.

Why deduplication matters for both performance and privacy: duplicated training data wastes compute (the model sees the same information multiple times) and dramatically increases verbatim memorization risk. Lee et al. (2021) showed that deduplication improves model perplexity because the model learns a more accurate data distribution rather than overfitting to repeated examples. The paper's finding that PaLM 2 shows significantly less memorization on sequences repeated fewer than three times (Figure 8b) is a direct consequence of the deduplication process.

Component 4: Toxicity control tokens. For a small fraction of pre-training data, the authors added special control tokens marking the toxicity level of the surrounding text, "using signals from a fixed version of the Perspective API" (Section 3). These tokens are not natural language; they are metadata tags inserted into the training sequence that the model learns to associate with the toxicity level of adjacent text. The paper states this is done "for a small fraction of pre-training data," implying the majority of data is untagged. The purpose is to enable inference-time control: by conditioning on a low-toxicity control token at generation time, the model can be steered toward safer outputs without modifying its architecture, retraining, or running an external toxicity classifier.

Why a fixed version of Perspective API: the paper notes they "used a version of the Perspective API to avoid drift as available signals improve over time (Pozzobon et al., 2023)." This is a methodological choice to ensure reproducibility—if the toxicity classifier used for tagging were updated over time, the control tokens in the training data would correspond to different underlying toxicity definitions, making it impossible to interpret or reproduce the control effect.

Component 5: Canary injection. Special "canary" token sequences were injected into the pre-training data to enable improved measures of memorization. The paper describes two types of canaries in Section 4.7:

Interleave canary: takes two real documents from the pre-training data and intersperses batches of $N = 50$ tokens from each document while preserving their internal ordering. This preserves some linguistic properties (the n-gram statistics within each batch are natural) while creating an overall sequence that would never occur naturally (alternating between two unrelated documents). The interleave structure makes the canary uniquely identifiable: the model can be prompted with the first document's prefix and tested on whether it continues with the second document's text, which would only happen if it memorized the specific interleaving pattern.
Shuffle canary: takes a single real training document and randomly shuffles all its tokens, removing all sequence-level ordering information while preserving the bag-of-words. This creates a sequence that is syntactically meaningless but uses real vocabulary.

The design intent is to balance two competing goals: canaries need to be sufficiently outlier-like that their memorization can be reliably attributed to rote learning rather than generalization (a completely natural paragraph might be "memorized" simply because the model learned the language well), but they also need to be sufficiently data-like that the model doesn't ignore them entirely during training (a sequence of completely random tokens would have no gradient signal because the model would assign it near-zero probability and the loss would not improve with training). The interleave and shuffle canaries provide this middle ground: they use real tokens and local n-gram structure (so they generate a training signal) but have an unnatural global structure (so any reproduction of that structure indicates memorization).

Languages are categorized into two bins—"large" and "small"—based on their total token count in the pre-training data. Large languages (English, Spanish, Chinese, Japanese, Russian) receive both shuffle and interleave canaries, with a total of 3,090 canaries per type per language distributed across repetition counts (40 at 1 repetition, 20 at 5, 20 at 10, 20 at 25, 15 at 50, and 15 at 100). Small languages (Turkish, Korean, Dutch, Farsi, Hindi, Greek, Hebrew, Slovak, Estonian, Bengali, Swahili, Tamil, Marathi) receive only interleave canaries with 60 per type at a single repetition count (10 each at 1 and 5 repetitions, none at higher counts). The asymmetry in canary design reflects the different research questions: for large languages, the goal is to study how memorization scales with repetition in general; for small languages, the goal is to study whether the "tail" of the data distribution exhibits elevated memorization risk.

Why repetition matters: prior work (Carlini et al., 2019; Lee et al., 2021) established that the number of times a sequence appears in training data is the dominant predictor of verbatim memorization. By injecting canaries at controlled repetition counts, the authors can measure the memorization rate as a function of repetition and compare it to the memorization rate on naturally occurring data where repetition counts are estimated post-hoc from the training corpus.

Component 6: Context length extension. The paper states that "PaLM 2 was trained to increase the context length of the model significantly beyond that of PaLM" (Section 3), though specific context lengths for either model are not disclosed. The paper asserts that "it is possible to increase the context length of the model without hurting its performance on generic benchmarks, which may not require longer contexts." This is a meaningful practical claim: extending context length typically requires architectural changes (e.g., different positional encodings) or training modifications that could trade off short-context performance for long-context capability. The claim that PaLM 2 achieves longer context without degrading standard benchmarks suggests that the model's increased capacity (from better data and objectives) absorbed the context extension cost.

Architectural and Objective Improvements: The UL2 Mixture

The PaLM 2 architecture is based on the Transformer, but departs from the dominant paradigm of using a single pre-training objective. The paper states: "Given the strong results of UL2 (Tay et al., 2023), we use a tuned mixture of different pre-training objectives in this model to train the model to understand different aspects of language."

What UL2 provides (as relevant to PaLM 2): UL2 (Unifying Language Learning, Tay et al., 2023) proposes that different pre-training objectives teach complementary aspects of language:

Causal language modeling (predicting the next token given previous tokens, as in GPT) teaches left-to-right generation fluency and is ideal for autoregressive decoding tasks.
Masked language modeling (predicting masked tokens from surrounding context, as in BERT) teaches bidirectional understanding and is ideal for comprehension tasks where the model needs to reason about both past and future context.
Span denoising (reconstructing corrupted spans of text) teaches the model to understand longer-range dependencies and to fill in missing information, which is useful for tasks like infilling, editing, and summarization.

UL2 unifies these objectives by randomly applying different "denoising" tasks to each training example during pre-training. A single training example might be processed as causal language modeling 60% of the time, masked language modeling 25% of the time, and span denoising 15% of the time—the exact mixture proportions are tuned and are not disclosed in the PaLM 2 report, which simply says "a tuned mixture."

Why a mixture rather than a single objective: the intuition is that no single objective teaches everything. A model trained purely on causal language modeling becomes excellent at fluent generation but may struggle with tasks requiring bidirectional reasoning (e.g., reading comprehension where the answer depends on context both before and after). Conversely, a model trained purely on masked language modeling (like BERT) excels at understanding but is not designed for autoregressive generation. By training on a mixture, the model develops both capabilities simultaneously—it learns to generate fluently (from the causal objective) and to reason about context bidirectionally (from the masked and span-denoising objectives). The paper's strong results on both generation tasks (translation, coding, creative writing) and comprehension tasks (classification, QA, reading comprehension) are consistent with this multi-objective training providing complementary benefits.

An important caveat: the paper does not disclose how the different objectives are reconciled during training—whether they use different loss weights, different token masking patterns, or task-specific prefixes that signal which objective is active. The UL2 paper specifies a mode-switching token ([NLU] for understanding tasks, [NLG] for generation tasks) that is prepended to the input to indicate the current objective. Whether PaLM 2 adopts this approach or a variant is not specified, making the exact mechanism of objective mixture a black box in this report.

A secondary claim: improved architecture beyond objectives. The paper states architectural and objective improvements as a combined contribution but provides almost no detail on what the architectural improvements entail beyond the Transformer backbone and UL2 objectives. The phrase "architectural and objective improvements" in Section 1 is the only mention, and no subsequent section elaborates on architectural changes (e.g., attention mechanism modifications, activation functions, normalization schemes, or positional encoding innovations). This is a notable opacity in an otherwise detailed report.

Inference-Time Control via Control Tokens

Section 5.1 describes a mechanism for controlling model behavior at inference time without modifying the model's weights. The approach is straightforward: during pre-training, a small fraction of documents are tagged with special control tokens that indicate the toxicity level of the surrounding text. At inference time, the user (or the application developer) prepends a control token to the prompt, and the model's generation is conditioned on this token.

The control tokens and their effect. The paper evaluates three control conditions—"low toxicity," "medium toxicity," and "high toxicity"—alongside a baseline with no control token. Table 14 shows the probability of producing a toxic continuation (toxicity probability ≥ 0.5 via Perspective API) from a non-toxic prompt:

No control token: 0.075
Low toxicity control: 0.033
Medium toxicity control: 0.116
High toxicity control: 0.203

The low-toxicity control token reduces toxic continuations by more than half (from 7.5% to 3.3%), while the high-toxicity token nearly triples toxic continuations (to 20.3%), demonstrating that the control token provides a graded "knob" for toxicity rather than a binary switch.

Mechanism of action. During pre-training, the model learns to associate the control token with the toxicity level of the surrounding text. For example, if a document tagged with <toxicity=low> consistently contains non-toxic language, the model learns that when it sees <toxicity=low>, the subsequent text should be non-toxic. This is an example of conditional generation —the control token acts as an additional conditioning variable that shifts the model's output distribution. The mechanism is identical in principle to how the model learns to condition on any other prefix: the prefix changes the model's hidden state, which changes the logits, which changes the sampled tokens.

Why this approach over alternatives: prior work on controlling LLM toxicity typically required either fine-tuning on curated safe data (Thoppilan et al., 2022; Ouyang et al., 2022), running an external classifier to filter or re-rank outputs (a compute-expensive inference-time operation), or reinforcement learning from human feedback (RLHF, which requires a separate training phase). Control tokens offer several advantages:

No additional compute at inference time: the control token is just another token in the prompt; conditioning on it requires no extra forward passes, classifier calls, or sampling.
No separate training phase: the control capability is learned during the standard pre-training run from naturally occurring (or naturally tagged) data, not from a separate fine-tuning stage.
Reversible and granular: the same model can produce high-toxicity or low-toxicity outputs simply by changing the control token, without retraining. This is useful for research and debugging, where developers may want to probe the model's behavior under different toxicity conditions.
Transparent to downstream fine-tuning: because the control token is just a token, downstream fine-tuning stages (instruction tuning, RLHF) can either preserve the control mechanism or overwrite it, depending on the developer's needs.

A key finding on effectiveness in different contexts. Figure 11 shows a comparison of control methods: control tokens alone, dialog prompting alone (a prompting template designed to elicit safe responses, adapted from Glaese et al., 2022), control tokens layered on dialog prompting, and a specialized dialog system (LaMDA). The surprising result is that dialog prompting alone is more effective than control tokens at reducing toxic generation in dialog contexts. On the standard ParlAI dataset, dialog prompting reduces the percentage of toxic responses from approximately 30% to approximately 12%, while control tokens alone on a conversational language modeling task reduce from roughly 30% to roughly 18%. Layering control tokens on dialog prompting provides a small additional gain (approximately 12% to 10%), but only on the standard dataset—not on the adversarial dataset.

This finding has a critical implication: general-purpose inference-time control mechanisms are significant but not a replacement for application-specific safety measures. The paper explicitly notes that "specialized downstream mitigation methods [like LaMDA] remain more effective than general-purpose inference time mitigations" and that developers should "continue to invest in application-specific mitigation methods."

An important limitation: no alignment tax observed. The paper notes that "we note no clear alignment tax nor penalty on other evaluation results, possibly from the small fraction of pre-training data that was tagged." This is important because training on control-tagged data could, in principle, degrade the model's performance on tasks unrelated to toxicity—the model might overfit to the control signal or learn spurious correlations. The absence of such degradation is positive but stated tentatively ("possibly from the small fraction"), reflecting the difficulty of conclusively ruling out subtle performance impacts without exhaustive ablation studies at scale.

4. Key Insights and Innovations

Innovation 1: Compute-Optimal Scaling Isn't Just a Recipe — It's a Refutation That Reshapes the Economics of LLM Development

The PaLM 2 report is, at its core, an argument that the field's dominant scaling paradigm was wrong in a way that had enormous practical consequences. The innovation here is not the scaling law itself — Hoffmann et al. (2022) established the 1:1 ratio — but rather the independent validation at an order of magnitude larger scale combined with the demonstration that this principle, when combined with data and objective improvements, produces a model that is simultaneously better and dramatically smaller than its predecessor.

What the field believed before. The Kaplan et al. (2020) scaling laws, which shaped the design of GPT-3, Gopher, and the original PaLM, prescribed that model size should grow faster than training data size. The practical consequence was a race to ever-larger parameter counts — from 175B to 280B to 540B — under the assumption that bigger models were the primary path to better capabilities. Even after Hoffmann et al. (2022) challenged this with Chinchilla (showing a 70B model could outperform Gopher-280B through better data scaling), the field's default remained "train the largest model you can."

What PaLM 2 demonstrates differently. The report shows that at $1 \times 10^{22}$ FLOPs — a scale substantially larger than Hoffmann et al.'s experiments — the 1:1 ratio holds (Figure 5, Table 1). But the deeper contribution is in Section 6's explicit reframing: "we find that it is generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget." This inverts the Kaplan-era logic entirely. The goal is no longer to maximize parameters for a given training budget, but to jointly optimize for training loss and inference efficiency — and the optimal solution is typically a smaller model trained on more data.

Why this is a conceptual shift, not just a recipe change. The report reframes model design as a total cost of ownership problem rather than a maximize training performance problem. The scaling law (Section 2) tells you how to minimize training loss; the downstream evaluation (Section 2.2, Table 15) shows that training loss is not downstream performance; the practical decision (Section 6) weighs serving cost, latency, and deployment breadth alongside accuracy. This three-part argument — scaling law, loss-vs-task divergence, inference economics — transforms compute-optimal scaling from an academic finding about perplexity into a pragmatic principle for production model design.

The report provides a concrete existence proof: PaLM 2-L, significantly smaller than PaLM 540B, outperforms it across essentially every benchmark while being cheaper and faster to serve. This is not a theoretical claim about what could be achieved; it's a demonstration that the Kaplan-era models were systematically over-parameterized and under-trained, and that reallocating compute toward data yields models that are better in both quality and practicality.

The significance beyond performance. This finding changes the calculus for organizations allocating LLM compute budgets. Rather than asking "how large a model can we train?", the question becomes "what combination of model size, data quantity, data diversity, and training objectives maximizes capability per unit of inference cost?" The report demonstrates that the answer often involves a smaller model with more diverse data and more sophisticated training — a fundamentally different design philosophy from the "scale parameters first" era.

Innovation 2: Multilinguality Is Not a Zero-Sum Tradeoff with English Performance — It's a Potential Synergy

Prior to PaLM 2, the dominant assumption in LLM data design was that adding non-English data to a pre-training corpus would dilute English performance. This was not irrational: if a model has a fixed training budget and a fixed capacity, spending more tokens on non-English text means spending fewer tokens on English text, and the model should, all else equal, become worse at English. The original PaLM's 78% English (non-code) data composition reflected this assumption — multilingual capability was secondary; English was the priority.

What PaLM 2 shows that contradicts this. The report's most counterintuitive finding is that dramatically increasing the proportion of non-English data — to the point where English is no longer the dominant language in some sub-corpora (Table 21 shows Spanish at 11.5%, Chinese at 10.2%, Russian at 8.7% of multilingual web documents) — improves English performance rather than degrading it. PaLM 2-L achieves an average of 76.9% accuracy on English QA and classification tasks versus PaLM's 70.4% (Table 2), a 6.5 percentage point improvement on a benchmark suite that is overwhelmingly English.

The mechanism is not fully explained, but the pattern is clear. The report attributes this to "higher data quality in the PaLM 2 mixture" (Section 3) and to the possibility that multilingual training provides complementary linguistic signal that strengthens general language understanding, not just non-English capabilities. The parallel data inclusion — source-target translation pairs covering hundreds of languages — may play a particularly important role: training on translation pairs explicitly teaches the model that different surface forms can encode the same meaning, which could improve the model's ability to abstract semantic content from language-specific surface patterns.

Why this reframes data design strategy. If multilinguality is not zero-sum, then the optimal pre-training data mixture is not a careful balance of English versus non-English but rather an aggressive inclusion of diverse languages, with the understanding that a sufficiently capable model will extract complementary signal from the diversity rather than being confused by it. This reframing is significant because it changes the cost-benefit calculation for including low-resource languages: under the zero-sum assumption, each token of Swahili or Telugu data "costs" a token of English data; under the synergy assumption, those tokens may provide linguistic signal that benefits English understanding as well as enabling Swahili or Telugu capabilities.

The language proficiency exam results (Figure 1) provide the most vivid evidence: PaLM 2 achieves passing grades on every evaluated language — Chinese, Japanese, French, Spanish, and Italian — at the C2 (mastery) level, something PaLM could not do. Yet simultaneously, English performance improved. The report demonstrates that the apparent tradeoff was an artifact of prior models' insufficient capacity or suboptimal training, not a fundamental constraint.

A critical caveat. The report does not provide a controlled ablation showing that adding multilingual data causes English improvement — it's possible that both English and multilingual improvements arise from the combination of better data quality filters, compute-optimal scaling, and the UL2 objective mixture, and that the multilingual data is correlated with rather than causal for the English gains. The claim of synergy is therefore suggestive rather than proven, but the refutation of the zero-sum assumption is solid: at minimum, adding substantial multilingual data does not degrade English performance, which was the prior concern.

Innovation 3: Inference-Time Control via Pre-Training Data Annotation Is a Lightweight, Effective Alternative to Post-Hoc Safety Mechanisms

The dominant approaches to controlling LLM output toxicity — RLHF (Ouyang et al., 2022; Bai et al., 2022), supervised fine-tuning on curated safe data (Thoppilan et al., 2022), and classifier-based output filtering or re-ranking — all share a common characteristic: they require additional training phases or additional inference-time computation beyond the base pre-trained model. The PaLM 2 control token mechanism (Section 5.1) represents a fundamentally different approach: embed the control capability directly into pre-training by tagging a small fraction of the training data with metadata tokens, and activate it at inference time through simple token-level conditioning.

What makes this distinct from prior approaches. The key conceptual move is treating behavior control as a pre-training design choice rather than a post-hoc modification. By inserting control tokens into a small fraction of pre-training data, the model learns to associate specific tokens with specific output distributions — in this case, toxicity level — without requiring a separate fine-tuning stage, without modifying the architecture, and without adding inference-time overhead. The control token is just another token in the prefix; conditioning on it costs nothing beyond what the model already does for any prefix.

Why the implementation details matter for the innovation's significance. The report emphasizes two design choices that make this approach practical: first, that "only a small fraction of pre-training data" needs to be tagged (Section 3), minimizing the annotation burden and the risk of degrading performance on unrelated tasks; second, that the tagging uses a fixed version of the Perspective API to ensure reproducibility and avoid definitional drift (Pozzobon et al., 2023). Table 14 demonstrates the effectiveness: the low-toxicity control token reduces toxic continuations from 7.5% to 3.3%, more than halving the risk, while the high-toxicity token nearly triples it to 20.3%, confirming that the control is graded and intentional rather than a binary safe/unsafe switch.

The significance is in the deployment model it enables. A pre-trained model with embedded control tokens can be deployed with different safety profiles for different applications or user populations simply by changing the token prepended to the prompt — no retraining, no separate model variants, no classifier infrastructure. This is particularly valuable in API settings where a single model serves diverse use cases with different safety requirements.

The limitations are equally instructive. Figure 11 reveals that dialog prompting alone is more effective at reducing toxic generation in dialog contexts than control tokens, and that the specialized LaMDA system outperforms both. The control token approach also shows effectiveness primarily on the "standard" toxicity dataset (explicitly offensive language) rather than the "adversarial" dataset (implicit, subtle harm). This establishes clear boundary conditions: control tokens are a general-purpose mechanism suitable as a baseline safety layer, but they do not replace application-specific safety engineering. The innovation is the lightweight, scalable, pre-training-integrated mechanism itself, not the claim that it solves all safety problems.

Innovation 4: Memorization Risk Is Not Uniform — the "Tail" of the Data Distribution Behaves Qualitatively Differently, Creating Language-Specific Privacy Vulnerabilities

Most prior work on LLM memorization (Carlini et al., 2021, 2022; Chowdhery et al., 2022) focused on aggregate memorization rates across English training data, establishing that larger models memorize more and that data duplication dramatically increases memorization risk. The PaLM 2 report's memorization analysis (Section 4.7) introduces a new dimension: language resource level matters independently of repetition count, and the mechanisms governing memorization in high-resource versus low-resource ("tail") languages are qualitatively different.

The core finding. Figure 9 reveals a striking pattern: for outlier canaries (artificially injected sequences designed to be rare), lower-resource languages require fewer repetitions for the canary to be extractable compared to higher-resource languages. That is, a canary repeated 5 times in a low-resource language might be memorized at rates comparable to a canary repeated 25 or 100 times in a high-resource language. However, for naturally occurring training data, there is no strong correlation between language size and memorization rate — in fact, under-represented languages often show less memorization on unique (single-repetition) sequences.

Why this matters conceptually. This dissociation between canary behavior and natural data behavior suggests that the tail's vulnerability is not a simple function of data scarcity. Low-resource languages have less data overall, which could lead to overfitting, but they also have less repeated data (because there are fewer documents to duplicate across), which reduces memorization. The canary results isolate the repetition effect: when a sequence is deliberately repeated in a low-resource language, it stands out more starkly against the sparse background and is more likely to be memorized. The natural data results show that this worst-case scenario may not be the common case — real low-resource data is often not highly repeated, so the aggregate memorization rate remains low.

The practical significance for multilingual model deployment. This finding identifies a specific, actionable risk: if a low-resource language corpus contains highly repeated sequences (e.g., the same template text appearing across many documents, or a frequently quoted passage), those sequences are at elevated risk of verbatim memorization compared to equivalently repeated sequences in high-resource languages. Developers deploying multilingual models should therefore prioritize deduplication and repetition analysis specifically for lower-resource languages in their training data, not just apply uniform deduplication across all languages.

The methodological innovation. The dual canary design — interleave and shuffle canaries, injected at controlled repetition counts across languages binned by resource level — represents a measurement framework for studying memorization that goes beyond aggregate extraction rates. By systematically varying the outlier-ness of the canary (interleave preserves some linguistic structure, shuffle destroys it), the repetition count, and the language resource level, the analysis decomposes memorization risk into constituent factors rather than treating it as a monolithic quantity. This methodology is transferable to future models and datasets.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans a wide range of benchmarks rather than a single dataset. For classification and question answering, the paper uses standard English tasks including TriviaQA, Natural Questions, WebQuestions, LAMBADA, HellaSwag, StoryCloze, WSC, WinoGrande, Winograd, SQuAD v2, RACE, PIQA, ARC, OpenBookQA, BoolQ, COPA, RTE, WiC, MultiRC, ReCoRD, CB, ANLI-R1/R2/R3, and TyDi QA (multilingual). For reasoning, it uses WinoGrande, ARC-C, DROP, StrategyQA, CommonsenseQA, XCOPA, and BIG-Bench Hard. For mathematics, it uses MATH, GSM8K, and MGSM. For coding, it uses HumanEval, MBPP, ARCADE, and BabelCode. For translation, it uses WMT21 and FRMT. For natural language generation, it uses XSum, WikiLingua, and XLSum. For language proficiency, it uses the latest publicly available C2-level exams in Chinese (HSK 7-9), Japanese (J-Test A-C), French (TCF), Spanish (DELE C2), and Italian (PLIDA C2), with the listening portions converted to additional reading questions. Responsible AI evaluations use CivilComments, Jigsaw Multilingual, ParlAI Dialogue Safety, BBQ, RealToxicityPrompts, and a custom multilingual representational bias benchmark. The memorization analysis uses 10,000 unique sampled English documents and canaries injected across 18 languages.
Base model(s). Three variants of PaLM 2 are evaluated: PaLM 2-S (Small), PaLM 2-M (Medium), and PaLM 2-L (Large). Unless otherwise specified, "PaLM 2" refers to the Large variant. For coding evaluations, a specialized variant called PaLM 2-S* is created by continuing to train PaLM 2-S on an extended, code-heavy, heavily multilingual data mixture. The predecessor model PaLM (540B) serves as the primary comparison point. The paper states that the models are "representative of the capabilities of many contemporary LLMs" and that results are averaged over the five last PaLM 2 checkpoints "for a more robust comparison for the large model." An instruction-tuned variant (Flan-PaLM 2) is used for reasoning evaluations.
Metrics. The primary metrics vary by task category. For classification and QA: accuracy (exact match for open-domain QA; standard accuracy for classification and multiple-choice tasks; F1 for MultiRC). For reasoning: accuracy or multiple choice grade, with some tasks using exact string match. For coding: pass@1 and pass@k, where a generated program passes a set of held-out test cases. For translation: BLEURT (version 0p2p1) as an automatic metric and MQM (Multidimensional Quality Metrics) from professional human translators, with errors weighted at 5 for major, 1 for minor, and 0.1 for minor punctuation errors. For NLG: ROUGE-2 for English and SentencePiece-ROUGE-2 (using the mT5 tokenizer) for all other languages. For language proficiency exams: scores computed by equally weighting reading and writing portions, with writing evaluated by three professional native speakers on a scale of 1-5, and a pass/fail assigned according to official guidelines. For responsible AI evaluations: AUC-ROC for toxicity classification; toxicity probability (Perspective API score ≥0.5) for generation tasks; percentage of toxic responses for dialog evaluations; gender agreement scores (0-3 scale) and translation quality scores (0-3 scale) for misgendering analysis. For memorization: percentage of 50-token suffixes exactly reproduced from 50-token prefixes using greedy decoding.
Baselines. The primary baseline is PaLM (540B) (Chowdhery et al., 2022), with results either reproduced from the original paper or recomputed for this work. For translation, Google Translate (production system) serves as an additional baseline. For reasoning comparisons, GPT-4 (OpenAI, 2023b) and state-of-the-art (SOTA) results are reported, where SOTA varies by dataset (e.g., QDGAT for DROP, DeBERTaV3-large+KEAR for CSQA, Minerva for MATH). For conversational safety, LaMDA (Thoppilan et al., 2022) serves as a specialized dialog system baseline. For memorization, PaLM models at comparable scales (small, medium, large) are compared. For coding, PaLM-Coder-540B is the primary comparison.
Generation budget / compute accounting. The paper does not use a uniform "generation budget" concept across all evaluations. For few-shot evaluations, the number of exemplars is specified per task (typically 1-shot for QA, 3-8 shots for reasoning, 5-shot for translation, 1-shot for NLG). For chain-of-thought reasoning, a fixed number of sample paths is used for self-consistency (64 for MATH, 40 for GSM8K). For coding pass@k, greedy sampling is used for pass@1 and temperature 0.8 with nucleus sampling p=0.95 for pass@k. For toxicity evaluations, greedy decoding is used for language modeling (single sample per prompt) while top-k (k=40, temperature=1.0) with multiple samples (10 or 20 per prompt) is used for dialog evaluations. For the scaling law experiments, compute is measured in FLOPs using the heuristic FLOPs ≈ 6ND, with experiments at four budgets: 1×10^19, 1×10^20, 1×10^21, and 1×10^22.
Cross-validation / statistical protocol. For the scaling law experiments (Section 2), the authors use a "leave-one-out" estimator to predict error bars on extrapolated predictions by estimating scaling coefficients with only 3 of the 4 compute budget points. For the main benchmark evaluations, results for PaLM 2-L are reported as the average over the five last model checkpoints to provide more robust estimates and capture checkpoint-level variance, with the paper noting that "there is some variance in results across model checkpoints." For human evaluation of language proficiency exams, three independent professional native speakers rate each writing portion, and the average score is used. For translation MQM evaluation, 7 professional translators evaluated English-to-German and 4 evaluated Chinese-to-English, with the final system-level score computed as an average over all annotations. For gender agreement in translation out of English, two raters evaluate each translation on 0-3 scales for both gender agreement and general quality independently, with results reported as averages. The standard deviation across the five checkpoints is reported for the memorization analysis (Section 4.7). For the large-scale benchmark evaluations, no cross-validation over data splits is performed—the test sets of established benchmarks are used directly.

Main Quantitative Results

English Classification and Question Answering

Table 2 presents the most comprehensive head-to-head comparison between PaLM 2 variants and PaLM 540B on 26 English QA and classification benchmarks in a 1-shot setting. The headline result: PaLM 2-M already outperforms PaLM 540B consistently (72.0% average vs. 70.4%), while PaLM 2-L achieves 76.9% average accuracy, a 6.5 percentage point absolute improvement over PaLM despite being significantly smaller.

Several individual results merit attention. On TriviaQA, PaLM 2-L achieves 86.1% exact match (EM), up from PaLM's 81.4%. On Natural Questions, performance rises from 29.3% to 37.5%—a relative improvement of 28%. On Adversarial NLI (ANLI), the improvements are particularly dramatic: ANLI-R1 jumps from 52.6% to 73.1% (+20.5 points), ANLI-R2 from 48.7% to 63.4% (+14.7 points), and ANLI-R3 from 52.3% to 67.1% (+14.8 points). These are challenging benchmarks specifically designed to probe model robustness, and the large improvements suggest the model learned more robust representations.

On reading comprehension, RACE-H (high school level) improves from 52.1% to 62.3% and RACE-M (middle school) from 69.3% to 77.0%. On commonsense reasoning, PIQA rises from 83.9% to 85.0%, ARC-C (challenge set) from 60.1% to 69.2%, and ARC-E (easy set) from 85.0% to 89.7%.

Two tasks show negligible or slightly negative change. On Winograd Schema Challenge (WSC), PaLM 2-L achieves 86.9% vs. PaLM's 86.3%—essentially flat. On WinoGrande, PaLM 2-L at 83.0% slightly underperforms PaLM's 83.7%. The paper notes this but does not analyze why Winograd-style tasks specifically do not benefit. On OpenBookQA, the improvement is modest: 58.5% vs. 53.6%.

A notable pattern across model sizes: PaLM 2-S (the smallest variant) achieves 69.9% average—competitive with PaLM 540B's 70.4%—despite being dramatically smaller. This is an early signal of the paper's central thesis: better data and training methodology can compensate for substantially fewer parameters.

Multilingual Question Answering

Table 3 presents TyDi QA results in both Gold Passage (reading comprehension with the relevant passage provided) and no-context (closed-book, relying on model parameters) settings, across 9 languages. All PaLM 2 variants consistently outperform PaLM in both settings.

In the Gold Passage setting, PaLM 2-L achieves 73.6% average F1, up from PaLM's 69.8%—a modest but consistent improvement. Differences between PaLM 2 variants are relatively small (PaLM 2-S at 73.3%, PaLM 2-M at 73.3%, PaLM 2-L at 73.6%), suggesting that even the smallest model has learned robust multilingual reading comprehension. The claim here is that reading comprehension with context provided is relatively easy for all model sizes once the data mixture supports multilinguality.

In the more challenging no-context setting, the performance spread across model sizes is larger: PaLM 2-L achieves 40.3% average F1, up from PaLM's 31.5%. PaLM 2-S (32.5%) is roughly comparable to PaLM (31.5%), while PaLM 2-M (36.2%) and PaLM 2-L (40.3%) show clear scaling benefits. The improvements are particularly pronounced for languages with limited data: Telugu rises from 9.6% to 12.2% (still very low, but a 27% relative improvement), Swahili from 39.7% to 50.3%, Bengali from 27.6% to 41.6%, and Korean from 35.0% to 46.9%. The paper highlights that improvements are "particularly pronounced for languages with limited data" and "languages with non-Latin scripts such as Arabic and Korean."

Multilingual Toxicity Classification

Table 4 in Section 4.2 presents toxicity classification AUC-ROC on the Jigsaw Multilingual dataset (French, Portuguese, Russian, Spanish, Turkish) and English Civil Comments, comparing PaLM and PaLM 2 in 0-shot and 10-shot settings. PaLM 2 significantly outperforms PaLM in almost all conditions. In the 0-shot setting, non-English average AUC-ROC jumps from 77.08% to 88.93%. Turkish shows the largest gain: from 84.10% to 93.42% in 0-shot. In the 10-shot setting, the gains are smaller but still present: non-English average rises from 89.21% to 91.10%. The one exception is Spanish in the 10-shot setting, where PaLM 2 (89.68%) slightly underperforms PaLM (91.23%)—the paper notes "slightly reduced performance in Spanish" but does not analyze this further.

Reasoning: BIG-Bench Hard

Table 6 provides per-task results on the 23 BIG-Bench Hard tasks, comparing PaLM and PaLM 2 with both direct prompting and chain-of-thought (CoT) prompting in a 3-shot setting. The headline: PaLM 2 with CoT achieves 78.1% average, up from PaLM's 65.2%—a 12.9 percentage point absolute gain, representing a 20% relative improvement.

Several tasks show improvements exceeding 50%:

multistep_arithmetic_two with CoT: from 19.6% to 75.6% (+56.0 points, +286% relative). This is a dramatic jump that the paper characterizes as "demonstrating new emerging abilities." The direct prompting version actually declined from 1.6% to 0.8%, suggesting the CoT prompting is essential for the capability to manifest.
temporal_sequences (direct): from 39.6% to 96.4% (+56.8 points, +143%). This task tests reasoning about when events occurred, and PaLM 2 approaches ceiling performance.
tracking_shuffled_objects (CoT): from 52.9% to 79.3% (+26.4 points, +50%).
dyck_languages (CoT): from 28.0% to 63.6% (+35.6 points, +127%). This task tests hierarchical reasoning using formal language syntax, and the large jump suggests PaLM 2 developed substantially stronger capacity for nested structure.
reasoning_about_colored_objects (direct): from 38.0% to 61.2% (+23.2 points, +61%).

A few tasks show degradation with CoT in PaLM 2 despite improving in PaLM: geometric_shapes drops from 43.6% to 34.8% with CoT in PaLM 2, and hyperbaton drops from 90.4% to 82.4%. The paper does not analyze these regressions but they suggest that CoT prompting can sometimes be counterproductive, possibly introducing irrelevant reasoning steps that confuse the model on certain task types.

Notable for its lack of improvement: snarks shows only 0.6 points gain with direct prompting (78.1% to 78.7%) but 23.0 points with CoT (61.8% to 84.8%), highlighting that CoT is essential for unlocking the gain on sarcasm detection. The task web_of_lies shows perfect performance (100%) with CoT for both PaLM and PaLM 2, suggesting ceiling effects; the direct improvement is modest (51.2% to 55.2%).

Reasoning: Mathematical and Commonsense Benchmarks

Table 5 provides aggregate results on WinoGrande, ARC-C, DROP, StrategyQA, CommonsenseQA, XCOPA, and BIG-Bench Hard, while Table 7 focuses on MATH, GSM8K, and MGSM.

On mathematical reasoning (Table 7):

MATH: PaLM 2 achieves 34.3% with CoT alone, rising to 48.8% with self-consistency (SC) using 64 sample paths. This is competitive with Minerva's 50.3% (a dedicated math model using continued pretraining on scientific content) and GPT-4's 42.5%. PaLM scored only 8.8%.
GSM8K: PaLM 2 achieves 80.7% with CoT, rising to 91.0% with SC (40 samples), outperforming GPT-4 (92.0%) and Minerva (78.5%). PaLM scored 56.5% with CoT and 74.4% with SC.
MGSM: PaLM 2 achieves 72.2% with CoT and 87.0% with SC, surpassing the prior SOTA of 72.0% (Flan-PaLM). PaLM scored 45.9% with CoT and 57.9% with SC.

The instruction-tuned version (Flan-PaLM 2) shows further improvements: GSM8K rises to 84.7% (CoT) and 92.2% (SC), MGSM to 75.9% (CoT) and 85.8% (SC). The MGSM result is notable because it tests mathematical reasoning across ten typologically diverse languages, demonstrating that the multilingual data mixture benefits reasoning capabilities across languages, not just language understanding.

On XCOPA (Table 5 and Table 17): PaLM 2 achieves 94.4% average accuracy in a 4-shot setting without chain-of-thought, compared to PaLM's 83.0% (standard) and 89.9% (with CoT). The per-language breakdown (Table 17) shows particularly large gains on under-represented languages: Estonian rises from 75.6% to 97.6%, Haitian from 77.2% to 91.4%, Swahili from 68.8% to 92.8%, and Cusco-Collao Quechua from 60.6% to 76.8%. PaLM 2 without CoT outperforms PaLM with CoT across all languages, indicating that the base model's multilingual commonsense reasoning is fundamentally stronger, not just better at leveraging chain-of-thought.

Coding

Table 8 presents coding results comparing PaLM 2-S* (the code-specialized variant) against PaLM-Coder-540B. On HumanEval, PaLM 2-S* achieves 37.6% pass@1 and 88.4% pass@100, versus PaLM-Coder-540B's 35.9% and 88.4%—matching the larger model's pass@100 while exceeding it at pass@1. On MBPP, PaLM 2-S* achieves 50.0% pass@1 and 86.6% pass@80, versus 47.0% and 80.8%—a clear improvement across both metrics. On ARCADE, PaLM 2-S* achieves 16.2% pass@1 and 43.6% pass@30, versus 7.9% and 33.6%—more than doubling pass@1.

Figure 6 and Table 18 show the multilingual coding results on BabelCode across 12 programming languages in the pass@1 setting. PaLM 2-S* outperforms PaLM-Coder-540B on all but two languages (C# and PHP, where performance is comparable). The most dramatic gains are in low-resource programming languages: Haskell jumps from 1.86% to 8.7% (4.7× improvement), and Julia from 4.35% to 16.77% (3.9× improvement). Remarkably, PaLM 2-S* performance on Java (31.06%), JavaScript (32.3%), and TypeScript (32.3%) exceeds Python (34.16%), despite Python being the original HumanEval language. This pattern suggests the code-heavy, multilingual training mixture created genuinely balanced multilingual code generation capabilities rather than Python-centric skills with weak generalization.

Translation

Table 9 presents WMT21 translation results for Chinese→English and English→German. The primary metric is MQM (human evaluation by professional translators, lower is better), with BLEURT as a supporting automatic metric. PaLM 2 achieves scores competitive with or better than Google Translate: Chinese→English MQM of 3.0 vs. Google Translate's 3.1 and PaLM's 3.7; English→German MQM of 0.9 vs. Google Translate's 1.0 and PaLM's 1.2. On BLEURT, PaLM 2 scores 69.2 (Chinese→English) and 73.3 (English→German), versus Google Translate's 68.5 and 73.0 respectively.

Table 10 presents FRMT (Few-shot Regional Machine Translation) results, testing dialect-specific translation for Portuguese (Brazil vs. Portugal) and Chinese (Mainland vs. Taiwan) in a 5-shot setting, scored with BLEURT. PaLM 2 improves over both PaLM and Google Translate in all four locale pairs. The largest gain is Chinese (Taiwan): PaLM 2 scores 72.0 versus Google Translate's 68.5 and PaLM's 68.6. This demonstrates that the multilingual training data provides not just language-level but dialect-level sensitivity, an important capability for producing natural-sounding translations for specific regional audiences.

The paper notes that parallel data was "a minor part of the mixture" (Section 3), yet PaLM 2 achieves translation quality on par with a dedicated production translation service. This is a striking finding about the efficiency of embedded translation learning at scale.

Natural Language Generation

Table 11 presents 1-shot NLG results on XSum (English), WikiLingua (6 languages), and XLSum (11 languages). The headline: PaLM 2-L achieves dramatic improvements over PaLM, ranging from 59.4% relative improvement on XSum to 100.8% on WikiLingua.

XSum (English only): PaLM 2-L achieves 23.2 ROUGE-2, up from PaLM's 14.5. PaLM 2-S (16.9) already exceeds PaLM, and PaLM 2-M (17.2) further widens the gap.
WikiLingua (average across ar, ja, ko, ru, th, tr): PaLM 2-L achieves 23.5 SentencePiece-ROUGE-2, up from 11.7. The per-language results (Table 19) show Thai benefits most dramatically: from 11.59 to 29.64.
XLSum (average across 11 languages): PaLM 2-L achieves 21.3, up from PaLM's 12.7. The per-language results show one surprising regression: Telugu drops from 5.59 to 6.86 for PaLM 2-L (both very low). Bengali shows a peculiar pattern: PaLM 2-M achieves 18.4 while PaLM 2-L achieves 15.87, though this may reflect variance rather than a systematic trend.

Table 12 shows the analysis of potential train/test overlap through 15-gram filtering. The "clean" proportion of training data (no 15-gram overlap with test) is 91.0% for WikiLingua, 53.1% for XLSum, and 58.0% for XSum. The filtered performance deltas are small and positive (0.3-0.6), indicating that "the model's performance is likely not inflated by memorizing the targets." This is an important robustness check given the scale of web-derived training data.

Language Proficiency Exams

Figure 1 presents the most visually striking result: PaLM 2 passes every evaluated C2-level language proficiency exam, including the writing portions, while PaLM failed or achieved lower scores across all exams. Specific scores:

HSK 7-9 Chinese: 82% (pass) vs. PaLM's 46% (fail). Writing: 62% (pass) vs. 33% (fail*).
J-Test A-C Japanese: 81% (pass) vs. PaLM's 94% (pass). Writing: 83% (pass) vs. 69% (pass). PaLM scored higher overall on Japanese (the one exception to uniform improvement), though both models passed.
TCF French: 94% (pass) vs. PaLM's 67% (fail*). PaLM 2 reached level C2 while PaLM reached only C1.
DELE C2 Spanish: 87% (pass) vs. PaLM's 70% (fail). Writing: 82% (pass) vs. 77% (pass).
PLIDA C2 Italian: 94% (pass) vs. PaLM's 25% (fail). Writing: 87% (pass) vs. 46% (fail).

The paper notes that exams were conducted under simulated conditions, writing portions were graded by three professional native speakers on a 1-5 scale, and "these are not official results." However, the dramatic gains—Italian improving from 25% to 94%, French from 67% to 94%—indicate genuine multilingual capability improvements beyond what benchmark datasets capture. The paper also notes that after removing contaminated examples (25-gram overlaps with pre-training data), Chinese scored 79% overall and Spanish scored 87% overall, with Italian and Japanese scores unchanged—suggesting that while some contamination existed, it did not substantively drive the passing results.

Memorization

Figure 8 presents verbatim memorization results on English training data. Panel (a) shows that PaLM 2 memorizes significantly less than PaLM on average across all three model sizes. The medium-sized PaLM 2 model memorized the most within the PaLM 2 family but "memorized significantly fewer sequences than even the least memorizing model from the PaLM family." Panel (b) disaggregates memorization by the number of repetitions of each 100-token sequence in the training data. When sequences are repeated only 1-2 times, PaLM 2 memorizes dramatically less than PaLM. However, when sequences are repeated more than ~10 times, PaLM 2 shows higher memorization rates than PaLM—a crossover effect that the paper hypothesizes "may be a side effect of de-duplication; because repeated n-grams are now both rarer and appearing in more unique contexts, this may lead to a higher likelihood of their memorization."

Figure 9 extends the analysis to multilingual memorization, comparing canary extraction rates and training data extraction rates across languages. Panel (a) shows that outlier canaries require fewer repetitions to be memorized in lower-resource languages—a 100-repetition canary in English has a similar extraction rate to a 5-repetition canary in some tail languages. Panel (b) shows that for real training data, there is "no strong correlation between language size and the training data extraction rate," and "under-represented languages often had less memorization on unique (single-repetition) sequences." However, highly repeated sequences in tail languages show elevated memorization risk.

Responsible AI: Toxicity and Bias Evaluations

Toxicity classification (Table 4): As reported above, PaLM 2 outperforms PaLM in both 0-shot and 10-shot settings on multilingual toxicity classification, with non-English 0-shot AUC-ROC rising from 77.08% to 88.93%.

Inference-time control (Table 14 and Figure 10): Conditioning on a low-toxicity control token reduces the probability of a toxic continuation from a non-toxic prompt from 0.075 (no control) to 0.033—a 56% reduction. Medium-toxicity and high-toxicity control tokens increase toxicity probability to 0.116 and 0.203 respectively, demonstrating graded controllability. Figure 10 shows that this effect holds across all strata of input prompt toxicity probability, and that PaLM 2 with no control token closely tracks the human baseline.

Dialog safety (Figure 11 and Figure 30): Dialog prompting significantly reduces toxic generation compared to conversational language modeling, but the effectiveness varies by dataset. On a single sample, dialog-prompted PaLM 2 performs similarly to the specialized LaMDA system. However, with 20 samples per query, over 30% of adversarial dataset queries and nearly 60% of standard dataset queries produce at least one toxic response, demonstrating that prompting-based control is "not effective at controlling toxic language harms when sampling multiple responses."

Multilingual representational bias (Figure 31 and Figure 32): Dialog-prompted PaLM 2 controls toxicity effectively for most languages (0-3.5% toxic responses), but English, German, and Portuguese show elevated rates (1-17.9% toxic responses). Disaggregating by identity terms reveals that in these three outlier languages, queries referencing "Black," "White," "Judaism," and "Islam" produce higher toxicity rates, demonstrating language-specific bias patterns that suggest "a bias in the level of toxic language produced for queries about different identity groups."

Translation misgendering (Tables 24, 25, 26, Figure 33): When translating into English, PaLM 2's performance is comparable to PaLM (97.2% overall accuracy vs. PaLM's 97.2%), with a minor improvement in disaggregated worst-case performance (58.7% vs. PaLM's 50.0% on Hindi "she"). When translating out of English (Table 26), results are mixed: PaLM 2 outperforms Google Translate on gender agreement in Spanish, Polish, and Portuguese, but shows lower gender agreement than PaLM in Telugu, Hindi, and Arabic. Translation quality broadly correlates with the percentage of pre-training data in each language (Figure 34), with Arabic as a notable exception showing lower quality than its data representation would predict.

BBQ bias benchmark for QA (Tables 27, 28, Figure 35): In disambiguated contexts (where the correct answer is specified), PaLM 2 achieves 91.4% accuracy. However, 3% of all disambiguated questions produce a biased response that reinforces a social stereotype. In ambiguous contexts (where the correct answer should be "unknown"), PaLM 2 is only 0.6% accurate, and biased responses occur 15.3 percentage points more often than anti-biased responses. A qualitative analysis (Section D.6) reveals cases where the model introduces stereotypes unrelated to the tested bias dimension—for example, in a question about who "exceeds expectations in class" between a Latino and Black student, the model answers "the white students," introducing a racial stereotype not present in the prompt.

Ablation Studies and Robustness Checks

Scaling law downstream performance vs. training loss (Table 15): The 9.50B parameter model, which achieves the lowest training loss (2.400) and is closest to the compute-optimal configuration, slightly underperforms the 16.1B model (loss 2.406) on downstream tasks: 57.68% average accuracy vs. 58.26%. This demonstrates that training loss is not a perfect proxy for downstream performance and that "scaling laws can be used to achieve optimal training loss for a given quantity of FLOPs, this does not necessarily transfer to achieving optimal performance for a given task."

Dataset contamination analysis for NLG tasks (Table 12 and Table 20): Filtering 15-gram overlaps between training data and test sets for WikiLingua, XLSum, and XSum shows small positive deltas in performance when using filtered data (0.3-0.6 ROUGE-2 points), providing evidence against score inflation from memorization. The per-language analysis (Table 20) shows that the Telugu subset has 92.5% clean data with a delta of 0.0, while Japanese XLSum has only 25.0% clean data with a delta of -0.8—one of the few slight negative deltas, though the paper notes this is within typical variance.

Control token effect across dialog and language modeling contexts (Figure 11 and Section 5.1): In conversational language modeling, control tokens reduce toxic responses from 30% to 12% (standard) and 18% to 7% (adversarial). However, in dialog uses, dialog-prompting alone is more effective than control tokens, and the specialized LaMDA system outperforms both. This demonstrates that the effectiveness of control tokens is context-dependent and that they complement rather than replace application-specific mitigations.

Continuous evaluation of toxicity during training (Figure 37): A "tiny" variant of the RealToxicityPrompts evaluation (500 examples, greedy decoding) run continuously throughout PaLM 2-L training shows stable performance: mean Toxicity Probability of Continuation of 0.0676 ± 0.0026, with worst-case performance of 0.0725 measured over 87% of training epochs. This confirms that the low toxicity baseline is not a cherry-picked checkpoint effect but a stable training property.

Comparison of RealToxicityPrompts evaluation variants (Table 29): The "small" variant (38k non-toxic prompts, greedy) and "full" variant (193k non-toxic + 55k toxic prompts, top-k sampling) show consistent patterns: PaLM 2 produces slightly lower toxicity probability than PaLM in all conditions (delta of -0.0020 for non-toxic prompts in the small variant, -0.0101 in the full variant), with toxic prompts showing a larger delta (-0.0527) suggesting PaLM 2 is more robust to toxic inputs. The consistency across evaluation variants supports the reliability of the finding.

Writing exam cross-contamination analysis (Appendix C): After removing questions with 25-gram overlap with pre-training data, the Chinese HSK score drops from the original to 79% overall, and Spanish DELE drops to 87% overall. Italian and Japanese scores are unchanged. The paper hand-checked each contaminated example and "ignored cases where the contamination was only at the level of standardised instructions." This analysis confirms that the pass rates are not entirely driven by memorization of exam content, though some contamination existed for Chinese and Spanish.

Canary design for memorization (Section 4.7 and Table 13): The dual canary design (interleave and shuffle) across two language bins (large and small) with controlled repetition counts enables a finer-grained analysis of memorization mechanisms than aggregate extraction rates. The finding that canary memorization rates differ sharply between high-resource and low-resource languages, while natural data memorization rates do not, would be invisible without the controlled canary methodology. Table 13 specifies the distribution: large languages receive 3,090 canaries per type with repetition counts from 1 to 100, while small languages receive 60 per type at only 1 and 5 repetitions.

Critical Assessment

Does PaLM 2 genuinely outperform PaLM by a large margin across diverse tasks?

The evidence supporting this claim is extensive and consistent. Table 2 shows a 6.5 percentage point improvement on the English QA/classification average (76.9% vs. 70.4%), with only 2 of 26 tasks showing negligible negative change (WSC, WinoGrande). Table 5 shows large improvements on reasoning tasks (90.9% vs. 85.1% on WinoGrande; 95.1% vs. 88.7% on ARC-C). Table 7 shows dramatic mathematical reasoning improvements (MATH: 48.8% vs. 8.8% with self-consistency). Table 11 shows NLG improvements ranging from 59% to 101% relative. Figure 1 shows universal passing of language proficiency exams where PaLM failed or scored substantially lower.

However, the strength of this claim is undermined by the opacity of the comparison conditions. The PaLM baselines are drawn from Chowdhery et al. (2022) and other prior work, which may have used different evaluation protocols, different numbers of few-shot exemplars, different decoding strategies, or different prompt formats. The PaLM 2 evaluations use "a custom 1-shot prompt for each dataset" (Section 4.6) and "the same prompts as in prior work" (various sections), but exact prompt compatibility is never systematically verified. It is possible that some fraction of the improvement reflects better prompt engineering rather than better model capability.

The per-checkpoint averaging for PaLM 2-L ("we average the results of the five last PaLM 2 checkpoints for a more robust comparison") is a strength for reliability but creates an asymmetry: PaLM results are typically single-checkpoint. If PaLM also showed checkpoint-level variance, this could inflate or deflate the apparent gain. The paper does not report standard deviations across checkpoints for most benchmarks.

The coding results (Table 8) are particularly convincing because the PaLM 2-S* model is dramatically smaller than PaLM-Coder-540B yet achieves comparable or better results—this is a strong falsification of the "bigger is always better" hypothesis. However, PaLM 2-S* was trained on "an extended, code-heavy, heavily multilingual data mixture" (Section 4.4), meaning it received additional code-specific training beyond the base PaLM 2-S model. This makes it a different comparison point: the improvement reflects not just architectural and data improvements but also task-specific continued training. The ARCADE result (16.2% vs. 7.9% pass@1) is the cleanest demonstration of genuine capability improvement, as ARCADE was designed with a "New Tasks split containing problems from newly curated notebooks to avoid evaluation data leakage."

Does the compute-optimal scaling finding hold at the scales tested?

This claim is well-supported by the scaling law experiments (Section 2.1), but the scope is narrower than the paper's narrative implies. The experiments test four compute budgets from 1×10^19 to 1×10^22 FLOPs, fitting power laws to the optimal model size at each budget. The 1:1 scaling ratio is clear in Figure 5, and the close agreement with Hoffmann et al. (2022) is striking given the different data mixtures and architectures.

However, the claim that PaLM 2-L is "significantly smaller than the largest PaLM model but uses more training compute" is not directly validated by the scaling law experiments. The scaling law experiments use different models and data mixtures than the final PaLM 2 models—the paper explicitly states: "It is important to note that the model sizes and total FLOPs in this section are only for the scaling law study, and do not reflect the model sizes and FLOPs used in PaLM 2 models." This means the scaling laws provide the principle for PaLM 2's design (smaller model, more data) but not the exact recipe. The actual PaLM 2 model sizes and training budgets are not disclosed, so we cannot verify that PaLM 2-L follows the 1:1 scaling ratio. The claim that compute-optimal scaling "enabled" PaLM 2's performance is therefore a design philosophy claim rather than a directly verified causal claim.

The divergence between training loss optimality and downstream performance (Table 15: 9.50B achieves lowest loss but 16.1B achieves highest accuracy) is a critical caveat that the paper acknowledges but does not deeply analyze. At a 0.6 percentage point difference on 1×10^22 FLOPs, this divergence may grow at larger scales, meaning the compute-optimal model for perplexity may not be the best model for tasks. Future work scaling to larger FLOPs budgets would need to track this divergence systematically.

Does improved data mixture drive the multilingual and English improvements?

The evidence is strong but circumstantial. PaLM 2 achieves dramatically better multilingual performance (Tables 3, 5, 9, 10, 11; Figure 1) while simultaneously improving English performance (Table 2). The paper attributes this to the "more multilingual and diverse pre-training mixture" (Section 1) and "higher data quality in the PaLM 2 mixture" (Section 3). However, there is no controlled ablation comparing the same model architecture and compute budget with different data mixtures. Without a "PaLM 2 with English-only data" baseline or a "PaLM 2 with PaLM's data mixture" baseline, we cannot separate the contribution of data mixture from the contributions of compute-optimal scaling and the UL2 objective mixture.

The translation results provide the strongest indirect evidence: PaLM 2 matches Google Translate on WMT21 (Table 9) and surpasses it on dialect-specific translation (Table 10), despite parallel data being "a minor part of the mixture." This capability was not present in PaLM, and the most parsimonious explanation is the inclusion of parallel multilingual data in pre-training. However, without ablating the parallel data specifically, we cannot rule out that the multilingual monolingual data alone would have produced comparable translation capabilities.

The finding that under-represented languages show particularly large gains (Telugu, Swahili, Quechua, Haitian) is consistent with the data diversity hypothesis—if the model previously saw very little data in these languages, adding even a small amount yields disproportionate improvements. But again, without controlled experiments varying the amount of low-resource language data, this remains a correlation.

Does PaLM 2 demonstrate genuinely new capabilities, or is this primarily benchmark saturation?

The BIG-Bench Hard results (Table 6) provide the strongest evidence for genuine capability emergence. Tasks like multistep_arithmetic_two (0.8% to 75.6% with CoT), temporal_sequences (39.6% to 96.4%), and dyck_languages (28.0% to 63.6% with CoT) show performance jumps from near-floor or moderate to near-ceiling or substantially improved, which is hard to explain as benchmark saturation. The dramatic MATH improvement (8.8% to 48.8%) similarly suggests genuine mathematical reasoning capability rather than overfitting to evaluation quirks.

However, the paper's reliance on few-shot prompting with hand-crafted prompts means that performance is confounded with prompt quality. The BIG-Bench Hard results use the "same 3-shot prompts" as Suzgun et al. (2022), providing some consistency, but the instruction-tuned variant (Flan-PaLM 2) was trained on a mixture that included WinoGrande, ARC, DROP, and COPA training data (Appendix A.2). The paper acknowledges this: "Training data (but not dev or test data) has thus been seen by the model as part of a very large mixture." For these tasks, the comparison with non-instruction-tuned models is not a clean measure of reasoning capability—the model may have benefited from memorizing task formats or patterns.

The language proficiency exam results (Figure 1) are compelling demonstrations of real-world capability, but the exams were "under simulated conditions" with the writing portions "graded by 3 third-party contractors." The sample sizes per exam are small (a single exam instance), the scoring rubrics are not described in detail, and no inter-rater reliability metrics are provided. The claim that PaLM 2 demonstrates "a level of language proficiency sufficient to teach that language" (Section 1) is therefore based on a very thin evidential foundation.

Does the memorization analysis demonstrate meaningful privacy improvements?

The memorization results (Figures 8 and 9) show consistent reductions in verbatim memorization on average compared to PaLM, which is a genuine improvement. However, the analysis reveals new risks that the paper partially acknowledges:

The crossover effect in Figure 8(b)—where PaLM 2 memorizes highly repeated sequences more than PaLM—is attributed to deduplication making rare sequences more salient, but no mechanism is experimentally verified.
The canary analysis (Figure 9a) demonstrates that tail-language canaries are memorized at lower repetition counts, identifying a specific vulnerability, but the paper's conclusion that "on average, we observe no strong correlation that under-represented languages may be memorized more" applies only to natural data without controlled repetition. The worst-case scenario (highly repeated tail-language data) is flagged but not quantified in terms of actual pre-training data risk.
The memorization analysis only covers 50-token verbatim sequences, which is one of many potential privacy attacks. Adversaries could exploit partial memorization, paraphrased memorization, or association-based leakage not captured by exact-match extraction.

The paper's framing that "PaLM 2 on average, memorizes less training data than PaLM" (Figure 8 caption) is accurate but incomplete without equal emphasis on the elevated tail risk.

Weaknesses in the experimental design:

Missing architectural ablations. The paper attributes improvements to "architectural and objective improvements" (Section 1) but provides zero detail on what architectural changes were made and no ablations separating the UL2 objective mixture from other architectural modifications. The UL2 contribution cannot be independently assessed.
Missing data mixture ablations. Without a controlled comparison varying the language distribution or data sources, the claim that improved data mixture drives multilingual and English gains remains a correlation. A "PaLM 2 trained on PaLM's data mixture" ablation would be the gold standard but is expensive; the report does not attempt even smaller-scale ablations.
Model size opacity. The actual parameter counts of PaLM 2-S, PaLM 2-M, and PaLM 2-L are not disclosed, making it impossible to verify the "significantly smaller" claim against PaLM 540B or to compute exact inference efficiency ratios. The scaling law experiments use models up to 16.1B parameters at 1×10^22 FLOPs, but these are explicitly stated to be different from the PaLM 2 models.
Single evaluation protocol per task. For most tasks, a single prompting strategy (1-shot, specific prompt format) is used. There is no sensitivity analysis to prompting variations, making it unclear whether the improvements generalize across prompt styles or are specific to well-tuned prompts.
Contamination analysis is limited. The 15-gram overlap analysis is only done for NLG tasks (XSum, WikiLingua, XLSum). For the many other benchmarks—including the English QA suite, reasoning benchmarks, and coding tasks—no contamination analysis is reported. Given that PaLM 2 was trained on a massive web corpus, the risk of benchmark contamination across the full evaluation suite is significant and largely unaddressed.
Human evaluation sample sizes. The MQM translation evaluation uses 4 annotators for Chinese→English and 7 for English→German. The language proficiency exams use 3 raters per writing portion. These are small sample sizes that generate point estimates without reported confidence intervals. The gender agreement evaluation (Table 26) uses 2 raters per translation, and no inter-rater reliability metric (e.g., Cohen's kappa) is reported.
No comparison with models of similar scale. The primary comparison is with PaLM 540B, a much larger model. Comparisons with models of PaLM 2-L's approximate size (whatever that is) from other organizations would contextualize whether the improvements come from the specific innovations or from general progress in the field.
Checkpoint averaging creates an asymmetric comparison. PaLM 2-L results are averaged over five checkpoints; PaLM baselines are typically single-checkpoint. If PaLM also exhibited checkpoint-level variance (which is likely), the apparent PaLM 2 advantage could be partially an artifact of using a more stable estimate for only one side of the comparison.

Experiments that would have strengthened the paper:

A controlled ablation training a model with PaLM 2's architecture and compute budget but PaLM's data mixture, to isolate data mixture effects.
A controlled ablation training a model with PaLM 2's data and compute budget but a single causal LM objective, to isolate UL2 effects.
A small-scale (e.g., 1B parameter) sweep of data mixture ratios (English/non-English proportions) to establish the dose-response curve for multilingual benefit vs. English impact.
Systematic prompt sensitivity analysis for 2-3 key benchmarks to quantify the variance attributable to prompt design.
Contamination analysis for the full evaluation suite, not just NLG tasks.
Reporting PaLM 2 model sizes to enable computation of exact inference efficiency ratios.
Head-to-head comparison with a non-Google model of similar parameter count (e.g., LLaMA, Falcon) to establish whether the innovations provide gains beyond field-wide progress.

6. Limitations and Trade-offs

Model Size and Architecture Opacity Prevents Independent Verification of the Core Efficiency Claim

The paper's central thesis—that PaLM 2 achieves superior performance while being "significantly smaller than the largest PaLM model"—cannot be independently verified because no actual parameter counts are disclosed for any PaLM 2 variant. The scaling law experiments in Section 2 use models ranging from 3.86B to 16.1B parameters, but the paper explicitly states that "the model sizes and total FLOPs in this section are only for the scaling law study, and do not reflect the model sizes and FLOPs used in PaLM 2 models" (Section 2.1). The actual PaLM 2-S, PaLM 2-M, and PaLM 2-L parameter counts are withheld.

The consequence. The claim that PaLM 2 is "significantly smaller" than PaLM 540B is unverifiable. If PaLM 2-L were, for example, a 400B parameter model, the efficiency gains would be modest rather than dramatic, and the paper's narrative about inference cost reduction would be substantially weaker. Similarly, the FLOPs-matched comparisons with larger models (a recurring theme throughout the report) cannot be replicated or scrutinized without knowing the baseline model sizes. For practitioners evaluating whether to adopt the PaLM 2 design philosophy, this opacity undermines the central economic argument: you cannot estimate the serving cost, latency, or hardware requirements of a model whose size is unknown.

What evidence exists in the paper. The scaling law experiments (Table 1, Figures 4 and 5) suggest that the compute-optimal model at $1 \times 10^{22}$ FLOPs is approximately 10.7B parameters, but the paper explicitly states this is not the PaLM 2 model size. The model card (Appendix D.10) states: "Further details of model size and architecture are withheld from external publication." There is zero disclosure of actual parameter counts for any PaLM 2 variant.

Mitigation status. Not addressed. The paper provides no justification for withholding model sizes (e.g., competitive concerns, safety considerations), acknowledges the withholding in the model card, and offers no proxy metrics—such as inference latency benchmarks, FLOPs-per-token measurements, or relative throughput comparisons—that would allow practitioners to estimate the practical efficiency gains independently. This is a fundamental transparency gap in what is otherwise a detailed technical report.

No Data Mixture Ablations Leave the Multilingual Synergy Claim as Correlation, Not Causation

The paper's most counterintuitive finding—that dramatically increasing the proportion of non-English pre-training data improves rather than degrades English performance—is presented as a key insight (Section 3: "we observe that larger models can handle more disparate non-English datasets without causing a drop in English language understanding performance"). However, the paper provides no controlled experiments isolating the effect of data mixture changes from the concurrent changes in compute-optimal scaling and the UL2 objective mixture.

The consequence. The improvements in both English (Table 2: 76.9% vs. 70.4% average) and multilingual tasks (Tables 3, 5, 9, 10, 11; Figure 1) could arise from:

The multilingual data itself (the claimed synergy mechanism),
The compute-optimal scaling recipe (training a smaller model on more tokens),
The UL2 objective mixture (which may improve language understanding generally),
Better data quality filtering applied to the entire corpus,
Some interaction among these factors.

Without ablating the data mixture—for instance, by training a PaLM 2-architecture model on the same compute budget but with PaLM's English-dominated data mixture, or by systematically varying the proportion of non-English data in smaller-scale experiments—the paper cannot distinguish these explanations. The claim that multilinguality is not zero-sum with English performance is therefore a correlation supported by compelling results, not a demonstrated causal mechanism. This matters for practitioners because if the English gains are primarily driven by better data quality filters and compute-optimal scaling (not the multilingual data itself), then the optimal data mixture for an English-first application might look very different from the one optimized for a general-purpose multilingual assistant.

What evidence exists in the paper. No data mixture ablation experiments are reported at any scale. Section 3 lists the data sources qualitatively but does not report proportions, compare alternative mixture ratios, or show what happens when non-English data is reduced. The paper notes that "even though PaLM 2 has a smaller proportion of English data than PaLM, we still observe significant improvements on English evaluation datasets" and attributes this "partially to the higher data quality in the PaLM 2 mixture"—an acknowledgment that multiple factors are confounded. Table 21 provides the language distribution of multilingual web documents but only for the non-English portion, leaving the overall English/non-English ratio undisclosed and unablated.

Mitigation status. Not addressed. The paper does not acknowledge the absence of data mixture ablations as a limitation, nor does it propose future work to disentangle the contributions of data diversity, data quality, and training methodology. For a report whose central thesis is that "meticulous data selection" is a key driver of performance, the lack of systematic data ablation experiments is a significant gap.

The Difficulty Estimation for Memorization Risk in Tail Languages Raises a Privacy Concern That the Paper Identifies But Cannot Quantify

The memorization analysis in Section 4.7 introduces a new dimension—language resource level—to the study of training data extraction. The canary experiments (Figure 9a) demonstrate that artificially injected outlier sequences in lower-resource languages require fewer repetitions to be memorized compared to high-resource languages. However, the paper's treatment of this finding as an edge case rather than a systemic risk leaves a critical gap for practitioners deploying multilingual models.

The consequence. The finding implies that if a low-resource language corpus contains highly repeated sequences—which is common when web data for a language is scarce and the same templates, news articles, or public-domain texts appear in many crawled documents—those sequences are at elevated risk of verbatim memorization compared to similarly repeated sequences in English or other high-resource languages. A developer building a multilingual product on PaLM 2 who assumes uniform memorization risk across languages (based on the aggregate finding that "PaLM 2 on average memorizes significantly less data" from Figure 8a) would miss this tail risk entirely. For languages with very small web presences, even natural (non-canary) training data may be repeated enough cross the elevated-risk threshold. The paper's conclusion that "on average, we observe no strong correlation that under-represented languages may be memorized more than well-represented languages" (Figure 9b caption) only applies to naturally occurring single-repetition sequences—it does not address the scenario where tail-language data is repeated, which is precisely the scenario that privacy-concerned deployments must consider.

What evidence exists in the paper. Figure 9a provides direct evidence: canaries repeated 5 times in some tail languages show extraction rates comparable to canaries repeated 25-100 times in high-resource languages. The paper acknowledges this: "in data sources with fewer documents, it takes fewer repetitions of these outlier canaries for extraction to succeed" (Section 4.7). However, the canary analysis uses only interleave and shuffle canaries at controlled repetition counts in 18 languages, and the natural data analysis in Figure 9b samples only up to 2,000 sequences per repetition bin per language—a small sample that may not capture worst-case repeated template text.

Mitigation status. Partially addressed through acknowledgment. The paper notes that "when sequences of a tail language are highly repeated, there are sometimes cases of significantly higher memorization rates" (Section 4.7) and that deduplication "may be a side effect" causing the elevated high-repetition memorization in PaLM 2 compared to PaLM (Figure 8b). However, the paper does not quantify how much tail-language data in the actual PaLM 2 training corpus falls into the high-repetition regime, does not provide per-language repetition distributions, and does not recommend language-specific deduplication thresholds for tail languages. The mitigation suggested is implicit (apply deduplication) but not tailored to the identified vulnerability.

The Single-Benchmark, Single-Model-Family Evaluation Strategy Limits the Generality of Every Major Claim

All results in Sections 4.2–4.6 and Appendix D are produced by PaLM 2 models evaluated on standard benchmarks, with PaLM (the predecessor from the same organization, trained on similar infrastructure) as the primary comparison point. While the breadth of benchmarks is impressive—26 English QA tasks, 9 TyDi QA languages, 23 BIG-Bench Hard tasks, 3 coding benchmarks, 2 translation directions, 3 NLG datasets, 5 language proficiency exams, and extensive responsible AI evaluations—they are evaluated on a single model family trained by a single organization, with no comparison to equivalently-sized models from other research groups.

The consequence. Several key claims cannot be assessed for generality:

The claim that compute-optimal scaling + data mixture + UL2 is the winning combination: Without comparisons to models like LLaMA, Falcon, or Mistral that achieve strong results with different design philosophies (e.g., LLaMA's parameter-over-data scaling, Falcon's standard causal LM objective), we cannot determine whether PaLM 2's specific innovations matter or whether the entire field has improved benchmark performance through better data curation, longer training, and more careful hyperparameter tuning—factors that would benefit any model family.
The claim that larger models can handle diverse non-English data without English degradation: This could be specific to PaLM 2's architecture, training infrastructure, or data processing pipeline. Other model families might show different English/multilingual tradeoffs at similar scales.
The claim that UL2 objective mixtures improve performance: Tay et al. (2023) showed benefits at up to 20B parameters; PaLM 2 extends this to a larger scale, but without comparing to a PaLM 2-equivalent model trained with a standard causal LM objective, we cannot attribute any specific fraction of the improvement to UL2.
The toxicity and bias evaluations (Section 5 and Appendix D): These are almost exclusively compared to PaLM, an older model. Without comparison to contemporary models from other organizations that may have invested differently in safety, the finding that PaLM 2 "exhibits stable performance on a suite of responsible AI evaluations" is relative to an internal baseline, not to the state of the field.

The paper acknowledges the representative claim: "we believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is stated about PaLM 2-S* specifically in the coding context and is not systematically argued for the full evaluation suite.

What evidence exists in the paper. The evaluation sections compare primarily to PaLM, with occasional comparisons to GPT-4 (Tables 5 and 7), Google Translate (Tables 9 and 10), Minerva (Table 7), and SOTA dataset-specific models (Table 5). These external comparisons are valuable but sparse—GPT-4 appears only for reasoning and math, Google Translate only for translation, Minerva only for math. There are no comparisons with publicly available models of comparable size (LLaMA-65B, Falcon-40B, Mixtral) on standard benchmarks, which would be the most informative test of whether PaLM 2's innovations produce gains beyond general field progress.

Mitigation status. Not addressed. The paper does not acknowledge the single-model-family limitation or argue for why comparisons with other organizations' models were not included. The exclusive reliance on PaLM as baseline is natural for an internal technical report but limits the external validity of every quantitative claim.

The Claimed Translation Parity with Google Translate Is Based on Extremely Narrow Evaluation

Section 4.5 reports that PaLM 2 achieves MQM scores competitive with or better than the Google Translate production system on two language pairs: Chinese→English (MQM 3.0 vs. 3.1) and English→German (MQM 0.9 vs. 1.0). The paper explicitly states that "at this scale, even though translation pairs were a minor part of the mixture, it allowed the model to be on par with production translation services" (Section 6). This is a headline claim with significant practical implications—it suggests that a general-purpose LLM with incidental parallel data can match a dedicated, production-optimized translation system.

The consequence. The evidence base for this claim is two language pairs evaluated by a total of 11 human annotators (4 for Chinese→English, 7 for English→German) on the WMT21 test sets. This is an extraordinarily thin foundation for a claim about parity with a production system that handles over 100 languages (Bapna et al., 2022). Translation quality is notoriously language-pair-dependent, domain-dependent, and sensitive to evaluation methodology—a system that matches or exceeds Google Translate on Chinese→English and English→German news-domain text may substantially underperform on other language pairs or domains. The paper provides no evidence that the translation parity generalizes beyond these two high-resource pairs.

Furthermore, the misgendering analysis (Section D.5, Table 26) reveals that when translating out of English in the zero-shot setting (which is how PaLM 2's translation is evaluated), PaLM 2 shows lower gender agreement than Google Translate in several languages including Arabic, Hindi, and Telugu, while outperforming Translate in Spanish, Polish, and Portuguese. This mixed pattern suggests that the "parity" claim masks significant variance across language pairs and evaluation dimensions—PaLM 2 is better at translation quality in some languages but worse at avoiding misgendering harms in others.

What evidence exists in the paper. The WMT21 evaluation (Table 9) covers two language pairs. The FRMT evaluation (Table 10) adds four locale-specific pairs (two Portuguese dialects, two Chinese dialects) using BLEURT automatic metrics only—no human evaluation for dialect translation. The misgendering evaluation (Tables 24, 26) covers 26 languages translating into English (automated evaluation) and 13 languages translating out of English (human evaluation with 2 raters per language). The out-of-English evaluation (Table 26) shows that PaLM 2's gender agreement scores range from 1.70 (Arabic) to 2.90 (French) on a 0-3 scale, and general quality from 1.61 (Arabic) to 2.85 (German)—a wide spread that suggests the "parity with production services" claim does not hold uniformly. Figure 34 shows a correlation between translation quality and the percentage of language tokens in pre-training data, but Arabic is a notable outlier (low quality despite moderate representation), indicating additional unidentified factors.

Mitigation status. The paper acknowledges limitations of the translation evaluation implicitly by reporting the misgendering results separately (Section D.5) and by noting that parallel data was "a minor part of the mixture" (Section 3). However, the headline claim of production parity (Section 6) is not qualified with the narrow evaluation scope, and no future work is proposed to validate translation quality across more language pairs, domains, or evaluation dimensions. The statement that PaLM 2 is "on par with production translation services" is substantially stronger than the evidence supports.

The Responsible AI Evaluations Are Primarily Decontextualized Benchmarks That May Not Predict Downstream Harms

The paper devotes extensive space to responsible AI evaluations (Section 5, Appendix D)—commendable in scope—but these evaluations share a structural limitation: they measure model behavior on standardized benchmarks rather than on real downstream applications with the pre- and post-processing steps that production systems employ. The paper itself acknowledges this in multiple places: "user-facing products typically include additional pre- and post-processing steps" (Section 1), "these evaluations are intended to help downstream developers assess potential harms" (Section 5.2), and "it is important to recognize that the models evaluated in this paper are pre-trained models" (Section 5).

The consequence. The gap between benchmark-measured harms and real-world harms in deployed systems is well-documented in the FAccT literature (Selbst et al., 2019; Blodgett et al., 2020; Raji et al., 2021). Several specific findings in the paper illustrate why this gap matters:

Dialog prompting substantially reduces toxic degeneration in the single-sample setting (Figure 30) but fails when sampling multiple responses—over 30% of adversarial queries and nearly 60% of standard queries produce at least one toxic response with 20 samples. A production dialog system that samples multiple candidates and selects the best is exposed to this multi-sample risk, but the paper's headline dialog safety results emphasize the single-sample improvement.
The multilingual representational bias evaluation (Figure 31) reveals that dialog prompting controls toxicity for most languages but fails for English, German, and Portuguese, with toxicity rates disaggregated by identity term (Figure 32) showing that queries referencing "Black," "White," "Judaism," and "Islam" produce elevated toxicity in these outlier languages. A developer targeting a Portuguese-speaking market who relies on the aggregate safety findings would miss this disaggregated risk entirely.
The BBQ QA bias evaluation (Section D.6) finds that in ambiguous contexts, PaLM 2 is only 0.6% accurate and introduces stereotypes not present in the prompt—for example, answering "the white students" to a question about a Latino and Black student's relative class performance. This type of hallucinated bias would not be captured by standard toxicity classifiers and would require application-specific testing to detect.

What evidence exists in the paper. The paper is transparent about the benchmark nature of its evaluations: it uses RealToxicityPrompts, CivilComments, ParlAI Dialogue Safety, BBQ, and Jigsaw Multilingual—all standard research benchmarks with known limitations (discussed in the CrowdWorksheets in Appendices D.9.1 and D.9.2). The measurement quality rubrics (Appendix D.8) explicitly score these evaluations on dimensions of relevance, validity, and inclusivity, with several dimensions rated "1 okay" (limited) rather than "2 good" or "3 great"—for example, the RealToxicityPrompts rubric notes that "more work is needed to show relationship with toxic language harms within tasks" and that the evaluation is "only in English."

Mitigation status. Addressed through acknowledgment and recommendations, but not through improved evaluation methodology. The paper repeatedly emphasizes that "application-specific analysis and evaluation of potential harms is critical" (Section 5.2), that "changes in decoding strategy and prompts can have a significant impact on generated responses" (Section 5.2), and that the results "should not be expected to exactly match user-facing products" (Section 5). The CrowdWorksheets and measurement quality rubrics represent an unusual and valuable transparency practice. However, the paper does not conduct any end-to-end application-specific harm evaluations, does not test the model with production-style pre- and post-processing, and does not provide guidance on how developers should map benchmark results to their specific use cases. The recommendation to "consider recent work demonstrating the promise of new methods for more precisely measuring application-specific harms" (Section 5.2) passes the responsibility to downstream developers without demonstrating how this mapping should be done.

7. Implications and Future Directions

How This Work Changes the Landscape

The PaLM 2 report does not introduce a single algorithmic breakthrough or a novel architecture. Instead, it functions as a large-scale existence proof that the field's prior scaling paradigm was suboptimal in ways that had direct, quantifiable economic and deployment consequences. The conceptual shift is deceptively simple—train a smaller model on more, better, and more diverse data—but the evidence that this produces a model which is simultaneously better and cheaper to serve across nearly every dimension reframes the design space for LLM development from "how large can we make it?" to "what combination of model size, data quality, and training methodology maximizes capability per unit of inference cost?"

This is a re-calibration of priorities, not a paradigm shift. The underlying technology (Transformer, autoregressive language modeling, scaling laws) remains unchanged. What changes is the answer to the optimization question: given a fixed total budget spanning both training and inference, where should the marginal dollar go? Prior to PaLM 2, the dominant answer—implicitly encoded in the architectures of GPT-3, Gopher, and the original PaLM—was "more parameters." Kapl et al. (2020) provided the theoretical justification; the economics of model scaling (train once, serve many times) made the tradeoff seem natural. PaLM 2 demonstrates that this answer was wrong for a wide range of practical deployment scenarios, and that the magnitude of the misallocation was large: the original PaLM 540B was approximately 50× larger than the compute-optimal model at $1 \times 10^{22}$ FLOPs (10.7B parameters, Table 1) yet underperformed PaLM 2-L—which is itself significantly smaller than 540B—across benchmarks. This is not a marginal correction; it is a finding that the field's default design heuristic was overshooting by more than an order of magnitude on parameter count.

The report resolves several tensions in the prior literature:

First, it reconciles the Kaplan et al. (2020) and Hoffmann et al. (2022) scaling law disagreement. By independently replicating the Hoffmann et al. finding at $1 \times 10^{22}$ FLOPs—an order of magnitude larger than Chinchilla's experiments—the paper establishes that the 1:1 parameter-to-token scaling ratio is robust across scale and data mixture, and that the Kaplan et al. prescription (models growing faster than data) was an artifact of undertrained small models. This is important replication work that converts a contested finding into a reliable engineering principle.

Second, it refutes the assumed zero-sum tradeoff between multilinguality and English performance. Prior work treated the inclusion of non-English data as a deliberate compromise—you accept weaker English to gain multilingual capability. PaLM 2 shows the opposite: dramatically increasing the non-English data proportion while simultaneously improving English benchmarks (Table 2: +6.5 percentage points average). This reframing has immediate practical consequences for data collection strategy. Under the zero-sum assumption, a product team targeting primarily English users would keep non-English data to a minimum. Under the synergy hypothesis that PaLM 2 suggests (but does not prove causally), including diverse language data may improve the model for all users, not just multilingual ones.

Third, it demonstrates that inference-time control can be embedded in pre-training at negligible cost. The control token mechanism (Section 5.1) shows that conditioning on a single token can halve toxic generation rates without additional inference overhead, separate fine-tuning, or architectural modification. This is not a research breakthrough—conditional generation is a well-understood mechanism—but it is a practical demonstration that safety features can be "baked in" during pre-training rather than bolted on post-hoc, and that this does not impose an alignment tax on other capabilities.

Research directions that become more attractive:

Data engineering as a first-class research investment. PaLM 2's results make clear that the returns to better data curation, filtering, deduplication, and mixture design are comparable to—or exceed—the returns to architectural innovation at current scales. This shifts the research portfolio: a team with fixed resources should likely spend more on data quality and less on novel architectures or training algorithms than the field's historical allocation would suggest.
Joint optimization of training and inference cost. The paper's framing of model design as a total-cost-of-ownership problem (training FLOPs + inference FLOPs over the model's lifetime) is a lens that the scaling laws literature has largely ignored. This opens a research agenda around deployment-aware model design: given a predicted inference volume, what model size and training budget minimize total cost for a target capability level? This is an optimization problem with clear economic value that the paper sketches but does not formalize.
Low-resource language inclusion as a general-purpose capability improver, not a niche feature. If the multilingual synergy finding generalizes, adding data from diverse languages becomes a strategy for improving model robustness and reasoning, not just for supporting those languages. This would shift the incentives for data collection in under-resourced languages from "charitable inclusion" to "strategic investment in model quality."

Research directions that become less attractive:

Scaling parameters as the default path to better models. PaLM 2 makes it harder to justify simply training a larger model without first exhausting data improvements. A 540B parameter PaLM was trained; PaLM 2 outperforms it with fewer parameters. The burden of proof now falls on advocates of larger models to demonstrate that the additional parameters provide benefits beyond what could be achieved by reallocating the same compute to better data and more training tokens.
Post-hoc safety patching as the primary mitigation strategy. The control token results suggest that safety considerations can be integrated into pre-training design rather than relegated to fine-tuning stages. This does not eliminate the need for application-specific safety measures—the paper explicitly shows that dialog prompting outperforms control tokens in dialog contexts—but it provides a baseline safety layer that reduces the burden on downstream mitigations.

Follow-Up Research This Work Enables

Causal decomposition of the multilingual synergy effect. PaLM 2 shows that adding multilingual data coincides with improved English performance, but the mechanism is unproven. A targeted follow-up would train a family of models at moderate scale (e.g., 1B–3B parameters) on a fixed compute budget while systematically varying the proportion of non-English data (e.g., 0%, 10%, 25%, 50%, 75%) while holding total tokens, data quality filters, and architecture constant. The key measurement would be English benchmark performance as a function of non-English data proportion, controlling for total training tokens. If English performance improves monotonically with non-English proportion up to some optimum and then plateaus or declines, this establishes a causal dose-response curve for the synergy effect. If English performance is flat or declining, the PaLM 2 English gains were due to other factors (data quality, compute-optimal scaling, UL2) and the synergy claim is refuted. A strong follow-up would also measure whether particular types of multilingual data (parallel translation pairs vs. monolingual non-English text vs. code) drive the effect, by varying data source composition independently of total non-English proportion.

Measuring the training loss vs. downstream performance divergence at larger scales. Table 15 shows that at $1 \times 10^{22}$ FLOPs, the compute-optimal model for training loss (9.50B parameters, loss 2.400) slightly underperforms a larger model (16.1B, loss 2.406) on downstream tasks by ~0.6 percentage points. This divergence could grow, shrink, or reverse at larger compute budgets, and the practical consequences are significant: if the gap widens, compute-optimal models become increasingly suboptimal for real applications, undermining the scaling law's practical utility. A systematic study would measure this gap across an order-of-magnitude range of compute budgets (e.g., $1 \times 10^{20}$ to $1 \times 10^{23}$ FLOPs), tracking both training loss and a standardized downstream benchmark suite for models at multiple sizes per budget. The key question: does the parameter count that maximizes downstream accuracy diverge systematically from the parameter count that minimizes perplexity as compute increases, and if so, by what functional form? The PaLM 2 data provides one point on this curve; a follow-up would fill in the rest.

Quantifying the tail-language memorization risk surface across real training data distributions. The canary experiments (Figure 9a) demonstrate that artificially repeated outlier sequences in low-resource languages are memorized at lower repetition counts than in high-resource languages. However, the paper does not quantify how much naturally occurring tail-language data falls into the "highly repeated" regime, making it impossible for practitioners to assess real-world privacy risk. A follow-up study would: (1) audit the actual PaLM 2 pre-training corpus (or a comparable large-scale multilingual web crawl) to measure the distribution of n-gram repetition counts per language as a function of language resource level; (2) train extraction attacks stratified by language and repetition count to produce a risk map showing the probability of extracting a sequence as a function of its repetition count and the language's total token count; (3) identify whether specific types of content (legal boilerplate, religious texts, widely-translated government documents) account for the bulk of high-repetition tail-language data, since these categories have different privacy implications than personal information. This would convert the paper's qualitative warning into an actionable risk assessment.

Combining control tokens with instruction tuning for steerable safety. The control token mechanism (Section 5.1) shows strong effectiveness in language modeling contexts (Table 14: 56% reduction in toxic continuations) but weaker effectiveness in dialog contexts where dialog prompting alone dominates (Figure 11). A natural extension would test whether control tokens can be integrated into instruction tuning: train a model where a fraction of instruction-tuning examples include control tokens specifying desired safety properties (e.g., "respond safely and avoid harmful content" prepended to safe responses, "respond edgily and push boundaries" prepended to intentionally unsafe responses). The hypothesis is that instruction tuning would strengthen the control token's effect in dialog contexts by providing explicit supervised signal for the desired conditional behavior, potentially closing the gap between general-purpose control tokens and application-specific safety prompting. The key measurement would be toxicity rates on the ParlAI adversarial dataset with and without control tokens after instruction tuning, compared to the dialog-prompting baseline.

Scaling the UL2 objective mixture: does it continue to help at 100B+ parameters? PaLM 2 demonstrates that the UL2 objective mixture contributes to improvements at whatever scale PaLM 2-L represents (which is undisclosed but likely in the tens to low hundreds of billions of parameters based on the scaling law context). Tay et al. (2023) validated UL2 up to ~20B parameters. An open question is whether the benefit of a multi-objective training regime diminishes as models scale—the intuition being that at very large scales, the raw signal from more data on a single objective might overwhelm the marginal benefit of objective diversity, or conversely, that larger models might benefit more from the regularization effect of training on multiple objectives. A systematic scale sweep—training UL2 vs. standard causal LM at 1B, 10B, 100B, and (if resources permit) ~500B parameters on comparable data—would answer this. The PaLM 2 report provides motivation but cannot disentangle the UL2 contribution from data and scaling improvements.

Open-vocabulary toxicity classification across languages to validate the multilingual safety findings. The multilingual representational bias evaluation (Figures 31, 32) identifies English, German, and Portuguese as outlier languages where dialog prompting fails to control toxic generation, and disaggregates toxicity by identity term. However, these evaluations rely on fixed template prompts translated into each language, which may not capture the full diversity of toxic or biased language in each linguistic context—template-based evaluations systematically miss emergent, context-specific, and culturally contingent forms of harmful language. A follow-up would collect open-ended user queries in the three outlier languages from native speakers interacting with the model adversarially (a structured red-teaming approach), then categorize the resulting harmful outputs to identify whether the template-based evaluation missed significant categories of harm. This would validate or refine the paper's finding that "queries referencing the 'Black' and 'White' identity group lead to higher toxicity rates" and determine whether additional mitigation strategies beyond dialog prompting are needed for specific languages or harm categories.

Practical Applications and Downstream Use Cases

Cost-efficient multilingual product deployment. The paper's demonstration that a model significantly smaller than PaLM 540B can outperform it across benchmarks—while simultaneously being cheaper and faster to serve—has immediate implications for organizations deploying LLM-powered products in multiple languages. Consider a customer support chatbot that must handle queries in English, Spanish, French, German, and Japanese. The naive approach is to deploy separate models fine-tuned per language, or a single very large multilingual model. PaLM 2's results suggest a third path: a single smaller model like PaLM 2-M (which already outperforms PaLM 540B at 72.0% average accuracy vs. 70.4% on English QA, Table 2, and achieves 73.3% F1 on TyDi QA Gold Passage vs. PaLM's 69.8%, Table 3) can serve all languages simultaneously while reducing per-query inference cost—potentially by a factor proportional to the parameter count reduction. For a product handling 10 million queries per day, a 10× reduction in model size translates directly to a ~10× reduction in serving infrastructure cost, assuming linear scaling of inference cost with parameter count.

On-device or edge deployment of capable language models. The report does not disclose PaLM 2-S's parameter count, but even PaLM 2-S achieves 69.9% average accuracy on English QA (Table 2)—competitive with PaLM 540B at 70.4%. If PaLM 2-S is in the single-digit billions of parameters (a reasonable extrapolation from the scaling law experiments showing optimal models at $1 \times 10^{20}$ FLOPs at ~1B parameters and $1 \times 10^{21}$ FLOPs at ~3.4B parameters, Table 1), it could plausibly run on consumer hardware (a high-end laptop or mobile device) with appropriate quantization. A developer building a privacy-sensitive application—such as a medical question-answering system that must process patient queries locally without sending data to cloud servers—could deploy PaLM 2-S or a quantized variant on-device and achieve PaLM 540B-level English performance. The TyDi QA no-context results (Table 3) further suggest that even the small model handles multilingual queries reasonably: PaLM 2-S achieves 32.5% average F1 vs. PaLM's 31.5%, with notably strong performance on Swahili (39.9% vs. 39.7%) and Indonesian (37.7% vs. 35.5%). This enables on-device multilingual applications in low-resource languages that previously required cloud offloading.

Pre-training data design as a strategic investment with measurable ROI. The paper's finding that data mixture and quality improvements drive performance gains comparable to or exceeding raw parameter scaling changes the calculus for organizations building proprietary LLMs. Rather than allocating budget primarily to compute infrastructure for larger models, teams should invest in data acquisition, cleaning, and diversity—specifically, in parallel multilingual data (which produced translation quality competitive with Google Translate despite being a "minor part of the mixture," Section 3), in code data for coding capabilities (PaLM 2-S* was created by continuing training PaLM 2-S on code-heavy data and dramatically outperformed the much larger PaLM-Coder-540B, Table 8), and in quality filtering and deduplication (which the paper credits for reduced memorization and improved performance). A practical heuristic: for a fixed total project budget, allocate a larger fraction to data engineering than the field's historical norm, and a correspondingly smaller fraction to training the largest possible model. The paper's specific results—MATH improvement from 8.8% to 48.8% (Table 7), coding pass@1 improvement from 35.9% to 37.6% with a smaller model (Table 8), language proficiency exam pass rates jumping from 25–70% to 82–94% (Figure 1)—provide concrete evidence that data investment pays off in task-specific capabilities.

Inference-time safety control as a lightweight baseline for API products. The control token mechanism (Section 5.1) enables API providers to offer multiple safety profiles from a single model instance without duplicating serving infrastructure. A provider could offer a "safe mode" endpoint (prepend the low-toxicity control token) and a "research mode" endpoint (no control token, higher toxicity but more creative) using the same model, same hardware, and same inference cost. Table 14 quantifies the differentiation: low-toxicity mode reduces toxic continuations by 56% (from 0.075 to 0.033 probability), while high-toxicity mode increases them to 0.203. The finding that "no clear alignment tax nor penalty on other evaluation results" was observed (Section 5.1) means the safe mode does not degrade general capability—a frequent concern with post-hoc safety interventions like RLHF, which can reduce model performance on capabilities benchmarks. For applications where the cost or latency of running a separate safety classifier or re-ranking outputs is prohibitive (real-time chat, high-throughput batch processing), the control token approach provides a zero-overhead safety baseline that can be combined with application-specific mitigations.

When to Prefer This Method

The PaLM 2 report establishes a specific design philosophy—train a smaller model on more, better, and more diverse data using a mixture of pre-training objectives—and positions it against the prior dominant approach of scaling parameters faster than data. The tradeoff is explicit throughout the report (Section 6: "it is generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget"), and the evidence for when each approach wins is grounded in the paper's results. The decision rule follows from the scaling law experiments (Section 2), the downstream evaluations (Section 4), and the inference efficiency argument (Section 6).

Prefer the PaLM 2 design philosophy (smaller model, more data, diverse mixture, multi-objective training) when:

Inference cost dominates the lifetime budget. If your model will serve many queries relative to training tokens (high $D_{\text{inference}} / D_{\text{train}}$ ratio), the inference savings from a smaller model compound. PaLM 2-M already outperforms PaLM 540B (72.0% vs. 70.4% average English accuracy, Table 2), so there is no capability sacrifice. The paper does not provide exact inference cost ratios, but the parameter count difference between PaLM 2-L and PaLM 540B—while undisclosed—is described as "significantly smaller," implying at least a several-fold reduction in per-token serving cost.
Multilingual capability is a product requirement. The report demonstrates that a diverse multilingual data mixture produces dramatic gains on translation (Table 9: matching Google Translate), multilingual QA (Table 3: TyDi QA no-context rising from 31.5% to 40.3% average), and language proficiency (Figure 1: passing all C2 exams). The synergy hypothesis—that this multilingual data may improve English performance rather than degrading it—further tips the scales toward diverse data for any application where users might benefit from non-English support, even if English is the primary language.
Latency constraints preclude large-model serving. The paper emphasizes "faster and more efficient inference" and "a more natural pace of interaction" (Section 1). For real-time applications (conversational AI, code completion in IDEs, interactive writing assistants), the lower latency of a smaller model is a hard requirement that cannot be compensated by better accuracy from a larger, slower model. The coding-specific PaLM 2-S* model demonstrates this directly: it outperforms PaLM-Coder-540B on pass@1 for HumanEval (37.6% vs. 35.9%, Table 8) and dramatically outperforms it on ARCADE (16.2% vs. 7.9%) while being "dramatically smaller, cheaper, and faster to serve" (Section 4.4).
Deployment hardware is limited (edge, on-device, or consumer-grade). PaLM 2-S achieves 69.9% average English accuracy (Table 2), competitive with PaLM 540B at 70.4%. If the small variant can run on constrained hardware, it enables applications that the large PaLM model could not serve at all without cloud offloading—privacy-sensitive medical QA, offline translation, on-device code completion, and any application where data cannot leave the device.

Prefer scaling parameters (larger model on less data, per Kaplan et al.) when:

Downstream task performance at the margin outweighs inference cost. Table 15 shows that at $1 \times 10^{22}$ FLOPs, the slightly larger 16.1B model achieves 58.26% average accuracy vs. 57.68% for the 9.50B compute-optimal model—a 0.58 percentage point advantage. In high-stakes applications where even fractional accuracy improvements translate to significant value (medical diagnosis support, legal document analysis, financial trading signals), the inference cost premium of a larger model may be justified by the downstream accuracy gain, however small. The scaling law optimizes for training loss; the application optimizes for task accuracy, and the paper shows these diverge.
The primary use case is pure English with no multilingual requirement, and data quality is already saturated. The multilingual synergy hypothesis is not proven causally—if English performance improvements come primarily from data quality and compute-optimal scaling rather than from multilingual data itself, and if an organization already has access to extremely high-quality, well-deduplicated English text at massive scale, the case for diluting the training mixture with non-English data weakens. The paper does not provide a data-only ablation to resolve this, so a conservative English-only deployment might default to the known scaling approach rather than betting on unproven synergy.
The model will serve very few queries relative to training cost (low inference-to-training ratio). If the model is trained for a specific one-time evaluation, a research benchmark submission, or a batch processing task where inference cost is negligible compared to the training investment, the inference efficiency argument for a smaller model evaporates. In this regime, the goal is to maximize capability for a given training budget regardless of serving cost, and the optimal model under a training-loss-minimization criterion might be larger than the compute-optimal model that accounts for inference cost. The paper does not explore this regime explicitly, but it follows logically from its distinction between training-loss optimality and downstream performance optimality.