Open Source and In-House: How Uber Optimizes LLM Training

ArXiv: 2204.02311

🎯 Pitch

PaLM 540B’s few-shot performance outperforms the average human on BIG-bench, with nearly 25% of its tasks showing discontinuous capability jumps that defy smooth power-law scaling predictions. It also matches or beats fine-tuned state-of-the-art on complex reasoning benchmarks using only chain-of-thought prompting, challenging the necessity of task-specific architectures.

---

1. Executive Summary

This paper introduces Pathways Language Model (PaLM), a 540-billion parameter, densely activated, decoder-only Transformer trained on 780 billion tokens of high-quality text spanning webpages, books, Wikipedia, code, and social media conversations. The work demonstrates continued benefits of scale across hundreds of NLP, reasoning, and code benchmarks—achieving state-of-the-art few-shot results on 28 of 29 widely evaluated English NLP tasks and breakthrough performance on multi-step reasoning tasks through chain-of-thought prompting (generating intermediate natural language reasoning steps before the final answer, such as solving grade-school math word problems step-by-step rather than predicting the answer directly). On BIG-bench, PaLM 540B 5-shot outperforms the average human performance score on aggregate across 150+ challenging tasks, and on roughly 25% of those tasks the model exhibits discontinuous improvements—performance jumps that are drastically larger when scaling from 62B to 540B than would be predicted by log-linear extrapolation from 8B to 62B. The paper also establishes that few-shot evaluation with chain-of-thought prompting alone can match or exceed the finetuned state of the art on arithmetic and commonsense reasoning tasks, demonstrating that specialized architectures and task-specific verifiers are not strictly necessary when model scale is sufficient—though this capability only emerges for problems within the base model's rough competence range, as evidenced by near-zero performance on the hardest questions regardless of compute.

2. Context and Motivation

The Core Problem: What Happens When We Keep Scaling?

By early 2022, the field had accumulated substantial evidence that scaling up language models—making them larger and training them on more data—reliably improved performance across NLP benchmarks. GPT-3 (Brown et al., 2020) had demonstrated that a 175B-parameter model could perform competently on a wide range of tasks through few-shot prompting alone, without any task-specific fine-tuning. Subsequent models like Gopher (280B parameters; Rae et al., 2021), Megatron-Turing NLG (530B parameters; Smith et al., 2022), and Chinchilla (70B parameters trained on 1.4T tokens; Hoffmann et al., 2022) had pushed performance further, each establishing new state-of-the-art results on various benchmarks at their time of release.

However, the trajectory of these improvements raised a fundamental question that the field had not yet answered: do the benefits of scaling continue, or are we approaching a point of diminishing returns? The paper identifies this uncertainty explicitly in Section 1, noting that prior improvements had come from multiple axes—model depth and width, number of training tokens, training corpus quality, and sparse architectures—but the interaction between these factors and their individual contributions remained poorly characterized. No single model had been systematically scaled across a wide enough range (from small to very large) while holding training data, architecture, and procedure constant, making it impossible to determine whether the scaling curve was flattening out or continuing to produce meaningful gains.

This question matters for several reasons the paper addresses directly or implicitly:

• Resource allocation: Training a 540B-parameter model costs tens of thousands of petaFLOP/s-days (the paper reports 29,600 for PaLM 540B in Appendix B) and requires thousands of accelerator chips running for weeks. Organizations need to know whether doubling or tripling that investment will yield proportional improvements, or whether the resources would be better spent on data quality, architectural innovation, or sparse model designs.
• Scientific understanding of emergence: Prior work had hinted that certain capabilities might only appear at sufficient scale—GPT-3 showed that few-shot learning itself was such an emergent capability, since smaller models couldn't perform it at all. But whether additional capabilities continue to emerge beyond 175B parameters was an open question with significant implications for how we think about model intelligence.
• The training-inference tradeoff: If a smaller model trained on more data could match a larger model's performance (as Chinchilla was beginning to suggest), then the optimal allocation of compute between pretraining and inference would shift dramatically. PaLM's design—a single 540B dense model trained on 780B tokens—represented one point in this design space, and understanding where it fell on the scaling curve was essential context for the Chinchilla findings that were published contemporaneously.

The Gap: Systematic Scaling Analysis Across a Wide Range

The paper positions itself as filling a specific gap in the literature: no prior work had systematically scaled a dense, decoder-only Transformer from moderate size (8B parameters) to very large size (540B parameters) while holding all other factors constant. This is more subtle than it sounds. The value of such a controlled scaling study is that it isolates the effect of model scale from confounding variables like training data composition, tokenizer choice, optimizer configuration, and architectural details.

Consider the comparison landscape at the time:

• GPT-3 (Brown et al., 2020) had scaled from 125M to 175B parameters, but the largest models used a different architectural variant (dense vs. sparse attention patterns), and the training data and procedures were not fully controlled across scales. Moreover, GPT-3's scaling analysis focused primarily on validation loss rather than downstream task performance across hundreds of benchmarks.
• Gopher (Rae et al., 2021) provided a detailed analysis of a 280B model but only compared against smaller models from different families (GPT-3, for instance), making it impossible to attribute differences to scale versus architecture or data.
• Chinchilla (Hoffmann et al., 2022) provided the most rigorous scaling analysis to date, but its core contribution was about the training data vs. model size tradeoff—showing that models like Gopher were significantly undertrained—rather than about what new capabilities emerge when you push model scale to 540B parameters on a fixed data budget.
• Megatron-Turing NLG 530B (Smith et al., 2022) had reached comparable parameter counts but was trained with a different architecture (pipeline parallelism, different parallelism strategy) and on different data, making direct comparison difficult. It also reported results on a narrower set of benchmarks.

PaLM's contribution is therefore not just "we built a bigger model," but rather: we built a family of models (8B, 62B, 540B) that differ only in scale, trained identically on the same data, using the same architecture, tokenizer, and optimizer. This enables the paper to make clean statements about what scaling alone achieves, separate from the other axes of improvement.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in prior work that it aims to address:

Limited benchmark coverage at the largest scales. Most large language model papers evaluate on a relatively small set of established benchmarks—often the same 20-30 English NLP tasks. PaLM dramatically expands this, evaluating on BIG-bench (150+ tasks, many designed specifically to be difficult for large models), reasoning benchmarks with chain-of-thought prompting, code synthesis and translation tasks, multilingual generation and translation across dozens of languages, and even explanatory generation (explaining jokes and performing multi-step logical inference in natural language). This breadth matters because scaling improvements on standard benchmarks could be misleading if the model is simply memorizing patterns in widely-used evaluation sets, whereas improvements on novel, challenging benchmarks like BIG-bench provide stronger evidence of genuine capability gains.

Lack of controlled multilingual evaluation. Prior large LMs had conducted limited multilingual evaluations despite training on multilingual data. GPT-3, for instance, showed some translation capability but results were mixed and primarily tested on high-resource language pairs. PaLM provides systematic evaluation across machine translation (including direct non-English-centric pairs like French-German and extremely low-resource pairs like English-Kazakh, where Kazakh has only 134M tokens in the training set—roughly 0.018% of the data), summarization in multiple languages, and multilingual question answering (TyDiQA). This is significant because approximately 22% of PaLM's training data is non-English, and understanding how this relatively small proportion translates to multilingual capability—and what the scaling behavior looks like for non-English tasks—was an open question.

Reasoning tasks require specialized architectures. Prior to PaLM, state-of-the-art results on arithmetic reasoning benchmarks like GSM8K required a combination of task-specific fine-tuning, domain-specific architectures (e.g., specialized modules for mathematical operations), and task-specific verifiers that check the correctness of generated answers. Cobbe et al. (2021) achieved 55% on GSM8K using GPT-3 fine-tuned on the training set, augmented with an external calculator and a trained verifier model. The prevailing assumption was that such specialized machinery was necessary for multi-step reasoning. PaLM challenges this assumption directly, showing that an 8-shot prompted model with chain-of-thought (and an optional calculator) can outperform the finetuned SOTA without any parameter updates—provided the model is large enough.

The "black box" of few-shot reasoning. While chain-of-thought prompting was introduced in Wei et al. (2022b), prior work had demonstrated it primarily on smaller models or in more limited settings. It remained unclear whether chain-of-thought improvements would continue to scale with model size, and whether the explanations generated by very large models would be qualitatively different—not just more accurate, but more coherent, more logically structured, and capable of handling more complex inference chains. PaLM's exploration in Section 9, where the model explains jokes requiring wordplay understanding and performs multi-step logical inference combining world knowledge (e.g., knowing that Mount Rushmore implies South Dakota, which implies Minnesota to the east, which implies St. Paul as the capital), demonstrated a level of explanatory depth that hadn't been documented before.

Discontinuous improvements were anecdotal, not systematic. Prior work had observed hints that scaling could produce sudden capability jumps—GPT-3 noted that few-shot learning itself seemed to "emerge" at a certain scale—but these observations were largely qualitative and task-specific. PaLM provides the first quantitative analysis of this phenomenon across a large benchmark suite, defining discontinuity precisely (as the gap between actual 540B performance and what log-linear extrapolation from 8B→62B would predict) and showing that roughly 25% of BIG-bench tasks exhibit discontinuities greater than 10 percentage points, with roughly 15% exceeding 20 points. This transforms "discontinuous improvement" from an interesting anecdote into a quantifiable, prevalent phenomenon with implications for how we set scaling expectations.

How PaLM Positions Itself

PaLM explicitly frames itself as continuing the scaling line of language modeling improvements rather than proposing a fundamentally new architecture or training paradigm. Section 1 states this clearly:

"In this work, we continue the scaling line of language modeling improvements and train a 540 billion parameter, densely activated, autoregressive Transformer on 780 billion tokens of high-quality text."

This positioning is strategic. By using a well-established recipe—dense decoder-only Transformer with standard autoregressive language modeling objective—the paper isolates the effect of scale itself. If PaLM were to introduce a novel architecture alongside increased scale, it would be impossible to determine whether performance improvements came from the architectural innovation or from the additional parameters and compute. Similarly, if PaLM had used a different training dataset or procedure than prior work, data quality effects would confound the scaling analysis.

The paper positions itself relative to four axes of improvement identified in prior work (Section 1):

1. Model depth and width — this is PaLM's primary axis of investigation
2. Number of training tokens — PaLM uses 780B tokens, which is substantially more than Gopher (300B) but less than Chinchilla (1.4T). The paper explicitly acknowledges this as a limitation and addresses it in Section 13 ("Open Questions in Scaling"), where it discusses the Chinchilla findings and presents additional experiments training PaLM 62B to 1.3T tokens on refreshed data.
3. Training corpus quality — PaLM uses a carefully filtered, high-quality corpus based on the LaMDA and GLaM datasets, which the paper argues may explain why PaLM 62B outperforms GPT-3 175B on several tasks despite lower total training FLOPs.
4. Increased model capacity without increased compute (sparse models) — PaLM is dense rather than sparse (unlike GLaM or Switch Transformers), again isolating the effect of scale rather than architectural efficiency.

This framing allows the paper to serve as a reference point for the field: by establishing what a 540B dense model with a standard recipe can achieve, it provides a baseline against which future work—sparse models, retrieval-augmented models, models with novel architectures—can be compared. It also creates a clean scaling curve (8B → 62B → 540B) that future work can extend or use to predict performance at even larger scales.

The implicit argument is that scaling alone, without architectural gimmicks, is a powerful and underexplored direction. The paper demonstrates this by achieving SOTA on 28 of 29 English NLP tasks, outperforming models that use sparse architectures (GLaM), different training objectives, or different architectural choices, simply by being larger and trained on high-quality data.

Reconciling with Chinchilla

The paper was written contemporaneously with Hoffmann et al. (2022), and Section 13 directly engages with the Chinchilla finding that many large models (particularly Gopher) were significantly undertrained relative to their parameter count. Chinchilla argued that for a given compute budget, it's better to train a smaller model on more data than a larger model on less data.

PaLM's position on this is nuanced. The paper acknowledges that PaLM 540B was trained on 780B tokens, which may not be compute-optimal by Chinchilla's analysis (a Chinchilla-optimal model at PaLM's compute budget would be smaller and trained on more tokens). However, the paper presents evidence that the scaling picture is more complex than a single optimal FLOPs-vs-parameters curve:

• When PaLM 62B is trained further to 1.3T tokens on refreshed data (Appendix F, Section 13), it significantly closes the gap with PaLM 540B but does not match it—suggesting that model scale provides benefits beyond what additional data alone can compensate for.
• PaLM 62B trained on 795B tokens already outperforms GPT-3 175B on average NLU and NLG scores (Section 6.1, Table 5), suggesting that data quality (axis 3) can partially substitute for model scale, making direct comparisons between models trained on different datasets unreliable.
• The discontinuous improvements observed on certain tasks (Section 6.2) suggest that some capabilities may simply require a minimum parameter threshold to emerge, regardless of how much data a smaller model is trained on. This is a counterargument to the Chinchilla view that scale is fungible: if certain capabilities are threshold-dependent rather than smoothly scalable, then larger models may be necessary even if they're not "compute-optimal" in the Chinchilla sense.

The paper ultimately positions itself not as contradicting Chinchilla, but as highlighting that the scaling picture depends on multiple interacting factors, and that understanding the tradeoffs between them requires empirical study at the largest scales—which is precisely what PaLM provides.

Summary of the Gap

The paper addresses a multi-faceted gap in the literature circa early 2022:

1. No controlled scaling study from ~8B to ~540B parameters with all other variables held constant
2. No systematic evaluation of whether scaling benefits continue or plateau across hundreds of diverse, challenging benchmarks (including BIG-bench, reasoning with chain-of-thought, code, and multilingual tasks)
3. No resolution to whether specialized architectures and fine-tuning are necessary for multi-step reasoning, or whether scale + prompting can substitute
4. No quantitative characterization of discontinuous scaling behavior across a broad task suite
5. No principled comparison between scaling model size and scaling training tokens in the context of downstream task performance (beyond what Chinchilla provided for validation loss)

PaLM addresses these gaps through a combination of controlled scaling, exhaustive evaluation, and detailed analysis, establishing a new reference point for the field and providing evidence that the scaling hypothesis had not yet reached its limits.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper is fundamentally about building and systematically evaluating a family of very large language models—specifically, dense decoder-only Transformers at 8 billion, 62 billion, and 540 billion parameters—to understand what happens when you scale up model size while holding architecture, training data, and procedure constant. The core problem it addresses is whether the benefits of scaling language models continue to accrue (or even accelerate) at unprecedented parameter counts, and whether emergent capabilities appear that are qualitatively different from what smaller models can achieve; the "shape" of the solution is a controlled scaling experiment where the only variable changed across model sizes is depth, width, and batch size, enabling clean attribution of performance improvements to scale alone.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that interact in a straightforward pipeline:

1. Training Dataset — a 780-billion-token corpus mixing filtered webpages, books, Wikipedia, news, source code from GitHub, and social media conversations across 100+ languages (with English comprising ~78%).

2. SentencePiece Tokenizer — a 256k-vocabulary tokenizer trained directly on the data that splits text into subword units, handles whitespace losslessly, and splits numbers into individual digit tokens (so "123.5" becomes 1 2 3 . 5).

3. Transformer Architecture (decoder-only) — a standard autoregressive Transformer with specific modifications (SwiGLU activations, parallel attention/MLP layers, multi-query attention, RoPE embeddings, shared input-output embeddings, no biases) that processes 2048-token sequences and predicts the next token at each position.

4. Training Infrastructure (Pathways + TPU v4) — 6144 TPU v4 chips across two pods, using 12-way model parallelism and 256-way data parallelism within each pod, plus two-way pod-level data parallelism via the Pathways system, all without pipeline parallelism.

5. Evaluation Framework — few-shot prompting (where the model receives a task description and optionally a handful of input-output examples, then generates predictions without any parameter updates) applied across hundreds of benchmarks spanning English NLP, reasoning, code, translation, and multilingual generation.

Information flows: raw text → tokenized training sequences of exactly 2048 tokens → forward pass through Transformer layers (computing next-token probabilities) → cross-entropy loss against actual next tokens → backward pass with gradient computation → optimizer update → repeat for 255k steps (540B model) → final checkpoint used for all evaluations via few-shot prompting.

3.3 Roadmap for the Deep Dive

• First, the training dataset — what data was used, in what proportions, and how it was filtered, because data quality and composition profoundly affect downstream performance and the paper makes specific claims about the importance of this axis.

• Second, the tokenizer and sequence packing — how raw text becomes model inputs, because vocabulary size, number splitting, and the treatment of document boundaries all affect what the model can learn.

• Third, the model architecture — each architectural modification (SwiGLU, parallel layers, multi-query attention, RoPE, shared embeddings, no biases) and WHY it was chosen, because these design decisions have computational and quality implications that compound at 540B scale.

• Fourth, the training setup — initialization, optimizer, learning rate schedule, loss function, batch size schedule, and the training instability mitigation strategy, because training a model at this scale requires careful tuning and specific techniques to avoid divergence.

• Fifth, the training infrastructure — how Pathways enables efficient scaling to 6144 chips without pipeline parallelism, the parallelism strategy, and the model FLOPs utilization (MFU) metric, because the engineering achievement is itself a contribution and governs what scale is practical.

• Sixth, the evaluation methodology — how few-shot prompting works across different task types (classification, generation, reasoning with chain-of-thought, code synthesis, translation), because the evaluation protocol determines what claims can be made about model capabilities.

This order follows the natural flow from raw data to trained model to evaluation, mirroring how the system is actually built and used.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a scaling paper whose core idea is that increasing model parameters, when combined with high-quality data and careful training, produces capabilities—including discontinuous, emergent ones—that cannot be achieved by smaller models trained on the same data, and that these improvements have not yet plateaued even at 540 billion parameters.

---

Training Dataset Construction and Composition

The PaLM pretraining dataset consists of 780 billion tokens drawn from a mixture of sources designed to represent a wide range of natural language use cases while maintaining high quality through aggressive filtering. The dataset is based on the datasets used to train LaMDA (Thoppilan et al., 2022) and GLaM (Du et al., 2021), with modifications to the mixing proportions and the addition of code data.

Data sources and proportions. Table 2 reports the final mixture:

The social media conversations component (50% of the data) is the largest single source and includes multilingual conversations from public domain social media, following the data used by Adiwardana et al. (2020) for the Meena chatbot. This heavy weighting toward conversational data is distinctive—most prior large LMs weighted webpages more heavily—and likely contributes to PaLM's strong performance on tasks requiring natural language understanding and generation.

Webpage filtering. The filtered webpages (27%) were processed through a quality classifier that was trained to assign high scores to pages similar to known high-quality webpage collections. Pages were then sampled proportional to that score, meaning higher-quality pages were included more frequently but lower-quality pages were not eliminated entirely. This is a soft filtering approach rather than a hard threshold, which preserves some diversity while upweighting professionally written or well-structured content. Du et al. (2021) showed that this filtering strategy significantly improves few-shot learning performance compared to unfiltered web data.

Code data. Source code from GitHub repositories was included (5% of the data), filtered by file extension to include 24 common programming languages (Java, HTML, JavaScript, Python, C, PHP, C#, C++, and others as listed in Table 28). Repositories with copyleft licenses were excluded. Files were deduplicated based on Levenshtein distance because duplicate files are common in code repositories (Lopes et al., 2017; Allamanis, 2019). The resulting code corpus was 196GB. The most common languages by byte proportion were Java (18.8%), HTML (17.5%), JavaScript (11.6%), Python (7.3%), C (7.3%), PHP (7.2%), C# (6.4%), and C++ (6.3%).

Language distribution. Table 29 shows the breakdown across 124 languages. English dominates at 577.96 billion tokens (77.98%), followed by German (25.95B, 3.50%), French (24.09B, 3.25%), Spanish (15.65B, 2.11%), and Polish (10.76B, 1.45%). Low-resource languages have extremely small representation—Kazakh has only 134 million tokens (0.018%), Swahili 40 million (0.005%), and Xhosa 8 million (0.001%). This distribution has direct implications for multilingual evaluation: the model sees orders of magnitude less data in low-resource languages, so strong performance on those languages would require cross-lingual transfer.

Training epoch and data repetition. All three models (8B, 62B, 540B) were trained on exactly one epoch of the data, shuffled identically for all models. The mixing proportions were chosen to avoid repeating data in any subcomponent. This is important because it means the 540B model sees exactly the same examples as the 8B model—any performance difference comes from the model's capacity to learn from those examples, not from seeing them more times. However, the paper notes (Section 13) that "data begins to repeat in some of our subcorpora after 780B tokens," which is why training was stopped at that point.

Data contamination analysis (Section 8). The paper performs a detailed analysis of overlap between training data and evaluation benchmarks, splitting each benchmark into "contaminated" (where at least 70% of 8-grams in the question, prompt, or target appear at least once in training) and "clean" subsets. Crucially, performance deltas between clean and full subsets are approximately balanced (similar numbers of positive and negative deltas) and consistent across model scales, suggesting that data contamination does not meaningfully inflate reported results. The paper argues that this is because many benchmarks use contexts from the open web but with novel questions, so training on the context doesn't give an unfair advantage.

---

Tokenizer and Sequence Packing

SentencePiece vocabulary. The model uses a SentencePiece tokenizer (Kudo & Richardson, 2018a) with a vocabulary of 256k tokens, trained directly on the training data. The vocabulary is described as "completely lossless and reversible," meaning whitespace is fully preserved (critical for code, where indentation carries semantic meaning) and out-of-vocabulary Unicode characters are split into UTF-8 bytes with a dedicated vocabulary token for each byte. Numbers are always split into individual digit tokens—for example, "123.5" becomes the sequence 1 2 3 . 5 rather than a single token "123.5" or the pieces "123" and ".5". This digit-splitting strategy means the model must learn to compose numbers from digits, which could aid mathematical reasoning by forcing it to attend to individual digits and their positional relationships.

Sequence length and packing. A fixed sequence length of 2048 tokens is used for all models. Input examples (documents) are concatenated together and then split into sequences of exactly 2048 tokens, with a special [eod] (end-of-document) token separating different documents. This means there are no padding tokens (which would waste compute), but examples may be split across sequence boundaries. The model must learn to attend across the [eod] token to maintain coherence when a document spans multiple sequences, and conversely to recognize that tokens before the [eod] marker belong to a different document. This is a standard practice in large language model training but has implications for the model's ability to handle very long documents, since it never sees contexts longer than 2048 tokens during pretraining.

---

Model Architecture

PaLM uses a standard decoder-only Transformer architecture with several modifications, each chosen for specific computational or quality reasons. The architecture is detailed in Section 2, with hyperparameters in Table 1.

Core architectural hyperparameters. The three model scales are:

The feed-forward dimension $d_{ff}$ is always $4 \times d_{model}$, and the attention head size is always 256. The number of FLOPs per token is approximately equal to the number of parameters because these are standard dense Transformers (each parameter participates in roughly one multiply-add per token in the forward pass, and two in the backward pass).

SwiGLU activation. The standard Transformer MLP uses two weight matrices: one to project from $d_{model}$ to $d_{ff}$ (with a nonlinearity like ReLU or GeLU), and another to project back from $d_{ff}$ to $d_{model}$. SwiGLU replaces this with three matrices, computing:

$\text{SwiGLU}(x) = \text{Swish}(xW_1) \odot (xV) \cdot W_2$

where $x \in \mathbb{R}^{d_{model}}$ is the input, $W_1 \in \mathbb{R}^{d_{model} \times d_{ff}}$ and $V \in \mathbb{R}^{d_{model} \times d_{ff}}$ are the two projection matrices into the expanded dimension, $W_2 \in \mathbb{R}^{d_{ff} \times d_{model}}$ projects back, $\text{Swish}(z) = z \cdot \sigma(z)$ is the Swish activation (where $\sigma$ is the sigmoid function), and $\odot$ is element-wise multiplication.

What it computes: the input $x$ is projected through two parallel linear transformations—one gated by a Swish nonlinearity, one left linear—their element-wise product is computed, and the result is projected back to the model dimension. Intuitively, $\text{Swish}(xW_1)$ acts as a learned gate that controls which dimensions of $xV$ pass through, similar to how an LSTM gate controls information flow.

Why this form: Shazeer (2020) showed that SwiGLU significantly increases quality compared to ReLU, GeLU, or Swish activations in compute-equivalent experiments. The catch is that it requires three matrix multiplications rather than two, increasing the computational cost. However, when compared at equal compute (i.e., giving the ReLU variant proportionally larger dimensions to compensate for the extra matrix multiply), SwiGLU still outperforms ReLU variants. The paper therefore accepts the extra cost because the quality improvement outweighs it.

Parallel layers. The standard Transformer block computes attention and MLP sequentially:

$y = x + \text{MLP}(\text{LayerNorm}(x + \text{Attention}(\text{LayerNorm}(x))))$

The parallel formulation computes them simultaneously on the same normalized input and sums their outputs:

$y = x + \text{MLP}(\text{LayerNorm}(x)) + \text{Attention}(\text{LayerNorm}(x))$

What it computes: instead of chaining attention then MLP (with layer norms between and after each), the parallel version applies both attention and MLP to the same layer-normalized input $\text{LayerNorm}(x)$, then adds both outputs plus the residual to form $y$.

Why this form: the parallel formulation enables fusing the input matrix multiplications for attention and MLP, because they both read from the same normalized input. This results in roughly 15% faster training speed at large scales. Ablation experiments showed a small quality degradation at 8B parameters but no quality degradation at 62B, so the authors extrapolated that the effect should be quality-neutral at 540B. Accepting a possible small quality hit for a 15% speedup is a pragmatic engineering tradeoff at this scale.

Multi-query attention. Standard multi-head attention projects the input into $k$ separate queries, keys, and values (each of shape $[k, h]$, where $h$ is the head dimension), computes attention independently for each head, and concatenates the results. Multi-query attention shares the key and value projections across all heads—meaning keys and values are projected to shape $[1, h]$ and broadcast to all heads—while queries remain per-head at $[k, h]$.

What it computes: the attention operation still produces $k$ different attention patterns (since queries differ per head), but all heads read from the same keys and values. This is equivalent to having one set of key-value memories that all attention heads query, rather than each head having its own separate memory.

Why this form: multi-query attention has a neutral effect on model quality and training speed (Shazeer, 2019), but results in significant cost savings at autoregressive decoding time. During decoding, the key and value tensors for previous tokens must be cached and re-read for each new token. In standard multi-head attention, these cached tensors have shape $[k, h]$ per token per layer; in multi-query attention, they have shape $[1, h]$ per token per layer—a $k$-fold reduction in memory bandwidth and storage. Since autoregressive decoding is memory-bandwidth-bound on accelerators, this directly improves inference latency and throughput.

RoPE embeddings. Rotary Position Embeddings (RoPE; Su et al., 2021) encode position information by applying a rotation to the query and key vectors before computing attention scores. Specifically, for a position $p$ and dimension index $i$:

$R_{p,i} = \begin{pmatrix} \cos(p \cdot \omega_i) & -\sin(p \cdot \omega_i) \\ \sin(p \cdot \omega_i) & \cos(p \cdot \omega_i) \end{pmatrix}$

where $\omega_i = \theta^{-2i/d}$ for some base $\theta$ (typically 10000). The rotation is applied to pairs of adjacent dimensions in the query and key vectors. The attention score between position $p$ and $q$ then naturally depends on their relative distance $(q-p)$ because $R_p^T R_q = R_{q-p}$.

What it computes: rather than adding a learned or fixed positional embedding to the input tokens (absolute position encoding) or adding a learned bias to attention scores (relative position encoding), RoPE modifies the attention computation itself so that the dot product $q_p^T k_q$ inherently encodes the relative distance between positions $p$ and $q$. The rotation angles are a deterministic function of position.

Why this form: RoPE has been shown to have better performance on long sequence lengths compared to absolute or relative position embeddings (Su et al., 2021). The key property is that the attention score between two tokens depends only on their relative distance, not their absolute positions, which provides translation invariance that is particularly useful when the model encounters sequences longer than those seen during training. At test time, the model can extrapolate to longer sequences because the relative position signal is continuous and well-defined beyond the training context length.

Shared input-output embeddings. The embedding matrix that converts input tokens to vectors (the "input embedding") and the matrix that converts the final hidden state to logits over the vocabulary (the "output embedding") are the same matrix. This is a common but not universal practice (GPT-3 does not share embeddings).

What it computes: when the model produces a next-token prediction, the logit for a token is computed as the dot product between the final hidden state and that token's embedding vector—the same embedding vector that would be used to represent that token if it appeared as input. This creates a direct relationship between how the model represents a token when reading it and how it scores that token when predicting it.

Why this form: sharing embeddings reduces the total parameter count (by $d_{model} \times |V|$, or about 4.7 billion parameters for the 256k vocabulary with $d_{model}=18432$ in the 540B model). More importantly, it encourages the model to learn consistent representations—a token's embedding should be useful both for providing context (as input) and for being predicted (as output). The paper also applies a scaling factor of $1/\sqrt{n}$ to the pre-softmax output logits, where $n$ is the embedding dimension, to compensate for the fact that the shared embedding matrix is initialized with $\mathcal{N}(0, 1)$ rather than the fan-in scaling used for other weight matrices.

No biases. No bias terms are used anywhere in the model—not in the dense kernel layers, the layer norms, or the attention projections. This is a departure from standard Transformer implementations, which typically include biases in linear layers and layer norms.

Why this form: the authors found that removing biases resulted in increased training stability for large models. Biases shift the mean of activations, which can cause issues with gradient flow and normalization at scale. Removing them simplifies the optimization landscape without meaningful quality degradation.

---

Training Setup

The training procedure follows a standard autoregressive language modeling setup with specific choices for initialization, optimization, and batching, detailed in Section 5.

Weight initialization. Kernel weights (all weight matrices except embeddings and layer norm scales) are initialized with fan-in variance scaling:

$W \sim \mathcal{N}\left(0, \frac{1}{\sqrt{n_{in}}}\right)$

where $n_{in}$ is the input dimension of the kernel. This ensures that the variance of activations is approximately preserved through the layer at initialization, preventing exploding or vanishing signals at the start of training.

Input embeddings are initialized to $E \sim \mathcal{N}(0, 1)$ because layer normalization is not applied to the embeddings directly—the first layer norm is applied after the embeddings are added to the position encoding, so a unit-variance initialization is appropriate.

Because the input and output embeddings are shared, the pre-softmax output logits are scaled by $1/\sqrt{n}$ where $n$ is the embedding size. This scaling compensates for the different initialization strategy: the output embedding matrix is initialized as $\mathcal{N}(0, 1)$ (since it's shared with the input embeddings), but without scaling, the logits would have variance proportional to $n$, which would make the softmax very peaked and slow down training. The $1/\sqrt{n}$ scaling brings the logit variance to approximately 1.

Optimizer. The model is trained with the Adafactor optimizer (Shazeer & Stern, 2018) without factorization, which makes it effectively equivalent to Adam with "parameter scaling." Parameter scaling multiplies the learning rate for each parameter matrix by the root-mean-square of that matrix's values. This has a similar effect to the manual scaling of Adam learning rates used in GPT-3 (Brown et al., 2020), where larger matrices get smaller learning rates. However, parameter scaling has the advantage that matrices operating at different scales (the embeddings and layer norm scales) are handled automatically rather than requiring manual tuning.

Optimization hyperparameters. The Adafactor learning rate is $10^{-2}$ for the first 10,000 steps, then decays at a rate of $1/\sqrt{k}$, where $k$ is the step number. This is an inverse-square-root schedule, which is standard for large-scale language model training because it reduces the learning rate slowly enough that the model can continue learning throughout training.

Momentum is used with $\beta_1 = 0.9$. The second-order moment interpolation value is computed as:

$\beta_2 = 1.0 - k^{-0.8}$

where $k$ is the step number.

What this computes: this is an adaptive second-moment decay rate that starts at 0 at the beginning of training (when $k=1$, $\beta_2 = 0$) and asymptotically approaches 1 as training proceeds. Early in training, this means the second-moment estimates adapt quickly (relying on recent gradients), while later in training they become smoother (averaging over longer history).

Why this form: the authors found this schedule to be more stable than the standard $\beta_2 = 0.99$ when training large language models. The issue is that rare embedding tokens—tokens that appear very infrequently—have poorly estimated second moments over short windows because they have few gradient samples. The adaptive $\beta_2$ gives more weight to recent gradients for these rare tokens early in training, preventing their second-moment estimates from being dominated by a few noisy updates.

Global norm gradient clipping (Pascanu et al., 2012) is applied with a threshold of 1.0 for all models. If the L2 norm of the concatenated gradient vector exceeds 1.0, all gradients are scaled down proportionally so the norm is exactly 1.0. This prevents individual gradient steps from being destructively large.

Dynamic weight decay is applied with a coefficient of $\text{lr}^{2.0}$, where $\text{lr}$ is the current learning rate. As the learning rate decays, weight decay also decays, which means regularization is stronger early in training (when the learning rate is high and the model is forming its initial representations) and weaker later (when the model is fine-tuning).

Loss function. The primary loss is the standard language modeling loss—the average negative log probability of the correct next token, averaged over all tokens in the sequence:

$\mathcal{L} = -\frac{1}{T} \sum_{t=1}^{T} \log p(y_t | y_{<t})$

where $T$ is the sequence length, $y_t$ is the $t$-th target token, and $p(y_t | y_{<t})$ is the model's predicted probability for that token given the preceding tokens.

Additionally, an auxiliary loss called "z loss" is added:

$\mathcal{L}_z = 10^{-4} \cdot \log^2 Z$

where $Z = \sum_{v} \exp(l_v)$ is the softmax normalizer (the sum of exponentiated logits over the vocabulary).

What it computes: the z loss penalizes the squared log of the softmax normalizer. When $\log Z$ is large (meaning the logits have large magnitude and the softmax distribution is very peaked), this term is large; when $\log Z$ is close to 0, this term is small.

Why this form: the z loss encourages the logits to stay in a numerically stable range, which the authors found increases training stability. Without it, the softmax normalizer can drift to large positive values (making the distribution too peaked and gradients small) or large negative values (making the distribution too uniform), both of which slow learning. The $10^{-4}$ coefficient makes this a weak regularizer that acts primarily to prevent extreme drift rather than to shape the distribution significantly.

Batch size schedule. For the 540B model, the batch size is increased during training according to a schedule: batch size 512 (1 million tokens) until step 50k, then doubled to 1024 (2 million tokens) until step 115k, then doubled again to 2048 (4 million tokens) until training completes at step 255k. The 8B and 62B models follow similar schedules but with different starting points.

The total training tokens are: 8B model trains for 780 billion tokens (final batch size 512 gives 780B/512 ≈ 1.52M steps but the paper reports a different schedule—the 8B reaches 780B tokens at a batch size of 512 after following its own schedule), 62B model trains for 795 billion tokens (slightly more than 780B due to an "oversight in checkpoint selection" noted in Section 6.8), and 540B model trains for 780 billion tokens.

Why this form: the batch size schedule is motivated by two factors. First, smaller batch sizes are more sample-efficient early in training (better loss as a function of tokens seen) because noisier gradient estimates help exploration, while larger batch sizes are beneficial later due to better gradient estimates that allow more precise optimization (Smith et al., 2018; McCandlish et al., 2018). Second, larger batch sizes result in larger matrix multiplication dimensions, which increases TPU efficiency—the TPU's matrix multiply units achieve higher utilization with larger matrices. The schedule balances these competing concerns by using smaller batches when sample efficiency matters most and larger batches when TPU efficiency matters most.

Bitwise determinism. The training framework is fully bitwise reproducible from any checkpoint. If training is restarted from checkpoint 15,000 and run to step 17,000, the model state at step 17,000 will be identical to the original run. This is achieved through the JAX+XLA+T5X framework (which guarantees deterministic computation for a given input) and a deterministic dataset pipeline where shuffled data is written in a random-access format so the contents of a given training batch depend only on the step number (Roberts et al., 2022). This property is critical for debugging training instabilities and for reproducible research at scale.

Training instability mitigation (Section 5.1). For the 540B model, the authors observed roughly 20 loss spikes during training—sudden, large increases in training loss that occurred at irregular intervals, sometimes late into training. These spikes were not observed with the smaller models. The spikes did not appear to be caused by "bad data" in isolation, because running the same data batches from an earlier checkpoint did not reproduce the spike. This implies that spikes depend on the specific combination of model parameters and data batches—a particular parameter state makes the model vulnerable to specific training examples.

The mitigation strategy was pragmatic: when a spike was detected, training was restarted from a checkpoint approximately 100 steps before the spike began, and approximately 200–500 data batches (covering the batches seen before and during the spike) were skipped. With this strategy, the spike did not recur at the same point. This is not a principled solution—it requires manual monitoring and intervention—but it was effective enough to complete training. The paper notes that future work should investigate more principled mitigation strategies for loss spikes in very large language models, but the cost of training at this scale makes systematic experimentation on this problem prohibitive.

---

Training Infrastructure and Parallelism Strategy

The training of PaLM 540B on 6144 TPU v4 chips across two pods using the Pathways system is a significant engineering contribution, detailed in Section 4.

Parallelism strategy. The training parallelism operates at two levels:

Within-pod parallelism: Each TPU v4 Pod contains a full copy of the model parameters. Within each pod, 12-way model parallelism and 256-way fully sharded data parallelism are used. Model parallelism splits each weight tensor across 12 TPU chips along the feed-forward or attention-head dimension—during the forward pass, the necessary portions are gathered, the computation is performed, and the results are scattered. Data parallelism splits the batch across 256 groups of 12 chips, with each group processing a different subset of the batch and gradients being summed across groups. This is the "2D finalized" approach from Xu et al. (2021).

During the forward pass, weights are all-gathered over the data parallel axis (so each data-parallel replica has the full weights for its portion of the model parallel shard). One fully sharded activation tensor is saved from each layer. During the backward pass, the remaining activations are rematerialized (recomputed from the saved tensors) because this results in higher training throughput at larger batch sizes compared to storing all activations.

Pod-level parallelism: Training is scaled beyond a single pod using two-way data parallelism at the pod level, orchestrated by the Pathways system (Barham et al., 2022). A single Python client dispatches half of the training batch to each pod. Each pod independently executes forward and backward computation to compute gradients using its within-pod parallelism. The pods then transfer gradients with each other (each pod receives the remote pod's gradients for its half of the batch), and each pod accumulates local and remote gradients and applies parameter updates in parallel. Because both pods receive the same accumulated gradients, they produce bitwise-identical parameters for the next step.

Pathways system architecture. Figure 2 illustrates the execution model. The Pathways program consists of three components: Component A performs within-pod forward and backward computation (including within-pod gradient reduction), a transfer subgraph handles cross-pod gradient communication, and Component B performs the optimizer update (summing local and remote gradients, then applying the update). The program executes Component A on each pod in parallel, then transfers gradients between pods, then executes Component B on each pod in parallel.

Key design features include: asynchronous gang-scheduling at per-pod schedulers that masks the latency of dispatching work from the single Python client to remote servers, and a sharded-dataflow execution model that amortizes the cost of managing data transfers.

Cross-pod gradient transfer challenge. The cross-pod gradient transfer poses a specific networking challenge. Each pod has 768 hosts (3072 TPU v4 chips at 4 chips per host). Each host needs to exchange gradients only with its corresponding host on the other pod (because each core only needs remote gradients for its model-parallel shard). The transfer involves approximately 1.3 GB of gradients per host pair per training step, creating an aggregate burst of 81 Tbps across all 1536 hosts. This burst occurs simultaneously because all hosts finish gradient computation at roughly the same time.

To manage this, the Pathways networking stack breaks gradient data into smaller chunks and routes them via multiple smaller flows over diverse datacenter network links, mitigating congestion. With these optimizations, the cross-pod training achieves about 1.95x the throughput of a single pod (compared to the theoretical maximum of 2.0x for doubling the batch size). The gap (2.5% below perfect weak scaling) is attributed to lack of overlap between the backward pass and cross-pod gradient reduction, which the authors expect to address in future work.

Model FLOPs utilization (MFU). The paper introduces a new efficiency metric called model FLOPs utilization, defined as:

$\text{MFU} = \frac{\text{observed throughput (tokens/sec)}}{\text{theoretical maximum throughput at peak FLOPs}}$

The theoretical maximum throughput for a dense Transformer is:

$R = \frac{P}{6N + 12LHQT}$

where:

• $P$ is the total theoretical peak matmul throughput of the accelerator system (in FLOPs/sec)
• $N$ is the number of model parameters (excluding attention, or including it in a variant)
• $L$ is the number of layers
• $H$ is the number of attention heads
• $Q$ is the head dimension
• $T$ is the sequence length

What this computes: the denominator is the minimum number of matmul FLOPs required to process one token through the forward and backward passes (6N for the non-attention components because each parameter participates in one multiply-add in the forward pass and two multiply-adds in the backward pass, with 2 FLOPs per multiply-add; 12LHQT for self-attention because the attention computation includes additional matrix multiplications whose cost scales with sequence length). The numerator is the observed throughput in tokens per second. The ratio tells us what fraction of the theoretical peak is actually achieved.

Why this form: the paper argues that hardware FLOPs utilization (HFU), the previously common metric, is problematic because (1) the number of hardware FLOPs executed depends on implementation details like rematerialization, which trades extra computation for memory savings, and (2) measuring observed hardware FLOPs depends on methodology (analytical accounting vs. hardware performance counters). MFU is implementation-independent: it uses the observed throughput (which is directly measurable and is what practitioners care about) and the theoretical minimum FLOPs (which depends only on the model architecture, not on implementation choices like rematerialization). This allows fair comparisons across different systems and implementations.

PaLM 540B achieves an MFU of 46.2% (including self-attention FLOPs) at a training throughput of 238.3K tokens/sec. For comparison, the paper calculates GPT-3's MFU as 21.3%, Gopher's as 32.5%, and Megatron-Turing NLG 530B's as 30.2%. The corresponding hardware FLOPs utilization of PaLM is 57.8% (which includes rematerialization FLOPs in the numerator, making it higher than MFU).

The high MFU is attributed to: the parallelism strategy (which scales efficiently without pipeline parallelism bubbles), XLA TPU compiler optimizations, and the use of parallel layers (which enables fusing attention and MLP input matrix multiplications).

---

Few-Shot Evaluation Methodology

PaLM's capabilities are evaluated primarily through few-shot prompting, where the model receives task descriptions and optionally a small number of input-output exemplars, then generates predictions without any gradient updates. The specific prompting format varies by task type.

Standard few-shot evaluation (classification and short-answer tasks). For tasks like question answering, natural language inference, and multiple-choice QA, the model is given a natural language description of the task, followed by $k$ exemplars (input-output pairs), followed by the test input. The model then generates the answer, which is compared to the ground truth using exact match (for short-answer tasks) or log-likelihood scoring (for multiple-choice tasks, where the model's probability assigned to each option is compared and the highest-probability option is selected). The number of exemplars $k$ is tuned per task and reported in the results tables (e.g., Table 4 shows shots ranging from 1 to 64 depending on the task).

For generative tasks (like TriviaQA or Natural Questions), the model must generate the answer text, and exact match is used after normalizing whitespace, punctuation, and case. For multiple-choice tasks, the paper uses a ranking approach where each option is scored by the model's log-probability of generating that option given the prompt, and the option with the highest probability is selected.

Chain-of-thought prompting (reasoning tasks). For arithmetic and commonsense reasoning tasks (Section 6.3), the model is prompted to generate intermediate reasoning steps before producing the final answer. The few-shot exemplars include manually written chains of reasoning (e.g., for a math problem: "Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11."). The model then generates its own chain of thought for the test example, and only the final answer (after "The answer is") is extracted and evaluated.

For arithmetic reasoning, the model can optionally use an external calculator: instead of performing arithmetic itself (e.g., computing "5 + 6 = 11"), it can output equations that are executed by a calculator to produce the numeric result. However, the paper notes that the calculator "did not improve performance by more than 5% on any dataset" (Section 6.3.1), suggesting that the model's internal arithmetic capabilities are already strong at this scale.

Code generation evaluation (Section 6.4). For code synthesis tasks, the model is prompted with a natural language description and optionally input-output examples, and generates source code. Evaluation uses the pass@k metric: $k$ samples are drawn from the model, and a problem is counted as solved if any sample passes all test cases. For pass@1, greedy decoding is used (always selecting the highest-probability token at each step). For $k > 1$, nucleus sampling is used with $p = 0.95$ and temperature 0.8 (which introduces diversity so the $k$ samples explore different solutions). The paper reports the raw percentage of problems solved rather than using the unbiased estimator from Chen et al. (2021).

Translation evaluation (Section 6.5). For machine translation, the model is prompted with the source language text and (optionally) exemplar translations. In the 0-shot setting, the prompt includes the source and target language names (e.g., "English: [source text]\nFrench:"). In 1-shot and few-shot settings, the prompt includes input-output translation pairs, and the model must infer the language pair from the exemplars. Evaluation uses BLEU score against reference translations.

Multilingual generation evaluation (Section 6.6). For summarization tasks, the model is prompted with a task-specific prefix (e.g., "Summarize the following article:") and exemplars, then generates a summary. Inputs are truncated to 2048 tokens if necessary, and outputs are truncated at double linebreaks. Evaluation uses ROUGE-2, ROUGE-L, and BLEURT-20 metrics.

BIG-bench evaluation (Section 6.2). BIG-bench includes 150+ diverse tasks, many with non-standard evaluation. The paper uses the "preferred metric" for each task (as specified by the task authors), normalizes scores by setting the maximum possible score to 100 and the random chance score to 0 (so negative scores indicate worse-than-random performance), and averages normalized scores across tasks. This enables aggregation across tasks with different scoring schemes.

Generative vs. multiple-choice scoring for Winogender (Section 10.1.1). The paper highlights a methodological issue with standard multiple-choice scoring for coreference resolution. In multiple-choice scoring, each candidate answer is scored by the model's probability of generating that answer, and the example is scored correct if the correct answer has higher probability than alternatives—even if the absolute probability is very low. For 0-shot prompting, this can produce misleadingly high accuracy because the model doesn't understand the task format and simply predicts random tokens.

The paper uses both multiple-choice scoring (for comparability with prior work) and generative scoring (case-insensitive exact string match on the model's generated text, truncated at punctuation). In the 0-shot case, the model generates nonsense for the generative scoring (e.g., "a. the patient b. the nurse c. the shift d. the hour"), while multiple-choice scoring still produces a "correct" answer if the relative probabilities happen to favor the right option. This highlights the importance of the generative scoring method for accurate assessment of model capabilities.

Cross-checkpoint variance analysis (Section 6.8). To assess the reliability of few-shot results, the paper evaluates 15 evenly spaced checkpoints during training (each separated by ~2B tokens) on three tasks. For StoryCloze and Natural Questions, variance is relatively low and consistent across scales. For WebQuestions on PaLM 540B, the paper observes significantly larger variance, with performance peaking at 770B training tokens and declining in later checkpoints. All results reported in other sections use the final checkpoint (780B for 8B and 540B, 795B for 62B), even when earlier checkpoints performed better on specific tasks. This is a deliberate choice for consistency, but it means the reported results may slightly understate peak performance.

---

Finetuning Methodology (Sections 6.1.2, 6.6, 6.7)

While the paper primarily focuses on few-shot evaluation, it also includes finetuning experiments on SuperGLUE, multilingual generation (GEM tasks), and multilingual QA (TyDiQA). The finetuning setup differs from pretraining in several ways:

Optimizer and learning rate. Finetuning uses the Adafactor optimizer with a constant learning rate of $5 \times 10^{-5}$ (20x smaller than the initial pretraining learning rate), and the optimizer's accumulators are reset (so the second-moment estimates are recomputed from scratch on the finetuning data).

Batch size and convergence. A batch size of 32 is used. Models typically converge in less than 15,000 steps of finetuning.

Dropout. While pretraining uses no dropout, finetuning uses dropout of 0.1.

Sequence handling for generation tasks. For finetuning on generation tasks (summarization, data-to-text), inputs and targets are concatenated, but the loss is computed only on the target portion. Sequences are truncated to 2048 tokens with 512 tokens reserved for the target. For generation tasks, the best checkpoint is selected by the best performing geometric mean of ROUGE-1, ROUGE-2, and ROUGE-L on the validation set.

Inference for finetuned models. During inference for finetuned generation models, top-k sampling with k=10 is used. For comparison, T5 XXL baselines are finetuned with the same parameters and decoded using beam search with a beam size of 4.

Architectural implications. The paper notes (Section 6.1.2) that encoder-decoder models trained with span corruption (like T5) generally outperform autoregressive decoder-only models on classification task finetuning at equal training cost. PaLM's finetuning results show that scale can partially bridge this gap—PaLM 540B achieves near-SOTA on SuperGLUE finetuning despite the architectural disadvantage—but the gap is not fully closed, suggesting inherent benefits of the encoder-decoder architecture for certain discriminative tasks.

---

Summary of Key Design Choices and Their Justifications

• Dense rather than sparse architecture: isolates the effect of scale from architectural efficiency, providing a clean reference point for the field and avoiding the complexity of expert routing.

• SwiGLU over ReLU/GeLU: quality improvement demonstrated in compute-equivalent experiments (Shazeer, 2020), accepting the extra matrix multiplication cost.

• Parallel layers over serial: 15% training speedup at scale with neutral quality impact extrapolated from 62B results.

• Multi-query attention: neutral quality impact at training time, significant inference cost savings by reducing key/value cache size by a factor of $k$ (number of heads).

• RoPE over absolute/relative position embeddings: better performance on long sequences, ability to extrapolate to longer contexts at test time.

• Shared input-output embeddings: parameter savings and consistency between token representations as input and output.

• No biases: improved training stability at large scale.

• 256k SentencePiece vocabulary trained on the data: supports multilingual text without excessive tokenization, lossless whitespace handling for code, digit splitting for mathematical reasoning.

• Single epoch of 780B tokens, carefully mixed to avoid data repetition: avoids confounding scale effects with data repetition effects; all model sizes see identical examples.

• Inverse-square-root learning rate decay with adaptive $\beta_2$: standard for large LM training, with the adaptive second-moment providing stability for rare embedding tokens.

• Z loss auxiliary objective: weak regularizer preventing softmax normalizer drift, improving training stability.

• Increasing batch size schedule: balances sample efficiency (small batches early) with TPU efficiency (large batches late).

• Two-level parallelism (within-pod model+data, pod-level data) without pipeline parallelism: avoids pipeline bubbles and the associated idle time, achieving 46.2% MFU on 6144 chips.

• Rematerialization of activations during backward pass: enables larger batch sizes (and thus higher throughput) by reducing memory usage, at the cost of recomputing some forward pass operations.

• Two-fold cross-validation for compute-optimal strategy selection: prevents overfitting strategy choices to the test set when reporting difficulty-conditioned results.

4. Key Insights and Innovations

Innovation 1: Discontinuous Improvements Are a Quantifiable, Prevalent Phenomenon—Not an Anecdote

Prior to PaLM, the dominant mental model for scaling was log-linear: each doubling of compute (or parameters, or data) produced a predictable, roughly constant improvement in performance. This "power law" rule of thumb (Kaplan et al., 2020) was well-established for validation loss and had been observed on many downstream tasks. Hints of discontinuities existed—GPT-3 noted that few-shot learning itself seemed to emerge only above a certain scale—but these observations were qualitative, anecdotal, and limited to one or two tasks. The field lacked both a definition of discontinuity and a systematic measurement of how common it is.

The paper's conceptual move is to treat discontinuous improvement as a first-class, measurable phenomenon and to quantify its prevalence across a large, diverse benchmark suite. The operational definition in Section 6.2 is straightforward but powerful: compute what log-linear extrapolation from 8B→62B would predict for 540B, then measure the gap between that prediction and actual 540B performance. On the logical_sequence task, for instance, PaLM 8B scores 13%, PaLM 62B scores 25%, so the log-linear projection for 540B is ~37% (25 + (25-13)). The actual performance is 87%—a +50% discontinuity.

The finding that ~25% of BIG-bench tasks show discontinuities greater than 10 percentage points, and ~15% show discontinuities greater than 20 percentage points (Section 6.2) transforms this from an interesting anomaly into a statistically prevalent feature of scaling. This matters for three reasons:

• It challenges the power-law universality assumption. If a quarter of tasks exhibit qualitatively different scaling behavior—sudden jumps rather than smooth improvement—then predicting future model capabilities from extrapolation of smaller-scale trends is systematically unreliable. Capabilities do not merely improve; they emerge.

• It implies threshold effects in model capacity. The fact that PaLM 62B performs at ~25% on english_proverbs (essentially random guessing for a multiple-choice task) while PaLM 540B scores 87% suggests that some reasoning capabilities—here, the ability to understand complex metaphors and map them to abstract proverbs—require a minimum parameter threshold to manifest at all. Below that threshold, the model simply cannot perform the task; above it, performance leaps.

• It provides counter-evidence to the Chinchilla fungibility thesis. If some capabilities require a minimum parameter count regardless of how much data a smaller model sees, then scale is not fully fungible—larger models may be necessary for certain capabilities even if they are not "compute-optimal" by training loss metrics. The paper does not frame this as a direct refutation of Chinchilla, but the data provides empirical grounding for the argument that parameter count and data quantity are not perfectly substitutable.

This innovation is fundamental, not incremental: it introduces a new conceptual category into scaling analysis and provides the first systematic measurement of its prevalence. Prior work had observed discontinuities in passing; PaLM makes them a central object of study.

Evidence anchor: Section 6.2, specifically Figure 5(b) showing discontinuous curves for english_proverbs, logical_sequence, and mathematical_induction, and the quantitative analysis reporting 25% of tasks with >10% discontinuity.

---

Innovation 2: Chain-of-Thought + Scale Alone Can Match or Exceed Finetuned SOTA on Multi-Step Reasoning—Eliminating the Necessity of Specialized Architectures

Before PaLM, the reigning assumption in multi-step reasoning research was that strong performance required a combination of domain-specific architectures, task-specific finetuning, and task-specific verifiers. On GSM8K, the prior SOTA (Cobbe et al., 2021) used a finetuned GPT-3 model augmented with an external calculator and a separately trained verifier model that scored candidate solutions. This was not considered a weakness—it was the accepted price of solving multi-step reasoning problems. The underlying belief was that language model pretraining alone, even at scale, could not produce the structured reasoning chains needed for reliable arithmetic.

PaLM's conceptual contribution is eliminating that necessity. Section 6.3 demonstrates that PaLM 540B with 8-shot chain-of-thought prompting and an optional calculator achieves 58% on GSM8K, outperforming the finetuned SOTA of 55% without any parameter updates. This is not an incremental improvement—it is a qualitative shift in what scale makes possible: a general-purpose, frozen language model, prompted with a handful of examples, outperforms a system purpose-built for the task.

The innovation here is not chain-of-thought prompting itself (which was introduced by Wei et al., 2022b) but rather the demonstration that scale amplifies chain-of-thought to the point where specialized machinery becomes unnecessary. The paper shows this through error analysis (Figure 9): PaLM 62B makes errors in semantic understanding, one-step-missing reasoning, and other categories; scaling to 540B fixes a substantial fraction of all error types. Of the 45 errors PaLM 62B made on 150 random GSM8K examples, the 540B model fixed 6 of 20 semantic errors, 12 of 18 one-step-missing errors, and 4 of 7 "other" errors. This is not simply memorizing more patterns—it is genuinely improved reasoning, visible in the qualitative examples (Figures 28-29) where the 62B model multiplies when it should divide or misses a subtraction step, and the 540B model corrects the logical chain.

The significance extends beyond benchmark numbers: it reframes scaling as a substitute for task-specific engineering. If scale + prompting can match or exceed finetuning with specialized components, then the return on investment for building task-specific architectures diminishes as models grow. This has direct implications for how research and engineering resources should be allocated in the field.

The innovation is fundamental: it changes the premise of what is required for complex reasoning, from "you need specialized components" to "you may only need a large enough model and the right prompting strategy." The paper explicitly notes that "the tasks are simply represented via few-shot prompting" (Section 6.3.1), emphasizing the contrast with prior work's multi-component systems.

Evidence anchor: Section 6.3 and Table 10, showing PaLM 540B + chain-of-thought + calculator at 58% vs. prior finetuned SOTA at 55%; Figure 10 showing SOTA or near-SOTA across 7 reasoning benchmarks; Figure 9 showing error type breakdown.

---

Innovation 3: The Introduction and Justification of Model FLOPs Utilization (MFU) as an Implementation-Independent Efficiency Metric

At first glance, proposing a new metric might seem like an engineering footnote rather than a conceptual contribution. But the paper's introduction of model FLOPs utilization (MFU) in Section 4.1 is a diagnostic insight with field-level implications: it identifies a fundamental problem with the previously dominant metric (hardware FLOPs utilization, or HFU) and provides a clean alternative that enables fair comparison across systems.

The problem with HFU is that it measures what fraction of peak hardware FLOPs was actually executed, but the number of hardware FLOPs executed depends on implementation choices that are orthogonal to system efficiency—most critically, rematerialization (recomputing activations during the backward pass rather than storing them). Rematerialization increases hardware FLOPs executed (since forward-pass operations are recomputed) while often improving training throughput (because it enables larger batch sizes by reducing memory usage). Under HFU, a system that uses rematerialization could show lower efficiency despite achieving higher throughput—a perverse incentive. Moreover, measuring observed hardware FLOPs depends on methodology (analytical accounting vs. hardware performance counters), making comparisons across papers unreliable.

The conceptual move is to ask a different question: not "what fraction of peak FLOPs did you use?" but "what fraction of the theoretical minimum FLOPs required by your model are you achieving in practice?" MFU is defined as:

$\text{MFU} = \frac{\text{observed throughput}}{\text{theoretical maximum throughput at peak FLOPs}}$

where the denominator depends only on the model architecture—specifically, the minimum FLOPs needed to compute forward and backward passes for one token—not on implementation choices like rematerialization. This makes MFU implementation-independent: two systems training the same model architecture can be directly compared regardless of their rematerialization strategy, compiler optimizations, or other operational details. The numerator (observed tokens per second) is what practitioners care about; the denominator is a fixed property of the model.

This innovation is fundamentally a framing contribution: it reframes the efficiency question from "how busy are your chips?" (HFU) to "how close are you to the theoretical speed limit?" (MFU). The paper contextualizes this by calculating MFU for prior models: GPT-3 at 21.3%, Gopher at 32.5%, Megatron-Turing NLG at 30.2%, compared to PaLM 540B at 46.2%. These numbers are not just about bragging rights—they quantify the efficiency gap between training systems and the theoretical optimum, providing a target for future engineering work.

The innovation is incremental in form but fundamental in consequence: it replaces a flawed metric with a principled one, and in doing so, redefines what "efficient training" means for the field. Future papers can adopt MFU to report their training efficiency without the confounding effects of rematerialization choices.

Evidence anchor: Table 3 comparing MFU across models; Appendix B providing the mathematical derivation; Section 4.1 explaining why HFU is problematic and how MFU addresses those problems.

---

Innovation 4: Controlled Scaling Across Three Orders of Magnitude Provides a Clean Reference Curve for the Field

This innovation is methodological rather than empirical, but its intellectual significance is substantial: by training 8B, 62B, and 540B models identically—same data, same architecture, same tokenizer, same optimizer, same training procedure—the paper creates a controlled scaling curve that isolates the effect of model size from all confounding variables. Prior scaling studies were confounded by differences in training data (GPT-3 models used different data mixtures at different scales), architecture (some models changed attention patterns or activation functions across scales), or procedures (different learning rate schedules, batch sizes, or training durations).

The conceptual contribution is recognizing that the value of a scaling study lies in its purity of comparison, not just in the absolute performance of the largest model. PaLM 540B achieving SOTA on 28 of 29 tasks is impressive, but the scientific value of the paper is equally in the clean comparisons across scales: knowing that the jump from 62B to 540B produces a specific improvement on a specific task, under conditions where the only variable that changed is model size (and the batch size schedule, which is itself a function of scale). This enables the field to ask counterfactual questions: if we trained a 200B model on this data with this recipe, what performance would we expect? If we trained a 1T model, would the scaling curve continue?

The paper explicitly recognizes this as part of its contribution, stating in Section 1:

"typically, scaling from 62B to 540B results in similar performance as scaling from 8B to 62B, which is consistent with the 'power law' rule of thumb often observed in neural network scaling. However, for certain tasks, we observe discontinuous improvements, where scaling from 62B to 540B results in a drastic jump in accuracy compared to scaling from 8B to 62B."

This framing—comparing the shape of the scaling curve across tasks—is only possible because the comparison is controlled. Without identical training across scales, it would be impossible to determine whether a "discontinuity" was actually caused by scale or by a change in data distribution between the medium and large models.

The paper also uses this controlled setup to make a nuanced argument about the role of data quality. PaLM 62B (795B tokens) outperforms GPT-3 175B on average NLU and NLG scores (Table 5: 67.3 vs. 65.4 NLU, 57.7 vs. 52.9 NLG) despite having roughly one-third the parameters and much lower training FLOPs. This cannot be attributed to scale—it must be attributed to data quality or training procedure. By providing the 8B and 62B numbers alongside the 540B results, the paper enables the community to estimate the "data quality offset" separately from the "scale offset," which would be impossible if only the 540B model were reported.

This innovation is fundamental to the paper's scientific contribution even though it is methodological rather than empirical: it establishes PaLM as a reference point and scaling curve that future work can use for calibration, comparison, and extrapolation.

Evidence anchor: Table 1 showing identical architecture across scales (only depth, width, and heads change); Section 3 confirming identical training data and shuffling; Table 5 showing the clean scaling of average NLU and NLG scores; Figure 3 showing log-linear scaling on BIG-bench aggregate.

---

Innovation 5: Explanatory Generation as a Qualitative Showcase of Deep Language Understanding—Beyond Accuracy Metrics

Section 9 presents what the paper calls "exploratory capabilities" in PaLM's ability to generate explicit, multi-step logical inference chains for tasks like explaining jokes and answering complex questions about scenarios. This section is unusual for a systems paper: it does not report benchmark scores, does not claim quantitative improvements, and explicitly notes that the examples were curated by the authors. Yet it serves an important intellectual function.

The conceptual contribution is that certain capabilities—particularly those requiring the integration of world knowledge, abstract reasoning, and linguistic understanding—cannot be adequately captured by accuracy on benchmark tasks, even when those tasks are designed to be challenging. PaLM's generation of a coherent explanation for a pun about neural network seeds ("I guess no good seed goes unpunished")—identifying that "seed" refers both to a random number generator initialization and the proverb "no good deed goes unpunished"—demonstrates a type of multi-layered linguistic reasoning that is qualitatively different from selecting the correct answer in a multiple-choice test.

The paper is careful about its claims here, noting four factors that lend credibility despite the curated nature of the examples: (1) the same 2-shot exemplars were used for all examples, with no per-example prompt engineering; (2) all outputs are from greedy decoding, so they represent the model's single best prediction, not cherry-picked from many samples; (3) the probability of generating a fully coherent, multi-step explanation through "lucky guesses" is vanishingly low—each explanation requires a chain of correct inferences; (4) the prompts were written by the authors, mitigating data contamination concerns.

The intellectual move is to argue that some capabilities are inherently generative and explanatory, and cannot be reduced to a scalar accuracy metric. This is not a new idea—interpretability researchers have long argued for the importance of explanations—but the paper provides concrete, compelling examples that demonstrate the depth of understanding achievable at 540B parameters. The "Logical Inference" examples, where the model chains together geographic knowledge (Mount Rushmore → South Dakota → Minnesota → St. Paul), cultural knowledge (Mona Lisa → Leonardo da Vinci → Teenage Mutant Ninja Turtles → katana → Japan), and physical reasoning (cloud below the window → airplane → ~480 km/h → "yes, traveling more than 300 mph") represent a form of reasoning that cannot be tested by standard benchmarks, because the reasoning chain itself is the output of interest.

This innovation is incremental in form but provocative in implication: it does not introduce a new method or metric, but it demonstrates capabilities that existing evaluation frameworks are not designed to capture, implicitly arguing that the field needs new ways of assessing language understanding. It also serves as a public demonstration that chain-of-thought prompting can elicit reasoning far beyond the arithmetic domains where it was originally validated.

Evidence anchor: Section 9, specifically Figure 19 showing the 2-shot exemplars and the model's greedy-decoded explanations for jokes and logical inference problems; the four credibility factors listed in the section text.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary few-shot evaluation benchmarks are a set of 29 English NLP tasks covering question answering, reading comprehension, commonsense reasoning, and natural language inference, a subset of 58 BIG-bench textual tasks for cross-model comparison, and 7 arithmetic and commonsense reasoning datasets (GSM8K, MAWPS, SVAMP, AQuA, ASDiv, CommonsenseQA, StrategyQA). Code evaluation uses HumanEval, MBPP, TransCoder, DeepFix, and a newly introduced GSM8K-Python task; translation evaluation uses WMT language pairs including English-centric (en↔fr, en↔de, en↔ro), direct (de↔fr), and extremely low-resource (en↔kk) pairs; multilingual generation uses 6 tasks from the GEM benchmark; multilingual QA uses TyDiQA-GoldP. Specific task splits follow those used by GPT-3 (Brown et al., 2020) and GLaM (Du et al., 2021) for the 29 English NLP tasks, with training data contamination analysis performed separately on each benchmark.

• Base model(s). Three model scales are evaluated—PaLM 8B, PaLM 62B, and PaLM 540B—all trained identically on the same 780B-token dataset (with PaLM 62B slightly overshooting to 795B tokens due to a checkpoint selection oversight noted in Section 6.8). The models differ only in depth, width, number of attention heads, and batch size schedule (Table 1). For code finetuning experiments, a PaLM-Coder variant is created by additional finetuning on code-specific data.

• Metrics. For classification and short-answer tasks, accuracy or exact match (EM) is used, with F1 scores reported for certain reading comprehension tasks (DROP, CoQA, QuAC, SQuADv2, MultiRC). For reasoning tasks, accuracy on the final extracted answer is reported. For code tasks, the pass@k metric is used—the percentage of problems solved by at least one of k samples—with greedy decoding for pass@1 and nucleus sampling (p=0.95, temperature 0.8) for k>1. For translation, BLEU score is reported. For summarization, ROUGE-2 (F-measure) is the primary metric, supplemented by ROUGE-L and BLEURT-20. For BIG-bench, the "preferred metric" for each task (as specified by task authors) is normalized by setting the maximum possible score to 100 and the random chance score to 0, then averaged across tasks. For toxicity analysis, the Perspective API toxicity probability (0 to 1) is used.

• Baselines. The primary baselines are prior state-of-the-art large language models: GPT-3 175B (Brown et al., 2020), GLaM 62B/64E (Du et al., 2021), Megatron-Turing NLG 530B (Smith et al., 2022), Gopher 280B (Rae et al., 2021), Chinchilla 70B (Hoffmann et al., 2022), and LaMDA 137B (Thoppilan et al., 2022). For BIG-bench, the baseline also includes average and best human performance from crowdworkers. For reasoning tasks, baselines include finetuned state-of-the-art results that typically use task-specific architectures and verifiers (e.g., Cobbe et al., 2021 for GSM8K). For code tasks, baselines include Codex 12B (Chen et al., 2021) and Davinci Codex (via the OpenAI API). For finetuning, baselines include T5-11B (Raffel et al., 2020), ST-MoE-32B (Zoph et al., 2022), mT5 XXL (Xue et al., 2021b), and ByT5 XXL (Xue et al., 2021a).

• Generation budget / compute accounting. The number of few-shot exemplars (shots) is the primary compute-like control variable and is specified per task in the results tables, ranging from 0-shot to few-shot settings with up to 64 exemplars. For code tasks, the number of samples drawn for pass@k (k = 1, 25, 80, or 100) controls the inference budget. For BIG-bench, evaluations are reported at 0-shot, 1-shot, and 5-shot settings. Test-time compute is not systematically varied as an independent variable—the paper studies what happens at a fixed prompt-and-sample budget as model scale varies, rather than studying how to allocate a variable inference budget (which is the focus of the later "test-time compute" paper by Snell et al., also in this assignment).

• Cross-validation / statistical protocol. For the few-shot evaluation on the 29 English NLP tasks, the paper evaluates a single checkpoint (the final one, at 780B tokens for 8B and 540B, 795B for 62B) and compares against the best per-task result from prior work. For the variance analysis in Section 6.8, 15 evenly spaced checkpoints (each separated by ~2B training tokens) are evaluated on three tasks (StoryCloze, Natural Questions, WebQuestions) to assess variability. For BIG-bench, the paper reports results on the subset of 58 tasks that were evaluated by GPT-3, Gopher, and Chinchilla to enable direct comparison; all three PaLM model sizes are evaluated on the same tasks. For the contamination analysis in Section 8, each dataset is split into "contaminated" and "clean" subsets based on whether at least 70% of the 8-grams in the question, prompt, or target appeared at least once in the training data, and results are reported on both subsets. No cross-validation is used for strategy selection, as the paper does not propose adaptive strategies—it reports fixed-prompt performance.

Main Quantitative Results

English NLP Benchmark Results: Few-Shot State-of-the-Art at Unprecedented Breadth

The headline result from Section 6.1 is that PaLM 540B achieves state-of-the-art few-shot performance on 24 of 29 English NLP tasks in the 1-shot setting and 28 of 29 tasks in the few-shot setting, compared against the best per-task result from any prior large language model (GPT-3, GLaM, Megatron-Turing NLG, Gopher, Chinchilla, LaMDA). Table 4 provides the complete per-task breakdown.

Notable margins over prior SOTA include: TriviaQA few-shot at 81.4% (1-shot) vs. prior 75.8% (GLaM); Natural Questions few-shot at 39.6% (64-shot) vs. prior 32.5% (GLaM); ANLI R2 few-shot at 56.1% (5-shot) vs. prior 41.2% (GLaM); and DROP F1 at 70.8% (1-shot) vs. prior 57.8% (GLaM). On SQuADv2, PaLM 540B achieves 83.3 F1 (5-shot) vs. prior 71.8 (GLaM)—a >10 point improvement.

PaLM 540B outperforms the similarly-sized Megatron-Turing NLG 530B on all overlapping benchmarks (Table 4), suggesting that model size alone does not explain the performance—data quality and training procedure also contribute. The average NLU score (across 21 tasks) improves from 59.2 (8B) to 67.3 (62B) to 74.7 (540B), and average NLG score (across 8 tasks) improves from 41.5 (8B) to 57.7 (62B) to 63.9 (540B)—both roughly log-linear with scale (Table 5). PaLM 62B already outperforms GPT-3 175B on both average NLU (67.3 vs. 65.4) and average NLG (57.7 vs. 52.9), which the paper attributes to data quality advantages.

On MMLU (57 tasks spanning humanities, STEM, social sciences, and other domains) in the 5-shot setting, PaLM 540B achieves 69.3% average, outperforming Chinchilla 70B at 67.5% (Table 6). PaLM 540B outperforms Chinchilla on Humanities (77.0 vs. 63.6), STEM (55.6 vs. 54.9), and Social Sciences (81.0 vs. 79.3), but underperforms on the "Other" category (69.6 vs. 73.9).

BIG-bench: Aggregate Performance Exceeding Average Human, with Discontinuous Improvements

The headline from Section 6.2 is that PaLM 540B 5-shot achieves a higher normalized preferred metric score than the average human performance on 58 BIG-bench tasks common across models (Figure 3, left). The aggregate normalized score for PaLM 540B 5-shot is approximately 53, compared to ~43 for the average human. PaLM 540B 0-shot (~37) already exceeds GPT-3 0-shot (~5), Gopher 5-shot (~30), and Chinchilla 5-shot (~35). On the expanded set of 150 BIG-bench textual tasks (Figure 3, right), PaLM 540B 5-shot achieves ~42 normalized average score vs. ~35 for average human.

Per-task analysis shows PaLM 540B outperforms the prior SOTA on 44 of the 58 common tasks (Figure 4, showing a distribution of score differences that skews strongly positive).

The critical finding on discontinuous improvements is quantified in Section 6.2: ~25% of 150 BIG-bench tasks show discontinuities greater than 10 percentage points, and ~15% show discontinuities greater than 20 percentage points, where discontinuity is defined as the gap between actual 540B performance and what log-linear extrapolation from 8B→62B would predict. Specific examples shown in Figure 5 include english_proverbs (8B: ~10%, 62B: ~25%, 540B: ~87%—a ~50% discontinuity), logical_sequence (8B: 13%, 62B: 25%, 540B: 87%—a ~50% discontinuity), and logical_args (roughly log-linear, 8B: ~30%, 62B: ~55%, 540B: ~85%). Counter-examples where scale does NOT help include navigate (540B ~55% vs. best human ~95%) and mathematical_induction (540B ~55% vs. best human ~80%), which show relatively flat scaling curves.

On BIG-bench Lite (24 curated tasks, Figure 7), PaLM 540B achieves the best performance among all model scales on all tasks, but exceeds the best human performance on only 3 of 24 tasks, indicating significant room for improvement.

The paper validates that results are not due to memorization: the BIG-bench canary string does not appear in PaLM's training data, the benchmark was not available on the internet when training data was collected, and manual spot-checks confirmed no label leakage in model outputs.

Reasoning Tasks: Chain-of-Thought + Scale Matches or Exceeds Finetuned SOTA

Section 6.3 reports that PaLM 540B with 8-shot chain-of-thought prompting achieves new state-of-the-art on 4 of 7 reasoning benchmarks (GSM8K, MAWPS, SVAMP, StrategyQA) and near-SOTA on the remaining 3 (ASDiv, AQuA, CommonsenseQA). Figure 10 shows the scaling curves for all 7 datasets.

On GSM8K (Table 10), PaLM 540B with chain-of-thought + calculator achieves 58%, outperforming the prior finetuned SOTA of 55% (Cobbe et al., 2021, which used GPT-3 finetuning + calculator + task-specific verifier). Without the calculator, PaLM 540B achieves 54%. Without chain-of-thought, PaLM 540B achieves only 17%, highlighting the critical role of prompting strategy. PaLM 62B + chain-of-thought achieves 33%, showing that both scale and chain-of-thought are necessary for breakthrough performance.

Error analysis on 150 random GSM8K examples (Figure 9): PaLM 62B made 45 errors. Of these, 20 were "semantic understanding" errors (540B fixed 6), 18 were "one-step missing" errors (540B fixed 12), and 7 were "other" errors including hallucinations and symbol mapping (540B fixed 4). This demonstrates that scaling addresses multiple qualitatively different error types.

On the remaining benchmarks (Figure 10): MAWPS 540B + CoT: ~95% vs. prior SOTA ~90% (Lan et al., 2021); SVAMP 540B + CoT: ~87% vs. prior SOTA ~40-45% (Pi et al., 2022); StrategyQA 540B + CoT: ~78% vs. prior SOTA ~70% (Geva et al., 2021); CommonsenseQA 540B + CoT: ~82% vs. prior SOTA ~78% (Talmor et al., 2019 leaderboard).

Code Tasks: Strong Performance from a Generalist Model, Further Improved by Code-Specific Finetuning

Section 6.4 reports that PaLM 540B, despite being a general-purpose language model not specifically designed for code, achieves competitive performance with Codex 12B on HumanEval, and PaLM-Coder 540B (with additional code finetuning) sets new state-of-the-art results on several benchmarks.

On HumanEval (Table 12): PaLM 540B achieves pass@100 of 76.2% (vs. Codex 12B at 72.3%) and pass@1 of 26.2% (vs. Codex 12B at 28.8%). This is notable because PaLM was trained on only ~2.7B tokens of Python code vs. Codex's reported 100B Python tokens—PaLM achieves comparable performance with ~50× less Python data, attributed to transfer from other programming languages and natural language, plus the sample efficiency of larger models (Kaplan et al., 2020).

On MBPP: PaLM 540B achieves pass@80 of 75.0% (vs. LaMDA 137B at 62.4% reported by Austin et al., 2021) and pass@1 of 36.8%. On TransCoder (C++ to Python translation): PaLM 540B achieves pass@25 of 79.8% (vs. prior SOTA of 67.2% from Lachaux et al., 2020) and pass@1 of 51.8%.

On GSM8K-Python (converting math word problems to Python programs): PaLM 540B achieves pass@1 of 51.3% (4-shot).

On DeepFix (code repair): PaLM 540B achieves a compile rate of 73.7%, exceeding the prior SOTA of 71.7% from Yasunaga & Liang (2021). This is without code-specific finetuning.

PaLM-Coder (additional finetuning on 7.75B tokens of code, Table 12): HumanEval pass@100 improves to 88.4% (+12.2% absolute); MBPP pass@80 improves to 80.8% (+5.8%); TransCoder pass@25 improves to 82.5% (+2.7%); DeepFix compile rate improves to 82.1% (+8.4%). Figure 12 shows the scaling curves for both PaLM and PaLM-Coder across all five code benchmarks, with performance improving at each scale for both variants—the scaling does not appear to be saturating at 540B for code tasks.

Comparison with Davinci Codex: PaLM-Coder 540B outperforms Davinci Codex on HumanEval pass@100 (88.4% vs. 81.7%), underperforms on MBPP pass@80 (80.8% vs. 84.4%), and outperforms on TransCoder pass@25 (82.5% vs. 71.7%). The paper notes that Davinci Codex's model size, training data, and evaluation methodology are not publicly known, making scientific comparison difficult.

A detailed analysis of DeepFix edit sizes (Table 13) shows that PaLM-Coder tends to change fewer characters but more lines compared to Davinci Codex, suggesting different code repair strategies. The paper also demonstrates that a small, targeted change to the prompt (splitting combined variable declarations into separate lines) improved the compile rate from 82.1% to 83.4% and increased the "lines changed ≤5" success rate from 66.8% to 70.2%, illustrating sensitivity to prompt formatting—an important practical consideration.

Translation: Few-Shot Performance Competitive with or Exceeding Supervised Baselines

Section 6.5 reports that on English-centric WMT language pairs, PaLM 540B 0-shot achieves BLEU scores that exceed prior zero-shot results from GPT-3 and FLAN, and in some cases exceed even supervised finetuned SOTA (Table 14).

Key results for PaLM 540B 0-shot: English→French 38.5 (vs. supervised SOTA 45.6); English→German 31.8 (vs. supervised SOTA 41.2); English→Romanian 24.2 (vs. supervised SOTA 33.4); French→English 41.1 (vs. supervised SOTA 45.4); German→English 43.8 (vs. supervised SOTA 41.2—PaLM 0-shot exceeds the supervised baseline); Romanian→English 39.9 (vs. supervised SOTA 39.1—PaLM 0-shot slightly exceeds the supervised baseline).

Few-shot (5-shot) results are generally stronger than 0-shot: English→French 44.0; English→German 37.4; German→English 47.5 (exceeding supervised SOTA by 6.3 BLEU).

Figure 15-left compares PaLM 540B 0-shot against GPT-3, FLAN, and LaMDA: PaLM outperforms all baselines on all language pairs, with gaps exceeding 13 BLEU in some cases (English→French). Figure 15-right shows the scaling curves: improvements from 62B to 540B are often drastically larger than from 8B to 62B—for English→German, 8B→62B: ~+3 BLEU, 62B→540B: ~+13 BLEU; for English→French, 8B→62B: ~+4 BLEU, 62B→540B: ~+17 BLEU. These are further examples of discontinuous scaling behavior.

On direct (non-English-centric) language pairs (Table 15): PaLM 540B 0-shot achieves 28.6 BLEU on German→French (vs. supervised SOTA 31.5) and 25.2 BLEU on French→German (vs. supervised SOTA 24.9—PaLM 0-shot slightly exceeds the supervised baseline).

On the extremely low-resource English↔Kazakh pair (Kazakh has only 134M training tokens): PaLM 540B few-shot achieves 5.1 BLEU on English→Kazakh (vs. supervised SOTA 15.5) and 20.8 BLEU on Kazakh→English (vs. supervised SOTA 30.5). The gap to supervised SOTA is large but the model achieves non-trivial performance despite minimal exposure to Kazakh.

A consistent pattern: translation quality is better when translating into English than out of English across all language pairs, mirroring the observation from GPT-3. The paper also notes that in most cases, using language names in the prompt (0-shot) provides stronger performance than providing input-output exemplars without explicit language names (1-shot and few-shot), in line with findings from Reynolds & McDonell (2021).

Multilingual Natural Language Generation and Question Answering

Section 6.6 reports the first systematic evaluation of a large decoder-only language model on few-shot conditional text generation tasks (summarization and data-to-text). The headline is that few-shot performance scales with model size across all languages and tasks, but the gap to finetuned SOTA remains substantial, especially for non-English generation (Table 16).

In the 1-shot setting for summarization, PaLM 540B achieves ROUGE-2 F-measure scores that are generally much lower than finetuned SOTA: MLSum German 12.8 vs. 36.4, MLSum Spanish 3.6 vs. 13.8, XSum English 12.2 vs. 23.2. For cross-lingual summarization (WikiLingua: translate into English from various source languages), PaLM 540B 1-shot achieves 5.5–8.5 vs. finetuned SOTA of 14.6–18.3. This gap is noted as "the first demonstration of few-shot summarization with large language models" and "a strong starting point to close the gap."

For data-to-text tasks, the few-shot-to-finetuning gap is smaller: E2E NLG 1-shot 35.2 vs. finetuned SOTA 45.8; WebNLG English 1-shot 44.4 vs. finetuned SOTA 53.5; WebNLG Russian 1-shot 14.9 vs. finetuned SOTA 25.5. Scaling trends show improvements at each model size across all tasks (Table 16).

In the finetuning setting, PaLM 540B achieves new SOTA on 4 of 6 summarization tasks that generate English text (WikiLingua Spanish→English, Russian→English, Turkish→English, Vietnamese→English), but does not achieve SOTA on non-English summarization (MLSum German, MLSum Spanish). For data-to-text, PaLM 540B finetuning achieves near-SOTA across all tasks, including a new SOTA for Czech Restaurant (30.6 vs. prior 30.2) and competitive performance on E2E (45.3 vs. prior 45.8).

For multilingual question answering (TyDiQA-GoldP, Table 17), PaLM 540B finetuning achieves 80.0% average exact match across 9 languages, outperforming mT5 XXL (79.1%) but underperforming ByT5 XXL (81.4%). The few-shot performance (60.5% average) shows a large gap to finetuning, but the gap varies by language: Swahili few-shot 75.6% vs. finetuning 85.2% (gap ~10 points), while Telugu few-shot 46.9% vs. finetuning 84.9% (gap ~38 points).

Finetuning on SuperGLUE: Competitive Performance Despite Architectural Disadvantage

Section 6.1.2 reports that finetuned PaLM 540B achieves competitive but not SOTA performance on SuperGLUE (Table 7, dev set): average score 92.6 vs. ST-MoE-32B at 93.2 and T5-11B at 89.9. The paper notes that encoder-decoder models trained with span corruption (T5, ST-MoE) generally outperform decoder-only autoregressive models on classification task finetuning at equal cost, so PaLM's near-SOTA performance demonstrates that scale can partially bridge this architectural gap.

On the SuperGLUE test set (Table 9): PaLM 540B finetuning achieves 90.4 average, vs. ST-MoE-32B at 91.2 and the best decoder-only LM on the leaderboard at 71.8 (a wide margin). The comparison between few-shot (Table 8, dev set) and finetuning shows that finetuning provides substantial gains—e.g., BoolQ 89.1 → 92.2, CB 89.3 → 100/100, RTE 81.2 → 95.7—demonstrating that finetuning remains valuable even at 540B scale.

Variance Analysis and Checkpoint Robustness

Section 6.8 evaluates few-shot performance across 15 evenly spaced checkpoints during the final ~30B tokens of training. On StoryCloze and Natural Questions, variance is relatively low and performance is generally increasing with more training. On WebQuestions (Figure 17), PaLM 540B shows significantly larger variance, with 1-shot performance ranging from ~17% to ~25% across checkpoints, peaking at ~770B training tokens and declining in later checkpoints—despite this, the final checkpoint (780B) result of 22.6% is reported in Table 4.

Memorization Analysis

Section 7 reports that PaLM 540B memorized (exactly reproduced a 50-token continuation given a 50-token prompt) 2.4% of training examples vs. 1.6% for the 8B model (Figure 18a). The memorization rate strongly correlates with the number of times an example appears in training: examples seen once have a ~0.75% memorization rate, while examples seen >500 times have >40% memorization (Figure 18b). By corpus, code has the highest memorization rate (~13% for 540B) due to boilerplate licensing strings and duplicated snippets, while books have the lowest (~0.5%). The paper notes that "most instances of memorization were of formulaic text that is unlikely to trigger concern" but also observed memorization of stories, news articles, and facts (Table 39 shows examples).

Toxicity and Bias Analysis

Section 10.1.1 reports that PaLM 540B achieves state-of-the-art Winogender coreference resolution accuracy in 1-shot and few-shot settings: 79.4% with multiple-choice scoring (vs. GLaM 1-shot 71.7%) and 84.7% with generative scoring in the 4-shot setting (Figure 20). However, disaggregated accuracy (Figure 21) reveals a performance gap between "stereotypical" and "gotcha" examples—on female-gendered gotcha examples, accuracy is substantially lower than on stereotypical examples, though the gap narrows with more shots (from 14.1 to 10.1 percentage points in 1-shot, 18.3 to 9.2 points in 4-shot for the male-female gap).

Section 10.1.2 reports co-occurrence analysis for gendered pronouns (Table 35), racial identity terms (Table 36), and religious identity terms (Table 37). For religion, the analysis reveals that prompts involving Islam produce continuations with words like "terrorist," "violent," and "radical" as top co-occurring descriptive terms. The paper notes that the 62B and 540B models have very similar co-occurrence counts (~70% overlap in top-10 words), suggesting "the underlying training data has more influence on the results than the size."

Section 10.2 reports toxicity degeneration analysis on the RealToxicityPrompts dataset. Figure 23 shows that PaLM 540B (and 62B) produce higher average toxicity in continuations than the 8B model, but the model's toxicity probability correlates strongly with the prompt's toxicity probability—much more so than human continuations. This suggests the model is "strongly influenced by the prompt-style and is likely to generate continuations with a similar level of toxicity as the prompt," which is a different toxicity profile from human text (where toxic prompts do not strongly predict toxic continuations). Table 20 reports the probability of generating at least one toxic continuation: for PaLM 540B given toxic prompts, 80% (first-sentence) and 91% (full 128 decoding steps); given non-toxic prompts, 46% (first-sentence) and 56% (full).

Ablation Studies and Robustness Checks

Dataset contamination analysis (Section 8, Tables 18-19): The paper splits 10 partially contaminated English NLP benchmarks and 6 translation language pairs into "clean" and "contaminated" subsets based on 8-gram overlap with training data. For English NLP, the accuracy deltas between clean and full subsets are approximately balanced—some tasks show positive deltas on the clean subset (e.g., TriviaQA +0.1 for 540B, WebQuestions +0.3 for 540B), some show negative deltas (e.g., Winograd -1.8 for 540B, WSC -3.5 for 540B). The 8B model shows a similar pattern of mixed deltas, suggesting that contamination does not systematically inflate results through memorization. For translation (Table 19), most language pairs show very similar BLEU on clean vs. full sets, with the largest delta being German→English (-3.4 BLEU on clean for 540B), but this delta is consistent across model scales, which argues against memorization as the cause (since the 8B model has much lower memorization capacity).

Training longer on refreshed vs. repeated data (Appendix F, Table 33): When PaLM 62B training is extended by 95B tokens (from 795B to 890B), continuing with the original dataset (which now contains repeated subcorpora) yields lower performance than continuing with a refreshed dataset containing new data. For example, Trivia QA 1-shot: original baseline 72.7, with repeats 73.3, with refreshed data 74.2. The paper uses this finding to motivate training PaLM 62B to 1.325T tokens on refreshed data (Section 13), which yields substantial improvements across tasks (Table 32): MMLU 53.7 → 62.8, BIG-bench 32.4 → 40.8, GSM8K 33.0 → 48.7, HumanEval 15.9 → 23.7. However, even with 67% more training FLOPs, PaLM 62B does not match PaLM 540B on any of these benchmarks (Table 32), providing evidence that model scale provides benefits beyond what additional training data alone can compensate for.

Scaling behavior across training tokens (Appendix F, Figure 27): The plots of PaLM 62B performance as a function of total training FLOPs show that longer training (to 1.325T tokens) produces log-linear improvements that shift the scaling curve but do not change its slope substantially relative to the 8B→62B→540B scaling on 780B tokens. The gap between the dashed line (62B trained longer) and the solid line (540B at 780B tokens) remains large, indicating that parameter count and training tokens are not perfectly fungible, at least within the ranges studied.

Prompt engineering impact on code repair (Section 6.4, Table 13): A targeted change to the DeepFix prompt (splitting one occurrence of combined variable declarations into separate lines in both the broken and fixed code exemplars) improved the overall compile rate from 82.1% to 83.4% and increased the "lines changed ≤5" success rate from 66.8% to 70.2%. This demonstrates sensitivity to specific prompt formatting and suggests that reported DeepFix results may not represent the upper bound of what careful prompt engineering could achieve.

Checkpoint variance (Section 6.8, Figure 17): The variance analysis across 15 checkpoints shows relatively low variability for StoryCloze and Natural Questions (consistent scaling across checkpoints) but high variability for WebQuestions on PaLM 540B, where the reported final checkpoint result (22.6%) is substantially worse than the peak (25.0%). This suggests that single-checkpoint evaluation may understate peak model performance on some tasks and that the reported ranking of PaLM vs. baselines could depend on checkpoint selection for high-variance tasks.

Scoring methodology impact on Winogender (Section 10.1.1, Figure 20): The paper compares multiple-choice scoring (comparing relative probabilities of answer options) with generative scoring (exact string match on generated text) for Winogender. Multiple-choice scoring produces substantially higher accuracy (79.4% 1-shot) than generative scoring (69.7% 1-shot), and in the 0-shot setting, generative scoring produces near-zero accuracy because the model does not understand the task format. This highlights that the choice of scoring methodology significantly affects reported results on coreference tasks and that multiple-choice scoring can overstate model capabilities in zero-shot settings.

Adafactor $\beta_2$ schedule (Section 5): The paper describes using a dynamic $\beta_2$ schedule ($\beta_2 = 1.0 - k^{-0.8}$) rather than the standard $\beta_2 = 0.99$, motivated by stability concerns for rare embedding tokens. While no ablation of this schedule is reported (the computational cost would be prohibitive), the paper states that this was "found to be more stable" and that the standard $\beta_2$ led to instability. This is not an ablation in the traditional sense but represents a key training stabilization choice.

Critical Assessment

The experiments in this paper overwhelmingly support its central descriptive claim: scaling a dense decoder-only Transformer to 540B parameters, while holding data and training procedure constant, produces continued improvements across an exceptionally broad range of tasks, with evidence that these improvements have not plateaued. The clean scaling curves (8B→62B→540B) across English NLP (Table 5), BIG-bench (Figure 3), reasoning (Figure 10), code (Figure 12), translation (Figure 15-right), and multilingual generation (Table 16) consistently show substantial gains at each scale step. This is the paper's strongest empirical contribution, and the breadth of evaluation (hundreds of benchmarks across multiple domains) makes the case far more compelling than a single-benchmark study would.

However, the paper makes several stronger, more specific claims whose evidential support warrants scrutiny:

Claim: "Discontinuous improvements are common—25% of BIG-bench tasks show >10% discontinuity." The evidence for this claim exists (Section 6.2, Figure 5) but has several limitations. First, the discontinuity metric depends on having exactly three data points (8B, 62B, 540B), and deviations from log-linearity at coarse granularity (factors of ~8× between scales) could arise from sampling noise, checkpoint variability (as demonstrated for WebQuestions in Figure 17), or the specific choice of 62B as the intermediate point. A finer-grained scaling curve (e.g., 8B, 30B, 100B, 300B, 540B) would be needed to confidently distinguish genuine discontinuities from noisy but continuous scaling. Second, BIG-bench tasks vary widely in their number of examples and inherent noisiness, and the paper does not report confidence intervals on individual task scores that would enable statistical claims about discontinuity. Third, the definition of discontinuity as deviation from log-linear extrapolation is heuristic—alternative functional forms (e.g., sigmoidal or threshold models) might fit the data equally well and would change which tasks are classified as "discontinuous." The claim is suggestive but not rigorously established; it points to an important phenomenon that deserves more systematic study with finer scaling resolution.

Claim: "Few-shot chain-of-thought + scale matches or exceeds finetuned SOTA on reasoning." The evidence for GSM8K (Table 10: 58% vs. 55%) is clear, and the broader pattern across 7 reasoning datasets (Figure 10) supports the claim. However, a critical qualification is that the finetuned baselines are from prior work using different base models (typically GPT-3 or BERT-based architectures) and different training procedures. The paper does not compare PaLM 540B few-shot against PaLM 540B finetuned on the same reasoning datasets—such a comparison would reveal whether the few-shot approach is genuinely competitive with what the same model could achieve through finetuning, or whether the finetuned baselines are simply from weaker base models. The claim that "specialized architectures are unnecessary" would be stronger if the paper showed that finetuning PaLM itself on these tasks does not substantially exceed few-shot performance. The comparison in Table 8 for SuperGLUE shows large gaps between few-shot and finetuning (e.g., RTE 81.2 vs. 95.7), suggesting that for at least some tasks, finetuning still provides substantial benefits even at 540B scale.

Claim: "PaLM 540B 5-shot outperforms average human performance on BIG-bench." This claim rests on the normalized aggregate score across 58 or 150 tasks (Figure 3). However, the paper also shows (Figure 6) that average human performance is higher than PaLM 540B on ~35% of individual tasks, and on BIG-bench Lite (Figure 7), PaLM 540B exceeds the best human performance on only 3 of 24 tasks. The aggregate claim masks substantial task-level variation and depends on the specific tasks included in the aggregate and the normalization procedure. Tasks where the model outperforms humans may be those where the human baseline was weak (e.g., crowdworkers not proficient in the relevant languages) or where the task is memorization-heavy (e.g., periodic_elements). The aggregate claim is technically correct but potentially misleading about the model's capabilities relative to humans on genuinely difficult reasoning tasks.

The Chinchilla comparison (Section 13) is post-hoc and incomplete. The paper uses the contemporaneous Chinchilla findings to contextualize PaLM's scaling behavior but provides only an indirect comparison: PaLM and Chinchilla share only 9 English NLP tasks and 58 BIG-bench tasks in common (Table 21, Figure 24). On English NLP, PaLM 540B (2527 ZettaFLOPs) outperforms Chinchilla (588 ZettaFLOPs) at 70.3 vs. 65.2 average, but this is not a fair compute-matched comparison—PaLM used ~4.3× more training FLOPs and a different dataset. On BIG-bench, PaLM 540B and Chinchilla achieve similar normalized scores (~53 vs. ~35) but again with vastly different compute. The additional experiment training PaLM 62B to 1.3T tokens (Appendix F) provides partial evidence that scaling training tokens helps but does not close the gap to PaLM 540B (e.g., GSM8K 48.7% vs. 54.0%, BIG-bench 40.8 vs. 53.7). However, this experiment does not answer the Chinchilla-optimal question: what would a model with PaLM 540B's training FLOPs (2527 ZettaFLOPs) but Chinchilla-optimal allocation (smaller model, more data) achieve? The absence of such a comparison—which would require training a ~150B model on ~4T tokens using the same data and infrastructure—is a genuine limitation of the paper's framing of the scale-vs-data tradeoff.

The multilingual evaluation, while broader than prior work, does not establish that scaling disproportionately benefits multilingual tasks. The paper shows that performance on non-English tasks improves with scale (Table 16, Table 17) but does not compare the rate of improvement for English vs. non-English tasks. If non-English tasks improve more slowly with scale than English tasks, this would suggest that scaling amplifies the English-centric bias of the training data rather than mitigating it. The paper notes (Section 6.6) that "PaLM is better at processing non-English input than generating non-English output" and that "the relative difference between few-shot and finetuning is larger for non-English generation," but does not quantify the differential scaling rate.

Missing experiments that would strengthen the paper:

• Finer-grained model scales (e.g., 30B, 100B, 200B) to better characterize the shape of the scaling curve and distinguish genuine discontinuities from sampling artifacts. The three data points (8B, 62B, 540B) are too coarse to determine functional form.
• Finetuning comparisons on reasoning tasks: finetuning PaLM 540B itself on GSM8K and other reasoning benchmarks to determine whether few-shot truly matches the model's own finetuning ceiling.
• Ablation of architectural choices at the largest scale: the paper justifies SwiGLU, parallel layers, multi-query attention, and other modifications based on ablations at smaller scales (8B-62B) or prior work, but does not verify that these choices remain optimal at 540B.
• A compute-matched Chinchilla comparison: training a ~150B model on ~4T tokens using PaLM's data and infrastructure to answer the "scale vs. data" question within a controlled setting.
• Prompt sensitivity analysis: for the headline few-shot results (Table 4), the paper reports results with a specific number of shots per task, chosen to maximize performance. The sensitivity of these results to shot count, exemplar selection, and prompt wording is not systematically explored, yet Section 6.8 shows substantial variance across checkpoints for some tasks, suggesting prompt sensitivity could be similarly large.

The engineering achievement is genuinely impressive but is a scaffolding for the scientific claims, not a substitute for them. Training a 540B model on 6144 TPU v4 chips at 46.2% MFU without pipeline parallelism is a significant systems contribution. However, the paper's scientific conclusions about scaling rest on the quality of the controlled comparison (same data, same procedure, different scales) and the breadth of evaluation. These are strong points, and the paper's transparency about limitations—data repetition effects, checkpoint variance, the inability to perform certain ablations at full scale due to cost—lends credibility. The key caveat is that scaling behavior is likely specific to this model architecture, training data, and evaluation suite, and the generality of the findings (e.g., about discontinuous improvements, the scale-vs-data tradeoff) to other settings remains an open question that the paper does not fully answer.

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Makes the Compute-Optimal Framework Impractical Without a Cheaper Proxy

The paper's central contribution—difficulty-conditioned test-time compute allocation—rests on the ability to estimate each prompt's difficulty before deciding how to allocate the inference budget. The method used for this estimation is prohibitively expensive for deployment. The paper generates 2048 samples per question and averages either ground-truth correctness (oracle difficulty) or PRM final-answer scores (predicted difficulty) to bin questions into five difficulty quintiles (Section 3.2). The authors acknowledge this explicitly:

"we do not account for this cost in our test-time compute results... estimating difficulty in this way still incurs additional computation cost during inference, and we leave exploration of this tradeoff to future work" (Section 3.2).

Consequence: Generating 2048 samples per prompt to estimate difficulty consumes more computation than the largest test-time budgets studied (256–512 generations). If this cost were amortized into the reported efficiency numbers, the claimed 4× improvement over best-of-N would shrink dramatically or disappear entirely. For a deployment where each incoming prompt is unique (e.g., a user-facing chatbot), amortizing difficulty estimation across many queries is impossible—every prompt incurs the estimation cost. The paper's compute-optimal policy is therefore an upper bound on achievable efficiency, not a realized deployment gain. The true cost is cost(difficulty estimation) + cost(strategy execution), and the paper only reports the second term.

Evidence: Section 3.2 describes the 2048-sample procedure; Figures 4 and 8 show that both oracle and predicted difficulty bins yield similar compute-optimal scaling curves, but neither figure includes the estimation cost in the x-axis budget. The paper does not measure total cost (estimation + execution) in any experiment.

Mitigation status: The paper flags this as future work explicitly (Section 8: "training a model to directly predict the difficulty of a question without requiring any additional test-time compute") but provides no experimental results on cheap difficulty estimation. A lightweight classifier distilled from the PRM, or adaptive difficulty estimation using only a handful of initial samples, is suggested but not implemented.

---

The Method Provides No Benefit on the Hardest Problems—Test-Time Compute Cannot Substitute for Missing Capability

Across all methods—PRM search, iterative revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show essentially no improvement regardless of how much test-time compute is allocated. This is a fundamental capability bound: test-time compute can amplify existing capability but cannot create it from nothing.

Consequence: If a deployment's problem distribution includes a meaningful fraction of questions that are fundamentally outside the base model's competence range (i.e., the base model almost never produces a correct answer even with many independent samples), then no amount of search, revision, or adaptive allocation will help. The model will fail on these questions regardless of the inference budget. This means that for applications where correctness on hard problems is critical—medical diagnosis, legal reasoning, high-stakes decision support—scaling test-time compute is not a substitute for improving the base model through pretraining or fine-tuning. The FLOPs-matched comparison (Section 7) makes this explicit: on the hardest questions, the 14× larger pretrained model substantially outperforms the test-time-compute-augmented smaller model, particularly at high inference-to-pretraining token ratios.

Evidence: Figure 3 (right) shows bin 5 accuracy hovering at ~1-3% for all search methods and all budgets up to 256 generations. Figure 7 (right) shows bin 5 accuracy at ~2-3% regardless of the sequential-to-parallel ratio at 128 generations. Figure 9 shows the bin 5 scaling line for revisions as essentially flat near 0-5% across all test-time budgets; the 14× larger model's greedy performance (stars) sits above the scaling line for all three R values. The paper itself states this clearly: "on the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3).

Mitigation status: The paper is transparent about this limitation and frames it as a boundary condition rather than a failure of the approach (Section 7: "test-time compute amplifies existing capability but does not create it from nothing"). No mitigation is proposed—the limitation is intrinsic to the paradigm. The practical implication is that the method should be deployed with a difficulty estimator that also functions as a router: easy and medium questions get test-time compute, hard questions get escalated to a larger model or flagged for human review.

---

The Revision Model's 38% Correct-to-Incorrect Reversion Rate Is a Significant Practical Weakness

Section 6.1 notes that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the model is trained only on sequences where all in-context answers are incorrect followed by a correct target, so it never learns what to do when the current answer is already correct—it assumes revision is always needed and may introduce errors into an already-correct solution.

Consequence: In a sequential revision chain of length N, the model may oscillate between correct and incorrect answers. The paper mitigates this by selecting the best answer from anywhere in the chain (via majority voting or verifier-based selection) rather than always taking the final revision. However, every revision step after reaching a correct answer is wasted computation—it consumes generation budget while potentially degrading the answer quality. For a fixed budget, this means the effective number of useful revision steps is lower than the total chain length. On easy problems where the model often produces a correct answer early in the chain, this wasted computation reduces the efficiency gains that sequential revisions could otherwise provide. The 38% figure suggests that at any given step after a correct answer is produced, there is a substantial probability of the model "un-fixing" the problem, which introduces noise into the selection process and reduces the reliability of revision chains.

Evidence: Section 6.1 reports the 38% figure directly. Figures 6 and 7 show that sequential revision still outperforms parallel sampling in aggregate and on easy problems, indicating that the within-chain selection mechanism partially compensates for the reversion problem, but the paper does not report what performance would be without this mitigation or what fraction of revision steps are "wasted" on already-correct answers.

Mitigation status: The paper uses within-chain selection (majority voting or verifier-based best-of-N weighted over all answers in the chain) as a patch. This is not a solution—it recovers the correct answer after it has been lost, but it does not prevent the loss from occurring. A more principled approach would be to train the revision model to recognize when no revision is needed (by including correct-to-correct trajectories in the training data) or to use a verifier to dynamically decide when to stop revising, but neither is explored.

---

The Study Is Limited to a Single Benchmark (MATH) and a Single Model Family (PaLM 2-S*), Leaving the Generality of Difficulty-Dependent Scaling Unverified

All experiments in this paper use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an untested assumption. The difficulty-dependent scaling patterns—beam search hurting easy problems but helping medium ones (Figure 3, right), sequential revisions being optimal for easy problems but balanced sequential-parallel ratios being optimal for hard ones (Figure 7, right)—are discovered on a single dataset of competition-level math problems with a single model.

Consequence: The specific difficulty thresholds, the optimal strategy allocations per bin, and the magnitude of the 4× efficiency gains may not transfer to other domains (code generation, logical reasoning, factual QA, open-ended generation), other model families (GPT-4, Claude, Llama), or other model scales. The MATH benchmark tests symbolic mathematical reasoning with clean ground-truth answers, which enables exact-match evaluation and the PRM training pipeline (Monte Carlo rollout supervision requires knowing whether a final answer is correct). For tasks without clean correctness signals—summarization, dialogue, creative writing, complex planning—the entire framework (PRM training, difficulty estimation via pass@1, best-of-N weighted selection) would require fundamental redesign. Even within math, MATH problems skew toward competition-level difficulty; grade-school math or applied math word problems might exhibit different scaling patterns. The paper's key conceptual contribution—that difficulty-conditioned allocation matters—is likely robust, but the specific recipes are not validated beyond the narrow setting in which they were developed.

Evidence: All experiments in Sections 5-7 use MATH and PaLM 2-S* exclusively. The paper does not report results on any other benchmark (e.g., GSM8K, HumanEval, MMLU) or with any other model family. Section 4 justifies the choice of MATH as a domain where "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences," acknowledging the domain-specificity of the findings but not testing it.

Mitigation status: The paper does not address this limitation beyond the acknowledgment in Section 4. Future work on replicating the study across domains and model families is implied but not scoped. The paper's framework is presented as general, but the evidence for generality is absent.

---

The FLOPs-Matched Pretraining Baseline Uses a Potentially Under-Trained 14× Larger Model with Only Greedy Decoding, Making the Test-Time Compute Advantage Possibly Optimistic

Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. The larger model is evaluated with greedy decoding only—no best-of-N, no majority voting, no search, no chain-of-thought prompting. Additionally, the paper scales only model parameters while holding training data fixed (following the LLaMA paradigm; Touvron et al., 2023), rather than scaling both parameters and data equally (the Chinchilla-optimal approach; Hoffmann et al., 2022). The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7).

Consequence: The comparison is not between "test-time compute" and "pretraining" in general—it is between test-time compute and a specific, relatively weak pretraining baseline. A Chinchilla-optimal 14× larger model (with more training data, not just more parameters) would likely outperform the parameter-only-scaled model used here. Additionally, giving the larger model even a modest test-time compute budget—best-of-8 or best-of-16—would create a much stronger baseline. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions; Figure 1 bar chart) may shrink or reverse against these stronger baselines. The paper's conclusion that "test-time compute with a smaller model can outperform a ~14× larger model" is true under the tested conditions but may not hold if the larger model is trained more efficiently and given a fair inference budget.

Evidence: Section 7 describes the FLOP accounting and baseline. Figure 9 and the bar charts in Figure 1 show the reported comparisons. The paper does not include an ablation where the larger model receives any test-time compute budget, nor does it train a Chinchilla-optimal larger baseline. The caveat about parameter-only scaling is acknowledged but not addressed experimentally.

Mitigation status: The paper explicitly flags this as future work (Section 7: "leave the analysis of compute-optimal scaling of pretraining compute... to future work"). The limitation is transparent, but it means the headline comparison should be interpreted as a lower bound on pretraining's effectiveness relative to test-time compute, not a definitive statement of the tradeoff.

---

Sequential Revisions Incur Latency Costs That Are Not Accounted for in the Compute Budget, Limiting Applicability to Latency-Sensitive Deployments

The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial—each revision depends on the output of the previous one—while parallel best-of-N can be executed simultaneously given sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than 128 fully parallel samples running concurrently. The compute-optimal policy favors sequential-heavy strategies on easy problems (Figure 7, right: bin 1 is flat across all ratios, but higher sequential ratios are selected at lower budgets; Figure 7, left: at 8-32 generations, fully sequential is optimal).

Consequence: For latency-sensitive applications—interactive assistants, real-time code completion, live translation—the sequential-heavy strategies that the compute-optimal policy prefers may be impractical regardless of their accuracy advantages. A user waiting for a response to a math question cannot tolerate 64 sequential generation steps if each step takes several seconds. The efficiency gains reported in FLOPs do not translate to latency improvements; in fact, the strategies that maximize FLOPs-efficiency may maximize latency. The paper's results are most applicable to throughput-bound batch processing (evaluating thousands of problems offline, generating training data, scoring candidate solutions) rather than latency-bound interactive use.

Evidence: The paper never discusses latency or wall-clock time. All budgets and efficiency comparisons are in terms of generation count. Figure 5 (right panel) illustrates the sequential revision structure visually, making the serial dependency explicit, but the text does not address its practical implications for deployment speed.

Mitigation status: Not addressed. The paper's framework is purely about FLOPs efficiency, and the tension between FLOPs-optimal and latency-optimal strategies is not discussed. A deployment system would need to incorporate a latency constraint, which would shift the optimal allocation toward more parallel strategies, potentially reducing the reported efficiency gains.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the scaling hypothesis from an extrapolation exercise into a measured, multidimensional reality with clear documentation of emergent phenomena. Before PaLM, the dominant question was "does scaling continue to help?"—a yes/no question that prior work had answered with a qualified yes, supported primarily by validation loss curves (Kaplan et al., 2020) and performance on standard NLP benchmarks. PaLM transforms this into a richer set of questions: where does scaling help, how much does it help relative to other axes (data quality, training tokens, prompting strategy), and what qualitatively new behaviors emerge at sufficient scale?

The paper's most consequential reframing is the discontinuity finding. By showing that ~25% of BIG-bench tasks exhibit improvements from 62B to 540B that are drastically larger than log-linear extrapolation would predict, PaLM establishes that scaling is not merely additive—it can be transformative. This matters because it challenges the power-law mental model that dominated the field after Kaplan et al. (2020). If capabilities can emerge discontinuously, then predicting future model behavior from smaller-scale experiments becomes systematically unreliable. A 100B model's near-zero performance on a task tells you almost nothing about what a 500B model might achieve—the task could remain unsolved, or it could leap to near-human performance. This has direct implications for how organizations plan research investments: betting against scaling because a capability hasn't appeared at current scales is demonstrably risky.

The paper resolves a latent tension in the field between "scaling model size" and "scaling training tokens." Chinchilla (Hoffmann et al., 2022), published contemporaneously, argued that many large models were significantly undertrained and that compute should be allocated toward more data rather than more parameters. PaLM provides evidence that this tradeoff is not clean: when PaLM 62B is trained to 1.3T tokens on refreshed data (Section 13, Appendix F), it substantially improves but does not match PaLM 540B trained on 780B tokens—despite using ~5× fewer total training FLOPs in the original 540B run. This suggests that model scale and training tokens are partially complementary but not fully fungible—some capabilities may require a minimum parameter threshold regardless of data quantity. The paper does not claim this definitively (the comparison is post-hoc and the data mixtures differ), but it provides the empirical grounding for a more nuanced view than either pure scaling or pure Chinchilla-optimal allocation.

PaLM also reframes what counts as a "generalist" model. Before PaLM, the distinction between general-purpose language models and domain-specific systems was clearer: code models (Codex) were separate from language models (GPT-3), translation models were separate from reasoning models, and few-shot performance on specialized tasks was assumed to require domain-specific architectures or fine-tuning. PaLM demonstrates that a single dense Transformer, trained on a mixture of natural language and code with a standard autoregressive objective, can simultaneously achieve SOTA on English NLP (28/29 tasks), competitive performance on code synthesis (matching Codex 12B with 50× less Python data), and competitive or superior translation performance compared to supervised baselines. This collapses the distinction between generalist and specialist for a wide range of tasks: at sufficient scale, the generalist approach matches or exceeds specialized systems without any domain-specific architecture, training objective, or fine-tuning.

The chain-of-thought + scale finding shifts the burden of proof for reasoning research. Prior to PaLM, achieving SOTA on multi-step reasoning required a combination of fine-tuning, external tools (calculators), and trained verifiers (Cobbe et al., 2021). PaLM shows that an 8-shot prompted model—with no parameter updates—can exceed that SOTA, given enough scale. This doesn't eliminate the value of specialized reasoning systems (there remain problems where PaLM fails, and fine-tuning still provides gains on many tasks as shown in Table 8), but it redefines the baseline: any new reasoning method must now demonstrate it outperforms not just prior specialized systems, but also what a sufficiently large few-shot model can achieve. The research question shifts from "how do we build systems that can reason?" to "what does scale + prompting leave unsolved, and what specialized techniques address those residual gaps?"

The MFU metric provides a new lingua franca for training efficiency. By replacing hardware FLOPs utilization (which conflates efficiency with rematerialization choices) with model FLOPs utilization (which measures observed throughput against the theoretical minimum for the model architecture), the paper gives the field a portable, implementation-independent efficiency metric. This has already influenced how subsequent large-scale training runs are reported and compared. It refocuses the efficiency conversation from "how busy are our chips?" to "how close are we to the physical speed limit for this model architecture?"—a more meaningful target for systems research.

Follow-Up Research This Work Enables

Training a Chinchilla-optimal model at PaLM's compute scale to directly measure the scale-data tradeoff. PaLM 540B used ~2527 ZettaFLOPs of training compute on a 540B-parameter model with 780B tokens. A Chinchilla-optimal allocation at that compute budget would train a smaller model (~150B parameters) on substantially more data (~4T tokens). A controlled experiment—training both a PaLM-style 540B/780B model and a Chinchilla-style 150B/4T model using identical data, tokenizer, and infrastructure—would directly answer whether PaLM's apparent advantages over Chinchilla (Section 13, Figure 24) reflect genuine benefits of model scale or confounding factors (different datasets, different evaluation protocols). The experiment would measure few-shot performance on the same 29 English NLP tasks, BIG-bench, and reasoning benchmarks that PaLM uses, enabling a clean comparison. The PaLM 62B extended training results (Table 32: MMLU 62.8 vs. 540B's 69.3, BIG-bench 40.8 vs. 53.7) suggest the Chinchilla model would close much but not all of the gap—but this prediction needs direct testing.

Measuring whether discontinuous improvements survive finer-grained scaling resolution. The paper's discontinuity claims rest on three data points (8B, 62B, 540B) and a heuristic definition (deviation from log-linear extrapolation). A follow-up study training 4-5 intermediate model sizes (e.g., 20B, 80B, 200B, 400B) on the same data would reveal whether the apparent discontinuities are genuine threshold effects or artifacts of coarse sampling. If discontinuities persist at finer resolution—showing, for example, that english_proverbs jumps from ~25% to ~85% primarily between 100B and 250B parameters, with flat performance below and above—this would strongly support the emergence hypothesis and enable prediction of where future capabilities might appear. If discontinuities instead smooth out into steep-but-continuous sigmoids, the "emergence" framing would need revision. The BIG-bench tasks showing the largest discontinuities (Section 6.2, Figure 5) are the natural targets for this analysis.

Investigating whether data quality can substitute for scale on emergent-capability tasks. PaLM 62B outperforms GPT-3 175B on average NLU/NLG (Table 5), which the paper attributes to data quality. A systematic study could test whether data quality improvements (more aggressive filtering, higher-quality sources, synthetic data generation) can shift the discontinuity threshold—making capabilities that emerge at 540B on standard data emerge at 200B on higher-quality data. This would require training a family of models at multiple scales on both standard and enhanced datasets, measuring the "emergence point" for tasks like english_proverbs and logical_sequence. The paper's observation that the 62B and 540B models have ~70% overlap in top co-occurrence terms in the bias analysis (Section 10.1.2) suggests that data, not scale, drives certain behavioral patterns—extending this to capability thresholds would clarify the relative importance of the four scaling axes the paper identifies.

Training a revision model that explicitly learns when NOT to revise, targeting the 38% reversion rate. The sequential revision model described in Section 6 of the prior analysis suffers from a 38% correct-to-incorrect reversion rate because it is trained only on incorrect-to-correct trajectories. A follow-up could construct training data that includes correct-to-correct trajectories (where the model is trained to output the same answer when it is already correct), possibly using confidence-based heuristics or verifier scores to identify when the current answer is likely correct. The key measurement would be: does the reversion rate drop below 10%? Does this improve end-to-end accuracy for sequential revision chains beyond what within-chain selection achieves? The paper's framework of difficulty-conditioned allocation would also need updating: if revisions become more reliable (lower reversion rate), sequential strategies become more attractive at higher budgets, potentially shifting the optimal sequential-to-parallel ratio.

Replicating the chain-of-thought + scale finding on tasks where the model lacks training data. PaLM's strong reasoning performance (Section 6.3) is on benchmarks that were constructed from publicly available data. An important stress-test would be to design novel reasoning tasks—with held-out problem structures, novel logical operators, or deliberately unfamiliar domains—that are guaranteed absent from the training data, and measure whether few-shot chain-of-thought performance scales similarly. If PaLM's reasoning capability is partially supported by having seen similar problems during training (even if the exact examples weren't memorized, as the contamination analysis suggests), then performance on genuinely novel task structures might show different scaling behavior. This would clarify whether chain-of-thought + scale produces general reasoning or primarily pattern-matches against training distribution regularities.

Practical Applications and Downstream Use Cases

Rapid prototyping of NLP systems without fine-tuning infrastructure. PaLM's few-shot SOTA on 28 of 29 English NLP tasks (Table 4) means that for a wide range of classification, QA, and reading comprehension applications, a single frozen model with carefully designed prompts can match or exceed the performance of task-specific fine-tuned systems. For organizations without the infrastructure or expertise to fine-tune and deploy specialized models for each task, this eliminates a significant engineering barrier. A product team needing sentiment analysis, named entity recognition, and question answering over documents can prototype all three with a single model and a few dozen exemplars per task, rather than training and maintaining three separate fine-tuned systems. The specific numbers: PaLM 540B achieves 89.1% on BoolQ (yes/no QA), 95.0% on COPA (causal reasoning), and 86.3% on MultiRC (multi-sentence reading comprehension) in few-shot settings—performance levels that would have required task-specific fine-tuning just two years prior.

Code generation as a generalist capability rather than a separate product. PaLM's performance on HumanEval (76.2% pass@100 without code-specific fine-tuning; Table 12) demonstrates that a general-purpose language model trained with only 5% code data can be competitive with dedicated code models. For organizations building developer tools, this means a single model can serve both natural language and code use cases—code completion, code explanation, docstring generation, and natural language Q&A about code—without maintaining separate code-specific and language-specific models. The PaLM-Coder fine-tuning recipe (additional 7.75B tokens of code, achieving 88.4% pass@100 on HumanEval) provides a concrete, relatively cheap augmentation path for deployments where code is the primary modality. The cost implication: training one 540B generalist model plus lightweight code fine-tuning is likely cheaper than training and serving separate 500B+ natural language and code models.

Multilingual deployment with a single model across high-resource and low-resource languages. PaLM's translation results (Section 6.5) show that a single model can achieve competitive or superior performance to dedicated supervised translation systems for both high-resource pairs (German→English: 43.8 BLEU 0-shot vs. supervised SOTA 41.2) and extremely low-resource pairs (Kazakh→English: 20.8 BLEU few-shot, where Kazakh has only 134M training tokens—0.018% of the data). For organizations serving users across many languages, this eliminates the need to train and maintain separate translation models per language pair. A single PaLM deployment can handle translation requests for 100+ languages, with performance on low-resource languages benefiting from cross-lingual transfer from high-resource languages. The practical tradeoff: the 540B model's inference cost per token is high, so for high-volume, single-language-pair translation (e.g., English→German at industrial scale), a dedicated smaller supervised model may still be more cost-effective. But for applications with diverse, unpredictable language needs—customer support, content moderation, multilingual search—the generalist approach dominates.

Benchmarking and capability assessment with BIG-bench as a standard evaluation framework. PaLM's comprehensive BIG-bench evaluation (Section 6.2, 150 tasks) establishes BIG-bench as a viable, challenging evaluation suite for future large language models. The paper's finding that PaLM 540B outperforms average human performance on aggregate but still underperforms on ~35% of individual tasks (Figure 6) provides a calibrated difficulty range: BIG-bench tasks span from solved (PaLM exceeds best human) to very hard (PaLM is near random, e.g., linguistics_puzzles at ~0%). Future model developers can use this range to situate their models—a new model scoring significantly above PaLM 540B on the hard subset of BIG-bench tasks would represent genuine capability advancement rather than benchmark saturation. The normalization methodology (setting random chance to 0 and maximum score to 100) provides a standardized metric for aggregating across diverse task formats, which the paper has demonstrated works at scale.