Emergent Abilities of Large Language Models

🎯 Pitch

Many capabilities of large language models appear suddenly at specific scales rather than improving smoothly—a phenomenon the authors term “emergence.” Performance on tasks like arithmetic and multi-step reasoning remains near-random across orders of magnitude of scale, then jumps sharply to well above chance at critical thresholds, meaning smaller-scale experiments fundamentally cannot predict what larger models will do.

1. Executive Summary

This paper surveys and formally characterizes emergent abilities of large language models—capabilities that are not present in smaller models but appear in larger models, following a definition rooted in Anderson (1972) that emergence is when "quantitative changes in a system result in qualitative changes in behavior." The authors analyze scaling curves across five model families—GPT-3, LaMDA, Gopher, Chinchilla, and PaLM—on benchmarks including BIG-Bench, TruthfulQA, MMLU, and WiC, documenting emergent phenomena in both few-shot prompting (e.g., 3-digit arithmetic jumps from near-random to sharply above random at ~2×10²² training FLOPs) and augmented prompting strategies such as chain-of-thought reasoning (which only surpasses standard prompting at ~10²³ training FLOPs), instruction-finetuning (which hurts performance below ~7×10²¹ FLOPs but improves it above ~10²³ FLOPs), and scratchpad-based program execution (which only helps above ~9×10¹⁹ FLOPs). The paper establishes that dozens of abilities exhibit a phase-transition pattern—performance is near-random until a critical scale threshold, after which it increases substantially—and that this phenomenon cannot be predicted by extrapolating the performance of smaller models, while also showing that cross-entropy loss improves smoothly even when downstream metrics remain flat, indicating that improvements in the log-likelihood of target sequences are masked by evaluation metrics like exact-match accuracy. The analysis further demonstrates that emergent risks—including bias amplification in ambiguous contexts, increased training data memorization, and mimicry of human falsehoods—can co-occur with emergent abilities, establishing that the unpredictability of emergence applies to both capabilities and potential harms as models scale.

2. Context and Motivation

The Core Problem: The Predictability Gap in Language Model Scaling

The central problem this paper addresses is deceptively simple: can we predict what a larger language model will be able to do by looking at smaller ones? The answer, as the paper documents extensively, is often no—and this unpredictability has profound consequences for how the field thinks about scaling, safety, and research prioritization.

To understand why this matters, we need to appreciate the state of language model research when this paper was written (mid-2022). The dominant narrative was one of predictable improvement. Kaplan et al. (2020) had demonstrated that cross-entropy loss scales as a smooth power law over more than seven orders of magnitude of compute. Hoffmann et al. (2022) had refined these scaling laws with the Chinchilla model, showing that compute-optimal training follows predictable relationships among model size, data quantity, and performance. The implicit message of this work—and a reasonable inference at the time—was: train a bigger model on more data, and everything gets proportionally better. You can look at the curve for your 1-billion-parameter model and confidently project what your 100-billion-parameter model will achieve.

This paper argues that this picture is incomplete in a practically critical way. While aggregate metrics like perplexity do scale smoothly, the downstream capabilities that people actually care about—can the model do arithmetic? can it answer questions truthfully? can it follow instructions?—often do not. Instead, many capabilities exhibit a phase transition: performance stays near random across multiple orders of magnitude of scale, then abruptly jumps to substantially above random at some threshold. Before that threshold, no amount of extrapolation from smaller models would have predicted the ability exists. The paper's core contribution is documenting that this phenomenon is widespread, occurs across model families and task types, and applies not only to beneficial capabilities but also to potential risks.

Why This Problem Is Important

The gap between smooth perplexity scaling and discontinuous capability emergence matters for several interconnected reasons:

Scientific understanding of what scale does. If scaling simply made models better in a uniform, predictable way, we could understand language model capabilities as a straightforward function of compute. The existence of emergent abilities suggests something qualitatively different is happening: scale is not just refining existing computations but unlocking entirely new computational mechanisms that are absent below a critical threshold. Understanding what those mechanisms are—and why they require a certain scale to manifest—touches on fundamental questions about what neural networks learn during training and how model capacity relates to the complexity of tasks that can be performed.

Research planning and resource allocation. In a world with smooth scaling, you could test an idea on a small model and confidently extrapolate to a large one. This would make research far cheaper and faster—everyone could iterate on 1-billion-parameter models and know what to expect at 100 billion. The reality of emergent phenomena means this strategy systematically fails: the paper documents multiple cases where a technique or capability shows zero signal until a threshold that may be 10× or 100× larger than affordable research models. For example, chain-of-thought prompting (Figure 3A) provides no benefit over standard prompting at smaller scales and only becomes useful at ~10²³ training FLOPs. A researcher testing chain-of-thought on a smaller model would have concluded it doesn't work and abandoned the idea—missing one of the most significant prompting innovations of the era. This has direct economic implications: organizations cannot cheaply prototype new techniques; they must invest in large-scale experiments to discover what is possible.

Safety and societal impact. The symmetry between emergent capabilities and emergent risks is perhaps the paper's most consequential observation. If a model's ability to generate toxic content, memorize training data, or produce convincing falsehoods also exhibits phase transitions, then safety testing on smaller models will systematically miss hazards that appear in larger ones. The paper documents precisely this pattern: on the TruthfulQA benchmark, GPT-3 models actually become more likely to mimic human falsehoods as they scale up (Lin et al., 2021), before Gopher eventually crosses a threshold where truthful answering emerges. A safety team evaluating a sub-threshold model would see no truthful-answering problem—and would not predict that a larger model would both amplify falsehoods and then surpass a threshold where truthfulness improves. This makes responsible scaling fundamentally about navigating unknown unknowns.

The theoretical question of whether scaling has limits. If many capabilities exhibit phase transitions, then the question "what can't language models do yet?" becomes difficult to answer definitively. The paper's enumeration of BIG-Bench tasks where no model performs above random (Appendix E.4) is explicitly not a list of impossible tasks—it is a list of candidate tasks for future emergence. Some of these tasks (e.g., checkmate-in-one prediction, multi-step arithmetic, program synthesis) may simply require a scale that hasn't been reached yet. Others may be genuinely out of reach for autoregressive language models regardless of scale. The problem is that we currently have no principled way to distinguish between these two categories. This uncertainty matters for both AI capability forecasting and safety planning.

Where Prior Work Falls Short

The paper identifies several specific limitations in how prior work approached the relationship between scale and capability:

Scaling laws focus on aggregate metrics that mask emergence. The foundational scaling law papers (Kaplan et al., 2020; Hoffmann et al., 2022) analyzed cross-entropy loss on held-out text—a smooth, continuous signal that is insensitive to whether improvements translate into task success. The paper's cross-entropy loss analysis in Appendix A explicitly demonstrates this masking effect: on all six emergent BIG-Bench tasks examined, cross-entropy loss improves continuously even across scales where downstream metrics (exact match, BLEU, accuracy) remain at chance levels. A model with 10²¹ FLOPs has lower cross-entropy loss than one with 10²⁰ FLOPs on these tasks, but both produce equally unusable outputs. The downstream metrics only jump above random when the accumulated improvements in log-likelihood cross some critical threshold.

The implication is that scaling laws derived from perplexity are necessary but insufficient for predicting capabilities. They tell you how much better the model's probability distribution over tokens will become, but not whether that improvement will manifest as a useful behavior change. This is not a flaw in the scaling laws work per se—perplexity remains a valid and useful metric—but it means the field had been implicitly treating perplexity improvements as a proxy for capability improvements, when in fact the mapping between them is nonlinear and task-dependent.

Individual papers documented emergent phenomena but did not synthesize them. The paper itself is primarily a survey, and its key contribution is recognizing a pattern scattered across many independent works. GPT-3 (Brown et al., 2020) had shown that few-shot performance on arithmetic, translation, and QA tasks improved with scale, and included the observation that some tasks showed sharp jumps rather than smooth improvement. The BIG-Bench collaboration (2022) had identified dozens of tasks with discontinuous scaling behavior. Wei et al. (2022b) had shown that chain-of-thought prompting only worked for large models. Nye et al. (2021) had demonstrated that scratchpad training only helped above a certain scale.

However, each of these works treated emergence as a finding within their specific experimental context rather than as a general phenomenon with shared characteristics that warranted its own analysis. The BIG-Bench paper, for instance, documented emergent tasks alongside smoothly-scaling and flat tasks, without developing a taxonomy or exploring the theoretical implications. This paper's contribution is pulling these scattered observations together and saying: look, this is a pattern. It happens across model families (GPT-3, LaMDA, Gopher, Chinchilla, PaLM), task types (generative, classification, reasoning, QA), and methodologies (few-shot prompting, finetuning, specialized prompting strategies). We should treat it as a first-class object of study.

Prior work lacked a clear definition and taxonomy. The term "emergence" was used informally and inconsistently across papers. Some works used it to describe any capability that improved with scale; others reserved it for surprising behaviors; still others used it to describe capabilities that seemed to appear "out of nowhere." The paper addresses this by offering an operational definition:

"An ability is emergent if it is not present in smaller models but is present in larger models."

This definition is deliberately narrow. It does not require that the ability is fundamentally unpredictable in principle—only that it is not predictable by simply extrapolating the performance trend from smaller models. It also anchors the definition to a specific visualization pattern: a scaling curve where performance is near-random until a critical threshold, after which it increases substantially. This operationalization makes emergence testable: plot the scaling curve, check for the phase transition pattern.

The paper also distinguishes emergent abilities from emergent sociological changes (§5.5), where scaling enables general-purpose models to surpass task-specific state-of-the-art through smooth, predictable improvements. GPT-3 175B achieving SOTA on TriviaQA via few-shot prompting is not an emergent ability under this definition—it reflects the continuous improvement of a general capability—but it represents an emergent shift in how the community approaches NLP tasks.

The relationship between scale and risk was underexplored. Prior work on language model risks had documented that larger models could exhibit more bias (Bender et al., 2021), more toxicity (Gehman et al., 2020), and more memorization (Carlini et al., 2021). However, these findings were not connected to the emergent abilities literature. The risks were discussed as continuous scaling problems—"bigger models are worse in these ways"—rather than as potential phase transitions. The paper argues (§5.4) that risks can also be emergent, and that the same unpredictability that makes it impossible to forecast beneficial capabilities from small models also applies to harms. This connection was not previously made explicit.

How This Paper Positions Itself

The paper explicitly positions itself as a descriptive survey and framework paper, not as presenting new methods or models. Its contribution is conceptual: drawing together evidence from multiple independent sources to establish that emergence is a widespread and important phenomenon, then using that synthesis to motivate a research agenda.

The paper's central framing borrows directly from Philip Anderson's 1972 essay "More Is Different," which argued that in complex systems, aggregate behavior cannot be understood simply by studying components in isolation—quantitative changes in scale lead to qualitative changes in behavior. The paper applies this lens to language model scaling, arguing that just as adding more particles to a physical system can produce phase transitions (liquid to solid, non-magnetic to magnetic), adding more parameters and training compute to a language model can produce phase transitions in capability (unable to do arithmetic → able to do arithmetic, unable to reason step-by-step → able to chain intermediate steps).

This framing is important because it implies that extrapolation from smaller systems is fundamentally insufficient—not just practically difficult. In Anderson's physics examples, you cannot predict the existence of superconductivity by studying individual electrons; the collective behavior emerges only at the macroscopic scale. The paper suggests, though does not prove, that something analogous may be happening in language models: certain capabilities may require a critical mass of parameters and training data before the network can represent the necessary computations.

The paper also explicitly limits its scope in two important ways. First, it does not claim to explain why emergence occurs—the discussion section (§5.1) lists potential contributing factors (multi-step reasoning requiring minimum depth, evaluation metrics masking incremental improvements, threshold frequencies in training data) but frames these as hypotheses rather than conclusions. Second, it does not claim that the specific scales at which abilities emerge are immutable properties—it emphasizes in §5.2 that "once an ability is discovered, further research may make the ability available for smaller scale models," citing examples where improved training data (PaLM's multilingual corpus unlocking Persian QA at smaller scale) or different architectures (encoder-decoder models achieving instruction-following at 11B parameters vs. 68B for decoder-only) shifted emergence thresholds downward. The phenomenon of emergence is robust; the specific numerical thresholds are contingent on model architecture, data quality, and training procedure.

A key methodological choice is the paper's use of training FLOPs as the primary x-axis for scaling curves, rather than model parameters alone. As the paper explains in §2, "using training FLOPs or model parameters as the x-axis produces curves with similar shapes due to the fact that most dense Transformer language model families have scaled training compute roughly proportionally with model parameters." However, the paper also acknowledges that "there is not a single proxy that adequately captures all aspects of scale," citing Chinchilla (fewer parameters than Gopher but similar training FLOPs) and sparse mixture-of-experts models (more parameters per FLOP than dense models) as cases where parameter count and training FLOPs diverge. This choice matters because it affects where emergence thresholds appear on the x-axis: a model with Chinchilla-optimal training might achieve the same capability at fewer parameters but similar FLOPs, shifting the apparent threshold when plotted against parameter count.

The paper's ultimate stance is that emergence poses an important open question for the field. The conclusion states it plainly:

"the questions of how they emerge and whether more scaling will enable further emergent abilities seem to be important future research directions for the field of NLP."

This is not a position of paralysis—the paper does not argue we should stop scaling because we cannot predict the results. Rather, it argues that we should study emergence directly, develop better understanding of why it occurs, and invest in forecasting methods that go beyond simple extrapolation. The paper's value is in establishing that this is a coherent, important problem worthy of dedicated attention.

3. Technical Approach

3.1 Reader Orientation

This paper is a descriptive survey and analytical framework, not a system that is built or a method that is proposed. The "thing being built" is a conceptual lens for understanding a widespread but previously under-synthesized phenomenon: certain capabilities of language models do not improve gradually with scale but instead appear abruptly—like a phase transition—at some critical threshold of model size or training compute. The problem it solves is the predictability gap: if you only test small models, you cannot know which capabilities larger models will possess, because the scaling curves for many important tasks are flat (near-random performance) across multiple orders of magnitude before jumping sharply upward. The "shape" of the solution is a rigorous definition of emergence, a systematic taxonomy of where it occurs, a methodology for distinguishing emergent from non-emergent scaling behavior using scaling curve visualization, and an analytical toolkit for probing whether the apparent emergence is real or an artifact of evaluation metrics.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's intellectual architecture has five major components:

Definition and operationalization of emergence (§2): A precise, testable criterion for what counts as an emergent ability—performance near random until a critical scale, after which it jumps substantially above random—distinguishing emergence from smooth improvement, flat scaling, or noise.
Evidence collection across model families and tasks (§3–4, Appendices D–E): A systematic collation of scaling curves from five model families (GPT-3, LaMDA, Gopher, Chinchilla, PaLM) on benchmarks including BIG-Bench (200+ tasks), TruthfulQA, MMLU, WiC, GSM8K, and others, spanning few-shot prompting and augmented prompting strategies.
Metric analysis to probe whether emergence is real (§5.1, Appendix A): An investigation into whether the apparent phase transitions are genuine changes in model capability or artifacts of evaluation metrics (exact match, BLEU, accuracy) that mask gradual improvement. The key analysis replaces downstream metrics with cross-entropy loss and examines whether improvements are present but hidden.
Taxonomy of where emergence does and does not occur (§5.2–5.3, Appendices B–C, E–F): A categorization of BIG-Bench tasks by scaling behavior (emergent, smoothly increasing, flat), analysis of which task types are most likely to be emergent, and identification of tasks that are candidates for future emergence (currently flat even at the largest scales).
Extension to risks and sociological changes (§5.4–5.5): An argument that the unpredictability of emergent abilities also applies to emergent risks (bias, toxicity, memorization), and a distinction between emergent model abilities and emergent sociological shifts in how the community uses language models.

Information flows as follows: raw scaling data (performance at multiple model sizes for multiple tasks) → filtering through the emergence definition (does the curve show a phase transition?) → categorization into emergent/smooth/flat → metric analysis to verify that emergence is not an artifact → synthesis into a research agenda identifying open questions.

3.3 Roadmap for the Deep Dive

The technical breakdown proceeds in this order:

First, the formal definition of emergence and its operationalization through scaling curves—this is the conceptual foundation that determines which phenomena "count."
Second, the methodology for constructing and interpreting scaling curves, including the choice of x-axis (training FLOPs vs. parameter count), the handling of multiple model families, and the statistical considerations in classifying a task as emergent.
Third, the cross-entropy loss analysis (Appendix A), which is the paper's primary methodological contribution for probing whether emergence is a genuine phenomenon—this requires understanding how loss and task metrics can diverge.
Fourth, the taxonomy construction for BIG-Bench tasks (Appendix A.3, E), including the classification criteria and the analysis of which keywords correlate with emergence.
Fifth, the multi-x-axis analysis using WikiText103 perplexity (§5.3, Appendix B), which examines whether emergence is better understood as a function of language modeling quality rather than raw scale.
Sixth, the treatment of emergent risks and the distinction between emergent abilities and emergent sociological changes—which are conceptually distinct despite both being "emergent" in a broad sense.

This order builds from definition to evidence to analysis to implications, ensuring each conceptual step is grounded in what preceded it.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a conceptual analysis and survey paper whose core idea is that language model capabilities can exhibit phase transitions as a function of scale—performance stays at near-random levels across multiple orders of magnitude of training compute or model parameters, then jumps sharply to substantially above random at some critical threshold—and that this phenomenon is widespread, occurs across model families and task types, and cannot be predicted by extrapolating from smaller models.

The Definition of Emergence and Its Operationalization

The paper offers a deliberately narrow, operational definition of emergence. Quoting directly from §2:

"An ability is emergent if it is not present in smaller models but is present in larger models."

The authors immediately ground this definition in a specific visual pattern on scaling curves:

"When visualized via a scaling curve (x-axis: model scale, y-axis: performance), emergent abilities show a clear pattern—performance is near-random until a certain critical threshold of scale is reached, after which performance increases to substantially above random. This qualitative change is also known as a phase transition—a dramatic change in overall behavior that would not have been foreseen by examining smaller-scale systems."

Several design choices in this definition are worth examining closely because they determine what does and does not count as emergent.

"Not present in smaller models" is operationalized as near-random performance. The paper does not require that smaller models have zero capability—only that their performance is indistinguishable from random guessing (or a trivial baseline) on the evaluation metric. For a classification task with 4 balanced classes, "near-random" means ~25% accuracy. For a generative task scored with exact match, "near-random" means ~0% exact match. This is a pragmatic choice: it avoids debates about whether a model that scores 26% on a 4-way classification (1% above chance) has "some" ability, by setting the threshold at a level where the model is not useful for the task.

"Substantially above random" is left qualitative. The paper does not specify a numerical threshold for what counts as "substantially above random." Examining Figure 2, emergent jumps range from ~10 percentage points above random (Figure 2D, Persian QA) to ~50+ percentage points above random (Figure 2A, modified arithmetic). This flexibility is both a strength (it accommodates tasks with different baseline difficulties) and a weakness (it introduces subjectivity into classification, which the paper partially addresses through annotation by multiple co-authors, as noted in Appendix A.3).

The definition is relative to a specific evaluation metric. The paper acknowledges this explicitly in §5.1:

"It is also important to consider the evaluation metrics used to measure emergent abilities. For instance, using exact string match as the evaluation metric for long-sequence targets may disguise compounding incremental improvements as emergence."

This is the central methodological tension in the paper. If emergence is partly an artifact of metrics that demand exact correctness and award no partial credit, then the phenomenon is partly about measurement rather than about genuine qualitative changes in model behavior. The cross-entropy loss analysis in Appendix A is designed to address exactly this concern.

The definition is about predictability, not about fundamental unpredictability. The paper carefully distinguishes between "cannot be predicted by simply extrapolating the performance of smaller models" and "cannot be predicted in principle." This is an important hedge: it leaves open the possibility that better understanding of the relationship between scale and capability might eventually enable prediction, while stating that current extrapolation methods (linear or log-linear fits to performance vs. scale) fail systematically for emergent tasks.

The broader conceptual framing draws from Philip Anderson's 1972 essay "More Is Different," as quoted in §1:

"Emergence is when quantitative changes in a system result in qualitative changes in behavior."

This physics-inspired framing implies that emergence is not merely a statistical artifact but reflects a genuine change in the system's computational properties. In Anderson's original context, you cannot predict the existence of superconductivity from the properties of individual electrons—the phenomenon only exists at the macroscopic scale. The paper suggests, without proving, that something analogous may be happening in language models: below a certain scale, the network may simply lack the representational capacity or training dynamics to implement certain computations, and adding more parameters or training data enables a qualitatively new computational regime.

Scaling Curve Methodology and the Choice of X-Axis

The paper's primary evidence for emergence comes from scaling curves: plots with model scale on the x-axis and task performance on the y-axis, where each point represents a different model. The methodology for constructing these curves involves several non-trivial choices.

Training FLOPs as the primary x-axis. The paper states in §2:

"In this paper, we will analyze scaling curves by plotting the performance of different models where training compute for each model is measured in FLOPs on the x-axis."

The training FLOPs for each model are drawn from the model cards in Appendix C (Table 2), which lists parameter counts, training tokens, and computed FLOPs for all models discussed. The FLOPs calculation uses the standard approximation from the scaling laws literature:

$\text{Training FLOPs} \approx 6 \times N \times D$

where $N$ is the number of model parameters (not including embedding parameters, following Kaplan et al., 2020) and $D$ is the number of training tokens. The factor of 6 comes from the standard Transformer forward-pass cost: approximately $2ND$ for the forward pass, plus an equal amount for the backward pass (gradient computation), with additional overhead bringing the total to roughly $6ND$ .

Why FLOPs rather than parameters. The paper justifies this choice in §2:

"Using training FLOPs or model parameters as the x-axis produces curves with similar shapes due to the fact that most dense Transformer language model families have scaled training compute roughly proportionally with model parameters."

However, the paper immediately acknowledges that this proportionality does not always hold:

"For example, Chinchilla has one-fourth as many parameters as Gopher but uses similar training compute; and sparse mixture-of-expert models have more parameters per training/inference compute than dense models."

This matters practically because the emergence threshold can shift depending on which scale metric is used. A Chinchilla-style model (smaller but trained on more data) might achieve an emergent capability at fewer parameters but similar FLOPs compared to a Gopher-style model (larger but trained on less data). The paper handles this by showing both axes for key figures (Figure 2 vs. Figure 11, Figure 3 vs. Figure 12) and noting that the shapes are similar, while acknowledging that this may not hold for future architectures.

Handling multiple model families on the same plot. A key methodological strength is that the paper aggregates scaling data across independent model families from different organizations. Figure 2 shows, for example, LaMDA, GPT-3, Gopher, Chinchilla, and PaLM all plotted on the same axes for several tasks. This cross-family replication is important because it addresses a potential confound: if emergence were only observed within a single model family, it could be attributed to a quirk of that family's architecture or training procedure. The fact that similar phase-transition patterns appear across independently trained models from different labs using different architectures, data, and training recipes suggests that emergence is a robust property of scaling rather than an artifact of any particular implementation.

Specific model scales plotted. Appendix C (Table 2) provides the exact specifications:

Model Family	Smallest	Largest
GPT-3	125M params, $2.25 \times 10^{20}$ FLOPs	175B params, $3.14 \times 10^{23}$ FLOPs
LaMDA	2.1M params, $3.30 \times 10^{18}$ FLOPs	137B params, $5.54 \times 10^{23}$ FLOPs
Gopher	417M params, $7.51 \times 10^{20}$ FLOPs	280B params, $5.46 \times 10^{23}$ FLOPs
Chinchilla	417M params, $7.86 \times 10^{20}$ FLOPs	70B params, $5.63 \times 10^{23}$ FLOPs
PaLM	8B params, $3.74 \times 10^{22}$ FLOPs	540B params, $2.53 \times 10^{24}$ FLOPs
Anthropic	800M params, $4.08 \times 10^{21}$ FLOPs	52B params, $2.65 \times 10^{23}$ FLOPs

The total span is roughly six orders of magnitude in training FLOPs (from $3.30 \times 10^{18}$ to $2.53 \times 10^{24}$ ) and six orders of magnitude in parameters (from 2.1M to 540B). This wide range is what enables the paper to observe phase transitions: if data were only available for a narrow range of scales, one might see only the flat pre-emergence region or only the improving post-emergence region, and miss the transition point.

The random baseline anchoring. Every emergent scaling curve is plotted with a "Random" baseline—a horizontal line showing the expected performance from random guessing. This baseline is essential to the emergence definition because it establishes what "not present" means. For generative tasks scored with exact match, random is near 0%. For classification tasks, random is typically 25% (4-way), 50% (2-way), or 1/ $K$ for $K$ classes. The paper consistently includes this baseline in all scaling curve figures, making the phase transition visually unambiguous when it occurs.

Cross-family comparison noise. One limitation the paper does not explicitly address is that different model families were evaluated under slightly different conditions (different prompts, different numbers of few-shot exemplars, different decoding strategies). For instance, the MMLU results in Figure 2G use the evaluations from each model's original paper, which may differ in prompt format. The paper implicitly handles this by focusing on the shape of scaling curves (flat-then-jump) rather than precise numerical comparisons between families, and by relying on the most salient feature—the jump from random to substantially above random—which is robust to moderate variations in evaluation protocol.

The Cross-Entropy Loss Analysis: Is Emergence Real or an Artifact of Metrics?

The single most important methodological contribution for probing the validity of emergence is the cross-entropy loss analysis in Appendix A. This analysis addresses the central skeptical concern: that apparent emergence might be an artifact of using harsh evaluation metrics (exact match, accuracy on final answers) that mask incremental improvements. If a model goes from getting 0% to 0% to 0% exact match, then suddenly to 40%, one might wonder whether intermediate models were actually getting better in ways that exact match fails to capture—for instance, getting more digits correct, producing more plausible reasoning steps, or assigning higher probability to correct answers even when not selecting them.

The experimental setup. The analysis focuses on six BIG-Bench tasks that were classified as emergent for LaMDA models: three generative tasks (modified arithmetic, IPA transliteration, word unscrambling) and three classification tasks (logical arguments, sports understanding, figure of speech detection). For each task, the paper constructs paired plots: the standard scaling curve (error rate vs. training FLOPs) alongside a new curve (cross-entropy loss vs. training FLOPs). The cross-entropy loss is computed on the same model outputs, measuring how well the model's predicted probability distribution over the target tokens matches the ground truth.

The two possible outcomes and what they would mean. The paper frames the analysis around two hypothesized outcomes:

Outcome 1: If cross-entropy loss also does not improve over the scales where error rate is random, this would imply that the models genuinely are not getting better at the task—their internal representations of the target outputs are not improving. This outcome would support a strong interpretation of emergence as a true qualitative change.
Outcome 2: If cross-entropy loss does improve while error rate remains random, this would imply that scaling produces genuine improvements in the model's understanding of the task, but these improvements are invisible to harsh metrics like exact match until they cross a threshold where the model can finally produce outputs that are entirely correct rather than partially or probabilistically correct. This outcome would support a weaker interpretation: emergence is partly a measurement phenomenon, though still practically significant.

The actual findings. Across all six tasks, the paper finds Outcome 2. The key empirical observation is described in Appendix A.1.1 for generative tasks:

"For all three tasks, notice that while the error rate is nearly 100% for small models (≤10²² FLOPs / ≤27B params), the cross-entropy loss does actually improve for these model sizes. At the point of emergence as measured by error rate, we also see an 'elbow' in performance improvement for cross-entropy loss."

The same pattern holds for classification tasks (Appendix A.1.2):

"Similar to the generative tasks, when the error rate is close to random, cross-entropy loss consistently still improves for models trained with more compute."

What the cross-entropy loss curves actually look like. Examining Figure 5 and Figure 6 reveals a distinctive pattern. For modified arithmetic (Figure 5, left column), the error rate sits at nearly 100% from $10^{20}$ to $10^{22}$ FLOPs, then drops sharply to ~60% at $10^{23}$ FLOPs. Meanwhile, the cross-entropy loss (bottom row) shows a smooth, continuous decline from approximately 15 to 3 across the entire range. There is no phase transition in the loss—only steady improvement. The "elbow" the paper refers to is a slight steepening of the loss curve around the same scale where the error rate jumps, but the loss improvement is monotonic and substantial even in the "pre-emergence" regime.

For the classification tasks (Figure 6), the paper adds a third analysis: the log-probabilities of correct and incorrect responses separately. This reveals a counterintuitive finding:

"We find that the cross-entropy loss decreases for both the correct and incorrect responses in the three emergent multiple choice tasks. Counterintuitively, both log-probabilities can decrease in tandem even when the probability across all available multiple choice responses is normalized. The reason is that larger models produce less-extreme probabilities (i.e., values approaching 0 or 1) and therefore the average log-probabilities have fewer extremely small values."

In plain language: smaller models assign extremely low probabilities to many answer options (producing very negative log-probabilities), while larger models spread probability more evenly (producing less negative log-probabilities on average). This means the average log-probability improves (becomes less negative) for both correct and incorrect answers simultaneously, which is why cross-entropy loss improves even when accuracy remains at chance. Only when the log-probability of the correct answer eventually diverges from incorrect answers does accuracy jump above random.

What this analysis establishes and what it leaves open. The cross-entropy loss analysis makes three contributions to the paper's argument:

It demonstrates that emergence is not purely a measurement artifact. The phase transition in downstream metrics is real and robust—models genuinely cannot produce correct outputs until a certain scale, even though their internal probability distributions are improving.
It provides a more nuanced understanding of what is emergent. What "emerges" is not necessarily the raw knowledge or pattern recognition required for the task (those improve smoothly), but rather the ability to operationalize that knowledge into correct outputs as measured by stringent evaluation criteria.
It explains why emergence is unpredictable from downstream metrics alone: if you only measure exact match or accuracy, you see zero signal across multiple orders of magnitude despite genuine underlying improvement.

However, the analysis also has important limitations. It does not explain why the downstream metric eventually jumps—why accumulated improvements in log-likelihood suddenly translate into correct outputs at a particular scale rather than improving gradually. The paper is explicit about this limitation in §5.1:

"However, this analysis does not explain why downstream metrics are emergent or enable us to predict the scale at which emergence occurs."

This means the cross-entropy loss analysis rules out the simplest skeptical explanation (that emergence is entirely a measurement artifact) but does not provide a positive explanation for the phase transition. The underlying question—what changes at the critical scale threshold?—remains open.

Multi-Metric Analysis for Generative Tasks

Beyond cross-entropy loss, the paper addresses the metric-dependence concern through a second analysis in Appendix A.2: examining whether emergent generative tasks look emergent under all evaluation metrics, not just exact match. This analysis targets the concern that exact match is a particularly harsh metric (requiring character-for-character correctness on potentially long outputs) and that emergence might disappear when using metrics that award partial credit.

The experimental setup. The paper selects three generative BIG-Bench tasks that are classified as emergent (modified arithmetic, IPA transliteration, and periodic elements) and plots scaling curves using seven different evaluation metrics: exact match, ROUGE-1, ROUGE-2, ROUGE-L-Sum, BLEURT, BLEU, and sequence F1 score. Each of these metrics captures a different aspect of output quality, from strict correctness (exact match) to semantic similarity (BLEURT) to n-gram overlap (ROUGE, BLEU).

The finding. The paper states the result clearly:

"For all three tasks, emergent behavior appears to be independent of which evaluation metric is used. Hence, we conclude that using exact string match instead of another evaluation metric that awards partial credit is not a complete explanation of emergence on generative tasks."

The implication is that the phase transition is not merely about the model finally crossing a threshold where its outputs become exactly correct. Even metrics that award partial credit for being "close" show the same flat-then-jump pattern. This suggests that small models are not producing outputs that are "almost correct" or "partially correct" in ways that these softer metrics detect—they are producing outputs that are fundamentally wrong in structure or content, and the transition to producing structurally appropriate outputs is abrupt.

A nuance on interpretability. The paper explicitly notes that two tasks (word unscrambling and repeat copy logic) are excluded from this analysis because "exact match is the only most sensible evaluation metric for those tasks, which measure the ability to manipulate words in the input (and hence metrics like BLEU and ROUGE that give word-level partial credit are not valid)." This is a careful methodological note: for tasks where the correct output is a deterministic manipulation of the input (e.g., unscrambling letters), partial-credit metrics based on word overlap would be misleading because they might give credit for outputs that contain the right words but in the wrong order, which is fundamentally incorrect for the task.

BIG-Bench Task Taxonomy and Keyword Analysis

One of the paper's most empirically intensive contributions is the systematic classification of all 210 BIG-Bench tasks by their scaling behavior and the subsequent analysis of which task types are most likely to be emergent. This analysis is described primarily in Appendix A.3 and Appendix E.

The classification procedure. The paper describes the methodology:

"We manually classified all 210 BIG-Bench tasks as thus far emergent or not. We used the definition of emergence given in §3, which is that the task had near-random performance until a certain scale, after which performance increases to substantially above random (as opposed to smoothly increasing). Because this definition is potentially subjective based on the definition of 'near-random' (and any heuristic we decide on would encode these subjective biases), two co-authors of the paper worked together and agreed with confidence on all the tasks labeled as emergent."

Several aspects of this methodology are notable. First, the classification is based on human judgment of scaling curve shapes, not an automated statistical test. This is a practical necessity—the number of data points per task is small (often 5–10 model sizes), and the appropriate statistical test would depend on assumptions about the functional form of the scaling relationship—but it does introduce subjectivity. The paper mitigates this by having two annotators agree with confidence, though inter-annotator agreement metrics are not reported.

Second, the classification uses four categories, not just emergent/non-emergent:

Emergent with LaMDA or GPT-3: Tasks that first showed the phase-transition pattern when evaluated on LaMDA 137B or GPT-3 175B (the largest models available at the time those benchmarks were run). These are listed in Appendix E.2 (25 tasks).
Emergent with PaLM: Tasks that remained at near-random performance for all previous models (including LaMDA 137B and GPT-3 175B) but showed the phase transition when evaluated on PaLM models (8B, 62B, or 540B). These are listed in Appendix E.3 (41 tasks).
Smoothly increasing: Tasks where performance improved predictably with scale, without a sharp phase transition. These are listed in Appendix E.1 (59 tasks).
Flat (no model better than random): Tasks where even the largest models (including PaLM 540B) do not perform above random. These are listed in Appendix E.4 (43 tasks).
Other: Tasks that don't fit cleanly into any category—those with performance not correlated with scale, those where performance gets worse with scale, those with too few evaluation examples, etc. (Appendix E.5, ~42 tasks).

The third category (emergent with PaLM) is particularly important for the paper's argument in §5.2 that "model scale is not the singular factor for unlocking an emergent ability." PaLM 62B has fewer parameters and fewer training FLOPs than GPT-3 175B and LaMDA 137B—yet it achieves above-random performance on 41 BIG-Bench tasks where those larger models failed. This demonstrates that factors other than raw scale (the paper hypothesizes training data quality and architectural choices like split digit-encodings) influence the emergence threshold.

The keyword analysis (Figure 8). Each BIG-Bench task is tagged with keywords by the task authors (e.g., "logical reasoning," "multilingual," "visual reasoning"). The paper aggregates these tags to ask: which types of tasks are most likely to be emergent? This is visualized in Figure 8, which shows a stacked bar chart of the four scaling categories for each keyword.

The analysis reveals several patterns:

Keywords with the highest fraction of emergent tasks: analogical reasoning, word sense disambiguation, truthfulness, social reasoning, and emotional understanding. Only two of these top five are explicitly "reasoning" tasks, which the authors note as somewhat unexpected.
Keywords with unexpectedly low emergence rates: arithmetic and mathematics had relatively few emergent tasks, which the paper notes is "unexpected since some of the earliest examples of emergence were on arithmetic." This may be because many BIG-Bench arithmetic tasks are simple enough that even medium-sized models can perform them (making them "smoothly increasing" rather than "emergent") or because some are sufficiently multi-step that even the largest models cannot perform them (making them "flat").
Keywords with the most "flat" tasks (candidates for future emergence): visual reasoning (8/13 tasks flat—expected since language models lack visual input), non-language, repeated interaction, context length, computer code, and multi-step. The paper frames these as "directions for future work in emergence in large language models."

A methodological limitation. The paper acknowledges that tracking the absolute number of emergent tasks per keyword is problematic because "it effectively just captures the most common keywords used across BigBench"—some keywords are applied to many tasks simply because they are broad tags. The paper therefore focuses on fraction of tasks per keyword that are emergent, but this too has limitations: keywords with very few tasks (fewer than 5, which the paper excludes from analysis) would have unstable fractions. The overall conclusion is measured:

"Overall, there are no clear trends for which types of tasks are most emergent."

This is itself an important finding: emergence is not cleanly predicted by task category, which reinforces the paper's central claim that emergence is difficult to predict from task characteristics alone.

The complete task lists (Appendix E). The paper provides, in full detail, the classification of every BIG-Bench task. Appendix E.4 is particularly noteworthy: it lists 43 tasks where "no model better than random," including tasks like checkmate-in-one prediction, multi-step arithmetic, program synthesis, sudoku, and many others. These are explicitly framed as "potential candidates for future emergence"—tasks that might exhibit a phase transition if future models are scaled further or trained differently. This list serves as a concrete benchmark for tracking whether and when emergence occurs for currently-unsolved tasks.

The MMLU Category Analysis: Stratifying Emergence by Domain

The paper extends the taxonomy approach to the MMLU benchmark in Appendix B. MMLU consists of 57 topics spanning four supercategories (Humanities, STEM, Social Science, Other), and the paper analyzes whether emergence varies by category.

The approach. Performance of Chinchilla and Gopher models is plotted separately for each of the four categories, with all three x-axis variants (training FLOPs, model parameters, WikiText103 perplexity). Figure 10 shows four rows of three plots each (twelve plots total). Figure 9 provides a summary comparison: the performance of the largest model (70B for Chinchilla, 280B for Gopher) minus the performance of the second-largest model (7B for both), broken out by category.

The key finding. The paper states:

"It is clear that Social Science and Humanities have the largest jump in performance from the second-largest to the largest model, and STEM has the smallest jump in performance."

Specifically examining Figure 9: the second-largest models for both Chinchilla and Gopher perform near random (~25%) on all four categories. The largest models jump to roughly 62% (Chinchilla) and 75% (Gopher) on Humanities, roughly 70% and 85% on Social Science, but only roughly 38% and 50% on STEM—a substantially smaller jump. The paper interprets this as an "emergent" pattern (flat-then-jump) that is strongest for humanities and social science, and weakest for STEM.

Interpretation and caveats. The paper does not offer a definitive explanation for why STEM shows the weakest emergence, but several possibilities are consistent with the broader findings. STEM questions may require more multi-step reasoning or mathematical computation that even the largest models struggle with—consistent with the finding that arithmetic and mathematics tasks show relatively low emergence rates in BIG-Bench. Alternatively, STEM knowledge may be less densely represented in the training data, making it harder to acquire through scale alone. The finding that STEM shows the smallest jump, combined with BIG-Bench results showing that multi-step tasks are often "flat" (no model achieves better than random), suggests a coherent picture: tasks requiring precise symbolic manipulation or multi-step deduction resist emergence in a way that knowledge-intensive tasks (humanities, social science) do not.

WikiText103 Perplexity as an Alternative X-Axis: The Multi-View Analysis of Emergence

One of the paper's more subtle methodological contributions is the analysis in §5.3 and Figure 4 that examines emergence as a function of language modeling quality (WikiText103 perplexity) rather than raw scale (FLOPs or parameters). This is motivated by an important conceptual question: is emergence fundamentally about compute, or about how well the model has learned the statistical structure of language?

The motivation. The paper states:

"While scale (e.g., training FLOPs or model parameters) has been highly correlated with language model performance on many downstream metrics so far, scale need not be the only lens to view emergent abilities."

The intuition is straightforward: if two models have the same training FLOPs but one was trained more efficiently and achieves lower perplexity, the lower-perplexity model might show emergent abilities that the other does not. In this case, perplexity would be a better predictor of emergence than FLOPs.

The empirical relationship between scale and perplexity. Figure 4 (top row) shows that for Chinchilla and Gopher—the two model families with available WikiText103 perplexity data—training FLOPs, model parameters, and WikiText103 perplexity are highly correlated. The plots show a smooth monotonic relationship: more FLOPs → lower perplexity, more parameters → lower perplexity. This correlation means that for currently available models, the three potential x-axes are largely interchangeable in terms of the shape of the scaling curves they produce.

The MMLU analysis under all three axes. Figure 4 (bottom row) shows MMLU performance plotted against all three x-axes. Because of the high correlation, the curves look similar—they all show the emergent flat-then-jump pattern. The paper is careful to note that this equivalence may not hold in the future:

"Because WikiText103 perplexity and training FLOPs happen to be highly correlated for the models considered here (Gopher and Chinchilla), the plots of emergent abilities look similar for both. However, this correlation between WikiText103 perplexity and scale may not hold in the future as new techniques beyond vanilla dense Transformer models are developed."

The paper gives the specific example of retrieval-augmented models, which "may have strong WikiText103 perplexity with less training compute and fewer model parameters." In such a scenario, a retrieval-augmented model operating at a given perplexity might show emergent abilities that a dense model at the same perplexity does not (or vice versa), making the choice of x-axis consequential.

The deeper conceptual point. This analysis serves a broader argument: emergence should be understood as a function of many correlated variables rather than a single cause. The paper states:

"Overall, emergent abilities should probably be viewed as a function of many correlated variables."

This is a hedge against overinterpretation. It acknowledges that while training FLOPs serve as a convenient summary statistic for current models, they do not isolate a single causal mechanism. The real drivers of emergence could be any combination of model capacity (parameters), amount of training signal (data), quality of training signal (data curation), architectural inductive biases, or some emergent property of the interaction among these factors. Perplexity is offered as one additional lens, not as a replacement explanation.

A methodological limitation. Comparing across model families using WikiText103 perplexity is complicated because "differences in training data composition" can affect perplexity independently of model quality. A model trained primarily on Wikipedia-like text might have artificially low WikiText103 perplexity compared to a model trained on a broader data mixture, even if the latter is better on downstream tasks. The paper does not account for this directly but flags it as a consideration.

Treatment of Emergent Risks and Sociological Changes

The final two components of the paper's methodological approach extend the emergence concept beyond model capabilities to two additional domains: risks and sociological shifts. These are conceptually distinct from each other and from emergent abilities, and the paper treats them with different levels of analytical rigor.

Emergent risks (§5.4). The paper argues that the same unpredictability that makes it impossible to forecast beneficial capabilities from smaller models also applies to harmful behaviors. The methodology here is primarily literature review: summarizing prior findings on how risks scale, and flagging cases where the scaling behavior is discontinuous.

The paper makes a careful distinction:

"Such risks are important considerations whether or not they can be precisely characterized as 'emergent' based on the definition in §2, and, in some scenarios, do increase with model scale."

This is an important hedge. Unlike emergent abilities—where the paper provides systematic scaling curve evidence for phase transitions—the evidence for emergent risks is more anecdotal and less systematically documented. The paper notes several specific findings:

On WinoGender (measuring gender bias), scaling has improved performance so far—this is not emergent but smoothly improving.
On the BBQ bias benchmark, "bias can increase with scaling for ambiguous contexts"—the paper does not specify whether this increase is smooth or phase-transitional.
For toxicity (RealToxicityPrompts dataset), "larger language models could produce more toxic responses"—again, without specifying whether this is smooth or emergent.
For memorization, "larger models were found to be more likely to memorize training data"—this appears to be a continuous trend rather than a phase transition, but the paper does not rule out a threshold effect.
On TruthfulQA, "GPT-3 models were more likely to mimic human falsehoods as they got larger"—this is an inverse scaling trend, where performance gets worse with scale. The paper notes that Gopher (280B) later showed "emergent performance substantially better than random" on a multiple-choice version, suggesting a possible U-shaped curve (first worse, then better) rather than a simple phase transition.

The paper also enumerates potential future emergent risks that "might only exist in future language models or that have not yet been characterized in current language models," including "backdoor vulnerabilities, inadvertent deception, or harmful content synthesis." This is explicitly speculative and is framed as a motivation for proactive safety research rather than as an established finding.

Sociological changes (§5.5). The paper introduces a second, entirely different concept of "emergence" that refers to changes in how the NLP community views and uses language models, rather than changes in model behavior. The key claim:

"Another notable type of qualitative change is sociological, in which increasing scale has shifted how the community views and uses language models."

The paper traces a shift from task-specific models (the dominant paradigm in NLP through the 2010s) to general-purpose models that perform many tasks via prompting without task-specific training. This shift is described as "emergent" in a sociological sense—it represents a qualitative change in the field's practices that was enabled by quantitative increases in model scale.

The paper is careful to distinguish this from the technical definition of emergence:

"These abilities are not necessarily emergent since they have smooth, predictable scaling curves—however, they do underscore an emergent sociological shift towards general-purpose models in the NLP community."

The evidence for this sociological shift is the observation that few-shot prompted general-purpose models have begun to surpass task-specific state-of-the-art on various benchmarks (Figure 13 in Appendix D). GPT-3 175B achieved SOTA on TriviaQA and PiQA; PaLM 540B achieved SOTA on three arithmetic reasoning benchmarks; Flamingo 80B achieved SOTA on six visual QA benchmarks. In each case, a general model with no task-specific training outperformed models specifically designed and trained for that task. The scaling curves for these achievements are shown in Appendix D (Figure 13), and they typically show smooth improvement rather than phase transitions.

This sociological dimension is conceptually important for the paper's broader argument because it suggests that emergence—understood broadly as qualitative change arising from quantitative scaling—operates at multiple levels. There are emergent model abilities (the phase transitions in task performance), emergent risks (harms that appear or intensify unpredictably), and emergent sociological changes (shifts in research paradigms and real-world applications). The paper's contribution is not to claim these are the same phenomenon but to argue that scaling language models produces consequences at all three levels, and that the unpredictability of these consequences is a unifying theme.

Summary of Key Methodological Choices and Their Justifications

Training FLOPs as primary x-axis: Provides a unified metric that accounts for both model size and training duration, enabling comparison across model families with different parameter-count-to-data ratios. Supplemented with parameter-count and perplexity axes to acknowledge that no single metric captures all relevant aspects of scale.
Manual classification of scaling curves as emergent vs. non-emergent: Necessitated by the small number of data points per task (making automated curve-fitting unreliable) and the definitional requirement of assessing whether performance is "near-random" and then "substantially above random"—both qualitative judgments. Mitigated by dual-annotator agreement.
Cross-entropy loss analysis alongside downstream metrics: Addresses the most significant skeptical concern (measurement artifact) by testing whether models exhibit any improvement at all in the pre-emergence regime. The finding that loss improves smoothly while accuracy remains flat supports the paper's interpretation that emergence is real but provides an incomplete explanation of why the transition occurs.
Multi-metric analysis for generative tasks: Rules out the alternative hypothesis that emergence is solely an artifact of exact-match scoring, by showing that multiple metrics with different stringency levels all exhibit the flat-then-jump pattern.
Four-category taxonomy (emergent-with-LaMDA/GPT, emergent-with-PaLM, smooth, flat) rather than binary: Acknowledges that emergence thresholds shift with better training recipes and architectures, and provides a framework for tracking whether currently-flat tasks become emergent in future models.
WikiText103 perplexity as alternative x-axis: Explores whether emergence is better predicted by realized language modeling quality than by raw compute, while acknowledging the current high correlation among these axes limits the ability to distinguish between them.
Explicit separation of emergent abilities, emergent risks, and emergent sociological changes: Prevents conflation of distinct phenomena while maintaining the unifying theme that quantitative scaling produces qualitative changes at multiple levels of analysis.

4. Key Insights and Innovations

Innovation 1: The Phase Transition as a First-Class Object of Study in Machine Learning

The paper's most fundamental intellectual contribution is not any specific capability it documents but the act of naming and operationalizing a phenomenon that was hiding in plain sight across dozens of papers: language model abilities do not always improve smoothly with scale—they exhibit phase transitions. Prior work had individually observed that specific tasks showed sharp performance jumps (GPT-3 on arithmetic; BIG-Bench on various benchmarks), but each observation was treated as a curious finding within its own experimental context. There was no recognition that this was a category of behavior with shared characteristics, that it occurred across model families and task types, or that it demanded its own conceptual framework.

The innovation is analogous to the moment in physics when researchers stopped treating each phase transition (water freezing, iron magnetizing, helium becoming superfluid) as a separate curiosity and recognized them as instances of a general class of critical phenomena governed by shared principles. The paper performs this synthesis for language model scaling, pulling together evidence from GPT-3, LaMDA, Gopher, Chinchilla, and PaLM across BIG-Bench (200+ tasks), TruthfulQA, MMLU, WiC, and augmented prompting benchmarks to argue that the flat-then-jump pattern is not an anomaly but a defining feature of how certain capabilities manifest under scaling.

What makes this intellectually distinctive is the epistemological shift it demands. Before this paper, the dominant mental model for scaling—implicit in work like Kaplan et al. (2020) and Hoffmann et al. (2022)—was one of continuous improvement: train a bigger model, get proportionally better performance, and use the smooth scaling curve to extrapolate what even larger models will achieve. Emergent abilities break this model. If a task shows near-random performance across two orders of magnitude of scale, then jumps to well-above-random at a third, then the scaling behavior of the first two orders of magnitude gives zero information about the third. Extrapolation fails systematically. The paper's key move is to say: this is not an edge case or a measurement artifact—it is a distinct regime of scaling behavior that requires its own analytical toolkit.

The paper also introduces a crucial refinement by distinguishing emergent abilities (phase transitions in model behavior) from emergent sociological changes (§5.5)—where general-purpose models surpass task-specific state-of-the-art through smooth, predictable improvements. GPT-3 achieving SOTA on TriviaQA is not emergent under the paper's technical definition—the scaling curve is smooth—but it represents a qualitative shift in how the community practices NLP. Drawing this distinction prevents the informal use of "emergence" from watering down the specific, testable claim about phase transitions.

The broader significance extends beyond NLP. If phase transitions are a general property of scaling neural networks—not specific to autoregressive language models, particular architectures, or certain benchmarks—then this changes how machine learning researchers should think about evaluating and forecasting model capabilities across all domains. You cannot prototype on small models and confidently project to large ones. This has profound implications for research methodology, resource allocation, and safety evaluation that the paper begins to articulate but that extend far beyond its empirical scope.

Innovation 2: The Cross-Entropy Loss Analysis as a Diagnostic for "True" vs. "Measurement" Emergence

The paper's second major conceptual contribution is a diagnostic framework for distinguishing genuine capability emergence from artifacts of evaluation metrics. This is not merely a methodological footnote—it addresses the most obvious skeptical objection to the paper's central claim and, in doing so, produces a more nuanced understanding of what is actually emerging.

The skeptical concern is straightforward: exact-match accuracy, BLEU score, and multiple-choice accuracy are all harsh metrics that award zero partial credit. A model could transition from producing completely wrong outputs to producing largely correct but slightly imperfect outputs to producing exactly correct outputs, and the harsh metric would register only the final transition, not the intermediate improvement. Under this view, the apparent "emergence" is an illusion created by the evaluation function—the model was getting gradually better all along, but the metric was blind to it.

Prior work had largely ignored this concern or addressed it obliquely. The BIG-Bench paper (2022) noted the issue but did not perform a systematic analysis across tasks. The scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) focused on cross-entropy loss as the primary metric but did not connect loss improvements to downstream metric emergence.

The paper's cross-entropy loss analysis in Appendix A provides the first systematic test of the metric-artifact hypothesis across multiple emergent tasks. By plotting cross-entropy loss alongside downstream metrics for the same models on the same tasks, the analysis creates a direct comparison: if emergence is purely a measurement artifact, then loss should be flat over the same range where accuracy is flat; if loss improves continuously while accuracy jumps, then the true story is more complex. The finding—that loss improves smoothly across the entire scale range on all six emergent tasks examined—simultaneously rules out the simplest version of the measurement-artifact hypothesis and produces a more interesting result: the model is getting better in ways that are invisible to the downstream metric, but those improvements eventually cross a threshold where they become visible. What "emerges" is not the underlying capability (that was improving all along) but rather the operationalization of that capability into outputs that satisfy a stringent correctness criterion.

The additional log-probability analysis on classification tasks (Figure 6, bottom row) introduces a second diagnostic: examining whether the probability assigned to correct answers diverges from the probability assigned to incorrect answers. The finding that both correct and incorrect log-probabilities improve in tandem for small models—because larger models produce less extreme probabilities—reveals a subtler measurement effect. The model's internal representations are improving in a way that doesn't yet discriminate between correct and incorrect answers. Only at a critical scale do the probabilities diverge, and at that point accuracy jumps. This suggests that what "emerges" might be the model's ability to disambiguate between plausible alternatives rather than its ability to recognize the task structure.

The intellectual significance of this contribution lies in its transformation of the emergence question from "is it real or fake?" to "what exactly is emerging, and why does it shift from invisible to visible at a particular scale?" It provides an empirical methodology—pairing downstream metrics with continuous-probability analyses—that future work can apply to characterize emergence on new tasks and model families. And it introduces a conceptual distinction between competence (the model's internal representation of the task, reflected in loss) and performance (the model's ability to produce correct outputs, reflected in downstream metrics) that was latent in the scaling literature but never explicitly articulated in this way.

Innovation 3: The Symmetry Between Emergent Abilities and Emergent Risks

The paper makes a conceptual move that, while occupying relatively little page space (§5.4), has outsized implications for AI safety: explicitly positing that the unpredictability of emergent beneficial capabilities and the unpredictability of emergent harmful behaviors are manifestations of the same underlying phenomenon. This is not simply an observation that "bigger models can be more dangerous"—which was already known from work on bias amplification (Bender et al., 2021), toxicity (Gehman et al., 2020), and memorization (Carlini et al., 2021)—but a specific claim that these risks can exhibit the same phase-transition pattern as capabilities, making them invisible in smaller models and therefore impossible to anticipate through standard safety testing at sub-threshold scales.

Prior work on language model risks had largely treated them as continuous scaling problems: as models get larger, memorization increases (Carlini et al., 2021), toxic outputs become more plausible (Askell et al., 2021), and certain forms of bias are amplified. The dominant framework was one of monitoring trends and establishing acceptability thresholds. The paper's contribution is to suggest that this framework may be insufficient—that some risks may follow the same flat-then-jump pattern as capabilities, and that safety evaluations on current models cannot rule out the emergence of currently-unobserved harms in future, larger models.

The paper's evidence for this claim is more suggestive than definitive—it acknowledges this explicitly ("Such risks are important considerations whether or not they can be precisely characterized as 'emergent' based on the definition in §2")—but the symmetry argument is compelling on conceptual grounds. If emergence is a consequence of scale enabling new computational mechanisms that were not present in smaller models, there is no principled reason why those mechanisms would be limited to beneficial capabilities. The same representational capacity that enables a model to suddenly perform multi-step reasoning could also enable it to suddenly engage in more sophisticated forms of deception, manipulation, or harmful planning.

The TruthfulQA results in Figure 2E provide the clearest concrete illustration: GPT-3 models become more likely to mimic human falsehoods as they scale (an inverse scaling trend), and only when Gopher reaches 280B parameters does the model suddenly jump to performing substantially better than random at truthful answering. A safety evaluator looking at the GPT-3 trend would have seen increasing untruthfulness with scale—a concerning but predictable trajectory—and would not have anticipated that further scaling would eventually reverse this trend. This specific example is about a capability (truthfulness) rather than a risk (deception), but the structural point holds: the scaling curve was not predictable from smaller models.

The broader significance lies in how this reframes the relationship between AI capability research and AI safety research. In a world where capabilities improve predictably with scale, safety can be a parallel track—monitor trends, establish thresholds, intervene when metrics cross lines. In a world where capabilities emerge unpredictably, safety work must contend with the possibility that currently unobservable behaviors will appear at scales not yet reached, and that testing on today's models provides no guarantee about tomorrow's. The paper does not solve this problem—it explicitly lists it as an open question—but by naming it and connecting it to the documented phenomenon of emergent abilities, it shifts the burden of proof: the default assumption should no longer be that scaling trends are smooth and predictable.

This symmetry argument can be understood as a warning against extrapolation-based safety assurance—the claim that because a model of size X shows no evidence of harmful behavior Y, a model of size 10X will also lack Y. The paper's evidence on beneficial capabilities shows that this extrapolation fails routinely; the extension to risks is a logical argument rather than an empirical one, but it gains force from the demonstrated ubiquity of emergence in the capability domain. If dozens of beneficial capabilities appeared "out of nowhere" at specific scale thresholds, the absence of currently-observed harmful behaviors does not license confidence that no such behaviors will emerge at larger scales.

Innovation 4: The Multi-Factor View of Emergence as a Refutation of Scale Determinism

The paper's fourth conceptual contribution is a subtle but important one: demonstrating that the scale at which an ability emerges is not a fixed property of the task or the ability, but is contingent on model architecture, training data composition, training procedure, and possibly other factors. This might seem obvious in retrospect, but it directly challenges an implicit assumption that pervaded early discussions of emergence—that specific tasks "require" specific parameter counts or compute budgets.

The evidence for this comes primarily from the contrast between the "Emergent with LaMDA/GPT" and "Emergent with PaLM" categories in the BIG-Bench taxonomy (Appendix E.2–E.3). PaLM 62B—with fewer parameters and fewer training FLOPs than GPT-3 175B and LaMDA 137B—achieves above-random performance on 41 BIG-Bench tasks where those larger models failed. The paper enumerates these tasks explicitly in Appendix F, listing cases like anachronisms, ASCII word recognition, conceptual combinations, cryptonite, disambiguation QA, emoji movie, goal step wikihow, GRE reading comprehension, linguistics puzzles, logic grid puzzle, metaphor boolean, metaphor understanding, odd one out, and parsinlu QA.

This finding is significant because it breaks the intuitive causal chain: scale (FLOPs/parameters) → capability. If a 62B-parameter model can do tasks that a 175B-parameter model cannot, then raw scale is not the sole determinant. The paper hypothesizes that factors like "high-quality training data (e.g., more multilingual and code data than LaMDA) and architectural differences (e.g., split digit-encodings)" contribute to lowering the emergence threshold, but these are post-hoc explanations—the critical point is that the threshold can shift, and therefore the specific scales documented in the paper should not be reified as requirements.

Prior work had inadvertently encouraged a deterministic view of scale requirements. GPT-3's paper (Brown et al., 2020) presented a family of models trained on the same data with the same architecture, making scale the primary varying factor. The finding that certain capabilities only appeared at 175B naturally led readers to conclude that 175B was necessary. Similarly, the chain-of-thought prompting paper (Wei et al., 2022b) showed the technique only working above ~100B parameters, suggesting a parameter-count threshold. The present paper systematically undermines this interpretation by showing that the threshold depends—on architecture (encoder-decoder vs. decoder-only), on training data (PaLM's multilingual corpus), and on training procedure (Chinchilla-optimal vs. Gopher-style training).

The implications extend beyond the specific findings. If emergence thresholds are contingent rather than fixed, then the research program shifts from documenting at what scale abilities appear to understanding the causal factors that determine the threshold. This is a more ambitious and potentially more productive agenda: rather than simply scaling and hoping, researchers can investigate how data composition, architecture design, and training objectives shift the emergence threshold downward. The paper explicitly points to this in §5.2, noting that "once an ability is discovered, further research may make the ability available for smaller scale models," and citing Sanh et al. (2022) achieving instruction-following at 11B parameters (vs. Wei et al., 2022a's 68B threshold) through encoder-decoder architecture as an example.

This insight also has direct implications for the accessibility of emergence research. If emergence thresholds were fixed at hundreds of billions of parameters, only a handful of organizations could study emergent abilities. If thresholds can be lowered through better architecture and data—as the PaLM 62B evidence suggests—then the research community more broadly can investigate emergence at attainable scales. The paper's framing of this as an open question ("there is not an empirical study ablating every difference between PaLM 62B and prior models (the computational cost would be too high)") is honest about the limitations of the current evidence while pointing clearly toward the needed future work: controlled experiments that isolate which factors shift emergence thresholds, by how much, and why.

In essence, this innovation transforms emergence from a descriptive phenomenon (this is what happens when you scale) into an intervention target (here are levers that affect when and whether emergence occurs). This is a fundamental shift in how the phenomenon is conceptualized, with practical consequences for research prioritization.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper draws evidence from multiple benchmarks rather than a single dataset. The primary source is BIG-Bench (2022), a crowd-sourced suite of over 200 tasks spanning reasoning, knowledge, multilingual capabilities, and more. The paper also analyzes MMLU (Massive Multi-task Language Understanding; Hendrycks et al., 2021a), which aggregates 57 tests covering math, history, law, and other topics; TruthfulQA (Lin et al., 2021), which measures truthful answering on adversarially-constructed questions; the WiC (Word in Context) benchmark (Pilehvar & Camacho-Collados, 2019) for semantic understanding; GSM8K for math word problems (used in the chain-of-thought analysis); and grounded conceptual mappings (Patel & Pavlick, 2022). The specific evaluation splits are those used in the original papers: BIG-Bench uses its standard evaluation protocol, MMLU uses the 57-topic test set, TruthfulQA uses the multiple-choice or generative evaluation depending on the model family, and WiC uses its standard test set. The paper does not introduce new datasets or splits.
Base model(s). The paper analyzes six model families spanning multiple organizations and training paradigms: GPT-3 (Brown et al., 2020; 125M to 175B parameters), LaMDA (Thoppilan et al., 2022; 2.1M to 137B parameters), Gopher (Rae et al., 2021; 417M to 280B parameters), Chinchilla (Hoffmann et al., 2022; 417M to 70B parameters), PaLM (Chowdhery et al., 2022; 8B to 540B parameters), and Anthropic's models (Kadavath et al., 2022; 800M to 52B parameters). The full specifications are provided in Appendix C (Table 2). The selection is deliberately cross-organization and cross-architecture to establish that emergence is not an artifact of any particular training recipe. The total span covers roughly six orders of magnitude in training FLOPs (from ~3.3×10¹⁸ to ~2.5×10²⁴) and parameter count (from 2.1M to 540B). The paper states in Section 4 that these models are "representative of the capabilities of many contemporary LLMs," though this claim is not independently verified beyond the models analyzed.
Metrics. Performance is measured using task-specific downstream metrics: accuracy for classification tasks (e.g., MMLU, TruthfulQA multiple-choice, WiC), exact match for generative tasks where the output has a single correct form (e.g., modified arithmetic, word unscrambling), and BLEU for tasks where output quality is graded on similarity to a reference (e.g., IPA transliteration). The paper also employs cross-entropy loss as a secondary metric in the analysis of Appendix A, computed as the negative log-likelihood of the target sequence under the model's predicted distribution, to probe whether improvements in downstream metrics are masked by harsh evaluation criteria. For classification tasks, log-probabilities of correct and incorrect responses are separately tracked. The random baseline for each metric is the expected performance from uniform guessing: 25% for 4-way classification, 50% for binary, ~0% for generative exact match.
Baselines. The primary baseline for emergence classification is "random performance"—the expected score from a strategy that guesses uniformly among available options. Emergence is defined relative to this baseline: an ability is emergent if performance is near-random for smaller models and substantially above random for larger ones. For the chain-of-thought analysis (Figure 3A), the baseline is standard few-shot prompting without intermediate reasoning steps (Wei et al., 2022b). For instruction-following (Figure 3B), the baseline is the same model without instruction-finetuning (Wei et al., 2022a). For scratchpad training (Figure 3C), the baseline is direct prediction without intermediate outputs (Nye et al., 2021). For calibration (Figure 3D), the baseline is the standard method of measuring calibration using answer-choice probabilities rather than the True/False technique (Kadavath et al., 2022). The paper does not introduce new baselines; it compiles comparisons already present in the cited works.
Generation budget / compute accounting. The paper uses training FLOPs as the primary scale metric for the x-axis of scaling curves. Training FLOPs are computed using the standard approximation: approximately 6 × N × D, where N is the number of non-embedding model parameters and D is the number of training tokens (following Kaplan et al., 2020). The factor of 6 accounts for the forward pass (~2ND), backward pass (~2ND), and additional overhead. The specific FLOPs for each model are provided in Table 2 (Appendix C). For example, GPT-3 175B used 300B training tokens, yielding 3.14×10²³ FLOPs; PaLM 540B used 780B training tokens, yielding 2.53×10²⁴ FLOPs. The paper also provides scaling curves with model parameters on the x-axis (Figures 11–12, Appendix D) and with WikiText103 perplexity on the x-axis (Figure 4) to demonstrate that the choice of scale metric does not qualitatively change the emergence patterns for currently available models. All comparisons between methods at a given scale are made by matching training FLOPs or parameter counts—there are no generation-budget-matched comparisons of the sort found in inference-time compute optimization papers, because the paper's scope is purely about how capabilities vary as a function of pretraining scale, not inference-time computation.
Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional machine learning sense, because it is not training models or selecting hyperparameters. The classification of BIG-Bench tasks as emergent, smoothly increasing, or flat is done through manual annotation by two co-authors who "worked together and agreed with confidence on all the tasks labeled as emergent" (Appendix A.3). The paper acknowledges the subjectivity inherent in this procedure—"any heuristic we decide on would encode these subjective biases"—but does not report quantitative inter-annotator agreement metrics. For the scaling curves themselves, the paper plots all available data points without error bars or confidence intervals. The statistical robustness of the emergence classification is therefore based on the visual distinctiveness of the flat-then-jump pattern and the replication of this pattern across independent model families, rather than on formal statistical tests. The cross-entropy loss analysis in Appendix A serves as a form of robustness check—testing whether the emergence pattern holds when the evaluation metric is changed—rather than a statistical validation protocol.

Main Quantitative Results

Few-Shot Prompted Tasks (Section 3)

The paper presents eight exemplar emergent abilities in the few-shot prompting setting, summarized in Figure 2. The headline finding is that multiple benchmarks across diverse model families exhibit a consistent pattern: performance remains at near-random levels across one to two orders of magnitude of training FLOPs, then jumps sharply to substantially above random at a critical scale threshold.

Modified arithmetic (Figure 2A). On a BIG-Bench task testing 3-digit addition/subtraction and 2-digit multiplication, both GPT-3 and LaMDA show near-zero accuracy for several orders of magnitude of training compute. GPT-3's performance jumps to sharply above random at approximately 2×10²² training FLOPs (the 13B parameter model). LaMDA's jump occurs at approximately 10²³ training FLOPs (the 68B parameter model). The jump is from near 0% accuracy to roughly 30–45% (exact values must be read from the log-scale plot; the paper does not provide numerical tables). This is one of the most visually dramatic phase transitions in the paper—the curve is essentially flat at 0% across three orders of magnitude, then rises steeply.

IPA transliteration (Figure 2B). Transliterating from the International Phonetic Alphabet shows a similar pattern: LaMDA models from ~10¹⁸ to ~10²² FLOPs score near 0% BLEU; at ~10²³ FLOPs, performance jumps to roughly 40–50% BLEU. GPT-3 and PaLM also contribute data points consistent with this pattern, though with fewer intermediate scales plotted.

Word unscrambling (Figure 2C). Recovering a word from its scrambled letters shows near-0% exact match for LaMDA models up to ~10²² FLOPs, then a jump to roughly 30–40% exact match at ~10²³ FLOPs. GPT-3 and PaLM data points are consistent with the same threshold.

Persian question-answering (Figure 2D). This task is notable because it illustrates the interaction between scale and training data composition. LaMDA models through 137B parameters achieve near-0% exact match. PaLM 62B—despite having fewer parameters and training FLOPs than LaMDA 137B—achieves substantially above-random performance purely through its improved multilingual training data. The paper cites this as evidence that "both using PaLM's training dataset and scaling to 62B parameters is required for question-answering in Persian" (Section 5.6).

TruthfulQA (Figure 2E). GPT-3 models of all sizes (up to 175B parameters, 3.14×10²³ FLOPs) fail to exceed random performance on TruthfulQA—in fact, the paper notes they become more likely to mimic human falsehoods with scale. Gopher models also fail until the largest model (280B parameters, 5×10²³ FLOPs), which jumps to "more than 20% above random" (Section 3). The exact accuracy values must be read from Figure 2E, which shows Gopher 280B at roughly 55–65% accuracy against a roughly 25% random baseline.

Grounded conceptual mappings (Figure 2F). Patel & Pavlick (2022) evaluated GPT-3 models on mapping cardinal directions in a textual grid world. Performance jumps to above random only with the largest GPT-3 model (175B, 3.1×10²³ FLOPs). The paper does not provide numerical exact-match values beyond what is visible in the figure.

MMLU (Figure 2G). This is one of the paper's most extensively analyzed benchmarks. Models of ~10²² FLOPs (~10B parameters) or smaller perform no better than random guessing (25%) averaged across all 57 topics. GPT-3 175B, Gopher 280B, and Chinchilla 70B all achieve substantially above-random performance at 3–5×10²³ FLOPs. The paper highlights that this emergence occurs across three independently developed model families, providing strong evidence that the phenomenon is not family-specific. The cross-family consistency is particularly notable because MMLU covers a broad range of topics (math, history, law, medicine, etc.), suggesting that the ability to answer knowledge-intensive questions across many domains may require a critical threshold of scale—at least for dense models without retrieval augmentation (the paper explicitly includes this caveat).

WiC (Figure 2H). This benchmark provides the paper's historical case study in emergence. GPT-3 at 175B (3.14×10²³ FLOPs) fails to achieve one-shot performance above random on WiC. The GPT-3 paper (Brown et al., 2020) had suggested that this failure might indicate a fundamental limitation of autoregressive decoder-only models for this task, hypothesizing that a bidirectional architecture might be necessary. However, PaLM 540B (2.5×10²⁴ FLOPs)—also a decoder-only model—achieves above-random performance, demonstrating that "further scaling a decoder-only language model was actually enough" (Section 5). This example is used to illustrate the unpredictability of emergence: an apparent ceiling at one scale turned out to be an intermediate plateau that further scaling surpassed.

The paper also notes that additional emergent few-shot prompted tasks from BIG-Bench are listed in Appendix E, with dozens of examples spanning categories including logical deduction, metaphor understanding, causal judgment, and multilingual reasoning.

Augmented Prompting Strategies (Section 4)

The paper documents four cases where specialized prompting or finetuning techniques show no benefit (or even harm) at smaller scales but become effective above a critical scale threshold.

Chain-of-thought prompting for math word problems (Figure 3A). On the GSM8K benchmark, Wei et al. (2022b) compared standard few-shot prompting against chain-of-thought prompting (which includes intermediate reasoning steps). For LaMDA models, chain-of-thought provides no benefit—and may even perform slightly worse—at scales below approximately 10²² FLOPs. Above approximately 10²³ FLOPs (~68B parameters), chain-of-thought significantly outperforms standard prompting, reaching roughly 15–20% accuracy versus roughly 5–10% for the no-chain-of-thought baseline at the largest scale. The paper describes the transition as chain-of-thought only "surpassing" standard prompting at the critical threshold.

Instruction tuning (Figure 3B). Wei et al. (2022a) finetuned LaMDA models on a mixture of tasks phrased as instructions and evaluated zero-shot performance on unseen tasks. For models of ~7×10²¹ FLOPs (8B parameters) or smaller, instruction tuning hurts performance compared to the pretrained baseline. Only when scaled to ~10²³ FLOPs (~68B parameters, in the FLAN work) does instruction tuning improve the 10-task NLU average, reaching roughly 60–65% versus roughly 50% for the no-instruction-tuning baseline. The paper notes that this threshold was later lowered: Sanh et al. (2022) achieved instruction-following behavior with an 11B-parameter encoder-decoder T5 model, demonstrating that architecture choice influences the emergence threshold.

Scratchpad training for 8-digit addition (Figure 3C). Nye et al. (2021) finetuned LaMDA models to predict intermediate computation steps before giving the final answer. On 8-digit addition, scratchpad training only helps for models of approximately 9×10¹⁹ FLOPs (40M parameters) or larger. Below this threshold, the no-scratchpad and scratchpad methods perform similarly at near-zero accuracy. Above the threshold, the scratchpad method achieves roughly 80–100% accuracy while the no-scratchpad baseline remains near zero. This is notable as one of the earliest documented emergence thresholds—at 40M parameters, far smaller than most other examples in the paper—suggesting that the critical scale depends heavily on the specific capability being measured.

Calibration via P(True) (Figure 3D). Kadavath et al. (2022) compared two methods for measuring model calibration: the standard approach (using answer-choice probabilities) and the True/False technique (models propose answers, then evaluate the probability that their answer is correct). The superiority of the True/False technique—measured by expected calibration error (ECE)—only emerges at the largest Anthropic model scale of approximately 3×10²³ FLOPs (52B parameters). Figure 3D shows ECE on a log scale; the True/False method achieves substantially lower ECE (better calibration) than the standard method only at the largest scale, with the two methods performing similarly at smaller scales.

The paper also includes an extensive table (Table 1) listing additional emergent abilities documented in prior work, including self-consistency decoding (Wang et al., 2022b), least-to-most prompting (Zhou et al., 2022), zero-shot chain-of-thought (Kojima et al., 2022), multilingual chain-of-thought (Shi et al., 2022), and leveraging explanations in prompting (Lampinen et al., 2022). For each, the table lists the emergent scale in both training FLOPs and parameter count, along with the model family and citation. These are not analyzed in detail with individual scaling curves in the main text but are included to demonstrate the breadth of the phenomenon.

Summary of Emergence Thresholds Across Tasks and Model Families

Table 1 in the paper provides a comprehensive listing of emergent abilities and the scale at which they were first observed. The thresholds span an enormous range:

Lowest documented emergence threshold: Scratchpad training on 8-digit addition emerges at 8.9×10¹⁹ FLOPs (40M parameters, LaMDA), per Nye et al. (2021). This is approximately four orders of magnitude smaller than the largest thresholds.
Highest documented emergence threshold: Word in Context (WiC) emerges at 2.5×10²⁴ FLOPs (540B parameters, PaLM), per Chowdhery et al. (2022). This is the largest model analyzed in the paper.
Intermediate thresholds: Modified arithmetic emerges at 2.3×10²² FLOPs (13B, GPT-3). MMLU emerges at 3.1×10²³ FLOPs (175B, GPT-3) and at slightly different scales for Gopher and Chinchilla. Chain-of-thought for math emerges at 1.3×10²³ FLOPs (68B, LaMDA). Instruction tuning improves performance at 1.3×10²³ FLOPs (68B, FLAN/LaMDA).

The broad range of thresholds—spanning four orders of magnitude of training compute—reinforces the paper's argument that emergence is not a single phenomenon occurring at a fixed scale, but rather a general pattern that manifests at different scales for different capabilities. The paper does not attempt to predict from task characteristics at what scale emergence will occur; this is left as a central open question.

Cross-Entropy Loss Analysis of Emergent Tasks (Appendix A)

The paper's most important analytical contribution for understanding whether emergence is real or an artifact of measurement is the cross-entropy loss analysis on six BIG-Bench tasks classified as emergent for LaMDA.

Generative tasks (Figure 5). Three generative tasks are analyzed: modified arithmetic, IPA transliteration, and word unscrambling. For each task, two plots are shown side by side: error rate (1 - downstream metric) versus training FLOPs, and cross-entropy loss versus training FLOPs. The key result:

Error rate remains near 100% for models up to ~10²² FLOPs, then drops sharply at ~10²³ FLOPs (for LaMDA models). This is the emergent pattern on downstream metrics.
Cross-entropy loss shows a completely different pattern: it declines continuously across the entire scale range, from approximately 15 (at 10¹⁹–10²⁰ FLOPs) down to approximately 3 (at 10²³ FLOPs) for modified arithmetic. There is no phase transition or flat region—the loss improves monotonically.

The paper describes an "elbow" in the loss curve around the same scale where error rate jumps—suggesting a possible acceleration in improvement—but emphasizes that the improvement is substantial and continuous even in the pre-emergence regime. The implication is that the model's underlying understanding of the task (as measured by the probability it assigns to the correct sequence) is improving steadily, but these improvements do not translate into correct outputs until a critical threshold is crossed.

The paper also notes that the analysis includes both greedy decoding (temperature T=0) and random sampling (T=1), with similar patterns for both. The random baseline is explicitly plotted, confirming that small-model performance is indistinguishable from random.

Classification tasks (Figure 6). Three classification tasks are analyzed: logical arguments, sports understanding, and figure of speech detection. The paper provides three plots per task:

Error rate: Remains near-random for small models, then drops at the emergence threshold. This replicates the standard emergent pattern.
Cross-entropy loss: Improves continuously across the entire scale range, similar to the generative tasks. The paper notes that cross-entropy loss improves "even for small model scales where the downstream metrics are close to random."
Log-probability of correct and incorrect responses: This is the most detailed diagnostic. For all three tasks, the log-probability of both correct and incorrect responses improves (becomes less negative) as scale increases—even while accuracy remains near random. The paper explains this counterintuitive result:

"The reason is that larger models produce less-extreme probabilities (i.e., values approaching 0 or 1) and therefore the average log-probabilities have fewer extremely small values."

In other words, small models assign very low probabilities to many answer options (highly negative log-probs), but this affects both correct and incorrect answers roughly equally. As models scale up, their probability distributions become less extreme—probabilities are spread more evenly—which improves average log-probability for all answer choices. The discrimination between correct and incorrect answers (the divergence of the two log-probability curves) only occurs at a certain scale, and this divergence coincides with the accuracy jump. The paper notes: "for each of these three tasks, the average log-probability of the correct and incorrect responses eventually deviates at a certain scale, during which performance on the task increases substantially."

This finding is significant because it identifies what is emergent: not the raw pattern recognition or knowledge required for the task (which improves smoothly, as evidenced by continuously improving cross-entropy loss), but rather the ability to disambiguate between correct and incorrect options—to assign systematically higher probability to the right answer than to wrong ones.

What the cross-entropy loss analysis does and does not establish. The paper is explicit about the limitations of this analysis (Section 5.1, Appendix A). It establishes that emergence is not purely a measurement artifact: the flat downstream-metric curves are not simply hiding smoothly improving outputs that would be revealed by a more forgiving metric. Even metrics that award partial credit (ROUGE, BLEU, BLEURT) show the same flat-then-jump pattern on generative tasks (Appendix A.2, Figure 7). However, it does not explain why the downstream metrics eventually jump—why accumulated improvements in log-likelihood suddenly translate into correct outputs at a particular scale threshold. The paper states this limitation explicitly:

"This analysis does not explain why downstream metrics are emergent or enable us to predict the scale at which emergence occurs."

Multi-Metric Analysis for Generative Tasks (Appendix A.2)

The paper addresses the concern that exact-match scoring might be uniquely harsh by evaluating three emergent generative BIG-Bench tasks (modified arithmetic, IPA transliteration, periodic elements) under seven different evaluation metrics: exact match, ROUGE-1, ROUGE-2, ROUGE-L-Sum, BLEURT, BLEU, and sequence F1. Figure 7 shows the result: all metrics exhibit the emergent flat-then-jump pattern. The paper states:

"For all three tasks, emergent behavior appears to be independent of which evaluation metric is used."

This finding is important for interpreting the nature of emergence. If only exact match showed the phase transition while softer metrics showed smooth improvement, that would strongly suggest emergence is an artifact of metric harshness. The fact that even semantic-similarity metrics (BLEURT) and n-gram overlap metrics (ROUGE, BLEU) show the same sudden jump implies that small models are not producing outputs that are "almost correct" or "partially correct" in any metric's sense—they are producing fundamentally incorrect outputs, and the transition to producing appropriate outputs is sharp regardless of how "appropriate" is measured.

The paper notes that two tasks (word unscrambling and repeat copy logic) are excluded because partial-credit metrics are invalid for them: for tasks where the correct output must manipulate input words exactly, metrics that award credit for containing the right words in any order would give misleading signals.

BIG-Bench Task Taxonomy (Appendix A.3, Appendix E)

The paper classifies all 210 BIG-Bench tasks into five categories based on their scaling behavior. The distribution is shown in Figure 8 and detailed in full in Appendix E. The category counts are:

Smoothly increasing (59 tasks): Performance improves predictably with scale, without a sharp phase transition. Examples include cause and effect, date understanding, simple arithmetic, and unit conversion.
Emergent with LaMDA 137B or GPT-3 175B (25 tasks): These tasks showed the flat-then-jump pattern when evaluated on the largest models available at the time (pre-PaLM). Examples include modified arithmetic, IPA transliteration, word unscrambling, logical deduction, and figure of speech detection.
Emergent with PaLM models (41 tasks): These tasks remained at near-random for all prior models but showed above-random performance with PaLM (8B, 62B, or 540B). Examples include anachronisms, ASCII word recognition, conceptual combinations, cryptonite, logic grid puzzle, and metaphor understanding. The fact that PaLM 62B—smaller than GPT-3 175B in both parameters and training FLOPs—achieves above-random on many of these tasks is evidence that factors beyond raw scale (data quality, architecture) influence emergence thresholds.
Flat (43 tasks): No model, including PaLM 540B, achieves above-random performance. Examples include checkmate-in-one, program synthesis, multi-step arithmetic, sudoku, and twenty questions. These are "candidate tasks for future emergence."
Other (~42 tasks): Tasks that do not fit the above categories—those with performance not correlated with scale, those where performance gets worse with scale, tasks with too few evaluation examples, etc.

The keyword analysis (Figure 8). The paper aggregates task classifications by BIG-Bench keyword tags to investigate which types of tasks are most likely to be emergent. The key findings:

Keywords with the highest fraction of emergent tasks: analogical reasoning, word sense disambiguation, truthfulness, social reasoning, and emotional understanding. Notably, only two of these top five are explicitly "reasoning" keywords.
Arithmetic and mathematics have a relatively low fraction of emergent tasks, which the paper notes is "unexpected since some of the earliest examples of emergence were on arithmetic." This may reflect the fact that many BIG-Bench arithmetic tasks are simple enough that even medium-sized models perform above random (making them "smoothly increasing"), while some are sufficiently complex that even the largest models fail (making them "flat").
Keywords with the largest fraction of flat tasks (no model above random): visual reasoning (8/13 tasks—expected for text-only models), non-language, repeated interaction, context length, computer code, and multi-step. The paper frames these as directions for future work.
The paper concludes: "Overall, there are no clear trends for which types of tasks are most emergent." This is itself an important negative result: task category is not a reliable predictor of whether a capability will emerge with scale, reinforcing the paper's central argument about unpredictability.

MMLU Category Analysis (Appendix B)

The paper stratifies MMLU performance by the benchmark's four supercategories (Humanities, STEM, Social Science, Other) for both Chinchilla and Gopher models. The results are shown in Figure 10 (detailed scaling curves per category) and summarized in Figure 9 (comparison of performance jumps from second-largest to largest model).

The key finding:

"It is clear that Social Science and Humanities have the largest jump in performance from the second-largest to the largest model, and STEM has the smallest jump in performance."

Specifically (from Figure 9): For both Chinchilla and Gopher, the second-largest model (7B parameters) performs near random (~25%) on all four categories. The largest model (70B for Chinchilla, 280B for Gopher) jumps to:

Humanities: ~62% (Chinchilla), ~75% (Gopher)
Social Science: ~70% (Chinchilla), ~85% (Gopher)
Other: ~58% (Chinchilla), ~73% (Gopher)
STEM: ~38% (Chinchilla), ~50% (Gopher)

STEM shows the weakest emergence by a substantial margin: the jump from random to the largest model is roughly half the magnitude of the Humanities and Social Science jumps. The paper does not offer a definitive explanation but notes that this pattern is consistent with BIG-Bench findings that multi-step reasoning and mathematical tasks often remain flat or show weak emergence. The implication is that knowledge-intensive domains (humanities, social science) benefit more from scale than domains requiring precise symbolic manipulation or multi-step deduction (STEM).

WikiText103 Perplexity as an Alternative Scaling Axis (Section 5.3, Figure 4)

The paper explores whether emergence is better predicted by language modeling quality (WikiText103 perplexity) than by raw scale metrics (training FLOPs, parameter count). Figure 4 shows:

Top row: The relationships among training FLOPs, model parameters, and WikiText103 perplexity for Chinchilla and Gopher. All three are highly correlated: more FLOPs → lower perplexity; more parameters → lower perplexity. The relationship is smooth and monotonic.
Bottom row: MMLU performance plotted against all three x-axes. Because of the high correlation, all three plots show similar emergent patterns—the flat-then-jump is visible regardless of the x-axis.

The paper emphasizes that this equivalence may not hold for future model families:

"Because WikiText103 perplexity and training FLOPs happen to be highly correlated for the models considered here (Gopher and Chinchilla), the plots of emergent abilities look similar for both. However, this correlation between WikiText103 perplexity and scale may not hold in the future."

The paper gives retrieval-augmented models as an example where a model might achieve low WikiText103 perplexity with far less training compute than a dense model, potentially shifting the relationship between perplexity and other capabilities. The broader conceptual point is that "emergent abilities should probably be viewed as a function of many correlated variables" rather than pinned to any single metric.

Ablation Studies and Robustness Checks

The paper, being a survey rather than an experimental study with a novel method, does not contain ablation studies in the standard sense (varying components of a proposed system and measuring the impact). The closest analogues are the metric-variation analyses and the cross-family replication checks, which serve as robustness verification for the emergence classification.

Cross-entropy loss vs. downstream metrics (Appendix A.1): Tests whether the observed phase transitions in task performance are artifacts of harsh evaluation metrics by plotting cross-entropy loss—a continuous, partial-credit signal—alongside downstream metrics. Finding: cross-entropy loss improves continuously across all scales for all six emergent BIG-Bench tasks examined, even where downstream metrics remain at chance (Figure 5, Figure 6). This rules out the simplest measurement-artifact hypothesis but does not explain why the downstream jump occurs. The paper explicitly notes this limitation.

Multiple evaluation metrics on generative tasks (Appendix A.2): Tests whether emergence is specific to exact-match scoring by evaluating three emergent generative BIG-Bench tasks under seven metrics (exact match, ROUGE-1/2/L-Sum, BLEURT, BLEU, sequence F1). Finding: "emergent behavior appears to be independent of which evaluation metric is used" (Figure 7). All metrics show the flat-then-jump pattern. Two tasks (word unscrambling, repeat copy logic) are excluded because partial-credit metrics would be invalid for their task structure.

Log-probability decomposition for classification tasks (Appendix A.1.2): Tests whether emergence in classification reflects improved discrimination between correct and incorrect answers or simply reduced probability extremity. Finding: log-probability of both correct and incorrect answers improves in tandem for small models (because larger models assign less extreme probabilities overall); the divergence between correct and incorrect log-probabilities—indicating genuine discrimination—only occurs at the scale where accuracy jumps (Figure 6, bottom row).

Training FLOPs vs. parameter count as x-axis (Figure 2 vs. Figure 11; Figure 3 vs. Figure 12): Tests whether the choice of scale metric qualitatively affects the emergence classification. Finding: "using training FLOPs or model parameters as the x-axis produces curves with similar shapes due to the fact that most dense Transformer language model families have scaled training compute roughly proportionally with model parameters" (Section 2). The paired figures confirm visual similarity. The paper acknowledges that this equivalence may not hold for sparse mixture-of-experts models or Chinchilla-style compute-optimal training where parameter count and FLOPs diverge.

WikiText103 perplexity as alternative x-axis (Figure 4): Tests whether language modeling quality is a better predictor of emergence than raw compute. Finding: for Chinchilla and Gopher, the three x-axes (FLOPs, parameters, perplexity) are highly correlated and produce similar scaling curves. The paper notes this correlation is contingent and may not hold across future model families with different architectures or training recipes.

Cross-model-family replication (Figures 2, 3, 11, 12 throughout): Tests whether emergence is specific to a particular model family's architecture, training data, or procedure. Finding: the flat-then-jump pattern appears across GPT-3, LaMDA, Gopher, Chinchilla, PaLM, and Anthropic models for multiple tasks (e.g., MMLU in Figure 2G, modified arithmetic in Figure 2A). The paper argues this cross-family replication establishes emergence as "a recently discovered outcome of scaling up language models" rather than an artifact of any specific implementation (Section 6).

PaLM 62B vs. GPT-3 175B and LaMDA 137B (Section 5.2, Appendix F): Tests whether raw scale is the sole determinant of emergence by comparing a smaller model (PaLM 62B) against larger predecessors. Finding: PaLM 62B shows emergence on 41 BIG-Bench tasks where GPT-3 175B and LaMDA 137B performed at near-random levels, despite having fewer parameters and fewer training FLOPs. The paper lists these 41 tasks in Appendix F and attributes the difference to "high-quality training data (e.g., more multilingual and code data than LaMDA) and architectural differences (e.g., split digit-encodings)." This is a critical finding for the paper's argument that "model scale is not the singular factor for unlocking an emergent ability" (Section 5.2).

MMLU category stratification (Figure 9, Figure 10): Tests whether emergence on MMLU is driven by specific topic categories. Finding: Humanities and Social Science show the largest emergence effects (largest jump from the 7B second-largest model to the 70B/280B largest model); STEM shows the weakest emergence. This suggests domain-specific factors influence whether scale produces phase transitions in capability.

Temperature variation (Figure 5): For the generative BIG-Bench tasks in the cross-entropy loss analysis, both greedy decoding (T=0) and random sampling (T=1) are shown. Finding: both decoding strategies produce the emergent flat-then-jump pattern. The paper does not dwell on this but includes it as a methodological check that the result is not specific to a particular decoding approach.

Negative results—tasks where no current model exceeds random (Appendix E.4): The paper enumerates 43 BIG-Bench tasks where even PaLM 540B fails to achieve above-random performance, including checkmate-in-one, multi-step arithmetic, program synthesis, and sudoku. These are explicitly framed as "potential candidates for future emergence" rather than as tasks that are impossible for language models. This is a negative result in the sense that it shows the limits of current scaling, but the paper does not treat it as evidence that these tasks will never be solved—only that they have not yet emerged at currently available scales.

Negative results—tasks where performance gets worse with scale (Appendix E.5): The paper notes that a small number of BIG-Bench tasks show inverse scaling (performance decreases as models get larger), including BBQ bias, diverse social bias, and movie recommendation. These are classified in the "Other" category and are not analyzed in detail, but their existence is flagged as relevant to emergent risks.

Limitation acknowledged but not ablated: The paper does not perform a controlled experiment that isolates which specific factor (data quality, architecture, training objective) causes PaLM 62B to outperform larger models on 41 BIG-Bench tasks. The paper states: "there is not an empirical study ablating every difference between PaLM 62B and prior models (the computational cost would be too high)." This is an honest acknowledgment of a genuine experimental limitation—the cost of training multiple 62B+ parameter models with controlled variations would be prohibitive—but it means the causal attribution of PaLM's improved performance to specific factors (multilingual data, code data, split digit-encodings) is post-hoc hypothesis rather than experimentally established.

Critical Assessment

The paper's central claim is descriptive rather than causal: emergent abilities—phase transitions in task performance as a function of model scale—are a widespread phenomenon in large language models that occur across model families, task types, and experimental paradigms, and they cannot be predicted by extrapolating from smaller models. The experiments presented, drawn from prior work and synthesized here, provide substantial evidence for this descriptive claim, but several important qualifications and gaps warrant examination.

The "widespread" claim is well-supported within the scope of the analyzed benchmarks. The paper documents phase transitions in eight few-shot prompted tasks (Figure 2), four augmented prompting strategies (Figure 3), and dozens of BIG-Bench tasks (Appendix E), spanning five independent model families. The cross-family replication—particularly for MMLU (GPT-3, Gopher, Chinchilla—Figure 2G) and modified arithmetic (GPT-3, LaMDA—Figure 2A)—strengthens the case that emergence is not an artifact of a particular training recipe. The BIG-Bench taxonomy, while manually annotated and therefore subject to annotator bias, provides the most systematic cataloguing of emergence patterns available at the time, and the inclusion of 41 tasks that emerged only with PaLM (Appendix E.3) provides a natural experiment demonstrating that emergence thresholds shift with improved training. The claim that emergence is "widespread" is reasonably supported by this body of evidence.

However, the evidence base has important scope limitations that the paper does not fully address. All analyzed tasks are drawn from a relatively narrow range of NLP benchmarks: BIG-Bench (~200 crowd-sourced tasks), MMLU (multiple-choice knowledge), TruthfulQA (truthfulness), WiC (word sense), GSM8K (math word problems), and a handful of others. These benchmarks are heavily weighted toward knowledge-intensive, reasoning-oriented, English-language tasks with clear correctness criteria. The paper does not analyze whether emergence occurs in open-ended generation (story writing, dialogue), creative tasks, tasks requiring real-world interaction, or tasks with ambiguous or subjective evaluation. This is a genuine limitation: the documented phenomenon may be specific to tasks where "correctness" is well-defined and can be measured with a single metric, and the appearance of phase transitions may partly reflect the discontinuity of correctness evaluation rather than the discontinuity of capability. The cross-entropy loss analysis partially addresses this concern but does not test whether continuous-valued tasks (e.g., human preference ratings, quality scores) show similar phase transitions.

The "cannot be predicted by extrapolating from smaller models" claim is demonstrated by example but not tested as a forecasting hypothesis. The paper's primary method for demonstrating unpredictability is to show scaling curves where performance is flat for several orders of magnitude and then jumps. This establishes that a naive linear or log-linear extrapolation from the flat region would fail to predict the jump. However, the paper does not formalize "extrapolation" or test alternative forecasting methods. Could a forecaster who has access to cross-entropy loss trends (which improve smoothly in the pre-emergence regime) predict that a jump is imminent? Could they predict at what scale the jump will occur? The paper explicitly says the cross-entropy loss analysis "does not enable us to predict the scale at which emergence occurs," but it does not test whether any signal—from loss curves, from similar tasks, from training data statistics—would enable prediction. The claim that emergence "cannot be predicted" is therefore stronger than the evidence directly supports: the paper demonstrates that a particular extrapolation method (naive projection from smaller-scale downstream metrics) fails, not that prediction is impossible in principle. This is a nuance that the paper acknowledges in passing—"Emergent abilities would not have been directly predicted by extrapolating a scaling law... from small-scale models" (Section 2)—but the distinction between "would not have been" (past subjunctive, describing what happened) and "cannot be" (present indicative, describing a necessary property) is sometimes blurred in the discussion.

The test-set sizes for per-task emergence classification are small and unstated. For the BIG-Bench tasks classified as emergent, the paper does not report the number of evaluation examples per task. BIG-Bench tasks vary widely in size—some have hundreds of examples, others have fewer than 50. For small test sets, the observed phase transition could be partly a consequence of statistical noise: with only 32 examples (as the paper notes for the "logical arguments" task in Figure 6), a model could jump from 25% to 50% accuracy (the difference between random and above-random) on the basis of getting a handful of additional questions right, and this jump could appear sharp on a scaling curve simply because the resolution of the accuracy measurement is coarse. The paper acknowledges this concern for one task ("Logical arguments only has 32 samples, which may contribute to noise") but does not systematically report test-set sizes or analyze how emergence classification depends on them. A task with a statistically small jump from random might be classified as emergent on a small test set but as smoothly improving on a larger one.

The "random" baseline is well-motivated but may not be the appropriate comparison for all tasks. The paper defines emergence relative to the expected performance of uniform random guessing. For tasks where the output space is large and unstructured (e.g., generative tasks with exact-match scoring), "random" performance is essentially 0%, and any consistent above-zero performance is evidence of nontrivial capability. However, for some classification tasks, the random baseline may underestimate the difficulty of the task. If a task has imbalanced classes or if the correct answer distribution is highly skewed, a model could achieve above-random performance through simple heuristics (e.g., always predicting the majority class) without possessing genuine task understanding. The paper does not analyze whether the observed above-random performance at the emergence threshold reflects genuine competence or superficial pattern matching. This is a broader limitation of benchmarking rather than specific to this paper, but it affects the interpretation of emergence: if a model jumps from 25% to 35% accuracy on a 4-way classification task, is this an "emergent ability" or simply the model learning to exploit a dataset bias that was not exploitable at smaller scales?

The paper does not disentangle pretraining data memorization from genuine capability acquisition. For knowledge-intensive tasks like MMLU and TruthfulQA, it is possible that performance improvements at larger scales reflect increased memorization of training data that overlaps with evaluation data, rather than the emergence of genuine reasoning or knowledge retrieval abilities. The paper notes that "larger models were found to be more likely to memorize training data" (Section 5.4, citing Carlini et al., 2021), but does not analyze whether the emergent tasks show evidence of memorization-driven performance. If a model's performance on MMLU jumps from 25% to 60% at a particular scale primarily because it has memorized more of the MMLU questions (or near-duplicates) from its training data, this would still be "emergence" under the paper's operational definition—the ability "is not present in smaller models but is present in larger models"—but it would be a very different kind of emergence than the acquisition of generalizable reasoning capabilities. The paper's definition deliberately avoids this distinction, but it matters for claims about what emergence implies about future models and for the connection between emergent abilities and emergent risks.

The cross-entropy loss analysis is performed on only six tasks from a single model family. The most methodologically important analysis in the paper—the comparison of downstream metrics with cross-entropy loss to probe whether emergence is a measurement artifact—is limited to six BIG-Bench tasks evaluated on LaMDA models (Appendix A). This is a small fraction of the dozens of tasks the paper classifies as emergent (66 total between the LaMDA/GPT and PaLM categories), and it does not include any augmented-prompting-strategy tasks (chain-of-thought, instruction tuning, scratchpad, calibration). The analysis also does not extend to non-LaMDA model families. It is possible that the continuous-improvement-in-loss pattern does not hold uniformly across all emergent tasks—some tasks might show genuine phase transitions even in cross-entropy loss, which would suggest a qualitatively different kind of emergence. Without broader coverage, the paper's claim that "for all six BIG-Bench tasks... cross-entropy loss does actually improve" must be interpreted as evidence that some emergent tasks show this property, not that all do.

The paper does not test whether emergence thresholds are stable under re-evaluation. The emergence classification for BIG-Bench tasks relies on a single evaluation run per model per task. Given that language model outputs are stochastic (depending on sampling temperature, prompt format, and even hardware-level nondeterminism in some implementations), a task classified as "flat" at a given scale might achieve above-random performance on a different evaluation run or with a different prompt. The paper does not report evaluation variance or perform multiple evaluation runs to establish the stability of the classification. For the few-shot prompted tasks, this is particularly relevant: different choices of few-shot exemplars can produce substantially different performance on the same model, and the paper does not control for or report exemplar sensitivity.

The MMLU "emergence" story is complicated by the Chinchilla vs. Gopher comparison. In Figure 2G and Figure 4, Chinchilla 70B and Gopher 280B both achieve above-random MMLU performance, but Chinchilla does so with one-fourth the parameters. This is consistent with the paper's narrative that training-data scaling (Chinchilla was trained on more tokens) can shift emergence thresholds. However, it also complicates the x-axis choice: if Chinchilla 70B emerges at roughly the same training FLOPs as Gopher 280B (~5×10²³), then the emergence threshold in FLOP terms is consistent, and the parameter-count difference reflects Chinchilla's superior efficiency. If instead Chinchilla emerges at a lower FLOP threshold, that would suggest training-data quality or quantity directly affects the emergence threshold. The paper does not resolve this ambiguity because the Chinchilla and Gopher MMLU data points are at similar FLOP ranges, making it difficult to determine whether FLOPs or parameters (or data quantity) is the better predictor.

Ablations that would strengthen the paper but were not performed (and are acknowledged as computationally prohibitive): The paper identifies but does not resolve the question of which specific factors cause PaLM 62B to outperform larger models on 41 BIG-Bench tasks. A controlled experiment training multiple model variants at the 62B scale—varying data composition (with/without multilingual data, with/without code), architecture (standard vs. split digit-encoding), and training objective—could isolate causal factors, but the paper correctly notes that the computational cost is prohibitive. This is a genuine limitation of emergence research: the experiments that would be most informative are also the most expensive, creating a structural barrier to causal understanding.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Emergence Framework

The assumption or constraint. The paper's central analytical move—classifying tasks as emergent by examining scaling curves across multiple model sizes—requires evaluating each model size on each task. For the BIG-Bench taxonomy in Appendix A.3, this means running every LaMDA, GPT-3, and PaLM model size (spanning six orders of magnitude) on 200+ tasks. The paper treats this evaluation cost as external to the analysis—it is a survey paper cataloguing results already computed by prior work—but the cost has direct implications for anyone attempting to apply the emergence framework prospectively. The paper acknowledges in §2 that "the scale at which an ability is first observed to emerge depends on a number of factors and is not an immutable property," and in §5.2 that "as the science of training large language models progresses, certain abilities may be unlocked for smaller models with new architectures, higher-quality data, or improved training procedures." However, it does not discuss the practical cost of determining whether a given task is emergent for a given model family: you must train (or at minimum evaluate) models across a wide enough scale range to observe the flat region and the jump. Without that range, you cannot distinguish an emergent task from a smoothly-scaling one, because both look identical if you only observe the post-emergence region or only the pre-emergence flat region.

The consequence. A practitioner or research team asking "is task X emergent for our model family?" faces a chicken-and-egg problem. To answer the question, they must evaluate models across multiple scales—potentially up to the largest scale they can afford—but the point of wanting to know whether a task is emergent is to decide whether scaling further is worth the investment. If the task is emergent, small-model results are uninformative and the only way to discover capability is to scale. If the task scales smoothly, small-model results can justify further investment via extrapolation. But distinguishing these cases requires the very scaling experiments whose value is in question. The paper provides no diagnostic short of full multi-scale evaluation that can predict whether a task will be emergent. The cross-entropy loss analysis (§5.1, Appendix A) shows that loss improves continuously even when downstream metrics are flat, but it does not provide a forecasting method—you still need multiple model sizes to observe whether the loss improvement will eventually cross the threshold where downstream metrics jump, and the paper explicitly states that "this analysis does not explain why downstream metrics are emergent or enable us to predict the scale at which emergence occurs."

What evidence exists in the paper. The paper does not measure or discuss evaluation cost directly. It is implicitly present in the scope of the data collection: Table 2 lists 30 distinct model sizes across six model families, and the BIG-Bench taxonomy in Appendix E classifies 210 tasks, implying thousands of model-task evaluations. The paper treats this as a given—it surveys results that were computed by the original model developers for their own purposes—but the cost of replication or extension is unstated. The difficulty-estimation cost is not accounted for in any budget or efficiency metric because the paper proposes no method or system; it is a descriptive survey. However, a reader trying to operationalize the paper's framework for a new model family would discover this cost immediately.

Mitigation status. Not addressed. The paper does not propose lightweight proxies for emergence classification (e.g., using cross-entropy loss trends at a single scale to predict whether downstream metrics will jump at larger scales; using training data statistics; using architectural properties). The cross-entropy loss analysis in Appendix A demonstrates that loss and downstream metrics diverge, but it does not convert this observation into a forecasting tool. §5.6 lists "understanding emergence" as a direction for future work, including the goal of predicting "what abilities future models may have," but no concrete forecasting method is developed or evaluated. This limitation is fundamental to the paper's framing: the central claim is that emergence is unpredictable from smaller models, so any method that successfully predicted emergence would paradoxically undermine the paper's thesis. The paper resolves this tension by leaving prediction as an open problem.

All Evidence Comes from a Narrow Slice of Task Types: English-Language Benchmarks with Well-Defined Correctness Criteria

The assumption or constraint. Every emergent ability documented in the paper is measured on a benchmark where "correctness" is crisply defined and evaluated with a single scalar metric: multiple-choice accuracy (MMLU, TruthfulQA, WiC), exact-match string comparison (modified arithmetic, word unscrambling, Persian QA), BLEU score (IPA transliteration), or BLEURT/ROUGE (Appendix A.2). The paper does not analyze tasks where output quality is multidimensional, subjective, or evaluated by human judgment—open-ended dialogue, creative writing, summarization quality beyond n-gram overlap, code generation assessed by functional correctness, or tasks requiring real-world interaction. This scope limitation is acknowledged implicitly in the task selection (all tasks are from standard NLP benchmarks) but never discussed as a potential confound for the generality of the emergence phenomenon.

The consequence. The paper's central claim—that emergence is a "widespread" and "recently discovered outcome of scaling up language models" (§6)—may not generalize beyond tasks with thresholded evaluation metrics. In fact, the very definition of emergence (performance is near-random until a critical scale, then jumps substantially above random) presupposes a metric that produces near-binary outcomes at the instance level. If each model output is scored on a continuous scale (e.g., human-rated quality from 1–5, or BLEURT semantic similarity from 0–1), then average performance would naturally appear continuous and smooth even if the underlying capability improved discontinuously—the metric itself smooths out the threshold. The paper's multi-metric analysis in Appendix A.2 partially addresses this by showing that even continuous metrics like BLEURT exhibit the flat-then-jump pattern on generative BIG-Bench tasks (Figure 7). However, these tasks still have well-defined correct outputs (a specific arithmetic answer, a specific IPA string)—the continuous metrics are measuring proximity to a single ground-truth target. This is fundamentally different from tasks where there is no single "correct" output and quality is inherently multidimensional (e.g., "write a compelling story," "have a natural conversation," "generate helpful code with good style"). The paper provides no evidence about whether emergence occurs in such settings, and the definition of emergence may not cleanly apply to them.

What evidence exists in the paper. The task selection is explicit throughout the paper. Figure 2 documents eight emergent abilities: modified arithmetic (exact match), IPA transliteration (BLEU), word unscramble (exact match), Persian QA (exact match), TruthfulQA (multiple-choice accuracy), grounded mappings (accuracy), MMLU (multiple-choice accuracy), WiC (accuracy). Figure 3 documents four augmented prompting abilities: math word problems (accuracy), instruction following (NLU task average accuracy), 8-digit addition (accuracy), calibration (ECE). The BIG-Bench taxonomy in Appendix E covers 210 tasks, all with defined evaluation metrics. The paper does not discuss the absence of open-ended tasks or argue that the emergence pattern would generalize to them. The Appendix A.2 multi-metric analysis (Figure 7) shows that the emergence pattern is robust to metric choice for tasks with well-defined targets, but this is not evidence that the pattern holds for tasks without well-defined targets.

Mitigation status. Not addressed. The paper does not acknowledge the task-type scope limitation or discuss its implications. §5.6 lists "frontier tasks" as a direction for future work, including "prompting in multiple modalities" and "multilingual emergence," but does not mention open-ended generation or subjectively-evaluated tasks as a gap. The broader impact statement (§6) does not discuss whether the documented emergence patterns are specific to benchmark-style evaluation. This limitation is not hidden—it is visible from the task selection—but the paper's claims about the "widespread" nature of emergence implicitly generalize beyond the evidence without flagging the scope restriction.

The Paper Provides No Causal Explanation for Why Emergence Occurs, Limiting Its Utility for Engineering Decisions

The assumption or constraint. The paper is explicitly descriptive rather than explanatory. Section 5.1 ("Potential explanations of emergence") lists hypotheses—multi-step reasoning requiring minimum depth, evaluation metrics masking incremental improvements, threshold frequencies in training data—but frames them as speculations rather than tested explanations. The paper states:

"Although there are dozens of examples of emergent abilities, there are currently few compelling explanations for why such abilities emerge in the way they do."

And later, after the cross-entropy loss analysis:

"This analysis does not explain why downstream metrics are emergent or enable us to predict the scale at which emergence occurs."

The paper does not attempt to establish causal relationships between specific model properties (depth, width, training data distribution, optimization dynamics) and the presence or absence of phase transitions. It documents that emergence happens but not why.

The consequence. For a practitioner deciding how to allocate resources, the descriptive nature of the paper creates an unsatisfying decision framework. The paper tells you that some capabilities will appear unpredictably at larger scales, but it gives you no principled way to determine which capabilities, at what scale, or whether an alternative intervention (more data, different architecture, better training recipe) would unlock a desired capability more efficiently than simply scaling parameters and compute. The PaLM 62B vs. GPT-3 175B comparison in §5.2 demonstrates that factors beyond raw scale matter—PaLM 62B shows emergent abilities on 41 BIG-Bench tasks where larger models failed—but the specific factors (multilingual data, code data, split digit-encodings) are identified post-hoc and not experimentally isolated. Without causal understanding, the only actionable recommendation the paper can make is the brute-force one: "further scaling up models has so far appeared to increase the capabilities of language models, and is a straightforward direction for future work" (§5.6). This is unsatisfying for resource-constrained settings where "just scale more" is not feasible, and it provides no guidance on how to allocate a fixed budget across competing interventions (parameters vs. data vs. architecture vs. training duration).

What evidence exists in the paper. The paper's descriptive nature is evident throughout. The scaling curves in Figures 2–4, 11–12 are purely correlational: they show that larger models perform better on emergent tasks, but they do not test why. The cross-entropy loss analysis (Appendix A) demonstrates that loss improves continuously while accuracy remains flat, but this is a characterization of the phenomenon, not an explanation—it tells us what is happening (improvements in log-likelihood accumulate but don't manifest in downstream metrics until a threshold) but not why the threshold exists or what determines its location. The keyword analysis (Appendix A.3, Figure 8) attempts to identify task characteristics that correlate with emergence, but the conclusion is that "there are no clear trends for which types of tasks are most emergent"—a negative result that reinforces the lack of predictive understanding. The PaLM 62B finding (§5.2, Appendix F) is perhaps the strongest evidence that causal factors beyond raw scale exist, but the paper cannot disentangle them.

Mitigation status. Partially addressed by explicitly framing causal explanation as future work. Section 5.6 lists "understanding emergence" as a primary research direction and suggests specific approaches: "analyze the relationship between emergent tasks and similar data in training; create a synthetic task that requires multiple compositional sub-tasks and evaluate how each of those sub-tasks improve with scale and unlock emergence when combined." The paper also acknowledges the computational barrier to causal experiments: "there is not an empirical study ablating every difference between PaLM 62B and prior models (the computational cost would be too high)" (§5.2). This is an honest acknowledgment of a genuine experimental limitation, but it means the paper's practical guidance is limited to documenting the phenomenon and speculating about its causes.

The Operational Definition of Emergence Is Sensitive to the Choice of Evaluation Metric and Statistical Threshold, and the Paper Does Not Formalize Either

The assumption or constraint. The paper defines emergence as performance that is "near-random until a certain critical threshold of scale is reached, after which performance increases to substantially above random" (§2). This definition relies on two qualitative judgments: what counts as "near-random" and what counts as "substantially above random." The paper does not operationalize either judgment with a formal statistical criterion. For BIG-Bench taxonomy construction (Appendix A.3), the classification is done by two co-authors who "agreed with confidence on all the tasks labeled as emergent," with the acknowledgment that "this definition is potentially subjective based on the definition of 'near-random' (and any heuristic we decide on would encode these subjective biases)." No inter-annotator agreement metrics, decision boundaries, or sensitivity analyses are reported.

The consequence. The subjectivity of the emergence classification has two practical consequences. First, it means that different annotators—or the same annotators applying slightly different implicit thresholds—could classify borderline tasks differently. A task where the largest model achieves 28% accuracy on a 4-way classification task (4% above random) might be classified as "still near-random" by one annotator and "beginning to emerge" by another. The paper's category boundaries (emergent with LaMDA/GPT, emergent with PaLM, smoothly increasing, flat) are treated as crisp, but the evidence for many individual classifications is a visual judgment of a scaling curve with often fewer than 10 data points. Second, the sensitivity to metric choice—even within the set of "hard" metrics—means that the classification could change if a different evaluation protocol were used. For example, the same model outputs on MMLU could yield different "accuracy" depending on how multiple-choice scoring handles ties, how prompts are formatted, or how many few-shot exemplars are used. The paper does not test whether emergence classifications are robust to such variations.

What evidence exists in the paper. The paper explicitly acknowledges the subjectivity in Appendix A.3. The multi-metric analysis (Appendix A.2, Figure 7) partially addresses metric sensitivity by showing that the emergence pattern holds across seven metrics for three tasks—but this analysis is limited to three generative tasks and does not test whether the classification of a task as emergent vs. smooth would be stable under metric variation. For classification metrics, the paper does not test whether slightly different accuracy thresholds (e.g., "above random" defined as >30% vs. >35% on a 4-way task) would reclassify tasks. The log-probability analysis for classification tasks (Appendix A.1.2) shows that the divergence between correct and incorrect log-probabilities is gradual, which implies that the exact point at which accuracy "jumps above random" depends on the interaction between this gradual divergence and the discrete threshold of the accuracy metric.

Mitigation status. Partially addressed through dual-annotator agreement, but the mitigation is incomplete. The paper reports that the annotators "agreed with confidence" but does not quantify agreement (e.g., Cohen's kappa, percentage agreement, or number of initially-disputed cases). The annotated task lists in Appendix E provide full transparency—readers can inspect individual classifications—but without a formal criterion, independent replication of the taxonomy would rely on the replicators' subjective judgment matching the original annotators'. The paper does not discuss sensitivity to the "near-random" vs. "substantially above random" thresholds or provide alternative taxonomies under different thresholds. This is a genuine methodological limitation: the paper's central contribution is a taxonomy of emergent phenomena, but the taxonomy's construction method is not fully reproducible.

The Paper Does Not Disentangle Emergence Driven by Scale from Emergence Driven by Training Data Overlap Between Pretraining and Evaluation Sets

The assumption or constraint. The paper treats emergence as a property of model scale: larger models, trained on more compute, exhibit capabilities that smaller models do not. However, the paper does not control for or analyze the possibility that performance improvements on evaluation benchmarks reflect increased memorization of training data that overlaps with those benchmarks, rather than the acquisition of generalizable capabilities. This is particularly relevant for knowledge-intensive benchmarks like MMLU (57 topics spanning math, history, law, medicine), TruthfulQA (adversarially constructed against GPT-3), and BIG-Bench tasks that probe factual knowledge. The paper notes in §5.4 that "larger models were found to be more likely to memorize training data" (citing Carlini et al., 2021; 2022), but it does not analyze whether the documented emergent abilities are contaminated by memorization. The paper refers to memorization only in the context of emergent risks, not as a potential confound for the interpretation of emergent abilities.

The consequence. If a model's MMLU performance jumps from 25% to 60% at a particular scale because a larger fraction of MMLU questions (or near-duplicates) appeared in the training data—and the larger model has greater capacity to memorize and reproduce those answers—then the "emergence" is fundamentally different from the acquisition of generalizable reasoning or knowledge retrieval. It would still satisfy the paper's operational definition of emergence ("not present in smaller models but present in larger models"), but it would imply very different things about what future models will be capable of. A memorization-driven jump might plateau once all memorizable evaluation instances are saturated, whereas a generalization-driven jump might continue to improve with scale on genuinely novel instances. The distinction matters for both capability forecasting and safety: memorization-driven emergence on a benchmark tells you little about real-world performance on novel inputs, and it could create a false sense of progress that does not transfer to deployment. Several BIG-Bench tasks listed as emergent (e.g., Hindu knowledge, Swahili-English proverbs, Persian QA—Appendix E.2–E.3) are explicitly knowledge-recall tasks where training-data overlap is a plausible performance driver.

What evidence exists in the paper. The paper provides no analysis of training data overlap for any benchmark. The training data composition for each model family is described only in general terms (e.g., PaLM "includes more multilingual and code data than LaMDA"—§5.2). The paper does not attempt to measure the extent to which MMLU, TruthfulQA, or BIG-Bench evaluation instances appear in the pretraining corpora of the analyzed models, nor does it test whether performance on evaluation instances that are likely to be memorized differs from performance on instances that are likely to be novel. The cross-entropy loss analysis in Appendix A could in principle speak to this question—if performance improvements were purely driven by memorization, one might expect the loss on specific evaluation instances to drop sharply when those instances are memorized, rather than improving continuously—but the paper does not use the loss analysis to probe memorization effects.

Mitigation status. Not addressed. The paper does not discuss memorization as a potential confound for emergence classification, despite flagging memorization as an emergent risk in §5.4. The training data for the analyzed models is not publicly available in most cases (GPT-3, PaLM, LaMDA, Gopher training data is proprietary), making independent analysis of data overlap infeasible—but the paper does not acknowledge this barrier or discuss how it affects the interpretation of results. Future work on decontamination studies and memorization-aware evaluation (e.g., comparing performance on canary strings vs. held-out data) is not mentioned as a direction in §5.6. This is a significant gap because it means the paper cannot distinguish between two very different mechanisms—memorization and generalization—that produce observationally equivalent scaling curves.

The Paper Treats Scale as a Monolithic Variable, But the Analyzed Models Confound Scale with Architecture, Data, and Training Recipe Changes

The assumption or constraint. The paper's primary analytical lens treats "scale" as the independent variable and "performance" as the dependent variable, with the emergence classification based on the shape of the resulting curve. However, across the six model families analyzed, increasing scale is systematically confounded with changes in:

Training data composition: Larger models are often trained on larger, more diverse datasets. PaLM's training data included more multilingual and code data than LaMDA's (§5.2). The paper acknowledges this but cannot isolate whether PaLM's emergent abilities on 41 BIG-Bench tasks (Appendix E.3) are due to increased scale or improved data composition.
Training duration relative to model size: Chinchilla was trained on ~1.4T tokens with 70B parameters (a compute-optimal regime per Hoffmann et al., 2022), while Gopher was trained on ~300B tokens with 280B parameters (undertrained relative to Chinchilla-optimal). A Chinchilla-style 70B model and a Gopher-style 70B model would have very different training FLOPs, yet the paper's framework plots both as points on a FLOPs curve without distinguishing these training paradigms.
Architectural details: GPT-3, LaMDA, Gopher, Chinchilla, and PaLM all use decoder-only Transformer architectures, but they differ in details like activation functions, normalization schemes, position encodings, and vocabulary construction. The paper notes that PaLM uses "split digit-encodings" as one potential explanation for its improved performance (§5.2), but this is not controlled.
Inference-time protocol: Few-shot prompting format, number of exemplars, and decoding strategy (temperature, nucleus sampling parameters) vary across the evaluations compiled from different papers. The paper does not standardize these and does not analyze their interaction with emergence.

The consequence. The paper's scaling curves aggregate data points that differ along multiple dimensions, making it impossible to attribute the observed phase transitions to scale per se rather than to correlated improvements in training recipes. A reader might conclude from Figure 2 that training FLOPs is the key variable driving emergence, but the same data is consistent with a world where improved training data composition at larger scales is the true driver, and training FLOPs merely correlates with data diversity because organizations invest more in data curation for larger models. This matters for practical decision-making: if data quality is the primary lever, then investing in better data for a medium-scale model might unlock emergent abilities more cost-effectively than scaling parameters. The paper provides suggestive evidence for this through the PaLM 62B comparison (§5.2, Appendix F), but it cannot quantify the relative importance of scale vs. data vs. architecture because these factors are not experimentally separated.

What evidence exists in the paper. The confound is visible throughout. Table 2 shows that model families differ substantially in training tokens per parameter: GPT-3 175B used 300B tokens (~1.7K tokens/parameter); Chinchilla 70B used ~1.4T tokens (~20K tokens/parameter). These models appear at different positions on the FLOPs axis primarily because of different parameter counts, but their training paradigms are radically different. The paper acknowledges the multiple-confounds problem in §2:

"Training dataset size is also an important factor, but we do not plot capabilities against it because many language model families use a fixed number of training examples for all model sizes."

And in §5.3, discussing the WikiText103 perplexity analysis:

"Also note that using WikiText103 perplexity to compare across model families can be complicated due to factors such as differences in training data composition."

However, these acknowledgments are presented as caveats rather than as limitations that affect the core claims. The paper does not systematically discuss the implications of the confound for the interpretation of emergence as a scale-driven phenomenon.

Mitigation status. Partially acknowledged but not resolved. The paper addresses one dimension of the confound—parameter count vs. FLOPs as the x-axis—by providing both sets of plots (Figures 2 vs. 11, Figures 3 vs. 12) and noting their similarity. It addresses the data composition dimension by comparing PaLM 62B against larger models and by including the WikiText103 perplexity analysis as an alternative to raw scale metrics. However, these are correlational adjustments, not causal decompositions. The paper does not propose or perform the controlled experiments that would be needed to separate the effects of scale, data, and architecture—and it acknowledges that such experiments are computationally prohibitive (§5.2). The consequence is that "emergence" as documented in this paper is an empirical regularity about a particular historical trajectory of language model development (models got bigger and training data improved and architectures were refined), not a demonstrated causal law about the effects of scale alone. The paper's framing sometimes blurs this distinction, as when §6 concludes that "emergent abilities are a recently discovered outcome of scaling up language models"—implying a causal relationship—while the evidence supports only a correlational one.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper performs a conceptual synthesis that transforms how the field understands the relationship between scale and capability in language models. Before this work, the dominant narrative—anchored by scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022)—was one of continuous, predictable improvement: train a bigger model on more data, and performance on downstream tasks improves smoothly in ways that can be extrapolated from smaller models. The paper does not refute this narrative for aggregate metrics like perplexity, but it demonstrates that it fails systematically for the downstream capabilities that people actually care about. The phase-transition pattern—performance near random across multiple orders of magnitude of scale, then a sharp jump to substantially above random—is not an edge case observed in one or two tasks. The paper documents it across five model families, dozens of BIG-Bench tasks, four augmented prompting strategies, and multiple evaluation paradigms. The implication is that smooth perplexity improvements and discontinuous capability emergence coexist as distinct scaling regimes, and the field's reliance on perplexity-based scaling laws as a proxy for capability forecasting was missing a major part of the picture.

The magnitude of this shift is best characterized as a reframing of the scaling problem, not a paradigm shift that invalidates prior work. The scaling laws literature remains correct about cross-entropy loss—the paper's own cross-entropy loss analysis (Appendix A) confirms that loss improves continuously, exactly as Kaplan et al. and Hoffmann et al. would predict. What changes is the interpretation of what those loss improvements mean for downstream utility. The paper provides the diagnostic framework—pairing downstream metrics with cross-entropy loss, log-probability decomposition, and multi-metric analysis—that allows the field to distinguish between "the model is getting better in ways that don't yet produce correct outputs" and "the model is not getting better at all." This framework is now available for any researcher studying a new task or model family.

The paper also resolves a latent tension in the literature that had not been explicitly articulated. The BIG-Bench collaboration (2022) had observed that some tasks showed sharp performance jumps while others scaled smoothly, but this observation was presented as one finding among many in a 200+ task benchmark paper. Individual works (Brown et al., 2020 on arithmetic; Wei et al., 2022b on chain-of-thought; Nye et al., 2021 on scratchpads) had documented emergent behavior in their specific contexts but did not connect these observations into a general phenomenon. The present paper performs that connection, establishing that emergence is not a quirk of specific tasks, models, or evaluation protocols—it is a robust property of how capabilities manifest under scaling. This synthesis retroactively makes sense of apparently contradictory findings. For example, the observation that self-correction via prompting fails for reasoning tasks (Huang et al., 2023) and the observation that chain-of-thought prompting succeeds (Wei et al., 2022b) can both be true because they operate at different effective difficulty levels relative to the base model's capabilities. The emergence framework provides a vocabulary and conceptual structure for understanding when and why certain techniques work, without requiring them to work universally.

In terms of research prioritization, this paper makes several directions more attractive and several less so:

More attractive. Understanding why emergence occurs and whether it can be predicted becomes a first-class research problem rather than a philosophical curiosity. The paper's cross-entropy loss analysis shows that loss improves continuously even when downstream metrics are flat, which localizes the mystery: the question is not "why do models suddenly learn the task?" but rather "why do continuous improvements in the model's internal probability distribution suddenly translate into correct outputs at a specific scale?" Answering this question requires new analytical tools—possibly from the physics of phase transitions, the theory of neural network representations, or the circuit-level interpretability of Transformers—and the paper's documentation of the phenomenon provides the empirical foundation for that investigation. Similarly, the finding that PaLM 62B achieves emergent performance on 41 BIG-Bench tasks where GPT-3 175B and LaMDA 137B failed (§5.2, Appendix F) makes the question of which factors lower the emergence threshold a central practical concern. Controlled experiments varying training data composition, architecture, and training objective at fixed scale—while computationally expensive—become a high-priority research investment rather than an academic exercise, because they could reveal how to achieve emergent capabilities without brute-force scaling.

Also more attractive is the study of emergent risks as a phenomenon symmetric to emergent abilities. The paper's argument in §5.4—that the same unpredictability that characterizes beneficial capabilities also applies to harmful behaviors—reframes AI safety from a monitoring-and-mitigation discipline into one that must contend with fundamental uncertainty about what larger models will be capable of. This does not make alignment work less important; it makes it more urgent and more difficult, because safety evaluations on current models cannot guarantee the absence of harms in future models. The paper provides a concrete agenda for empirical safety research: systematically characterize scaling curves for risk-related metrics (bias, toxicity, deception, situational awareness) across model scales, identify which risks show emergent patterns, and develop early-warning indicators analogous to the cross-entropy loss analysis for capabilities.

Less attractive. The paper effectively refutes the strategy of prototyping techniques on small models and confidently extrapolating to large ones. For tasks that exhibit emergence, small-model experiments provide literally zero signal about whether a technique will work at scale—this is the paper's central empirical claim, not a speculative concern. Researchers and organizations that have built their workflow around iterative development on affordable model sizes must either restrict themselves to tasks with smooth scaling (which the paper provides no reliable method to identify ex ante) or accept that some experiments require large-scale execution. This has direct economic implications: the cost of discovering new capabilities is higher than previously assumed, because sub-threshold experiments are uninformative. The paper does not overstate this—it explicitly notes that some tasks do scale smoothly and that the threshold can sometimes be lowered through better data and architecture—but it establishes that assuming smooth scalability is an unreliable default.

The paper also complicates the narrative that scaling is a reliable path to increasingly general intelligence. If capabilities emerge unpredictably, then the research program of "scale and observe" becomes more of a gamble than an engineering roadmap. Some capabilities that the field currently considers out of reach might emerge at the next order of magnitude of compute; others might remain flat indefinitely. The 43 BIG-Bench tasks where no current model exceeds random performance (Appendix E.4)—including checkmate-in-one, program synthesis, and multi-step arithmetic—are explicitly framed as candidates for future emergence, not tasks proven impossible. But the paper provides no way to predict which of these will eventually yield to scale and which will not. This uncertainty is uncomfortable for long-term planning, and the paper's contribution is to make that discomfort explicit and well-documented rather than to resolve it.

Follow-Up Research This Work Enables

Controlled ablation of factors that shift the emergence threshold. The paper's most suggestive but least experimentally grounded finding is that PaLM 62B outperforms GPT-3 175B and LaMDA 137B on 41 BIG-Bench tasks despite having fewer parameters and fewer training FLOPs (§5.2, Appendix F). The paper hypothesizes that training data composition (more multilingual and code data) and architectural choices (split digit-encodings) are responsible, but no controlled experiment isolates these factors. A strong follow-up would train a set of models at fixed scale (~10B–70B parameters) with controlled variations: (a) standard English-heavy pretraining data vs. data augmented with code; (b) standard English-heavy data vs. data augmented with multilingual text; (c) standard architecture vs. architectural modifications like split digit-encodings. The dependent variable would be the emergence threshold on a battery of BIG-Bench tasks known to be emergent for PaLM but not for earlier models. If a specific factor (e.g., code data) systematically lowers the emergence threshold across multiple tasks, this would transform emergence from a descriptive phenomenon into an intervention target. The computational cost—training multiple models at the 10B+ scale—is substantial but tractable for well-resourced labs, and the scientific payoff would be high.

Emergence prediction from pretraining loss trajectories. The paper's cross-entropy loss analysis (Appendix A) demonstrates that loss improves continuously on emergent tasks even when downstream metrics remain flat, but it explicitly states that "this analysis does not explain why downstream metrics are emergent or enable us to predict the scale at which emergence occurs" (§5.1). A natural follow-up asks: given the pretraining loss curve and the downstream-task loss curve at sub-emergence scales, can we predict whether a task will be emergent and at approximately what scale? This would involve fitting functional forms to the loss trajectory (e.g., power laws with estimated exponents), measuring how quickly the gap between the model's predicted distribution and the correct target distribution is closing, and testing whether tasks where this gap closes slowly relative to the harshness of the evaluation metric are more likely to show emergent patterns. A concrete experiment: take the six emergent BIG-Bench tasks from Appendix A plus a matched set of six smoothly-scaling tasks, extract their loss trajectories at sub-emergence scales, and train a classifier to predict emergence status from loss trajectory features alone. If features like the rate of loss improvement, the divergence rate between correct and incorrect log-probabilities, or the elbow point in the loss curve predict emergence, this would provide a practical diagnostic tool that does not require evaluating at the emergence scale itself.

Synthetic tasks with controlled compositional structure to isolate the mechanism of emergence. The paper speculates that emergence may be related to the compositional nature of certain tasks: "if a multi-step reasoning task requires l steps of sequential computation, this might require a model with a depth of at least O(l) layers" (§5.1). This hypothesis is testable with synthetic data. Construct a family of tasks where the number of required compositional steps is systematically varied (e.g., multi-hop reasoning over synthetic knowledge graphs with 1, 2, 3, ..., k hops; or arithmetic with 1-digit, 2-digit, ..., n-digit operands). Train models of varying depth and width on these tasks, controlling for total parameters and training tokens, and measure whether the emergence threshold (in terms of model depth) scales linearly with the required number of compositional steps, as the hypothesis would predict. If the relationship holds, it provides a mechanistic explanation for at least one class of emergent abilities and a predictive framework: for a new task with known compositional depth, the required model scale can be estimated. If it does not hold, the compositional-depth hypothesis is ruled out and alternative mechanisms (training data statistics, optimization dynamics, representation learning phase transitions) become more plausible.

Systematic characterization of emergent risks with the same diagnostic toolkit applied to capabilities. The paper's symmetry argument in §5.4—that emergent risks may exhibit the same phase-transition pattern as emergent abilities—is currently more conceptual than empirical. A direct follow-up would apply the paper's diagnostic framework to risk-related metrics. For a range of model scales within a single family, measure: (a) toxicity generation rate on RealToxicityPrompts (Gehman et al., 2020); (b) bias scores on BBQ (Parrish et al., 2022) across disambiguated and ambiguous contexts; (c) memorization rates using the exposure metric from Carlini et al. (2021); (d) TruthfulQA accuracy and truthfulness scores; (e) calibration error on knowledge-intensive tasks. For each metric, construct scaling curves with both the standard risk metric and an underlying continuous signal (e.g., for toxicity, the probability assigned to toxic continuations rather than a binary toxic/safe classification; for bias, the raw model probabilities for stereotypical vs. anti-stereotypical associations rather than accuracy on a bias benchmark). The key question: do any risk metrics show the flat-then-jump pattern, where small models appear safe by a binary metric but the continuous signal reveals steady deterioration that crosses a threshold at larger scales? This would provide the first direct evidence for emergent risks as a phenomenon parallel to emergent abilities, rather than an extrapolation. It would also identify which risks are candidates for future emergence in models larger than currently available.

Inference-time compute scaling as a window into emergence mechanisms. The paper focuses exclusively on how capabilities vary with pretraining scale, but recent work on inference-time compute scaling (e.g., best-of-N sampling, chain-of-thought, verifier-guided search) provides a complementary lens. A strong follow-up would ask: for a fixed model size that is below the emergence threshold on a given task, can allocating additional inference-time compute (through more samples, longer reasoning chains, or search) unlock performance that is otherwise only observed above the pretraining threshold? If yes, this suggests that the sub-threshold model possesses the relevant knowledge or pattern-recognition capability (as evidenced by improving cross-entropy loss in the paper's analysis) but lacks the ability to operationalize it into correct outputs under standard decoding—and inference-time compute bridges the gap. This would directly test the paper's implicit competence-vs.-performance distinction and provide a practical method for extracting emergent-like capabilities from smaller models. If no—if inference-time compute cannot substitute for pretraining scale on emergent tasks—this would suggest that the phase transition reflects a genuine representational capacity threshold that cannot be bypassed at inference time, strengthening the case for scale as a necessary condition.

Cross-domain replication of the emergence pattern. The paper's evidence is entirely from English-language text-based benchmarks evaluated on decoder-only Transformer language models. Whether emergence is a general property of scaling neural networks—or specific to autoregressive language modeling on text—is an open question. Strong follow-up studies would replicate the emergence analysis in: (a) multilingual settings, testing whether the same tasks show emergent behavior in non-English languages and whether the emergence threshold depends on the amount of training data in that language—this is partially anticipated by the Persian QA result in Figure 2D but needs systematic coverage; (b) code generation models, testing whether benchmarks like HumanEval or MBPP show phase transitions in functional correctness as a function of model scale, and whether code-specific training data shifts the threshold; (c) vision-language models, testing whether multimodal capabilities (e.g., visual question answering, image captioning) emerge at specific scales and whether the emergence threshold for visual reasoning tasks is higher than for text-only reasoning, as the BIG-Bench finding that visual reasoning tasks are disproportionately "flat" (8/13 tasks in Appendix E.4) would suggest; (d) architectures beyond dense Transformers, including mixture-of-experts models (where parameter count and FLOPs diverge) and retrieval-augmented models (where knowledge access does not require parameter storage), to test whether the phase-transition pattern is architecture-dependent. Each of these replications would either extend the scope of the emergence phenomenon or identify boundary conditions where it does not hold—both outcomes advance understanding.

Practical Applications and Downstream Use Cases

Safety evaluation protocols for frontier model development. The paper's central finding—that the absence of a capability (or risk) in current models does not license confidence in its continued absence at larger scales—has direct operational implications for organizations developing increasingly large language models. A concrete safety application: before training a model at a new scale (e.g., 10× the training compute of the current largest model), evaluate the current largest model on a comprehensive battery of risk-related metrics, compute the cross-entropy loss or continuous-probability version of each metric (following the paper's diagnostic approach in Appendix A), and track whether the continuous signal is improving in ways that might foreshadow a downstream threshold crossing. If the continuous toxicity probability or the confidence-weighted bias score or the log-probability of memorized training sequences is steadily increasing even though the binary metric remains acceptable, this is an early warning that the next scale jump may produce a qualitative change in behavior. The paper's framework does not eliminate the unpredictability of emergence, but it provides a structured way to identify which risks are candidates for phase transitions, enabling more informed decisions about whether to proceed with scaling, what monitoring to put in place, and what mitigation strategies to prepare.

Resource allocation for research teams deciding between scaling and alternative investments. The paper provides language for a decision framework that was previously absent. For a team with a fixed compute budget, the question "should we train a larger model or invest in better data/architecture for our current model scale?" maps onto the paper's distinction between scaling-driven emergence (where larger models unlock qualitatively new capabilities) and threshold-shifting interventions (where improved data or architecture enables the same capability at smaller scale). The PaLM 62B vs. GPT-3 175B comparison in §5.2 and Appendix F provides the key empirical precedent: investments in training data quality and diversity may lower emergence thresholds more cost-effectively than scaling parameters, at least for knowledge-intensive tasks. A practical workflow: (1) characterize which desired capabilities are currently flat vs. smoothly improving vs. already emergent at current scale; (2) for flat capabilities, examine whether the cross-entropy loss is improving (if so, the model is making progress invisible to the downstream metric, and either further scaling or inference-time techniques may eventually unlock the capability); (3) benchmark against models like PaLM that achieved emergence at smaller scale through data improvements, to estimate whether data investment could shift the threshold into an achievable range. The paper does not provide a formula for this decision—the causal factors remain poorly understood—but it provides the conceptual categories that make the decision discussable in principled terms rather than as pure speculation.

Benchmark design and capability forecasting for the research community. The paper's taxonomy of BIG-Bench tasks into emergent, smooth, and flat categories (Appendix E) serves as a template for how benchmark suites should be constructed and interpreted. Current benchmark development often treats all tasks as equally informative about model progress; the paper demonstrates that tasks differ systematically in how they respond to scale. A practical implication: benchmark suites intended to track progress toward general capabilities should explicitly include tasks across the emergence spectrum—some that already show smooth scaling (to measure incremental progress), some that are emergent at current scales (to detect phase transitions), and some that are flat (to serve as canaries for future emergence). The 43 flat BIG-Bench tasks in Appendix E.4—including checkmate-in-one, program synthesis, multi-step arithmetic, and sudoku—provide a concrete list of such canary tasks. If a future model suddenly achieves above-random performance on several of these, that would be strong evidence of a qualitatively new capability regime, even if aggregate benchmark scores improve only modestly. The paper's framework also implies that benchmarking should report not just final performance but performance across multiple model scales within a family when possible, to distinguish emergent from smooth scaling—a practice that is rare in current benchmark papers but would substantially improve their informativeness for capability forecasting.

Communication with policymakers and the public about AI progress. The paper's distinction between smooth scaling and emergent phase transitions provides a vocabulary for explaining why AI capabilities can seem to "suddenly" appear despite years of incremental progress. For policymakers and non-technical stakeholders who may hear about language models passing the bar exam or solving competition math problems and wonder whether these are surprising breakthroughs or predictable milestones, the emergence framework offers an explanation: these capabilities were likely improving continuously in ways invisible to standard benchmarks, and scaling crossed a threshold where the improvement became visible. This is not to say the progress was predictable—the paper's central claim is that it was not, at least from downstream metrics alone—but it provides a mechanistic story that is more nuanced than either "AI is suddenly intelligent" or "nothing has changed, the benchmarks just got saturated." The paper's documentation of emergent risks (§5.4) serves a parallel communication function: it provides a scientifically grounded argument for why safety testing on current models is insufficient to guarantee the safety of future models, without resorting to speculative claims about superintelligence. The concrete examples—TruthfulQA accuracy jumping from below random to ~20% above random at a specific scale (Figure 2E), PaLM 540B achieving above-random WiC performance where GPT-3 175B had failed (Figure 2H)—make the abstract concept of emergence tangible for non-experts.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: The Predictability Gap in Language Model Scaling

Why This Problem Is Important

Where Prior Work Falls Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Definition of Emergence and Its Operationalization

Scaling Curve Methodology and the Choice of X-Axis

The Cross-Entropy Loss Analysis: Is Emergence Real or an Artifact of Metrics?

Multi-Metric Analysis for Generative Tasks

BIG-Bench Task Taxonomy and Keyword Analysis

The MMLU Category Analysis: Stratifying Emergence by Domain

WikiText103 Perplexity as an Alternative X-Axis: The Multi-View Analysis of Emergence

Treatment of Emergent Risks and Sociological Changes

Summary of Key Methodological Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: The Phase Transition as a First-Class Object of Study in Machine Learning

Innovation 2: The Cross-Entropy Loss Analysis as a Diagnostic for "True" vs. "Measurement" Emergence

Innovation 3: The Symmetry Between Emergent Abilities and Emergent Risks

Innovation 4: The Multi-Factor View of Emergence as a Refutation of Scale Determinism

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Few-Shot Prompted Tasks (Section 3)

Augmented Prompting Strategies (Section 4)

Summary of Emergence Thresholds Across Tasks and Model Families

Cross-Entropy Loss Analysis of Emergent Tasks (Appendix A)

Multi-Metric Analysis for Generative Tasks (Appendix A.2)

BIG-Bench Task Taxonomy (Appendix A.3, Appendix E)

MMLU Category Analysis (Appendix B)

WikiText103 Perplexity as an Alternative Scaling Axis (Section 5.3, Figure 4)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Emergence Framework

All Evidence Comes from a Narrow Slice of Task Types: English-Language Benchmarks with Well-Defined Correctness Criteria

The Paper Provides No Causal Explanation for Why Emergence Occurs, Limiting Its Utility for Engineering Decisions

The Operational Definition of Emergence Is Sensitive to the Choice of Evaluation Metric and Statistical Threshold, and the Paper Does Not Formalize Either

The Paper Does Not Disentangle Emergence Driven by Scale from Emergence Driven by Training Data Overlap Between Pretraining and Evaluation Sets

The Paper Treats Scale as a Monolithic Variable, But the Analyzed Models Confound Scale with Architecture, Data, and Training Recipe Changes

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

BIG-bench: Beyond the Imitation Game Benchmark

OPT: Open Pre-trained Transformer Language Models