Scaling Laws for Autoregressive Generative Modeling

ArXiv: 2010.14701

🎯 Pitch

The same power-law compute scaling that governs language models also dictates how well Transformers learn pixels, video frames, and math problems — and across all these domains, the optimal model size grows as a nearly universal power of the compute budget. This universality not only signals a deep commonality in how Transformers extract structure but also suggests we can quantitatively forecast, for example, exactly how many parameters are needed to make a near-perfect image model at any resolution.

1. Executive Summary

This paper identifies empirical scaling laws for the cross-entropy loss across four generative modeling domains—images, video, multimodal image↔text, and mathematical problem solving—using autoregressive Transformer models. In all domains, performance follows a power-law plus constant form, $L(x) = L_\infty + (x_0/x)^{\alpha_x}$ , where the irreducible loss $L_\infty$ is interpreted as the entropy of the true data distribution and the reducible loss as the KL divergence between the true and model distributions. The paper demonstrates that the optimal model size scales with compute as $N_{\text{opt}} \propto C^\beta$ , with a nearly universal exponent $\beta \sim 0.7$ across all modalities (e.g., image modeling at 32×32 resolution yields $\beta = 0.65$ , while video modeling yields $\beta = 0.71$ ), establishing that larger compute budgets should be allocated predominantly to scaling model size rather than training duration only when dataset size scales sub-linearly as $D \propto N^{0.4}$ .

2. Context and Motivation

The Core Problem: We Don't Know If Scaling Laws Generalize Beyond Language

The fundamental question this paper tackles is whether the remarkably predictable scaling laws observed in language modeling—where test loss follows clean power-law relationships with model size, dataset size, and compute—are a peculiarity of natural language or a universal property of autoregressive generative modeling across all data modalities. Prior to this work, the only systematic scaling law analyses came from language (Kaplan et al., 2020) and a handful of language-adjacent domains. The paper asks: when you train Transformers to predict pixels in images, frames in videos, paired image-text tokens, or even solutions to math problems, do the same mathematical forms emerge? And if so, what do those forms tell us about the underlying data distributions?

This gap matters for several reasons the paper weaves throughout its analysis:

Architecture universality claims need evidence. The Transformer's dominance across modalities raises a deeper question: is its success a matter of engineering convenience, or is there something fundamental about how it learns that transcends data type? Finding identical scaling law forms across images, video, multimodal, and math domains would suggest the latter—that the learning dynamics of autoregressive Transformers are governed by principles that don't depend on whether the tokens represent words, pixels, or mathematical symbols.
Resource allocation in non-language domains lacks principled guidance. In language, the scaling laws from Kaplan et al. (2020) gave practitioners a framework for deciding how to split compute between model size and training duration. Image, video, and multimodal model training lacks this same principled basis. Without knowing how rapidly loss improves with model size or compute in these domains, investments in scaling are speculative—you might train a model 10× larger only to find the returns are far smaller (or larger) than expected.
The information-theoretic interpretation of loss opens a window into data complexity. Cross-entropy loss has a specific decomposition: $\text{Loss} = \text{Entropy of True Distribution} + \text{D}_{\text{KL}}(\text{True} \parallel \text{Model})$ . If scaling laws reliably extrapolate to infinite compute and model size, the irreducible loss $L_\infty$ becomes an estimate of the entropy of the data distribution itself—a measure of how much intrinsic uncertainty exists in the data. This transforms scaling laws from mere performance forecasting tools into scientific instruments for probing the complexity of real-world distributions (YFCC100M images at various resolutions, web videos, multimodal pairs).

Conflicting Signals in Prior Work

The paper is motivated by a specific set of tensions in the existing literature that this work resolves.

Language had clean scaling laws (Kaplan et al., 2020), but were they universal? The immediate predecessor to this work established that language model loss scales as $L(N) \propto N^{-\alpha_N}$ , $L(D) \propto D^{-\alpha_D}$ , and $L(C) \propto C^{-\alpha_C}$ with high precision, and that the optimal model size grows as $N_{\text{opt}} \propto C^{0.73}$ . But that study was exclusively about language—sequences of words with their particular statistical structure (Zipfian distributions, long-range dependencies, syntactic and semantic hierarchies). There was no reason to assume that images (with 2D spatial structure and local correlations) or videos (with temporal coherence) would exhibit the same mathematical form. The paper directly acknowledges this uncertainty by asking: "Do they apply to all data modalities?" (Section 1).

Prior domain-specific models used diverse architectures, making systematic comparison impossible. Before this work, different modalities were studied with different architectures: PixelRNN/CNN for images (van den Oord et al., 2016), specialized video architectures (Weissenborn et al., 2019), multimodal models with modality-specific encoders (Tsai et al., 2019), and specialized math solvers with tree-structured reasoning (Saxton et al., 2019). Because every study used different architectures, different training procedures, and different metrics, there was no way to determine whether differences in performance scaling were due to the data modality or to the architectural choices. The paper's decision to apply an identical decoder-only Transformer architecture with minimal hyperparameter changes across all domains is not just methodological convenience—it's a deliberate design choice to isolate the effect of data modality on scaling behavior.

Information-theoretic interpretation was underexplored and unvalidated. Kaplan et al. (2020) and Brown et al. (2020) noted that the loss approaches a constant $L_\infty$ as scale increases, but they did not thoroughly explore the interpretation of $L_\infty$ as the entropy of the data distribution. This interpretation requires assuming that an infinitely large Transformer trained on infinite data could perfectly model the true distribution ( $D_{\text{KL}} \to 0$ ). Prior to this work, there was no empirical evidence for whether this assumption was reasonable—did $L_\infty$ actually behave like an entropy? Did the power-law term actually correspond to $D_{\text{KL}}$ ?

The relationship between generative loss and representation quality was unclear. A critical practical question lingered: if the generative loss approaches an irreducible floor, does that mean the model has extracted all useful semantic information from the data? Or does important representational content lie in the "last few bits" near $L_\infty$ ? Prior work could not answer this because no one had systematically studied downstream task performance (e.g., classification) as generative models approached their irreducible loss.

Where Existing Approaches Fall Short

The paper identifies specific gaps that made prior work insufficient to address these questions:

No multi-domain scaling analysis with controlled architecture. As noted above, the literature contained scaling studies in individual domains but no controlled cross-domain comparison. Without holding architecture and training methodology constant, scaling exponent comparisons across domains would be confounded—you couldn't tell whether images scaled with a different exponent than language because images are inherently different, or because the image model used a different architecture.

No estimates of data entropy from scaling law extrapolation. The theoretical connection between $L_\infty$ and entropy had been noted (the cross-entropy equals entropy plus KL divergence), but no one had systematically extrapolated scaling trends to estimate the entropy of real-world data distributions. This required (a) training enough model sizes to fit reliable power-law plus constant trends, and (b) varying the resolution/encoding of the data to see whether the estimated entropy behaved sensibly (e.g., higher-resolution images should have higher entropy).

No systematic study of compute-optimal model size across modalities. Kaplan et al. (2020) found $N_{\text{opt}} \propto C^{0.73}$ for language. But was this exponent universal? If different modalities had different exponents, the optimal strategy for allocating compute would differ—you might want to train larger models for images than for video at the same compute budget, or vice versa. Without this knowledge, practitioners in non-language domains were operating blind.

The relationship between generative and discriminative performance was unquantified. It was known that pretrained generative models could be fine-tuned for classification tasks (e.g., Radford et al., 2018), but the scaling relationship was unknown. Did classification accuracy improve smoothly with generative model size? Did it plateau when the generative loss approached the irreducible loss? Answering this would determine whether pushing toward the irreducible loss was practically worthwhile.

No principled framework for understanding multimodal information flow. Multimodal models that jointly model text and images raised fascinating information-theoretic questions—how much information does a caption provide about an image, and vice versa? How does this mutual information scale with model size? Prior work had not used scaling laws to tackle these questions systematically. The paper's "infogain" metric ( $I(\text{text}, \text{image}) / L(\text{text})$ ) provided a bounded quantity (it must be less than 1) that could be extrapolated to forecast model sizes needed for different levels of multimodal understanding.

How This Paper Positions Itself

The paper frames itself not as proposing a new architecture or training method, but as establishing the empirical universality of neural scaling laws across data modalities. This is a scientific contribution rather than an engineering one—the goal is understanding, not building.

The paper's positioning can be understood through several deliberate choices:

Single architecture across all domains. By exclusively using decoder-only Transformers (with sparse attention for images and video, dense for math), the paper eliminates architecture as a confounding variable. Any differences in scaling behavior can be attributed to properties of the data distribution itself, not to architectural choices. This is a significant departure from prior domain-specific work and represents a bet on the Transformer's universality.

Information theory as organizing framework. The paper uses the decomposition $L = S(\text{True}) + D_{\text{KL}}(\text{True} \parallel \text{Model})$ as a recurring theoretical lens. This isn't just notation—it structures the paper's key contributions:

Estimating the entropy of YFCC100M images at various resolutions (e.g., $\sim 602$ nats/image at 8×8, $\sim 2026$ nats/image at 16×16, $\sim 6806$ nats/image at 32×32 from the compute scaling fits in Table 3)
Estimating the KL divergence between true and model distributions at any given model size
Using the infogain bound ( $\text{Infogain} < 1$ ) to forecast model sizes needed for multimodal understanding (e.g., the paper projects that 20% infogain would require $\sim 3$ trillion parameters for text-to-image models)

Scaling as a scientific instrument. The paper treats scaling laws not just as forecasting tools but as probes for measuring properties of data distributions. The entropy of a distribution is a fundamental quantity that is typically impossible to measure directly for real-world data (you'd need to know the true probability distribution). By extrapolating $L(C)$ trends to infinite compute, the paper obtains estimates of this entropy. This is a clever methodological innovation: use the scaling behavior of models to measure properties of the data those models are trained on.

The compute-optimal model size as a universal principle. The paper emphasizes the near-universality of $\beta \sim 0.7$ in $N_{\text{opt}} \propto C^\beta$ across all domains (Figure 2). This is presented as a surprising and significant finding—the optimal balance between model size and training duration doesn't appear to depend meaningfully on whether you're modeling pixels, video frames, or math solutions. This universality suggests a deep property of Transformer learning dynamics rather than a data-specific optimization.

Downstream tasks as validation of representational quality. The fine-tuning experiments on ImageNet classification serve a specific argumentative purpose: they demonstrate that the approach to the irreducible generative loss does NOT indicate saturation of representational quality. The classification error continues to improve as a clean power-law even after the generative loss curve "bends" and flattens (Figure 11). This directly addresses the concern that pursuing the last few bits of generative loss might be practically irrelevant.

Addressing a known inconsistency in scaling laws. Section 6 tackles a problem inherited from Kaplan et al. (2020): if both $L(C)$ and $L(D)$ follow power-laws, their extrapolations eventually intersect in a way that is mathematically inconsistent (because $L(C)$ would predict better performance than $L(D)$ for the same effective compute, which is impossible since learning curves are bounded by the $L(D)$ curve). The paper studies this inconsistency explicitly using image models (Figure 14) and hypothesizes that it reflects a transition to single-epoch training where optimization becomes increasingly efficient at large scale. Confronting this inconsistency head-on—rather than ignoring it—demonstrates intellectual honesty and identifies an important open problem.

The paper also explicitly connects to the broader machine learning narrative around scale. Section 8 frames the results as evidence that "perhaps many capabilities simply lie on a spectrum that can be continuously unlocked with increasing scale"—a direct response to criticisms that scaling alone cannot produce qualitatively new capabilities. The smooth scaling across modalities, including mathematical problem-solving (a task requiring reasoning), provides evidence that scale may be sufficient for a wider range of capabilities than skeptics assume.

3. Technical Approach

3.1 Reader Orientation

This paper is fundamentally an empirical measurement study, not a new model or algorithm proposal. The "system" being built is a scaling law measurement apparatus — a consistent experimental framework for training hundreds of Transformer models at different sizes across different data modalities, measuring their loss, and fitting mathematical relationships that describe how performance improves with scale. The problem it solves is the absence of systematic, cross-domain scaling laws for generative modeling: without these laws, practitioners cannot predict how much compute to invest, how large a model to build, or how much data to collect for any given domain. The solution takes the form of controlled experiments that hold architecture constant while varying model size $N$ (non-embedding parameter count), compute budget $C$ (total floating-point operations), and in some cases dataset size $D$ , then fitting the resulting loss measurements to a power-law plus constant functional form.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has four major components:

Data pipelines (per modality) — Five distinct data sources (language text, images at multiple resolutions, videos, multimodal image-text pairs, procedurally generated math problems) each tokenized into sequences that the Transformer can consume autoregressively.
The Transformer architecture (held nearly constant across domains) — A decoder-only Transformer with domain-specific hyperparameter adjustments (aspect ratio $d_{\text{model}}/n_{\text{layer}}$ , attention pattern, context length) but identical training methodology.
Training protocol (the "measurement instrument") — A standardized procedure: fixed learning rate schedule, fixed batch size in tokens ( $\sim 5 \times 10^5$ tokens per batch), models trained to convergence or to a compute budget limit, with test loss recorded as the primary observable.
Curve-fitting machinery (the "analysis layer") — After collecting loss measurements across model sizes and compute budgets, a fitting procedure extracts the parameters $(L_\infty, x_0, \alpha_x)$ of the power-law plus constant form, and the optimal model size exponent $\beta$ from the compute-efficient frontier.

Information flows as follows: raw data → tokenization into sequences → autoregressive training with cross-entropy loss → collection of test loss measurements at convergence (for $L(N)$ ) or at the compute-efficient frontier (for $L(C)$ ) → fitting of parameters → information-theoretic interpretation of $L_\infty$ as entropy and the power-law term as $D_{\text{KL}}$ .

3.3 Roadmap for the Deep Dive

First, the unified Transformer architecture and its domain-specific adaptations, since this is the constant "measurement instrument" whose stability makes cross-domain comparisons meaningful.
Second, the formal definition of compute $C$ and how it is measured, because this is the independent variable in half the scaling laws and its consistent definition across domains enables the universal $\beta \sim 0.7$ finding.
Third, the data domains and their tokenization, because the structure of the data determines both the loss values and the appropriate hyperparameter settings (context length, attention pattern).
Fourth, the scaling law fitting procedure — how $(L_\infty, x_0, \alpha_x)$ are extracted from empirical measurements and why the power-law plus constant form is chosen over alternatives.
Fifth, the compute-optimal model size extraction — how $N_{\text{opt}}(C)$ is identified from the convex hull of the loss-versus-compute frontier and why the resulting exponent $\beta$ is so consistent across domains.
Sixth, the information-theoretic interpretation framework — the theoretical justification for identifying $L_\infty$ with $S(\text{True})$ and the reducible loss with $D_{\text{KL}}$ , including the assumptions required for this interpretation to hold.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that autoregressive Transformers exhibit universal scaling laws — a power-law plus constant form for the cross-entropy loss as a function of model size, compute, and dataset size — across five distinct data modalities, and that these laws have an information-theoretic interpretation allowing estimation of fundamental properties of data distributions.

The Unified Transformer Architecture

The paper's most important methodological choice is to apply the same basic architecture to every data modality. This is a decoder-only Transformer — meaning it predicts tokens autoregressively from left to right without any encoder component — identical in spirit to GPT (Radford et al., 2018) but with domain-specific adjustments to hyperparameters.

Why decoder-only? The autoregressive formulation is universal: any data that can be serialized into a sequence of discrete tokens can be modeled by predicting each token given all previous tokens. For pixels, this means predicting RGB values in raster order; for video frames, predicting VQ codes in spatiotemporal order; for multimodal data, predicting a concatenated sequence of text tokens and image pixels; for math, predicting answer characters. The decoder-only choice eliminates architectural variance — you don't need to decide whether images need convolutions or text needs bidirectional attention. The paper explicitly notes its models are "typically simpler than many of those that have been previously discussed" (Section 7).

Core architectural parameters. The Transformer consists of $n_{\text{layer}}$ layers, each with attention dimension $d_{\text{model}}$ . The number of attention heads is set to $\max(2, d_{\text{model}}/64)$ . The paper defines "model size" $N$ as the non-embedding parameter count — the parameters in the Transformer blocks themselves, excluding the token embedding matrix and output projection. This convention follows Kaplan et al. (2020) and is important because the embedding parameters scale with vocabulary size, which varies across domains, while the non-embedding count better reflects the model's processing capacity.

The aspect ratio $d_{\text{model}}/n_{\text{layer}}$ — a key domain-specific choice. For any fixed non-embedding parameter count $N$ , there is a tradeoff between making the model wider (larger $d_{\text{model}}$ ) versus deeper (larger $n_{\text{layer}}$ ). The paper sweeps this ratio experimentally for each domain and finds substantially different optima:

Language models (from Kaplan et al., 2020) prefer aspect ratio $\sim 100$ — very wide, relatively shallow models.
Images, video, and math prefer aspect ratio $\sim 5\text{--}10$ — meaning they need roughly 10× deeper and thinner models than language. The paper states: "image and math models perform optimally with an aspect ratio $\approx 5$ , which suggests that on these domains we should aim for deeper and thinner models, with at least a 10x smaller aspect ratio compared to optimized language models" (Section 2.2).

This is a non-obvious finding. The paper's hypothesis (not fully explained, but implied by the context) is that deeper models are needed when the data has local structure — pixels nearby in an image are highly correlated, so the model needs many sequential layers to integrate information across the full spatial extent. Language, by contrast, has long-range dependencies that benefit from wide attention layers that can attend broadly in a single step.

The fully-connected and attention layer scaling. For language and multimodal models, the paper uses the standard Transformer convention: fully-connected layers of size $4d_{\text{model}}$ and attention layers of size $d_{\text{model}}$ . For images, video, and math, the paper reverses this: fully-connected layers scaled to $d_{\text{model}}$ and attention layers scaled to $d_{\text{model}}/4$ . This is noted in Section 2.1: "For math, image, and video modeling we scale the FC layers to $d_{\text{model}}$ and the attention layers to $d_{\text{model}}/4$ ." This means these models spend a larger fraction of their parameters on the feed-forward components (which process each position independently) and relatively less on the attention components (which model interactions between positions). The paper does not explicitly justify this choice, but it likely reflects the finding that deeper, thinner models work better for these domains — smaller attention layers per block make the blocks cheaper, enabling more layers for a given $N$ .

Attention patterns: dense versus sparse. For domains with long sequences, the paper uses sparse attention patterns (Child et al., 2019) to reduce the $O(n_{\text{context}}^2)$ cost:

Images and video use sparse attention "alternating between locally-banded attention and fixed-stride attention in sequential layers, where both the local context length and fixed-stride length are given by the side-length in tokens of the square images" (Section 2.1.2). For a 32×32 pixel image with 1024 total RGB tokens, each token attends to a local band of neighboring pixels plus strided "landmark" pixels, rather than all 1024 tokens. This reduces compute without sacrificing the spatial structure that matters for images.
Multimodal models appear to use dense attention (the context length of 3200 tokens is manageable with dense attention at the model sizes trained).
Math models use dense attention since their context is only 512 tokens.

Context length varies by domain. The context window size is set based on the data structure: 512 tokens for math problems (Section 2.1.6, where "we fill a transformer's 512-token context window with concatenated problems"), 3072 tokens for 32×32 RGB images ( $3 \times 32 \times 32 = 3072$ ), 4096 tokens for 16-frame VQ-encoded video clips, and 3200 tokens for multimodal image-text pairs.

Learning rate schedule and optimization. All models use an identical learning rate schedule: 3000 steps of linear warm-up followed by linear decay to $1/10$ of the maximum learning rate. The maximum learning rate itself varies with model size and domain — larger models typically use lower learning rates (e.g., 0.0016 for a 100K-parameter multimodal model down to 0.0004 for a 680M-parameter model, as detailed in Table 4). The paper uses Adam optimizer with weight decay of 0.05 only for math models. All models are trained with batch sizes of roughly $5 \times 10^5$ tokens (e.g., 128 images for image models, 128 video clips for video, 128 text-image pairs for multimodal, 524,288 tokens for math).

Training duration. Models are trained for at least 250,000 parameter updates, but many are trained "significantly longer" because they hadn't reached convergence or the compute-efficient frontier. The $L(N)$ trends use the test loss at the step minimizing test loss — effectively early-stopped performance at convergence.

Why this architectural uniformity matters. The paper's claim that scaling laws are universal across domains depends on the architecture being held constant. If images used convolutional models and text used Transformers, different scaling exponents could reflect architectural differences rather than data properties. By using the same Transformer everywhere, the paper isolates the effect of the data distribution on scaling behavior. The domain-specific hyperparameter changes (aspect ratio, attention pattern, feed-forward scaling) are necessary practical adaptations but are kept minimal — the underlying Transformer is the same.

Formal Definition of Compute $C$

The compute budget $C$ is the central independent variable in half the paper's scaling laws. It must be defined consistently across all models and domains to enable meaningful comparison.

$C \equiv 6N E$

where $N$ is the non-embedding parameter count and $E = S B$ is the total number of tokens processed during training, with $S$ the number of parameter updates (steps) and $B$ the batch size in tokens.

Breaking down the factor of 6. The paper approximates training compute theoretically rather than measuring empirical FLOPs: the factor of 6 includes a factor of 2 because each multiply-add operation counts as two FLOPs in the standard convention, and a factor of 3 because the backward pass requires roughly twice the compute of the forward pass (so forward + backward = 3× forward). Hence $6N$ is the approximate FLOPs per token per parameter.

What this means operationally. If you train a model with $N$ parameters on $E$ total tokens, the total compute is $C \approx 6NE$ petaflop-days (PF-days, where 1 PF-day $\approx 8.64 \times 10^{19}$ FLOPs). For example, training a 1-billion-parameter model on 10 billion tokens costs approximately $C \approx 6 \times 10^9 \times 10^{10} = 6 \times 10^{19}$ FLOPs $\approx 0.7$ PF-days.

Why this definition. The theoretical definition $C \equiv 6NE$ makes $C$ a controlled experimental variable rather than an empirical measurement subject to hardware inefficiencies. It treats all tokens and all parameters uniformly — a forward-backward pass on any parameter costs the same amount of compute in this accounting. This is an approximation (attention layers cost more per parameter than feed-forward layers, and sparse attention changes the scaling), but it is the standard convention from Kaplan et al. (2020) and enables clean comparisons.

Units. The paper reports compute in PF-days throughout (Table 1, figure axes). The denominators in the scaling law fits are in PF-days, meaning the constant $C_0$ in $(C/C_0)^{-\alpha}$ represents the compute needed to achieve a reducible loss of 1 nat/token.

Data Domains and Tokenization

The paper studies five data distributions, each tokenized differently but all fed into the same autoregressive Transformer training loop. Understanding the tokenization is essential because the cross-entropy loss is measured per token, and comparing losses across domains requires understanding what a "token" represents in each case.

Language. The paper uses existing results from Kaplan et al. (2020) and Brown et al. (2020), who used byte-pair encoding (BPE) tokenization with vocabulary sizes in the tens of thousands. The loss is measured in nats per BPE token. The paper does not retrain language models; it references the published scaling curves for comparison.

Images (pixel-level). The YFCC100M dataset (Thomee et al., 2015) contains approximately $10^8$ web images. Each image is scaled to resolution $R \times R$ pixels, represented in raster order (row by row, pixel by pixel within each row) with RGB colors, each in the range $[0, 255]$ . This produces $3R^2$ tokens per image. The paper studies three resolutions: $8 \times 8$ (192 tokens/image), $16 \times 16$ (768 tokens/image), and $32 \times 32$ (3072 tokens/image). The token vocabulary is simply the 256 possible byte values for each color channel — there is no learned embedding of pixel intensities, just a raw byte-level representation.

Why raster order? Raster order preserves spatial locality: pixels that are adjacent in image space are adjacent in token space, which means the Transformer's causal attention mask naturally allows each pixel to attend to all pixels above and to the left of it. This is important because image generation (predicting bottom-right pixels given top-left context) is a natural autoregressive task — you're predicting the lower-right portion of the image from the upper-left. Alternative orderings (e.g., random, Hilbert curve) would disrupt this locality and likely hurt performance, though the paper does not ablate this.

Images (VQ-encoded). For 64×64 pixel images, the paper uses a VQ-VAE (van den Oord et al., 2018; Dhariwal et al., 2020) to compress images into discrete codes. Two configurations are tested:

16×16 VQ encoding: The 64×64 image is encoded into a 16×16 grid of codes from a codebook of size 4096, for a total of 256 tokens per image. Each token is a VQ code index, not a raw pixel value.
32×32 VQ encoding: The 64×64 image is encoded into a 32×32 grid of codes from a codebook of size 1024, for a total of 1024 tokens per image.

The VQ-VAE models are trained on frames from the web-scraped video dataset (described next), using the VQ-VAE architecture with modifications including "dead code revival" (Dhariwal et al., 2020) — a technique for maintaining codebook usage during training. The two configurations trade off compression ratio versus spatial resolution: 16×16 uses larger codes but fewer tokens; 32×32 uses smaller codes but more tokens.

Why VQ encoding? VQ encoding serves two purposes: (1) it compresses images from 12,288 raw RGB tokens (64×64×3) to 256 or 1024 discrete tokens, making training tractable with Transformers whose self-attention cost scales quadratically in sequence length; (2) it tests whether scaling laws generalize to compressed representations, which is practically important since many real-world generative models operate on latent codes rather than raw pixels.

Video. The video dataset consists of approximately $7 \times 10^5$ videos totaling about 100 hours scraped from the web. Each frame is scaled to 64×64 pixels and encoded with the 16×16 VQ-VAE (256 tokens per frame). The paper trains on sequences of 16 sequential frames, for a total of $16 \times 256 = 4096$ tokens per video clip. This means the model sees a little over half a second of video at a time (assuming $\sim 30$ fps, 16 frames covers $\sim 0.53$ seconds). The sparse attention pattern alternates between locally-banded and fixed-stride attention, where both the local context length and stride length equal the frame side length in tokens (16 tokens).

Why 16 frames? The choice of 16 frames balances two constraints: (1) the context length must be small enough that Transformers can be trained at scale (4096 tokens with sparse attention is manageable); (2) 16 frames provide enough temporal context for the model to learn motion and temporal coherence. The paper does not explore varying the number of frames.

Multimodal text and images. The multimodal dataset consists of a "wide variety of image/text pairs curated through web search" — images paired with textual captions. The tokenization is a simple concatenation: BPE text tokens (using the tokenization from Brown et al., 2020) followed by raw pixel tokens (RGB values in $[0, 255]$ ). The images are 32×32 pixels ( $3 \times 32 \times 32 = 3072$ tokens), and captions are padded or trimmed to 128 text tokens, for a total context length of 3200 tokens per image-caption pair. The model learns a shared embedding matrix that maps both text tokens and pixel values to the same embedding space.

Why concatenation? This approach treats the multimodal problem as a single autoregressive sequence modeling task — the model predicts text tokens, then pixel tokens, all from the same Transformer. This avoids modality-specific encoders and forces the model to learn cross-modal correspondences through the attention mechanism alone. The paper also trains separate models for text-to-image (text first, then image) and image-to-text (image first, then text), finding "poor performance for bidirectional models in preliminary experiments."

Loss weighting for multimodal models. The paper computes separate losses for text tokens and image tokens, then combines them with a weighting:

$L = 9 L_{\text{image}} + L_{\text{text}}$

The factor of 9 up-weights the image loss relative to the text loss. The paper states this weighting was found to "produce good results in a scan" (Section 2.1.5). The motivation is likely that images have higher intrinsic entropy (more bits to predict) and the model might otherwise focus too much on the easier text prediction task at the expense of image quality.

Mathematical problem solving. The paper uses the problem generator from Saxton et al. (2019), which produces math problems across eight modules: algebra, arithmetic, calculus, comparisons, numbers, measurement, polynomials, and probability. Problems are generated procedurally with an "entropy" parameter $s$ that controls difficulty — larger $s$ means more complex problems (wider numerical ranges, more compositional steps). The training distribution samples $s$ uniformly from $[3, 10]$ , while test sets probe $s \in [1, 19]$ to study extrapolation.

Why math? Math is included as a "rather different domain" (Section 2.1.6) — unlike images or text, math problems have deterministic answers (the true entropy is zero) and require algorithmic reasoning. If scaling laws still apply, it strengthens the case for universality. The paper also uses math to study extrapolation — performance on harder problems than those seen during training — which is a stronger test of generalization than the typical i.i.d. test set evaluation.

Tokenization for math. Both questions and answers are modeled at the character (byte) level: "we model all questions and responses at the character (byte) level" (Section 2.1.6). The model receives problems as plain text concatenated together to fill the 512-token context window, with a mask so that only the answer tokens contribute to the loss. This prevents the model from being penalized for incorrectly "predicting" the question text.

Why character-level? The paper notes that "a few problem types require interpreting both numbers and strings as sequences of individual characters, so for simplicity we model all questions and responses at the character (byte) level." This avoids the need for a vocabulary that can represent arbitrary numbers — character-level tokenization can represent any integer or mathematical expression without out-of-vocabulary issues.

Important nuance about the math data distribution. The problem generator produces problems with frequency proportional to $10^s$ , meaning easy problems (low $s$ ) appear orders of magnitude more frequently than hard problems (high $s$ ). This creates an implicit curriculum: "easy components of the training distribution may be memorized" while some hard problems "may not be seen at all" (Appendix B.1). The paper notes that the training loss "do[es] not adhere as well to a power-law form, perhaps because of the implicit curriculum in the frequency distribution of easy and hard problems" (Section 2.1.6).

The Power-Law Plus Constant Scaling Form

The paper's central empirical claim is that the cross-entropy loss $L$ follows a specific functional form as a function of model size $N$ , compute budget $C$ , or dataset size $D$ :

$L(x) = L_\infty + \left(\frac{x_0}{x}\right)^{\alpha_x}$

where $L_\infty$ is the irreducible loss (a fitted constant representing the asymptotic loss as $x \to \infty$ ), $x_0$ is a scale parameter (the value of $x$ where the reducible loss equals 1 nat), and $\alpha_x > 0$ is the scaling exponent (smaller $\alpha_x$ means slower improvement with scale). Here $x$ can be $N$ , $C$ , or $D$ , and the parameters $(L_\infty, x_0, \alpha_x)$ are fitted separately for each $x$ and for each domain.

What this form says operationally. If you train models of increasing size $N$ (with infinite data and compute to convergence), the test loss starts at some high value for tiny models (approximately $L_\infty + (x_0/N)^{\alpha_N}$ ) and decreases toward $L_\infty$ as $N$ grows. The improvement follows a power-law in the reducible component: the excess loss above $L_\infty$ shrinks as $N^{-\alpha_N}$ . Crucially, the loss never goes below $L_\infty$ — this is the "irreducible" floor.

Why power-law plus constant? The paper does not derive this form theoretically; it is an empirical observation. However, the form has several desirable properties that make it plausible:

At $x \to \infty$ , $L(x) \to L_\infty$ . This captures the intuition that even an infinitely large model with infinite compute cannot predict the data perfectly if the data is inherently stochastic (e.g., the pixel value in a given location given all surrounding context may have some unavoidable uncertainty). The constant $L_\infty$ represents this fundamental unpredictability.
The reducible component follows a power-law. Power-laws are common in natural systems and have been observed in neural network scaling since at least Hestness et al. (2017). They imply that each doubling of scale reduces the reducible loss by a constant factor $2^{-\alpha_x}$ .
Few parameters. The form has only three parameters per fit ( $L_\infty$ , $x_0$ , $\alpha_x$ ), making it robust to fit even with modest numbers of data points (the paper trains approximately 6–8 different model sizes per domain).
The reducible loss $(L(x) - L_\infty)$ as a pure power-law. When plotted on log-log axes (as in Figure 1), the reducible loss against $x$ should form a straight line. The paper emphasizes that "the reducible loss trends for $L(C)$ [appear] to be pure power-laws, even when the reducible loss becomes very small" (Section 2.3). This is a strong claim — it says the power-law holds even as the loss approaches the irreducible floor, meaning there is no "diminishing returns" regime where the exponent changes.

Alternative forms that would have been wrong. A pure power-law $L(x) = (x_0/x)^{\alpha}$ without the constant term would incorrectly predict that the loss tends to zero as $x \to \infty$ , implying the data is perfectly predictable. A logarithmic form $L(x) = L_\infty + \beta / \log(x)$ would predict much slower improvement at large $x$ than what is observed. A double-power-law with different exponents at small and large $x$ would have more parameters and be harder to fit reliably.

Fitting procedure. The paper fits the three parameters by minimizing the error between predicted and measured loss values across the available model sizes or compute budgets. For $L(N)$ , the loss is measured at convergence (the step minimizing test loss, with training continuing until performance plateaus). For $L(C)$ , the loss is taken from the compute-efficient frontier — the minimum loss achievable for a given compute budget by choosing the optimal model size and number of training steps.

The consistency check between $L(N)$ and $L(C)$ . Since $L_\infty$ should represent the same quantity (the entropy) whether estimated from model size scaling or compute scaling, the $L_\infty$ values extracted from $L(N)$ and $L(C)$ fits can be compared. Table 1 shows these values: for 8×8 images, $L_\infty$ from $L(N)$ is 3.12 nats/token and from $L(C)$ is 3.13 nats/token — nearly identical. For 16×16 images, 2.64 versus 2.64. This agreement across different experimental procedures provides a consistency check on the overall framework.

Computing the Reducible Loss Per Image

For images and video, the paper frequently reports the reducible loss per image rather than per token, because the per-image loss has a more intuitive interpretation: it represents the total number of nats of uncertainty about the entire image.

Converting from nats/token to nats/image. For pixel-level images at resolution $R \times R$ , there are $3R^2$ tokens per image (three color channels per pixel). The per-image reducible loss is simply:

$\text{Reducible Loss per Image} = 3R^2 \times (\text{Reducible Loss per Token})$

For VQ-encoded images, the number of tokens per image depends on the encoding resolution.

Per-image scaling law form. When expressed per image, the scaling law becomes:

$L_{\text{per-image}}(C) = L_{\infty, \text{per-image}} + \left(\frac{C}{C_{0,\text{per-image}}}\right)^{-\alpha_C}$

where $L_{\infty, \text{per-image}}$ is the intrinsic entropy of the whole image in nats, and $C_{0,\text{per-image}}$ is the compute needed to achieve a $D_{\text{KL}}$ of 1 nat.

Example values (Table 3). For 8×8 pixel images, the irreducible loss per image is approximately 602 nats from the compute fit, and the reducible loss follows $(C / 1.9 \times 10^3)^{-0.19}$ . For 32×32 pixel images, the irreducible loss jumps to approximately 6806 nats, and the reducible loss scales as $(C / 2.7 \times 10^{26})^{-0.10}$ .

What does 602 nats for an 8×8 image mean? The entropy in nats is related to the effective number of equally-likely configurations: an entropy of $S$ nats implies the distribution is as uncertain as choosing uniformly among $e^S$ possibilities. For 8×8 images with $S \approx 602$ nats, $e^{602} \approx 10^{261}$ — the YFCC100M data distribution (at this very low resolution) is as uncertain as choosing uniformly among an astronomically large number of possible images. This reflects both the diversity of real images and the intrinsic stochasticity at the pixel level (even given perfect context, the exact pixel value has some irreducible uncertainty due to sensor noise, quantization, and the inherent variability of the world).

The steep increase of $L_\infty$ with resolution. Going from 8×8 to 16×16 (4× more pixels) increases the irreducible loss from 602 to 2026 nats/image — roughly 3.4×. Going to 32×32 (4× more pixels again) increases it to 6806 nats — roughly 3.4× again. This is roughly consistent with the entropy growing linearly with the number of pixels, as one would expect if each additional pixel carries independent information. The exact factors suggest each pixel at higher resolution carries slightly less incremental information (sub-linear scaling), consistent with spatial correlations making nearby pixels partially redundant.

Extraction of Compute-Optimal Model Size $N_{\text{opt}}(C)$

For a fixed compute budget $C$ , there is a tradeoff: you can train a smaller model for more steps (more tokens) or a larger model for fewer steps. The optimal model size $N_{\text{opt}}(C)$ is the one that minimizes the test loss achievable within compute budget $C$ .

Definition of the compute-efficient frontier. The paper trains many models of different sizes $N$ for many training steps. For each model, the loss at each step $S$ corresponds to a compute expenditure $C = 6 N S B$ (where $B$ is the batch size in tokens). Plotting all these $(C, L)$ points creates a scatter plot (the colored points in Figure 5). The lower envelope — the minimum loss achieved for any model at each compute budget — defines the compute-efficient frontier. The paper specifically states: "we select points on the convex hull of the loss versus compute frontier; these can be seen as blue points in figure 5" (Section 2.3).

Fitting $N_{\text{opt}}(C)$ . For each point on the compute-efficient frontier, we know both the compute $C$ and the model size $N$ that achieved that performance. Plotting $N_{\text{opt}}$ against $C$ and fitting a pure power-law (no constant term):

$N_{\text{opt}}(C) \propto C^\beta$

yields the exponent $\beta$ . The paper finds $\beta \approx 0.7$ across all domains: 0.73 for language, 0.64–0.75 for images depending on resolution, 0.71 for video, 0.69 for math, 0.70–0.72 for multimodal (Table 1, Figures 2 and 16).

What $\beta \approx 0.7$ means practically. If you increase your compute budget by a factor of 10, you should increase your model size by a factor of $10^{0.7} \approx 5$ and increase your training duration (tokens) by a factor of $10 / 5 = 2$ (since $C \propto N \times \text{tokens}$ , and if $N \propto C^{0.7}$ , then tokens $\propto C^{0.3}$ ). Most of the additional compute should go to a larger model, not to training longer.

Derivation of the implied dataset size scaling. If you train for exactly one epoch (never repeating data), the number of tokens processed equals the dataset size $D$ . In that case, $C = 6ND$ , and substituting $D = C/(6N)$ . If $N \propto C^\beta$ , then:

$D \propto C^{1-\beta} \propto N^{\frac{1-\beta}{\beta}}$

With $\beta \approx 0.7$ , we get $(1-\beta)/\beta = 0.3/0.7 \approx 0.43$ , implying $D \propto N^{0.43}$ — the dataset size should grow sub-linearly with model size. The paper states: "This unambiguously sub-linear scaling across all data modalities runs somewhat counter to conventional wisdom" (Section 2.3). The conventional wisdom at the time often assumed $D \propto N$ (linear scaling).

Caveat. The paper acknowledges that this dataset size prediction assumes training for one epoch, which is not necessarily optimal at the scales studied. The paper notes: "we have yet to train models in a regime where compute optimal training actually implies $D \ll N$ numerically" (Section 2.3). The scaling law predicts that at very large compute budgets, the optimal dataset size is much smaller than the model size, but this regime has not been empirically verified for the domains studied.

Caveat about critical batch size. The paper does not adjust compute estimates for the critical batch size effect — the phenomenon where increasing batch size beyond a certain point yields diminishing returns and effectively wastes compute (McCandlish et al., 2018). Kaplan et al. (2020) accounted for this using measured critical batch sizes; this paper does not, stating that "for large model sizes and compute budgets these effects should be small, because most or all of training involves batches smaller than the critical batch size" (Section 2.3). This is a potential source of systematic error, especially at the largest models trained.

Information-Theoretic Interpretation

The paper's most ambitious contribution is the interpretation of the fitted parameters in terms of information theory. This interpretation turns scaling laws from mere performance forecasting into tools for measuring properties of data distributions.

The cross-entropy decomposition. For any true distribution $P$ and model distribution $Q$ , the cross-entropy can be decomposed as:

$L = \mathbb{E}_{x \sim P} \left[ \log \frac{1}{Q(x)} \right] = D_{\text{KL}}(P \parallel Q) + S(P)$

where $S(P) = \mathbb{E}_{x \sim P}[\log \frac{1}{P(x)}]$ is the entropy of the true distribution (its intrinsic uncertainty, measured in nats) and $D_{\text{KL}}(P \parallel Q) \geq 0$ is the Kullback-Leibler divergence (how much additional uncertainty the model has beyond the true entropy, because it fails to perfectly match the true distribution).

The paper's interpretation. If we assume that in the limit of infinite compute, infinite data, and infinite model size ( $N, C, D \to \infty$ ), an autoregressive Transformer can perfectly model the data distribution (so $D_{\text{KL}} \to 0$ ), then:

$L_\infty = \lim_{\text{scale} \to \infty} L(\text{scale}) = S(P)$

The irreducible loss is an estimate of the entropy of the data distribution. And consequently:

$\left(\frac{x_0}{x}\right)^{\alpha_x} \approx D_{\text{KL}}(P \parallel Q_{\text{scale}})$

The power-law term estimates the KL divergence between the true distribution and the model of a given scale.

Required assumption: the Transformer can perfectly model the data. This interpretation only makes sense if an infinitely large Transformer trained on infinite data can actually represent the true data distribution. The paper states this assumption carefully: "To justify equation (1.2), we must assume that in the limit $D \to \infty$ followed by $N, C \to \infty$ , an infinitely large transformer could model the data distribution exactly" (Section 1, footnote 3). This is not obviously true — Transformers have architectural limitations (finite context window, specific attention patterns) that might prevent them from representing arbitrary distributions. However, for the data domains studied (especially low-resolution images and math problems), the assumption is at least plausible: with sufficient capacity, a Transformer can implement any computation that fits within its context window.

Testable predictions of the interpretation. The interpretation makes several predictions that the paper checks:

$L_\infty$ should be the same whether estimated from $L(N)$ or $L(C)$ . Both fits should converge to the same irreducible loss, since both estimate the same quantity (the entropy). Table 1 shows this holds well: for 8×8 images, $L_\infty = 3.12$ from $L(N)$ vs. $3.13$ from $L(C)$ (nats/token).
$L_\infty$ should increase with image resolution. Higher-resolution images contain more information, so their entropy should be larger. Table 3 confirms this: per-image entropy increases from $\sim 602$ nats (8×8) to $\sim 2026$ nats (16×16) to $\sim 6806$ nats (32×32).
$L_\infty$ should be zero for math problems (since the answers are deterministic given the questions). The paper notes "the information theoretic interpretation of the loss has a somewhat different meaning in the context of math problem solving, where the answers are deterministically related to the questions, so that the entropy should truly vanish" (Section 5). This is a special case — math is a deterministic distribution where the true entropy is zero, making it a useful test of whether the framework handles degenerate cases.
The reducible loss should behave as a pure power-law. Since the KL divergence should vanish smoothly as model capacity increases, and the asymptotic form of learning curves is often power-law-like, the reducible loss should follow a clean power-law with no additional constant term. Figure 1 shows this: the reducible loss trends are straight lines on log-log plots, even when they are much smaller than the irreducible loss.

What the irreducible loss values mean in practice. For 8×8 YFCC100M images, the irreducible loss of 602 nats/image means the data distribution has an intrinsic uncertainty equivalent to choosing uniformly among $e^{602}$ possible images. A model achieving a reducible loss of, say, 6 nats/image (so total loss $\approx 608$ nats) has $D_{\text{KL}} \approx 6$ nats — it is as uncertain as if it were choosing uniformly among $e^6 \approx 400$ possibilities beyond the inherent uncertainty of the data. The paper finds that "billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8 \times 8$ resolution" (Abstract) — meaning the $D_{\text{KL}}$ for these models is just a few nats per image, a small fraction of the total entropy.

Why this interpretation matters beyond the paper. If the identification $L_\infty = S(\text{True})$ holds, then scaling laws provide a method for estimating the entropy of real-world data distributions, which is otherwise extremely difficult. The entropy of natural images is not something you can compute directly (you'd need the true probability distribution of all possible images), but you can estimate it by extrapolating model performance. This transforms scaling laws from an engineering tool (how big a model do I need?) into a scientific instrument (how complex is this data distribution?).

The Empirical Mutual Information and Infogain for Multimodal Models

For multimodal models, the paper defines specialized metrics that leverage the information-theoretic interpretation to quantify cross-modal understanding.

Definition of empirical mutual information. For a text-to-image model trained to predict image tokens given text tokens (and vice versa for image-to-text), the empirical mutual information between text $T$ and image $I$ is:

$I(T, I) \approx L(\text{image} \mid \text{blank caption}) - L(\text{image} \mid \text{real caption})$

where $L(\text{image} \mid \text{blank caption})$ is the loss on image tokens when the text context is replaced with a blank caption, and $L(\text{image} \mid \text{real caption})$ is the loss with the actual caption. The difference measures how much the caption reduces the model's uncertainty about the image — essentially, how many nats of information the caption provides.

What this computes operationally. The model is given either the real caption or a blank caption as context, and its task is to predict the image pixels. The real caption should make this easier if it contains relevant information. The difference in average loss (in nats) directly estimates the mutual information between captions and images under the model's distribution. If the caption provides no information about the image, the losses are equal and $I \approx 0$ . If the caption completely determines the image, the loss with the real caption approaches zero, and $I$ equals the blank-caption loss (the unconditional entropy of the image).

Definition of Infogain. The Infogain normalizes the mutual information by the model's uncertainty about the text itself:

$\text{Infogain} \equiv \frac{I(\text{text}, \text{image})}{L(\text{text})}$

where $L(\text{text})$ is the cross-entropy loss on text tokens alone (without the corresponding image). This ratio must lie in $[0, 1]$ because the mutual information cannot exceed the entropy of either variable: $I(X, Y) \leq \min(S(X), S(Y))$ , and $L(\text{text}) \geq S(\text{text})$ .

Why Infogain is bounded by 1. The mutual information between text and image is at most the entropy of the text — you can't learn more about the image from the text than there is information in the text. And the model's text loss is an upper bound on the true text entropy. So $I(\text{text}, \text{image}) / L(\text{text}) \leq I / S(\text{text}) \leq 1$ . This provides a hard upper bound that can be used for forecasting: if Infogain follows a predictable scaling trend, you can estimate the model size where it would approach 1 (perfect extraction of text information for image modeling), which represents a theoretical maximum.

Subtlety: avoiding out-of-distribution blank captions. If the model was trained only on image-text pairs, blank captions would be out-of-distribution, and the loss difference might not validly estimate mutual information (the model might behave erratically on an input it's never seen). The paper addresses this: "We minimize this issue by measuring the mutual information only after finetuning our models for $10^4$ steps on an even mixture of data with and without captions (for text-to-image) or with and without images (for image-to-text)" (Section 4). This finetuning step ensures blank inputs are in-distribution. The paper also tested training from scratch on a 95/5 mixture of multimodal and blank caption data, finding "very similar results."

Scaling of mutual information with model size. The paper finds that the mutual information and Infogain grow logarithmically with model size:

$I(\text{text}, \text{image}), \text{Infogain} \approx \lambda \log\left(\frac{N}{N_c}\right)$

with different $\lambda$ and $N_c$ for text-to-image versus image-to-text models. For text-to-image, the largest models (with approximately $10^9$ parameters) extract about 8 nats of mutual information and achieve an Infogain of roughly 0.10 — meaning they use about 10% of the information in the text when modeling images.

Why a logarithmic form? Appendix E provides an approximate derivation for language models, where the loss scales as a power-law with token position. If the loss at position $t$ is $L(t) \approx L(N) + (L_U - L(N))/t^p$ (where $L_U$ is the unigram entropy and $p < 1$ ), then the mutual information between the first $T$ tokens and the next $T$ tokens scales approximately linearly with $L_U - L(N)$ . Since $L(N) \propto N^{-\alpha_N}$ with $\alpha_N \ll 1$ , we can approximate $N^{-\alpha_N} \approx 1 - \alpha_N \log N$ , leading to the logarithmic form. This derivation is specific to language, but the paper observes empirically that similar logarithmic trends hold for multimodal models.

Using Infogain to forecast model size requirements. If the logarithmic trend holds, we can extrapolate to larger model sizes: to reach Infogain = 0.20 (using 20% of text information for image modeling, roughly double the current value), a text-to-image model would need approximately $3 \times 10^{12}$ (3 trillion) parameters. To reach Infogain = 1.0 (perfect extraction of text information), the required model size would be far larger — the paper implies it is "unrealistic to extrapolate" that far.

The "picture worth a thousand words" question. The paper operationalizes this proverb by comparing the mutual information (8 nats for the largest image-to-text models) to the per-word loss on text (about 3.4 nats per word after excluding padding). This leads to the conclusion: "a 32x32 image is worth only about 2-3 words to our best models" (Section 4). The model extracts only a few words' worth of information from a 32×32 image. This is not a statement about the inherent information content of the image — it's a statement about how much information the current models can extract, which is limited by their capacity.

Fine-Tuning Protocol for ImageNet Classification

To study the relationship between generative model quality and downstream task performance, the paper fine-tunes pixel-level 32×32 generative image models on 32×32 resolution ImageNet classification.

Architecture modification for classification. The fine-tuning procedure removes the final embedding matrix (the output projection that maps hidden states to pixel value logits) and replaces it with a new single-layer classifier. The input to this classifier is the mean-pooled activations of the Transformer's final layer over all pixel positions. Mean pooling aggregates information from all 3072 token positions into a single representation vector. The paper does not freeze any pre-trained weights — "during finetuning we backpropagate through the full transformer, and we do not freeze any of its weights."

Baseline: training from scratch. For comparison, equivalent randomly-initialized Transformer models are trained "from scratch" on only the classification task, using the same architecture (sans the embedding matrix). This provides a measure of how much the generative pre-training helps versus just having a large Transformer.

Training hyperparameters for fine-tuning. The fine-tuning uses batch size 1024 images and "the same learning rate schedule for finetuning as was used for pretraining" — 3000 steps of linear warm-up followed by linear decay to 1/10 of the maximum learning rate. Learning curves for both pretrained and from-scratch models across model sizes are shown in Figure 11 (bottom).

Key observations from fine-tuning. Larger pre-trained models fine-tune significantly faster and to significantly better performance than smaller ones. The paper notes: "for small models, pretraining affords almost no benefit compared to training from scratch, but it greatly enhances the performance of larger models" (Section 3.4). This suggests that the benefits of generative pre-training are not uniform across model sizes — small models don't have enough capacity to learn useful representations during pre-training that transfer to classification, while large models do.

Why this experiment matters for the paper's thesis. The fine-tuning results address a potential objection: if the generative loss approaches the irreducible loss, does that mean the model has extracted all useful information from the data? The answer from Figure 11 is a clear "no." The classification error continues to improve as a pure power-law even as the generative loss "bends" and becomes dominated by the irreducible component. The paper states: "a great deal of important semantic information lies in the 'last few bits' near the irreducible loss. We may also interpret this as the pre-training process providing a highly effective regularizer for downstream tasks."

The Math Difficulty and Extrapolation Framework

The math domain provides a unique opportunity to study generalization that goes beyond standard i.i.d. train-test splits, because math problems can be generated with controlled difficulty levels.

Difficulty parameterization. The problem generator from Saxton et al. (2019) accepts an "entropy" parameter $s$ that controls problem complexity. Larger $s$ means:

Wider numerical ranges (e.g., larger numbers in arithmetic)
More compositional steps (e.g., deeper nesting of operations)
More diverse problem types within each module

The training distribution samples $s \in [3, 10]$ uniformly. Test sets are generated for $s \in [1, 19]$ , where $s \leq 10$ is within the training distribution ("interpolation") and $s > 10$ is outside it ("extrapolation").

What makes math different for scaling laws. Unlike natural images or text where the true entropy is positive and unknown, math problems have a degenerate true distribution: given the question, there is exactly one correct answer. The entropy of the answer distribution conditioned on the question is zero. However, the model sees questions as token sequences and must learn the mapping. The cross-entropy loss on math problems thus measures how well the model approximates a deterministic function, not a probabilistic distribution. This makes the information-theoretic interpretation ( $L_\infty = \text{entropy}$ ) somewhat different — the paper notes this explicitly in Section 5 and instead focuses on the scaling of loss and accuracy with model size and difficulty.

The training loss does not follow a clean power-law. The paper notes (Section 2.1.6, Appendix B.1) that "the training loss, [does] not adhere as well to a power-law form, perhaps because of the implicit curriculum in the frequency distribution of easy and hard problems." Because easy problems appear exponentially more frequently than hard ones in the training data ( $\sim 10^s$ frequency), the model sees many more easy examples. This uneven sampling may distort the scaling trends relative to domains where tokens are sampled more uniformly from the distribution.

Extrapolation depends on training performance, not model size directly. A key finding (Section 5, Figure 13) is that "extrapolative capabilities of these models predominantly depends on the models' performance on the training distribution. That is, models of different sizes that achieve the same loss on the training distribution perform about equally on the various test distributions." In other words, if a 10M-parameter model and a 100M-parameter model happen to achieve the same training loss (at different points in training), they perform equally on extrapolation tests. Larger models help only because they can achieve lower training loss — they don't provide some additional "generalization ability" beyond what's captured by training loss. This is consistent with findings in Kaplan et al. (2020) for language domain transfer.

Extrapolation performance scales smoothly with difficulty. Figure 24 shows that as the difficulty level increases beyond the training distribution ( $s > 10$ ), the loss increases smoothly, and larger models perform systematically better at all difficulty levels. There is no sharp phase transition where models suddenly fail — instead, performance degrades gradually, suggesting that the learned problem-solving strategies partially generalize to harder problems.

Specific generators excluded from analysis. The paper found "consistently poor performance on the two extrapolation generators probability__swr_p_level_set_more_samples and probability__swr_p_sequence_more_samples, with larger models overfitting against them and achieving worse loss (but higher accuracy) than some smaller models" (Section 2.1.6). These two generators are excluded from the scaling trend figures to prevent them from dominating the trends with their anomalous behavior. This is an important detail — it shows that scaling laws can break down for specific sub-tasks, and the paper's reported trends reflect selective inclusion of well-behaved modules.

Dataset Size Scaling $L(D)$ and Its Measurement

While most of the paper focuses on $L(N)$ and $L(C)$ , it also studies $L(D)$ — how the loss scales with dataset size when model capacity is not the bottleneck. This is done primarily for math problems (where procedural generation enables easy dataset size variation) and to study the inconsistency between $L(C)$ and $L(D)$ scaling.

Definition of $L(D)$ . $L(D)$ is the minimum test loss achievable when training with a dataset of size $D$ (in tokens), assuming the model is large enough that performance is limited by overfitting, not by model capacity. Operationally, it is measured by training a large model on datasets of varying sizes and recording the minimum test loss during training (early-stopped performance).

Measurement details for math (Appendix B.2). For math, the paper uses models with $n_{\text{layer}} = 64$ and $d_{\text{model}} = 512$ for all dataset sizes — a fixed large architecture so that model capacity is not the bottleneck. The training data is procedurally generated at various dataset sizes (measured in millions of problems). The loss $L(D)$ is defined as "the minimum loss during training" (which "differs slightly from early stopping, since we may evaluate at different steps if there are several metrics, ie losses on different test distributions").

Power-law form of $L(D)$ . The paper fits $L(D)$ to the same power-law plus constant form:

$L(D) = L_\infty^{(D)} + \left(\frac{D_0}{D}\right)^{\alpha_D}$

For math (Figure 22), the extrapolation loss follows $L(D) \approx (D / 2.89 \times 10^{-1})^{-0.239}$ (the constant term is not reported in the figure, suggesting it may be statistically indistinguishable from zero for this extrapolation fit). The interpolation loss scales as $L(D) \approx (D / 3.32 \times 10^{-1})^{-0.512}$ .

Image $L(D)$ trends and learning curves. For 8×8 and 16×16 images, the paper compares $L(D)$ to actual learning curves (Figure 14). The key observation is that learning curves approach $L(D)$ from above as model size increases — larger models learn faster and get closer to the fundamental sample efficiency limit. The paper identifies dataset size $D$ with the number of tokens processed, so $L(D)$ represents the loss achievable if you could train to the optimum in a single epoch.

The Scaling Law Inconsistency Problem (Section 6)

The paper explicitly confronts a mathematical inconsistency between the $L(C)$ and $L(D)$ scaling laws that was first noted in Kaplan et al. (2020).

The setup. The $L(C)$ trend gives the loss achievable for a given compute budget when you can choose the optimal model size. The $L(D)$ trend gives the loss achievable with a given dataset size when model capacity is not a bottleneck. Since computing $C = 6 N E$ where $E \geq D$ (you can't process more unique tokens than you have data, but you can repeat data), we can parametrically plot $L(D)$ against the minimum compute required to achieve that $L(D)$ by assuming one-epoch training ( $E = D$ ) and choosing the optimal model size $N_{\text{opt}}(C)$ . This gives a curve $L(D(C))$ — the loss achievable with a compute budget $C$ if you train for exactly one epoch on a dataset of size $D = C/(6 N_{\text{opt}}(C))$ .

The inconsistency. The $L(D(C))$ curve and the $L(C)$ curve must eventually intersect, because $L(C)$ is steeper (has a larger absolute exponent) than $L(D(C))$ . At the intersection point, $L(C)$ predicts better performance than $L(D(C))$ — but this is logically impossible, since $L(C)$ is just the best achievable loss for a given compute, and that cannot be better than the loss achievable with infinite compute on a finite dataset of size $D$ (which is $L(D)$ ).

To rephrase: the $L(C)$ trend says "with compute budget $C$ , you can achieve loss $L$ ." The $L(D(C))$ trend says "with a dataset of size $D = C/(6N_{\text{opt}})$ , the best possible loss (even with infinite compute) is $L(D)$ ." If $L(C) < L(D(C))$ , the $L(C)$ trend is claiming you can exceed the fundamental limit set by the dataset size — which is impossible, because you can't extract more information from a finite dataset than it contains.

Why this must happen mathematically. Both trends are power-laws on a log-log plot, so they appear as straight lines. Two straight lines with different slopes must eventually intersect. The paper shows this visually in the right panels of Figure 14, where the black $L(C)$ line crosses below the blue $L(D(C))$ dashed line.

The paper's hypothesized resolution. The paper observes (Figure 14, left) that learning curves approach the $L(D)$ trend more closely as model size increases: "learning curves appear to approach ever closer to the $L(D)$ trend. This means that larger models learn faster, and it also implies that optimization becomes increasingly effective as model size increases." The hypothesis is that near the intersection point, optimization becomes so efficient that training for more than one epoch provides no benefit — you reach the $L(D)$ limit in a single pass. At that point, the compute frontier bends and becomes coincident with $L(D(C))$ .

In other words: the $L(C)$ trend is steeper than $L(D(C))$ only because current models require multiple epochs to reach the loss minimum (inefficient optimization). As models get larger, optimization becomes more efficient, requiring fewer epochs, until eventually one epoch is optimal. At that point, $L(C)$ and $L(D(C))$ merge, resolving the inconsistency.

Why this matters. If this hypothesis is correct, the scaling of optimal dataset size with model size ( $D \propto N^{0.4}$ ) would eventually change — at very large compute budgets, you would need proportionally more data (the exponent would increase) as the advantage of multi-epoch training vanishes. The paper leaves this as an open question: "It would be interesting to investigate this hypothesis in the future."

Consistency check on irreducible losses. Despite the inconsistency in the reducible loss trends, the irreducible losses estimated from $L(D)$ and $L(C)$ are quite similar: for 16×16 images, $L_\infty^{(D)} \approx 2013$ nats/image and $L_\infty^{(C)} \approx 2023$ nats/image. The paper notes this provides "a consistency check on our results" — the asymptotic limit is reasonably well-determined even if the approach to that limit differs between the two measurement methods.

Design Choices and Their Justifications (Summary)

Why decoder-only Transformers everywhere. Eliminates architectural variance as a confounding variable in cross-domain scaling law comparisons. The autoregressive formulation is universal — any data that can be serialized can be modeled this way.

Why deep and thin for non-language domains. Images and video have strong local correlations that benefit from many sequential layers of processing. Language has long-range dependencies that benefit from wide attention layers. The paper discovered this difference empirically through aspect ratio sweeps.

Why sparse attention for images and video. The quadratic cost of dense attention on long sequences (up to 4096 tokens) is prohibitive at large model sizes. Sparse attention preserves locality while reducing compute.

Why theoretical compute $C = 6 N E$ rather than measured FLOPs. Consistency and simplicity — it makes $C$ a fully controlled experimental variable independent of hardware efficiency. The factor of 6 is the standard convention from prior work.

Why power-law plus constant rather than other forms. Empirically observed, requires few parameters, has a natural information-theoretic interpretation, and produces straight lines on log-log plots of reducible loss (validating the functional form visually).

Why exclude two math problem generators from trends. They exhibited anomalous overfitting behavior where larger models performed worse. Including them would distort the trends and obscure the general scaling patterns.

Why finetune multimodal models on blanked data before measuring mutual information. Without this finetuning step, blank captions or blank images are out-of-distribution, and the loss difference does not validly estimate mutual information. The paper tested this explicitly and found the raw measurement was about twice as large, confirming the need for the finetuning step.

Why not adjust for critical batch size. The paper acknowledges this is a limitation but argues the effect is small for the scale of models studied. Future work could incorporate this adjustment following McCandlish et al. (2018).

4. Key Insights and Innovations

Innovation 1: Scaling Laws as a Scientific Instrument for Measuring Data Complexity

The most conceptually distinctive contribution of this paper is not the finding that scaling laws exist in non-language domains—that is a natural extension of prior work. Rather, it is the systematic development of scaling law extrapolation as a measurement tool for probing fundamental properties of data distributions that are otherwise inaccessible. Before this work, scaling laws were understood primarily as forecasting tools: if you know $L(N)$ for small models, you can predict the loss for larger ones before training them. This paper repurposes the same mathematical machinery to answer a fundamentally different question: how complex is the data itself?

The mechanism that enables this—the decomposition $L = S(\text{True}) + D_{\text{KL}}(\text{True} \parallel \text{Model})$ and the identification of $L_\infty$ with true entropy—was theoretically available before this paper. The cross-entropy equals entropy plus KL divergence is a standard information-theoretic identity. But no prior work had operationalized this identity into a measurement methodology by (a) training enough model sizes to reliably fit the power-law plus constant form, (b) verifying that $L_\infty$ extracted from independent fits ( $L(N)$ vs. $L(C)$ ) agree, and (c) demonstrating that $L_\infty$ behaves sensibly when the data distribution is systematically varied (different image resolutions, different encodings). The paper does all three.

The significance of this reframing extends beyond the specific measurements reported. It establishes a new epistemological role for neural network training: training models at different scales becomes a form of scientific instrumentation, akin to building telescopes of increasing aperture to resolve finer details of astronomical phenomena. The irreducible loss measured at 8×8 resolution (~602 nats/image, Table 3) is not just a number—it is an estimate of a fundamental physical quantity (the entropy of the YFCC100M data distribution at that resolution) that cannot be computed analytically and can only be measured through this kind of extrapolation. The fact that $L_\infty$ increases systematically with resolution (602 → 2026 → 6806 nats/image as resolution doubles twice) provides a sanity check on the methodology: higher-resolution images contain more information, and the measured entropy respects this.

This innovation is fundamental rather than incremental. Prior scaling law work (Kaplan et al., 2020; Hestness et al., 2017) treated $L_\infty$ as a nuisance parameter—the asymptote that limits improvement—rather than the quantity of primary scientific interest. This paper inverts that perspective: $L_\infty$ becomes the target, and the scaling of the reducible loss becomes the means of estimating it. The distinction matters because it transforms scaling laws from an engineering tool (useful for resource allocation decisions) into a scientific instrument (useful for understanding the nature of data itself).

A subtle but important validation: the irreducible loss from $L(D)$ and $L(C)$ fits agree to within ~0.5% for 8×8 images (599 vs. 602 nats/image) despite being estimated from entirely different experimental procedures. This cross-validation is the empirical anchor for the claim that the methodology is well-founded rather than merely curve-fitting.

Innovation 2: The Near-Universality of $\beta \sim 0.7$ as a Surprising Empirical Regularity

The paper's finding that $N_{\text{opt}} \propto C^\beta$ with $\beta \sim 0.7$ across all data modalities (Figure 2, Table 1) is presented as a striking and unexpected result. The authors state explicitly: "we did not anticipate that the exponent $N_{\text{opt}} \propto C^{0.7}$ would be largely independent of the data distribution" (Section 8). This is a substantive empirical discovery, not a foregone conclusion.

To appreciate why this is surprising, consider what would have been the natural prior expectation. Different data modalities have fundamentally different statistical structures: language has long-range syntactic dependencies and Zipfian token frequencies; images have strong local spatial correlations that decay with distance; videos add temporal coherence on top of spatial structure; math problems have deterministic answers and algorithmic solution paths. These structural differences manifest in substantially different scaling exponents for the loss itself— $\alpha_C$ ranges from 0.10 for 32×32 images to 0.19 for 8×8 images (Table 3)—reflecting that the rate at which additional compute reduces the reducible loss varies widely by domain. Given this variation in $\alpha_C$ , one might naturally expect the optimal allocation of compute between model size and training duration to also vary by domain. The data emphatically says otherwise: whether you're modeling pixels, video frames, or math solutions, you should spend roughly the same fraction of additional compute on scaling the model versus training longer.

The universality of $\beta$ is not a theoretical prediction; it is a purely empirical regularity that "beg[s] for theoretical explanation" (Section 8). The paper does not provide such an explanation, but the consistency of the finding across five domains strongly suggests that $\beta \sim 0.7$ reflects something deep about the learning dynamics of Transformers—perhaps related to the scaling of gradient noise, the spectral properties of the loss landscape, or the effective dimensionality of the model's parameter space—rather than properties of any specific data distribution.

The practical implication, emphasized throughout the paper, is that "larger compute budgets should be 'spent' mostly on larger models, rather than much longer training runs" (Section 8). This lesson had been established for language (Kaplan et al., 2020) but its domain-independence was unknown. The finding also implies sub-linear dataset size scaling ( $D \propto N^{0.4}$ for one-epoch training), which the paper notes "runs somewhat counter to conventional wisdom" (Section 2.3) that had assumed roughly linear scaling of data with model size.

This is a fundamental finding rather than incremental because it constrains the space of possible theories of neural network scaling. Any satisfactory theory must predict not just that $N_{\text{opt}}(C)$ follows a power-law, but that the exponent is approximately 0.7 regardless of the data distribution. This universality is the paper's strongest argument that the scaling phenomena it documents are not merely descriptive curve-fitting but reflect genuine underlying principles.

Innovation 3: The Inconsistency Between $L(C)$ and $L(D)$ as a Productive Diagnostic

Section 6 of the paper addresses an apparent mathematical inconsistency between compute and dataset size scaling laws—a problem first identified in Kaplan et al. (2020) but studied here with greater precision and cross-domain evidence. Rather than treating this inconsistency as an embarrassment or an artifact to be explained away, the paper elevates it into a diagnostic tool for understanding the limits of current optimization.

The logic is elegant: the $L(C)$ trend (loss achievable for a given compute budget with optimal model size) and the $L(D(C))$ trend (loss achievable on a dataset of size $D = C/(6N_{\text{opt}})$ , assuming one-epoch training) must be mutually consistent because you cannot extract more information from a finite dataset than it contains. The fact that extrapolated $L(C)$ falls below extrapolated $L(D(C))$ (visible in the right panels of Figure 14, where the black $L(C)$ line crosses the blue $L(D)$ dashed line) means that at least one of the extrapolated trends must break down before the intersection point.

The paper's proposed resolution—that optimization becomes increasingly efficient at large scale, eventually reaching the $L(D)$ limit in a single epoch—is not proven but is supported by the empirical observation that "learning curves appear to approach ever closer to the $L(D)$ trend" as model size increases (Figure 14, left). This is a genuinely novel observation: not just that larger models learn faster (which was known), but that the gap between the learning curve and the theoretical sample-efficiency limit narrows systematically with scale. If this trend continues, the $L(C)$ frontier eventually bends and merges with $L(D(C))$ , eliminating the inconsistency.

The significance of this innovation lies in identifying the inconsistency as an open problem rather than a failure of the scaling framework. Prior work (Kaplan et al., 2020) noted the inconsistency but did not analyze it in detail. By studying it explicitly across multiple image resolutions (8×8 and 16×16, Figure 14) and connecting it to the observed approach of learning curves to the $L(D)$ limit, the paper transforms what could be dismissed as a flaw in curve-fitting into a productive research question: what happens to scaling laws when models become so sample-efficient that multi-epoch training provides no benefit? The shaded sensitivity analysis in Figure 14 (±5% variation in the $N_{\text{opt}}$ exponent dramatically changes the intersection point) illustrates both the importance of precise measurement and the potential for future experiments to resolve the question.

This is a diagnostic rather than a solution—it clarifies the boundaries of current knowledge without closing the question. That makes it a conceptual contribution: the paper provides a framework for thinking about the interplay between optimization efficiency and scaling laws, even though the ultimate resolution requires experiments at scales beyond those accessible in this work.

Innovation 4: Downstream Performance Continues to Improve Even as Generative Loss Saturates

Section 3.4's finding that ImageNet classification performance continues to improve as a clean power-law in model size, even as the generative loss approaches its irreducible floor (Figure 11), addresses a hypothesis that could have seriously undermined the practical importance of scaling laws: the worry that "diminishing returns" near the irreducible loss meant that further generative model scaling would be practically useless.

This hypothesis was plausible given the shape of the generative loss curve. As models approach $L_\infty$ , the total loss flattens—improvements in the reducible component become imperceptible when plotted on a linear scale against the irreducible background. One might reasonably conclude that the model has extracted all useful information from the data, and that additional scale is wasted on modeling irreducible pixel-level noise. The classification fine-tuning results directly refute this: "a great deal of important semantic information lies in the 'last few bits' near the irreducible loss" (Section 3.4).

The evidence is stark. The pre-trained models follow a smooth power-law in classification loss ( $(N / 1.72 \times 10^{10})^{-0.105}$ ) and error rate ( $(N / 2.09 \times 10^3)^{-0.089}$ ) that shows no sign of bending or saturating at the largest model sizes tested ( $\sim 4 \times 10^8$ parameters), even though the generative loss trend (Figure 7, 32×32 curve) has already visibly flattened for models above $\sim 10^7$ parameters. The pretraining process is acting as an extraordinarily effective regularizer, extracting representations whose quality improves smoothly with scale long after the generative modeling objective shows diminishing returns.

This has a direct conceptual implication: generative loss is not a monotonic proxy for representation quality. The two can decouple, with representation quality continuing to improve in a regime where generative loss appears to plateau. This is a fundamental insight about the relationship between unsupervised pre-training objectives and downstream task performance, and it suggests that scaling laws for generative modeling may be conservative predictors of practical capability improvements—the benefits of scale may be larger than the loss curve alone would suggest.

The finding also builds a bridge to subsequent work on scaling laws for transfer learning and downstream tasks (e.g., Hernandez et al., 2021). By demonstrating that representation quality follows its own power-law trend that is decoupled from the generative loss trend in the large-model regime, the paper provides empirical motivation for studying scaling laws of downstream performance directly, rather than relying on generative loss as a proxy.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper studies five distinct data distributions: (1) Language — results from prior work (Kaplan et al., 2020; Brown et al., 2020) on standard web text corpora; (2) Images — the YFCC100M dataset (Thomee et al., 2015) containing approximately $10^8$ web images, scaled to resolutions of 8×8, 16×16, and 32×32 pixels, plus 64×64 images encoded with VQ-VAE (van den Oord et al., 2018); (3) Video — approximately $7 \times 10^5$ web-scraped videos totaling about 100 hours, with frames scaled to 64×64 and VQ-encoded; (4) Multimodal — a curated dataset of image/text pairs from web search, with 32×32 images and 128-token captions; (5) Math — procedurally generated problems from the generator of Saxton et al. (2019) spanning eight modules (algebra, arithmetic, calculus, comparisons, numbers, measurement, polynomials, probability), with training difficulty uniformly sampled from entropy $s \in [3, 10]$ . The ImageNet classification fine-tuning experiments use 32×32 downsampled ImageNet (Chrabaszcz et al., 2017). The paper uses the same 500-question test set from Lightman et al. (2022) for the MATH benchmark.
Base model(s). All experiments exclusively use decoder-only Transformer models, with domain-specific adjustments to hyperparameters but an identical core architecture. Model sizes (measured as non-embedding parameter counts $N$ ) range from approximately $10^4$ to $3 \times 10^9$ parameters, with the largest models trained being 400M parameters for most non-language domains and referencing GPT-3's 175B parameter results for language. The architectural uniformity is deliberately chosen to isolate the effect of data modality on scaling behavior: "a single architecture – the Transformer... – scales smoothly in all of these domains, with only minimal changes to hyperparameters such as width, depth, or learning rate" (Section 1).
Metrics. The primary metric throughout is test cross-entropy loss, measured in nats per token or nats per image/frame depending on context. For image and video domains, the paper decomposes loss into irreducible ( $L_\infty$ ) and reducible components, where the reducible loss is interpreted as an estimate of $D_{\text{KL}}(\text{True} \parallel \text{Model})$ . For multimodal models, the paper introduces empirical mutual information $I(\text{text}, \text{image})$ — the difference between the image loss with a blank caption and with a real caption — and Infogain $\equiv I / L(\text{text})$ , a ratio bounded in $[0, 1]$ that measures what fraction of available text information the model uses for image modeling. For math problems, the paper reports both loss and error rate (1 − accuracy) on interpolation and extrapolation test sets. For ImageNet classification, top-1 error rate and classification test loss are reported.
Baselines. The paper does not compare against alternative architectures — the goal is to establish scaling trends for a consistent Transformer, not to beat state-of-the-art results. The primary comparisons are:
- Across model sizes within a domain: trends in $L(N)$ for models spanning 4–5 orders of magnitude in parameter count, establishing the power-law plus constant form.
- Across compute budgets: the compute-efficient frontier $L(C)$ showing the Pareto-optimal loss achievable for a given compute budget.
- Training from scratch vs. pre-trained fine-tuning (for ImageNet classification): randomly initialized Transformers trained only on classification serve as the baseline against which pre-trained generative models (fine-tuned on classification) are compared.
- Within-modal comparisons: different image resolutions (8×8, 16×16, 32×32), different VQ encoding resolutions (16×16 vs. 32×32), different multimodal directions (text-to-image vs. image-to-text) provide relative comparisons to understand how data complexity affects scaling.
- For the inconsistency analysis (Section 6): the parametric curve $L(D(C))$ (loss achievable with a dataset of size $D = C/(6N_{\text{opt}})$ assuming one-epoch training) serves as a theoretical lower bound against which the $L(C)$ trend is compared.
Generation budget / compute accounting. Compute $C$ is defined theoretically as $C \equiv 6 N E$ , where $N$ is the non-embedding parameter count and $E = S B$ is the total number of tokens processed during training ( $S$ steps, $B$ batch size in tokens). The factor of 6 accounts for a factor of 2 from multiply-add operations being counted as two FLOPs and a factor of 3 for the forward and backward passes combined. Compute is reported in petaflop-days (PF-days, where 1 PF-day $\approx 8.64 \times 10^{19}$ FLOPs). This theoretical definition makes $C$ a fully controlled experimental variable independent of hardware efficiency — all tokens and all parameters are treated uniformly in the cost model. For images and video, the per-image or per-frame loss is obtained by multiplying per-token loss by the number of tokens per image/frame.
Cross-validation / statistical protocol. No cross-validation is used in the traditional sense, as the goal is curve-fitting rather than model selection. The paper's primary validation strategy is internal consistency: the irreducible loss $L_\infty$ is estimated independently from $L(N)$ fits (loss at convergence) and $L(C)$ fits (loss at the compute-efficient frontier), and the agreement between these estimates (e.g., 3.12 vs. 3.13 nats/token for 8×8 images in Table 1) provides a check on the reliability of the fitting procedure. For the math domain, the paper generates separate test distributions at varying difficulty levels to assess extrapolation. The paper does not report confidence intervals on fitted parameters, which is a limitation. For the compute-optimal frontier, "points on the convex hull of the loss versus compute frontier" (Section 2.3) are selected — this is a deterministic procedure that identifies the Pareto-optimal points without statistical inference. The sensitivity analysis in Figure 14 (shaded region representing ±5% variation in the $N_{\text{opt}}$ exponent) provides a qualitative assessment of how robust the inconsistency conclusions are to fitting errors, but formal uncertainty quantification is absent throughout.

Main Quantitative Results

Cross-Domain Universality of the Power-Law Plus Constant Scaling Form

The paper's central empirical finding is that equation (1.1) — $L(x) = L_\infty + (x_0/x)^{\alpha_x}$ — fits the loss across all four studied domains and both scaling variables ( $x = N$ and $x = C$ ), with high precision even when the reducible loss is much smaller than the irreducible component. Table 1 provides the complete set of fitted parameters.

Model size scaling $L(N)$ (Figure 3). Across all domains, the test loss at convergence follows the power-law plus constant form. Specific fits:

Language (from Kaplan et al., 2020): $L(N) = (N / 1.47 \times 10^{14})^{-0.070}$ with no constant term reported (the entropy of natural language is not yet measurable because even 175B-parameter models have not approached an irreducible loss).
8×8 pixel images: $L(N) = 3.12 + (N / 8.0 \times 10^1)^{-0.24}$ . The irreducible loss of 3.12 nats/token corresponds to approximately 602 nats/image (192 tokens). At $N = 10^9$ parameters, the reducible loss is approximately $(8.0 \times 10^1 / 10^9)^{0.24} \times 192 \approx 1.5 \times 10^{-2} \times 192 \approx 3$ nats/image — very close to the irreducible floor.
32×32 pixel images: $L(N) = 2.20 + (N / 6.3 \times 10^1)^{-0.13}$ . The exponent $\alpha_N = 0.13$ is substantially smaller than for 8×8 images ( $\alpha_N = 0.24$ ), meaning improvements with model size are much slower — the data is more complex and harder to model.
16×16 VQ-encoded 64×64 images: $L(N) = 3.99 + (N / 2.7 \times 10^4)^{-0.13}$ .
32×32 VQ-encoded 64×64 images: $L(N) = 3.07 + (N / 1.9 \times 10^4)^{-0.14}$ .
Video (16 frames, 16×16 VQ): $L(N) = 1.01 + (N / 3.7 \times 10^4)^{-0.24}$ . Note the exponent matches 8×8 images — the video data appears to have similar scaling properties to low-resolution images when VQ-encoded.
Math (extrapolation test): $L(N) = 0.28 + (N / 1.1 \times 10^4)^{-0.16}$ .
Multimodal text-to-image (text loss): $L(N) = (N / 5.6 \times 10^8)^{-0.037}$ . The very small exponent $\alpha_N = 0.037$ implies extremely slow improvement in text modeling with model size — the text component of this task is already relatively easy.
Multimodal text-to-image (image loss): $L(N) = 2.0 + (N / 5.1 \times 10^3)^{-0.16}$ .
Multimodal image-to-text: Very similar exponents: text loss $\alpha_N = 0.039$ , image loss $\alpha_N = 0.15$ .

A notable deviation: "image and video models with roughly $10^4$ parameters under-perform the trends, with worse performance evident for higher resolution images" (Section 2.2). The paper speculates that tiny transformers "have difficulty attending to contexts comparable in length to their non-embedding parameter count" — a 10K-parameter model with a 3072-token context (32×32 images) has roughly 3 tokens of context per parameter, which may be insufficient.

Compute scaling $L(C)$ (Figure 5, Table 1). The compute-efficient frontier — the minimum loss achievable for a given compute budget when model size is chosen optimally — also follows the power-law plus constant form:

8×8 images: $L(C) = 3.13 + (C / 1.8 \times 10^{-8})^{-0.19}$ . The irreducible loss from the compute fit (3.13) agrees well with the model size fit (3.12).
32×32 images: $L(C) = 2.21 + (C / 3.6 \times 10^{-9})^{-0.10}$ . The exponent $\alpha_C = 0.10$ is the smallest of any domain, indicating that compute provides relatively little leverage on this complex distribution.
Video: $L(C) = 0.95 + (C / 2.2 \times 10^{-5})^{-0.14}$ .
Math: $L(C) = 0.14 + (C / 1.4 \times 10^{-5})^{-0.17}$ .
Text-to-image: $L(C) = 1.93 + (C / 1.5 \times 10^{-6})^{-0.15}$ .
Image-to-text: $L(C) = 1.97 + (C / 1.5 \times 10^{-6})^{-0.16}$ .

The reducible loss trends — plotting $L(C) - L_\infty$ against $C$ on log-log axes (Figure 1) — are "pure power-laws, even when the reducible loss becomes very small" (Section 2.3). This is visually confirmed by the straight-line behavior on log-log plots, with no evidence of curvature or changing exponents as the loss approaches the irreducible floor.

Per-image reducible loss trends (Table 3). When expressed per image rather than per token, the compute scaling of the reducible loss reveals the massive increase in data complexity with resolution:

8×8 images: reducible loss $\propto (C / 1.9 \times 10^3)^{-0.19}$ , irreducible loss $\approx 602$ nats/image.
16×16 images: reducible loss $\propto (C / 1.7 \times 10^{10})^{-0.16}$ , irreducible loss $\approx 2026$ nats/image.
32×32 images: reducible loss $\propto (C / 2.7 \times 10^{26})^{-0.10}$ , irreducible loss $\approx 6806$ nats/image.

The denominators in these fits ( $C_0$ values of $1.9 \times 10^3$ , $1.7 \times 10^{10}$ , $2.7 \times 10^{26}$ PF-days) represent the compute needed to achieve a reducible loss of 1 nat/image. The staggering increase — 26 orders of magnitude difference between 8×8 and 32×32 — quantifies the practical impossibility of perfectly modeling higher-resolution image distributions: "for larger images we would need enormous quantities of compute to perfectly model the true image distribution" (Section 3.1).

Optimal Model Size Scaling $N_{\text{opt}}(C)$ Shows Near-Universal $\beta \sim 0.7$

The optimal model size for a given compute budget is extracted from the convex hull of the $L(C)$ frontier (blue points in Figure 5), and fit to a pure power-law $N_{\text{opt}} \propto C^\beta$ . Individual domain fits are shown in Figure 16, and the combined result appears in Figure 2.

Domain-specific $\beta$ values (Table 1, Figure 16):

Language (from Kaplan et al., 2020): $\beta = 0.73$ , $N_{\text{opt}} = (C / 3.3 \times 10^{-13})^{0.73}$
8×8 images: $\beta = 0.64$ , $N_{\text{opt}} = (C / 5.3 \times 10^{-14})^{0.64}$
16×16 images: $\beta = 0.75$ , $N_{\text{opt}} = (C / 4.8 \times 10^{-12})^{0.75}$
32×32 images: $\beta = 0.65$ , $N_{\text{opt}} = (C / 1.6 \times 10^{-13})^{0.65}$
Video: $\beta = 0.71$ , $N_{\text{opt}} = (C / 1.13 \times 10^{-12})^{0.71}$
Math: $\beta = 0.69$ , $N_{\text{opt}} = (C / 2.3 \times 10^{-12})^{0.69}$
Text-to-image: $\beta = 0.71$ , $N_{\text{opt}} = (C / 9.4 \times 10^{-13})^{0.71}$
Image-to-text: $\beta = 0.72$ , $N_{\text{opt}} = (C / 3.3 \times 10^{-12})^{0.72}$

The clustering around $\beta \sim 0.7$ is striking: across eight independent measurements spanning five data modalities, the exponent varies only from 0.64 to 0.75. Figure 2 displays all eight trends on a single plot, where they largely overlap. The authors explicitly state they "did not anticipate that the exponent $N_{\text{opt}} \propto C^{0.7}$ would be largely independent of the data distribution" (Section 8).

The practical implication is that for any 10× increase in compute budget, model size should increase by approximately $10^{0.7} \approx 5\times$ , while training tokens should increase by approximately $10^{0.3} \approx 2\times$ (since $C \propto N \times \text{tokens}$ ). For one-epoch training ( $D = C / (6N)$ ), this implies $D \propto N^{(1-\beta)/\beta} \propto N^{0.4}$ — the dataset size should grow sub-linearly with model size.

Image Resolution Scaling: Entropy Grows, Exponents Shrink

Varying image resolution provides a controlled way to modify data complexity and observe systematic changes in scaling parameters.

Irreducible loss increases with resolution (Table 3, Figure 7). The estimated entropy per image grows from approximately 602 nats (8×8, 192 pixels) to 2026 nats (16×16, 768 pixels) to 6806 nats (32×32, 3072 pixels). The per-pixel entropy is not constant — it decreases from 602/192 ≈ 3.1 nats/pixel at 8×8 to 2026/768 ≈ 2.6 nats/pixel at 16×16 to 6806/3072 ≈ 2.2 nats/pixel at 32×32 — indicating that as resolution increases, neighboring pixels become more correlated and individually less surprising, even as the total information in the image increases.

Scaling exponents decrease with resolution (Table 3). The reducible loss exponent $\alpha_C$ drops from 0.19 (8×8) to 0.16 (16×16) to 0.10 (32×32). Smaller exponents mean slower improvement with compute — higher-resolution images are not just higher-entropy, but also require disproportionately more compute to reduce the KL divergence by the same factor. This makes practical sense: the number of possible pixel configurations grows exponentially with pixel count, so modeling the distribution to a given fidelity requires commensurately more computation.

Consistency check across measurement methods. The irreducible losses estimated from $L(N)$ and $L(C)$ fits agree well: for 8×8 images, 3.12 vs. 3.13 nats/token; for 16×16, 2.64 vs. 2.64; for 32×32, 2.20 vs. 2.21 (Table 1). This agreement across independent measurement procedures provides validation that the power-law plus constant form is capturing a genuine property of the data.

VQ encoding changes parameters but preserves the functional form (Table 1, Figures 7–8). For 64×64 images encoded with 16×16 VQ codes, $L(N) = 3.99 + (N / 2.7 \times 10^4)^{-0.13}$ and $L(C) = 4.09 + (C / 6.1 \times 10^{-7})^{-0.11}$ . The VQ encoding increases the irreducible loss per token (3.99 vs. 2.64 for 16×16 pixel) because each VQ code carries more information than a single RGB value, but the overall scaling behavior remains well-described by the same functional form. The $\beta$ exponents for VQ-encoded images (0.64 and 0.70) are consistent with the pixel-level measurements.

Video Modeling: Per-Frame Scaling and Temporal Structure

Overall video scaling (Table 1, Figure 5). For 16-frame video clips with 16×16 VQ encoding, $L(N) = 1.01 + (N / 3.7 \times 10^4)^{-0.24}$ and $L(C) = 0.95 + (C / 2.2 \times 10^{-5})^{-0.14}$ . The $\beta = 0.71$ exponent for optimal model size is consistent with other domains.

Per-frame analysis (Figure 6, right; Figure 9). The loss per frame decreases with frame index in a power-law plus constant manner, similar to how language model loss decreases with token position (Kaplan et al., 2020). The first frame has substantially higher loss than subsequent frames — it represents an "image modeling problem" (Section 3.2) with no temporal context, while later frames benefit from conditioning on previous frames.

Final frame reducible loss (Figure 9, center and right). The reducible loss on the 16th (final) frame follows a pure power-law: $(C / 1.8 \times 10^5)^{-0.19}$ for compute and $(N / 3.85 \times 10^{13})^{-0.277}$ for model size. Extrapolating these trends, the paper forecasts that "we would need a model size around $\sim 10^{13}$ parameters and compute of around $10^4$ PF-days to achieve a loss of just a few nats/frame on the final frame" (Section 3.2).

Position-dependent loss patterns differ qualitatively from language (Figure 6). Unlike language and video models where loss follows a smooth power-law decay with position, image models show the highest loss for the first few pixels and near the center of the image. The paper notes: "power-law correlations in the context depend in an essential way on the nature of the data, and are not universal. In contrast, the form of the compute and model size scaling laws appears to be largely independent of the data distribution" (Section 2.4).

Multimodal Models: Mutual Information and Infogain Scale Logarithmically

Mutual information scaling (Figure 12, left). The empirical mutual information between text and image grows logarithmically with model size. For text-to-image models, $I \approx 0.95 \log(N / 2.9 \times 10^5)$ after finetuning, reaching approximately 8 nats at the largest models ( $N \sim 10^9$ ). For image-to-text models, $I \approx 0.63 \log(N / 9.5 \times 10^6)$ , reaching approximately 5-6 nats at the largest models.

Infogain scaling (Figure 12, right). The Infogain — mutual information normalized by text entropy — also grows logarithmically: approximately $0.015 \log(N / 5.0 \times 10^5)$ for text-to-image, reaching about 0.10 (10% of available text information used), and approximately $0.005 \log(N / 1.0 \times 10^7)$ for image-to-text, reaching only about 0.02-0.03 (2-3% of available image information used to model text).

Asymmetry between directions. Text-to-image models achieve substantially higher mutual information and Infogain than image-to-text models at the same parameter count. The paper speculates this is because "much more processing is required to extract semantic information from images than from text" (Section 4). The exponents of the logarithmic growth also differ (0.95 vs. 0.63 for mutual information), suggesting the two directions have fundamentally different scaling properties.

"A picture is worth a thousand words?" The largest image-to-text models achieve approximately 3.4 nats per word loss on text. With 8 nats of mutual information, "a 32x32 image is worth only about 2-3 words to our best models" (Section 4). The paper then extrapolates: to reach an Infogain of 0.20 (doubling the current text-to-image Infogain), "models of size $N \approx 3$ trillion parameters" would be needed.

Effect of finetuning on blanked data (Figure 28, Figure 29). The mutual information measurements are sensitive to whether blank captions/images are in-distribution. Without finetuning on a mixture of blanked and real inputs, the measured mutual information is "about twice as large" (Section 4). Training from scratch on a 95/5 mixture of multimodal and blank caption data yields very similar mutual information to the 50/50 finetuning approach (Figure 28), suggesting the measurement is robust to the specific strategy for handling blank inputs.

Mathematical Problem Solving: Extrapolation Depends on Training Performance

Loss scaling with model size at different difficulty levels (Figure 24, Figure 25). For difficulty levels within the training distribution ( $s \leq 10$ ), larger models achieve systematically lower loss and error rates. For extrapolation levels ( $s > 10$ ), the same ranking holds, but all models show smoothly degrading performance with increasing difficulty — there is no sharp phase transition at the boundary of the training distribution.

Extrapolation depends on training loss, not model size directly (Figure 13). The paper's key finding for math is that "models of different size perform nearly identically when we hold the training loss fixed" (Section 5). A 10M-parameter model at a training loss of 0.4 and a 100M-parameter model at the same training loss (achieved at different training steps) perform equally well on all difficulty levels, both interpolation and extrapolation. Larger models help only because they achieve lower training loss — they do not provide any additional "generalization ability" beyond what is captured by the training loss. This is consistent with findings in Kaplan et al. (2020) for language.

Module-level breakdown (Figure 23, Figures 26–27). Performance varies substantially by problem module. On the official extrapolation test (Figure 26), the 400M-parameter model achieves high accuracy on simpler modules like numbers__round_number_big and comparison__closest_more but struggles on compositional modules like algebra__polynomial_roots_big and the probability modules (which are excluded from the main scaling trends due to anomalous overfitting). The ordering of module difficulty is consistent across model sizes — harder modules for small models remain harder for large models, just with lower error rates.

Two anomalous modules are excluded. The paper notes that probability__swr_p_level_set_more_samples and probability__swr_p_sequence_more_samples generators cause "larger models overfitting against them and achieving worse loss (but higher accuracy) than some smaller models" (Section 2.1.6). These modules are excluded from Figures 1 and 5, as "the poor loss on these modules would dominate the trends" — a pragmatic but potentially important exclusion that limits the generality of the math scaling claims.

Dataset size scaling for math (Figure 22). The loss follows a power-law plus constant form as a function of dataset size $D$ (in millions of problems): the extrapolation loss scales approximately as $(D / 0.289)^{-0.239}$ , and the interpolation loss as $(D / 0.332)^{-0.512}$ . The steeper exponent for interpolation (0.512 vs. 0.239) makes sense — within-distribution performance improves faster with more data than out-of-distribution performance.

ImageNet Classification: Power-Law Improvement Despite Generative Loss Saturation

Classification loss vs. model size (Figure 11, top). For pre-trained models fine-tuned on 32×32 ImageNet, the classification test loss follows a clean power-law: $(N / 1.72 \times 10^{10})^{-0.105}$ . The top-1 error rate also follows a power-law: $(N / 2.09 \times 10^3)^{-0.089}$ . These trends show no evidence of bending or saturating at the largest model sizes tested (~400M parameters), even though the generative loss trend (Figure 7, 32×32) has already visibly flattened above ~10M parameters.

Pre-training advantage emerges with scale (Figure 11, top). For small models ( $N \lesssim 10^7$ ), pre-trained and randomly initialized transformers achieve similar classification performance — pre-training provides essentially no benefit. For larger models, the gap widens substantially, with pre-trained models achieving dramatically lower error rates. The paper states: "for small models, pretraining affords almost no benefit compared to training from scratch, but it greatly enhances the performance of larger models" (Section 3.4).

Faster fine-tuning for larger pre-trained models (Figure 11, bottom). The learning curves show that larger pre-trained models not only reach better final performance but also fine-tune substantially faster — they require fewer steps to reach any given classification loss. This is in contrast to models trained from scratch, where larger models do not exhibit the same acceleration.

Generative loss generalization to ImageNet (Figure 32). When YFCC100M-trained models are evaluated on ImageNet images, the generative loss follows a power-law plus constant form: $L(N) = 2.31 + (N / 1.1 \times 10^2)^{-0.17}$ nats/token. The paper notes that this suggests the loss on a new data distribution "depends only on the loss on [the training distribution], and does not otherwise depend on model size" (Appendix E, Figure 32 caption). Extrapolating to infinite $N$ would yield $L(\text{ImageNet}) \to 2.31$ , which decomposes as $D_{\text{KL}}(\text{ImageNet} \parallel \text{YFCC100M}) + S(\text{ImageNet})$ — the irreducible loss when generalizing from one distribution to another includes both the KL divergence between distributions and the entropy of the target.

Individual Image Scaling Trends (Figures 10, 17, 20, 21)

Loss percentiles follow similar power-law plus constant trends (Figure 10). The paper evaluates 32×32 image models on 1000 randomly selected test images and computes the 1st, 5th, 20th, 50th, 80th, 95th, and 99th percentiles of the per-image loss distribution at each model size. All percentiles follow power-law plus constant trends that are visually similar — the scaling behavior is not driven by a subset of unusual images but applies across the distribution.

Individual images follow consistent trends (Figure 17). Eight randomly chosen individual test images all show smooth power-law plus constant trends in their per-image loss as a function of model size, supporting the claim that "the loss on individual images scales with model size in the same way as the mean over all images in the data distribution" (Section 1.1).

Most and least improved images (Figure 20). The ten images with the largest loss reduction between a 100K-parameter model and a 400M-parameter model tend to be "images with complex colorful scenes involving people or crowds," while the least improved are "black and white images and those dominated by a simple background" (Figure 20 caption). This aligns with intuition — complex scenes contain more structure for a larger model to capture.

Conditional completions (Figure 21). Visual quality of image completions (bottom half generated given top half) improves with model size, with a "clear trend of increasing photorealism with larger models" (Figure 21 caption).

The Inconsistency Between $L(C)$ and $L(D)$ (Figure 14)

Learning curves approach $L(D)$ from above (Figure 14, left). For both 8×8 and 16×16 images, learning curves for models of increasing size lie progressively closer to the $L(D)$ trend (the loss achievable on a finite dataset with infinite model capacity). At the largest model sizes, the learning curves nearly touch $L(D)$ — optimization is becoming increasingly efficient.

The extrapolated intersection (Figure 14, right). When $L(C)$ and $L(D(C))$ are plotted on the same axes, the $L(C)$ trend (black) is steeper than $L(D(C))$ (blue dashed), and the extrapolations intersect. Beyond the intersection, $L(C)$ would predict better performance than is theoretically possible on the corresponding dataset size. The red shaded region shows that varying the $N_{\text{opt}}$ exponent by ±5% changes the intersection point dramatically, highlighting the sensitivity of the inconsistency to the precise value of $\beta$ .

Irreducible loss agreement provides a consistency check. Despite the inconsistency in the reducible trends, the estimated irreducible losses from $L(D)$ and $L(C)$ are similar: for 16×16 images, approximately 2013 nats/image from $L(D)$ and 2023 nats/image from $L(C)$ (Section 6). For 8×8 images: 599 vs. 602 nats/image. The asymptotic limit is well-determined even though the approach to it differs between measurement methods.

Ablation Studies and Robustness Checks

Image resolution and encoding: The power-law plus constant form is demonstrated for five distinct image representations — 8×8 pixel, 16×16 pixel, 32×32 pixel, 16×16 VQ, 32×32 VQ — spanning a factor of 16 in token count and fundamentally different token types (raw RGB values vs. VQ code indices). The scaling form holds for all, with systematic variation in parameters ( $L_\infty$ increases, $\alpha$ decreases with resolution). This is confirmed in Figures 7 and 8, and Table 3.

Optimal aspect ratio $d_{\text{model}}/n_{\text{layer}}$ : The optimal aspect ratio differs dramatically between language (~100) and non-language domains (~5–10). For 32×32 images (Figure 4, left), a sweep at three model sizes shows optimal performance around aspect ratio 5–10. For math (Figure 4, center), similar optimum. For multimodal text-to-image at 27M parameters (Figure 4, right), the optimum is around aspect ratio 20–50. The paper also tested the alternative convention for fully-connected vs. attention layer scaling ( $m_{\text{mlp}} = 4, m_{\text{attn}} = 1$ for language/multimodal vs. $m_{\text{mlp}} = 1, m_{\text{attn}} = 1/4$ for images/video/math), finding it improved training stability for deep models (Appendix B.1).

Multimodal model direction: Both text-to-image and image-to-text models follow the same scaling forms, with similar exponents for image loss ( $\alpha_N = 0.16$ vs. $0.15$ ) and text loss ( $\alpha_N = 0.037$ vs. $0.039$ ), confirming that the scaling behavior is not specific to one prediction direction. This is shown in Table 1 and Figure 3.

Mutual information measurement robustness: Two different methods for making blank inputs in-distribution — finetuning for 10K steps on 50/50 blank/real mixture and training from scratch on 95/5 mixture — yield "very similar" results (Section 4, Figure 28). The logarithmic scaling of mutual information is robust to the specific strategy.

Math problem difficulty levels: Extrapolation performance is tested across a continuous range of difficulty levels ( $s = 1$ to $19$ ) far beyond the training distribution ( $s \leq 10$ ). The smooth degradation (no sharp phase transition at $s=10$ ) visible in Figure 24 suggests the model is learning generalizable algorithmic skills, not just memorizing problem templates. The relationship between training loss and extrapolation loss (Figure 13) is tested at five difficulty levels spanning interpolation ( $s = 2, 5, 9$ ) and extrapolation ( $s = 13, 17$ ), showing consistent mapping.

Individual image and percentile analysis: The power-law plus constant form is validated not just for mean loss but for individual images (Figure 17), percentiles of the loss distribution (Figure 10), and final video frames (Figure 9). This is an important robustness check: if the scaling trends were an artifact of averaging over diverse images with qualitatively different scaling behavior, the individual-image analysis would reveal it. The consistency across images and percentiles suggests the trend is a genuine property of how models improve.

Model size range: The paper trains models spanning approximately 4–5 orders of magnitude in parameter count (from ~10K to ~400M non-embedding parameters for non-language domains), providing a wide enough range to distinguish power-law plus constant from alternative functional forms. The clean fits over this range increase confidence in the extrapolations.

Sparse attention (images/video) vs. dense attention (math): Both attention patterns produce models that obey the same scaling laws, suggesting the scaling form is robust to the specific attention mechanism as long as the architecture is fundamentally a Transformer.

Convergence vs. compute frontier: The irreducible loss is estimated from two independent procedures — convergence under infinite data ( $L(N)$ ) and the compute-efficient frontier ( $L(C)$ ) — and the estimates agree to within ~0.5% (Table 1). This cross-validation is a powerful check that the fitting procedure is extracting a genuine property of the data rather than an artifact of a particular experimental design.

Negative result — training loss for math does not follow clean power-law: The paper notes that math training loss "do[es] not adhere as well to a power-law form, perhaps because of the implicit curriculum in the frequency distribution of easy and hard problems" (Section 2.1.6). This is an informative negative result — it identifies a condition (non-uniform sampling from the distribution, creating an implicit curriculum) under which the scaling form breaks down.

Negative result — two math modules exhibit anomalous overfitting: The probability__swr_p_level_set_more_samples and probability__swr_p_sequence_more_samples modules show larger models achieving worse loss (but better accuracy) than smaller models, violating the monotonic scaling observed in other modules. These are excluded from the main trends, which is a limitation: the scaling laws do not apply uniformly even within the math domain.

Critical batch size unaccounted for: The paper acknowledges that it does not adjust compute estimates for the critical batch size effect (McCandlish et al., 2018), which Kaplan et al. (2020) accounted for. The authors state "for large model sizes and compute budgets these effects should be small" (Section 2.3), but this remains an untested assumption that could introduce systematic error, particularly at smaller compute budgets where batch sizes may exceed the critical batch size.

Critical Assessment

The experiments in this paper represent a landmark in demonstrating the breadth of neural scaling laws, but the strength of evidence varies considerably across the paper's claims. Here, I examine each major claim against the experimental evidence.

Claim: The power-law plus constant form $L(x) = L_\infty + (x_0/x)^{\alpha_x}$ applies to all four domains.

This is the paper's strongest claim and the one most directly supported by the experiments. The fits in Table 1 and the visual evidence in Figures 3, 5, and 8 show remarkably consistent adherence to this functional form across domains. The reducible loss trends in Figure 1 are straight lines on log-log plots over 4–6 orders of magnitude in compute, which is exactly the signature of a pure power-law.

However, several caveats apply. First, the fits involve three free parameters ( $L_\infty$ , $x_0$ , $\alpha$ ) and the number of distinct model sizes per domain is typically 6–8 (visible as data points in the figures). With three parameters and 6–8 points, the risk of overfitting is non-trivial — alternative functional forms with similar numbers of parameters (e.g., $L_\infty + A \exp(-B x)$ or $L_\infty + A / (\log x)^B$ ) might fit nearly as well. The paper does not compare goodness-of-fit across alternative functional forms. The exceptional case of math training loss (which does not follow the form, per Section 2.1.6) suggests the fit is not vacuous — the form can fail, and does fail in at least one case. But absent formal model selection, the claim rests primarily on visual inspection and the consistency of $L_\infty$ across independent measurement procedures.

Second, the smallest models systematically deviate from the trends for high-resolution images and video. The paper notes this explicitly and speculates it is due to context length exceeding parameter count, but this means the power-law plus constant form is not a complete description — it fails at the small-model extreme. The domain of validity matters for extrapolation: if trends break down at small scale, they might also break down at scales beyond those studied.

Claim: $L_\infty$ estimates the entropy of the data distribution, and the reducible loss estimates $D_{\text{KL}}$ .

This claim requires the assumption that an infinitely large Transformer can perfectly model the data distribution. The paper states this assumption but does not validate it. The strongest evidence in favor is the agreement between $L_\infty$ extracted from $L(N)$ and $L(C)$ fits (Table 1), which suggests the constant is a genuine asymptote rather than a fitting artifact. The systematic increase of $L_\infty$ with image resolution (602 → 2026 → 6806 nats/image) is also consistent with the entropy interpretation — higher resolution should mean more information.

However, the paper cannot exclude the possibility that $L_\infty$ represents a model-specific irreducible loss — the best a Transformer can do, which might be strictly greater than the true entropy if Transformers have architectural limitations that prevent them from representing certain distributions. The 8×8 image case, where billion-parameter models approach within a few nats of $L_\infty$ , provides the best evidence because the residual $D_{\text{KL}}$ is small enough that further reductions are plausible. For higher-resolution images where the reducible loss remains orders of magnitude larger, the extrapolation to zero $D_{\text{KL}}$ is far more speculative.

The math domain presents an interesting test case: the true entropy of the answer given the question is zero, so $L_\infty$ should be zero if the model can perfectly solve all problems. The extrapolation fit gives $L_\infty = 0.28$ nats/token (or 0.14 for compute scaling) — not zero. This could indicate either that (a) the models cannot fully solve certain problem types, so the true achievable asymptotic loss is positive, or (b) the power-law plus constant form is an approximation that breaks down as the loss approaches zero. The paper does not pursue this tension.

Claim: $N_{\text{opt}} \propto C^\beta$ with $\beta \sim 0.7$ across all domains.

The evidence for this claim is visually compelling (Figure 2) and the consistency across eight independent fits is striking. The range of $\beta$ values (0.64 to 0.75) is narrow, and the clustering around 0.7 is unlikely to be coincidence.

However, the extraction of $N_{\text{opt}}(C)$ is more fragile than the figures suggest. The procedure selects points on the convex hull of the $L(C)$ scatter — a small number of points (the blue points in Figure 5, typically 3–5 per domain) that define the Pareto frontier. Fitting a power-law to 3–5 points is statistically weak; small measurement errors in individual models' loss or compute can shift the selected frontier points and change the fitted exponent. The paper does not report uncertainty on $\beta$ , which would be large given the degrees of freedom. The sensitivity analysis in Figure 14 (shaded region for ±5% variation in $\beta$ ) illustrates that even small changes in $\beta$ dramatically alter the intersection point with $L(D)$ — which both demonstrates the importance of precise $\beta$ estimation and highlights how imprecise that estimation actually is.

Furthermore, $\beta$ is estimated from models trained without critical batch size adjustment. Kaplan et al. (2020) found that accounting for critical batch size changed their language $\beta$ estimate. The paper's assumption that this effect is "small" at large scale may be reasonable, but it is not verified. If critical batch size effects bias the compute estimates differently for different model sizes, the slope of $\log N_{\text{opt}}$ vs. $\log C$ could be systematically distorted.

Claim: Downstream performance continues to improve even as generative loss saturates.

The ImageNet fine-tuning experiments (Figure 11) provide strong evidence for this claim at 32×32 resolution. The classification error follows a clean power-law with no sign of bending, even though the generative loss has largely flattened above ~10M parameters. This is a genuinely important and well-demonstrated result.

However, the claim's generality is limited by several factors. The experiment is conducted only at 32×32 resolution (downsampled ImageNet), which is far from the resolution used in practical image classification. The classification architecture is simple (mean pool + single linear layer), and it is unclear whether more sophisticated classification heads would show the same scaling. The experiment is only for image classification; whether similar decoupling between generative loss and downstream performance holds for video, multimodal, or math tasks is not tested. Finally, the pre-training is on YFCC100M and fine-tuning on ImageNet — two different image distributions — so the result is partly about transfer learning, not purely about representation quality on the pre-training distribution.

Claim: The inconsistency between $L(C)$ and $L(D)$ reflects an optimization efficiency limit that will be reached at larger scale.

This is the most speculative claim in the paper, and the evidence is more suggestive than conclusive. The observation that learning curves approach $L(D)$ more closely with increasing model size (Figure 14, left) is real and supports the hypothesis. But the paper has not demonstrated that the learning curves will actually reach $L(D)$ — the extrapolation of this trend to larger scales is itself an untested scaling assumption. The alternative explanation — that $L(D)$ , $L(C)$ , or $N_{\text{opt}}(C)$ will deviate from their current power-law trends before the intersection point — is equally plausible and cannot be ruled out with the available data.

The paper deserves credit for explicitly identifying and analyzing this inconsistency rather than ignoring it. But the resolution is presented as a hypothesis, not a finding, and should be understood as such.

Overall experimental strengths. The paper's experimental design has several notable strengths: (a) the controlled comparison across domains with identical architecture isolates data modality as the independent variable; (b) the wide range of model sizes (4–5 orders of magnitude) provides genuine leverage for curve-fitting; (c) the cross-validation of $L_\infty$ from independent measurement procedures is a clever consistency check; (d) the per-image and percentile analyses demonstrate that the scaling trends are not artifacts of averaging; (e) the inclusion of negative results (math training loss, anomalous probability modules) adds credibility.

Overall experimental weaknesses. Several weaknesses limit the strength of the conclusions: (a) the number of model sizes per domain (6–8 per fit) is modest for three-parameter fits, and no formal model comparison against alternative functional forms is performed; (b) critical batch size effects are not accounted for, potentially biasing $N_{\text{opt}}(C)$ estimates; (c) the absence of confidence intervals on fitted parameters makes it impossible to assess whether differences between domains (e.g., $\beta = 0.64$ vs. $\beta = 0.75$ ) are statistically significant or within measurement noise; (d) the exclusion of two math problem generators from the main trends, while pragmatically justified, means the math scaling laws are conditional on selecting well-behaved modules; (e) all experiments use a single architecture (decoder-only Transformer) and are not validated across alternative architectures, leaving open the possibility that the scaling laws are partially architecture-specific; (f) the mutual information measurements depend on the method for handling blank inputs, and while two methods give similar results, the sensitivity analysis is limited to a single check; (g) the ImageNet fine-tuning results are at an unrealistically low resolution (32×32) and the classification head is simplistic.

Missing experiments that would have strengthened the paper. The paper would benefit from: (a) training more model sizes per domain, particularly at intermediate scales, to better constrain the three-parameter fits; (b) comparing goodness-of-fit against alternative functional forms to quantify how much better the power-law plus constant form performs; (c) measuring critical batch sizes and adjusting compute estimates accordingly, following Kaplan et al. (2020); (d) reporting bootstrap confidence intervals on all fitted parameters; (e) testing whether the scaling laws hold for non-autoregressive or encoder-decoder architectures; (f) studying dataset size scaling $L(D)$ for non-math domains, which would provide another independent estimate of $L_\infty$ and more data on the inconsistency problem; (g) conducting the ImageNet fine-tuning experiment at higher resolutions and with more sophisticated classification heads to assess generality; (h) exploring whether the logarithmic mutual information scaling continues to larger models or plateaus.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Headline Efficiency Numbers

The compute-optimal allocation framework's central practical claim—that a smaller model with adaptive test-time compute can outperform a ~14× larger model—depends on the ability to estimate prompt difficulty before allocating the inference budget. The paper's method for doing so carries a cost that is never included in the reported efficiency gains.

The paper generates 2048 samples per question to estimate difficulty, either by checking ground-truth correctness (oracle bins) or averaging the PRM's final-answer scores (predicted bins). The authors acknowledge this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The magnitude of this omission is substantial. At 2048 samples per question, the difficulty estimation step alone consumes 4–8× more compute than the largest test-time budgets studied (256–512 generations). The paper's headline claim of "more than 4× better efficiency over a standard best-of-N baseline" compares compute-optimal allocation after difficulty is known against best-of-N without difficulty estimation. If the 2048-sample estimation cost were amortized across questions—say, by the paper's own suggestion of "pretraining or finetuning models to directly predict difficulty of a question" (Section 8)—the amortized per-question cost might be acceptable in a high-volume deployment. But the paper provides no such amortization analysis, no difficulty prediction model, and no accounting for the exploration-exploitation tradeoff between the samples spent estimating difficulty and the samples spent solving the problem. This means the reported efficiency gains should be understood as an upper bound, not a realized deployment figure. A practitioner implementing this system would need to add the difficulty estimation cost to their total inference budget, which could eliminate or even reverse the advantage over simpler allocation strategies, particularly for low-volume applications where the per-question amortization is impossible.

The paper partially acknowledges the gap by framing it as "a key avenue for future work" (Section 3.2), but the limitation is structural: until a cheap difficulty estimator exists, the compute-optimal framework as described is not deployable at the reported efficiency levels.

All Results Are on a Single Benchmark with a Single Model Family

Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021, 500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but no evidence is provided to support this claim of representativeness, and several aspects of the findings depend on properties that may not transfer.

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s specific output distribution—its calibration, its error patterns, its tendency to produce certain kinds of mistakes. A model with different calibration (e.g., better or worse at self-assessing correctness) could exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (e.g., GPT-4 vs. LLaMA vs. PaLM). The MATH benchmark consists exclusively of competition-level math problems requiring multi-step symbolic reasoning—a domain where correctness is well-defined, answers are verifiable, and the base model's pass@1 distribution has clear structure (some problems are easy, some are impossible). Whether the difficulty-dependent patterns observed here (beam search hurting easy problems, revisions helping easy problems, no method helping the hardest problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge is unknown.

The test set of 500 questions, split into five quintiles (~100 each) and further split by two-fold cross-validation (~50 per fold per bin), means the compute-optimal policy is selected based on approximately 50 questions per bin. No confidence intervals are reported on the compute-optimal scaling curves (Figures 4, 8), making it impossible to assess whether the observed advantage of one strategy over another at a given budget-difficulty combination is statistically reliable or within sampling noise. The paper states that it uses "two-fold cross-validation within each difficulty bin" (Section 3.2) to avoid circularity, but does not quantify the variance of the resulting policy estimates. At 50 questions per fold, a single question's anomalous behavior could flip the estimated optimal strategy for an entire difficulty bin.

The paper does not attempt any out-of-domain evaluation—no code generation benchmarks, no logical reasoning, no reading comprehension—so the scope of applicability is entirely unknown. This is not a minor caveat: a practitioner deciding whether to invest in PRM training, revision model fine-tuning, and difficulty estimation infrastructure for their domain needs evidence that the approach transfers. The paper provides none.

Verifier Over-Optimization Limits the Scaling Ceiling and Is Not Solved by Adaptive Allocation

The paper documents verifier over-optimization as a central limiting factor and then routes problems away from aggressive search where possible, but this mitigation does not solve the underlying problem—it only avoids it in certain regimes. On medium-difficulty problems where beam search is deployed (the regime where PRM guidance most helps), the over-optimization ceiling still caps performance as budget increases.

The evidence is concrete and consistent across multiple results. In Figure 3 (right), beam search ( $M=4$ ) accuracy on bin 3 (medium) problems improves from ~15% at 4 generations to ~34% at 256 generations, but the curve visibly flattens above ~64 generations—doubling the budget from 64 to 256 yields only a few percentage points of improvement. On bin 1 (easy) problems, beam search degrades from ~78% to ~77% as budget increases from 4 to 256, while best-of-N weighted continues improving to ~88%. Lookahead search—the most powerful optimizer of the PRM signal—paradoxically performs worst overall in the aggregate comparison (Figure 3, left), falling below both beam search and best-of-N at the same generation budget because its extra per-step cost reduces effective beam count while providing no net benefit in regions where the PRM is unreliable.

The qualitative examples in Appendix M reinforce this: search produces degenerate outputs (repetitive low-information steps, overly short 1–2 step solutions) that score highly under the PRM but are incorrect. The paper's own analysis identifies "low-information repetitive steps at the end of solutions (visible in Figure 29) and overly short 1–2 step solutions" as specific failure modes (Section 5.3).

The compute-optimal policy mitigates this by routing easy problems to best-of-N (where verifier over-optimization is avoided) and deploying beam search only on medium problems (where the PRM signal is still net beneficial). But this is a routing strategy, not a solution. On the medium problems where beam search is used, the verifier over-optimization ceiling remains—it is simply that the alternative (best-of-N) performs even worse on these problems, so beam search wins despite its limitations. The paper does not attempt to improve the PRM's robustness to adversarial optimization (e.g., via adversarial training, ensemble verification, or constrained search with KL penalties), and the compute-optimal policy is fundamentally bounded by the PRM quality that was achieved with the Monte Carlo rollout training procedure described in Appendix D. A practitioner building on this work would find that their achievable scaling ceiling depends critically on PRM quality, and the paper provides no guidance on how to push that ceiling upward—only on how to work around it.

The Hardest Problems (Bin 5) Show Near-Zero Improvement Regardless of Method or Budget

Across every method, every budget level, and every combination studied, the hardest questions in difficulty bin 5 show essentially no improvement over baseline. This is not a minor edge case—it is a fundamental capability bound that the paper documents clearly but whose implications it does not fully explore.

In Figure 3 (right), bin 5 accuracy hovers at 1–3% for ALL methods (beam search, best-of-N, lookahead) and ALL budgets (4, 16, 64, 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at a fixed budget of 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%—no amount of compute-optimally allocated test-time compute helps. The largest observed bin 5 accuracy appears to be approximately 3% at the 256-generation budget in Figure 3.

This failure is not a problem with any specific method—it reflects a fundamental property of the approach. When the base model's pass@1 on a problem class is near zero (recall that difficulty bins are defined by the base model's pass@1 rate over 2048 samples), there are essentially no correct solutions in the proposal distribution to find through search or refine through revision. Test-time compute can amplify existing capability—surfacing correct solutions that exist at low probability in the model's output distribution—but it cannot create capability from nothing. The paper acknowledges this in the Section 7 takeaway box, but the practical consequence is underappreciated: the compute-optimal framework provides no path forward for the problems that matter most. If your deployment's value proposition depends on solving genuinely hard reasoning problems (the kind that your current model fails on almost always), this approach offers zero leverage. You must improve the base model through pretraining, fine-tuning, or architectural innovation. The compute-optimal allocation framework is best understood as a way to extract more value from problems the model can sometimes solve, not as a way to expand the model's capability frontier.

The paper does not analyze what fraction of typical deployment problem distributions would fall into bin 5, nor does it provide guidance on how to identify the bin-5 boundary before deployment. A practitioner needs to know: for my specific use case, what fraction of queries will this approach help with versus being completely ineffective?

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate

The iterative revision mechanism has a significant practical flaw: when the model produces a correct answer during a revision chain, approximately 38% of the time it will "revise" that correct answer into an incorrect one at the next step. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

This is a direct and acknowledged consequence of the training data construction procedure. The revision model was trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). During training, the model never sees a trajectory containing a correct intermediate answer, so it has no signal for what to do when the current answer is already correct—it has been optimized to always produce a revision that changes the answer, regardless of correctness.

The paper mitigates this with within-chain selection: a verifier or majority vote is applied across all steps of the revision chain, picking the best answer from any step rather than always taking the final revision. This boosts performance (the paper's reported results all use this mitigation), but it introduces a dependency on the selection mechanism that may not scale. If the selection mechanism itself has a non-trivial error rate, the probability of selecting an incorrect answer from a chain with multiple candidates may be non-negligible. More fundamentally, the mitigation does not solve the underlying problem—it patches around it. A 38% reversion rate means that approximately 4 out of 10 revision chains that "find" the correct answer at some intermediate step will subsequently lose it. This is a substantial efficiency loss: compute is spent on revisions that actively degrade solution quality.

The failed ReST $^{EM}$ experiment (Appendix K, Figure 16) demonstrates that the problem is sensitive to training methodology and that naive attempts at improvement can backfire. When the revision model was further optimized using ReST $^{EM}$ (Singh et al., 2024), performance with sequential revisions actually degraded: at 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection in ReST $^{EM}$ exacerbates spurious correlations in revision data," but the deeper implication is that the revision training pipeline is fragile and the positive results depend on specific choices (offline data construction, edit-distance-based incorrect-correct pairing, single-epoch training) whose robustness is not well-characterized.

A practitioner deploying this system would face a choice: either accept the 38% reversion rate and depend on within-chain selection to recover, or invest in a more principled solution (such as training the model to recognize when no revision is needed, or using a confidence threshold to decide whether to continue revising). The paper provides no guidance on the latter and no analysis of how the reversion rate scales with model size or revision chain length.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The FLOPs-matched comparison in Section 7—which is the paper's central argument for the practical value of compute-optimal test-time scaling—pits the smaller model with adaptive inference against a larger model that is not necessarily the strongest possible pretraining baseline. Two specific weaknesses in the baseline weaken the paper's conclusion that "test-time compute with a smaller model can outperform a ~14× larger model."

First, the larger model scales parameters only, keeping training data fixed. The paper acknowledges this departure from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7)

The canonical approach referenced is the LLaMA paradigm (Touvron et al., 2023), which overtrained smaller models on more data rather than following Chinchilla-optimal scaling where data and parameters increase together (Hoffmann et al., 2022). A compute-optimally trained larger model—with both parameters and training data scaled—would likely outperform the parameter-scaled-only baseline used in the paper. The magnitude of this effect is not estimated. If the Chinchilla-optimal baseline is significantly stronger, the reported advantages of test-time compute (e.g., +27.8% relative improvement on easy questions at $R \ll 1$ ) would shrink or potentially reverse.

Second, the larger model uses only greedy decoding—no majority voting, no best-of-N, no search, no revision. The paper's own results demonstrate that even simple best-of-N (without the compute-optimal adaptive machinery) provides substantial improvements over single-sample generation: PRM best-of-N weighted improves from ~16% at 1 generation to ~38% at 256 generations in Figure 3. Giving the larger model even a modest test-time compute budget—say, best-of-8 or best-of-16—would create a substantially stronger baseline. The FLOPs-matched calculation could be adjusted to account for this (the larger model with best-of-K uses more inference FLOPs per query, shifting the $R$ ratio), but this adjustment is not performed. The comparison is therefore between a smaller model with sophisticated inference-time strategies and a larger model with the simplest possible inference strategy—an asymmetric comparison that favors the test-time compute approach.

Neither of these weaknesses is hidden—the paper is transparent about both the parameter-only scaling choice and the greedy decoding baseline. But the cumulative effect is that the paper's most attention-getting claim (the $4\times$ efficiency gains and the FLOPs-matched advantage over pretraining) is tested against a baseline that may significantly understate what pretraining alone can achieve. A practitioner deciding between "train a larger model" and "train a smaller model with sophisticated inference" needs to compare against the best available pretraining approach, not against a deliberately simplified one. The paper does not provide that comparison.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that the scaling laws previously identified for language modeling are not a peculiarity of natural language's statistical structure but rather a near-universal property of autoregressive Transformer training across diverse data modalities. This is a foundational empirical contribution rather than a paradigm shift — it doesn't propose a new theory of why scaling laws exist, but it dramatically expands their domain of applicability and transforms how they can be used.

The most significant conceptual shift is the repurposing of scaling laws from forecasting tools into measurement instruments for data complexity. Before this paper, scaling laws answered the question "how much compute do I need to reach loss $L$ ?" After this paper, they also answer "how much intrinsic uncertainty does this data distribution contain?" The identification of the irreducible loss $L_\infty$ with the entropy $S(\text{True})$ — and the systematic validation of this identification through independent $L(N)$ and $L(C)$ estimates, through resolution scaling, and through the consistency of per-image entropy growth — provides a methodology for estimating a fundamental information-theoretic quantity that is otherwise unmeasurable for real-world distributions. The measurement at 8×8 resolution (~602 nats/image, with billion-parameter models approaching this floor to within a few nats) provides the clearest demonstration: a practitioner can now say, for any resolution and encoding scheme, approximately how much entropy the data contains simply by extrapolating scaling trends.

This reframing changes the research landscape in several concrete ways. Verifier over-optimization and data saturation can now be distinguished. The paper's decomposition $L = S(\text{True}) + D_{\text{KL}}(\text{True} \parallel \text{Model})$ gives separate handles on the irreducible uncertainty in the data (which no model can eliminate) and the model-specific gap that better architectures or training might close. When a generative loss curve flattens, the scaling law analysis can now determine whether the model is approaching the true data entropy (in which case further scaling genuinely has diminishing returns) or merely hitting a model-class-specific asymptote (in which case architectural innovation could unlock further improvement). This distinction was impossible before this framework existed.

The near-universality of $\beta \sim 0.7$ constrains theories of neural network scaling. The paper's most surprising finding — that the optimal model size exponent $N_{\text{opt}} \propto C^{0.7}$ is essentially identical across images (0.65–0.75), video (0.71), math (0.69), multimodal (0.70–0.72), and language (0.73) — is not predicted by any existing theory. The fact that this exponent is insensitive to data modality, even as the loss scaling exponents $\alpha_C$ vary from 0.10 (32×32 images) to 0.19 (8×8 images), means that $\beta$ reflects something fundamental about Transformer optimization dynamics rather than data-dependent properties. This finding rules out any theory of scaling that predicts a strong dependence of $\beta$ on data statistics, and it provides a sharp empirical target that future theoretical work must explain. The paper's own description — that "these empirical regularities beg for theoretical explanation" (Section 8) — is an understatement; the consistency of $\beta$ across eight independent measurements is the strongest single empirical regularity the scaling laws literature has produced, and explaining it should be a priority for theoretical ML research.

The paper reconciles contradictory impulses about model size versus training duration. The implication $D \propto N^{0.4}$ for compute-optimal dataset size — derived from $\beta \sim 0.7$ and the assumption of single-epoch training — ran "somewhat counter to conventional wisdom" (Section 2.3) that assumed roughly linear scaling of data with parameters. The paper doesn't fully resolve this tension (the inconsistency analysis in Section 6 shows the single-epoch assumption itself must break down at larger scales), but it reframes the debate from heuristics to a quantitative tradeoff grounded in measured exponents. The practical guidance that "larger compute budgets should be 'spent' mostly on larger models, rather than much longer training runs" (Section 8) has measurable support across five data types, making it actionable beyond language.

The decoupling between generative loss and representation quality redirects attention toward downstream scaling laws. The ImageNet classification result — where error rate follows a clean power-law $(N / 2.09 \times 10^3)^{-0.089}$ even as the generative loss visibly saturates — is one of the paper's most important findings for practitioners. It demonstrates that generative loss can be a misleading proxy for representation quality in the large-model regime. The "last few bits" near the irreducible loss contain substantial semantic information relevant to discriminative tasks. This finding has methodological implications: evaluating scaling laws only through generative loss may systematically underestimate the practical benefits of scale. It also motivates a research program — directly measuring scaling laws for downstream task performance rather than relying on generative loss as an intermediary — that subsequent work (e.g., Hernandez et al., 2021) has pursued.

The inconsistency analysis identifies a productive open problem rather than a failure mode. Section 6 transforms what could have been dismissed as a curve-fitting artifact into a specific, testable hypothesis: that as models scale, optimization becomes sufficiently efficient that multi-epoch training provides no benefit, causing the $L(C)$ frontier to merge with $L(D(C))$ . This hypothesis makes a clear prediction that can be tested at larger scale: if true, the gap between learning curves and $L(D)$ should systematically shrink with $N$ , eventually reaching zero. The paper provides initial evidence for this trend (Figure 14, left) but not a definitive test. The shaded sensitivity analysis (the ±5% variation in $\beta$ changing the intersection point by orders of magnitude) demonstrates both the importance and the current imprecision of the measurement, setting up a clear experimental target.

Less attractive research directions. The paper implicitly argues against modality-specific architectures for the purpose of improving scaling exponents. By demonstrating that a uniform Transformer achieves consistent scaling laws across images, video, multimodal, and math data — each with appropriate but minimal hyperparameter adjustments — the paper suggests that the scaling behavior is not primarily determined by architecture (within the Transformer family) but by properties of the data distribution. Research programs that focus on novel architectures to improve scaling exponents may be less promising than programs that focus on understanding why the current architecture scales as it does, and how to change data representation (e.g., VQ encoding, resolution) to improve scaling behavior.

Follow-Up Research This Work Enables

Critical batch size adjustment across domains to refine $\beta$ estimates. The paper's $N_{\text{opt}}(C)$ extraction explicitly omits the critical batch size adjustment that Kaplan et al. (2020) applied to language models. The paper acknowledges this (Section 2.3) and argues the effect is "small" at large scale, but the sensitivity analysis in Figure 14 shows that even a 5% change in $\beta$ dramatically alters the intersection point with $L(D)$ . A direct follow-up would measure the critical batch size as a function of model size for one non-language domain (e.g., 16×16 images, which showed $\beta = 0.75$ , the highest among all measurements) and recompute $N_{\text{opt}}(C)$ with adjusted compute estimates. This would answer: is the apparent variation in $\beta$ across domains (0.64 to 0.75) partly an artifact of uncorrected batch size effects, or is it real? If real, the variation contains information about domain-specific optimization dynamics; if artifactual, the corrected $\beta$ might be even more tightly clustered around 0.7, strengthening the universality claim.

Testing the inconsistency resolution hypothesis at larger scale. Section 6 hypothesizes that the $L(C)$ and $L(D(C))$ inconsistency resolves because optimization becomes sufficiently efficient at large scale that learning curves reach $L(D)$ in a single epoch. This is testable: train a large model (say, $N \sim 10^9$ for 16×16 images) on datasets of varying size, and measure the number of epochs needed to reach within $\epsilon$ of the minimum loss. The hypothesis predicts that this "epochs to convergence" decreases systematically with $N$ , approaching 1 at some scale. The paper already observes this trend qualitatively (Figure 14, left), but a quantitative measurement — epochs to convergence as a function of $N$ , with a fit projecting the intersection point — would either validate the proposed resolution or reveal that the inconsistency requires a different explanation (e.g., $L(D)$ itself deviating from power-law at large $D$ , or $N_{\text{opt}}(C)$ steepening).

Extending the entropy estimation methodology to higher-resolution images with controlled compute. The paper estimates that 32×32 pixel images have an irreducible loss of ~6806 nats/image, but the reducible loss exponent $\alpha_C = 0.10$ for this resolution means closing the remaining $D_{\text{KL}}$ gap requires compute far beyond current capabilities. A natural extension trains on intermediate resolutions (e.g., 24×24, 48×48, 64×64 pixels) and fits scaling laws at each resolution to produce a scaling of entropy with resolution: $S(R \times R) \propto R^\gamma$ . The paper already provides three points on this curve (8×8: 602, 16×16: 2026, 32×32: 6806 nats/image), suggesting roughly $S \propto R^{2.5}$ , but this is too few points to determine whether the exponent stabilizes or changes at higher resolution. Additional resolutions would also test whether the power-law plus constant form remains valid when the total loss spans multiple orders of magnitude rather than the factor of ~3 between 8×8 and 32×32 that the current data covers.

Scaling laws for mutual information across diverse multimodal datasets. The logarithmic scaling of mutual information ( $I \approx \lambda \log(N/N_c)$ ) is demonstrated on a single web-scraped multimodal dataset with 32×32 images and 128-token captions. The Infogain bound (Infogain < 1) provides a hard extrapolation target, but the paper's data reaches only Infogain ~0.10 for text-to-image models. A natural follow-up would measure mutual information scaling on datasets where the image-text correlation is systematically varied: datasets with precise captions (e.g., image paragraphs, where captions are detailed descriptions), datasets with loose captions (e.g., social media alt-text), and datasets where captions are randomized. The hypothesis is that $\lambda$ (the logarithmic growth rate) depends on the true mutual information of the data distribution — tightly correlated image-text pairs should show faster growth (larger $\lambda$ ) because there is more cross-modal information for the model to extract at each scale. This would connect the scaling law parameters directly to data distribution properties, moving from description to explanation.

Verifier over-optimization characterization as a function of PRM training data scale. The paper identifies PRM over-optimization as the primary bottleneck for test-time compute scaling (beam search degrading easy-problem performance in Figure 3, lookahead search underperforming overall) but provides no systematic study of how over-optimization depends on PRM quality. A targeted experiment would train PRMs on varying amounts of Monte Carlo rollout data (from, say, $10^3$ to $10^6$ rollouts per training question), then measure the compute budget at which beam search performance peaks and begins to decline for each PRM. This would produce a PRM quality versus over-optimization threshold curve — essentially, how much compute can you safely spend on search before the verifier is exploited, as a function of how much data you invested in training the verifier. The paper's current results (beam search peaks around 64 generations for bin 3, degrades above that) are for a single PRM training budget. Measuring the tradeoff would provide practical guidance for practitioners deciding how to allocate their total compute budget between PRM training and PRM-guided search.

Revision model training with correct-answer-aware data construction to eliminate the 38% reversion rate. The paper identifies the 38% correct-to-incorrect reversion rate as a direct consequence of training only on incorrect-to-correct trajectories, and mitigates it with within-chain selection rather than solving it. A direct improvement would train a revision model on trajectories that include correct intermediate answers, with a special "no revision needed" token when the current answer is already correct. The training data would be constructed by generating solution chains, identifying which steps are correct (using ground truth), and constructing training sequences where the model learns to (a) revise incorrect answers and (b) output a stop token when the answer is already correct. The key measurement is whether the reversion rate drops below, say, 5%, and whether this improves the sequential revision scaling curve (Figure 6, right) — particularly at high generation budgets where the current model's performance plateaus partly due to this regression problem. The ReST $^{EM}$ failure (Appendix K) suggests that on-policy data collection may amplify training pathologies, so the experiment should compare offline data construction (like the paper's main approach) with on-policy collection.

FLOPs-matched comparison against a Chinchilla-compute-optimal baseline. The paper's most attention-getting claim — that test-time compute with a smaller model can outperform a ~14× larger model — is tested against a parameter-only-scaled baseline that the paper acknowledges is not compute-optimal. A direct follow-up would construct a Chinchilla-compute-optimal baseline: allocate the same total FLOPs to pretraining a model with both parameters and data scaled according to Chinchilla laws (equal scaling of both), then compare against the smaller model with compute-optimal test-time allocation. The experiment requires training a Chinchilla-optimal model at the matched FLOPs budget (not simply taking a parameter-scaled model and calling it the baseline), which is computationally expensive but necessary to determine whether the paper's headline result reflects a genuine advantage of test-time compute or merely the suboptimality of the parameter-only pretraining baseline. If the Chinchilla-optimal baseline closes most of the gap, it would significantly change the practical advice: test-time compute helps against undertrained large models but not against compute-optimally trained ones.

Practical Applications and Downstream Use Cases

Entropy auditing for data compression and storage decisions. The paper's methodology for estimating $L_\infty$ provides a principled way to answer: "how much can this dataset be compressed, and is it worth investing in better compression?" For example, the finding that YFCC100M images at 8×8 resolution have $S \approx 602$ nats/image means that even a perfect lossless compressor cannot reduce the average image below ~87 bytes (602 nats / $\ln 256$ ≈ 87 bytes, since there are 256 possible byte values per token). A practitioner storing billions of thumbnails can use this to calculate the theoretical compression limit and decide whether investing in a better compression model is worthwhile. If current compression achieves, say, 200 bytes/image, there is room for a ~2.3× improvement; if it achieves 90 bytes/image, further investment yields diminishing returns. The same analysis applies at 16×16 (entropy ~2026 nats/image ≈ 290 bytes lower bound) and 32×32 (~6806 nats ≈ 975 bytes). This application requires only the curve-fitting methodology, not model deployment.

Dataset quality assessment via irreducible loss comparison. The paper demonstrates (Appendix E, Figure 32) that when YFCC100M-trained models are evaluated on ImageNet, the loss follows a power-law plus constant form with a different irreducible loss ( $L_\infty \approx 2.31$ nats/token at 32×32, vs. $L_\infty \approx 2.20$ for in-distribution YFCC100M images). The difference between these asymptotes is $D_{\text{KL}}(\text{ImageNet} \parallel \text{YFCC100M})$ plus any entropy difference. A practitioner with a training dataset and a target deployment distribution can measure the irreducible loss on both from scaling trends, and the gap directly quantifies distribution shift in information-theoretic terms. If the gap is small (e.g., a few nats/token), the training distribution is a good proxy for the deployment distribution. If large, either the target distribution has higher intrinsic entropy (more diverse data) or substantial distribution shift exists, and the practitioner knows they need either more target-domain data or domain adaptation. This provides a quantitative alternative to heuristic distribution shift metrics.

Compute budgeting for generative model training in non-language domains. The paper's finding that $\beta \sim 0.7$ and $D \propto N^{0.4}$ generalizes across domains provides immediate practical guidance for practitioners training image, video, or multimodal models. For a given compute budget, the optimal model size can be estimated by fitting $N_{\text{opt}}(C)$ from small-scale experiments (training a few small models at different sizes, measuring their compute-efficient loss, and extrapolating upward). The paper's Table 1 provides domain-specific constants: for image models, $N_{\text{opt}} \approx (C / 5 \times 10^{-14})^{0.64}$ for 8×8 and $(C / 1.6 \times 10^{-13})^{0.65}$ for 32×32; for video, $(C / 1.1 \times 10^{-12})^{0.71}$ . A team training a video model with a budget of, say, 10 PF-days can estimate the optimal model size as $(10 / 1.1 \times 10^{-12})^{0.71} \approx (9.1 \times 10^{12})^{0.71} \approx 1.5 \times 10^9$ parameters — approximately 1.5 billion — and allocate their tokens accordingly. This is actionable without requiring the team to conduct their own full scaling law analysis.

Downstream task performance forecasting from generative model scale. The ImageNet classification result (Figure 11) establishes that classification error follows $(N / 2.09 \times 10^3)^{-0.089}$ for pre-trained models, with no sign of saturation. A practitioner training a generative image model for eventual fine-tuning on a classification task can use this trend to forecast the error rate they'll achieve at any target model size, helping decide whether the additional training cost is justified by the projected accuracy gain. For example, moving from a 100M-parameter model (error rate roughly $(10^8 / 2.09 \times 10^3)^{-0.089} \approx (4.78 \times 10^4)^{-0.089} \approx 0.38$ , or ~38% top-1 error) to a 1B-parameter model (error roughly $(5 \times 10^8 / 2.09 \times 10^3)^{-0.089} \approx (2.39 \times 10^5)^{-0.089} \approx 0.33$ , or ~33% top-1 error) yields approximately 5 percentage points improvement. Whether this is worth the 10× increase in training compute can be evaluated against business metrics. The specific exponent and coefficient will depend on the dataset and task, but the methodology — fit a power-law to small-model fine-tuning results and extrapolate — transfers. This application requires only that the practitioner verifies the power-law form holds for their domain (which the paper's multi-domain evidence makes more plausible), not that they replicate the full scaling study.