PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs

🎯 Pitch

Training the same language model multiple times with only the random seed changed reveals that while pre-training is remarkably stable overall, a handful of outlier runs catastrophically diverge—and a new 'training map' method can predict these failures in larger models from smaller ones.

1. Executive Summary

This paper analyzes the stability of language model pre-training across randomness factors by introducing the PolyPythias, an extension of the Pythia model suite comprising 45 new training runs—9 additional random seeds across 5 model sizes from 14M to 410M parameters—resulting in approximately 7k new checkpoints released for research. Using these multi-seed runs, the authors study stability through three lenses: downstream performance and prediction consistency (quantified via inter-seed agreement using Cohen's κ and self-consistency across checkpoints), representational stability of linguistic information (measured through information-theoretic probing with minimum description length, yielding macro-F1, codelength ratio, and principal subspace angles), and training phase dynamics (characterized by training maps—Hidden Markov Models fitted to statistics of model parameters such as L2-norm and singular value variance—that reveal linear state transitions in stable runs versus regressive "forks" in unstable ones). The paper finds that language model pre-training is largely stable and follows consistent scaling laws across seeds, with two distinct learning phases—an initial phase between steps 10³–10⁴ where linguistic information begins to be encoded, and a critical phase between steps 10⁴–10⁵ where most performance improvements occur—while identifying only two outlier training runs (410M parameter models with seeds 3 and 4) that exhibit loss spikes and premature exits from key state transitions, establishing that training maps from smaller models can zero-shot predict the training trajectories and final performance of larger models with R² > 0.9 for all sizes except the smallest 14M model.

2. Context and Motivation

The Core Problem: We Don't Know How Much LLM Pre-Training Results Depend on Luck

The fundamental question this paper tackles is straightforward but rarely asked systematically: if you train the same language model architecture with the same hyperparameters, the same code, and the same dataset—but different random seeds—how different will the results be? This matters because training contemporary large language models is enormously expensive, and in standard practice, researchers and practitioners often train a model once and draw sweeping conclusions about architectures, training recipes, or emergent capabilities from that single run.

The paper defines stability as the change in a metric of interest (e.g., validation loss, downstream accuracy, representational quality) caused by changes in randomness factors—specifically, parameter initialization and data order (which the paper refers to as "batch composition" due to the packing procedure in the GPT-NeoX codebase). Stability is then quantified using the standard deviation of that metric across multiple training runs with different seeds. If this standard deviation is small, the training process is stable with respect to that metric; if it is large, then individual training runs are unreliable indicators of expected behavior, and conclusions drawn from single runs may be misleading.

The gap the paper identifies is that the research community still lacks sufficient resources and tools to systematically investigate pre-training stability, particularly for decoder-only language models (Section 1). This is not a minor oversight—it has direct consequences for how the field evaluates models, compares architectures, and draws scientific conclusions.

Why This Problem Is Important

Real-world impact on model development and deployment. Training a single large language model can cost millions of dollars in compute. If performance varies substantially depending on the random seed, then organizations face an uncomfortable reality: the model they deploy may be significantly better or worse than the "expected" performance of that architecture and training recipe, purely due to randomness. Without understanding the variance, it is impossible to know whether a seemingly superior model reflects a genuinely better design choice or just a lucky seed. This has direct implications for:

Architecture comparison: When researchers propose a new architecture and compare it against a baseline trained once, they cannot distinguish genuine improvements from seed-dependent noise unless the baseline's variance is known.
Hyperparameter tuning: If the training process is unstable, the optimal hyperparameters found in a tuning study may be artifacts of the particular seeds used, not generalizable properties of the training recipe.
Reproducibility: A core tenet of science. If two labs train the "same" model and get different results, is it because of implementation differences, or simply because they used different random seeds? Without multi-seed baselines, this question is unanswerable.

Theoretical significance for understanding learning dynamics. Beyond the practical concerns, stability analysis opens a window into the learning process itself. If a behavior (e.g., a particular linguistic capability, a phase transition in representations, or a performance benchmark score) emerges at the same training step across all seeds and model sizes, that suggests it is a robust, predictable property of the training algorithm and data distribution—something fundamental about how neural networks learn from language. Conversely, if a behavior varies wildly across seeds, it suggests sensitivity to initial conditions that may reflect chaotic dynamics, poor optimization landscapes, or fragile inductive biases.

The paper explicitly connects this to prior work on learning phases in language models. Saphra and Lopez (2019) showed that LSTMs exhibit distinct phases during training (e.g., memorization, generalization, forgetting) identifiable through representational similarity metrics. Chang and Bergen (2022) demonstrated that LMs progress from unigram-level predictions to context-dependent predictions at a consistent point in training. Olsson et al. (2022) identified that induction heads—the mechanistic basis of in-context learning—emerge around 2.5B tokens of training in transformer LMs. The paper asks: are these phase transitions invariant to random seeds? If so, they reflect genuine algorithmic properties of the learning process. If not, they may be contingent on initialization or data order, requiring more careful interpretation.

The paper also identifies a gap in our understanding of outlier training runs—runs that, for reasons not fully understood, deviate substantially from the norm, exhibiting loss spikes, performance degradation, or other pathologies. Understanding what distinguishes these runs from stable ones could enable early detection and intervention, saving compute by terminating or restarting problematic runs before they fully train.

Where Existing Approaches Fall Short

Single-run analyses are the norm, not the exception. The paper points out that the original Pythia suite (Biderman et al., 2023b)—despite being one of the most widely used resources for studying LM training dynamics—provides only one training run per model size. Researchers using Pythia to study memorization patterns (Biderman et al., 2023a; Lesci et al., 2024), learning dynamics (Michaelov and Bergen, 2023; Arnold et al., 2024), biases (Hu et al., 2024), or circuit formation (Tigges et al., 2024) have had no way to assess whether their findings are seed-specific or general. A correlation between some property and training step observed in a single Pythia run could be a robust feature of the learning process or a quirk of that particular initialization and data shuffle. Without multiple seeds, there is no way to tell.

Prior multi-seed releases are limited in scope. The paper acknowledges prior work that released multi-seed model suites but argues each falls short in critical ways:

MultiBERTs (Sellam et al., 2022): Released 25 BERT-base checkpoints (encoder-only architecture) trained with different seeds, but only the final checkpoints are available for most runs; intermediate checkpoints exist for only 5 of the 25 runs (28 per run). This makes it impossible to study training dynamics—you can see where the models end up, but not how they got there or whether they followed similar trajectories. Additionally, BERT uses an encoder-only architecture, and the paper notes that the field has increasingly shifted to decoder-only models (GPT-style), whose training dynamics may differ substantially.
GPT-2 multi-seed runs (Karamcheti et al., 2021): Trained 10 runs of GPT-2 at two sizes (124M and 355M parameters) with 600 intermediate checkpoints each. This is the closest predecessor to PolyPythia, but is limited to only two model sizes, making it impossible to study how stability scales with model size (e.g., are larger models more or less stable? Do phase transitions shift with parameter count?).
Llama-2-7B multi-seed runs (Madaan et al., 2024): Trained 10 runs of Llama-2-7B with different seeds on 210B tokens, analyzing 21 intermediate checkpoints. This operates at a larger scale than PolyPythia, but the checkpoints remain publicly unavailable, and the analysis is limited to a single model size. Furthermore, 21 checkpoints is coarse-grained for studying phase transitions—the paper notes (Section 1) that this provides insufficient temporal resolution to identify when specific capabilities emerge or how training dynamics evolve.

The field lacks a systematic framework for stability analysis. Beyond the specific resource gaps, the paper identifies a conceptual gap: prior work has studied stability mostly as an afterthought—a variance estimate on final performance metrics—rather than as a first-class object of study that spans the entire training pipeline. Section 1 explicitly frames the paper's contribution as covering stability across three levels: downstream performance (the output), intermediate representations (the internal processing), and training parameter dynamics (the optimization process). This holistic framing is absent from prior work, which typically examines only final benchmark scores or validation loss.

Stability may depend on model scale in unknown ways. The paper notes that as models grow larger, the computational cost of multi-seed training becomes prohibitive, creating a tension: the models where stability matters most (because training is most expensive) are the hardest to study. Prior work provides no way to extrapolate stability properties from smaller to larger models—if a 160M model shows low variance across seeds on a given metric, does a 7B model trained on the same data exhibit similarly low variance? The paper explicitly aims to provide resources at intermediate scales (14M to 410M) that can help answer this question, while acknowledging that its largest model is still small compared to frontier LLMs.

How This Paper Positions Itself

A resource contribution, not a method contribution. The paper frames PolyPythia primarily as an infrastructure contribution—"an extension of the Pythia model suite" (Section 2) designed to enable future research, not a novel method or algorithm. The release includes 45 new training runs (9 new seeds × 5 model sizes) plus the original 5 runs (seed 0 for each size), yielding approximately 7,000 checkpoints with the same granularity as the original Pythia—154 checkpoints per run, log-spaced early in training and every 1,000 steps thereafter.

The paper explicitly positions this as filling a gap in the research ecosystem:

"Our aim is to provide an additional resource for researchers unable to train even 410M parameter models from scratch, thus enabling them to study training stability across model sizes."

This is an important framing: the paper is not claiming to answer all questions about stability, but rather to provide the raw material—the checkpoints, the training logs, the pre-shuffled datasets—that other researchers need to ask those questions.

Comparison to prior suites. The paper positions PolyPythia as comparing favorably to existing multi-seed suites along several dimensions (Section 2):

Scale diversity: 5 model sizes vs. 1 (MultiBERTs, Llama-2-7B) or 2 (GPT-2 suites). This is the key advantage—it enables analysis of how stability changes with model size.
Checkpoint granularity: 154 per run vs. 28 (MultiBERTs), 21 (Llama-2-7B). This enables fine-grained study of learning phases and when behaviors emerge.
Architecture: Decoder-only transformers, matching the dominant paradigm in contemporary LLM research, unlike the encoder-only MultiBERTs.
Accessibility: All checkpoints, datasets, and training logs are publicly released, unlike the Llama-2-7B suite.

Connecting stability to scaling laws. The paper draws an implicit connection to the scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) by framing stability as a dimension of scaling behavior. Much as scaling laws describe how performance predictably improves with model size and compute, the paper asks whether the variance of performance also follows predictable patterns across scales. This is explored in Section 4 (representational stability correlates across model sizes with r > 0.94) and Section 5 (training maps from smaller models predict those of larger models), but the conceptual motivation is established in the introduction: understanding whether conclusions drawn from smaller-scale experiments hold at larger scales.

A deliberate choice of scale. The paper is transparent about its scale limitation—models up to 410M parameters, far smaller than contemporary frontier LLMs—and frames this as a deliberate tradeoff:

"A limitation of our suite is that it spans model sizes up to 410M parameters. This choice reflects computational constraints, prioritising seed exploration and checkpoint granularity over scaling up model size."

This is a crucial design decision: given a fixed compute budget, the authors chose to study many seeds at modest scale rather than few seeds at large scale. This reflects their hypothesis that understanding variance requires statistical power in the number of runs, not just parameter count. Whether the stability patterns observed at 14M–410M parameters generalize to billion-parameter models is an empirical question the paper leaves open, but the choice to maximize seed diversity provides the strongest possible signal at the chosen scale.

Specific training choices. The paper makes several explicit decisions that position the release for maximum comparability and reuse:

Same hyperparameters, codebase, and data as the original Pythia suite: Every training run uses the identical configuration as Biderman et al. (2023b)—the GPT-NeoX codebase, the Pile dataset, a cosine learning rate schedule with warm-up, batch size of 1,024 sequences (approximately 2M tokens), and 143k optimization steps. The only variation is the random seed, which controls both parameter initialization and the shuffling of documents prior to packing into training sequences. This isolation of randomness as the sole independent variable is what makes the suite useful for stability analysis.
Non-deduplicated Pile: The paper uses the standard (non-deduplicated) version of the Pile, matching the original Pythia release. This is important for consistency—researchers comparing PolyPythia results to prior Pythia-based studies need the same data distribution.
Release of pre-shuffled datasets: Beyond model checkpoints, the paper releases the tokenized and pre-shuffled datasets corresponding to each seed. This enables researchers to study the relationship between data order and learned behaviors—for instance, investigating whether specific capabilities emerge when the model encounters particular types of documents, or whether memorized examples appear at consistent positions across seeds.

The nested definition of randomness: seeds vs. data order vs. initialization. A subtle but important aspect of the paper's positioning is how it treats randomness factors. A "seed" in the PolyPythia suite controls both parameter initialization and data order simultaneously. This means that when the paper reports variance across seeds, it captures the combined effect of both sources of randomness. This is a practical choice—in real training runs, these factors co-vary—but it means the paper cannot disentangle which source of randomness drives any observed instability. The release includes some additional runs for the 160M model size that vary only the data seed or only the weight initialization seed (Table 5), enabling researchers to begin teasing apart these factors, but the main analysis treats them jointly.

A note on terminology: "batch composition" vs. "data order." The paper uses the term "batch composition" rather than "data order" because of a technical detail in the GPT-NeoX codebase:

"the GPT-NeoX codebase... shuffles documents before packing them into sequences. This results in sequences that are not simply reshuffled across seeds; they are unique due to the different packing."

In standard data shuffling, the same set of documents appears in different orders across seeds, but each individual sequence of 2,048 tokens contains exactly the same tokens (just in different positions). In the Pythia setup, documents are first shuffled and then packed into fixed-length sequences by concatenation, so a different shuffling results in sequences that contain different combinations of documents—not merely the same sequences in a different order. This means the model sees genuinely different token sequences across seeds, not just a permutation of identical sequences. This subtlety matters because it means the variance across PolyPythia seeds may be larger than variance from order alone in a non-packed setting, since the packing process amplifies the effects of shuffling.

The paper's relationship to existing training dynamics research. The paper does not present itself as the first to study training stability—it extensively cites prior work on fine-tuning stability (Dodge et al., 2020; McCoy et al., 2020; Mosbach et al., 2021), pre-training underspecification (D'Amour et al., 2022), and benchmark sensitivity (Alzahrani et al., 2024). Rather, it positions itself as filling a specific gap in this literature: the lack of resources for studying stability throughout the training process for decoder-only architectures at multiple scales with dense checkpointing. Each prior work covers some subset of these desiderata; PolyPythia aims to cover all of them simultaneously.

This positioning is strategic: by providing a resource that combines scale diversity, seed diversity, checkpoint density, and architectural relevance to contemporary LLMs, the paper enables a class of research questions that were previously difficult or impossible to address without training models from scratch—a barrier that excludes most academic researchers. The paper's analyses in Sections 3–5 serve as existence proofs for the suite's utility, but the explicit intent is that other researchers will use it to ask questions the authors did not anticipate.

3. Technical Approach

3.1 Reader Orientation

The "system" being built in this paper is not a single model or algorithm, but rather a controlled experimental framework—the PolyPythia suite—that lets researchers isolate and quantify the effects of randomness on every stage of language model pre-training. Instead of training one model and hoping it is representative, the authors train the exact same architecture with the exact same hyperparameters and data ten different times (varying only the random seed), at five different sizes, producing a grid of 50 training runs with dense intermediate checkpoints. The problem it solves is the fundamental uncertainty about whether any single training run's behavior—its benchmark scores, its learned representations, its loss trajectory, its phase transitions—is a reliable signal about the underlying learning process or a fluke of initialization and data order. The "shape" of the solution is a resource contribution: release the checkpoints, the pre-shuffled datasets, and analysis tools so that the broader research community can ask stability questions without training models from scratch.

3.2 Big-Picture Architecture (Diagram in Words)

The PolyPythia framework can be understood as a pipeline with four interconnected components:

Multi-seed training infrastructure — The GPT-NeoX codebase configured to train the same model architecture at five sizes (14M, 31M, 70M, 160M, 410M parameters), each with 10 different random seeds, producing 50 complete training trajectories from initialization to 143k steps. The seeds control both parameter initialization and the shuffling-then-packing of documents into training sequences.
Dense checkpointing — Each training run saves 154 checkpoints at identical intervals: log-spaced across the first 1,000 steps (steps 1, 2, 4, 8, . . . , 512), then every 1,000 steps from step 1k to 143k. This granularity is what makes studying when behaviors emerge possible, not just the final outcomes.
Multi-level evaluation pipeline — Three distinct analysis frameworks applied to the same checkpoints, spanning the full model stack: task performance and prediction agreement on downstream benchmarks (the output layer), information-theoretic probing of token representations (the internal hidden states), and training maps constructed from model parameter statistics (the weights and biases themselves).
Stability quantification methods — A suite of metrics that operationalize "stability" across runs: standard deviation of metrics across seeds, inter-seed agreement (Cohen's κ between predictions of different seeds), self-consistency (κ between a model's early and final predictions), representational shift (principal subspace angles between consecutive checkpoints' probes), and training map comparison (whether HMM state transitions follow linear or forked trajectories).

Information flows as follows: the training process produces checkpoints → checkpoints are evaluated on benchmarks, probed for linguistic representations, and have their parameter statistics extracted → within each model size, metrics are computed per-seed and then aggregated (median, interquartile range) across seeds → cross-size comparisons examine whether stability patterns scale predictably with model size → outlier detection identifies seeds whose behavior deviates from the per-size norm.

3.3 Roadmap for the Deep Dive

First, the multi-seed training procedure and checkpointing strategy, because everything downstream depends on what was trained, how it was trained, and when snapshots were taken. Understanding the packing-based data pipeline is particularly important—the paper's use of "batch composition" rather than "data order" reflects a technical detail that affects how randomness propagates.
Second, the downstream evaluation protocol—which benchmarks are used, why those specific ones, how predictions are extracted, and how inter-seed agreement and self-consistency are computed—because these are the most practitioner-facing metrics and the level at which outlier seeds are first identified.
Third, the information-theoretic probing framework for representational stability—how probes are trained with the MDL objective, how global token representations are constructed through learned layer weighting, and how the three metrics (macro-F1, codelength ratio, principal subspace angles) capture different aspects of what representations encode and how they change—because this bridges the gap between observable outputs and internal processing.
Fourth, the training maps methodology—how Hidden Markov Models are fitted to parameter statistics, how latent states are assigned to checkpoints, and how bag-of-states representations enable regression-based prediction of final performance—because this is the most novel analysis method in the paper and the one that enables zero-shot cross-model-size prediction.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a resource and empirical analysis paper whose core idea is that by training multiple instances of the same model architecture with different random seeds and releasing dense intermediate checkpoints, the research community can systematically study which aspects of language model pre-training are stable (invariant to randomness) and which are fragile (seed-dependent), and can develop methods to detect and predict instability early in training.

Multi-Seed Training Procedure and Checkpointing

The foundation of the PolyPythia suite is a simple experimental design: take the exact training recipe from the original Pythia suite (Biderman et al., 2023b) and re-run it with different random seeds, keeping everything else identical. The paper trains models at five sizes—14M, 31M, 70M, 160M, and 410M parameters—with the original seed (seed 0) plus 9 new seeds (seeds 1–9) per size, for a total of 50 training runs. The 45 newly released runs (9 seeds × 5 sizes) add to the 5 already available from the original Pythia release, which included only seed 0 for each size.

What the seed controls. In this training setup, the random seed governs two sources of randomness simultaneously. First, it determines the initial values of all model parameters (weights and biases in every linear layer, embedding matrix, and layer normalization parameters). These are drawn from the same distribution across seeds—the standard initialization scheme used in the GPT-NeoX codebase—but the specific sampled values differ. Second, it determines the order in which documents are presented during training via the shuffling of the Pile dataset before packing documents into fixed-length sequences. The paper uses the term "batch composition" rather than "data order" because of a technicality in how the GPT-NeoX codebase processes data:

"the GPT-NeoX codebase shuffles documents before packing them into sequences. This results in sequences that are not simply reshuffled across seeds; they are unique due to the different packing."

In a standard data pipeline, shuffling permutes pre-existing fixed-length sequences, so each seed sees the same set of sequences but in a different order. In the Pythia pipeline, documents are first shuffled and then concatenated into sequences of exactly 2,049 tokens (one extra token is needed because target tokens are the right-shifted input tokens, so a sequence of 2,048 input tokens requires 2,049 total tokens in the concatenated stream before the shift). Because the concatenation boundaries depend on document lengths and the shuffle order, different shuffles result in sequences that contain different combinations of documents—not merely the same sequences permuted. This amplification effect means that PolyPythia seeds may exhibit larger variance than would be observed in a non-packed setting, since the model sees genuinely different token sequences across runs, not just a reordering of identical sequences.

Training hyperparameters. All runs use the identical configuration as the original Pythia suite, which the paper summarizes in Section 2:

Dataset: The Pile (Gao et al., 2021), a 300B-token curated collection of English documents. The paper uses the standard non-deduplicated version to maintain consistency with the original Pythia release and with prior research that used those checkpoints. The pre-shuffled and tokenized datasets for each seed are released alongside the model checkpoints, enabling researchers to study the relationship between data exposure order and learned behaviors.
Sequence length: 2,048 tokens (with an additional token for the right-shifted target, making the packed sequences 2,049 tokens long before the shift). This is a standard length for models in this parameter range and matches the original Pythia configuration.
Batch size: 1,024 sequences per optimization step. At 2,048 tokens per sequence, each step processes approximately 2.1 million tokens. Over the full 143,000-step training run, the model sees roughly 300B tokens, corresponding to one epoch over the Pile.
Learning rate schedule: Cosine schedule with a warm-up phase. The paper does not report the exact peak learning rate, warm-up duration, or minimum learning rate in the main text, but these match the original Pythia configuration, which uses a peak learning rate that varies by model size (larger models use lower peak learning rates) and a warm-up of approximately 1% of total steps (roughly 1,430 steps, or about 3B tokens).
Total training steps: Exactly 143,000 optimization steps. This was chosen in the original Pythia suite to match one epoch over the 300B-token Pile at a sequence length of 2,048 and batch size of 1,024: 143,000 × 1,024 × 2,048 ≈ 300B tokens.
Codebase: GPT-NeoX v1.0 (Andonian et al., 2023), the same version used for the original Pythia release. This ensures that any differences between runs are attributable to the seed, not to code changes.

Why this configuration? The paper deliberately reuses the original Pythia configuration to maximize comparability and reuse. Researchers who have already developed analysis pipelines for Pythia models can apply them to PolyPythia with minimal modification. The choice to use exactly one epoch—unlike many contemporary LLMs that are trained for multiple epochs on smaller datasets—means that each training example is seen exactly once per run, making the data order effect a pure function of when each document appears rather than how many times it appears. This simplifies the interpretation of seed-dependent differences: if two seeds produce different behaviors, those differences must arise from the order in which the model encountered training examples (and the initial parameter values), not from differential exposure frequency.

Checkpointing schedule. Each training run saves 154 checkpoints following a two-phase schedule designed to provide dense coverage of early training (where learning dynamics change most rapidly) and regular coverage of later training (where changes are more gradual). The phases are:

Log-spaced checkpoints from step 0 to step 1,000: The initial checkpoint at step 0 (random initialization before any training), then checkpoints at steps 1, 2, 4, 8, 16, 32, 64, 128, 256, and 512. This logarithmic spacing captures the rapid changes that occur in the first few hundred optimization steps, when the loss drops most steeply and basic linguistic patterns are first acquired.
Linearly-spaced checkpoints from step 1,000 to step 143,000: One checkpoint every 1,000 steps starting at step 1k (steps 1000, 2000, 3000, . . . , 143000). This provides 143 checkpoints in the later phase, giving a resolution of roughly 2.1B tokens between snapshots (1,000 steps × 1,024 sequences/step × 2,048 tokens/sequence ≈ 2.1B tokens).

The total number of checkpoints per run is 154 (11 log-spaced through step 512, step 1000, and 142 from step 2000 to 143000 inclusive). Across 50 training runs, this yields approximately 7,700 checkpoints, of which roughly 7,000 are new (the original Pythia release already included seed-0 checkpoints for each size). The paper uses the term "about 7k new checkpoints" to refer to the 45 newly released runs × 154 checkpoints ≈ 6,930, reflecting the reality that some runs may have minor checkpoint count variations.

Evaluation checkpoint subset. For computational efficiency, the paper does not evaluate all 154 checkpoints per run for every analysis. Instead, it uses a coarser subset (Section 3): log-spaced checkpoints at steps 0, 1, 2, . . . , 512, 1k, and then every 10,000th step from step 3k to 143k (steps 3000, 13000, 23000, . . . , 143000). This results in approximately 26 evaluation checkpoints per run for the downstream performance and representational stability analyses. The training maps analysis (Section 5) uses all 154 checkpoints because HMM fitting benefits from temporal density. The paper justifies this subsetting by noting that the goal is to track model behavior across training while limiting computational costs—evaluating all 154 checkpoints for all 50 runs on all benchmarks and probing tasks would be prohibitively expensive, and the 10k-step spacing in later training captures the slower changes in that phase at sufficient resolution.

Disentangling initialization from data order. The paper includes additional training runs for the 160M model size that isolate the two sources of randomness (Table 5). Specifically, there are runs where only the data seed is varied while keeping the weight initialization seed fixed ("data-seed1," "data-seed2," "data-seed3"), and runs where only the weight initialization seed is varied while keeping the data seed fixed ("weight-seed1," "weight-seed2," "weight-seed3"). These ablation runs are not used in the main paper analyses but are released as part of the suite to enable future research to disentangle which source of randomness drives observed instability. The paper does not report separate variance components for data order versus parameter initialization in its analyses.

Downstream Evaluation Protocol and Agreement Metrics

The paper evaluates model performance using the Language Model Evaluation Harness framework (Gao et al., 2024; Biderman et al., 2024), the standard open-source toolkit for standardized LLM benchmarking. The evaluation protocol follows the few-shot prompting approach established by Brown et al. (2020), where models are presented with a task description and (optionally) examples, and answer selection is performed by comparing the log-likelihood of each candidate continuation.

Benchmark selection. The paper selects eight multiple-choice benchmarks that span different reasoning and knowledge domains. The selection rationale, following Biderman et al. (2023b), is to cover a range of task difficulties so that both ceiling effects (all models score near random, providing no signal about differences) and saturation effects (all models score near perfect) can be identified:

ARC (Easy) and ARC (Challenge) (Clark et al., 2018): Natural science question answering drawn from standardized exams. The Easy subset contains questions that are generally answerable with basic science knowledge; the Challenge subset contains questions deliberately selected to be difficult for retrieval-based and simple statistical methods. The paper uses these as probes for factual knowledge and basic reasoning.
LAMBADA (Paperno et al., 2016): A language modeling benchmark that requires predicting the final word of a passage given the full preceding context. Unlike standard next-word prediction benchmarks where local context suffices, LAMBADA passages are constructed so that the final word is not predictable from the immediately preceding sentence alone—it requires integrating information from earlier in the passage. This makes it a test of long-range contextual understanding.
Logiqa (Liu et al., 2020): Logical reasoning questions derived from the LSAT (Law School Admission Test), requiring deductive reasoning, identification of logical fallacies, and inference from premises. This is the most demanding reasoning task in the benchmark suite and serves as a test of whether models can perform structured logical operations beyond pattern matching.
Piqa (Bisk et al., 2020): Physical commonsense reasoning, where questions ask about everyday physical interactions (e.g., "how to separate egg yolks from whites"). This tests whether models have acquired intuitive physical knowledge from text, a form of understanding that goes beyond linguistic pattern recognition.
SciQ (Welbl et al., 2017): Multiple-choice science questions sourced from educational materials at various grade levels. This complements ARC by providing a broader coverage of scientific topics at varying difficulty levels.
WinoGrande (Sakaguchi et al., 2020): An expanded and adversarially filtered version of the Winograd Schema Challenge, testing pronoun resolution that requires commonsense reasoning. The adversarial construction means that superficial statistical cues are deliberately removed, forcing models to rely on deeper understanding.
WSC (Levesque et al., 2012): The original Winograd Schema Challenge, a smaller set of carefully constructed pronoun resolution problems. Despite its small size (273 examples), it is included for historical comparability.

Prediction extraction. For each benchmark, the model is presented with a prompt containing the question and candidate answers. The model's prediction is the candidate answer whose continuation receives the highest log-probability under the model. This is the standard "likelihood-based" approach for multiple-choice evaluation in autoregressive LMs: the model does not generate free-form text but instead scores each candidate as if it were the next tokens to be generated, and the highest-scoring candidate is selected. The paper reports accuracy as the fraction of questions for which the selected candidate matches the ground-truth answer.

Cross-seed aggregation. For each model size, checkpoint step, and benchmark, the paper reports the median and interquartile range (IQR, the range between the 25th and 75th percentiles) of accuracy across the 10 seeds. The median is chosen over the mean because it is robust to outliers—a single seed with unusually high or low accuracy will not dominate the summary. The IQR provides a measure of dispersion that is similarly robust. Together, they give a picture of both the central tendency (what a "typical" training run achieves) and the spread (how much variation exists around that typical value).

Inter-seed agreement (Cohen's κ). To go beyond aggregate accuracy and examine whether models with different seeds make the same predictions—not just achieve similar average accuracy—the paper computes Cohen's κ (Cohen, 1960) between the individual answer choices of different seeds. The computation is defined as follows:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed proportion of agreement between two models' predictions (the fraction of test examples where both models select the same answer), and $p_e$ is the expected proportion of agreement by chance (what would be expected if the two models were making independent random predictions with the same marginal answer distribution).

What it computes: Cohen's κ is a measure of agreement between two raters (here, two models with different seeds) that corrects for the agreement that would be expected by random chance. A κ of 1.0 means the two models make identical predictions on every example. A κ of 0 means they agree exactly as often as would be expected if they were guessing independently and randomly (i.e., their agreement provides no evidence of genuine consistency beyond what chance would produce). A negative κ would mean they disagree more than expected by chance, which in practice almost never occurs in this setting.

Why this form: The correction for chance agreement is essential because raw agreement percentages are misleading when the task has few answer choices. On a 4-option multiple-choice task, two random models would agree 25% of the time purely by chance. Raw agreement of 75% might seem high, but corresponds to a κ of only 0.67 (since 50 percentage points of the 75% could be attributed to skill beyond the 25% chance baseline, and 50/75 = 0.67). Using κ avoids conflating trivial agreement from limited answer spaces with genuine consistency in model predictions. The paper adopts the standard interpretation where κ ≈ 0.5 represents "moderate agreement" and κ ≈ 0.7 represents "substantial agreement."

Inter-seed agreement computation procedure. For each model size, the paper defines inter-seed agreement as the κ between the predictions of a model trained with a given seed (1–9) and the predictions of the model trained with seed 0 (the original Pythia seed). Seed 0 serves as a fixed reference point, enabling direct comparison across seeds: if seed 3 has κ = 0.6 with seed 0 on a given benchmark at a given checkpoint, and seed 7 has κ = 0.4, then seed 3's predictions are more similar to the reference than seed 7's. The paper aggregates these per-seed κ values by reporting the median and IQR across the 9 seeds (1–9) for each model size, checkpoint, and benchmark.

Self-consistency (within-run agreement across time). To examine whether a model's predictions stabilize during training, the paper computes κ between each model's predictions at an earlier checkpoint and its predictions at the final checkpoint (step 143k). This is defined as:

$\kappa_{\text{self}}(c) = \frac{p_o(c, 143\text{k}) - p_e(c, 143\text{k})}{1 - p_e(c, 143\text{k})}$

where $p_o(c, 143\text{k})$ is the observed agreement between the model's predictions at checkpoint $c$ and its predictions at step 143k, and $p_e$ is the corresponding chance-level agreement.

What it computes: For each checkpoint $c$ , the self-consistency score quantifies how similar the model's current "opinion" about each test example is to its final opinion. Early in training, when the model is making essentially random predictions, self-consistency will be near 0. As training progresses and the model converges to stable answer choices, self-consistency increases. However, the trajectory is not necessarily monotonic—if the model undergoes a phase where it changes its "mind" about many examples (e.g., moving from one incorrect strategy to a different incorrect strategy), self-consistency could temporarily decrease even as the model improves.

Why this form: This metric captures a different aspect of stability than accuracy or inter-seed agreement. A model could achieve high accuracy early in training and maintain it, yielding high self-consistency; or it could achieve the same final accuracy through a circuitous path of changing predictions, yielding lower self-consistency. The trajectory reveals whether learning is gradual (the model settles into its final answer pattern early and refines) or involves major reorganizations (the model substantially revises which examples it gets right and wrong).

Gender bias evaluation. In addition to task performance, the paper evaluates gender bias using three benchmarks that capture different aspects of how models encode gender-related information:

BLiMP (Gender Agreement) (Warstadt et al., 2020): A set of minimal sentence pairs that differ only in gender-marked words (e.g., "the actress . . . she" vs. "the actor . . . he"), testing whether the model assigns higher probability to the grammatically correct continuation. This measures the grammatical encoding of gender—whether the model has learned English gender agreement rules—not social bias per se. Accuracy is reported as the fraction of pairs where the model prefers the grammatically correct form.
CrowS-Pairs (Gender) (Nangia et al., 2020): A set of minimal sentence pairs contrasting stereotypical and anti-stereotypical associations (e.g., "women are nurturing" vs. "men are nurturing"). The paper uses the adaptation by Névéol et al. (2022) which removes instances identified as problematic by Blodgett et al. (2021). The metric reported is the proportion of times the model assigns higher probability to the stereotypical continuation, where 0.5 indicates no bias (equal preference for stereotypical and anti-stereotypical forms) and values above 0.5 indicate a bias toward stereotypes.
Simple Co-occurrence Bias (Smith et al., 2022): A set of template-based prompts that measure the model's likelihood of completing a context with male- vs. female-identifying words. The metric reported is the proportion of times the male option is preferred. Like CrowS-Pairs, 0.5 indicates no gender skew.

Why these benchmarks? The paper selects these three to cover different "levels" of gender encoding: BLiMP captures whether the model has learned the grammatical concept of gender agreement (a linguistic phenomenon); Simple Co-occurrence captures whether the model has absorbed crude statistical associations between genders and particular words or contexts (a distributional phenomenon); CrowS-Pairs captures whether the model has internalized more semantically rich cultural stereotypes (a social phenomenon). By tracking all three across training, the paper can examine whether these different types of gender information emerge at the same time or follow distinct developmental trajectories.

Benchmark reliability caveat. The paper explicitly notes that the observed variance in bias metrics may reflect poor benchmark reliability rather than actual differences in model bias:

"We posit that the large variance observed for the bias measures reflects the poor reliability (e.g., due to small benchmark size, poor quality test items, etc) rather than actual bias differences."

This is an important methodological note: bias benchmarks like CrowS-Pairs are small (the Gender subset contains only a few hundred examples) and have been criticized for containing ambiguous or poorly constructed items (Blodgett et al., 2021). The observed variance across seeds on these benchmarks may therefore be a function of the test set's noisiness rather than genuine instability in the model's learned gender associations. The paper does not attempt to decompose the observed variance into benchmark noise vs. model noise.

Information-Theoretic Probing for Representational Stability

The paper analyzes the stability of what models learn internally—not just what they output—by applying the information-theoretic probing framework from Müller-Eberstein et al. (2023). This framework trains lightweight classifiers ("probes") on top of frozen model representations to predict linguistic properties of input tokens, then measures how much linguistic information those representations contain and how that information changes during training.

Linguistic probing tasks. Seven token-level classification tasks are selected to span different levels of linguistic structure, from low-level syntax to high-level semantics:

Part-of-Speech tagging (PoS) (Pradhan et al., 2013): Classify each token into one of 51 syntactic categories (noun, verb, adjective, etc.). This is the most basic syntactic task—it asks whether the model's representations distinguish the fundamental grammatical roles of words.
Dependency parsing (Dep) (Silveira et al., 2014): Predict the syntactic dependency relation between each token and its head (e.g., subject, object, modifier). This is a more structurally complex syntactic task that requires understanding the hierarchical relationships between words.
Coreference resolution (Coref) (Pradhan et al., 2013): Link mentions of the same entity across a text (e.g., identifying that "the president" and "he" refer to the same person). This spans both syntax (finding the mentions) and semantics (determining referential identity).
Named Entity Recognition (NER) (Pradhan et al., 2013): Classify tokens into 18 entity categories (person, organization, location, date, etc.). This tests whether the model's representations encode real-world knowledge about what types of entities words refer to.
Semantic Tagging (SemTag) (Abzianidze et al., 2017): Classify tokens into 69 fine-grained semantic categories (e.g., temporal entity, event, modal operator). This tests whether the model's representations encode compositional semantic information.
Sentiment Analysis (Senti) (Socher et al., 2013): Classify each token's sentiment polarity (positive, negative, neutral) in context. This tests whether the model's representations capture the affective/evaluative dimension of meaning.
Topic Classification (Topic) (Lang, 1995): Classify documents into 20 topic categories. This tests whether the model's token representations carry information about the document's high-level subject matter.

Probe architecture. The probing framework constructs a global token representation by learning a weighted combination of the model's hidden states across all layers. For a model with $L$ layers producing hidden states $\mathbf{h}_l(x) \in \mathbb{R}^d$ for each token $x$ at each layer $l$ , the global representation is:

$\mathbf{h}(x) = \sum_{l=1}^{L} \alpha_l \mathbf{h}_l(x)$

where $\alpha_l \in \mathbb{R}$ is a learned scalar weight for layer $l$ , and the weights are normalized such that $\sum_l \alpha_l = 1$ (the paper uses a softmax parameterization to enforce this). This global representation is then passed to a linear classifier:

$\hat{y} = \text{softmax}(\boldsymbol{\theta}^\top \mathbf{h}(x))$

where $\boldsymbol{\theta} \in \mathbb{R}^{d \times |\mathcal{Y}|}$ is a learned weight matrix mapping the $d$ -dimensional representation to logits over the $|\mathcal{Y}|$ output classes.

What the layer weighting does. Rather than using only the final layer's representations (which might be optimized for the language modeling objective rather than general linguistic features) or averaging all layers equally, the learned $\alpha_l$ weights allow the probe to discover which layers are most informative for each linguistic task. This is important because prior work (e.g., Tenney et al., 2019; Hewitt and Manning, 2019) has shown that different linguistic phenomena are encoded most prominently at different depths: syntactic information tends to peak in middle layers, while semantic information is more prominent in later layers. The learned weighting automatically adapts to this layer-wise specialization without requiring manual layer selection.

Why a linear probe? The linear classifier is chosen deliberately as the simplest architecture that can read out information from the representations. A non-linear probe (e.g., an MLP) could potentially extract more information, but it would also be capable of computing that information from the representations rather than merely reading out what is already encoded. The linear probe provides a conservative estimate: if a linear classifier can correctly predict a linguistic property from the representations, then that property must be encoded in a linearly separable way, which is a strong form of encoding. This choice aligns with the standard practice in probing literature (Alain and Bengio, 2017; Hewitt and Liang, 2019) where linear probes are used to establish a lower bound on the information content of representations.

Training objective: Minimum Description Length (MDL). The probes are trained not with standard cross-entropy minimization alone, but with a Minimum Description Length objective (Voita and Titov, 2020) that incorporates a regularization term encouraging sparsity:

$\mathcal{L}_{\text{MDL}} = -\mathbb{E}_{\boldsymbol{\theta} \sim p(\boldsymbol{\theta})} \left[ \sum_{(x, y) \in \mathcal{D}} \sum_{x \in \mathbf{x}, y \in \mathbf{y}} \mathcal{L}_{\text{CE}}\left(\boldsymbol{\theta}^\top \mathbf{h}(x), y\right) \right] + \text{KL}\left(p(\boldsymbol{\theta}) \parallel q(\boldsymbol{\theta})\right)$

where:

$\mathcal{D}$ is the probing dataset, containing input-output pairs $(\mathbf{x}, \mathbf{y})$ (token sequences with per-token labels)
$\mathcal{L}_{\text{CE}}$ is the standard cross-entropy loss between the probe's predicted class probabilities and the ground-truth label
$p(\boldsymbol{\theta})$ is the probe's parameter distribution (learned during training)
$q(\boldsymbol{\theta})$ is a fixed prior distribution that encodes the inductive bias toward sparsity

The outer sum over $(x, y) \in \mathcal{D}$ iterates over all examples in the dataset; the inner sum over $x \in \mathbf{x}, y \in \mathbf{y}$ iterates over all tokens and their corresponding labels within each example.

What the MDL objective computes: The first term (the expectation over $\boldsymbol{\theta} \sim p(\boldsymbol{\theta})$ of the cross-entropy loss) measures how well the probe predicts the linguistic labels from the model's representations—lower values mean the representations contain more easily extractable linguistic information. The second term (the Kullback-Leibler divergence between the learned probe distribution $p(\boldsymbol{\theta})$ and the sparsity-inducing prior $q(\boldsymbol{\theta})$ ) penalizes the probe for using many non-zero parameters. Together, they implement the minimum description length principle: find the simplest probe (fewest effective parameters) that adequately explains the data.

Why MDL instead of standard cross-entropy? Standard cross-entropy training of probes has a well-documented flaw: a sufficiently expressive probe can memorize random label assignments, achieving perfect training accuracy on shuffled labels, which demonstrates that the learned "information" was an artifact of the probe's capacity, not genuinely present in the representations (Voita and Titov, 2020; Pimentel et al., 2020). The MDL objective addresses this by penalizing complex probes: if the representations genuinely encode linguistic information in a structured way, a simple (sparse) linear classifier will be able to extract it. If the representations are effectively random with respect to the linguistic labels, the probe would need many non-zero parameters to fit the training data, and the regularization penalty would outweigh any cross-entropy improvement, causing the probe to prefer a near-zero solution. This built-in Occam's razor makes MDL probes more reliable indicators of genuine representational content.

The prior $q(\boldsymbol{\theta})$ : The paper does not specify the exact form of the sparsity-inducing prior in the main text, but following Voita and Titov (2020), it is typically a zero-mean Gaussian with learnable variance per parameter, where the KL divergence term encourages the posterior variances to be small for parameters that are not strongly supported by the data—effectively "turning off" unnecessary probe parameters.

Three metrics of representational stability. Once the probes are trained on each checkpoint of each model, the paper extracts three complementary metrics:

1. Macro-F1 score. The F1 score is computed separately for each class (precision × recall × 2 / (precision + recall)), and then averaged across classes (the "macro" average, which treats all classes equally regardless of frequency). This is the standard metric for multi-class classification when class imbalance is present, as it is in linguistic tasks (e.g., the noun class in PoS tagging is far more frequent than the interjection class).

What macro-F1 captures: The amount of linguistic information that is extractable from the model's representations. Higher macro-F1 means the model's internal representations make the linguistic distinctions more linearly separable—the model "knows" more about syntax, semantics, or entity types at that point in training. This is the representational analog of downstream task accuracy.

2. Codelength ratio. The MDL objective itself has an information-theoretic interpretation: it corresponds to the codelength (in nats or bits) needed to transmit the labels given the representations under the optimal probe. The paper defines the codelength ratio as:

$\text{Codelength Ratio} = \frac{\mathcal{L}_{\text{MDL}}^{\text{trained}}}{\mathcal{L}_{\text{MDL}}^{\text{random}}}$

where $\mathcal{L}_{\text{MDL}}^{\text{trained}}$ is the codelength achieved by the probe trained on the current checkpoint's representations, and $\mathcal{L}_{\text{MDL}}^{\text{random}}$ is the codelength achieved by a probe trained on the randomly initialized (step 0) model's representations.

What codelength ratio captures: How efficiently the model encodes linguistic information relative to the random baseline. A ratio near 1.0 means the representations are no more informative than random noise for this task—transmitting the labels given the representations is just as expensive as transmitting them without any help from the model. A ratio near 0 means the representations are extremely efficient: the labels are almost perfectly predictable from the representations, requiring near-zero additional information to transmit. The paper notes that "values close to 0 indicate high representational efficiency" (Section 4). This metric is complementary to macro-F1 because it penalizes representations that require a complex probe to extract information: two representations might have the same macro-F1, but the one that achieves it with a simpler probe will have a lower (better) codelength ratio.

3. Principal subspace angles (SSAs). To measure how much representations change between consecutive checkpoints, the paper computes the principal angles between the subspaces spanned by the probes' weight matrices at neighboring time steps. Given two probe parameter matrices $\boldsymbol{\theta}_t$ and $\boldsymbol{\theta}_{t+1}$ at checkpoints $t$ and $t+1$ , the principal subspace angles (Knyazev and Argentati, 2002) quantify how much the most important directions in the classification weight space have rotated:

$\text{SSA}(\boldsymbol{\theta}_t, \boldsymbol{\theta}_{t+1}) = \text{arccos}\left(\sigma_{\min}\left(\mathbf{Q}_t^\top \mathbf{Q}_{t+1}\right)\right)$

where $\mathbf{Q}_t$ and $\mathbf{Q}_{t+1}$ are orthonormal bases for the top- $k$ principal components of $\boldsymbol{\theta}_t$ and $\boldsymbol{\theta}_{t+1}$ (typically $k$ is chosen as the number of probe parameters explaining some large fraction of variance, e.g., 95%), and $\sigma_{\min}$ is the smallest singular value of the matrix $\mathbf{Q}_t^\top \mathbf{Q}_{t+1}$ .

What SSA captures: The degree of rotation in the probe's decision boundaries between two checkpoints. An angle of 0° means the probe's weight subspaces are perfectly aligned—the linguistic distinctions the model makes are encoded in exactly the same directions in representation space at both checkpoints. An angle of 90° means the subspaces are orthogonal—the model has completely reorganized how it encodes the linguistic property. Intermediate angles indicate partial reorganization. The paper notes that SSA provides a measure of representational shift that is complementary to the information-content metrics: a model could maintain stable macro-F1 while undergoing substantial internal reorganization (high SSA), or could change its macro-F1 while keeping its representational geometry relatively stable (low SSA).

Why SSAs between probe weights, not between representations directly? Computing the principal angles between the probe subspaces rather than between the representation subspaces has an important advantage: the probe incorporates both the layer weighting $\alpha$ and the classification directions $\boldsymbol{\theta}$ , so changes in either (a shift in which layers are informative, or a rotation in how different classes are separated) will be reflected in the SSA. This provides a more holistic measure of representational change than angles computed on raw hidden states, which would be sensitive to the scale and rotation of the entire representation space, not just the linguistically relevant subspaces.

Computational procedure for probing. For each model size, each seed, and each checkpoint in the evaluation subset (approximately 26 checkpoints per run), the paper trains separate probes for each of the 7 linguistic tasks. The probes are trained on the standard training splits for each task, with early stopping based on validation set MDL (the specific splits are not detailed in the main text but are standard for these tasks). The trained probes are then evaluated on the held-out test sets to compute macro-F1, codelength ratio, and (between consecutive checkpoints) SSAs. Results are aggregated across the 10 seeds by computing the mean and standard deviation for each metric at each checkpoint, shown as lines (mean) with shaded regions (±1 standard deviation) in Figure 3.

Cross-model-size correlation analysis. To quantify whether the trajectories of representational metrics are similar across model sizes, the paper computes Pearson correlations between the full time series of each metric for each pair of model sizes. Specifically, for a given metric (e.g., macro-F1 on PoS), the "trajectory" for a model size is the concatenation of that metric's values across all 7 tasks and all evaluation checkpoints—a vector of length 7 × 26 = 182 values per seed per model size. The paper then computes the Pearson correlation $r_{i,j}$ between model sizes $i$ and $j$ for that metric, and aggregates across all 25 pairs of model sizes using the Fisher z-transformation:

$\bar{r} = \tanh\left(\frac{1}{|\mathcal{M}|^2} \sum_{i,j \in \mathcal{M} \times \mathcal{M}} \text{arctanh}(r_{i,j})\right)$

where $\mathcal{M} = \{14\text{M}, 31\text{M}, 70\text{M}, 160\text{M}, 410\text{M}\}$ is the set of model sizes, and $|\mathcal{M}|^2 = 25$ is the number of pairs. The Fisher transformation (Fisher, 1970) is necessary because Pearson correlation coefficients are not normally distributed (they are bounded in [-1, 1]), so simple averaging would produce biased estimates. The $\tanh$ of the average $\text{arctanh}$ of the correlations produces an unbiased estimate of the mean correlation in Fisher-z space, which is then mapped back to the correlation scale.

What this analysis tests: If the trajectory of (say) NER macro-F1 across training is highly correlated between the 14M and 410M models (i.e., both models start improving at the same step, peak at the same step, and exhibit the same pattern of saturation), then the representational learning process is scale-invariant—understanding how a small model learns could predict how a large model will learn, enabling researchers to prototype and debug training dynamics at smaller, cheaper scales. The paper reports that these correlations are extremely high: $\bar{r} = 0.99$ for macro-F1, $0.98$ for codelength ratio, and $0.94$ for SSA (all $p < 0.001$ ), suggesting that representational learning dynamics are indeed highly scale-invariant for these model sizes.

Training Maps: Hidden Markov Models of Parameter Dynamics

The deepest level of analysis examines not what the model outputs or represents, but how its parameters themselves evolve during training. The paper adapts the training maps methodology from Hu et al. (2023), which treats the sequence of model checkpoints as observations from a latent dynamical system and uses Hidden Markov Models (HMMs) to infer the underlying states and transitions.

Parameter statistics. For each checkpoint of each training run, the paper extracts a vector of 15 statistics computed from the model's weights and biases, following (and partially extending) the feature set in Hu et al. (2023). These statistics are listed in Appendix C, Table 4, and are computed separately for the weight matrices and bias vectors of each layer, then averaged across layers to produce a single per-checkpoint feature vector:

Statistic	Description
`L1`	The L1-norm (sum of absolute values), averaged over weight matrices
`L2`	The L2-norm (square root of sum of squares), averaged over weight matrices
`L1/L2`	Ratio of L1 to L2 norm, a measure of weight sparsity (the closer to $1/\sqrt{d}$ , the denser the weights; the closer to 1, the sparser)
`μ_w`	Sample mean of the weights
`median_w`	Median of the weights
`σ_w`	Sample variance of weights
`μ_b`	Sample mean of the biases
`median_b`	Median of the biases
`σ_b`	Sample variance of biases
`trace`	The average trace (sum of diagonal elements) of the weight matrices, which equals the sum of eigenvalues and provides a measure of the total "magnitude" of each weight matrix's linear transformation
`λ_max`	The average spectral norm (largest singular value) of the weights, which determines the maximum amount a weight matrix can stretch any input direction
`trace/λ_max`	The ratio of trace to spectral norm; for a matrix with all eigenvalues positive, this is related to the effective rank
`μ_λ`	The average singular value over the weights
`σ_λ`	Sample variance of singular values over the weights

What these statistics capture. These 15 statistics are designed to summarize different aspects of the model's parameter distribution: the norms (L1, L2, trace, λ_max) capture the overall magnitude of the weights and how much they can amplify or attenuate signals; the sparsity ratio (L1/L2) captures whether weights are mostly near zero with a few large outliers or more uniformly distributed; the means and medians capture central tendency; the variances (σ_w, σ_b, σ_λ) capture how spread out the values are; the spectrum statistics (μ_λ, trace/λ_max) capture the shape of the singular value distribution, which relates to the effective dimensionality of the weight matrices.

Why these specific statistics? Hu et al. (2023) selected these statistics through a combination of prior knowledge about neural network training dynamics and empirical validation. The L2-norm and spectral norm are known to be important for generalization and stability (weight decay directly controls L2-norm; spectral normalization is used to stabilize GAN training). The ratio L1/L2 is a classic sparsity measure used in compressed sensing. The singular value variance (σ_λ) has been identified by Godey et al. (2024) as important for understanding "representation degeneration" in small language models, where the singular value distribution first becomes more uniform and then abruptly collapses around a few dominant directions. The paper's inclusion of these specific statistics is thus grounded in prior findings about which properties of weight matrices are informative about learning dynamics.

Standardization across seeds. Before fitting the HMM, the statistics are standardized within each model size: for each statistic and each checkpoint, the paper computes the mean and standard deviation across the 10 seeds, then transforms each seed's value to have zero mean and unit variance. This standardization is necessary because "HMMs are sensitive to the scale of the inputs" (Section 5). Without standardization, statistics with naturally larger numerical ranges (e.g., L2-norm, which grows with model size) would dominate the HMM's state assignments, drowning out the signal from statistics with smaller ranges (e.g., L1/L2 ratio). Standardization ensures that each statistic contributes equally to the similarity metric used by the HMM, regardless of its natural scale.

Hidden Markov Model (HMM) fitting. An HMM models a sequence of observations (here, the 15-dimensional standardized statistic vectors at each checkpoint) as being generated by a sequence of unobserved ("hidden") discrete states that follow a Markov process. The model is defined by three components:

Initial state distribution $\boldsymbol{\pi} \in \mathbb{R}^K$ : The probability of starting in each of $K$ latent states at step 0.
Transition matrix $\mathbf{A} \in \mathbb{R}^{K \times K}$ : The probability of transitioning from state $i$ to state $j$ between consecutive steps. Since the training process is progressive (checkpoints are visited in fixed temporal order), the HMM models the transitions between consecutive checkpoints.
Emission distributions $p(\mathbf{x}_t \mid s_t = k)$ : The probability of observing the 15-dimensional statistic vector $\mathbf{x}_t$ at step $t$ given that the model is in latent state $s_t = k$ . The paper uses Gaussian emission distributions, where each state $k$ is associated with a mean vector $\boldsymbol{\mu}_k \in \mathbb{R}^{15}$ and covariance matrix $\boldsymbol{\Sigma}_k \in \mathbb{R}^{15 \times 15}$ .

The HMM is trained using the Baum-Welch algorithm (Baum et al., 1970), an iterative Expectation-Maximization (EM) procedure. In the E-step, the algorithm computes the posterior probability of each state at each time step given the current parameter estimates (using the forward-backward algorithm). In the M-step, it updates the parameters ( $\boldsymbol{\pi}$ , $\mathbf{A}$ , $\boldsymbol{\mu}_k$ , $\boldsymbol{\Sigma}_k$ ) to maximize the expected complete-data log-likelihood. The process alternates until convergence.

Choosing the number of states $K$ . The primary hyperparameter in an HMM is the number of latent states $K$ . The standard approach to select $K$ is to fit HMMs with different values and choose the one that minimizes an information criterion like the Bayesian Information Criterion (BIC; Schwarz, 1978) or Akaike Information Criterion (AIC; Akaike, 1998), which balance goodness-of-fit against model complexity. The paper states:

"To enable comparisons across model sizes, we use 5 states, as this value is near optimal for the Bayesian Information Criterion across all sizes."

By fixing $K = 5$ for all model sizes, the paper ensures that the resulting state sequences (training maps) are directly comparable: state 1 in the 14M model means the same thing (in terms of the relative position in the training process and the HMM's learned state ordering) as state 1 in the 410M model. If each model size used a different $K$ , cross-model-size comparison would be more challenging because the state spaces would have different cardinalities.

From HMM to training map. Once the HMM is fitted to the 10 seed trajectories for a given model size, each checkpoint is assigned to the latent state with the highest posterior probability under the fitted HMM. This produces, for each seed, a sequence of 154 state labels—one per checkpoint—which constitutes that seed's training map. Training maps are typically visualized as sequences of colored blocks (each color corresponding to a latent state) ordered by checkpoint step (Figure 4, right panels), allowing visual inspection of whether the state transitions are consistent across seeds and whether any seeds exhibit unusual trajectories.

What makes a training map "linear" vs. "forked"? In a stable training run, the sequence of latent states follows a simple linear progression: the model starts in state 0, transitions to state 1 after some number of steps, then to state 2, and so on, never returning to an earlier state. This reflects the monotonic nature of gradient-based optimization—the model's parameters move progressively through parameter space rather than cycling back to earlier configurations. In an unstable run, the map may exhibit "forks" where the HMM assigns a later checkpoint to an earlier latent state, indicating that the model's parameter statistics have regressed to a configuration characteristic of an earlier training phase. These forks correspond to instability in the optimization process, such as loss spikes, where the optimizer takes a step that substantially worsens the loss and effectively "resets" some aspect of the model's parameter distribution.

What drives state transitions? To understand what distinguishes stable from unstable training, the paper analyzes which parameter statistics change most dramatically at each state transition. For each transition (e.g., state 2 → state 3), the paper computes the standardized difference in each of the 15 statistics between the typical values in the source state and the target state. The statistics with the largest absolute standardized differences are the primary "drivers" of that transition—the properties of the model parameters that change most at that phase boundary. The paper reports the top three drivers for each transition in Table 1, using a notation where an upward arrow (↑) indicates the statistic increases during the transition, a downward arrow (↓) indicates it decreases, and the subscript number is the standardized effect size (the difference divided by the pooled standard deviation). For example, for the transition 2 → 3 that is present in stable maps but missing in outlier maps:

"median_b ↓0.44, L2 ↓0.42, σ_w ↓0.50"

This means that in a normal 2 → 3 transition, the median bias decreases by 0.44 standard deviations, the L2-norm of weights decreases by 0.42 standard deviations, and the weight variance decreases by 0.50 standard deviations. The fact that all three effects are negative (decreasing) indicates that the 2 → 3 transition is characterized by a general "tightening" or regularization of the parameters.

Identifying outlier states and transitions. The paper compares the training maps of stable and outlier seeds to identify which transitions are missing or abnormal in the outliers. For the 410M model size, the outlier seeds (3 and 4) exhibit several distinctive features:

They fail to make the 2 → 3 transition that all the stable seeds make. This transition is characterized by decreases in median bias, L2-norm, and weight variance—a "regularization" phase that the outliers skip.
Instead, they make an abnormal 2 → 4 transition characterized by increases in the maximum singular value (λ_max ↑1.41), bias variance (σ_b ↑1.79), and singular value variance (σ_λ ↑1.71). This suggests the model enters a state where the weight matrices are becoming more anisotropic (some singular values growing large while others shrink) and the biases are becoming more variable across layers.
From state 4, they transition to state 0 or state 1—earlier states in the progression—rather than continuing forward to state 3 or a new state. These regressive transitions (4 → 0 and 4 → 1) are characterized by sharp decreases in singular value variance (σ_λ ↓2.99 and σ_λ ↓2.21, respectively), which the paper connects to the "representation degeneration" phenomenon described by Godey et al. (2024):

"the distribution of singular values first becomes increasingly uniform and then abruptly degenerates around a point."

In other words, the outlier runs first enter a state where the singular value spectrum spreads out (becomes more uniform), and then suddenly collapses—the effective rank of the weight matrices drops dramatically as most singular values shrink to near zero and only a few remain large. This degeneration coincides with the loss spikes and performance drops observed in these seeds (Figure 5 in Appendix A).

Bag-of-states representation. To use training maps predictively, the paper represents each training run as a bag-of-states: a vector of length $K$ (here, $K = 5$ ) where the $k$ -th entry is the number of checkpoints (out of the total 154) that were assigned to state $k$ . This is analogous to a histogram of state visits and captures the proportion of training time the model spent in each phase of the training dynamics. The bag-of-states is a lossy representation—it discards the temporal order of state visits—but the paper finds that for 410M it is sufficient to distinguish stable from outlier runs.

Predicting final performance from bag-of-states. The paper uses the bag-of-states as input features to a linear regression model that predicts the final performance of each training run. The outcome variable is a standardized accuracy score (z-score) computed as follows:

For each model size, the paper selects four benchmarks for which all models perform better than random: ARC (Easy), LAMBADA, Piqa, and SciQ.
For each benchmark, the paper computes the accuracy of the final checkpoint (step 143k) for each seed.
The accuracies are averaged across the four benchmarks to produce a single "average accuracy" for each seed.
Within each model size, the average accuracies are standardized to have mean 0 and standard deviation 1 (by subtracting the mean across seeds and dividing by the standard deviation across seeds). The resulting z-scores indicate how many standard deviations above or below the per-size mean each seed performs.

The linear regression model is:

$z_s = \beta_0 + \sum_{k=1}^{5} \beta_k \cdot \text{count}_s(k) + \epsilon_s$

where $z_s$ is the z-score for seed $s$ , $\text{count}_s(k)$ is the number of checkpoints where seed $s$ was in state $k$ , $\beta_k$ is the regression coefficient for state $k$ , $\beta_0$ is an intercept term, and $\epsilon_s \sim \mathcal{N}(0, \sigma^2)$ is a Gaussian error term.

What the regression computes: For each state $k$ , the coefficient $\beta_k$ tells us whether spending more checkpoints in that state is associated with higher or lower final performance. A positive $\beta_k$ means that models that dwell longer in state $k$ tend to have above-average performance; a negative $\beta_k$ means the opposite. The $R^2$ of the regression measures what fraction of the variance in final z-scores across seeds can be explained by the bag-of-states features alone.

Why this form: The linear model is the simplest possible predictor that can use the bag-of-states to make quantitative predictions. The paper is not claiming that the relationship between state visits and performance is strictly linear—this is an operational assumption made for simplicity and interpretability. The main purpose is to test whether the bag-of-states contains any signal about final performance: a significant $R^2$ (which the paper confirms for 410M, $R^2 = 0.99$ ) means that even this crude representation of the training trajectory carries substantial predictive information. The coefficients, reported in Table 3 (row "410M"), reveal that the outlier seeds (3 and 4) are the only ones with negative coefficients—the model correctly assigns them low predicted z-scores based on their unusual state visitation patterns.

Zero-shot prediction across model sizes. The paper tests a more ambitious hypothesis: can training maps learned from smaller models predict the training trajectory and final performance of a larger model? The procedure is:

Fit an HMM on all 10 seeds of a small model (e.g., 14M). This produces a set of emission distributions and transition probabilities that characterize the small model's training dynamics.
Take the 410M model's checkpoint sequences (the standardized parameter statistics at each step for each of its 10 seeds) and, for each checkpoint, compute the posterior probability of each of the 5 HMM states under the small model's HMM. Assign each 410M checkpoint to the state with the highest posterior probability. This is "zero-shot" because the HMM was never trained on 410M data—it only knows about the statistics of the small model's parameters.
For each 410M seed, compute a bag-of-states based on these zero-shot state assignments.
Regress the 410M z-scores on these zero-shot bag-of-states and report $R^2$ .

What this tests: If the zero-shot $R^2$ is high, it means that the parameter dynamics of small models are informative about the parameter dynamics of large models—the "states" that an HMM learns to recognize in a 14M model correspond to meaningful phases in the training of a 410M model, even though the absolute scales of the parameters are very different (which is why the standardization step is essential). The paper reports $R^2$ values in Table 3:

14M HMM → 410M prediction: $R^2 = 0.18$ (poor)
31M HMM → 410M prediction: $R^2 = 0.97$ (excellent)
70M HMM → 410M prediction: $R^2 = 0.97$ (excellent)
160M HMM → 410M prediction: $R^2 = 0.96$ (excellent)
410M HMM → 410M prediction: $R^2 = 0.99$ (excellent, for reference)

The failure of the 14M HMM to predict 410M performance ( $R^2 = 0.18$ ) is notable. The paper does not provide a detailed explanation, but possibilities include: the 14M model's training dynamics are qualitatively different from larger models (e.g., it saturates much earlier, as shown in Figure 1), so its HMM states capture phases that do not correspond to meaningful divisions in larger models; or the 14M model's parameter statistics have different variance structure due to its small size, causing the standardization to produce HMM states that do not transfer well.

The regression coefficients. Table 3 reports the per-state regression coefficients for each zero-shot prediction. The key finding is that the coefficients for the outlier seeds (3 and 4, highlighted in gray) are consistently large and negative across all HMMs that achieve good $R^2$ (31M and above):

"Seed 3: −0.56 to −1.12 (always negative). Seed 4: −2.74 to −2.80 (strongly negative)."

This means that regardless of which HMM is used to assign states, the outlier seeds' bag-of-states patterns consistently trigger low predicted z-scores—the "signature" of instability is robust enough to be detected by HMMs trained on completely different model sizes. For the stable seeds (0, 1, 2, 5, 6, 7, 8, 9), the coefficients are generally positive (range ~0.01 to ~0.69), indicating that spending time in the "normal" states is associated with above-average performance.

Temporal consistency of state transitions. The paper also examines when state transitions occur during training, reported in Table 2 (right columns) as the mean step ± standard deviation across seeds (excluding outlier seeds for 410M). The key findings:

Within each model size, transitions are highly consistent across seeds: The standard deviations are typically 1–5% of the mean step. For example, at 160M, the 0 → 1 transition occurs at step 2,000 ± 0 (all seeds within 1 checkpoint's resolution), and the 1 → 2 transition at step 18,000 ± 800.
Across model sizes, transitions follow a similar schedule but with size-dependent timing: The first transition (0 → 1) occurs between steps 2k and 7k for all sizes, aligning with the end of the learning rate warm-up phase (which the paper notes in Section 4). The second transition (1 → 2) occurs between 18k and 42k steps. The third transition (2 → 3) occurs between 58k and 73k steps—around the midpoint of training—and is the one missing in outlier maps. The final transition (3 → 4) occurs between 94k and 114k steps, in the later stages of training.
The 410M model is an exception: Even excluding the outlier seeds, the 410M model shows substantially higher variance in transition timing than the smaller models (e.g., the 0 → 1 transition has a standard deviation of 22,600 steps at 410M vs. 0–700 steps at smaller sizes). The paper leaves this as an open question for future work.

Limitation: early prediction is not yet possible. A practically important finding is that partial training maps (constructed from only the first $t$ checkpoints, for $t < 120\text{k}$ ) do not reliably predict final performance. The paper states:

"when we construct the bag-of-states using only a partial training run, we fail to predict the average z-score accurately. Empirically, we find that at least 120k steps (out of the total 143k) are required for a reliable prediction."

This means that while the training map provides a retrospective characterization of stability, it does not (in its current form) enable early stopping of problematic runs. The features that distinguish stable from unstable trajectories only become apparent in the later stages of training, after the critical transitions have (or have not) occurred. The paper frames improving early prediction as an important direction for future work.

Summary of Design Choices and Their Justifications

Training 10 seeds per size rather than 2 or 3: Provides sufficient statistical power to estimate variance (with 10 runs, the standard error of the estimated standard deviation is approximately $\sigma / \sqrt{18} \approx 0.24\sigma$ , giving reasonable precision for outlier detection). Fewer seeds would make it impossible to distinguish genuine outliers from normal variation.
Dense log-spaced early checkpoints (every power of 2 through 512): Captures the rapid learning dynamics in the first few hundred steps, when the loss drops fastest and basic capabilities emerge. Linear spacing would miss this region entirely or sample it too coarsely.
Using Cohen's κ rather than raw agreement: Corrects for chance-level agreement, which is essential when comparing models on tasks with few answer options. A raw agreement of 50% on a binary task is no better than random (κ = 0), but on a 100-way task, it would be extremely significant. κ normalizes for the number of classes and the marginal distribution of predictions, making it comparable across benchmarks with different numbers of answer choices.
Learned layer weighting in probing (α in the global representation): Avoids the need to manually select which layer(s) to probe, which would introduce experimenter degrees of freedom and potentially bias the analysis toward finding the results the experimenter expects. The learned weighting also adapts to the fact that different linguistic phenomena are encoded at different depths.
MDL objective rather than standard cross-entropy for probe training: Addresses the well-documented problem that high-capacity probes can "cheat" by memorizing label patterns rather than reading out information genuinely present in the representations. The MDL penalty ensures that high probe accuracy reflects efficient, structured encoding of linguistic information, not probe overfitting.
Standardizing HMM input features across seeds: Ensures that the HMM's state assignments are driven by the relative patterns within each model size (e.g., "this checkpoint has unusually high singular value variance") rather than the absolute scale of the statistics (e.g., "this checkpoint has large L2-norm because it's a large model"). This is what enables the zero-shot cross-model-size prediction.
Fixing K = 5 states across all model sizes: Enables direct comparison of training maps across sizes. If each model size used a different $K$ , the meaning of "state 2" would differ, making cross-size comparison of state transitions impossible. The paper validates this choice by showing that $K = 5$ is near optimal for BIC across all sizes, confirming that the data supports (roughly) 5 distinct phases of training dynamics.
Using bag-of-states rather than full state sequence for regression: The bag-of-states is a dimensionality reduction (from 154-step sequences to 5-element vectors) that makes regression feasible with only 10 data points (seeds). A full sequence model (e.g., an RNN or transformer) would have far more parameters than data points and would severely overfit. The bag-of-states trades temporal resolution for statistical tractability, and the paper shows this tradeoff is acceptable for distinguishing outliers.

4. Key Insights and Innovations

Innovation 1: Stability as a First-Class Object of Study Across the Full Pre-Training Pipeline

The paper's most distinctive conceptual move is elevating stability from an afterthought—a variance estimate tacked onto a performance table—to a primary object of scientific investigation that spans the entire model training pipeline, from parameter dynamics through internal representations to downstream behavior. This is not merely "measuring variance across seeds," which prior work (Sellam et al., 2022; Dodge et al., 2020) already did at final checkpoints. The intellectual shift is studying stability longitudinally, asking not just whether different seeds converge to different endpoints, but when and how their trajectories diverge, and whether divergence at one level (e.g., parameter statistics) propagates to others (e.g., benchmark accuracy).

Prior work treated stability as a property of final outputs. The MultiBERTs (Sellam et al., 2022) reported variance in downstream task performance across 25 BERT-base checkpoints but provided almost no intermediate checkpoints, making it impossible to ask when instability emerges or whether unstable final outcomes are preceded by detectable early signals. Dodge et al. (2020) showed that fine-tuning is sensitive to random seeds, but again only at final checkpoints. The field's implicit assumption was that training trajectories are essentially monotonic and that studying endpoints suffices to characterize stability.

PolyPythia fundamentally challenges this assumption by demonstrating that stability is temporally structured. The training maps (Figure 4, Table 2) reveal that stable and unstable runs follow identical trajectories through the first two HMM states, diverging only at the 2 → 3 transition around step 58k–73k—well into training. The self-consistency metric (Figure 1, right column) shows that models' predictions stabilize at different rates across tasks, with some benchmarks achieving κ ≈ 0.5 by step 10³ and others not converging until step 10⁵. The representational shift metrics (SSAs in Figure 3) show that linguistic representations undergo rapid reorganization early in training (SSA increasing through step 10³) before settling into slower, more stable refinement (SSA decreasing after step 10⁴). None of this temporal structure would be visible from final-checkpoint comparisons alone.

This reframing matters because it changes what questions researchers can ask about training dynamics. Instead of "is this model stable?" (a binary question answerable with final-checkpoint variance), the PolyPythia framework enables "at what phase does instability arise?", "can we detect it before it affects performance?", and "does representational instability precede or follow parameter instability?" These are mechanistic questions about the causal structure of training instability, not merely descriptive ones about outcome variance. The paper's finding that outlier 410M runs enter HMM state 2 prematurely (Section 5) and fail to make the 2 → 3 transition that stable runs make—long before their performance diverges visibly—is a concrete example of how temporal analysis reveals structure invisible in endpoint comparisons.

This is a fundamental advance rather than incremental because it establishes a new axis of analysis (temporal stability) that was previously inaccessible, much as scaling laws (Kaplan et al., 2020) established model size as an axis for studying performance rather than treating each model as an isolated artifact. The PolyPythia resource enables a research program in training dynamics stability analogous to what scaling laws enabled for performance prediction.

Innovation 2: Training Maps as a Diagnostic Tool That Unifies Parameter Dynamics, Performance, and Outlier Detection

The paper adapts training maps (Hu et al., 2023) not merely as a descriptive visualization but as a diagnostic instrument that can retrospectively characterize why some runs fail and—in a zero-shot transfer setting—can be learned from small models and applied to larger ones to predict final performance. The conceptual innovation is the demonstration that the simple bag-of-states representation, derived from an HMM fitted to parameter statistics, captures sufficient information about the quality of a training trajectory to distinguish stable from unstable runs with $R^2 = 0.99$ for the 410M model (Table 2).

Prior work on training dynamics diagnostics focused on post-hoc analysis of individual runs: Saphra and Lopez (2019) used representational similarity to identify learning phases in LSTMs, Michaelov and Bergen (2023) documented saturation in Pythia models, and Godey et al. (2024) characterized representation degeneration through singular value analysis. All of these identify what happens during training but not whether it is normal—there is no baseline of expected behavior against which to compare an individual run. Hu et al. (2023) introduced training maps as a visualization but did not connect them to performance prediction or demonstrate cross-model-size transfer.

The PolyPythia analysis makes three leaps beyond this prior work. First, it establishes that training maps have a normative structure: linear maps (simple sequential state transitions) correspond to stable, well-performing runs, while maps with forks (regressive transitions to earlier states) signal pathology. This is an empirical finding, not an assumption—the paper shows that the only runs with forked maps are the outlier 410M seeds 3 and 4, which also exhibit loss spikes and degraded performance (Figure 4, top vs. bottom). Second, it shows that the specific transition missed by outliers (2 → 3) has a characteristic signature: decreases in median bias, L2-norm, and weight variance (Table 1)—essentially, a regularization phase that unstable runs skip. This moves beyond pattern-matching ("this map looks different") toward mechanistic interpretation ("this specific phase of parameter tightening is critical"). Third, it demonstrates that training maps are transferable across model scales: HMMs fitted on models as small as 31M parameters can assign states to the 410M model in a way that preserves the diagnostic signal, achieving $R^2 = 0.97$ for performance prediction (Table 3). The failure of the 14M HMM ( $R^2 = 0.18$ ) is itself informative—it establishes a lower bound on the model size needed for meaningful cross-scale transfer, suggesting that the smallest models have qualitatively different training dynamics (consistent with their earlier saturation, visible in Figure 1).

The significance of this innovation is practical as much as scientific. If training maps from small models can predict the performance trajectory of large models mid-training, this opens the possibility of early termination of problematic runs—saving substantial compute by detecting instability before the full training budget is expended. The paper's finding that at least 120k out of 143k steps are needed for reliable prediction (Section 5) tempers this optimism, but the conceptual framework is in place: the bag-of-states representation is shown to contain the relevant signal; the remaining challenge is extracting it from partial trajectories rather than full ones.

This is a fundamental advance in diagnostic methodology rather than an incremental improvement, because it converts training maps from a post-hoc visualization into a predictive instrument with demonstrated cross-scale generalization. The connection to the singular value distribution (the "representation degeneration" link to Godey et al., 2024) suggests a physical interpretation of the HMM states that could guide future interventions to prevent instability, rather than merely detecting it.

Innovation 3: The Identification of Two Robust, Scale-Invariant Learning Phases

Through the multi-level stability analysis, the paper identifies two distinct learning phases that are remarkably consistent across model sizes, seeds, and analysis methods: an initial phase between steps 10³ and 10⁴ (roughly 2B–20B tokens) where linguistic information begins to be encoded, and a critical phase between steps 10⁴ and 10⁵ (roughly 20B–200B tokens) where most performance improvements occur. The conceptual contribution is not the existence of phases—prior work had identified phase transitions in specific models—but the demonstration that these phases are robust to randomness factors and to the choice of measurement instrument.

The evidence for this robustness is triangulated across three independent analysis frameworks. Downstream accuracy (Figure 1, left column) shows performance improvements concentrated between steps 10³ and 10⁵, with a performance dip between 10⁴ and 10⁵ for smaller models (the "saturation" effect from Michaelov and Bergen, 2023). The information-theoretic probing metrics (Figure 3) show that macro-F1, codelength ratio, and representational shift all undergo their most rapid changes in this same window: linguistic information content increases sharply, representational efficiency begins improving, and SSAs reach their peak (indicating maximum representational reorganization) around step 10⁴ before declining. The training maps (Table 2) show that HMM state transitions consistently occur within this window across all model sizes: the 0 → 1 transition at steps 2k–7k, the 1 → 2 transition at steps 18k–42k, and the 2 → 3 transition at steps 58k–73k all fall between approximately 10³ and 10⁵ steps.

What makes this finding an innovation rather than a confirmation of known results is its invariance properties. Prior work on phase transitions often studied single models (Saphra and Lopez, 2019), or compared across model sizes but only with single seeds (Biderman et al., 2023b; Tigges et al., 2024). These studies could not distinguish between behaviors that are universal properties of the learning algorithm and those that are contingent on initialization or data order. For example, Olsson et al. (2022) identified that induction heads emerge around 2.5B tokens in transformer LMs—but this was based on a single training run. The PolyPythia analysis shows that the broader phase structure in which induction heads emerge (the initial learning phase) is consistent across 50 training runs spanning a 30× range of model sizes, providing strong evidence that this temporal structure is a fundamental property of autoregressive language model training on natural language data, not an artifact of specific initial conditions.

The paper also connects these phases to prior findings about what emerges during them—induction heads and in-context learning (Olsson et al., 2022) in the initial phase, grammatical gender agreement and gender bias (BLiMP, CrowS-Pairs, Simple Co-occurrence Bias in Figure 2) around step 10²–10³, and the settlement of inter-seed agreement (Figure 1, middle column) peaking at κ ≈ 0.7 around step 10³ before converging to κ ≈ 0.5. This synthesis across behavioral, representational, and parameter-level analyses produces a coherent picture: the initial phase is about acquiring core linguistic and semantic capabilities (representations become informative, basic syntactic distinctions emerge), while the critical phase is about refining and stabilizing those capabilities (representations become more efficient, predictions become more self-consistent, models settle on their final answer patterns).

This is an incremental advance in terms of the individual observations (each phase-related finding has precedents in prior work) but a fundamental contribution in terms of synthesis and generalization. By demonstrating that the same phase boundaries appear across three independent measurement frameworks and across five model sizes with ten seeds each, the paper transforms phase transitions from an interesting phenomenon observed in specific models to a robust, generalizable property of the pre-training process—one that future research can treat as a reliable structural feature rather than a contingent finding to be re-verified in each new setting.

Innovation 4: The Discovery That Stability Patterns Are Scale-Invariant (With a Lower Bound), Enabling Small-Model Prototyping

The paper demonstrates that representational stability trajectories are highly correlated across model sizes (Pearson $\bar{r} = 0.99$ for macro-F1, $0.98$ for codelength ratio, $0.94$ for SSA; all $p < 0.001$ ), and that training maps from models as small as 31M parameters can predict the performance of 410M models with $R^2 = 0.97$ . This is not merely an empirical observation—it is a finding with direct methodological implications for how the field studies training dynamics, because it means that expensive large-scale training runs are not always necessary to understand the qualitative structure of the learning process.

The dominant assumption in the field—implicit in the scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) and explicit in many large-model releases—is that larger models may exhibit qualitatively different behaviors than smaller ones. This assumption is reasonable: larger models have more capacity, can memorize more, and might undergo different phase transitions or develop capabilities that smaller models cannot. The PolyPythia findings challenge this assumption for the specific property of training stability and representational learning dynamics at the scales studied (14M–410M). The key evidence is that the shapes of the learning curves—when metrics begin to improve, when they plateau, when they exhibit saturation—are essentially identical across model sizes, differing only in their asymptotic values (larger models plateau higher) but not in their temporal structure.

The important caveat is the lower bound: the 14M model is the exception, achieving only $R^2 = 0.18$ for zero-shot prediction of 410M performance (Table 3). This establishes that scale invariance does not hold for the smallest model. The paper does not fully explain this failure, but the evidence points to qualitative differences in the 14M model's training dynamics: it saturates earlier and more severely (Figure 1, where 14M accuracy drops between 10⁴ and 10⁵ while larger models continue improving), and its training map transitions occur at different relative positions in the training process (Table 2, where 14M's 0 → 1 transition at step 5k is proportionally later than 160M's at step 2k). The existence of this lower bound is itself informative—it suggests that there is a minimum model size below which training dynamics become qualitatively different, and that 31M parameters is above this threshold for the Pythia architecture and the Pile dataset.

The practical significance of this innovation is that it enables a research methodology previously unavailable: prototype and debug training dynamics at small scale, then transfer insights to larger models. If a researcher wants to understand whether a new training intervention (e.g., a different learning rate schedule, a new data mixture, a regularization technique) will affect the phase structure of training or the stability of representations, they can test it on a 70M model and have reasonable confidence that the qualitative patterns will transfer to a 410M model. This dramatically lowers the computational barrier to entry for training dynamics research—a 70M model can be trained on a single GPU in hours, while a 410M model requires substantially more resources. The paper's release of the PolyPythia checkpoints at multiple scales provides the raw material to validate this transfer methodology for a wide range of analyses beyond those presented in the paper.

This finding is incremental in the sense that it extends the known property of performance scaling laws (larger models perform better, but performance improves monotonically with scale) to the domain of training dynamics. However, it is fundamental in its implications for research methodology: it provides empirical justification for a scale-down-then-transfer approach to studying learning dynamics, reducing the compute barrier for a class of research questions that previously required training at the scale of interest. The failure at 14M also provides a cautionary boundary—researchers should be wary of drawing conclusions about large-model dynamics from models below roughly 30M parameters, at least for this architecture and dataset.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the Pile dataset (Gao et al., 2021) for pre-training—a 300B-token curated collection of English documents, using the standard non-deduplicated version as in the original Pythia release (Biderman et al., 2023b). For downstream evaluation, the paper uses a set of eight multiple-choice benchmarks: ARC (Easy) and ARC (Challenge) (Clark et al., 2018), LAMBADA (Paperno et al., 2016), Logiqa (Liu et al., 2020), Piqa (Bisk et al., 2020), SciQ (Welbl et al., 2017), WinoGrande (Sakaguchi et al., 2020), and WSC (Levesque et al., 2012). For representational analysis, seven linguistically motivated token-classification tasks are used: coreference resolution (Coref), dependency parsing (Dep), named entity recognition (NER), part-of-speech tagging (PoS), semantic tagging (SemTag), sentiment analysis (Senti), and topic classification (Topic), all drawn from standard OntoNotes 5.0 and other widely-used linguistic corpora (Pradhan et al., 2013; Silveira et al., 2014; Abzianidze et al., 2017; Socher et al., 2013; Lang, 1995). For gender bias evaluation, three benchmarks are used: BLiMP (Gender Agreement) (Warstadt et al., 2020), CrowS-Pairs (Gender) using the Névéol et al. (2022) adaptation, and Simple Co-occurrence Bias (Smith et al., 2022).
Base model(s). The PolyPythia suite extends the original Pythia model suite (Biderman et al., 2023b), training decoder-only transformer language models at five sizes: 14M, 31M, 70M, 160M, and 410M parameters. For each size, 10 training runs are available (the original seed 0 plus 9 new seeds 1–9), each using the identical GPT-NeoX codebase (Andonian et al., 2023), hyperparameters (cosine learning rate schedule with warm-up, batch size of 1,024 sequences of 2,048 tokens each, 143k optimization steps total), and training data as the original Pythia release, varying only the random seed which controls both parameter initialization and the shuffling of documents prior to packing into training sequences. The choice of these specific sizes reflects a deliberate tradeoff prioritizing seed exploration (10 seeds) and checkpoint granularity (154 per run) over scaling to larger models, motivated by the goal of enabling stability research at modest compute budgets while still spanning a meaningful range of scales.
Metrics. The paper quantifies stability through a multi-level suite of metrics spanning the full training pipeline. At the output level, downstream accuracy (fraction of correct multiple-choice answers via log-likelihood scoring), inter-seed agreement (Cohen's κ between each seed's predictions and seed 0's predictions, where κ = 1 indicates perfect agreement and κ = 0 indicates chance-level agreement), and self-consistency (Cohen's κ between a model's predictions at an earlier checkpoint and its final checkpoint) are computed for each benchmark. At the representation level, macro-F1 (per-class F1 averaged across classes) measures information content extractable by linear probes trained with an MDL objective; codelength ratio (the ratio of probe codelength on trained vs. randomly initialized representations, where values near 0 indicate high efficiency) measures representational efficiency; and principal subspace angles (SSAs) between consecutive checkpoints' probe weight matrices (ranging 0°–90°, where lower values indicate more similar representations) measure representational shift. At the parameter level, training maps are constructed from Hidden Markov Models (HMMs) fitted to 15 statistics of model weights and biases (L1-norm, L2-norm, L1/L2 ratio, means, medians, variances, trace, spectral norm, singular value statistics), and the resulting bag-of-states representation (counts of how many checkpoints visit each HMM state) is used to predict final performance via linear regression, with goodness-of-fit measured by $R^2$ and per-state coefficients. For gender bias, accuracy (BLiMP Gender Agreement), proportion of stereotypical choices (CrowS-Pairs Gender, where 0.5 indicates no bias), and proportion of male-preferred completions (Simple Co-occurrence Bias, where 0.5 indicates no skew) are reported.
Baselines. The original Pythia suite (Biderman et al., 2023b) serves as the single-seed baseline—one training run per model size with seed 0, which is included in the PolyPythia release alongside the 9 new seeds. For the analysis of stability, the key baseline is the distribution across seeds: each metric is reported as the median and interquartile range across the 10 seeds for each model size and checkpoint, and outlier seeds are identified as those falling more than 2 standard deviations from the per-size mean on standardized accuracy across four benchmarks (ARC Easy, LAMBADA, Piqa, SciQ). The paper does not introduce new algorithmic baselines (e.g., alternative training recipes or stability interventions) because the contribution is a resource and analysis framework, not a method; the comparison is between seeds within the same training configuration, not between different training strategies.
Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of test-time compute scaling, since all analysis is on pre-trained models. Instead, the implicit compute budget is the fixed 143,000 optimization steps (~300B tokens) per training run, identical across all seeds and model sizes. Compute accounting is not formalized in FLOPs terms (as would be done in scaling laws work), because the goal is to study variance across runs with identical compute expenditure, not to compare the efficiency of different compute allocations. The evaluation itself uses a subset of checkpoints (log-spaced early checkpoints through step 1k, plus every 10k-th step from 3k to 143k) to limit computational costs of downstream and probing evaluation, but the training maps analysis (Section 5) uses all 154 checkpoints per run because HMM fitting benefits from temporal density.
Cross-validation / statistical protocol. For outlier identification (Section 3), the paper uses a heuristic based on standardizing per-task accuracy across seeds within each model size and flagging model-seed combinations outside a 2-standard-deviation region—the authors explicitly note this is a simple heuristic rather than a formal statistical test (e.g., ANOVA with Tukey's test), which they acknowledge would require a larger sample size of tasks. For representational stability analysis (Section 4), Pearson correlations between metric trajectories across model sizes are computed on the concatenation of all-task, all-checkpoint values per seed, with Fisher z-transformation used to average correlations across pairs correctly. For training map prediction (Section 5), linear regression with bag-of-states features is evaluated by $R^2$ , and zero-shot cross-model-size prediction uses HMMs trained entirely on smaller model data, with no fine-tuning or adaptation to the target model size. No formal cross-validation over folds of seeds is performed, because the small number of seeds (10 per size) makes train/test splits impractical—the analyses are descriptive rather than predictive in the machine learning sense, with the regression serving as a quantification of association strength rather than a deployed predictive model.

Main Quantitative Results

Downstream Performance Stability

Aggregate accuracy trajectories show consistent scaling but reveal saturation in small models. Across all benchmarks where performance exceeds random chance (ARC Easy, Piqa, SciQ, LAMBADA; Figure 1, left column; Figure 6 in Appendix D), larger models consistently outperform smaller ones, with accuracy improving most rapidly between steps 10³ and 10⁴ before stabilizing. Specifically, on ARC (Easy), the 410M model reaches a median accuracy of approximately 0.50 at the final checkpoint (step 143k), compared to approximately 0.40 for the 160M, 0.38 for the 70M, and dropping further for smaller models—a monotonic scaling relationship consistent with the original Pythia findings (Biderman et al., 2023b). On SciQ, the pattern is similar: 410M achieves approximately 0.70 final accuracy, 160M approximately 0.62, and smaller models lower still. However, a notable non-monotonicity emerges for the smallest models: accuracy on ARC (Easy) for the 14M and 31M models drops between steps 10⁴ and 10⁵ (e.g., 14M declines from approximately 0.34 to 0.30), a "saturation" phenomenon consistent with prior observations by Michaelov and Bergen (2023) and Godey et al. (2024), attributed to a mismatch between the model's representational dimension and the high rank of the output embedding matrix. On the more challenging benchmarks (ARC Challenge, Logiqa, WSC, WinoGrande; Figure 6), all models perform at or near random chance throughout training, providing no signal for stability analysis—a finding consistent with Biderman et al. (2023b) that these model sizes are below the threshold for meaningful performance on these tasks.

Inter-seed agreement reveals that models converge to similar predictions but not identical ones. Across benchmarks, inter-seed agreement (Cohen's κ between each seed's predictions and seed 0's predictions; Figure 1, middle column) peaks at κ ≈ 0.7 around step 10³, then declines and stabilizes at κ ≈ 0.5 ("moderate agreement") from step 10⁴ onward. This pattern is visible on both ARC (Easy) and SciQ: the median κ across the 9 non-reference seeds rises from near 0 at initialization to approximately 0.7 around step 1k–2k, then settles to approximately 0.5–0.55 for the remainder of training. The peak at step 10³ is hypothesized to reflect all models initially converging on the same incorrect answer before diverging as they learn—the fact that inter-seed agreement is higher early in training than later suggests that early-stage models share common failure modes, and that true differentiation in what models "know" occurs during the critical learning phase (10⁴–10⁵ steps). The final κ ≈ 0.5 indicates that while different seeds achieve similar aggregate accuracy, they get different individual examples wrong—the models are not merely noisy copies of each other but have genuinely different "opinions" about borderline cases.

Self-consistency trajectories show that predictions stabilize gradually, with distinct phases. Self-consistency (κ between earlier and final predictions; Figure 1, right column) increases steadily to κ ≈ 0.5 by step 10³, plateaus through step 10⁴, and then resumes increasing to higher values (κ ≈ 0.7–0.8 on ARC Easy, κ ≈ 0.6–0.7 on SciQ) by the end of training. This two-phase pattern—rapid initial convergence to moderate self-consistency, a plateau, then a second phase of further convergence—mirrors the learning phase structure identified across other metrics. The plateau between 10³ and 10⁴ coincides with the period when inter-seed agreement declines (models are differentiating from each other), while the late-training increase in self-consistency reflects models settling into their final answer patterns. The smaller models (14M, 31M) show lower final self-consistency than larger ones, consistent with their saturation and ongoing representational instability.

Outlier seeds are rare but identifiable: only two model-seed combinations across all sizes and tasks fall outside 2 standard deviations. Using the heuristic of standardizing final-checkpoint accuracy across seeds for the four tasks where all models exceed random performance (ARC Easy, LAMBADA, Piqa, SciQ), the paper identifies only two outlier model-seed combinations: 410M seeds 3 and 4 (Section 3). These are the only runs that exhibit "loss spikes" during training (Figure 5 in Appendix A), confirming that the outlier detection method based on downstream performance successfully identifies the same problematic runs that are visible in training loss. The fact that no outliers are detected at smaller model sizes (14M–160M) suggests that training instability is more prevalent at larger scales, though the limited sample (10 seeds per size) precludes strong statistical conclusions about this trend.

Gender bias emerges sharply around step 10²–10³ with high variance across seeds. On BLiMP (Gender Agreement) (Figure 2, left), all model sizes start near chance accuracy (~0.50) and improve to 0.80–0.95 by the end of training, with the sharpest improvement occurring between steps 10² and 10³. The 410M model reaches the highest accuracy (~0.95), while smaller models saturate at lower levels (~0.80 for 14M), mirroring the general scaling pattern. On CrowS-Pairs (Gender) (Figure 2, middle), the proportion of stereotypical choices increases from ~0.50 at initialization to ~0.55–0.60 by training end, indicating a modest bias toward stereotypes that emerges during training. The interquartile range across seeds is notably wide (spanning ~0.10–0.15) compared to accuracy metrics, which the paper attributes to the small size and known reliability issues of CrowS-Pairs rather than genuine variance in model bias (Section 3). On Simple Co-occurrence Bias (Figure 2, right), a strong male skew emerges abruptly around step 10²–10³, with the proportion of male-preferred completions jumping from ~0.50 to ~0.60–0.75 depending on model size, and remaining elevated thereafter—suggesting that crude gendered associations are learned early and persist.

Representational Stability of Linguistic Information

Macro-F1 shows consistent scaling and a shared temporal profile across model sizes. Across all seven linguistic probing tasks (Figure 3, top row), macro-F1 improves monotonically with model size and follows a characteristic trajectory: initial improvements beginning around step 10³, rapid increase through step 10⁴, and relative stability thereafter, with smaller models showing some degradation in the final phase (consistent with saturation). Lower-level syntactic tasks (PoS, Dep) achieve higher absolute macro-F1 (reaching ~0.85–0.95 for the largest models) than higher-level semantic tasks (NER, Senti reaching ~0.60–0.80), consistent with the established finding that syntax is more linearly decodable from transformer representations than semantics. The standard deviation across seeds (shaded regions) is narrow for most tasks, indicating that the amount of linguistic information encoded in representations is highly stable across randomness factors—different seeds learn to encode similar amounts of linguistic structure.

Codelength ratio reveals that representational efficiency improves most between 10³ and 10⁴ steps, with a size-dependent asymptote. The codelength ratio (Figure 3, middle row) starts near 1.0 for all tasks and model sizes (representations are no more efficient than random noise at initialization), begins decreasing around step 10³, and reaches close-to-final values by step 10⁴. The asymptotic codelength ratio depends on both task and model size: for syntactic tasks (PoS, Dep, Coref), the ratio converges to similar values (~0.55–0.65) across all model sizes above 31M, while for semantic tasks (NER, Senti, SemTag, Topic), larger models achieve lower (better) ratios—the 410M model reaches ~0.40–0.50 for NER and Senti, while the 14M model plateaus at ~0.50–0.60. This 5–10 percentage point absolute difference indicates that larger models encode semantic information not only more accurately (higher macro-F1) but also more efficiently (requiring fewer probe parameters to extract the same information, as captured by the MDL regularization). The paper highlights this as a distinction between models below and above ~100M parameters (Section 4).

Principal subspace angles show that representational reorganization peaks at step 10³–10⁴ and then declines, following the learning rate schedule. SSAs between consecutive checkpoints' probe subspaces (Figure 3, bottom row) follow a remarkably consistent trajectory across model sizes and tasks: they increase rapidly from near 0° at initialization to a peak of ~60°–80° around step 10³–10⁴ (coinciding with the end of the learning rate warm-up phase at step ~2k and the subsequent high-learning-rate period), then decline steadily to ~15°–25° by the end of training. The paper notes (Section 4) that SSA "roughly follows the learning rate schedule," suggesting that the rate of representational change is driven primarily by the optimizer's step size rather than by task-specific learning dynamics. The fact that SSA does not directly correlate with macro-F1 or codelength ratio—large SSA early in training corresponds to improvements in those metrics, while small SSA late in training can correspond to degradation in the smallest models—highlights that SSA captures a distinct aspect of the learning process: the amount of representational change, not its direction (beneficial vs. harmful).

Representational stability trajectories are highly correlated across model sizes. The paper computes the average Pearson correlation (via Fisher z-transformation) between each metric's trajectory (concatenated across all tasks and checkpoints) for pairs of model sizes (Section 4). Macro-F1 trajectories achieve $r = 0.99$ , codelength ratio achieves $r = 0.98$ , and SSA achieves $r = 0.94$ (all $p < 0.001$ ), indicating that the shape of the learning curves—when each metric begins to change, when it peaks, when it stabilizes—is essentially identical across scales from 14M to 410M parameters, even though the absolute values differ by size. This finding is central to the paper's claim that stability patterns are scale-invariant: a researcher studying the temporal structure of representational learning on a 70M model would observe the same phase boundaries and qualitative dynamics as on a 410M model.

Training Phase Dynamics and Outlier Detection via Training Maps

Training maps for stable runs are linear with consistent state transitions. Across all model sizes and seeds, the HMM fitted to parameter statistics (with $K = 5$ states) produces training maps that are linear graphs—a simple sequential progression through states 0 → 1 → 2 → 3 → 4—for all stable runs (Figure 4, bottom-left panel for 410M stable seeds). The timing of these transitions is consistent across seeds within each model size (Table 2, right columns): for example, at 160M, the 0 → 1 transition occurs at step 2,000 ± 0 (all seeds within one checkpoint's resolution of 1,000 steps), the 1 → 2 transition at step 18,000 ± 800, the 2 → 3 transition at step 61,000 ± 1,500, and the 3 → 4 transition at step 100,000 ± 1,600. The standard deviations are typically 1–5% of the mean transition step, indicating high temporal consistency.

The three main learning phases identified across sections map onto HMM state transitions. The initial learning phase (10³–10⁴ steps) corresponds to the 0 → 1 and 1 → 2 transitions, which occur at steps 2k–7k and 18k–42k respectively across model sizes (Table 2). The critical learning phase (10⁴–10⁵ steps) corresponds to the 2 → 3 transition (steps 58k–73k) and the beginning of the 3 → 4 transition (starting at steps 94k–114k). The paper notes (Section 6) that these HMM-derived phase boundaries align with the phases identified independently from downstream performance (Figure 1), representational metrics (Figure 3), and prior work on circuit formation (Olsson et al., 2022; Tigges et al., 2024)—a triangulation across three independent analysis methods that strengthens the claim that these are genuine, robust properties of the training process.

Outlier 410M seeds (3 and 4) exhibit forked training maps and fail to make the critical 2 → 3 transition. The training maps for the two outlier seeds (Figure 4, top-left) are the only ones across all 50 training runs that contain "forks"—regressive transitions to earlier states. Specifically, both seeds enter state 2 prematurely and fail to transition to state 3 (the purple → brown transition visible in stable maps; Figure 4, bottom-left). Instead, they make an abnormal 2 → 4 transition, followed by regressive transitions 4 → 0 or 4 → 1, which coincide with the loss spikes and sharp performance drops visible in the corresponding accuracy trajectories (Figure 4, top-right, where seeds 3 and 4 show performance degradation) and training loss (Figure 5).

The failed 2 → 3 transition is characterized by missing parameter regularization. Table 1 identifies the top three drivers of the 2 → 3 transition that stable runs make but outliers skip: decreases in median bias (standardized effect ↓0.44), L2-norm of weights (↓0.42), and weight variance (↓0.50). These are all negative-direction changes indicating a "tightening" or regularization phase—the model's parameters become smaller in magnitude and less variable across layers. The abnormal 2 → 4 transition that outliers make instead is driven by increases in spectral norm (λ_max ↑1.41), bias variance (σ_b ↑1.79), and singular value variance (σ_λ ↑1.71)—the model enters a state where weight matrices become more anisotropic (some singular values grow large, others shrink) and biases become more variable. The subsequent regressive transitions (4 → 0 and 4 → 1) are driven by sharp decreases in singular value variance (σ_λ ↓2.99 and ↓2.21 respectively), which the paper links to the "representation degeneration" phenomenon from Godey et al. (2024): "the distribution of singular values first becomes increasingly uniform and then abruptly degenerates around a point" (Section 5). This suggests a causal sequence: the model enters an unstable regime where the singular value spectrum spreads out, followed by a collapse that correlates with catastrophic performance degradation.

Bag-of-states regression achieves $R^2 = 0.99$ for predicting 410M final performance. Using the bag-of-states (counts of checkpoints in each HMM state) as features in a linear regression predicting the standardized average accuracy z-score across ARC (Easy), LAMBADA, Piqa, and SciQ (Table 2, 410M row), the paper achieves $R^2 = 0.99$ , meaning the simple count of how many checkpoints visited each state explains nearly all variance in final performance across the 10 seeds. The regression coefficients (Table 3, 410M row) for the outlier seeds 3 and 4 are consistently negative (−0.57 and −2.80 respectively), while all stable seeds (0, 1, 2, 5, 6, 7, 8, 9) have positive coefficients (ranging 0.29 to 0.49), indicating that the bag-of-states clearly separates high-performing from low-performing runs.

Regression is not predictive for smaller model sizes where no performance variance exists. For the 14M–160M models, $R^2$ values range from 0.03 (160M) to 0.75 (14M) (Table 2, R² column). However, these low $R^2$ values do not indicate that training maps are uninformative for these sizes—rather, they reflect the near-absence of performance variance to predict. Table 2 (σ² column) shows that the variance of z-scores (standardized average accuracy) is 0.65 for 14M, 0.84 for 31M, 0.71 for 70M, 0.58 for 160M, and 0.98 for 410M. Only 410M has substantial performance differences across seeds to explain. The paper does not report whether alternative performance metrics (e.g., validation loss, individual task accuracy) show more variance at smaller sizes, which could potentially yield higher $R^2$ values.

Zero-shot cross-model-size prediction achieves $R^2 > 0.96$ for all HMMs from models 31M and above. Table 3 shows the key cross-model-size transfer results: HMMs trained on 31M, 70M, and 160M models, when applied zero-shot to assign states to the 410M checkpoints and then used to predict 410M z-scores via bag-of-states regression, achieve $R^2$ of 0.97, 0.97, and 0.96 respectively. The outlier seeds (3 and 4, highlighted in gray) consistently receive large negative coefficients regardless of which smaller model's HMM is used: seed 3 coefficients range from −0.56 to −1.12, seed 4 coefficients from −2.74 to −2.80 (the 14M HMM is an exception, producing uninterpretable coefficients due to its poor overall fit, $R^2 = 0.18$ ). This demonstrates that the "signature" of instability—the pattern of state visitations that characterizes a failing run—is robust enough to be detected by HMMs trained on models 13× smaller (31M vs. 410M).

The 14M HMM fails to transfer to 410M, establishing a lower bound for scale invariance. The 14M-trained HMM achieves only $R^2 = 0.18$ for predicting 410M performance (Table 3, first data row), and its regression coefficients do not consistently identify the outlier seeds (seed 4 receives a coefficient of −1.12, but seed 3 receives +0.39, and several stable seeds receive near-zero coefficients). This failure establishes that the scale invariance of training dynamics has a lower bound: below approximately 30M parameters, the qualitative structure of parameter dynamics becomes sufficiently different from larger models that HMMs trained at this scale no longer transfer. The paper does not definitively explain this threshold, but evidence from other analyses points to the 14M model's severe saturation (Figure 1), distinct transition timing (Table 2: 14M's 0 → 1 transition at step 5,000 is proportionally later in training than 160M's at step 2,000), and overall fragility as potential causes.

Early prediction from partial training maps is not currently possible. A practically important null result: "when we construct the bag-of-states using only a partial training run, we fail to predict the average z-score accurately. Empirically, we find that at least 120k steps (out of the total 143k) are required for a reliable prediction" (Section 5). This means that while the training map retrospectively characterizes stability, it does not enable early termination of problematic runs—the features that distinguish stable from unstable trajectories (the 2 → 3 transition and its abnormal alternatives) only become apparent after ~84% of training is complete. The paper identifies improving early prediction as a key direction for future work.

Ablation Studies and Robustness Checks

Cross-benchmark replication of accuracy, inter-seed agreement, and self-consistency patterns: The paper reports results for all eight downstream benchmarks in Appendix D (Figures 6, 7, 8). While the main text focuses on ARC (Easy) and SciQ, the qualitative patterns hold across all benchmarks where models perform above random chance: accuracy improves most between 10³ and 10⁴, inter-seed agreement peaks around 10³ before settling to moderate levels, and self-consistency increases in two phases. Benchmarks where all models perform at chance (ARC Challenge, Logiqa, WinoGrande, WSC) show flat trajectories near random accuracy and near-zero inter-seed agreement and self-consistency, confirming that these tasks are simply too difficult for models of this scale and provide no signal about stability (Figure 6). This replication across tasks strengthens the claim that the identified learning phases are general properties of the training process rather than artifacts of specific benchmarks.

Gender bias benchmark triangulation reveals shared temporal structure but variable magnitude: The three gender bias benchmarks (Figure 2) show distinct trajectories that nevertheless share a common onset: all three metrics begin changing around step 10²–10³. BLiMP (Gender Agreement) shows a steady accuracy increase (learning grammatical gender), Simple Co-occurrence Bias shows an abrupt jump in male preference, and CrowS-Pairs (Gender) shows a more gradual increase in stereotypical associations with much wider variance. The paper explicitly acknowledges the reliability concerns with CrowS-Pairs (Section 3): "We posit that the large variance observed for the bias measures reflects the poor reliability (e.g., due to small benchmark size, poor quality test items, etc) rather than actual bias differences," citing Van der Wal et al. (2024) and Delobelle et al. (2024). This is a measured interpretation: the paper does not claim that bias is stable or unstable, but rather notes that the measurement instruments themselves may be too noisy to distinguish genuine variance from measurement error.

Disentangling data order from parameter initialization (160M ablations): The PolyPythia release includes additional 160M training runs that vary only the data seed (keeping weight initialization fixed) or only the weight initialization seed (keeping data order fixed)—three runs of each type (Table 5). These ablation runs are publicly released but not analyzed in the main paper, representing a deliberate choice to leave the decomposition of variance sources to future research. The absence of this analysis from the current paper is a limitation: the paper's central claims about stability do not distinguish between "the training process is stable because data order doesn't matter" and "the training process is stable because parameter initialization doesn't matter," which are mechanistically different claims.

Choice of K = 5 HMM states is validated by BIC near-optimality: The paper uses 5 latent states for all model sizes, stating that "this value is near optimal for the Bayesian Information Criterion across all sizes" (Section 5). No BIC curves or alternative-K comparisons are shown, but the claim is that 5 states provide a reasonable balance between capturing the temporal structure (too few states would merge distinct phases) and avoiding overfitting (too many states would create spurious state distinctions that don't replicate). The paper does not report whether results (linearity of stable maps, forking in outliers, zero-shot transfer $R^2$ ) are sensitive to the choice of $K$ , which would be a useful robustness check.

Standardization of HMM input features across seeds is necessary for cross-size transfer: The paper notes (Section 5) that HMMs are sensitive to input scale, and that standardizing the 15 parameter statistics to have mean 0 and standard deviation 1 across seeds (within each model size) is essential for training maps to be comparable. Without standardization, statistics with larger numerical ranges (e.g., L2-norm, which grows with model size) would dominate the HMM's state assignments. The successful zero-shot transfer ( $R^2 > 0.96$ for 31M+ models) validates that this standardization preserves the relevant temporal structure—the relative patterns within each model size (e.g., "this checkpoint has unusually high singular value variance") are sufficient for the HMM to identify meaningful states, and these relative patterns are shared across scales.

Fisher z-transformation for averaging correlations: The paper uses the Fisher z-transformation ( $\bar{r} = \tanh(\frac{1}{|\mathcal{M}|^2} \sum \text{arctanh}(r_{i,j}))$ ) when averaging Pearson correlations across pairs of model sizes (Section 4). This is statistically necessary because Pearson's $r$ is bounded in [-1, 1] and not normally distributed, making simple arithmetic averaging biased. The Fisher transformation maps correlations to an unbounded scale where standard averaging is valid, and the inverse transformation returns the result to the correlation scale. This methodological detail matters for the reported $\bar{r} = 0.99$ , 0.98, and 0.94 values—without the transformation, the averages could be misleading, though given the extremely high raw correlations, the difference would likely be small.

Performance metric selection for outlier detection (only tasks above random): The paper's outlier detection heuristic (Section 3) uses only the four benchmarks where all model sizes achieve above-random final accuracy (ARC Easy, LAMBADA, Piqa, SciQ), excluding ARC Challenge, Logiqa, WSC, and WinoGrande where models perform at chance. This is methodologically sound: including chance-level benchmarks would add noise (since variance on these tasks reflects random variation around a chance baseline, not genuine differences in model capability) and could mask or dilute the signal from tasks where models actually differ. The standardization within each task before averaging ensures that each task contributes equally regardless of its absolute accuracy range.

Checkpoint subset for evaluation (every 10k steps after 3k): The paper uses a coarser evaluation subset (~26 checkpoints per run) for downstream and probing analyses, rather than all 154 checkpoints, to manage computational costs. The 10k-step spacing in later training (steps 3k to 143k) is justified by the observation that metrics change more slowly after step 10⁴—the temporal resolution is sufficient to capture the critical phase transitions, though finer-grained phenomena occurring between evaluated checkpoints (e.g., rapid transients) would be missed. The training maps analysis uses the full 154-checkpoint sequence, providing denser temporal coverage at the parameter level, so the most temporally sensitive analysis is not affected by this subsetting.

Negative result: ReST $^{EM}$ optimization degrades revision model performance (relevance check): This is not an analysis in the PolyPythia paper—it is a finding from a different paper entirely (the reference example discussing revision models and ReST $^{EM}$ ). The PolyPythia paper does not involve revision models, reinforcement learning, or iterative fine-tuning; it is purely an analysis of standard pre-training stability. This item should not appear in this paper's ablation list.

Negative result: early checkpoints insufficient for performance prediction (training maps): As discussed in Section 5, bag-of-states constructed from partial training runs (fewer than 120k of the 143k steps) fail to predict final z-scores accurately. This is an important negative result because it constrains the practical utility of training maps for early stopping: while the full training map contains strong diagnostic signal, this signal is concentrated in the later stages of training (specifically, whether the 2 → 3 transition occurs correctly). The paper does not explore whether alternative representations (e.g., the full state sequence rather than bag-of-states, or features derived from early-checkpoint statistics directly rather than via HMM) could achieve earlier prediction.

Critical Assessment

Claim from executive summary: Language model pre-training is largely stable and follows predictable scaling laws with respect to downstream performance. This claim is well-supported by the downstream evaluation results (Figure 1, Figure 6) with an important nuance: stability is high for aggregate accuracy (narrow interquartile ranges across seeds, only 2 outliers across 50 runs), but moderate for which specific examples each model gets right or wrong (inter-seed agreement κ ≈ 0.5, Figure 1, middle column). The paper defines stability as "the change in a metric of interest caused by changes in randomness factors, quantified using standard deviation" (Section 1), and by this definition, downstream accuracy is indeed stable—the standard deviation of final accuracy across seeds (visible as the IQR width in Figure 1) is small relative to the differences between model sizes and the improvement over training. However, the moderate κ values reveal that this aggregate stability masks substantial example-level variability: two seeds with the same 50% accuracy on ARC (Easy) might get different subsets of questions correct, meaning that using a single seed as a "representative" model for that architecture and training recipe would be misleading if the goal is to understand what the model knows, not just how much it knows. This is a genuine limitation that the paper acknowledges implicitly through its focus on inter-seed agreement as a key metric, but does not fully discuss: aggregate accuracy stability does not imply example-level prediction stability.

The scaling laws aspect of the claim—that stability follows predictable patterns across model sizes—is supported at the representational level (correlations r > 0.94 between metric trajectories across sizes; Section 4) and at the parameter level (zero-shot training map transfer with R² > 0.96 for models 31M+; Table 3). However, the downstream performance scaling is only weakly tested: the paper shows that larger models have higher accuracy (consistent with known scaling laws), but does not systematically test whether the variance across seeds scales predictably with model size. The outlier seeds only appear at 410M, which could indicate that larger models are more prone to instability, but with only one size showing outliers and only 10 seeds per size, this is anecdotal. A proper scaling law for stability would require many more model sizes and seeds to estimate the functional relationship between parameter count and performance variance.

Claim: Two distinct learning phases are identified—an initial phase (10³–10⁴ steps) and a critical phase (10⁴–10⁵ steps). This claim is strongly supported by triangulation across three independent analysis levels. The downstream accuracy curves (Figure 1, left) show the bulk of improvement between 10³ and 10⁵; the probing metrics (Figure 3) show macro-F1, codelength ratio, and SSA all undergoing maximum change in this window; and the HMM state transitions (Table 2) consistently fall within this range (0 → 1 at 2k–7k, 1 → 2 at 18k–42k, 2 → 3 at 58k–73k). The identification of these phases is robust to both the random seed (consistent across all 10 seeds at all sizes) and the measurement instrument (the phases appear across accuracy, probing, and training map analyses). The paper also provides a plausible mechanistic interpretation of what occurs in each phase—initial phase: encoding of basic linguistic information, emergence of grammatical capabilities, formation of induction heads; critical phase: refinement of representations, stabilization of predictions, performance improvements—by connecting its findings to prior work on induction heads (Olsson et al., 2022), circuit formation (Tigges et al., 2024), and word acquisition (Chang and Bergen, 2022). This synthesis of evidence across levels and connections to prior work makes a compelling case that these phases are genuine, robust features of autoregressive LM pre-training.

However, the claim that these phases are "identified" by the PolyPythia analysis somewhat overstates the novelty. Prior work on phase transitions (Saphra and Lopez, 2019; Müller-Eberstein et al., 2023; Godey et al., 2024) already documented similar temporal structure in LMs. The PolyPythia contribution is not the discovery of phases per se, but the demonstration that they are invariant to randomness factors and model scale—something prior work could not establish with single-seed analyses. A fairer characterization would be that the paper validates and generalizes previously observed phase structure rather than discovering it de novo.

Claim: Training maps from smaller models can zero-shot predict the training trajectories and final performance of larger models with R² > 0.9 for all sizes except 14M. This claim is supported for the 31M, 70M, and 160M → 410M transfer (R² of 0.97, 0.97, and 0.96 respectively; Table 3), with the 14M failure (R² = 0.18) explicitly noted as a lower bound. The claim is genuine—these are zero-shot predictions where the HMM is trained only on small-model data and applied without adaptation to 410M. However, three limitations temper the practical significance:

First, the "prediction" is retrospective: it uses the full 143k-step training trajectory of 410M to construct its bag-of-states, and the paper's own negative result shows that partial trajectories (fewer than 120k steps) do not yield accurate predictions. This means the method cannot currently be used for early stopping or mid-training intervention—it identifies unstable runs only after they have completed (or nearly completed) training, at which point the compute has already been spent.

Second, the only model size with substantial performance variance is 410M (σ² = 0.98; Table 2). The smaller models (14M–160M) have σ² ranging 0.58–0.84, and their own regression R² values are low to moderate (0.03–0.75). The impressive R² values for predicting 410M from smaller models are, in a sense, predicting variance that only exists at the largest scale—it is not demonstrated that the method can predict performance differences at the smaller scales themselves. The paper does not perform the reverse experiment (predicting 31M performance from a 410M HMM) or the cross-prediction among smaller sizes, which would strengthen the claim of bidirectional transferability.

Third, the paper does not compare the HMM bag-of-states predictor to simpler baselines. Could one predict 410M performance just as well from, say, the final validation loss, or the average gradient norm, or the L2-norm of weights at some intermediate checkpoint? The claim that training maps specifically are informative would be stronger if the paper demonstrated that the HMM states capture information beyond what simpler, computationally cheaper statistics provide. Without such a comparison, $R^2 = 0.99$ may simply reflect that the bag-of-states is correlated with some underlying property (e.g., "the training run was generally healthy") that many different metrics would also capture.

Claim: Only two outlier training runs were identified (410M seeds 3 and 4), which exhibit loss spikes and premature exits from key state transitions. This claim is supported by converging evidence from multiple methods: these seeds are the only ones flagged by the 2-standard-deviation heuristic on downstream accuracy (Section 3); they are the only ones with visible loss spikes (Figure 5); and they are the only ones with forked training maps missing the 2 → 3 transition (Figure 4). The triangulation is convincing. However, the claim that there are "only two" outliers is contingent on the specific outlier detection method: a 2-standard-deviation threshold with standardization across only 4 benchmarks and 10 seeds. With a more sensitive test (e.g., 1.5 standard deviations) or a different set of benchmarks, additional seeds might be flagged. The paper's approach is conservative (high threshold, robust median/IQR reporting), which is appropriate for avoiding false positives, but it means the reported number of outliers is a lower bound—there may be additional runs with mild instability that do not cross the 2-standard-deviation threshold but still exhibit suboptimal behavior.

The paper's characterization of why these seeds fail—the missed 2 → 3 transition and the link to singular value degeneration (Godey et al., 2024)—is suggestive but correlational. The training map analysis shows that outlier seeds follow a different state sequence, but it does not establish causation: did the failure to make the 2 → 3 transition cause the performance degradation, or did some underlying pathology (e.g., a bad data shuffle that introduced harmful correlations) cause both the missed transition and the performance drop? The connection to "representation degeneration" provides a plausible mechanism (singular value collapse → reduced model capacity → degraded performance), but the evidence is observational, not interventional.

Genuine weaknesses and missing experiments:

No disentanglement of data order vs. parameter initialization variance. The paper acknowledges that 160M ablation runs varying only data seed or only weight seed are released but not analyzed (Table 5). Without this decomposition, the paper's findings about "stability to randomness factors" conflate two potentially very different sources of variance. It is possible that downstream accuracy is stable because data order doesn't matter much, but parameter initialization does (or vice versa), and the paper cannot distinguish these cases.
Single dataset (the Pile), single architecture (Pythia), single training recipe. All findings are conditional on the specific data distribution, model architecture, and hyperparameters used. The Pile is a diverse curated dataset, and the Pythia architecture uses standard transformer design—these are reasonable choices for a general-purpose study—but the paper cannot claim that the identified learning phases or stability properties generalize to other datasets (e.g., code-heavy corpora, multilingual data) or architectures (e.g., architectures with sparse mixture-of-experts layers, different positional encodings, or different normalization schemes). A comparison with MultiBERTs (encoder-only, different data) or the GPT-2 multi-seed suite (Karamcheti et al., 2021; different data, slightly different architecture) would help establish which findings are architecture/dataset-specific and which are universal, but such cross-suite comparisons are absent.
Model scale limited to 410M parameters. The paper is transparent about this limitation (Section 2), but it constrains the generalizability of several key findings. The observation that training maps from small models can predict large-model performance might not extend to the 1B–10B+ parameter range, where qualitatively new phenomena (e.g., emergent abilities, grokking, phase transitions not seen at smaller scales) could change the training dynamics. The finding that only the 410M model has outlier seeds could mean that instability increases with scale—a trend that would be critically important for practitioners training billion-parameter models, but which the paper can only hint at, not establish.
No intervention experiments. The paper identifies signatures of instability (forked training maps, missed state transitions, singular value degeneration) but never tests whether intervening to prevent these signatures—for example, by adjusting the learning rate schedule when a model enters the abnormal 2 → 4 state, or by penalizing high singular value variance—would rescue an unstable run. Without intervention experiments, the causal status of the identified patterns remains uncertain. This is not a flaw per se (the paper is a resource and analysis contribution, not a methods paper), but it leaves open the question of whether training maps are merely diagnostic or can be prescriptive.
Small number of seeds for statistical power. With 10 seeds per size, estimates of variance have limited precision. The standard error of the standard deviation is approximately σ/√(2×9) ≈ 0.24σ, meaning a 95% confidence interval for the true standard deviation spans roughly ±0.5σ. Outlier detection via a 2-standard-deviation threshold with only 10 samples is noisy—a seed that is genuinely from the same distribution could appear as an outlier by chance roughly 5% of the time. More seeds (20–30) would substantially improve the reliability of variance estimates and outlier detection, but the computational cost is substantial. The paper's approach of using robust statistics (median, IQR) mitigates this somewhat, but the 10-seed sample size remains a limitation for any quantitative claim about variance magnitudes.

What would strengthen the paper:

Analysis of the 160M data-only and weight-only ablation runs to decompose variance sources.
A comparison of HMM-based performance prediction against simpler baselines (e.g., predicting z-scores from final validation loss, from the mean or variance of parameter statistics at a single intermediate checkpoint, or from a linear trend fit to early-checkpoint loss).
A formal statistical test for outliers (e.g., Grubbs' test or a robust z-score with correction for multiple comparisons) rather than the heuristic 2-standard-deviation threshold.
Experiments testing whether the learning phases identified (10³–10⁴, 10⁴–10⁵) shift systematically with model size, and whether the phase boundaries can be predicted from simpler quantities (e.g., tokens seen relative to model parameter count).
Reverse cross-size prediction (large → small) to test symmetry of the training map transfer finding.
A demonstration that training maps can identify instability in a held-out training run not used during HMM fitting, establishing out-of-distribution detection rather than in-distribution characterization.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Not Accounted for in the Reported Stability Analysis

The assumption or constraint. The PolyPythia analysis framework treats stability as measurable through variance across 10 independent training runs per model size. However, the paper's outlier detection heuristic (Section 3) and training map analysis (Section 5) rely on access to the full 143k-step training trajectories for all 10 seeds—including the outliers themselves. The computational cost of training these runs to detect instability is enormous: 10 seeds × 5 sizes × 143k steps × ~2M tokens/step ≈ 15 trillion tokens processed for the 50 training runs. The paper does not provide a method for detecting instability early in training or from a smaller number of seeds; indeed, Section 5 explicitly states that "at least 120k steps (out of the total 143k) are required for a reliable prediction" of final performance from partial training maps.

The consequence. If a practitioner wanted to use the PolyPythia methodology to determine whether a specific training run is stable or unstable, they would need to either (a) train multiple complete runs and retrospective identify outliers, defeating the purpose of early detection, or (b) complete at least 84% of training (120k/143k steps) before the training map signal becomes reliable, by which point most of the compute has already been expended. The paper's central diagnostic instrument—training maps—currently functions as a post-hoc characterization tool, not a deployment-time early warning system. The $4×$ efficiency gains reported in the reference example are not relevant here (that was a different paper), but the analogous failure mode applies: the headline finding that training maps can identify outliers with $R^2 = 0.99$ is based on analyzing runs that have already completed, providing no mechanism to prevent the compute waste associated with unstable training.

What evidence exists in the paper. The negative result is reported explicitly in Section 5:

"when we construct the bag-of-states using only a partial training run, we fail to predict the average z-score accurately. Empirically, we find that at least 120k steps (out of the total 143k) are required for a reliable prediction."

The paper also notes (Section 5) that the outlier seeds' training maps deviate from stable maps only at the 2 → 3 transition, which occurs at steps 58k–73k for the 410M model (Table 2)—well past the halfway point of training. The loss spikes visible in Figure 5 (Appendix A) appear to onset around step 40k–60k for seed 4, confirming that instability manifests relatively late in the training process for these models.

Mitigation status. The paper does not attempt to solve this problem—it frames improving early prediction as future work (Section 5). The release of the full checkpoints at all intermediate steps is, in a sense, the mitigation: by providing the raw material, the paper enables other researchers to develop early-detection methods that may require fewer training steps or alternative statistical features not captured by the current HMM framework. However, as of this paper, there is no demonstrated method for early termination of unstable runs based on PolyPythia-style training maps.

Limitation 2: Single Dataset and Single Architecture—No Evidence of Cross-Domain or Cross-Architecture Generalization

The assumption or constraint. All 50 training runs use the Pile dataset (Gao et al., 2021) and the Pythia decoder-only transformer architecture (Biderman et al., 2023b) with the identical GPT-NeoX codebase, hyperparameters, and training recipe. The paper's claims about learning phases, stability properties, and outlier characteristics are therefore conditional on this specific combination of data distribution, model architecture, and optimizer configuration. No experiments vary the training data (e.g., a different corpus with different domain composition, language, or deduplication status), the architecture (e.g., encoder-decoder vs. decoder-only, or alternative attention mechanisms), or key hyperparameters (e.g., learning rate schedule, batch size, or sequence length).

The consequence. Several of the paper's headline findings may not transfer to other settings. The identified learning phases at steps 10³–10⁴ and 10⁴–10⁵ are stated in terms of absolute step counts under the specific batch size of ~2M tokens per step—these correspond to roughly 2B–20B tokens and 20B–200B tokens, respectively. The paper notes (Section 6) that these phases align with prior observations: induction heads emerge around 2.5B tokens (Olsson et al., 2022), circuits form between 2B–10B tokens (Tigges et al., 2024), and LSTMs exhibit similar phase structure (Saphra and Lopez, 2019). This concordance suggests some universality, but the critical caveat is that the token counts at which phases occur may depend on data quality, data diversity, vocabulary size, and training objectives in ways the paper cannot assess from a single-dataset, single-architecture study. A medical-domain LM trained on PubMed might exhibit entirely different phase timing; a multilingual model trained on imbalanced language distributions might show seed-dependent phase structure that the English-only Pile masks.

Furthermore, the finding that "stability is largely consistent across seeds" (Section 6) may be a property of the specific data distribution. The Pile is a curated, high-quality, diverse dataset. A noisier dataset with more toxic content, more template-generated text, or more extreme domain imbalance might produce greater seed-to-seed variance, as certain initializations or data orders could amplify pathological learning dynamics that the clean Pile distribution suppresses. The paper cannot speak to this.

What evidence exists in the paper. The paper directly acknowledges the architecture constraint (Section 2):

"A limitation of our suite is that it spans model sizes up to 410M parameters. This choice reflects computational constraints, prioritising seed exploration and checkpoint granularity over scaling up model size."

However, it does not similarly acknowledge the single-dataset and single-architecture limitations. The dataset choice is justified by consistency with prior Pythia-based research, not by any claim that the Pile is representative of all training corpora. The paper does not compare its findings to any multi-seed study on a different dataset (e.g., the GPT-2 multi-seed runs trained on WebText; Karamcheti et al., 2021) to assess whether the stability properties are consistent across data distributions.

Mitigation status. The paper partially mitigates the architecture limitation by citing prior work that observed similar phase structure in different architectures: Saphra and Lopez (2019) for LSTMs and Müller-Eberstein et al. (2023) for encoder-only models (Section 6). This suggests—but does not prove—that the phase structure may be a general property of neural language model training rather than a decoder-only-transformer-specific phenomenon. However, this cross-architecture evidence is from single-seed studies and different datasets, so it cannot establish that stability (variance across seeds) is similarly architecture-invariant. The single-dataset limitation is unaddressed—the paper does not cite any multi-seed study on a non-Pile dataset that could provide comparative evidence.

Limitation 3: The Two Sources of Randomness (Initialization and Data Order) Are Conflated Throughout the Main Analysis

The assumption or constraint. In the main PolyPythia training runs, the "seed" controls both parameter initialization and the shuffling-then-packing order of documents. This is an intentional design choice reflecting realistic training conditions—in practice, both sources of randomness vary simultaneously—but it means that all variance estimates and outlier identifications conflate two mechanistically distinct sources of randomness. When the paper reports that a metric has a certain standard deviation across seeds, it cannot distinguish whether that variance arises from different initial parameter values, different training data orders, or an interaction between the two.

The consequence. The claim that "language modelling is largely stable" (abstract, Section 6) is ambiguous about what it is stable to. Stability to parameter initialization (the model converges to similar representations and performance regardless of starting weights) and stability to data order (the model converges similarly regardless of the sequence in which it sees training examples) are mechanistically different properties with different implications. If downstream accuracy is stable because it is insensitive to both factors, that is a strong robustness guarantee. If it is stable because parameter initialization matters but data order doesn't (or vice versa), that provides a weaker guarantee—a different training recipe that is more sensitive to the dominant randomness source could break the observed stability. Similarly, identifying outlier seeds (410M seeds 3 and 4) without knowing whether the pathology was caused by unlucky initialization, unlucky data order, or a specific interaction between the two severely limits the practical utility of the finding. If the instability is driven by parameter initialization, mitigating it might require better initialization schemes or different optimizer configurations. If it is driven by data order, the solution might involve different shuffling strategies, curriculum learning, or data deduplication.

What evidence exists in the paper. The paper is transparent about this limitation (Section 2):

"Each training run uses the same hyperparameters, codebase, and data as Biderman et al. (2023b) but varies the seeds for parameter initialisation and batch composition."

And in a footnote:

"We will use the term 'batch composition' instead of 'data order' because the GPT-NeoX codebase... shuffles documents before packing them into sequences. This results in sequences that are not simply reshuffled across seeds; they are unique due to the different packing."

The paper also releases additional 160M training runs that isolate the two factors (Table 5): "data-seed1," "data-seed2," "data-seed3" (varying only data order with fixed initialization) and "weight-seed1," "weight-seed2," "weight-seed3" (varying only initialization with fixed data order). However, these ablation runs are not analyzed anywhere in the paper—they are released as raw material for future research.

Mitigation status. The paper acknowledges the limitation by releasing the disentangled 160M runs but does not itself perform the variance decomposition. The main text analyses (Sections 3–5) treat "seed" as a monolithic randomness factor and report aggregate variance across seeds without source attribution. This is a significant gap: the paper's central resource contribution is designed to enable this analysis, but the paper does not demonstrate it. A variance components analysis (e.g., estimating the fraction of total variance attributable to initialization vs. data order vs. their interaction, perhaps via a random-effects model) would have substantially strengthened the interpretability of the stability findings and provided actionable guidance about which randomness source practitioners should worry about most.

Limitation 4: Scale Range Is Limited to 410M Parameters—No Evidence That Findings Extend to Practical LLM Scales (1B–100B+)

The assumption or constraint. The PolyPythia suite spans model sizes from 14M to 410M parameters. While this covers a 30× range in parameter count, it is an order of magnitude below the scale of models commonly deployed in practice (7B parameters and above) and two orders of magnitude below frontier models (70B–100B+). The paper's findings about learning phases, stability properties, and outlier characteristics are therefore established only for small-to-moderate-scale decoder-only transformers trained on a single epoch of a 300B-token dataset.

The consequence. Three findings are particularly vulnerable to scale-dependent effects that this paper cannot assess. First, the claim that training dynamics are "largely stable" may not hold at larger scales where loss landscapes are sharper, where small initialization differences can be amplified by the longer training trajectories, or where data order effects compound differently due to the model's greater capacity to memorize and fit to specific example sequences. Second, the finding that outlier seeds appear only at 410M (Section 3) hints at a trend—instability may increase with scale—but with only a single size showing outliers (and only 2 out of 10 seeds at that size), this is anecdotal. At 7B or 70B parameters, the outlier rate could be substantially higher, or the nature of instability could change qualitatively (e.g., loss spikes becoming more frequent, or new failure modes like training collapse emerging). Third, the zero-shot cross-model-size transfer result—that an HMM trained on a 31M model can predict 410M performance with $R^2 = 0.97$ (Table 3)—may not extend to the 410M → 7B or 31M → 70B regime, because the parameter dynamics of billion-parameter models could involve qualitatively new phases (e.g., emergent circuit formation, grokking, or loss-of-plasticity phenomena) not present in sub-billion-parameter models.

The paper's observation (Section 6) that learning phases occur at similar token counts across model sizes—and align with prior work on larger models (e.g., induction heads at 2.5B tokens in models up to several billion parameters; Olsson et al., 2022)—provides some suggestive evidence of scale invariance. However, Olsson et al. studied a range that includes models larger than PolyPythia's largest, and the PolyPythia paper's own findings already show that the smallest model (14M) is qualitatively different (it fails to transfer zero-shot, $R^2 = 0.18$ ; it exhibits severe saturation not seen in larger models). The fact that 14M is different from 31M–410M suggests that there is a lower bound on scale invariance. It would be surprising if there were no corresponding upper bound or gradual degradation in the transferability of training dynamics at scales beyond 410M.

What evidence exists in the paper. The paper explicitly identifies the scale limitation in Section 2:

"A limitation of our suite is that it spans model sizes up to 410M parameters. This choice reflects computational constraints, prioritising seed exploration and checkpoint granularity over scaling up model size. Our aim is to provide an additional resource for researchers unable to train even 410M parameter models from scratch, thus enabling them to study training stability across model sizes."

The paper also provides the only within-study evidence of scale-dependent qualitative change: the 14M model's failure to transfer zero-shot ( $R^2 = 0.18$ vs. $0.96$ – $0.97$ for larger models; Table 3) and its pronounced saturation (Figure 1, where 14M accuracy declines between 10⁴ and 10⁵ steps while larger models continue improving). The 410M model shows higher transition timing variance than smaller models (Table 2, standard deviation of 22,600 steps for the 0 → 1 transition at 410M vs. 0–700 at smaller sizes), even excluding outlier seeds—another hint that scale affects dynamics.

Mitigation status. The paper justifies its scale choice as a deliberate tradeoff (many seeds and dense checkpoints vs. larger models) and positions the suite as a resource for researchers who cannot train even 410M-parameter models, not as a definitive answer to stability questions at all scales. The cross-model-size correlation analyses (Section 4: $r > 0.94$ for representational metrics) and zero-shot training map transfer (Section 5: $R^2 > 0.96$ for models ≥31M) provide within-the-studied-range evidence of scale invariance, which is encouraging but does not constitute evidence about scales beyond 410M. The paper does not cite or compare against any multi-seed study at larger scales (e.g., the Llama-2-7B 10-seed suite from Madaan et al., 2024) to assess whether similar stability properties hold. Given that the Madaan et al. checkpoints are not publicly available, such a comparison would be difficult, but the conceptual gap remains: the PolyPythia findings are validated only up to 410M parameters, and the paper provides no principled argument for why they should extend to practical LLM scales.

Limitation 5: No Disentanglement of Benchmark Noise from Genuine Model Variance

The assumption or constraint. Throughout the downstream evaluation (Section 3) and representational probing (Section 4), the paper reports the median and interquartile range (or mean and standard deviation) of metrics across the 10 seeds for each model size and checkpoint. These summary statistics are interpreted as measures of how much "genuine" model behavior varies due to randomness factors. However, every metric is computed on a finite test set of fixed size—the downstream benchmarks range from a few hundred examples (WSC: 273; CrowS-Pairs Gender: a few hundred) to a few thousand (ARC Easy: ~2,400; SciQ: ~1,000; LAMBADA: ~5,000). The observed variance across seeds therefore conflates two sources of variation: (1) genuine differences in what the model has learned (the quantity of interest), and (2) binomial sampling noise from the finite test set (if two identically-performing models are evaluated on different random subsets of a test set, their accuracies will differ due to finite-sample effects, even if the models are identical).

The consequence. The paper's claim that "language modelling is largely stable" (Section 6) could be an artifact of small test sets masking meaningful variance. If the test sets are too small to reliably detect differences between seeds—i.e., if the confidence intervals around per-seed accuracy estimates are wide—then the observed narrow interquartile ranges may reflect insufficient measurement precision rather than genuine stability. This is a well-known issue in benchmark evaluation: with a test set of size $n$ , the standard error of an accuracy estimate is $\sqrt{p(1-p)/n}$ , where $p$ is the true accuracy. For a benchmark with 1,000 test examples and an accuracy of 0.50, the standard error is approximately 0.016, meaning a 95% confidence interval spans roughly ±3.2 percentage points. If the true across-seed standard deviation of model accuracy is, say, 2 percentage points, it would be largely buried within the measurement noise of a 1,000-example test set—the paper's IQR would appear narrow not because seeds are genuinely similar, but because the test set lacks the statistical power to resolve their differences.

The paper itself provides evidence consistent with this concern in its gender bias analysis (Section 3):

"We posit that the large variance observed for the bias measures reflects the poor reliability (e.g., due to small benchmark size, poor quality test items, etc) rather than actual bias differences."

This is an acknowledgment that for small benchmarks like CrowS-Pairs Gender, the observed variance is dominated by measurement noise. But the paper does not apply this same skepticism to its other benchmarks. The downstream accuracy metrics are computed on larger test sets (thousands of examples), so the noise is smaller, but it is not zero, and the paper never estimates what fraction of the observed across-seed variance is attributable to finite-test-set sampling vs. genuine model differences.

What evidence exists in the paper. The paper provides the benchmark sizes implicitly through citations (ARC Easy has ~2,400 test examples; SciQ has ~1,000; LAMBADA has ~5,000; Piqa has ~3,000; Logiqa has ~650; WSC has 273; WinoGrande has ~1,800) but does not report these explicitly in the main text or compute confidence intervals for per-seed accuracy estimates. The representational probing metrics (macro-F1, codelength ratio) are similarly computed on finite test sets for each linguistic task (OntoNotes test splits for PoS, Dep, NER, Coref; the standard test splits for Senti, Topic, SemTag), but test set sizes are not reported. The training maps analysis (Section 5) operates on parameter statistics computed directly from model weights, so it is not subject to finite-test-set noise—this is the one analysis level where the paper's variance estimates unambiguously reflect genuine model differences rather than measurement noise.

Mitigation status. The paper partially mitigates this concern through its multi-metric triangulation: the finding that learning phases are consistent across downstream accuracy, probing metrics, and training maps (Section 6) means that the observed temporal structure cannot be solely an artifact of benchmark noise, since the training maps are computed from model parameters directly (no test set involved). The identification of outlier seeds is also triangulated across performance (Section 3), training loss (Figure 5), and training maps (Figure 4), reducing the risk that outliers are false positives from noisy benchmarks. However, the magnitude of reported stability (e.g., the narrowness of IQRs in Figure 1) remains uncalibrated—the paper does not report what fraction of the observed IQR width would be expected from finite-test-set sampling alone if all seeds had identical true accuracy. Without this calibration, the claim of "largely stable" is qualitative rather than quantitative—the paper cannot say whether the stability it observes is tighter or looser than what measurement noise alone would produce.

Limitation 6: No Demonstration That Training Maps Enable Intervention or Prevention

The assumption or constraint. The training map analysis (Section 5) demonstrates that the HMM-based latent state framework can retrospectively characterize stable vs. unstable training trajectories and predict final performance with high $R^2$ (0.99 for 410M). However, the paper stops at diagnosis: it identifies that outlier seeds follow forked training maps with a missing 2 → 3 transition, but it does not test whether intervening during training—for example, adjusting the learning rate, modifying the optimizer, or restarting from an earlier checkpoint when a forked transition is detected—can rescue an unstable run or prevent the performance degradation.

The consequence. The training maps framework, as presented, is a retrospective analytical tool rather than a prescriptive engineering tool. A practitioner training a 410M-parameter model who observes (at step 80k) that the model has not made the expected 2 → 3 transition gains no actionable guidance from this paper about what to do next. Should they continue training and hope the model recovers? Restart with a different seed? Modify the learning rate schedule? Switch to a different optimizer? The paper provides no evidence about which (if any) of these interventions would be effective, because it never intervenes.

This limitation is particularly consequential given the paper's own characterization of the outlier runs. The finding that the 2 → 3 transition is driven by decreases in L2-norm, median bias, and weight variance (Table 1)—effectively, a parameter regularization phase—and that the abnormal 2 → 4 transition is associated with increases in singular value variance and spectral norm, suggests a plausible intervention: if a model enters state 4 instead of state 3, increasing weight decay or applying explicit spectral regularization might nudge it back onto a stable trajectory. However, the paper does not test this or any similar hypothesis.

What evidence exists in the paper. The paper identifies the mechanistic drivers of state transitions in Table 1, noting that the stable 2 → 3 transition is characterized by "median_b ↓0.44, L2 ↓0.42, σ_w ↓0.50" (all decreases, i.e., regularization), while the abnormal 2 → 4 transition in outliers is characterized by "λ_max ↑1.41, σ_b ↑1.79, σ_λ ↑1.71" (all increases, i.e., growing anisotropy and variance). The paper also connects the regressive 4 → 0 and 4 → 1 transitions to "representation degeneration" (Godey et al., 2024), providing a plausible mechanistic narrative for what goes wrong. The zero-shot cross-model-size transfer result (Table 3) demonstrates that the HMM state structure is robust enough to be detected by models trained on different scales, confirming that the state definitions capture real, generalizable properties of training dynamics, not artifacts of the 410M model specifically.

Mitigation status. The paper does not attempt any intervention experiments. It frames the training maps analysis as diagnostic (Section 5: "we find that it is possible to predict training maps in a 'zero-shot' fashion") and suggests that "interventional studies of training dynamics may be necessary" (Section 6) for future work, but does not perform them. The release of the full checkpoints at all intermediate steps enables other researchers to simulate retrospective interventions (e.g., "what if we had increased weight decay starting at step 60k for seed 3?"), but the paper itself provides no evidence that such interventions would be effective. This leaves a substantial gap between the diagnostic capability the paper demonstrates and the preventive capability a practitioner would need to justify the computational cost of running the training map analysis on a live training run.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the study of language model training dynamics from a single-run inferential model—where researchers train one model, observe its behavior, and draw conclusions about the underlying learning process—to a population-level statistical framework where training runs are treated as samples from a distribution, and only findings that replicate across seeds and scales are considered reliable. This is not a paradigm shift in the Kuhnian sense—the core methods (benchmarking, probing, HMM-based dynamics analysis) all existed before PolyPythia—but it is a methodological reframing whose practical consequences are substantial. Before PolyPythia, a researcher who wanted to claim that "induction heads emerge at step X" or "saturation occurs in small models" had no way to distinguish between a robust property of the learning algorithm and an artifact of a specific random seed. After PolyPythia, such claims can be tested against 50 training trajectories spanning a 30× range of model sizes, and the paper's own analyses demonstrate that some widely-held assumptions (e.g., that training dynamics are seed-sensitive) are less supported than the field assumed, while others (e.g., that learning occurs in discrete phases) are more robust than single-run studies could establish.

The most consequential empirical finding is that language model pre-training at this scale is substantially more stable than the field's prior literature would suggest. The paper identifies only 2 outlier runs out of 50 (Section 3), finds that downstream accuracy variance across seeds is small relative to the differences between model sizes (Figure 1), and shows that representational learning trajectories are nearly identical across seeds (the shaded standard-deviation regions in Figure 3 are narrow). This directly challenges a narrative—fed by work on fine-tuning instability (Dodge et al., 2020; Mosbach et al., 2021), benchmark sensitivity (Alzahrani et al., 2024), and underspecification (D'Amour et al., 2022)—that neural network training is inherently high-variance and that single-run conclusions are unreliable. The paper's data suggests a more nuanced picture: at the scale studied (14M–410M parameters, one epoch on a curated 300B-token dataset), the pre-training process is largely deterministic in its aggregate outcomes, even though the specific examples each model gets right or wrong vary substantially (inter-seed κ ≈ 0.5; Figure 1, middle column). This reconciles the apparent contradiction between "BERT fine-tuning is unstable" (Dodge et al., 2020) and "Pythia pre-training is stable" (this paper) by suggesting that pre-training and fine-tuning have fundamentally different stability properties—pre-training, with its massive data volume and long optimization trajectory, averages out the initialization and data-order effects that dominate fine-tuning on small datasets.

The paper also validates and generalizes the phase-transition model of LM training that has been accumulating in the literature. Saphra and Lopez (2019) identified phases in LSTMs, Müller-Eberstein et al. (2023) identified them in encoder-only models, and Michaelov and Bergen (2023) and Godey et al. (2024) documented saturation in Pythia models. But each of these studies was single-run, leaving open the possibility that the observed phases were contingent on initialization or data order. The PolyPythia analysis shows—via triangulation across downstream accuracy, probing metrics, and training maps (Section 6)—that the two-phase structure (initial learning at 10³–10⁴ steps, critical learning at 10⁴–10⁵ steps) is reproducible across all 10 seeds at all 5 model sizes. This converts phase transitions from an interesting phenomenon to a structural fact about autoregressive LM pre-training on natural language data that future research can treat as a reliable scaffold: if a new capability emerges at step 5k in one seed, it will emerge at roughly the same step in all seeds. The paper's connection of these phases to prior mechanistic findings (induction heads at ~2.5B tokens from Olsson et al., 2022; circuit formation at 2B–10B tokens from Tigges et al., 2024) provides a coherent timeline linking phase boundaries to specific circuit-level phenomena.

The methodological shift the paper enables is toward scale-down-then-transfer research. The demonstration that representational stability trajectories are correlated across model sizes with r > 0.94 (Section 4) and that training maps from 31M models can predict 410M final performance with R² = 0.97 (Table 3) means that researchers can prototype and debug training dynamics at small, cheap scales (70M parameters, trainable on a single GPU in hours) and transfer insights to larger models (410M parameters) with reasonable confidence. This dramatically lowers the compute barrier for training dynamics research. The failure of the 14M model to transfer (R² = 0.18) also provides a critical boundary condition: below roughly 30M parameters, the dynamics become qualitatively different enough that small-scale prototyping is unreliable. This is itself a useful finding—it tells the field where the "minimum viable model size" lies for studying dynamics that generalize to larger scales.

The paper also introduces training maps as a diagnostic instrument with demonstrated cross-scale validity, moving them from a visualization tool (Hu et al., 2023) toward a candidate for production monitoring. The finding that the bag-of-states representation achieves R² = 0.99 for predicting 410M performance (Table 2) and that the outlier seeds are consistently identified with large negative coefficients regardless of which model size's HMM is used (Table 3) demonstrates that training maps capture a robust "signature of instability" that generalizes across scales. However, the critical limitation—that at least 120k out of 143k steps are needed for reliable prediction (Section 5)—means this remains a retrospective diagnostic, not a real-time early warning system. The landscape shift is therefore toward using training maps for post-hoc analysis of training failures (to understand what went wrong after the fact) rather than for preventing them (intervening mid-training), at least until early-prediction methods improve.

Follow-Up Research This Work Enables

Developing early-warning instability detectors from partial training trajectories. The paper's central negative result—that the current bag-of-states approach requires 120k+ steps to predict final performance (Section 5)—opens a clear research question: can alternative features or models detect the signature of instability earlier in training? A strong follow-up would extract the same 15 parameter statistics at each checkpoint (available at every step in the PolyPythia release), then train a sequence model (e.g., an LSTM or transformer) to predict final z-score from the first t checkpoints for t = 10k, 20k, . . . , 120k, using leave-one-seed-out cross-validation across the 10 seeds per model size. The key metric would be the minimum t at which prediction R² exceeds, say, 0.8—if this t is substantially smaller than 120k (e.g., 40k–60k), it would demonstrate that the temporal sequence of parameter statistics (not just the aggregate bag-of-states) contains earlier instability signals that the HMM's discrete state assignments miss. The 410M outlier seeds 3 and 4 are the natural test cases: can a model trained on seeds 0–2, 5–9 detect that seeds 3 and 4 are diverging by step 60k, when the 2 → 3 transition should be occurring but isn't? The paper's release of all intermediate checkpoints makes this feasible without retraining models.

Disentangling initialization variance from data-order variance using the 160M ablation runs. The PolyPythia release includes six additional 160M training runs that isolate the two sources of randomness: three varying only the data seed (with fixed weight initialization) and three varying only the weight seed (with fixed data order) (Table 5). The paper explicitly does not analyze these runs, leaving a clear next step: for each downstream benchmark, probing task, and parameter statistic, compute the variance attributable to initialization vs. data order using a random-effects model (or, given the small sample size, a simple variance-components decomposition via ANOVA). This would answer a question the paper raises but cannot address: is the observed stability driven primarily by robustness to initialization (consistent with the "lottery ticket" hypothesis that good subnetworks exist at initialization and training merely selects them), or by robustness to data order (consistent with the "data determines everything" hypothesis that any reasonable initialization converges to the same solution given enough diverse data)? A finding that one source dominates would guide practitioners toward mitigating the high-variance source (e.g., if initialization dominates, invest in better initialization schemes; if data order dominates, invest in curriculum learning or adaptive batching). A finding that both contribute comparably would motivate joint optimization.

Testing whether training map state transitions correspond to circuit-level milestones. The paper notes (Section 6) that the initial learning phase (10³–10⁴ steps) aligns with prior findings about induction head formation (Olsson et al., 2022), circuit emergence (Tigges et al., 2024), and syntactic capability acquisition (Chang and Bergen, 2022; Jumelet et al., 2024). A natural follow-up would apply circuit-analysis techniques (activation patching, attention pattern analysis, logit attribution) to multiple seeds of the same model size at checkpoints just before and just after each HMM state transition. For example, does the 1 → 2 transition (steps 18k–42k at 160M; Table 2) coincide with the first appearance of induction heads across all seeds, or is the timing seed-dependent even though the broader phase structure is consistent? If circuit formation timing varies across seeds but the HMM transitions are locked to step count, that would suggest the transitions are driven by the optimizer schedule rather than by emergent computational capabilities—a finding that would revise the interpretation of what training maps actually measure. The dense PolyPythia checkpoints around the key transitions (log-spaced early, then every 1k steps) provide sufficient temporal resolution for this analysis.

Replicating the stability findings on a different dataset and architecture. The paper's findings are conditional on the Pile and the Pythia architecture (see Limitation 2). A direct stress-test would train 5–10 seeds of a comparable-size model (e.g., 160M parameters) on a substantially different dataset—say, a code-heavy corpus like The Stack, a multilingual corpus like mC4, or a domain-specific corpus like PubMed—using the same architecture and hyperparameters, and evaluate whether the two-phase learning structure, the narrow across-seed variance, and the linear training maps replicate. A negative result (e.g., code training shows different phase timing, multilingual training shows higher seed variance due to language imbalance, domain-specific training shows earlier saturation) would establish boundary conditions on the paper's claims and provide a richer picture of when stability holds. A positive result (stability across diverse corpora) would substantially strengthen the claim of universality. Additionally, testing on an encoder-decoder architecture (e.g., T5) or a non-transformer architecture (e.g., a state-space model like Mamba) would probe whether the phase structure is transformer-specific. The paper's release of the pre-shuffled datasets and training code makes it feasible for other researchers to swap the data pipeline while keeping the training infrastructure identical.

Developing methods for post-hoc "rescue" of unstable checkpoints. The paper identifies that outlier 410M seeds 3 and 4 fail to make the 2 → 3 transition, which is characterized by decreases in L2-norm, median bias, and weight variance (Table 1). A direct intervention experiment would take a checkpoint from seed 3 or 4 just before the divergence (e.g., step 50k, before the abnormal 2 → 4 transition) and continue training with modified hyperparameters designed to encourage the missing regularization: increased weight decay, gradient clipping, or explicit spectral normalization to penalize the λ_max and σ_λ increases observed in the abnormal transition (Table 1; λ_max ↑1.41, σ_λ ↑1.71). The outcome metric would be whether the modified training trajectory recovers to match the performance of stable seeds by step 143k, and whether the intervention eliminates the forked training map. Even a partial recovery (e.g., reducing but not eliminating the performance gap) would demonstrate that training maps are not merely diagnostic but prescriptive—they identify actionable pathologies. A complete failure (no intervention helps) would suggest the pathology is set earlier in training, perhaps during the initial learning phase, and that mid-training intervention comes too late.

Investigating whether the 14M model's failure to transfer reflects a general "minimum scale" threshold or an artifact of the Pythia architecture. The paper finds that the 14M HMM fails catastrophically at zero-shot prediction of 410M performance (R² = 0.18 vs. ≥0.96 for 31M+ models; Table 3), and that the 14M model exhibits pronounced saturation not seen in larger models (Figure 1). This raises the question: is there a fundamental minimum scale (in parameters, or in parameters relative to data) below which training dynamics become qualitatively different? A systematic experiment would train multiple seeds at even smaller scales (e.g., 5M, 10M parameters) and at intermediate scales between 14M and 31M (e.g., 20M, 25M) to identify where the transition from "transferable dynamics" to "non-transferable dynamics" occurs. Concretely: fit an HMM at each size, compute the zero-shot R² for predicting 410M performance, and identify the size at which R² crosses, say, 0.8. If the transition is sharp (e.g., R² jumps from <0.2 to >0.9 between 20M and 25M), that suggests a phase transition in the learning dynamics themselves—a minimum representational capacity needed for the two-phase structure to emerge. If the transition is gradual (R² increases smoothly with model size), that suggests the failure is simply a function of how different the small model's parameter trajectories are from the large model's, with no sharp threshold. The PolyPythia release already provides the 14M–410M data; additional runs at 5M–20M would fill in the lower range.

Practical Applications and Downstream Use Cases

Training run quality assurance for medium-scale model development. Organizations that train custom language models in the 100M–1B parameter range—for domain-specific applications, on-premise deployment, or research prototyping—can use the PolyPythia training map methodology as a post-hoc quality audit without incurring additional training costs. By extracting the 15 parameter statistics from saved checkpoints and fitting an HMM (using the released code and methodology from Section 5), a team can compare their training map against the normative linear pattern: if the map shows forks or regressive transitions (like the 410M outlier seeds in Figure 4, top-left), the run was unstable and its final checkpoint should be treated with skepticism—even if validation loss looks acceptable, downstream performance may be degraded. The paper's finding that the failed 2 → 3 transition is characterized by missing regularization (Table 1) gives a concrete signal to look for: if L2-norm, weight variance, and median bias do not decrease at the expected step (around 60k–70k for a 143k-step run; Table 2), the run may be diverging. This is actionable today for any team that saves intermediate checkpoints—the analysis requires only parameter statistics, not retraining or additional evaluation.

Calibrating confidence in single-run research findings. A researcher who has drawn conclusions from a single Pythia training run (e.g., "capability X emerges at step Y" or "metric Z correlates with model size") can use the PolyPythia release to retrospectively bound the seed-dependence of their claims. Specifically, for any finding established on seed 0 of a given model size, the researcher can check whether the same pattern holds across seeds 1–9 at the same size. If it does (narrow IQR, consistent timing), the finding is robust to randomness factors and can be reported with greater confidence. If it fails (wide IQR, phase shifts across seeds), the finding should be qualified as possibly seed-specific. The paper's own analyses demonstrate this calibration in action: the claim that "saturation occurs in small models" (Section 3) is validated by showing it appears in all 10 seeds of the 14M and 31M models (Figure 1, left column, where the 14M accuracy drops between 10⁴ and 10⁵ across the full IQR), while a claim about the exact step at which saturation begins might be more seed-dependent. This is a low-cost use case—it requires only downloading the released checkpoints and running the same evaluation pipeline the researcher already has, but on multiple seeds instead of one.

Benchmarking the reliability of evaluation metrics themselves. The paper's observation that observed variance on bias benchmarks may reflect "poor reliability (e.g., due to small benchmark size, poor quality test items)" rather than genuine model differences (Section 3) suggests a concrete use of the PolyPythia suite: use the 10-seed distribution as a null model for expected variance if models are identical. For any benchmark where researcher A claims that model X is more biased (or more accurate, or more capable) than model Y, a skeptic can evaluate 10 seeds of model X and 10 seeds of model Y on that benchmark and compute whether the between-model difference exceeds the within-model, between-seed variance. If the between-model difference is smaller than the typical across-seed range for a single model, the claimed difference is not statistically meaningful—it is indistinguishable from seed noise. The paper's data already provides this baseline for the eight downstream benchmarks (Figures 1 and 6) and three gender bias metrics (Figure 2) on the Pythia architecture, and the methodology transfers to any benchmark evaluable with the LM Evaluation Harness. This is particularly valuable for small benchmarks (like CrowS-Pairs Gender, with only a few hundred examples) where measurement noise is known to be high, but where researchers nonetheless draw comparative conclusions—the PolyPythia IQRs provide an empirical noise floor that can be cited to qualify or challenge such claims.

When to Prefer This Method

The paper does not propose a method in the traditional sense—it provides a resource (multi-seed training runs) and an analysis framework (downstream evaluation across seeds, representational probing with MDL, training maps with HMMs) for studying pre-training stability. It does not position itself against named alternatives such as "use single-seed analysis instead" or "use the original Pythia suite instead" in a way that creates a clear decision rule. The implicit tradeoff—captured in the paper's explicit limitation discussion (Section 2)—is between studying dynamics at larger scales with single seeds (e.g., analyzing a 7B model trained once, as in the Llama-2-7B multi-seed work by Madaan et al., 2024) versus studying dynamics at modest scales with many seeds and dense checkpoints (PolyPythia). The paper argues for the latter when the research question concerns variance, phase structure, or outlier detection, since these questions cannot be answered with single runs regardless of scale. It also argues that for dynamics research, scale diversity (5 sizes) can substitute for absolute scale when the goal is to understand qualitative features of learning (the two-phase structure, the saturation phenomenon, the driver statistics of state transitions) rather than quantitative predictions about specific benchmark scores at 70B parameters. However, the paper does not formalize this tradeoff or provide explicit criteria for when 10 seeds at 410M is preferable to 1 seed at 7B, and I will not invent a decision matrix the paper does not support.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: We Don't Know How Much LLM Pre-Training Results Depend on Luck

Why This Problem Is Important

Where Existing Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Multi-Seed Training Procedure and Checkpointing

Downstream Evaluation Protocol and Agreement Metrics

Information-Theoretic Probing for Representational Stability

Training Maps: Hidden Markov Models of Parameter Dynamics

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Stability as a First-Class Object of Study Across the Full Pre-Training Pipeline

Innovation 2: Training Maps as a Diagnostic Tool That Unifies Parameter Dynamics, Performance, and Outlier Detection

Innovation 3: The Identification of Two Robust, Scale-Invariant Learning Phases

Innovation 4: The Discovery That Stability Patterns Are Scale-Invariant (With a Lower Bound), Enabling Small-Model Prototyping

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Downstream Performance Stability

Representational Stability of Linguistic Information

Training Phase Dynamics and Outlier Detection via Training Maps

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Not Accounted for in the Reported Stability Analysis

Limitation 2: Single Dataset and Single Architecture—No Evidence of Cross-Domain or Cross-Architecture Generalization

Limitation 3: The Two Sources of Randomness (Initialization and Data Order) Are Conflated Throughout the Main Analysis

Limitation 4: Scale Range Is Limited to 410M Parameters—No Evidence That Findings Extend to Practical LLM Scales (1B–100B+)

Limitation 5: No Disentanglement of Benchmark Noise from Genuine Model Variance

Limitation 6: No Demonstration That Training Maps Enable Intervention or Prevention

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention