Simple and Scalable Strategies to Continually Pre-train Large Language Models

ArXiv: 2403.08763

🎯 Pitch

Large language models can be updated on massive new datasets without full retraining: a simple recipe of learning rate re-warming, re-decay, and a small fraction of replay data matches the loss and downstream performance of training from scratch—even for a 10B parameter model under real-world distribution shift. This holds across hundreds of billions of tokens, yet fails when the tokenizer or data distribution is too far out-of-domain, revealing a sharp boundary for current continual pre-training.

---

1. Executive Summary

This paper empirically studies how to update large language models on newly available data without re-training from scratch—a process it terms continual pre-training—using decoder-only transformer models up to 10B parameters on large-scale datasets (hundreds of billions of tokens) under both a weak distribution shift (English Pile → English SlimPajama) and a stronger shift (English Pile → German Common Crawl). It demonstrates that a simple and scalable combination of three techniques—learning rate re-warming and re-decaying (re-increasing the learning rate from its small final value and cosine-decaying it again on the new dataset), compute-equivalent replay (interleaving a fraction of previous-dataset tokens into the new training mix while reducing new tokens to hold total compute constant), and infinite learning rate schedules (schedules that maintain a constant learning rate across dataset boundaries to avoid re-warming entirely)—enables continually pre-trained models to match the final validation loss and average downstream benchmark performance of the expensive re-training-from-scratch baseline on the union of all data, achieving this parity while using substantially less compute (e.g., matching the 600B-token joint-training loss while seeing only 300B new tokens with 5% replay, and matching the 500B-token joint-training loss with 25% replay under the stronger shift), establishing that full re-training is unnecessary for updating LLMs when the incoming data distribution remains within the model's existing capability range—but with the finding that on the hardest distribution shifts, tokenizer limitations and fundamentally out-of-distribution data may require additional techniques beyond the three studied here.

2. Context and Motivation

The Core Problem: LLM Pre-training Is a One-Shot Affair in a Continuously Updating World

The fundamental problem this paper addresses is the economic and computational inefficiency of the standard LLM development cycle. In current practice, when an organization wants to update a language model with new data—a freshly scraped Common Crawl snapshot, a higher-quality filtered corpus, or data in a new language—the dominant approach is to throw away the trained model and start over from random initialization, training on the union of all data, old and new. This process is described explicitly in Section 1:

"Currently, LLMs are re-trained on a combination of old and newly collected data."

This is deeply wasteful. The cost to pre-train a competitive LLM is enormous and growing. Meta's LLaMA 2 family (Touvron et al., 2023b), for instance, was trained on 2 trillion tokens. The cost of repeating such training runs every time a new dataset becomes available multiplies the total compute budget—and therefore the financial cost, energy consumption, and carbon footprint—of maintaining state-of-the-art LLMs. The paper frames this inefficiency as the central motivation: as the community relentlessly produces "new and higher-quality datasets" (Section 1), the pressure to update models grows, yet the default method for doing so is to ignore all prior investment and begin again.

This gap is significant for several overlapping reasons:

• Economic pressure on model providers: Organizations that deploy LLMs—whether open-source (Touvron et al., 2023a;b; Jiang et al., 2023; Gemma Team et al., 2024) or proprietary—must keep their models current to remain competitive. If the only reliable method for incorporating new data is full re-training, the cost of staying current grows with the number of updates, potentially making it uneconomical for all but the largest players.

• Environmental implications: The paper explicitly notes in its Broader Impact Statement that continual pre-training is "a promising method to significantly reduce the compute associated with updating a model and, hence, the energy required to maintain foundation models." The environmental calculus is clear: if continual training can match re-training performance at a fraction of the compute, it directly reduces the carbon footprint of the LLM lifecycle.

• Democratizing model maintenance: If updating models requires full re-training, only organizations with massive compute budgets can keep their models current. Continual pre-training could lower this barrier, enabling smaller research groups and companies to maintain competitive models incrementally as new data becomes available.

• Temporal scaling of data availability: The paper notes that datasets are being produced at an accelerating rate—SlimPajama (2023) is a newer, deduplicated version of RedPajama, which itself was based on LLaMA's training data. As web-scale datasets improve in quality and coverage, the gap between a model's training data and the best available data widens over time, making the need for updates more frequent. Without efficient updating mechanisms, model providers face a growing "staleness tax."

Prior Approaches and Where They Fall Short

The paper identifies several lines of existing work, each of which addresses a piece of the problem but fails to provide a complete, validated solution for the specific setting of continual pre-training of large autoregressive LLMs on hundreds of billions of tokens.

1. Training from Scratch on the Union of All Data

This is the default approach in the literature and industry. Methods like Chinchilla scaling laws (Hoffmann et al., 2022) and hyperparameter transfer (Yang et al., 2022) optimize how to train from random initialization under a compute budget, but they "assume that models will be trained from random initialization" (Section 1). The paper directly questions this assumption:

"Should practitioners always combine existing datasets and train from random initialization to obtain the best performance? Doing so for every update of the models quickly becomes expensive."

This approach falls short because it ignores the computational value of the existing trained checkpoint. Every re-training discards the knowledge already encoded in the model's weights, treating the problem as if starting from scratch is the only path to optimal performance. The paper's central empirical question is whether this assumption holds under controlled conditions.

2. Domain-Adaptive Continual Pre-training (DACPT) for Smaller-Scale MLMs

Existing work by Ke et al. (2022) and Gururangan et al. (2020) studies a related but fundamentally different setting: adapting a pre-trained masked language model (typically BERT or RoBERTa) to a sequence of domain-specific corpora (biomedical papers, legal documents, etc.) using self-supervised objectives. While the high-level idea—"continuing to pre-train on new data without forgetting the old"—is shared, the paper identifies several critical mismatches that make this prior work insufficient for the LLM setting:

• Dataset scale: DACPT work typically deals with domain-specific datasets of a few billion tokens or less. The paper explicitly distinguishes its setting: "we highlight that it is distinct from existing settings in the literature (Gururangan et al., 2020; Ke et al., 2022; Scialom et al., 2022; Xie et al., 2023) due to the large amount of incoming data we consider" (Section 1). When datasets are hundreds of billions of tokens, the dynamics of adaptation, forgetting, and the value of replay may be qualitatively different.

• Model architecture and scale: The DACPT literature primarily uses BERT-style masked language models of a few hundred million parameters. This paper studies autoregressive decoder-only transformers (the dominant LLM architecture) at up to 10B parameters. The paper notes that pre-training objective and model scale both influence continual learning behavior (Section 3.2), so findings from MLMs may not transfer.

• Focus on single domains: DACPT methods assume each new dataset represents a single domain, and they often develop domain-specific adaptation strategies (importance masks, adapter layers, sample selection). The paper studies general-purpose pre-training datasets (Pile, SlimPajama) that already mix many domains, making domain-specific techniques inapplicable.

• Assumption of no data access: Ke et al. (2022) explicitly assume data from previous domains is unavailable when training on new domains and develop parameter-importance masking to prevent forgetting without replay. The paper makes the more practical assumption that previous data is available (Section 4.2), making replay-based approaches viable and removing the need for more complex architectural interventions.

3. Continual Learning for Small-Scale LLMs Applied to Specific Domains

Several works (Sun et al., 2020; Jang et al., 2022a;b; Gong et al., 2022; Zan et al., 2022; Yadav et al., 2023a; Ma et al., 2023; Yang et al., 2024) apply continual pre-training to specific tasks and domains—e-commerce, temporal knowledge bases, math problem understanding, plant science—using autoregressive LMs. The paper acknowledges this body of work but identifies key limitations that make it insufficient for general-purpose LLM continual pre-training:

• Small dataset scale: Most of these works operate on datasets of "< 10B tokens" (Section 3.4). The paper's setting involves 200B–300B token datasets, which is 20–30× larger. At these scales, the cost of poor adaptation is much higher (if you spend enormous compute on a large dataset but don't adapt to it, the investment is wasted), and the dynamics of forgetting may be different (more tokens means more opportunities for new learning to overwrite old knowledge).

• Small model scale: These works use models smaller than the 405M–10B parameter range studied in this paper. The paper cites evidence that model scale and pre-training both reduce forgetting (Cossu et al., 2022; Ramasesh et al., 2022; Mehta et al., 2023; Mirzadeh et al., 2022) and that scaling laws for transfer show positive transfer improves with parameter count (Hernandez et al., 2021). Therefore, conclusions drawn at smaller model scales may not hold—and may be overly pessimistic—at the scale of modern LLMs.

• No systematic ablation of techniques: The paper explicitly notes of Yang et al. (2024), the closest concurrent work, that "unlike our work, they do not build a controlled experimental framework to systematically evaluate the validity of these approaches for continual pre-training" (Section 3.4). Without controlled ablations, it is impossible to determine whether the techniques (replay, LR re-warming) are individually necessary or whether their success is specific to the domain.

The only work approaching the paper's dataset scale is Gogoulou et al. (2023), which explores continual autoregressive language modeling across four languages (English, Danish, Icelandic, Norwegian) with 73B-token datasets each. While they use LR re-warming and re-decaying (which this paper also advocates), they do not use replay and do not systematically evaluate the combination of techniques across controlled distribution shifts. The paper builds on this precedent but significantly expands the scope and rigor.

4. Learning Rate Schedule Design for Single-Task Pre-training

The paper identifies a critical practical barrier to continual pre-training that arises from the learning rate schedules used by virtually all performant LLMs. The de facto standard—linear warmup followed by cosine decay to a small value (Hoffmann et al., 2022; Rae et al., 2021; Touvron et al., 2023a;b)—is designed for single-dataset, fixed-budget training. The learning rate ends at $\eta_{min} = 0.1 \cdot \eta_{max}$ or lower, meaning the model is essentially finished learning by the end of training.

When a practitioner loads such a model to continue training on new data, they face an immediate dilemma:

• Keep the learning rate at $\eta_{min}$ : The model trains at a glacial pace, spending enormous compute on the new dataset while making minimal progress. The paper demonstrates this directly (Section 6.1.2, Figure 4): a model continuing with constant $\eta_{min}$ adapts far less to the new dataset than one that re-warms.

• Re-warm the learning rate: Push the learning rate back up to a high value ( $\eta_{max}$ or similar) to enable meaningful learning on the new data, then decay it again. But this creates a new problem that the paper is the first to systematically study: re-warming causes catastrophic forgetting on the previous data, and the severity of forgetting scales with the re-warm magnitude (Section 6.1.2, Section 7.1).

• Use a constant learning rate throughout: This avoids the re-warming problem but sacrifices the known benefits of learning rate decay—models trained with a constant learning rate converge to worse final performance than those trained with proper decay (Hoffmann et al., 2022).

Prior work on learning rate schedules did not address this dilemma because it was focused on single-task training. Raffel et al. (2023) used inverse square root decay for LLMs, and Zhai et al. (2022) used "infinite learning rate schedules" for vision transformers, but neither work studied these schedules in a continual learning context with multiple sequentially arriving datasets. The paper's introduction of infinite schedules specifically for continual LLM pre-training (Section 7.2) is a direct response to this gap.

5. Replay in the Context of Large-Scale Autoregressive LM Pre-training

Replay—mixing data from previous tasks into the current training stream—is one of the oldest and most reliable techniques in continual learning (French, 1999; Rolnick et al., 2019; Buzzega et al., 2020). However, its applicability to the paper's setting was unclear for several reasons:

• Scale of forgetting: While replay is well-studied for supervised classification tasks where forgetting means accuracy dropping by a few percentage points, the paper deals with a setting where forgetting on the previous dataset means a validation loss increase from 2.17 to 3.56 nats (Table 3, stronger shift without replay)—a catastrophic degradation that raises questions about whether realistic amounts of replay can address forgetting of this magnitude.

• Cost of replay: In standard continual learning, replay often involves storing and re-processing small amounts of data (e.g., a few thousand examples). In LLM pre-training, even 1% replay of a 300B-token dataset means re-processing 3B tokens—a non-trivial but manageable cost. The paper's concept of compute-equivalent replay (Section 4.2) formalizes the tradeoff: replay tokens displace new tokens from the budget, so the question becomes whether the net effect (reduced forgetting + reduced new-data exposure) outperforms pure new-data training.

• Interaction with learning rate dynamics: The paper hypothesizes that replay is particularly important because of the re-warming dynamics. The rapid forgetting observed when re-warming (Figure 4, top left) is partly a learning rate artifact—the model's weights are being pushed hard in new directions—and partly due to distribution shift. Replay provides a regularizing signal that keeps the model anchored to the old distribution even as the learning rate is high. Without re-warming, replay might be unnecessary (the constant $\eta_{min}$ model forgets very little in Figure 4), but without replay, re-warming causes catastrophic forgetting. The combination is what makes the approach work.

Scialom et al. (2022) provided the closest precedent, showing that "as little as 1%" replay can mitigate forgetting in an LLM fine-tuning setting. But the paper's setting is fundamentally more challenging: continual pre-training on hundreds of billions of tokens subjects the model to far more optimization steps on the new distribution than fine-tuning, creating much greater opportunity for forgetting. The paper's demonstration that replay scales to this regime—and that different distribution shifts require different replay fractions (5% for weak, 25% for strong; Section 6.2)—extends this prior finding into a practically usable guideline.

How This Paper Positions Itself

The paper positions itself not as proposing a fundamentally new learning algorithm, but as providing the first systematic empirical validation that simple, existing techniques—when properly combined—solve the continual LLM pre-training problem at scale. The introduction states this explicitly:

"In this work, we do not intend to improve on the performance of models trained from a random initialization on all of the available data. Instead, we consider models trained on the union of existing datasets as baselines whose performance we seek to match using a combination of continual learning strategies at scale."

This framing is important. The paper sets a pragmatic goal—matching, not beating, the re-training baseline—and argues that reaching this bar with substantially less compute is a meaningful achievement. This contrasts with much of the continual learning literature, which often proposes novel algorithms to minimize forgetting under strict memory or computation constraints. Here, the constraints are looser (previous data is available, compute is available but should be minimized), and the goal is to match the gold standard of full re-training rather than to achieve state-of-the-art under artificial restrictions.

The paper positions itself as the bridge between two largely disconnected literatures:

1. The continual learning literature, which has developed sophisticated techniques (replay, regularization, dynamic architectures) but has primarily tested them on small-scale supervised learning benchmarks (split CIFAR-100, split mini-ImageNet) and has not systematically evaluated them on large-scale self-supervised pre-training of transformer models.

2. The LLM pre-training literature, which has developed best practices for learning rate schedules (cosine decay), batch sizes, and architecture choices, but has assumed training from random initialization on the complete dataset. This literature has not engaged with the problem of incremental updates.

The paper argues that LLMs are actually well-suited to continual learning for several reasons that the continual learning literature has underappreciated:

• Pre-training helps: Cossu et al. (2022) and Mehta et al. (2023) found that pre-trained models forget less than randomly initialized models, likely because their weights lie in flatter regions of the loss landscape. Since LLMs are heavily pre-trained by definition, they may be naturally more resistant to forgetting than the small models typically studied in CL benchmarks.

• Scale helps: Ramasesh et al. (2022) and Mirzadeh et al. (2022) found that larger models and wider architectures forget less than their smaller counterparts. The paper explicitly cites this trend and argues that "the trend of increasing language model capacity and pre-training dataset size in tandem... will yield models increasingly capable of continual learning" (Section 1).

• The autoregressive objective helps: Scialom et al. (2022) showed that autoregressive LLMs have a "strong ability to learn continually" which they hypothesize is related to the pre-training objective itself—next-token prediction may naturally encourage representations that are robust to distribution shift.

By bringing these observations together with the practical techniques of LR re-warming, re-decaying, and replay, the paper positions itself as demonstrating that the LLM and continual learning communities have been talking past each other: LLM practitioners assume re-training is necessary, while continual learning researchers assume their small-scale findings won't transfer. The paper's empirical results are intended to correct both misconceptions simultaneously.

A crucial aspect of the paper's positioning is its emphasis on controlled experimental design. Unlike many works that continually pre-train on whatever data is available and compare to a single baseline, the paper carefully constructs:

• Two distribution shifts (weak: English→English, strong: English→German) to test the generality of findings,
• Multiple compute-equivalent baselines (trained on individual datasets, trained on the union of all data) to isolate the effect of continual pre-training,
• Systematic ablations of LR warmup duration, re-warming magnitude, and replay percentages to identify which components are necessary and how they interact,
• Two model scales (405M and 10B) to test whether findings hold as models grow.

This experimental rigor is a deliberate response to what the paper views as a lack of controlled comparisons in the existing continual pre-training literature. The authors want to establish not just that these techniques work, but when, why, and under what conditions they work—transforming the question of continual LLM pre-training from anecdotal evidence and folk wisdom into an empirically grounded engineering practice.

3. Technical Approach

3.1 Reader Orientation

The system being built is a methodology—a set of training recipes and schedules—for updating an already-trained large language model on a new dataset without starting from scratch. The problem it solves is that standard LLM training ends with the learning rate decayed to a near-zero value, making the model incapable of efficiently learning from new data unless the learning rate is deliberately increased again, yet doing so causes catastrophic forgetting of previously acquired knowledge; the solution is a combination of learning rate re-warming with re-decaying (to enable adaptation to new data) and compute-equivalent replay of old data (to mitigate the forgetting that re-warming induces), optionally replaced by infinite learning rate schedules that avoid the re-warming problem entirely.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three interacting components, each addressing a distinct aspect of the continual pre-training problem:

1. Learning Rate Scheduler — determines the learning rate at each training step during both initial pre-training and subsequent continual pre-training phases. In the standard approach, this is a linear warmup followed by cosine decay to a small minimum value, then re-warmed and re-decayed for each new dataset. In the infinite schedule variant, it transitions from warmup through a cooldown phase to a constant value that persists across dataset boundaries, with an optional final annealing phase at deployment time. This component controls the plasticity side of the stability-plasticity tradeoff.

2. Replay Mechanism — mixes tokens from previously seen datasets into the training stream of the current dataset. When the model trains on dataset $\mathcal{D}_1$, a fraction $r$ of each training batch consists of tokens from $\mathcal{D}_0$, and the remaining $(1 - r)$ consists of tokens from $\mathcal{D}_1$. The total number of training tokens is held constant (compute-equivalent replay), so replay tokens displace new tokens. This component addresses the stability side of the stability-plasticity tradeoff by keeping the model anchored to the old distribution.

3. Optimizer State Reset Policy — at each dataset boundary, the AdamW optimizer states (first and second moment estimates) are reset to their default initialization values. This simulates the realistic scenario where a practitioner loads an open-weight model from HuggingFace, which does not include optimizer states, and must begin training on new data without the momentum and variance estimates from prior training.

Information flows as follows: a pre-trained checkpoint on $\mathcal{D}_0$ is loaded → the optimizer states are reset → the learning rate scheduler is configured with a new warmup phase, target $\eta_{max}$, and decay schedule → the data loader provides batches that interleave $\mathcal{D}_1$ tokens with $\mathcal{D}_0$ replay tokens according to the configured replay fraction → the model is trained on this mixture under the new learning rate schedule → at deployment time, the model is evaluated on both datasets and on downstream benchmarks to assess adaptation (performance on new data) and forgetting (performance on old data).

3.3 Roadmap for the Deep Dive

• First, the learning rate schedule mechanisms—linear warmup, cosine decay, re-warming, and re-decaying—because they are the primary lever controlling how much the model can learn from new data and how much it forgets.
• Second, the compute-equivalent replay mechanism, including how it is implemented and how its cost is accounted for, because replay is the countermeasure that prevents the forgetting induced by re-warming.
• Third, the infinite learning rate schedule design, including its four phases and two cooldown variants, because it represents an alternative to the re-warming-plus-replay combination that may be more elegant and scalable to many dataset transitions.
• Fourth, the optimizer state reset policy and the training configuration details common to all experiments, because these are the fixed infrastructure on which the other components operate.
• Fifth, the evaluation protocol—both validation loss measurements and downstream benchmark selection—because it defines what "matching the re-training baseline" means operationally, and understanding this protocol is essential for interpreting the paper's main results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that a simple and scalable combination of learning rate re-warming, learning rate re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, and that infinite learning rate schedules offer a promising alternative that avoids the pathologies of re-warming.

---

3.4.1 Linear Warmup and Cosine Decay: The Default Schedule That Creates the Problem

The paper first establishes the standard learning rate schedule used by virtually all performant LLMs, because understanding its mechanics is a prerequisite for understanding why continual pre-training is difficult and why the proposed solutions work.

The schedule consists of two sequential phases, illustrated in Figure 2:

Phase 1: Linear warmup. The learning rate $\eta_t$ at training iteration $t$ is increased linearly from zero to a maximum value $\eta_{max}$ over $T_{warmup}$ timesteps, ending at timestep $t_{ann} = T_{warmup}$:

$\eta_t = \eta_{max} \cdot \frac{t}{T_{warmup}}$

where $\eta_t$ is the learning rate at iteration $t$, $\eta_{max}$ is the maximum learning rate (a hyperparameter, typically $3 \cdot 10^{-4}$ in this paper's experiments), and $T_{warmup}$ is the number of warmup iterations.

What it computes: a ramp from zero learning rate to the full maximum learning rate, linearly interpolated by the fraction of warmup completed. At iteration 0, the learning rate is 0; at iteration $T_{warmup}$, it equals $\eta_{max}$.

Why this form: starting training at full learning rate with randomly initialized weights and noisy gradients can cause instability and divergence, particularly with large batch sizes (Goyal et al., 2018) and Post-LN transformer architectures (Popel & Bojar, 2018). The warmup allows the optimizer to establish reasonable gradient statistics before applying large weight updates. The linear form is simple and empirically effective.

Phase 2: Cosine decay. After warmup, the learning rate is annealed from $\eta_{max}$ to a small minimum value $\eta_{min}$ over $T_{ann}$ timesteps according to a cosine function, ending at timestep $t_{end} = T_{ann} + t_{ann}$:

$\eta_t = \eta_{min} + \frac{(\eta_{max} - \eta_{min})}{2} \cdot \left( \cos \left( \pi \cdot \frac{t - t_{ann}}{t_{end} - t_{ann}} \right) + 1 \right)$

where $\eta_t$ is the learning rate at iteration $t$, $\eta_{min}$ is the minimum learning rate (typically $0.1 \cdot \eta_{max}$), $t_{ann}$ is the iteration where annealing begins (the end of warmup), and $t_{end}$ is the total number of training iterations (set to match the token budget of the dataset).

What it computes: a smooth decay from $\eta_{max}$ to $\eta_{min}$ following a half-cosine shape. At the start of annealing ($t = t_{ann}$), the cosine term evaluates to $\cos(0) = 1$, making the bracket term equal to $(1 + 1) = 2$, yielding $\eta_t = \eta_{min} + (\eta_{max} - \eta_{min}) = \eta_{max}$. At the end of annealing ($t = t_{end}$), the cosine term evaluates to $\cos(\pi) = -1$, making the bracket term $(-1 + 1) = 0$, yielding $\eta_t = \eta_{min}$. The transition between these points follows the smooth, S-like shape of the half-cosine.

Why this form: Hoffmann et al. (2022) established that cosine decay is the best practice for LLM pre-training, significantly outperforming constant learning rates. The decay enables the model to converge to a better final solution—the high learning rate early in training allows the model to explore the loss landscape and escape narrow local minima, while the decaying learning rate allows it to settle into a wide, flat basin that generalises well. The cosine shape has the property that the learning rate changes slowly near the extremes (close to $\eta_{max}$ and $\eta_{min}$) and fastest in the middle, which empirically produces better final performance than linear decay.

The critical consequence for continual pre-training: the model ends training at $\eta_{min} \ll \eta_{max}$. For the paper's experiments, $\eta_{max} = 3 \cdot 10^{-4}$ and $\eta_{min} = 3 \cdot 10^{-5}$ ($0.1 \cdot \eta_{max}$). This means the learning rate in the final phase of pre-training on $\mathcal{D}_0$ is one-tenth of its peak value, and the model's weights have been optimized at this low learning rate for a substantial fraction of total training. Most open-source LLMs (LLaMA, LLaMA 2, Falcon, Mistral) follow this convention, making the problem studied in this paper broadly applicable.

The paper explicitly notes: "most performant open-source LLMs (Touvron et al., 2023a;b; Jiang et al., 2023; Gemma Team et al., 2024) decay their learning rate to a small value by the end of training. We hypothesize, therefore, that the learning rate must be re-increased and re-decayed to improve adaptation per compute spent when training on a new dataset."

---

3.4.2 Learning Rate Re-Warming and Re-Decaying for Continual Pre-training

When a new dataset $\mathcal{D}_1$ arrives, the practitioner faces a choice: what learning rate to use for continued training on $\mathcal{D}_1$ starting from the checkpoint that ended training on $\mathcal{D}_0$ at $\eta_{min}$? The paper identifies three natural strategies, two of which serve as baselines and one of which is the proposed approach:

Strategy 1: Constant $\eta_{min}$ . Continue training at the same small learning rate with which pre-training on $\mathcal{D}_0$ ended. This is the simplest approach and requires no schedule modification. The paper tests this as a baseline (Section 6.1.2, Figure 4).

Strategy 2: Re-warm to $\eta_{max}$ without decay. Linearly warm the learning rate from $\eta_{min}$ back up to the original $\eta_{max}$ and keep it constant for the remainder of training on $\mathcal{D}_1$. This re-enables meaningful weight updates but sacrifices the benefits of learning rate decay. The paper tests this as a second baseline.

Strategy 3: Re-warm and re-decay (the proposed approach). Linearly warm the learning rate from $\eta_{min}$ to $\eta_{max}$ (or a different target maximum), then cosine-decay it back to $\eta_{min}$ (or a different target minimum) over the token budget of $\mathcal{D}_1$. This is essentially restarting the cosine schedule on the new data.

The paper provides a formal description: the re-warming follows the same linear warmup as Equation 1 but starts from $\eta_{min}$ rather than zero and covers 1% of the total training iterations on $\mathcal{D}_1$. The re-decay follows the cosine schedule of Equation 2 but is fitted to the token budget of $\mathcal{D}_1$ — that is, $t_{end}$ corresponds to the total number of training steps for $\mathcal{D}_1$, and the learning rate reaches $\eta_{min}$ (or a specified fraction of $\eta_{max}$) at exactly the end of training on $\mathcal{D}_1$.

Specifically, for the paper's experimental configuration (Section 5.3): all models use a batch size of 1104 and sequence length of 2048 for a total of 2.26M tokens per step. For a 300B token dataset (SlimPajama), this corresponds to approximately 132,366 training iterations. For a 200B token dataset (German Common Crawl), it corresponds to approximately 86,000 training iterations. The cosine decay schedule is fitted to these iteration counts, reaching $\eta_{min} = 0.1 \cdot \eta_{max}$ at the final iteration.

The paper also evaluates what happens when the re-warm target $\eta_{max}$ is varied relative to the original pre-training maximum (Section 6.1.2). Three settings are tested:

• Half $\eta_{max}$ : re-warm to $1.5 \cdot 10^{-4}$, half the original $3 \cdot 10^{-4}$.
• Same $\eta_{max}$ : re-warm to $3 \cdot 10^{-4}$, identical to the original schedule.
• Double $\eta_{max}$ : re-warm to $6 \cdot 10^{-4}$, twice the original maximum.

In all cases, $\eta_{min}$ is set to $0.1 \cdot \eta_{max}$, giving minima of $1.5 \cdot 10^{-5}$, $3 \cdot 10^{-5}$, and $6 \cdot 10^{-5}$ respectively.

The paper also sweeps warmup duration (Section 6.1.1): warmups of 0% (immediate re-warming — the learning rate starts at $\eta_{max}$ and decays immediately), 0.5%, 1%, and 2% of total $\mathcal{D}_1$ training iterations. The default warmup duration selected for all subsequent experiments is 1% of training iterations, which for a 300B-token dataset corresponds to roughly 1,324 steps (approximately 3B tokens with the paper's batch size and sequence length).

What the re-warming and re-decaying operation achieves: it enables the model's weights to undergo large gradient updates again, moving the model from the local basin it settled into during the final low-learning-rate phase of $\mathcal{D}_0$ training toward regions of the loss landscape that are better for $\mathcal{D}_1$. The re-decay then allows the model to settle into a new local basin that represents a compromise between the two distributions. Without re-warming, the model is trapped near its $\mathcal{D}_0$ solution because the learning rate is too small to escape; with re-warming but without re-decay, the model wanders without settling. The combination is necessary for efficient adaptation.

Why this design and not alternatives: the alternatives—using a constant high learning rate throughout or using a non-decaying cyclic schedule—were tested and found wanting. The constant $\eta_{max}$ baseline in Figure 4 adapts well initially but finishes with worse validation loss on $\mathcal{D}_0$ (more forgetting) and often worse on $\mathcal{D}_1$ (no settling benefit). The constant $\eta_{min}$ baseline avoids forgetting but barely adapts to $\mathcal{D}_1$. The re-warming-plus-re-decaying approach is therefore a direct application of the known benefits of cosine decay to the continual learning setting: maximize plasticity early, maximize stability late, within each dataset phase.

---

3.4.3 Compute-Equivalent Replay

The re-warming and re-decaying strategy solves the adaptation problem but creates a forgetting problem: Figure 4 (Section 6.1.2) shows that models re-warmed to $\eta_{max}$ suffer substantial increases in $\mathcal{D}_0$ validation loss during $\mathcal{D}_1$ training. The paper therefore introduces replay—mixing tokens from $\mathcal{D}_0$ into the $\mathcal{D}_1$ training stream—as a countermeasure.

The key innovation in the paper's treatment of replay is the concept of compute-equivalent replay (Section 4.2). This ensures fair comparison between models with and without replay by holding the total number of training tokens constant. Formally, for a replay fraction $r$ (where $0 \leq r < 1$):

• Total tokens processed: $N_{total} = N_{\mathcal{D}_1}^{unique} + N_{replay} = \text{tokens}(\mathcal{D}_1)$
• Replay tokens: $N_{replay} = r \cdot N_{total}$
• Unique $\mathcal{D}_1$ tokens: $N_{\mathcal{D}_1}^{unique} = (1 - r) \cdot N_{total}$

In operational terms, when training on $\mathcal{D}_1$ with $r = 0.05$ (5% replay) and a total budget of 300B tokens, the model sees 285B unique tokens from $\mathcal{D}_1$ and 15B tokens replaying $\mathcal{D}_0$. The replay tokens from $\mathcal{D}_0$ are replayed in the same order they were originally seen during pre-training on $\mathcal{D}_0$ (Section 4.2), which the paper notes is a deliberate choice: in preliminary experiments, re-shuffling the replay data did not produce noticeable differences, so the simpler sequential-replay approach was adopted to minimize implementation complexity.

What compute-equivalent replay computes: given a fixed total number of training steps and a replay fraction, it determines how many of those steps process new data (driving adaptation) and how many process old data (driving stability). The replay fraction $r$ is the primary hyperparameter controlling the stability-plasticity tradeoff: higher $r$ means more stability (less forgetting) but less plasticity (fewer tokens of new data to learn from).

Why this form: the paper argues that simple random sampling as a replay strategy is justified by two considerations. First, at the scale of hundreds of billions of tokens, sophisticated replay sample selection strategies (e.g., selecting the most "forgettable" examples, or those near decision boundaries) are computationally expensive to implement and of uncertain benefit given the self-supervised nature of the objective—there are no class boundaries or per-sample importance weights. Second, the paper's goal is to establish a lower bound on continual pre-training performance achievable with the simplest possible techniques; if even random replay suffices, then more sophisticated methods can only improve results.

The replay percentages tested (Section 6.2) are:

• Weak shift (Pile → SlimPajama): 0.5%, 1%, 5%, 10%, 50%
• Strong shift (Pile → German): 1%, 5%, 10%, 25%, 50%

The selection of these specific values reflects a logarithmic sweep from extremely cheap replay (1%, which for a 300B dataset means only 3B tokens of replay) to extremely heavy replay (50%, which cuts the new data exposure in half). The paper's main results use 5% replay for the weak shift and 25% replay for the strong shift, selected because these values achieve near-minimal forgetting with minimal reduction in $\mathcal{D}_1$ adaptation (Table 2, Figure 5).

A subtle implementation detail: replay tokens are interleaved at the batch level, not the epoch level. Each training batch draws $r \cdot B$ tokens from $\mathcal{D}_0$ (in original training order) and $(1 - r) \cdot B$ tokens from $\mathcal{D}_1$ (in standard training order), where $B$ is the batch size in tokens. This ensures that every gradient update sees a mixture of old and new data, providing a consistent regularizing signal rather than alternating between epochs of pure new data and epochs of pure replay. The paper's total batch size is 1104 sequences × 2048 tokens = 2.26M tokens per batch. For 5% replay, this means each batch contains approximately 113K tokens from $\mathcal{D}_0$ and 2.15M tokens from $\mathcal{D}_1$.

---

3.4.4 Infinite Learning Rate Schedules

Section 7 introduces a fundamentally different approach to the learning rate problem: instead of re-warming and re-decaying (which requires replay to mitigate re-warming-induced forgetting), redesign the learning rate schedule so that re-warming is never necessary. This is inspired by the "infinite learning rate schedules" of Zhai et al. (2022) for vision transformers, but adapted and evaluated for the first time in the context of autoregressive LLM continual pre-training.

The core insight motivating infinite schedules (Section 7.1) is that re-warming causes forgetting even without a distribution shift. Figure 8 demonstrates this: when the model continues training on the same dataset (Pile → Pile) but with a re-warmed learning rate, the Pile validation loss initially increases—a transient forgetting effect that is purely a consequence of the large learning rate, not of distribution shift. The higher the re-warm, the larger this transient spike (re-warming to $6 \cdot 10^{-4}$ causes a peak loss increase of roughly 0.2, versus roughly 0.1 for $3 \cdot 10^{-4}$). This shows that re-warming is inherently damaging to the model's current solution, regardless of what it will learn next.

The paper proposes infinite learning rate schedules as a way to avoid re-warming by maintaining a constant, moderately high learning rate across all dataset boundaries, with only an initial warmup (on the first dataset) and a final annealing (before deployment). The schedule has four phases, formally defined in Section 7.2:

Phase 1: Linear warm-up. Identical to the standard schedule, performed only once at the start of training on the very first dataset ($\mathcal{D}_0$):

$\eta_t = \eta_{\max} \cdot \frac{t}{T_{\text{warmup}}} \quad \text{for } t \in [0, t_{\text{cd}}]$

where $t_{\text{cd}} = T_{\text{warmup}}$ is the end of the warmup phase. This phase exists for the same stability reasons as in standard pre-training.

Phase 2: Cooldown phase. The learning rate decays from $\eta_{max}$ to a constant value $\eta_{const}$ according to a decay function $f_{cd}$ over $T_{cd}$ timesteps. The paper considers two decay functions:

Cosine cooldown:

$f_{\text{cd}}(t) = \eta_{\text{const}} + \frac{\eta_{\max} - \eta_{\text{const}}}{2} \cdot \left( 1 + \cos \left( \pi \left( \frac{t - t_{\text{cd}}}{t_{\text{const}} - t_{\text{cd}}} \right) \right) \right)$

where $t_{\text{const}} = t_{\text{cd}} + T_{cd}$ is the end of the cooldown. This follows the same half-cosine shape as the standard schedule, but instead of decaying to a near-zero minimum, it decays to a non-trivial constant $\eta_{const}$.

Inverse square root cooldown:

$f_{\text{cd}}(t) = \eta_{\max} + \frac{\eta_{\text{const}} - \eta_{\max}}{h(1)} \cdot h \left( \frac{t - t_{\text{cd}}}{t_{\text{const}} - t_{\text{cd}}} \right)$

where $h(x) = \frac{1}{\sqrt{1 + \alpha x}} - 1$ and $\alpha$ controls the steepness of the decay. The function $h(x)$ is a shifted and scaled inverse square root: when $x = 0$, $h(0) = 0$, so $f_{cd}(t_{cd}) = \eta_{max}$; when $x = 1$, $h(1)$ is some negative value (since $\frac{1}{\sqrt{1+\alpha}} < 1$), and the fraction $\frac{\eta_{const} - \eta_{max}}{h(1)} \cdot h(1)$ yields exactly $\eta_{const} - \eta_{max}$, so $f_{cd}(t_{const}) = \eta_{const}$. The shape is a smooth monotonic decay whose curvature is determined by $\alpha$.

What the cooldown phase computes: a transition from the high learning rate needed for initial rapid learning ( $\eta_{max}$ ) to a lower but still substantial constant rate ( $\eta_{const}$ ) that will be maintained throughout all subsequent continual pre-training phases. This is distinct from standard cosine decay, which transitions to a near-zero rate that would be useless for learning on future datasets.

Why include a cooldown rather than staying at $\eta_{max}$ permanently: the paper does not provide a direct ablation of this choice, but the design follows the principle that the optimal learning rate for a converged model is lower than the optimal learning rate for a randomly initialized model (as evidenced by the universal adoption of learning rate decay in LLM pre-training). The cooldown finds a sustainable "cruising" rate that balances ongoing learning with solution stability. The paper does not report the specific value of $\eta_{const}$ used in experiments; this is a gap in the provided paper text.

Phase 3: Constant phase. The learning rate remains at $\eta_{const}$ for all remaining training on $\mathcal{D}_0$ and across all subsequent datasets $\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_{N-1}$:

$\eta_t = \eta_{\text{const}} \quad \text{for } t \in (t_{\text{const}}, t_{\text{ann}}]$

where $t_{\text{ann}}$ is the start of the final annealing phase.

This is the key innovation for continual learning: when a new dataset arrives, the model can begin training on it immediately with no schedule modification—no re-warming, no new cooldown, no change at all. The constant learning rate provides enough plasticity to adapt to the new data, and the lack of a learning rate spike avoids the re-warming-induced forgetting documented in Section 7.1.

Phase 4: Annealing phase. When the practitioner decides that continual pre-training is complete (all datasets have been seen and the model is ready for deployment), the learning rate is rapidly decayed to a small value $\eta_{min}$ over $T_{ann}$ timesteps:

$\eta_t = \eta_{\text{const}} \cdot \left( \frac{\eta_{\min}}{\eta_{\text{const}}} \right)^{\frac{t - t_{\text{ann}}}{t_{\text{end}} - t_{\text{ann}}}} \quad \text{for } t \in (t_{\text{ann}}, t_{\text{end}}]$

where $t_{\text{end}} = t_{\text{ann}} + T_{\text{ann}}$ is the final training step.

What this final annealing phase computes: an exponential decay from the constant cruising rate to a near-zero rate. At $t = t_{\text{ann}}$, the exponent is 0, so $\eta_t = \eta_{const} \cdot 1 = \eta_{const}$. At $t = t_{\text{end}}$, the exponent is 1, so $\eta_t = \eta_{const} \cdot (\eta_{min} / \eta_{const}) = \eta_{min}$. Between these points, the learning rate follows an exponential interpolation: $\eta_t = \eta_{const} \cdot (\eta_{min} / \eta_{const})^{\tau}$ where $\tau = (t - t_{ann}) / (t_{end} - t_{ann})$ is the fraction of the annealing phase completed.

Why exponential decay rather than cosine: the paper does not provide explicit justification, but the exponential form has the property that the ratio $\eta_{t+1} / \eta_t$ is constant throughout annealing, meaning the relative decrease per step is uniform. This is distinct from cosine decay, where the absolute decrease is fastest in the middle of the schedule. For a final convergence phase, an exponential schedule may provide more consistent settling behavior.

Why this form matters for continual learning: the practitioner can deploy the model at any point by running the annealing phase on-demand. If new data arrives after annealing, the practitioner loads the pre-annealing checkpoint (the model as it was at the end of the constant phase, $t = t_{ann}$) and continues training at $\eta_{const}$ on the new data, then anneals again. The schedule never commits to a specific total token budget, making it suitable for open-ended continual learning scenarios.

Important caveat (Section 8, Limitation 5): the paper acknowledges that infinite schedules may have a fundamental limitation when the number of datasets grows large. During the constant phase, the model is effectively trained with a constant learning rate, which Section 6.1.2 showed is suboptimal compared to decaying schedules. The final annealing recovers some of this suboptimality (as shown in Figure 10), but it's unclear how well this scales: if the model spends hundreds of billions of tokens at $\eta_{const}$, it may converge to a worse solution than one that underwent periodic decay. The paper explicitly flags this: "it is very possible that over enough tokens, the infinite schedules may end up being suboptimal due to only having a single phase of warmup and cooldown, as the learning on all subsequent datasets may just be equivalent to using a constant learning rate, which proved to be suboptimal."

---

3.4.5 Optimizer State Reset Policy

At each dataset boundary (when transitioning from $\mathcal{D}_i$ to $\mathcal{D}_{i+1}$), the paper resets the AdamW optimizer states to their default initialization values. This means the first moment estimate ( $m$ ) and second moment estimate ( $v$ ) are set to zero vectors, equivalent to the state at the start of training.

Why this design: the paper states explicitly (Section 5.2): "At the start of each phase, we reset the optimizer states, since optimizer states may not always be available, e.g. when using open-weight models from HuggingFace." This is a practical concern: when a practitioner downloads a pre-trained model from a public repository, they receive the model weights (stored in formats like PyTorch checkpoints or safetensors) but typically not the optimizer states, which are much larger (two full-precision floating-point tensors per parameter for Adam) and are rarely distributed. A continual pre-training recipe that assumes access to optimizer states would therefore be inapplicable to most open-weight models.

The consequences of resetting optimizer states are visible in Figure 10: when transitioning between IID splits of SlimPajama, the validation loss jumps up sharply at the dataset boundary despite the learning rate not changing (for infinite schedules). This spike is attributed to the optimizer state reset—without the momentum and adaptive learning rate statistics from the previous dataset, the first few steps on the new data are taken with poorly calibrated gradient estimates, causing a transient performance hit. The model recovers quickly (within a few billion tokens), but this is a fundamental cost of the practical assumption that optimizer states are unavailable.

An important interaction: this reset interacts with the learning rate schedule. If the optimizer states were preserved, re-warming the learning rate might be less damaging because the optimizer would "remember" the parameter-specific scaling from $\mathcal{D}_0$ training. Without states, re-warming applies the full learning rate to each parameter independently, which may exacerbate the "chaotic phase" observed in the no-warmup condition (Figure 3, 0% warmup curves) and the forgetting spike in Figure 4.

---

3.4.6 Training Configuration and Hyperparameters

The paper provides detailed training hyperparameters in Section 5.3 and Table 13 of the appendix. All experiments share a common infrastructure:

Model architecture: autoregressive decoder-only transformers using Pre-LN (pre-layer normalization), the standard architecture for modern LLMs. Two sizes are tested: 405M parameters and 9.6B parameters (rounded to "10B" in the paper), including embedding parameters.

Tokenizer: the same BPE (byte-pair encoding) tokenizer as Black et al. (2022), trained exclusively on the Pile. This is an important design choice: the tokenizer is optimized for English text and is kept fixed across all experiments, including the English-to-German shift. The paper acknowledges (Section 5.2) that "as the domain strays farther and farther away from the tokenizer's training corpus, the tokenizer may become a key bottleneck to performance" but "leave the treatment of the tokenizer to future work." This is a deliberate scoping decision: the paper aims to study learning rate and replay strategies in isolation, not to solve tokenizer mismatch.

Optimizer: AdamW (Loshchilov & Hutter, 2019), the standard optimizer for LLM pre-training, with decoupled weight decay.

Batch size: 1104 sequences × 2048 tokens per sequence = 2.26M tokens per global batch. This large batch size (over 2 million tokens per step) is typical for LLM pre-training and reduces gradient variance, which the paper notes (Section 8, Limitation 4) means "there is little variance in the gradient estimates" and results should be relatively stable across random seeds.

Sequence length: 2048 tokens. This is moderate by modern standards (many models now use 4096 or 8192) but was standard at the time the experiments were conducted using GPT-NeoX.

Precision: mixed precision FP16/FP32. The model weights, activations, and gradients are stored in FP16 for memory efficiency and computational speed (using NVIDIA tensor cores), while a master copy of weights is maintained in FP32 for optimizer updates to preserve numerical precision.

Parallelism for 405M models: data parallelism across 46 nodes, each with 6 GPUs, using a micro-batch size of 4. This is a straightforward data-parallel setup appropriate for smaller models.

Parallelism for 10B models: a combination of tensor parallelism (spanning 6 GPUs within a node, partitioning individual matrix multiplications across GPUs), pipeline parallelism (spanning 4 nodes, each handling a subset of transformer layers), and data parallelism (across 276 nodes total). Each model replica spans 24 GPUs across 4 nodes, and training uses gradient accumulation of 4 steps (meaning the effective micro-batch is smaller than the global batch, and gradients are accumulated over 4 forward/backward passes before an optimizer step).

Memory optimization techniques: ZeRO-1 optimizer sharding (Rajbhandari et al., 2020), which partitions the optimizer states across data-parallel ranks; activation checkpointing (Chen et al., 2016), which trades computation for memory by recomputing activations during the backward pass rather than storing them; and activation partitioning across tensor-parallel ranks, which reduces memory by distributing activations across GPUs within a tensor-parallel group.

Learning rate hyperparameters (standard cosine schedule):

• $\eta_{max} = 3 \cdot 10^{-4}$ for initial pre-training and the "same $\eta_{max}$" re-warm condition
• $\eta_{min} = 0.1 \cdot \eta_{max} = 3 \cdot 10^{-5}$ (the paper states that "most works" decay to $10 \times$ the maximum value)
• Warmup duration: 1% of training iterations (selected after the sweep in Section 6.1.1, corresponding to roughly 1,324 steps for a 300B dataset)
• Cosine decay fitted to the token budget of each dataset

Learning rate hyperparameters (infinite schedules):

• The paper does not provide explicit values for $\eta_{const}$, $T_{cd}$, $\alpha$ (for inverse square root cooldown), or $T_{ann}$. This is a notable gap in the paper's documentation.

Dataset sizes:

• Pile training set: 300B tokens (subset of the full ~330B Pile training set)
• SlimPajama training set: 299.28B tokens (subsampled from the full 606B SlimPajama; Table 1 provides the domain breakdown: Common Crawl 155.89B tokens at 52.09% sampling, C4 79.87B at 26.69%, GitHub 15.63B at 5.22%, Arxiv 13.25B at 4.43%, Book 12.58B at 4.20%, Wikipedia 11.96B at 4.00%, Stack Exchange 10.09B at 3.37%)
• German Common Crawl training set: 195.43B tokens
• SlimPajama validation set: default tokenized validation set (size not specified)
• German validation set: 982.6M tokens (the paper notes that deduplication between German training and validation was not performed—Limitation 2 in Section 8—but argues this is mitigated by taking distinct shards and by the fact that all compared models have access to the same data)

Training paradigm: standard autoregressive causal language modeling. The model is trained to predict the next token given all previous tokens, with the loss computed as cross-entropy between predicted token probabilities and the true next token. No instruction tuning, fine-tuning, or RLHF is applied—all models are evaluated as base models.

Data ordering: for each dataset, the paper follows "standard practice in LLM pre-training" (Section 5.1) and samples from domains proportionally to the amount of data available in each domain, such that one pass over the dataset does not repeat samples from any domain. This domain-proportional sampling is standard for multi-domain pre-training datasets to avoid over-representing small high-quality domains and under-representing large web-scrape domains. The Pile dataset comes pre-shuffled and mixed; SlimPajama is randomly sub-sampled and then sampled proportionally by domain as specified in Table 1. For replay, data from $\mathcal{D}_0$ is replayed in the order it was originally seen during pre-training on $\mathcal{D}_0$, without re-shuffling.

---

3.4.7 Evaluation Protocol

The paper evaluates continual pre-training using two complementary measurement approaches:

Validation loss. The primary metric in most experiments is the per-token cross-entropy loss on held-out validation sets for each dataset. This measures how well the model predicts the next token on data from each distribution. Loss is computed on:

• Pile validation set (for $\mathcal{D}_0$), measuring forgetting
• SlimPajama or German validation set (for $\mathcal{D}_1$), measuring adaptation

The "final loss" values reported in tables (Tables 2, 3, 4) are averaged over the last 100 training iterations sampled at intervals of 10 iterations. The paper notes that standard error was computed but not reported because it was less than 0.001 for all models, indicating very stable measurements at the end of training.

The "average final loss" (AVG column in tables) is the arithmetic mean of the Pile final loss and the $\mathcal{D}_1$ final loss. This single scalar summarises the stability-plasticity tradeoff: a model with low average loss has achieved good adaptation to $\mathcal{D}_1$ without catastrophic forgetting on Pile.

Downstream LM evaluation benchmarks. The paper evaluates models on a suite of standard benchmarks, with the caveat that all models are base models (no instruction tuning, no fine-tuning, no RLHF adaptation). The benchmarks are:

• English Commonsense Reasoning (0-shot): HellaSwag (normalized accuracy), Winogrande (unnormalized accuracy), PIQA (unnormalized accuracy), OpenBookQA (unnormalized accuracy), ARC-Easy and ARC-Challenge (unnormalized accuracy). These tasks test whether the model possesses basic world knowledge and reasoning capabilities.
• English World Knowledge (5-shot): NaturalQuestions (exact match), TriviaQA (exact match). These tasks test the model's factual knowledge.
• English Reading Comprehension (0-shot): BoolQ (unnormalized accuracy).
• English Math: MathQA (unnormalized accuracy).
• English Aggregated: MMLU (5-shot, unnormalized accuracy). MMLU is a broad benchmark covering 57 subjects from STEM, humanities, social sciences, and more.
• TruthfulQA: Evaluated in two configurations (MC1 and MC2), measuring the model's tendency to reproduce common misconceptions.
• German Benchmarks: Translated versions of HellaSwag, ARC-Challenge, TriviaQA, and MMLU, created by Plüster (2023) using the GPT-3.5 API for translation.

The "average accuracy" reported in Table 5 is the arithmetic mean across all English evaluation tasks. The paper also reports German HellaSwag normalized accuracy separately in Table 3 for the English-to-German shift experiments.

Why this evaluation strategy: the paper uses validation loss as the primary metric for comparing models and learning rate/replay configurations because it is directly optimized during training and is available at every training step, enabling continuous monitoring of forgetting and adaptation. Downstream benchmarks are reported at the final checkpoint and serve as a more ecologically valid measure of whether the model's acquired knowledge and reasoning capabilities survive continual pre-training. The combination provides both a fine-grained training-dynamics view (validation loss curves) and a practically meaningful endpoint evaluation (benchmark scores).

4. Key Insights and Innovations

Innovation 1: Re-Framing LLM Updates as a Stability-Plasticity Tradeoff at Scale, Resolvable by Simple Methods Previously Tested Only at Small Scale

The dominant assumption in the LLM pre-training community has been that re-training from scratch on the union of all data is the only reliable method for updating models on new data—an assumption baked into the scaling laws frameworks of Hoffmann et al. (2022) and the hyperparameter transfer methods of Yang et al. (2022), both of which focus exclusively on training from random initialization. The continual learning community, meanwhile, has developed numerous sophisticated techniques (elastic weight consolidation, progressive networks, dynamic architectures, knowledge distillation-based regularization) but has almost exclusively tested them on small-scale supervised learning benchmarks. This paper's central conceptual move is to identify the gap between these two communities as unnecessary and to demonstrate that at the scale of modern LLM pre-training, the problem is in fact easier than the smaller-scale CL literature would suggest—because pre-training and model scale both naturally mitigate forgetting (as shown by Cossu et al., 2022; Ramasesh et al., 2022; Mehta et al., 2023; Mirzadeh et al., 2022), and because the self-supervised autoregressive objective itself appears to produce representations resilient to distribution shift (Scialom et al., 2022). Rather than requiring complex interventions, the stability-plasticity tradeoff at scale can be managed by three off-the-shelf techniques—learning rate re-warming, re-decaying, and replay—none of which are novel individually, but whose combination at this scale and in this setting had never been systematically validated.

This is fundamentally a reframing contribution. Prior work implicitly assumed that the difficulty of continual learning scales with model and dataset size, requiring ever more elaborate solutions. The paper inverts this assumption, arguing—and providing controlled evidence—that the properties making LLMs powerful (scale, pre-training) also make them naturally better continual learners. The evidence supporting this reframing includes: the finding that at 10B parameters, the performance gap between continual pre-training (with only 5% replay) and full re-training narrows to an average validation loss difference of just 0.02 nats (Table 4); the finding that larger models forget less even without replay (10B forgets 0.23 nats versus 405M forgetting 0.27 nats, Section 6.4.1); and the finding that the same techniques work across both a weak and a strong distribution shift (Figure 6, Table 3), suggesting generality. By reframing LLMs as inherently well-suited to continual learning, the paper paves the way for a new research agenda where continual pre-training is the default update strategy rather than a risky alternative to re-training.

The significance beyond raw performance lies in the practical implications of this reframing. If the paper had instead shown that continual pre-training requires exotic, brittle, or computationally expensive techniques to match re-training, the finding would be of academic interest only. The demonstration that simple techniques suffice—and that they are already being adopted successfully by large-scale industrial efforts (Glorioso et al., 2024 using the same re-warming + replay recipe on a 7B model; DeepSeek-AI et al., 2024 using variations of these techniques for a Mixture-of-Experts model trained on 6T additional tokens, as noted in the "Recent works employing our techniques" box)—transforms continual pre-training from a research curiosity into a deployable engineering practice. This is an incremental advance in the techniques themselves but a fundamental shift in the perceived viability of continual LLM pre-training, with direct economic and environmental implications (the Broader Impact Statement explicitly frames continual pre-training as a mechanism to "significantly reduce the compute associated with updating a model").

Innovation 2: Diagnosing Re-Warming as a Primary Cause of Forgetting, Independent of Distribution Shift

The paper makes a subtle but consequential diagnostic contribution: it isolates learning rate re-warming itself as a significant independent cause of forgetting, separable from the forgetting caused by distribution shift between datasets. Section 7.1 provides the clean experimental evidence (Figure 8): continuing to pre-train on the same Pile dataset with a re-warmed learning rate causes a measurable increase in validation loss on that very dataset—a transient spike of ~0.1 nats with the standard ηmax of 3×10⁻⁴, and ~0.2 nats with a doubled ηmax of 6×10⁻⁴. This forgetting occurs despite zero distribution shift, isolating the learning rate as the causal mechanism.

Prior work in continual learning has attributed forgetting primarily to gradient interference between tasks (the weights optimal for task B differ from those optimal for task A) and, in the case of replay-based methods, to an insufficient or poorly chosen replay buffer. The diagnostic that the learning rate schedule itself induces forgetting is rarely articulated explicitly. Scialom et al. (2022) observed that replay helps, and the CL literature knows that high learning rates cause more forgetting, but the clean demonstration in Figure 8—same data, different learning rate, measurable and lasting forgetting—provides a mechanistic understanding that had been missing. The paper shows that the initial forgetting spike when transitioning to a new dataset (seen in all the training curves in Sections 6.1 and 6.2) is a compound effect of two separable phenomena: (1) learning-rate-induced destabilization of the existing solution, and (2) genuine distribution-shift-induced overwriting of old knowledge by new. By demonstrating that phenomenon (1) exists even in isolation, the paper provides a diagnostic framework that clarifies what replay is fighting against: it is not just reminding the model of old patterns to prevent representational drift, but also providing a stabilizing gradient signal that counteracts the destructive effect of large weight updates driven by the re-warmed learning rate.

This is a fundamental diagnostic advance rather than an algorithmic one. Its significance lies in how it redirects solution design: if re-warming is the root cause of a substantial fraction of forgetting, then the solution space expands beyond "better replay strategies" to include "schedule designs that avoid re-warming entirely"—which the paper pursues in Section 7.2 with infinite learning rate schedules. Without this diagnostic, the natural response to forgetting would be more replay, more regularization, or more architectural interventions; with it, a simpler and more elegant solution (infinite schedules) becomes conceptually available. The finding also provides a principled explanation for the paper's empirical observation that replay fractions need to be higher under stronger distribution shifts (25% for German versus 5% for SlimPajama to achieve comparable forgetting mitigation, Table 2): stronger shifts make both the re-warming-induced forgetting and the distribution-shift-induced forgetting worse, requiring more replay to counteract the compound effect.

Innovation 3: Infinite Learning Rate Schedules as an Architectural Solution to the Re-Warming Pathology

The paper introduces infinite learning rate schedules to LLM continual pre-training—not as a new mathematical construct (the concept originates from Zhai et al., 2022, for vision transformers), but as a conceptually distinct solution to the specific problem of schedule-induced forgetting. The standard approach to continual learning schedule design is reactive: accept that re-warming is necessary on each new dataset, then add replay to mitigate the forgetting it causes. The infinite schedule approach is preventive: redesign the schedule so that the learning rate never drops to a level that requires re-warming in the first place.

The four-phase design (warmup → cooldown → constant → annealing) embodies a clear conceptual separation of concerns that is absent from standard cosine decay. The warmup addresses initialization instability. The cooldown transitions from the high learning rate needed for initial convergence to a sustainable "cruising" rate—acknowledging, correctly, that a constant-high learning rate is suboptimal (as Figure 4 demonstrates: constant ηmax baselines finish with worse loss than decaying schedules). The constant phase maintains a learning rate sufficient for ongoing adaptation across arbitrarily many dataset transitions. The annealing phase—deferred until deployment time—recovers the known benefits of learning rate decay for convergence and generalization. Each phase addresses a distinct requirement of the continual pre-training lifecycle, and none of them conflict with the others across dataset boundaries.

This is a fundamental conceptual contribution because it reframes the schedule design problem from "how do we restart the optimization process on each dataset?" to "how do we construct a single schedule that accommodates an open-ended sequence of datasets without re-initialization?" The practical implications are substantial: with infinite schedules, there is no need to decide in advance how many tokens of each dataset to train on (a limitation the paper explicitly notes of standard cosine decay, which requires fitting the decay to the token budget); no need to carefully balance re-warm magnitude against replay fraction for each dataset transition; and no need to store or re-access previous data for replay at all (at least to address re-warming-induced forgetting—distribution-shift-induced forgetting may still require some replay, though the paper's infinite schedule experiments in Section 7.4 show negligible forgetting across IID dataset boundaries without any replay). The paper's evidence for the viability of infinite schedules is preliminary (Section 7.4, Figure 10, only covers the no-shift setting on 405M models), and the paper is candid about the limitations (constant phase may be suboptimal over very long horizons; stability during extended constant-rate training is untested at larger scales). But the conceptual move—from treating each dataset as requiring its own optimization cycle to treating all datasets as part of a single extended training run with a single, carefully designed schedule—is the kind of reframing that opens up new research directions, including schedule designs that dynamically adjust the constant rate based on detected distribution shift, or that interleave periodic brief annealing phases without full re-warming.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses three large-scale pre-training datasets: The Pile (Gao et al., 2020), a 300B-token subset of the full ~330B training set for initial pre-training ($\mathcal{D}_0$); SlimPajama (Soboleva et al., 2023), a 299.28B-token subsample of the full 606B dataset for the weak-distribution-shift $\mathcal{D}_1$, with domain proportions detailed in Table 1; and German Common Crawl (Laippala et al., 2022), a 195.43B-token training set drawn from the Oscar dataset for the strong-distribution-shift $\mathcal{D}_1$, with a 982.6M-token German validation set. For the 3-dataset no-shift setting (Section 7.4), SlimPajama is partitioned into three IID 100B-token splits. All datasets use the same BPE tokenizer as Black et al. (2022), trained exclusively on the Pile; the paper notes that domain-proportional sampling is used so that one pass over each dataset does not repeat samples from any domain.

• Base model(s). All experiments use autoregressive decoder-only transformer models with Pre-LN (pre-layer normalization) at two scales: 405M and 9.6B (referred to as 10B) parameters, including embeddings. The architecture and training framework are based on GPT-NeoX (Andonian et al., 2021) with Megatron-DeepSpeed (Shoeybi et al., 2019; Microsoft, 2020). The paper states these models were chosen to be "representative of the capabilities of many contemporary LLMs" (Section 4, inferred) and to test whether findings hold across an order-of-magnitude scale difference. The models are pre-trained from random initialization on $\mathcal{D}_0$ using the standard linear warmup + cosine decay schedule, simulating the state of publicly available open-weight LLMs at the start of continual pre-training.

• Metrics. The primary metric is per-token cross-entropy validation loss, measured separately on held-out validation sets for $\mathcal{D}_0$ (Pile) and $\mathcal{D}_1$ (SlimPajama or German Common Crawl). "Final loss" values reported in Tables 2, 3, and 4 are averaged over the last 100 training iterations sampled at intervals of 10 iterations; the paper reports that standard error was <0.001 for all models. The "AVG" column is the arithmetic mean of the $\mathcal{D}_0$ and $\mathcal{D}_1$ final losses, used as a single scalar capturing the stability-plasticity tradeoff. Downstream evaluation uses a suite of English and German LM benchmarks (Section 5.4): English commonsense reasoning (HellaSwag, Winogrande, PIQA, OpenBookQA, ARC-Easy, ARC-Challenge, all 0-shot), world knowledge (NaturalQuestions exact match, TriviaQA exact match, both 5-shot), reading comprehension (BoolQ 0-shot), math (MathQA 0-shot), aggregated knowledge (MMLU 5-shot), and TruthfulQA (MC1 and MC2). German benchmarks are translated versions of HellaSwag, ARC-Challenge, TriviaQA, and MMLU from Plüster (2023). For HellaSwag, normalized accuracy is reported; for NaturalQuestions and TriviaQA, exact match; all other tasks report unnormalized accuracy. The paper cautions that all models are evaluated as base models without instruction tuning, fine-tuning, or RLHF adaptation.

• Baselines. The paper uses three natural baselines for the $N=2$ settings: (a) a model trained from random initialization on $\mathcal{D}_0$ alone (e.g., 300B tokens of Pile), representing the starting checkpoint before any update; (b) a model trained from random initialization on $\mathcal{D}_1$ alone (e.g., 300B tokens of SlimPajama, or 200B tokens of German), representing a compute-equivalent alternative to continual pre-training where one discards the $\mathcal{D}_0$ model entirely; and (c) a model trained from random initialization on $\mathcal{D}_0 \cup \mathcal{D}_1$ (e.g., 600B tokens of Pile+SlimPajama, or 500B tokens of Pile+German), representing the expensive full re-training baseline that the paper considers an upper bound on achievable performance. For learning rate schedule experiments (Section 6.1.2), two additional baselines are included: constant $\eta_{min}$ (continuing training on $\mathcal{D}_1$ at the small learning rate with which $\mathcal{D}_0$ training ended, $3 \cdot 10^{-5}$) and constant $\eta_{max}$ (re-warming to the original maximum $3 \cdot 10^{-4}$ but not re-decaying).

• Generation budget / compute accounting. Compute is measured and held constant in terms of total training tokens processed. The paper's central accounting mechanism is "compute-equivalent replay" (Section 4.2): when comparing a model trained with replay fraction $r$ to one without replay, the total number of training tokens is kept identical by reducing the number of unique $\mathcal{D}_1$ tokens to accommodate replay tokens from $\mathcal{D}_0$. For example, a model with 5% replay on a 300B-token budget sees 285B unique $\mathcal{D}_1$ tokens and 15B replay tokens from $\mathcal{D}_0$, totaling 300B tokens—the same total as the no-replay baseline which sees 300B unique $\mathcal{D}_1$ tokens. Training FLOPs are approximately proportional to total tokens processed, so this controls for total computational cost. The paper does not account for the cost of difficulty estimation (unlike the reference example paper), nor does it include optimizer state storage or data loading overhead in compute calculations. For the FLOPs-matched comparison (there is no Section 7 FLOPs-matched comparison between pretraining and test-time compute in this paper—this is a continual pre-training paper, not a test-time compute paper, so the "generation budget" framing applies differently), the relevant accounting is simply that the continually pre-trained models process fewer total unique tokens than the $\mathcal{D}_0 \cup \mathcal{D}_1$ baseline (e.g., the 5% replay model processes 300B Pile + 300B total tokens on $\mathcal{D}_1$ = 600B total tokens, same as the union baseline, but only 585B unique tokens of which 300B are Pile and 285B are SlimPajama). The compute savings come from the fact that a pre-trained checkpoint on $\mathcal{D}_0$ is assumed to already exist—Figure 1 illustrates this as "PT on Pile" starting from an existing model, so the $\mathcal{D}_0$ pre-training cost is amortized or considered sunk.

• Cross-validation / statistical protocol. The paper does not use cross-validation or multiple random seeds due to computational cost (explicitly noted as Limitation 4 in Section 8). All results are from single training runs. The paper argues this is mitigated by the large batch size (2.26M tokens per step), which "means there is little variance in the gradient estimates," and by the fact that dataset samples are processed in the same order across all compared runs. The paper acknowledges that "there is likely a stochastic element to some results" but does not quantify it. No confidence intervals, error bars, or statistical significance tests are reported on validation loss or benchmark scores beyond noting that standard error on final loss measurements was <0.001.

Main Quantitative Results

6.1.1: The Effect of Linear Warmup Duration

The paper investigates whether the duration of the linear warmup phase during learning rate re-warming affects forgetting or adaptation, testing warmup durations of 0% (immediate re-warming to $\eta_{max}$ and beginning decay), 0.5%, 1%, and 2% of total $\mathcal{D}_1$ training iterations on 405M parameter models. Results are reported in Figure 3 for the first 50B tokens of continual pre-training.

Headline finding: Warmup duration has negligible impact on final adaptation and forgetting after 50B tokens, though it affects early training dynamics. The paper states: "in the settings explored, the duration of the linear warm-up phase does not appear to affect forgetting or adaptation as measured by the validation loss when continuing to pre-train, although it can prevent initial transient spikes in the loss."

For the weak shift (Pile → SlimPajama, Figure 3a-b): the 0% warmup model experiences a sharp transient spike in both Pile validation loss (forgetting) and SlimPajama validation loss (adaptation) in the first few billion tokens, then recovers and converges to approximately the same loss as the longer-warmup models by 50B tokens. The 0.5%, 1%, and 2% warmup models show progressively smoother initial transitions, with the 2% warmup model exhibiting the most gradual changes in both forgetting and adaptation. By 50B tokens, Pile validation loss for all warmup durations converges within roughly 0.02 nats of each other (visual estimate from Figure 3b), and SlimPajama validation loss differences are similarly negligible.

For the strong shift (Pile → German, Figure 3c-d): the same pattern holds but is amplified. The 0% warmup model shows a dramatic initial spike in Pile validation loss (rising from approximately 2.2 to approximately 4.5 nats in the first ~2B tokens, visual estimate from Figure 3d), followed by partial recovery. Longer warmups substantially reduce the magnitude of this initial spike (the 2% warmup model peaks at approximately 3.0 nats on Pile validation loss). However, by 50B tokens, all models converge to similar Pile and German validation losses, with differences under ~0.1 nats. The paper notes that the 0% warmup model "undergo[es] an initial chaotic phase (as seen in the continual learning literature (De Lange et al., 2022))."

The paper concludes that 1% warmup duration is a safe default and uses this value for all subsequent experiments. This corresponds to approximately 1,324 steps for a 300B-token dataset with the paper's batch size and sequence length (2.26M tokens per step).

6.1.2: The Effect of Re-Warming, Re-Decaying, and Varying ηmax

The paper evaluates five learning rate strategies for continual pre-training on the full $\mathcal{D}_1$ datasets (300B tokens for SlimPajama, 200B tokens for German), all starting from a checkpoint pre-trained on 300B tokens of Pile. The five strategies are: constant $\eta_{min}$ ($3 \cdot 10^{-5}$, the learning rate at the end of Pile pre-training), constant $\eta_{max}$ (re-warmed to $3 \cdot 10^{-4}$ and held constant), and three re-warm + re-decay strategies with $\eta_{max}$ set to $1.5 \cdot 10^{-4}$ (half), $3 \cdot 10^{-4}$ (same), and $6 \cdot 10^{-4}$ (double) the original pre-training maximum, all decaying via cosine schedule to $\eta_{min} = 0.1 \cdot \eta_{max}$ by the end of $\mathcal{D}_1$ training. Results are in Figure 4 and summarized below from visual inspection and the paper's discussion.

Headline finding for adaptation: Re-warming and re-decaying the learning rate is necessary to best adapt to the new dataset. The constant $\eta_{min}$ model adapts the least to $\mathcal{D}_1$ in both shifts, achieving substantially higher final $\mathcal{D}_1$ validation loss than any re-warmed model. For the weak shift (Figure 4b), the constant $\eta_{min}$ model's SlimPajama validation loss plateaus at approximately 2.65 nats while re-warmed models reach 2.50–2.55 nats. For the strong shift (Figure 4d), the gap is larger: constant $\eta_{min}$ reaches approximately 1.30 nats on German validation versus 1.10–1.15 nats for re-warmed models. The constant $\eta_{max}$ model adapts well initially but finishes with worse $\mathcal{D}_1$ loss than the re-warm + re-decay models in the weak shift case, and comparably in the strong shift case.

Headline finding for forgetting: The constant $\eta_{min}$ model experiences the least forgetting on Pile across both shifts—its Pile validation loss remains near the starting value of approximately 2.17 nats throughout training (Figure 4a, 4c). All re-warmed models experience substantial increases in Pile validation loss, with higher $\eta_{max}$ values causing more forgetting. For the weak shift: $\eta_{max} = 1.5 \cdot 10^{-4}$ causes Pile loss to rise from 2.17 to approximately 2.30 nats; $\eta_{max} = 3 \cdot 10^{-4}$ rises to approximately 2.44 nats; $\eta_{max} = 6 \cdot 10^{-4}$ rises to approximately 2.55 nats. The constant $\eta_{max}$ model shows intermediate forgetting. For the strong shift: the effects are amplified, with $\eta_{max} = 6 \cdot 10^{-4}$ driving Pile loss above 3.5 nats by the end of training, and even the lowest re-warm ($1.5 \cdot 10^{-4}$) causing significant degradation. The paper explicitly notes: "small increases or decreases in $\eta_{max}$ allow to trade-off between more or less adaptation. A stronger distribution shift seems to be a catalyst for both forgetting and adaptation."

Headline finding relative to union baseline: No model matches the $\mathcal{D}_0 \cup \mathcal{D}_1$ baseline. For the weak shift, the union-trained model achieves Pile validation loss of approximately 2.17 nats and SlimPajama loss of approximately 2.53 nats (AVG ≈ 2.35). The best re-warmed model ($\eta_{max} = 3 \cdot 10^{-4}$) achieves Pile loss ≈ 2.44 and SlimPajama loss ≈ 2.50 (AVG ≈ 2.47)—a gap of 0.12 nats in average loss. For the strong shift, the union-trained model achieves Pile loss ≈ 2.26 and German loss ≈ 1.25 (AVG ≈ 1.75), while the best re-warmed model achieves significantly worse Pile loss (above 3.5 nats without replay). The paper concludes that "re-warming and re-decaying are necessary to maximize adaptation to the new dataset" but that "these models experience significant forgetting on $\mathcal{D}_0$, showing the need for replay to make these models competitive with the union baseline."

6.2: The Effect of Replay

The paper evaluates compute-equivalent replay at multiple fractions (weak shift: 0.5%, 1%, 5%, 10%, 50%; strong shift: 1%, 5%, 10%, 25%, 50%) for 405M parameter models, with all models using re-warming and re-decaying at $\eta_{max} = 3 \cdot 10^{-4}$. Results are in Figure 5 and Table 2. The key comparison is whether replay can close the gap to the $\mathcal{D}_0 \cup \mathcal{D}_1$ baseline that was observed in Section 6.1.2.

Headline finding for weak shift (Pile → SlimPajama, 300B tokens on $\mathcal{D}_1$): Even 1% replay substantially reduces forgetting relative to the 0% replay model, and 5% replay brings average validation loss within 0.02 nats of the union baseline while seeing 15B fewer unique SlimPajama tokens. The 0% replay model reaches final Pile loss = 2.44, SlimPajama loss = 2.50, AVG = 2.47. Adding 5% replay reduces Pile loss to 2.23, keeps SlimPajama loss at 2.51 (negligible adaptation cost), yielding AVG = 2.37—very close to the union baseline at AVG = 2.35 (Pile = 2.17, SlimPajama = 2.53). At 50% replay, Pile loss improves further to 2.16 but SlimPajama loss degrades to 2.54, yielding AVG = 2.35, exactly matching the union baseline's average loss, though the paper notes this comes at the cost of seeing 150B fewer unique SlimPajama tokens (the model only sees 150B unique SlimPajama tokens out of the 300B budget due to 50% replay). The paper observes that "for both shifts, the use of replay seems to negligibly affect adaptation to the downstream dataset, showing that reducing forgetting via replay comes at very little cost when continually pre-training LLMs."

Headline finding for strong shift (Pile → German, 200B tokens on $\mathcal{D}_1$): Substantially more replay is needed to combat the more severe forgetting. The 0% replay model suffers catastrophic forgetting: Pile loss rises from 2.17 to 3.56 (a 1.39 nat increase), while adapting well to German (loss = 1.11). At 25% replay, Pile loss is reduced to 2.33 and German loss is 1.16, yielding AVG = 1.75, which exactly matches the union baseline's AVG = 1.75 (Pile = 2.26, German = 1.25). At 50% replay, Pile loss further improves to 2.24 and German loss degrades to 1.22, producing AVG = 1.73, marginally better than the union baseline—but the model sees only 100B unique German tokens compared to the union baseline's full access to all 500B tokens (300B Pile + 200B German). The paper notes: "the stronger shift requires more replay to mitigate forgetting to the same extent."

Difficulty-dependent behavior: The paper observes that replay fractions exhibit a saturation effect: beyond a certain threshold (roughly 5% for weak shift, 25% for strong shift), additional replay provides diminishing returns on forgetting reduction while increasingly cutting into adaptation budget. The 50% replay models for both shifts achieve the lowest Pile validation loss among continual pre-training runs but begin to show adaptation degradation on $\mathcal{D}_1$. The paper explicitly notes this tradeoff: "when using an extreme amount of replay (50%), we observe that the model adapts relatively significantly worse to $\mathcal{D}_1$."

6.3: Final Performance for Weak and Strong Distribution Shifts (405M Models)

The paper selects two representative continual pre-training configurations—0% replay (re-warm and re-decay only) and replay (5% for weak shift, 25% for strong shift)—and compares them exhaustively against the three baselines (Pile-only, SlimPajama/German-only, and union-trained). Results are in Figure 6, Table 3, and Table 5 (bottom section).

Validation loss results (Table 3, Figure 6):

• Weak shift (Pile → SlimPajama): The 5% replay model achieves AVG = 2.37, within 0.02 of the union baseline's AVG = 2.35. The 0% replay model achieves AVG = 2.47, falling short by 0.12 nats. Both continual pre-training models adapt better to SlimPajama (loss = 2.50 and 2.51) than the union baseline (2.53), suggesting that the sequential training regime may actually benefit $\mathcal{D}_1$ performance slightly at the cost of $\mathcal{D}_0$ degradation. The Pile-only baseline achieves the best Pile loss (2.17) but worst SlimPajama loss (2.70, AVG = 2.44), while the SlimPajama-only baseline achieves the mirror image (Pile = 2.51, SlimPajama = 2.53, AVG = 2.52). Both individually-trained baselines are outperformed by the 5% replay model on average.

• Strong shift (Pile → German): The 25% replay model achieves AVG = 1.75, exactly matching the union baseline's AVG = 1.75, with nearly identical per-dataset losses (replay: Pile = 2.33, German = 1.16; union: Pile = 2.26, German = 1.25). The replay model actually achieves slightly better German loss (1.16 vs. 1.25) at the cost of slightly worse Pile loss (2.33 vs. 2.26). The 0% replay model achieves AVG = 2.34, far worse than the union baseline due to catastrophic Pile forgetting (Pile = 3.56, German = 1.11). The German-only baseline achieves terrible Pile loss (3.97) but best German loss (1.17, though marginally worse than the 0% replay model's 1.11), highlighting that continual pre-training from an English model provides better German adaptation than training from scratch on German alone—a form of positive transfer.

Downstream benchmark results (Table 3, right columns; Table 5 bottom section):

• English evaluation average (Table 5, 405M section): The 5% replay model achieves average English accuracy of 35.14%, slightly outperforming both the union baseline (34.30%) and the 0% replay model (34.93%), though the paper notes "differences are likely not significant" due to evaluation noise at this model scale. The Pile-only and SlimPajama-only baselines score 33.95% and 34.11% respectively—both lower than the 5% replay model. Task-level differences are modest: the replay model shows notable advantages on ARC-Challenge (23.55% vs. 23.55% for union—a tie) and ARC-Easy (55.01% vs. 52.99%), while the union model edges ahead on BoolQ (55.57% vs. 57.92%)—but here the replay model has the advantage. The paper does not report task-level statistical significance.

• German HellaSwag (Table 3, far right column): The 25% replay model achieves 31.04% normalized accuracy on HellaSwag-DE, compared to 30.45% for the union baseline and 31.23% for the 0% replay model. The German-only baseline achieves 29.53%, and the Pile-only baseline achieves 27.09% (near random). The continually pre-trained models thus outperform all baselines on this metric, though the paper notes that on all other German evaluation tasks (ARC-Challenge-DE, TriviaQA-DE, MMLU-DE), German-trained models were "near-random chance accuracy" and results are not reported in the main body (Table 11 in the appendix). A qualitative study of German generation (Appendix A.5, Table 8) confirms "an observable improvement in the generative quality of German-language outputs from the models trained on German Common Crawl when compared to the Pile baseline."

• English evaluation for German-trained models (Table 3, English column): The 25% replay model achieves 32.48% average English accuracy, closely matching the union baseline at 32.43%—evidence that replay successfully preserves English capabilities despite 200B tokens of German training. The 0% replay model drops to 29.20%, showing significant English degradation without replay. The German-only baseline scores 27.74%, and the Pile-only baseline scores 33.95%, providing lower and upper bounds respectively. The paper notes that "the strong replay used with the 25% replay German model helps to reduce this gap" between continually pre-trained models and the Pile-only baseline.

6.4: Final Performance at 10B Parameter Scale (Weak Shift Only)

The paper scales the weak-shift experiment (Pile → SlimPajama) to 10B parameters, training both the full suite of baselines and the two continual pre-training configurations (0% replay, 5% replay) from scratch. Results are in Figure 7, Table 4 (top section), and Table 5 (top section).

Validation loss results (Table 4, Figure 7):

• At 10B scale, the 5% replay model achieves AVG = 1.89, within 0.02 nats of the union baseline's AVG = 1.87. This gap is identical to the 405M-scale gap (both 0.02 nats), suggesting the relative benefit of continual pre-training with replay does not degrade with model scale. The 0% replay model achieves AVG = 1.99, slightly better than the Pile-only baseline (AVG = 1.99) and substantially better than the SlimPajama-only baseline (AVG = 2.07).

• The absolute losses are substantially lower than at 405M scale: Pile loss for the union baseline drops from 2.17 (405M) to 1.72 (10B), and SlimPajama loss drops from 2.53 to 2.02—a roughly 20% reduction in perplexity (exp(2.17)/exp(1.72) ≈ 1.57× lower perplexity).

• An interesting scale-dependent effect emerges: at 405M, the union baseline achieves identical per-dataset loss to the individually-trained baselines (Pile: 2.17 vs. 2.17; SlimPajama: 2.53 vs. 2.53). At 10B, the union baseline outperforms both individually-trained baselines on their respective datasets (Pile: 1.72 vs. 1.75; SlimPajama: 2.02 vs. 2.05). The paper hypothesizes this occurs "due to larger models having more capacity, thus being capable of learning at a higher rate for longer." This positive transfer effect benefits the union baseline but does not disadvantage continual pre-training, since the replay model also benefits from the larger capacity.

• Forgetting behavior with scale: the 0% replay 10B model's Pile loss degrades by 0.23 nats (from 1.75 to 1.98), while the 0% replay 405M model degrades by 0.27 nats (from 2.17 to 2.44). The paper interprets this as weak evidence that "larger models forget less, confirming our hypothesis," consistent with prior work (Ramasesh et al., 2022; Mirzadeh et al., 2022). The replay benefit is similar across scales: 5% replay reduces Pile forgetting by 0.19 nats at 10B (1.98 → 1.79) and by 0.21 nats at 405M (2.44 → 2.23)—a "negligible difference in forgetting-reduction from replay despite the order of magnitude difference in parameters."

Downstream benchmark results (Table 5, top section):

• The 5% replay model achieves average English accuracy of 47.68%, essentially matching the union baseline's 48.00% (difference of 0.32 percentage points) and the 0% replay model's 47.53%. The SlimPajama-only baseline scores 45.37% and the Pile-only baseline scores 43.45%, indicating that updating the model on SlimPajama improves downstream performance regardless of the training method.

• Task-level differences between the 5% replay and union models are mixed: the replay model wins on ARC-Easy (74.24% vs. 73.57%), OpenBookQA (30.60% vs. 27.80%), and Winogrande (68.67% vs. 67.32%), while the union model wins on MMLU (37.78% vs. 28.79%) and BoolQ (72.05% vs. 70.80%). The MMLU gap of approximately 9 percentage points is notable and the paper acknowledges it as a potential concern: "While this degradation in MMLU performance between both models could be cause for concern, we suspect it is due to the limited amount of training data used in our study." The paper cites Glorioso et al. (2024) as evidence that larger-scale applications of these techniques do not suffer MMLU degradation. The 0% replay model wins on HellaSwag (73.66% vs. 73.24% for replay, 73.39% for union) and BoolQ (73.18% vs. 70.80% and 72.05%), while underperforming on several knowledge-intensive tasks (NaturalQuestions EM: 12.99% vs. 13.32% for replay, 12.41% for union; TriviaQA EM: 57.94% vs. 57.86% for replay, 56.73% for union—essentially tied).

• The replay model's average accuracy of 47.68% represents a substantial improvement over the starting Pile checkpoint (43.45%, +4.23 percentage points), while the union baseline improves by a comparable +4.55 points. The individually-trained SlimPajama model scores 45.37%, suggesting that even training on SlimPajama alone (without Pile knowledge) outperforms the Pile-only model on these English-centric benchmarks—likely because SlimPajama (2023) contains newer, higher-quality data.

7.3–7.4: Infinite Learning Rate Schedules

The paper evaluates two variants of infinite learning rate schedules—cosine cooldown ("Cosine Inf") and inverse square root cooldown ("InvSqrt Inf")—against a standard cosine decay baseline in two settings.

Single-dataset pre-training on 300B SlimPajama (Section 7.3, Figure 9): All three schedules achieve similar final validation loss, demonstrating that infinite schedules are viable for standard single-dataset pre-training and do not sacrifice final performance. The cosine baseline and Cosine Inf schedule converge to nearly identical validation losses by 300B tokens (both reaching approximately 2.53, visual estimate from Figure 9c). The InvSqrt Inf schedule reaches a marginally higher final loss (approximately 2.55). The paper notes that the infinite schedules "have the advantage that one can start annealing at any time in the constant phase to efficiently improves the loss when deciding to finalize pre-training, and a pre-annealing checkpoint can be loaded to continue pre-training."

Three-dataset no-shift continual pre-training (Section 7.4, Figure 10): Models are trained from random initialization on three IID 100B-token splits of SlimPajama, simulating a setting where new data from the same distribution arrives over time. Both infinite schedules and a repeated-cosine-decay baseline are evaluated. Key findings:

• Negligible forgetting at dataset boundaries for infinite schedules: When transitioning between splits, the infinite schedule models show sharp but small and rapidly recovering validation loss spikes attributed to optimizer state reset (Section 7.4: "the losses initially increase sharply due to re-initializing the optimizer states, the infinite schedules models immediately recover from this"). After recovery, validation loss continues improving smoothly as if no dataset boundary had occurred. The repeated-cosine model shows a different pattern: it must re-warm at each boundary, which initially increases loss more substantially, then re-decay to bring it back down.

• Similar final performance: All three schedules achieve comparable final validation loss after 300B total tokens (all within ~0.05 nats of each other, visual estimate from Figure 10c). The annealing phase at the end of training provides a substantial final loss reduction—Figure 10 shows the "decayed" checkpoints (those that underwent Phase 4 annealing) achieving losses of approximately 2.40–2.45, while the final constant-phase checkpoints are at approximately 2.50–2.55.

• Overlap of schedules: Figure 10b notes that after approximately 80B tokens of training on the first split, the Cosine Inf and InvSqrt Inf constant-phase curves overlap, indicating that the specific cooldown function has negligible long-term effect—only the constant rate $\eta_{const}$ matters for asymptotic behavior.

The paper does not test infinite schedules under distribution shift (weak or strong), which it acknowledges as a limitation (Section 8, Limitation 5): "experiments involving distribution shifts, and a larger scale of models and datasets would be important to further test these infinite schedules."

Ablation Studies and Robustness Checks

• Warmup duration sweep (0%, 0.5%, 1%, 2% of training iterations): As detailed in Section 6.1.1, warmup duration does not affect final adaptation or forgetting after 50B tokens, though shorter warmups cause transient loss spikes early in training. The paper selects 1% as the default for all subsequent experiments. Figure 3.

• Re-warming magnitude sweep ($\eta_{max} \in \{1.5, 3.0, 6.0\} \times 10^{-4}$): Higher re-warming increases both adaptation (lower final $\mathcal{D}_1$ loss) and forgetting (higher final $\mathcal{D}_0$ loss) across both distribution shifts. The effect is monotonic and more pronounced under the strong shift. The paper selects the "same" $\eta_{max}$ ($3 \cdot 10^{-4}$) as the default for most experiments. Figure 4.

• Replay fraction sweep (weak shift: 0.5%, 1%, 5%, 10%, 50%; strong shift: 1%, 5%, 10%, 25%, 50%): Replay reduces forgetting at all tested fractions, with diminishing returns beyond ~5% (weak) and ~25% (strong). Adaptation to $\mathcal{D}_1$ is minimally affected up to 10% replay for weak shift and 25% for strong shift, then degrades noticeably at 50%. The interaction with distribution shift severity is non-trivial: stronger shifts require disproportionately more replay (25% vs. 5%) to achieve comparable forgetting mitigation. Table 2, Figure 5.

• Model scale comparison (405M vs. 10B): Scaling model size by approximately 25× (405M → 10B) moderately reduces absolute forgetting (0.27 nats → 0.23 nats without replay) and does not diminish the effectiveness of replay (forgetting reduction of 0.21 vs. 0.19 nats with 5% replay). The performance gap between continual pre-training with replay and the union baseline is identical at both scales (0.02 nats AVG loss). Figure 7, Table 4.

• Constant learning rate baselines vs. re-warm + re-decay: The constant $\eta_{min}$ baseline achieves the least forgetting but poorest adaptation. The constant $\eta_{max}$ baseline achieves intermediate forgetting and adaptation, underperforming re-warm + re-decay on both metrics for the weak shift and on adaptation for the strong shift. This ablation establishes that both re-warming and re-decaying are individually necessary components. Figure 4.

• Re-warming on the same data (no distribution shift): When the model is re-warmed on Pile after pre-training on Pile (zero distribution shift), the Pile validation loss still increases—0.1 nats peak increase for $\eta_{max} = 3 \cdot 10^{-4}$, 0.2 nats for $\eta_{max} = 6 \cdot 10^{-4}$—and does not fully recover within 100B tokens. This isolates re-warming as an independent cause of forgetting, separable from distribution shift. Figure 8.

• Infinite schedule cooldown function comparison (cosine vs. inverse square root): Both cooldown functions produce nearly identical validation loss trajectories during the constant phase, with curves overlapping after approximately 80B tokens. The cosine cooldown achieves marginally better final validation loss in single-dataset training, but the difference is small (~0.02 nats). This suggests the specific cooldown shape is not critical; what matters is the transition to a constant rate. Figures 9, 10.

• Optimizer state reset: All experiments reset optimizer states between datasets. The validation loss spikes at dataset boundaries in Figure 10 (even for infinite schedules where the learning rate does not change) are attributed to this reset, confirming that optimizer state absence imposes a measurable but transient cost. The paper does not ablate this by comparing with and without state preservation; this is left as an implicit design choice justified by practical deployment constraints.

• Domain-incremental continual pre-training (Appendix A.1): The paper tests consuming SlimPajama domains sequentially (Wikipedia → Book → C4 → StackExchange → GitHub → CommonCrawl → Arxiv) rather than mixed, finding "poor results" that "suggest that general-purpose LLMs should be continually pre-trained on a mixture of domains if possible, not updated per domain." This negative result is reported only briefly in Section 5.2 and Appendix A.1, without detailed quantitative tables in the provided paper excerpt.

• Replay data ordering (sequential vs. shuffled): The paper states in Section 4.2 that "we replay the data in the order it was seen when pretraining on $\mathcal{D}_0$, as we did not observe noticeable differences when reshuffling the replay data in preliminary experiments." This is a brief but important robustness check: the precise ordering of replay data is not a critical hyperparameter.

• Downstream evaluation robustness: Table 5 reports per-task scores for all models, enabling assessment of whether continual pre-training benefits are uniform across tasks or concentrated in specific categories. The pattern is mixed: no single model dominates all tasks, and average scores obscure task-level tradeoffs. For instance, at 10B, the union model wins MMLU by ~9 points while the replay model wins OpenBookQA by ~3 points. This suggests that claims of "matching the re-training baseline" are accurate at the aggregate level but mask task-specific variance.

Critical Assessment

Does the paper demonstrate that the three-technique combination matches re-training performance? Yes, but with important scope limitations. The claim is primarily supported by Tables 3 and 4, which show average validation loss differences of 0.00–0.02 nats between the best continual pre-training configuration and the union-trained baseline. The 5% replay model at 405M achieves AVG = 2.37 vs. 2.35 for the union baseline (Table 3, weak shift), and the 25% replay model achieves AVG = 1.75 vs. 1.75 for the union baseline (Table 3, strong shift)—an exact match. At 10B, the 5% replay model achieves AVG = 1.89 vs. 1.87 for the union baseline (Table 4). These gaps are extremely small and likely within the noise floor of large-batch LLM training, though the paper provides no formal statistical test to confirm this.

However, the claim is narrower than it might appear. The "re-training baseline" is specifically a model trained from random initialization on $\mathcal{D}_0 \cup \mathcal{D}_1$ using the same architecture, tokenizer, batch size, and training duration. It is not a comparison to the best possible model one could train with that compute budget—the paper explicitly does not tune the union baseline's hyperparameters differently from the continual pre-training runs. It is also not a comparison to a model that might allocate compute differently across datasets (e.g., a union model that samples more heavily from the higher-quality dataset). The baseline is fair and controlled, but it is a specific instantiation of "re-training," not a fully optimized one.

Does the paper demonstrate that the techniques work across different distribution shifts? Yes, this is one of the paper's strengths. The weak shift (English → English, with overlapping domains) and strong shift (English → German, different language) produce qualitatively different patterns: the strong shift causes substantially more forgetting (1.39 vs. 0.27 nats without replay, Table 3) and requires proportionally more replay (25% vs. 5%) to achieve parity with the union baseline. The fact that the same three techniques work in both regimes—with only the replay fraction needing adjustment—is evidence of robustness. However, both shifts share the property that the model already possesses knowledge relevant to $\mathcal{D}_1$: English pre-training provides transferable linguistic knowledge for German (as evidenced by the German-trained continual models outperforming the German-from-scratch baseline), and SlimPajama shares domains with Pile. A truly "out-of-distribution" shift—to a domain completely absent from pre-training, like protein sequences or mathematical proofs—is not tested.

Does the paper demonstrate that the findings hold at scale (10B)? Partially. The 10B experiments only cover the weak shift (Pile → SlimPajama) and only test two continual pre-training configurations (0% and 5% replay). The strong shift, the replay fraction sweep, and the infinite schedule experiments are all restricted to 405M. The paper's claim that results "suggest that this may also be the case for models with an order of magnitude more parameters (e.g. for 100B+ parameters)" (Section 6.4.2) is extrapolation beyond the tested range. The 10B results are encouraging—the gap to the union baseline is unchanged from 405M (0.02 nats), and larger models appear to forget slightly less—but the evidence for scalability is suggestive rather than conclusive. A 100B-scale experiment would be needed to confirm the trend, and the paper does not provide it.

Does the paper demonstrate meaningful downstream benchmark parity? The evidence is mixed and weaker than the validation loss evidence. Table 5 shows that at 405M, the 5% replay model (35.14% average) actually outperforms the union baseline (34.30%) on average English benchmark accuracy, while at 10B, the replay model (47.68%) slightly underperforms the union baseline (48.00%). The paper appropriately notes that differences are small and potentially attributable to noise, but the 10B MMLU discrepancy (28.79% for replay vs. 37.78% for union, a 9-point gap) is large enough to warrant concern. If this gap is real rather than noise, it would undermine the claim of "matching" the re-training baseline on practically meaningful metrics, even if validation loss parity is achieved. The paper's citation of Glorioso et al. (2024) as showing no MMLU degradation in a similar setup provides external validation but is not part of the paper's own experiments.

A broader concern: the paper's evaluation benchmarks are entirely English-centric (except for the German translations, which mostly show near-random performance). For the strong shift (English → German), the primary demonstration of retaining English capability is the English benchmark average (33.95% for Pile-only, dropping to 29.20% without replay, recovering to 32.48% with 25% replay—Table 3). This is a reasonable but narrow measure of "not forgetting." A model might preserve factual knowledge while losing more subtle capabilities (stylistic range, reasoning depth, cultural knowledge) that these benchmarks do not capture. This is a fundamental limitation of evaluation at the 405M scale—base models of this size perform poorly enough on most benchmarks that floor effects may mask real degradation.

Does the paper demonstrate that infinite learning rate schedules are a viable alternative? The evidence is preliminary. The infinite schedules match cosine decay on single-dataset pre-training (Figure 9) and on three IID dataset splits (Figure 10), both at 405M scale and without distribution shift. These are necessary but not sufficient conditions. The paper's central motivation for infinite schedules is to avoid re-warming-induced forgetting, yet this benefit is only tested in the no-shift setting where forgetting is minimal anyway. The real test—applying infinite schedules to the strong distribution shift (Pile → German) and comparing forgetting with and without replay—is not conducted. The paper acknowledges this candidly in the Limitations (Section 8, point 5), and the infinite schedule results should be interpreted as a proof-of-concept rather than a validated alternative to the re-warming + replay recipe.

Missing experiments that would strengthen the paper:

• 100B+ scale experiments: The paper claims scalability but stops at 10B. Given that modern LLMs are 70B–400B+ parameters, the extrapolation is untested. Training costs at 100B scale are enormous, and the paper's 10B results required 276 nodes × 24 GPUs—scaling another order of magnitude may reveal qualitatively different behaviors (e.g., optimization instability at constant learning rates, memory constraints on replay buffer management).

• Infinite schedules with distribution shift: As noted above, this is the critical missing experiment. The paper introduces infinite schedules specifically to address re-warming-induced forgetting, which is most severe under distribution shift (Section 7.1), yet never tests them in that setting.

• More than two datasets ($N > 2$) with distribution shift: The paper acknowledges (Limitation 3) that it "primarily study[s] the transition between two subsequent tasks." The 3-dataset experiment (Section 7.4) uses IID splits with no shift. How do the techniques scale to, say, 5 or 10 sequential datasets of 100B tokens each, with varying degrees of shift? Does replay from $\mathcal{D}_0$ remain sufficient when the model is on $\mathcal{D}_3$, or does one need a growing replay buffer spanning all prior datasets?

• Ablation of optimizer state reset: All experiments reset optimizer states between datasets. A comparison with and without state preservation would quantify the cost of this practical constraint and might inform whether practitioners should invest in storing and distributing optimizer states alongside model weights.

• Replay sample selection strategies: The paper uses simple sequential replay as a "lower bound" but never tests whether more sophisticated selection (e.g., prioritizing high-loss or high-gradient examples from $\mathcal{D}_0$) could reduce the required replay fraction. This is acknowledged as future work.

• Tokenizer adaptation for the strong shift: The German experiments use an English-trained tokenizer, which the paper acknowledges is suboptimal ("the tokenizer may become a key bottleneck to performance"). A comparison with a tokenizer adapted to German (e.g., by extending the vocabulary with high-frequency German subwords) would separate the effects of tokenizer mismatch from genuine distribution shift.

• Statistical significance: No confidence intervals, error bars, or significance tests are reported for any result. The large batch size argument (low gradient variance) addresses training stability but does not account for variability from random initialization, data ordering, or evaluation noise—all of which could produce the 1–2 percentage point differences seen in some benchmark results. Running even 2–3 seeds on a subset of experiments would substantially strengthen claims of parity.

Where the claims hold conditionally:

• "Matches re-training": Holds for validation loss (within ~0.02 nats) across weak and strong shifts and two model sizes. Holds for average downstream benchmark accuracy at 405M and approximately holds at 10B (within 0.32 percentage points), though the 10B MMLU gap is a notable outlier. Does not hold if "matching" is interpreted as dominating on all individual tasks—task-level tradeoffs exist.

• "Works across distribution shifts": Holds for the two shifts tested, but the replay fraction must be tuned per shift (5% vs. 25%). No evidence for shifts beyond English-language data with an English-trained tokenizer.

• "Scales to larger models": Weakly supported. 10B results are consistent with 405M, but the tested range (405M → 10B) represents a 25× scale increase; extrapolating another 10× to 100B is speculative.

• "Infinite schedules avoid re-warming pathologies": Supported for the no-shift case, untested under distribution shift. The conceptual argument is sound, but the empirical validation is incomplete.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. The paper's central claim—that continual pre-training with re-warming, re-decaying, and replay matches the re-training baseline while using "substantially less compute"—assumes that a pre-trained checkpoint on $\mathcal{D}_0$ is already available and that its training cost is a sunk cost not attributed to the continual pre-training process. Figure 1 makes this explicit: the compute comparison shows "PT on Pile" (the $\mathcal{D}_0$ pre-training phase) as a separate, already-completed step, and the baseline it compares against re-trains from scratch on $\mathcal{D}_0 \cup \mathcal{D}_1$. The paper states in Section 1: "We consider models trained on the union of existing datasets as baselines whose performance we seek to match using a combination of continual learning strategies at scale."

The analogy to the test-time compute reference paper is instructive but imperfect in a different way. Here the unaccounted cost is not sample-based difficulty estimation but the assumption that a specific starting checkpoint exists and is optimized for the practitioner's eventual continual pre-training needs. The paper's framework is designed for the situation where "a model is available (e.g. via hugging face or pre-trained in-house) that has been pre-trained... using a linear warmup and cosine decay LR schedule" (Section 5.2). But the paper does not account for the possibility that the available checkpoint may have been trained on a different data mixture, with a different learning rate schedule, to a different convergence point, or using different hyperparameters than what would be optimal for the eventual continual pre-training task.

The consequence. In practice, a practitioner does not simply "have" a pre-trained checkpoint that is ideally suited for the specific continual pre-training task at hand. If the pre-existing checkpoint was trained with a cosine schedule that decayed to $\eta_{min}$, the re-warming-plus-replay recipe applies directly. But if the checkpoint was stopped early (before full convergence), was trained with a different schedule (e.g., constant learning rate, inverse square root decay), or was trained on a different data distribution than the practitioner's $\mathcal{D}_0$ (e.g., the practitioner wants to continually pre-train a LLaMA model on a new dataset, but LLaMA was trained on a private data mixture whose replay data is unavailable), the paper's recipe may not transfer. Section 2's "Rules of thumb" explicitly caveat with "If the learning rate was cosine-decayed from a large value $\eta_{max}$ to a small value $\eta_{min}$ during pre-training," acknowledging this conditionality but not quantifying its impact.

A related unaccounted cost: the practitioner must run a sweep to determine the appropriate replay fraction $r$ for their distribution shift and model. Section 2's "Rules of thumb" recommend: "We recommend experimenting with different replay fractions since relative differences between them appear very early during training. For example, one may experiment with different replay fractions for a limited token budget, using evaluations relevant to their use case, to find a sweet spot between adapting to the new data and mitigating performance loss due to the distribution shift." This sweep—training multiple models with different replay fractions, each for enough tokens to observe relative differences (the paper's ablation in Figure 5 shows clear separation by ~20B tokens for the weak shift and ~10B tokens for the strong shift)—is itself a form of "difficulty estimation" cost not included in the headline compute savings. At the 10B scale, even 20B tokens of experimentation represents a non-trivial compute investment (approximately 7% of the full 300B-token $\mathcal{D}_1$ budget).

What evidence exists in the paper. The paper's own Figure 1 illustrates the cost accounting: the "PT on Pile" bar is positioned to the left of the comparison, implying it is a prior investment. The headline text states continual pre-training "require[s] substantially less compute" but the compute savings are never quantified as a percentage reduction relative to full re-training in a way that includes the $\mathcal{D}_0$ pre-training cost. For the weak shift case at 405M: the union baseline trains on 600B tokens total (300B Pile + 300B SlimPajama). The 5% replay model trains on 300B Pile (sunk) + 300B tokens on $\mathcal{D}_1$ (285B unique SlimPajama + 15B replay Pile). If the $\mathcal{D}_0$ pre-training is counted, both models process 600B tokens; the saving is only realized if the $\mathcal{D}_0$ model already exists for other purposes. If the practitioner must first pre-train on Pile specifically to enable continual pre-training on SlimPajama, there is no compute saving at all relative to union training—both paths process 600B tokens.

Table 2 and Figure 5 show that replay fraction sweeps require meaningful training to evaluate: by 50B tokens, the 1%, 5%, and 10% replay curves for the weak shift are clearly separated from 0% replay on Pile validation loss, but determining the optimal tradeoff point requires running each replay fraction for substantially longer—potentially to convergence—to assess the final adaptation-forgetting balance.

Mitigation status. The paper partially acknowledges this limitation in its framing (Section 2: "We note that this setting assumes that a pre-trained model is available"), but does not account for the $\mathcal{D}_0$ pre-training cost in its headline efficiency comparisons, nor does it quantify the cost of replay fraction sweeps. The "Rules of thumb" guide for replay selection is qualitative and depends on the practitioner having access to "evaluations relevant to their use case" and enough compute to run exploratory experiments. No method is proposed for predicting the optimal replay fraction from dataset characteristics or a small pilot study, and no meta-learning or difficulty-prediction model is developed to reduce this overhead. This is a pragmatic gap: the paper demonstrates that the sweet spot exists, but finding it for a new dataset pair still requires a non-trivial search, the cost of which is externalized from the paper's compute accounting.

---

Limitation 2: The 10B-Parameter Scaling Experiments Only Cover the Weak Distribution Shift

The assumption or constraint. The paper's strongest "at scale" results—the 10B parameter experiments in Section 6.4—are exclusively conducted under the weak distribution shift (Pile → SlimPajama). The strong distribution shift (Pile → German), the replay fraction sweep, the $\eta_{max}$ variation study, the warmup duration sweep, and the infinite learning rate schedule experiments are all conducted at the 405M scale only. The paper's central claim about matching the re-training baseline "across two distribution shifts (weak & strong) and two decoder-only transformer LLM scales (405M & 10B)" (Section 9) is slightly misleading in its conjunctive structure: the strong shift was tested at 405M but not at 10B, and the 10B scale was tested with the weak shift but not the strong shift.

The consequence. The extrapolation from 405M to larger scales for the strong distribution shift is untested. The strong shift produces qualitatively different dynamics than the weak shift at 405M: catastrophic forgetting on Pile (1.39 nat increase in validation loss without replay, Table 3), the need for substantially more replay (25% vs. 5%), and a positive transfer effect where English pre-training improves German adaptation (Table 3: Pile→German achieves lower German loss than German-from-scratch). How these dynamics change at 10B scale—and particularly whether the 25% replay fraction remains sufficient or the positive transfer strengthens or weakens—is unknown.

This matters practically because strong distribution shifts (new languages, new domains, new modalities) are precisely the cases where continual pre-training is most valuable: re-training from scratch on a new language from random initialization is enormously expensive and may underperform compared to starting from a multilingual or English-pretrained base. If the strong-shift recipe breaks at the 10B+ scales where it would be most economically impactful, the paper's most important practical claim is undermined. The paper itself notes that "as the domain strays farther and farther away from the tokenizer's training corpus, the tokenizer may become a key bottleneck to performance" (Section 5.2)—a factor that could interact with model scale in non-obvious ways (larger models may be better at overcoming tokenizer inefficiency, or tokenizer mismatch may impose a hard ceiling on transfer that scale cannot overcome).

What evidence exists in the paper. The 10B experiments are restricted to the weak shift (Figure 7, Table 4, Table 5 top section). The strong shift results appear only in the 405M sections (Figure 6 bottom, Table 3 bottom, Figure 5 bottom). Section 6.4 explicitly frames the scaling experiments as being in the "two dataset weak shift (Pile → SlimPajama)" setting. The paper's conclusion generalizes from these partial experiments to the claim that "these results show that the benefits of continual pre-training hold at the 10B parameter scale, suggesting that this may also be the case for models with an order of magnitude more parameters (e.g. for 100B+ parameters)" (Section 6.4.2), without conditioning on distribution shift type.

Mitigation status. The paper does not acknowledge this as a limitation in Section 8 (Limitations), which focuses on other issues (only two model scales studied, tokenizer limitations, number of sequential tasks, lack of multiple seeds, infinite schedule experiments restricted to 405M and no-shift). The Limitation that comes closest is point 1: "we only studied two model sizes (405M and 10B)" and "we could not extend the study to another order of magnitude due to computational limitations (e.g., 100B parameter scale)." But this framing omits the distribution-shift dimension: the issue is not merely that the 10B scale was only tested at all, but that it was only tested under the easier of the two distribution shifts, leaving the harder and more economically consequential case untested at scale. The computational limitation is understandable given the paper's resource constraints, but the generalisation gap should be explicitly bounded rather than implicitly elided by the conjunction "across two distribution shifts... and two model scales" in the conclusion.

---

Limitation 3: The Union-Trained Baseline Is a Necessary but Insufficient Yardstick for Practical Deployment

The assumption or constraint. The paper's evaluation framework treats the model trained from random initialization on $\mathcal{D}_0 \cup \mathcal{D}_1$ as the gold-standard baseline whose performance must be matched. This is a scientifically clean baseline: it controls for total data seen, total compute expended, architecture, and training hyperparameters. But as a proxy for what a practitioner actually cares about, it has two embedded assumptions that weaken its practical relevance.

First, the paper does not optimize the union baseline independently. The baseline uses the same learning rate schedule (linear warmup + cosine decay fitted to the combined token budget), same $\eta_{max}$ ($3 \cdot 10^{-4}$), same batch size, and same data mixing proportions (equal sampling from both datasets) as the continual pre-training runs. It is entirely possible—and indeed likely given the Chinchilla scaling laws and standard pre-training practice—that a practitioner allocating 600B tokens to a joint training run could achieve better performance by adjusting the learning rate schedule, the data mixing ratio, or the token allocation between datasets. The paper's claim of "matching the re-training baseline" therefore means matching this specific re-training configuration, not matching the best possible re-trained model one could produce with that compute budget.

Second, the downstream evaluation of "matching" relies heavily on aggregate metrics (average validation loss, average English benchmark accuracy) that can mask per-task degradation. Section 5's Experimental Analysis already documented the 9-percentage-point MMLU gap at 10B (28.79% for 5% replay vs. 37.78% for union, Table 5) and noted the paper's response: "we suspect it is due to the limited amount of training data used in our study" and a citation to Glorioso et al. (2024) showing no MMLU degradation at larger scale. But this defense is external to the paper's own experiments and essentially concedes that the paper's own 10B setup fails to match the union baseline on at least one practically important benchmark.

The consequence. A practitioner reading this paper might conclude from the headline claims that continual pre-training with 5% replay is a drop-in replacement for full re-training, with no meaningful performance sacrifice. The MMLU gap at 10B—and the unknown gaps on other high-stakes capabilities that the paper's evaluation suite does not measure (reasoning, safety, instruction-following, code generation)—suggests this conclusion is premature. If a deployed model exhibits subtle degradation on specific capabilities that aggregate metrics obscure, the cost of that degradation (in user trust, downstream task performance, or safety) may far outweigh the compute savings from avoiding re-training.

Furthermore, if the union baseline can be improved by re-optimizing its training recipe (e.g., by using a Chinchilla-optimal token allocation that spends more of the 600B budget on the higher-quality SlimPajama data rather than equal sampling), the paper's continual pre-training approach might fall short of a practitioner's actual alternative—not the paper's baseline, but a better-tuned version of it. The paper's compute-equivalent replay framework (Section 4.2) forces a token tradeoff (replay tokens come at the expense of new $\mathcal{D}_1$ tokens), but the union baseline has no such constraint: it can allocate tokens freely between datasets. This structural asymmetry means continual pre-training is competing against a baseline that can be further optimized in ways the continual approach cannot easily replicate.

What evidence exists in the paper. Table 5 at 10B shows the MMLU gap. The paper's acknowledgment: "While this degradation in MMLU performance between both models could be cause for concern, we suspect it is due to the limited amount of training data used in our study" is in Section 6.4.2. The paper does not report per-task statistical significance, confidence intervals, or task-level ablation to determine whether the MMLU gap is systematic or stochastic. No union baseline hyperparameter optimization is attempted or discussed—the baseline is simply the default training recipe applied to the combined dataset.

Mitigation status. The paper partially addresses this by citing external evidence (Glorioso et al., 2024; DeepSeek-AI et al., 2024) showing that similar techniques work at larger scale without MMLU degradation. Section 2's "Recent works employing our techniques" box provides this external validation. But from an epistemic standpoint, this is an appeal to external results rather than internal evidence. The paper's own experiments at 10B show a gap on a high-profile benchmark, and the paper's explanation (insufficient data) is a hypothesis, not a demonstrated resolution. A more cautious claim would acknowledge that downstream parity is benchmark-dependent and that certain capabilities (particularly those tested by knowledge-intensive, multi-subject benchmarks like MMLU) may be more vulnerable to continual pre-training than aggregate metrics suggest, at least at the data scales tested here.

---

Limitation 4: Infinite Learning Rate Schedules Under Distribution Shift Are Entirely Untested

The assumption or constraint. Section 7 introduces infinite learning rate schedules as a "promising alternative" to cosine decay that "help circumvent forgetting induced by LR re-warming" and are "not bound to a fixed token budget." The paper demonstrates their viability in two controlled settings: single-dataset pre-training on 300B SlimPajama (Section 7.3, Figure 9) and three-dataset continual pre-training on IID 100B splits of SlimPajama with no distribution shift (Section 7.4, Figure 10). Both experiments are at 405M parameter scale only.

The paper explicitly states in Section 8, Limitation 5: "our experiments on infinite learning rate schedules are limited to 405M scale with no distribution shift." It further acknowledges: "it is very possible that over enough tokens, the infinite schedules may end up being suboptimal due to only having a single phase of warmup and cooldown, as the learning on all subsequent datasets may just be equivalent to using a constant learning rate, which proved to be suboptimal (see Fig. 4)." And crucially: "experiments involving distribution shifts, and a larger scale of models and datasets would be important to further test these infinite schedules."

The consequence. This is the most significant open question in the paper, because infinite schedules are presented as the architectural solution to the re-warming pathology that the paper itself diagnosed (Section 7.1). If infinite schedules work under distribution shift, they could eliminate the need for both re-warming and the replay that re-warming necessitates—at least for the re-warming-induced component of forgetting. If they fail under distribution shift, the three-technique combination (re-warm, re-decay, replay) remains the only validated recipe, with all its associated overhead (replay buffer management, per-shift replay fraction tuning, optimizer state resets).

The failure modes under distribution shift are not difficult to anticipate. The constant phase's learning rate $\eta_{const}$ is a compromise: high enough to enable learning on new data, low enough to avoid destabilizing existing knowledge without replay. But Section 6.1.2 (Figure 4) demonstrated that under distribution shift, even a decaying schedule starting from $\eta_{max} = 3 \cdot 10^{-4}$ causes substantial forgetting without replay. An infinite schedule's constant rate would need to be lower than this peak to avoid re-warming-induced forgetting, but if it is too low, it may fail to provide sufficient plasticity for strong distribution shifts where the model must learn fundamentally new patterns (e.g., German grammar and vocabulary). The strong-shift experiments at 405M showed that models with re-warming to $3 \cdot 10^{-4}$ and replay adapted well to German (loss = 1.16 with 25% replay, Table 3). Would a constant rate of, say, $1 \cdot 10^{-4}$ (chosen to minimize forgetting without replay) enable comparable German adaptation? The paper provides no evidence either way.

Even without distribution shift, the long-term behavior is untested. Figure 10 shows infinite schedules performing well across three 100B-token IID splits (300B tokens total in the constant phase). What happens after 1T tokens at constant learning rate? Does the model's loss plateau at a higher level than a model that underwent periodic decay? Does training stability degrade without the regularizing effect of learning rate reduction? The paper's own evidence (Figure 4) shows that constant $\eta_{max}$ training finishes with worse loss than decaying schedules, and the constant phase of infinite schedules is—by design—a constant learning rate, albeit at a lower value than $\eta_{max}$.

What evidence exists in the paper. The infinite schedule experiments are Figures 9 and 10, both 405M, no distribution shift. Section 7.4's Figure 10 shows the clearest relevant evidence: when transitioning between IID splits of SlimPajama (zero distribution shift), infinite schedules show "negligible forgetting across dataset boundaries... the infinite schedules models immediately recover from [optimizer state reset-induced spikes]." The performance of the "decayed" checkpoints (those that underwent Phase 4 annealing) is comparable across all three schedules (cosine baseline, Cosine Inf, InvSqrt Inf), within approximately 0.05 nats of each other. These are encouraging preliminary results but do not address the distribution shift case.

Mitigation status. The paper is unusually transparent about this limitation, explicitly calling it out in Section 8 and framing it as a direction for future work. Section 9's conclusion states: "In follow-up work, it will be important to further investigate infinite learning rate schedules." This honesty is a strength of the paper's scientific communication, but it also means that the infinite schedule contribution—one of the paper's four main contributions listed in Section 1—is currently a promissory note rather than a validated alternative. A practitioner reading this paper today should treat the re-warming + re-decay + replay combination as the only empirically validated recipe for continual pre-training under distribution shift; infinite schedules remain an interesting research direction whose practical viability is unproven.

---

Limitation 5: No Statistical Quantification of Variability Across Training Runs

The assumption or constraint. All experiments in the paper are single training runs. No results are averaged over multiple random seeds, no confidence intervals are reported for validation loss or benchmark scores, and no statistical hypothesis tests are conducted to assess whether observed differences between models are significant. The paper acknowledges this as Limitation 4 in Section 8: "we did not run experiments over multiple seeds due to high computational cost, meaning that there is likely a stochastic element to some results." The paper's defense is that "our LLMs are trained with a large batch size (2M+ tokens) and, thus, there is little variance in the gradient estimates" and "the samples from each dataset are processed in the same order in all cases, we believe that our results should be relatively stable to changes in random initialization dictated by the seed."

The consequence. The paper's central empirical claim—that continual pre-training with replay matches the re-training baseline—relies on comparisons where the measured performance gaps are extremely small. The final average validation loss gap between the 5% replay model and the union baseline at 10B is 0.02 nats (1.89 vs. 1.87, Table 4). At 405M under the weak shift, the gap is also 0.02 nats (2.37 vs. 2.35, Table 3). The strong shift gap at 405M is 0.00 nats (1.75 vs. 1.75, Table 3)—a numerical tie. On downstream benchmarks, the 10B 5% replay model's average accuracy differs from the union baseline by 0.32 percentage points (47.68% vs. 48.00%, Table 5), and the 405M 5% replay model exceeds the union baseline by 0.84 percentage points (35.14% vs. 34.30%).

Without any estimate of the variance in these measurements, it is impossible to determine whether:

• The 0.02 nat gap at 10B is a real but small performance deficit, or could easily be reversed (or widened) by a different random initialization or data ordering.
• The 0.00 nat tie under the strong shift is genuine equivalence or a coincidence of this particular training run.
• The per-task benchmark differences (e.g., the 9-point MMLU gap at 10B, or the 3-point OpenBookQA advantage for the replay model) are systematic or noise-driven.
• The apparent pattern that larger models forget less (0.23 vs. 0.27 nats forgetting at 10B vs. 405M, Section 6.4.1) is a reliable trend or a single-run artifact.

The paper's defense—that large batch sizes reduce gradient variance and that fixed data ordering reduces variability—addresses only part of the problem. Large batch sizes reduce the variance of each gradient estimate, but do not eliminate the sensitivity to initial conditions: two models with different random initializations trained with the same large-batch schedule can converge to different local minima with different generalization properties. Fixed data ordering removes one source of variability (data sampling order) but does not address initialization variability or the inherent stochasticity of the training process (dropout, mixed-precision rounding, non-deterministic GPU operations). The batch size of 2.26M tokens is large, but not unprecedented; comparable-scale LLM training papers routinely report multiple seeds or confidence intervals for their headline results, or at minimum acknowledge the limitation explicitly rather than arguing it away.

What evidence exists in the paper. Section 8, Limitation 4 provides the acknowledgment. Table 2 and Table 3 note that "the standard error for these measurements was computed but is not reported as it was < 0.001 for all models." This is a within-run standard error of the final loss estimate (variance over the last 100 training iterations), not a between-run estimate of training variability. It tells us that the model's validation loss is stable at the end of training (the last 100 iterations are consistent with each other), but nothing about whether an independent training run with a different seed would achieve the same final loss.

Mitigation status. Not addressed. The paper acknowledges the limitation but provides no quantitative mitigation (e.g., a pilot multi-seed experiment on a smaller configuration, or estimation of variability from checkpoint differences during training). The claim that results "should be relatively stable" is a judgement, not an empirical finding. For a paper whose core message is that simple techniques enable continual pre-training to match full re-training, the absence of any statistical framework for evaluating what "match" means weakens the precision and force of that claim. A more conservative framing—"continual pre-training achieves final validation loss within the measurement noise of the re-training baseline"—would be more appropriate given the evidence presented, but would also be a weaker claim than the paper's actual assertion of matching performance.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the LLM pre-training paradigm from a single-shot, fixed-budget operation to an incremental, open-ended process—not by proposing a fundamentally new algorithm, but by providing the first rigorous, controlled empirical demonstration that the problem of updating LLMs on new data is already solvable with techniques the community already possesses. The shift is primarily practical and methodological rather than algorithmic: the paper's core message is that the barrier to continual pre-training has been not a lack of sophisticated methods, but a lack of systematic validation of simple ones at realistic scale.

The conceptual reframing operates on two levels. First, the paper recasts LLM updates as a natural extension of pre-training rather than a separate "continual learning" problem requiring specialized solutions. By showing that learning rate re-warming and re-decaying—essentially restarting the cosine schedule on the new data—combined with modest replay of old data suffices to match full re-training across distribution shifts and model scales, the paper dissolves the perceived boundary between "training" and "updating." The implication is that the optimization machinery that produced the original model (cosine schedules, AdamW, large-batch training) is sufficient to update it, provided it is applied correctly. This undercuts the motivation for many complex continual learning interventions (elastic weight consolidation, progressive networks, dynamic architectures) in the LLM regime: if replay and schedule manipulation already achieve parity with re-training, more complex methods must demonstrate benefits beyond parity to justify their overhead.

Second, and more subtly, the paper reframes the stability-plasticity tradeoff as being partially separable into schedule-induced and distribution-shift-induced components. The diagnostic experiment in Section 7.1—re-warming on the same data causes measurable, lasting forgetting even without distribution shift—is the key finding that enables this reframing. Before this paper, forgetting in continual pre-training was attributed holistically to "the model learning new things and overwriting old things." The paper shows that a non-trivial fraction of forgetting is purely an artifact of the learning rate schedule—the model's existing solution is destabilized by large weight updates, regardless of what those updates are about. This reattribution matters because schedule-induced forgetting and distribution-shift-induced forgetting have different solutions: the former can be addressed by schedule design (infinite schedules), while the latter likely requires data mixing (replay). By separating the two, the paper opens up a more precise research agenda where these components are studied and mitigated independently.

The paper also resolves a latent contradiction in prior work between the continual learning literature (which often finds that pre-training and scale reduce forgetting—Cossu et al., 2022; Ramasesh et al., 2022; Mehta et al., 2023) and the LLM community's default assumption that re-training is necessary. The contradiction was not explicit—the two communities largely did not engage—but it was practically significant: if pre-training helps continual learning, why does the LLM community retrain from scratch? The paper's answer is that the problem was never that LLMs can't learn continually, but that the standard training recipe (cosine decay to near-zero LR) actively prevents it. The fix is not to make LLMs better continual learners through architectural changes, but to modify the training recipe so the model can learn when new data arrives. This is a much simpler and more actionable insight than the CL literature's typical prescriptions.

Some research directions become less attractive as a result. Methods that assume previous data is unavailable (Ke et al., 2022; parameter-importance masking approaches) lose motivation when the paper shows that storing and replaying 5% of previous data is sufficient and cheap. Complex replay sample selection strategies (prioritizing high-loss or high-gradient examples, generative replay) become harder to justify when random sequential replay already achieves parity. Architectural modifications that freeze or expand parts of the model (Wu et al., 2024; LLaMA Pro) face a higher bar: if full-model fine-tuning with replay works, why pay the inference and implementation cost of a growing architecture? The paper does not definitively close these avenues—more sophisticated replay might reduce the needed fraction below 5%, and architectural freezing might eliminate the need for replay entirely—but it shifts the burden of proof onto them.

Conversely, several directions become more attractive. Learning rate schedule design for multi-phase training—previously a niche concern—becomes a first-class research problem. The paper's infinite schedules are a proof-of-concept that schedules designed for continual learning can avoid the re-warming pathology, but they are clearly preliminary. Verifier and replay quality interactions become interesting: if distribution shift is large enough that replay is needed, how should the replay buffer be constructed when the original training data is unavailable (e.g., for open-weight models)? Tokenizer adaptation for distribution shift becomes salient given the paper's acknowledgment that tokenizer mismatch may be the bottleneck for strong shifts like English-to-German. And scaling law analysis for continual pre-training—predicting the optimal replay fraction and schedule parameters as a function of dataset similarity, model size, and token budget—becomes a clearly defined empirical target.

Follow-Up Research This Work Enables

Scaling laws for optimal replay fraction as a function of distribution shift strength and model size. The paper demonstrates that the optimal replay fraction depends on the distribution shift (5% for weak shift, 25% for strong shift at 405M) and that this fraction may interact with model scale (the 10B experiments only tested 5% replay on the weak shift). A comprehensive scaling study would train 405M, 1B, 3B, and 10B models on a range of controlled distribution shifts—from near-IID (different subsamples of the same dataset) through weak shift (English-to-English with overlapping domains) to strong shift (English-to-different-language, English-to-code, English-to-math) to extreme shift (English-to-proteins, English-to-random token sequences)—while sweeping replay fractions from 0% to 50%. The goal would be to produce a predictive relationship: given a measured distribution divergence between $\mathcal{D}_0$ and $\mathcal{D}_1$ (e.g., n-gram overlap, embedding space distance, or a proxy like validation loss of the $\mathcal{D}_0$ model on $\mathcal{D}_1$), what replay fraction $r$ achieves >95% of the union baseline's average validation loss at minimum compute cost? Such a study would transform replay fraction selection from a hyperparameter sweep (the paper's current "experiment with different replay fractions" guideline) into a principled design choice. The paper's existing replay fraction sweeps (Figure 5, Table 2) provide the template; scaling it to more model sizes and shifts would be a direct extension.

Infinite learning rate schedules under distribution shift with and without replay. The paper identifies this as its most important missing experiment (Section 8, Limitation 5). A strong follow-up would replicate Figure 6 (the final-performance comparison for weak and strong shifts) but replace the cosine re-warming + re-decay schedule with infinite schedules (both Cosine Inf and InvSqrt Inf variants). The critical comparisons are: (a) infinite schedule with no replay versus cosine re-warming with no replay—does the constant learning rate eliminate the re-warming-induced forgetting component, and if so, by how much? (b) infinite schedule with modest replay (5% for weak shift, 25% for strong shift) versus cosine re-warming with the same replay—does the constant schedule match or exceed the cosine schedule's adaptation to $\mathcal{D}_1$? (c) infinite schedule with reduced replay—can the constant rate's inherent stability allow less replay than the cosine schedule requires, since re-warming-induced forgetting is eliminated? If the answer to (c) is yes (e.g., infinite schedule achieves parity with only 10% replay on the strong shift instead of 25%), infinite schedules would be a clear improvement on the state of the art. If the infinite schedule underperforms the cosine schedule on adaptation to $\mathcal{D}_1$ (because the constant rate is too low to drive sufficient plasticity for a strong shift), the result would clarify the fundamental plasticity limits of constant-rate training and motivate hybrid schedules (periodic brief re-warming, or adaptive constant-rate adjustment based on detected shift magnitude). The experiment should be run at both 405M and 10B to assess scale interactions.

Dynamic replay buffer construction for open-weight models with undisclosed training data. The paper's replay mechanism assumes that the exact training data from $\mathcal{D}_0$ is available for replay. This holds for in-house models but fails for the increasingly common case where a practitioner downloads an open-weight model (LLaMA, Mistral, Falcon) whose training data mixture is partially or completely undisclosed. A research direction would develop methods for constructing a synthetic replay buffer when the original data is unavailable. Candidates include: (a) distillation-based replay—use the downloaded model to score candidate web-text documents and select those with low perplexity under the model, on the assumption that low-perplexity text resembles the training distribution; (b) model-inversion replay—generate synthetic text by sampling from the model itself with prompts designed to elicit pre-training-like continuations (e.g., prompts drawn from Common Crawl), creating a replay buffer from the model's own output distribution; (c) proxy-data replay—use a publicly available dataset known to overlap with the model's likely pre-training data (e.g., C4, The Pile, RedPajama for most English LLMs) as an approximate replay buffer. The evaluation would compare these methods against the paper's exact-replay baseline (where possible) and against each other, measuring both forgetting on $\mathcal{D}_0$ evaluation tasks and adaptation to $\mathcal{D}_1$. The paper's own suggestion (Section 9) that "using the available model for synthetic data or distillation may be a promising direction" directly motivates this line of work. A concrete experiment: take a Pile-trained 405M checkpoint, simulate "undisclosed data" by deleting the Pile replay buffer, then apply the three candidate methods above to construct replay buffers, continually pre-train on SlimPajama, and compare forgetting against the 5% exact-replay baseline from Table 3.

Tokenizer adaptation as an orthogonal axis to learning rate and replay strategies. The paper explicitly identifies tokenizer mismatch as a potential bottleneck for strong distribution shifts (Section 5.2, Section 8) and leaves its treatment to future work. A direct follow-up would combine the paper's validated learning rate and replay recipe with tokenizer adaptation for the English-to-German shift. The experiment: take the English-trained BPE tokenizer (trained on the Pile), extend its vocabulary with the top-$k$ most frequent German subwords (identified from the German training corpus) that are not already present, resize the model's embedding matrix and output layer accordingly (with the new embeddings initialized as the weighted average of existing subword embeddings, or from scratch, or via a brief warm-start training phase), and then apply the paper's 25% replay + re-warming recipe for continual pre-training on German. The metric is German validation loss and downstream German benchmark performance (HellaSwag-DE, MMLU-DE) compared to the paper's existing results (Table 3, bottom section) which used the frozen English tokenizer. If tokenizer adaptation closes the gap between the continual pre-training model and a hypothetical German-native-tokenizer model (or substantially improves German benchmark scores above the near-random performance the paper observes), it would validate tokenizer work as a high-return complement to the paper's optimization-focused techniques. If it provides minimal benefit, it would suggest that the existing tokenizer is "good enough" even for strong language shifts and that the paper's observed poor German benchmark performance (Table 11 in the appendix) is due to model capacity (405M), training data scale (200B tokens), or evaluation difficulty, not tokenization.

Compute-optimal allocation of total tokens between $\mathcal{D}_0$ pre-training and $\mathcal{D}_1$ continual pre-training. The paper's framework assumes a fixed pre-training budget on $\mathcal{D}_0$ and a fixed continual pre-training budget on $\mathcal{D}_1$. But a practitioner designing a model lifecycle from scratch faces a joint optimization problem: given a total compute budget, how much should be spent on initial pre-training (to establish strong base capabilities and a good starting point for future updates) and how much should be reserved for future continual pre-training phases (when new and potentially higher-quality data will become available)? This is analogous to the Chinchilla scaling law question (how to allocate compute between model size and data), but translated into the temporal dimension. A scaling study would: fix a total token budget (e.g., 600B tokens), partition it between an initial pre-training phase on $\mathcal{D}_0$ ($T_0$ tokens) and a continual pre-training phase on $\mathcal{D}_1$ ($600B - T_0$ tokens, with optimal replay fraction from $\mathcal{D}_0$), and measure final average validation loss as a function of $T_0$. The hypothesis might be that there is an optimal $T_0$ where the $\mathcal{D}_0$ model is "good enough" to provide strong transfer to $\mathcal{D}_1$ without overspending tokens that could have been used on the higher-quality or more relevant $\mathcal{D}_1$ data. The paper's existing results provide partial evidence: the 10B model trained on 300B Pile then 300B SlimPajama achieves AVG = 1.89, while the 600B Pile ∪ SlimPajama baseline achieves 1.87. What if the split were 150B Pile + 450B SlimPajama (with appropriate replay)? The paper's positive transfer observations (Pile→German models adapt better to German than German-from-scratch, Table 3) and the finding that at 10B scale the union baseline benefits from positive transfer between datasets (Table 4) suggest that less initial pre-training might be optimal if $\mathcal{D}_1$ is the primary deployment target.

Continual pre-training as a function of the number of sequential dataset transitions ($N$). The paper primarily studies $N=2$ (two datasets in sequence) with a brief $N=3$ IID experiment for infinite schedules (Section 7.4). A critical stress test is: how does the gap between continual pre-training and the union baseline scale as $N$ grows from 2 to, say, 10? Consider a sequence of 10 datasets of 30B tokens each (totaling 300B), each drawn from a slightly shifted data distribution (e.g., successive years of Common Crawl from 2015 to 2024). Does the replay fraction needed per transition remain constant, or do later transitions suffer from "replay debt" where earlier datasets are underrepresented in the replay buffer (since 5% replay on dataset $i$ only replays $\mathcal{D}_{i-1}$, not $\mathcal{D}_0, \dots, \mathcal{D}_{i-2}$)? Does catastrophic forgetting accumulate across multiple transitions despite per-transition replay? The experiment would compare: (a) sequential training with 5% replay of the immediately previous dataset at each transition; (b) sequential training with a growing replay buffer that samples from all previous datasets proportionally to their size; (c) sequential training with a fixed-size replay buffer that uses reservoir sampling to maintain a balanced representation across all prior datasets; and (d) the union baseline trained on all 300B tokens jointly. The paper's existing domain-incremental experiment (Appendix A.1, briefly mentioned) suggests that sequential domain updates without replay perform poorly; this extension would determine whether replay solves the $N$-dataset problem or merely defers it.

Practical Applications and Downstream Use Cases

Updating deployed LLMs on newly collected data without full re-training. A model provider (e.g., an organization maintaining an internal LLM for code generation, customer support, or document analysis) scrapes a new 200B-token corpus of higher-quality or more recent data and wishes to update their existing 10B-parameter model. Using the paper's recipe—re-warm the learning rate to the original $\eta_{max}$ ( $3 \cdot 10^{-4}$ ), cosine-decay it over the new data, and mix in 5% replay tokens from the original pre-training corpus—they can achieve final validation loss within ~0.02 nats of what a full re-training from scratch on the combined 500B+ token corpus would achieve (Table 4, 10B results), while processing only the 200B new tokens plus 10B replay tokens instead of 500B+ tokens from scratch. At 10B scale, this represents roughly a 60% reduction in training compute per update cycle (210B tokens processed versus 500B+ for full re-training, though the exact saving depends on how the initial pre-training cost is amortized across updates). The paper's 10B model training configuration (276 nodes × 24 GPUs, Section 5.3) cost approximately 3.5 × 10^20 FLOPs for 300B tokens of pre-training; avoiding even one such re-training run saves millions of GPU-hours. The environmental implications are direct: the Broader Impact Statement frames continual pre-training as "a promising method to significantly reduce the compute associated with updating a model and, hence, the energy required to maintain foundation models." Organizations maintaining models updated quarterly on new web-scrapes would reduce their annual training compute by a factor of 2–3× compared to the re-training-from-scratch approach (three updates using continual pre-training at ~210B tokens each versus three full re-trainings at 500B+ tokens each, assuming the initial model is pre-trained once). The paper's citation of Glorioso et al. (2024) and DeepSeek-AI et al. (2024) as organizations already adopting these techniques at scale validates that this application is not hypothetical.

Adding a new language or domain to an existing LLM without losing English capabilities. An organization with a strong English-language LLM (trained on English web data, code, academic text) wishes to expand it to German, Japanese, or a specialized domain like legal or medical text. The paper's strong-shift results show that with 25% replay of English data, the model can be trained on 200B tokens of German text and achieve average validation loss matching the union-trained baseline (Table 3: 25% replay AVG = 1.75 vs. union baseline AVG = 1.75), while preserving English downstream benchmark accuracy (32.48% for 25% replay vs. 32.43% for union baseline, Table 3). This is substantially cheaper than the alternative—training a multilingual model from scratch on English + German would require processing the full 500B token corpus, with the English portion being a redundant re-learning of capabilities the model already possesses. The result that continual pre-training on German from an English checkpoint actually produces better German validation loss than training on German from scratch (1.16 vs. 1.17, Table 3) is a form of positive transfer that makes continual pre-training a better choice than the German-only baseline for domain expansion, not just an equal one. A concrete scenario: a European AI startup has access to a pre-trained 7B English model but needs to serve German-speaking customers. Rather than training a German model from scratch or paying for a multilingual re-training, they can apply the paper's 25% replay recipe on a 200B-token German corpus. The paper notes (Limitation 2) that tokenizer adaptation is left to future work, so the startup may need to extend the vocabulary if substantial German performance is required, but the core optimization recipe is validated.

Efficient training data curation and model improvement cycles. A research lab periodically improves their data filtering, deduplication, or quality scoring pipeline and wishes to re-train their model on the improved version of the same underlying corpus. This is the weak-shift scenario (Pile → SlimPajama, where SlimPajama is a newer, deduplicated, higher-quality version of essentially the same data distribution). The paper's weak-shift results (5% replay, 300B new tokens) show that continual pre-training can match the union baseline's average validation loss (2.37 vs. 2.35, Table 3) while achieving better performance on the new dataset than the union baseline (SlimPajama loss of 2.51 for replay vs. 2.53 for union). The downstream evaluation confirms this: the 5% replay 10B model scores 47.68% average accuracy, within 0.32 points of the union baseline (48.00%, Table 5), and actually outperforms it on several tasks (ARC-Easy, OpenBookQA, Winogrande). For a lab that iterates on data quality monthly, this means each improvement cycle can be deployed as a continual update rather than a full re-training, reducing the cost of each cycle by approximately 50% (training on 300B new tokens with 5% replay versus 600B tokens from scratch for the union). The finding that replay generalizes well even when the new dataset is simply a better-filtered version of the old one validates a very common industrial use case.

Pre-training curricula and staged training. Some LLM training recipes already use staged training—Gemma Team et al. (2024) describe pre-training on a broad web corpus followed by a decay phase on a smaller, higher-quality curated mixture, and the paper explicitly notes (Section 9) that their strategies "might also be relevant for pre-training curricula such as the ones used by Gemma Team et al. (2024)." The paper's techniques provide a principled optimization framework for such curricula: in the final high-quality phase, re-warm the learning rate to a fraction of the original maximum, use replay of the broad pre-training data to prevent catastrophic forgetting of general capabilities while specializing on the curated data, and decay the learning rate over the curated phase. The infinite schedule variant is particularly well-suited to curricula: the model can be trained on the broad corpus during the constant phase, then the curated data can be introduced as a new "dataset" with its own optional replay fraction, and the final annealing can be applied only after the curated phase completes. The paper's finding (Section 7.1) that re-warming causes forgetting even on the same data suggests that curriculum transitions should avoid large learning rate spikes; the infinite schedule's smooth constant rate across phases directly addresses this. The DeepSeek-AI et al. (2024) adoption described in the paper—"select a non-decayed checkpoint from the initial pre-training phase to ensure a smooth LR transition into continual pre-training"—is exactly this curriculum application.

When to Prefer This Method

The paper does not frame its contributions as a choice between named alternative methods, and the decision logic it provides (Section 2, "Rules of thumb") is a prescriptive guide for applying the proposed techniques rather than a comparative framework against alternatives. The "when to prefer" matrix is therefore less about "continual pre-training vs. method X" and more about the conditions under which the paper's specific recipe (re-warming + re-decay + replay) is expected to work, versus conditions where different approaches or additional techniques should be considered. The paper itself articulates several of these boundary conditions:

• When the incoming dataset is large (hundreds of billions of tokens) and adaptation is important: the paper explicitly distinguishes its setting from fine-tuning and small-scale domain adaptation (Section 1, Section 3.3), noting that re-warming and re-decaying are "necessary to best adapt to the new dataset" at this scale. If the new dataset were only a few billion tokens (typical of domain-adaptive continual pre-training work by Gururangan et al., 2020; Ke et al., 2022), the learning rate decay profile would be different—the model would spend most of training at low learning rates—and the adaptation benefit of re-warming might be less pronounced. The paper's recipe is specifically optimized for the regime where $\text{tokens}(\mathcal{D}_i) \geq 100\text{B}$ (Section 5.2).

• When the previous pre-training used a cosine schedule decaying to a small $\eta_{min}$: the paper's central diagnosis—that standard LLM training leaves the learning rate too low for further learning—depends on this being true. If a model was trained with a constant high learning rate, with an infinite schedule, or was stopped early before significant decay, the re-warming step might be unnecessary or even harmful. The paper's infinite schedule experiments (Section 7.4) show that pre-annealing checkpoints can resume training at constant rate without re-warming, and future work might extend this to models trained with other non-decaying schedules.

• When some replay data from $\mathcal{D}_0$ is available: the paper's replay mechanism requires access to the original training data (or a representative sample). As discussed in the Follow-Up Research section, this condition is not always met for open-weight models with undisclosed training data. In those cases, the paper suggests (Section 9) synthetic data or distillation to construct a proxy replay buffer, but this is a proposed direction, not a validated technique. If no replay data can be constructed, the paper's 0% replay results (Tables 3, 4) show that re-warming alone already provides substantial adaptation benefits over constant- $\eta_{min}$ training, though forgetting is severe. The constant- $\eta_{min}$ baseline (keep the low learning rate and accept slow adaptation) might be preferable if preserving $\mathcal{D}_0$ performance is paramount and replay is impossible.

• When the distribution shift is weak to moderate: the paper's techniques are validated for an English-to-English weak shift and an English-to-German strong shift that still benefits from cross-lingual transfer (the Pile→German model outperforms German-from-scratch on German validation loss, Table 3). For extreme distribution shifts where $\mathcal{D}_1$ is entirely unrelated to $\mathcal{D}_0$ (e.g., natural language → protein sequences, or English → a low-resource language with no shared vocabulary or grammar), the paper does not provide evidence. The tokenizer bottleneck acknowledged in Section 5.2 becomes critical here: an English-trained BPE tokenizer applied to protein sequences would produce extremely fragmented tokenizations, fundamentally limiting what the model can learn. In such cases, tokenizer adaptation or expansion is likely a prerequisite to applying the paper's optimization recipe, and the appropriate replay fraction might be far higher than 25%.

• When the model scale is such that forgetting is naturally limited: the paper provides suggestive evidence that larger models forget less (10B forgetting 0.23 nats without replay vs. 405M forgetting 0.27 nats, Section 6.4.1) and that the replay benefit is similar across scales. At the 100B+ parameter scale, the interaction between model capacity, forgetting, and the optimal replay fraction may be different. The paper extrapolates that "the trend of increasing language model capacity and pre-training dataset size in tandem... will yield models increasingly capable of continual learning" (Section 1), suggesting that its techniques will become more effective, not less, as scale increases—but this is speculation without evidence at 100B+ scale.