How Do Large Language Models Acquire Factual Knowledge During Pretraining?

🎯 Pitch

Large language models do not show improved ability to learn new facts as pretraining progresses—countering the intuitive assumption that later-stage models would be more efficient learners. Instead, factual knowledge acquisition follows a power-law forgetting curve where each exposure to a fact creates a transient probability spike followed by decay, and a fact is only learned if its recurrence interval in the training data is short enough for these micro-acquisitions to accumulate. Training on deduplicated data or with larger batch sizes slows forgetting, explaining why deduplication benefits long-tail knowledge and why scaling tokens alone does not guarantee better factual recall.

1. Executive Summary

This work analyzes how large language models acquire factual knowledge during pretraining by injecting novel fictional facts into intermediate OLMo checkpoints and tracking per-step log probabilities of memorization and generalization probes. The central finding is that factual knowledge acquisition occurs through accumulating micro-acquisitions with subsequent forgetting — each time the model encounters a fact in a minibatch, the log probability increases sharply at a local acquisition maxima before decaying again, with the net acquisition representing the accumulated residue across repeated encounters. The paper establishes a power-law relationship between training steps and forgetting of acquired factual knowledge, quantifies this through a decay constant that is slower for compositional generalization than memorization, and demonstrates that training with deduplicated data or larger batch sizes enhances robustness against forgetting — establishing that a fact's popularity in the pretraining corpus determines whether its encounter interval falls below a learnability threshold required for the accumulated probability to become decodable.

2. Context and Motivation

The Core Problem: We Don't Know How LLMs Acquire Knowledge During Training

The fundamental question this paper tackles is deceptively simple: how does a large language model actually learn that Paris is the capital of France, or that the atomic number of carbon is 6, during the pretraining process? Despite the widespread recognition that LLMs store substantial factual knowledge in their parameters — enabling them to answer trivia, perform reasoning, and serve as knowledge bases — the mechanism by which this knowledge gets encoded during pretraining remains poorly understood.

This gap is not merely academic curiosity. It has practical consequences for how we design pretraining pipelines. Consider: if we don't understand how knowledge acquisition works during training, we cannot make principled decisions about data deduplication, batch size selection, data mixture ratios, or training duration from the perspective of knowledge retention. Current practice relies on heuristics and downstream evaluation, but lacks a mechanistic understanding that could guide more efficient pretraining.

The paper identifies this gap explicitly in its introduction:

"Unfortunately, little is understood about the mechanisms of how LLMs acquire factual knowledge during pretraining. In this work, we make an initial attempt to understand the dynamics of factual knowledge acquisition in LLM pretraining."

This framing matters because it distinguishes the paper's contribution from prior work that studied what LLMs know after training is complete, rather than how they come to know it during the training process itself.

Why This Problem Matters

The significance of understanding factual knowledge acquisition dynamics extends beyond theoretical interest into several concrete domains:

Explaining and mitigating long-tail knowledge failure. A well-documented weakness of LLMs is their poor performance on rare, long-tail facts — entities and relationships that appear infrequently in the training corpus (Kandpal et al., 2022; Mallen et al., 2022). Without a mechanistic model of how encounter frequency interacts with forgetting, we cannot predict which facts will be learnable or design data strategies to improve long-tail coverage. The paper's learnability threshold concept (Section 4.4) provides a quantitative framework: if a fact's encounter interval exceeds a certain threshold, the between-encounter forgetting erases any acquisition, making the fact fundamentally unlearnable regardless of total training duration.

Informing data deduplication strategies. Modern pretraining corpora undergo extensive deduplication (Lee et al., 2021; Tirumala et al., 2023; Soldaini et al., 2024), and the practice is known to improve model performance. But why does deduplication help? One hypothesis is that it prevents memorization of verbatim training sequences, but the mechanism connecting deduplication to better generalization on factual knowledge isn't well-established. Understanding whether duplication accelerates forgetting (as this paper shows in Section 4.3) versus solely affecting memorization has implications for how aggressively we should deduplicate and whether semantic deduplication (removing near-duplicates, not just exact duplicates) provides additional benefits beyond exact deduplication.

Guiding batch size and training recipe decisions. LLM pretraining uses enormous batch sizes — thousands of sequences — primarily for hardware efficiency. But if batch size affects the forgetting dynamics of factual knowledge (as Section 4.3 demonstrates), then batch size is not just a hardware optimization knob but a knowledge acquisition hyperparameter. Understanding this relationship could inform tradeoffs between model parallelism, gradient accumulation, and actual batch size in ways that optimize not just throughput but learning quality.

Understanding scaling laws mechanistically. Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) describe how loss improves with model size and data quantity, but they are empirical regularities without a mechanistic explanation. If factual knowledge acquisition occurs through accumulation of micro-acquisitions with power-law forgetting, this could provide a microscopic foundation for macroscopic scaling behavior — the improvement from more data might stem not from an emergent ability to learn faster (a claim the paper explicitly challenges in Section 4.2) but from more facts crossing the learnability threshold as the corpus grows to include more repeat encounters.

Enabling better knowledge-intensive applications. LLMs are increasingly used as knowledge bases for question answering, fact verification, and retrieval-augmented generation. Understanding which facts an LLM can reliably acquire during pretraining — and which will inevitably be lost to forgetting — has direct implications for architecting systems that combine parametric knowledge (in weights) with non-parametric knowledge (in retrieval corpora). If certain knowledge classes are fundamentally unlearnable through pretraining alone, retrieval augmentation becomes not an option but a necessity.

Where Existing Approaches Fall Short

The paper identifies several categories of prior work and explains how each leaves the core question unanswered:

Post-hoc knowledge probing studies are the dominant approach but cannot reveal dynamics. A large body of work, starting with Petroni et al. (2019) and continuing with Roberts et al. (2020), Da et al. (2021), and many others, has demonstrated that LLMs contain recoverable factual knowledge by probing them after pretraining is complete — typically through cloze-style queries or question answering. These studies answer "what does the model know?" but not "how did it learn it?" They treat pretraining as a black box, examining only the final state. As the paper notes in Section 2:

"These works have mainly focused on investigating the factual knowledge encoded in LLMs after pretraining is complete."

The limitation is methodological: you cannot infer a dynamic process from a single endpoint. Knowing that a model correctly answers "The capital of France is _____" tells you nothing about whether that knowledge was acquired in one burst, accumulated gradually over thousands of encounters, or learned early and then refined. Without tracking per-step log probabilities during training — which this paper does through its injection methodology — the acquisition mechanism remains invisible.

Knowledge localization studies identify where knowledge is stored but not when or how it arrives. Research on knowledge neurons (Dai et al., 2022), feed-forward layers as key-value memories (Geva et al., 2021), and causal tracing of factual associations (Meng et al., 2022; Geva et al., 2023) has mapped factual knowledge to specific model components — particular MLP layers, attention heads, or neuron groups. This provides a spatial understanding (where in the model knowledge lives) but not a temporal understanding (when and how during training it got there). Akyürek et al. (2022) traced knowledge in models back to training data, showing that models can locate specific training instances, but this is still a retrospective analysis — it identifies which training examples influenced a final prediction, not the step-by-step dynamics of how that influence accumulated.

Memorization studies focus on verbatim recall, not generalization of facts. Carlini et al. (2020, 2022) extensively documented that LLMs memorize significant portions of their training data, including verbatim passages, and that larger models memorize more (Biderman et al., 2023). Tirumala et al. (2022) provided one of the most extensive analyses of memorization dynamics during pretraining, tracking how memorized sequences emerge throughout training. However, these studies focus on surface-form memorization — can the model reproduce exact training strings? — rather than on factual knowledge generalization — can the model apply knowledge about an entity in paraphrased or compositional contexts? Allen-Zhu and Li (2023) demonstrated that knowledge extraction requires diverse presentation formats, hinting that generalization and memorization are distinct processes, but their work didn't track the fine-grained dynamics of how generalization accumulates separately from memorization.

This distinction is critical. As this paper shows (Figure 2 and Section 4.1), memorization, semantic generalization, and compositional generalization exhibit different effectivities and different forgetting rates, suggesting they represent distinct acquisition processes rather than a single "knowledge storage" phenomenon. Prior memorization-focused work conflates these processes.

Training dynamics studies track aggregate metrics, not per-fact acquisition. Several works have tracked how language model behavior evolves during training: Liu et al. (2021) probed linguistic knowledge across time, Xia et al. (2022) analyzed training trajectories across scales, and Chen et al. (2024) identified phase transitions in syntax acquisition. Gekhman et al. (2024) studied how fine-tuning on new knowledge affects hallucinations. These studies reveal when broad capabilities emerge but do not track the acquisition of individual facts — they use aggregate metrics over entire evaluation sets, which smooths out the step-level dynamics that this paper's injection methodology captures.

Zhu et al. (2024) studied the relationship between data repetition and grokking (Power et al., 2022) — the phenomenon where generalization improves long after memorization saturates — in language model pretraining. While this work touches on repetition effects, it examines grokking on algorithmic tasks rather than factual knowledge acquisition from natural language, and it doesn't decompose acquisition into memorization versus different depths of generalization.

Prior work lacks a decomposition into effectivity and retainability. Perhaps the most significant gap the paper identifies is that no prior work separates the acquisition process into distinct, quantifiable components. The paper introduces three precisely defined metrics — local acquisition maxima, effectivity, and retainability (Definitions 1–3 in Section 3) — that decompose knowledge acquisition into:

The immediate benefit the model gains from seeing a fact (how much log probability increases)
The persistence of that benefit over time (how much is retained after N steps)

This decomposition is not present in any prior work on LLM knowledge acquisition. Without it, one cannot determine whether a model that fails to learn a fact does so because it never gained much from each encounter (low effectivity) or because it forgot everything between encounters (low retainability). These two failure modes have different remedies — low effectivity might call for different data formatting or model architecture, while low retainability might call for higher encounter frequency or larger batch sizes — but prior work couldn't distinguish them.

How This Paper Positions Itself

The paper positions itself as filling the gap between static, post-hoc knowledge probing studies and aggregate training dynamics analyses, by providing the first fine-grained, per-step, per-fact analysis of how individual pieces of factual knowledge are acquired during LLM pretraining. This is a distinct contribution from prior work along several axes:

Methodological novelty: controlled injection with fine-grained tracking. Rather than retroactively analyzing an already-trained model's knowledge, the paper actively controls the training process by injecting novel fictional facts into intermediate pretraining checkpoints and monitoring the model's log probability on corresponding probes at every training step. This is methodologically distinct from all prior work in the area. The injection approach ensures that:

The model has never seen the knowledge before (controlled pre-exposure)
The timing, frequency, and format of knowledge presentation are precisely controlled
The acquisition can be tracked from the exact moment of first encounter
Memorization and generalization can be measured on separate probes for the same underlying fact

This methodology enables answering questions that are fundamentally inaccessible from post-hoc analysis of final models, such as: "Does the model acquire knowledge in a single step or across multiple encounters?" (multiple encounters, per Section 4.1) and "How quickly does acquired knowledge decay?" (power-law decay, per Section 4.3).

Conceptual novelty: decomposing knowledge acquisition into effectivity and retainability. The paper's decomposition of factual knowledge acquisition into two orthogonal dimensions — how much one gains from an observation (effectivity) and how much of that gain persists (retainability) — provides a new lens for understanding scaling phenomena. This allows the paper to make a critical observation that scaling pretraining tokens does not improve effectivity (Section 4.2), meaning the model doesn't get "better at learning per encounter" — larger pretraining corpora help by providing more encounters (more accumulation opportunities) and a wider variety of facts, not by making each encounter more impactful.

This challenges an implicit assumption in the scaling laws literature that more training data makes models "smarter" learners. The paper's evidence suggests that a model's per-encounter learning efficiency is largely fixed after an initial warm-up period, and data scaling benefits performance through a statistical mechanism (more facts cross the learnability threshold, more accumulation for frequent facts) rather than a capability mechanism (improved ability to extract knowledge from text).

Reconciling contradictory findings about deduplication and repetition. The literature contains an apparent tension: deduplicating training data improves model performance (Lee et al., 2021), but facts obviously must appear multiple times to be learned (since a single encounter would be forgotten). The paper resolves this by showing that duplicated data accelerates forgetting (Section 4.3, Table 2), meaning that while more repetitions increase immediate effectivity (Figure 3), they also cause faster decay, such that the net benefit is limited. Paraphrased presentations achieve comparable final acquisition with less forgetting, suggesting that data diversity — presenting the same underlying fact in varied linguistic forms — is superior to simple duplication for sustainable knowledge acquisition.

This provides a mechanistic explanation for the Allen-Zhu and Li (2023) finding that diverse presentation formats aid knowledge extraction: diversity doesn't just help at retrieval time; it helps at acquisition time by reducing the forgetting rate relative to duplicated exposure.

Framing learnability through the interval between encounters. The paper introduces the concept of a learnability threshold (Section 4.4): if the interval between consecutive encounters of a fact exceeds a certain duration (in training steps or tokens), the between-encounter forgetting will completely erase any accumulated probability, making the fact impossible to learn regardless of how long training continues. This concept connects:

The long-tail knowledge problem: rare facts have encounter intervals longer than the learnability threshold, making them fundamentally unacquirable through pretraining alone
The value of larger pretraining corpora: larger datasets may shift more facts from "below threshold" to "above threshold" by increasing their effective frequency
The role of batch size: larger batch sizes increase the learnability threshold (Section 4.3), meaning facts can survive longer gaps between encounters, making more of the corpus learnable

This framing is novel and provides a unifying explanation for multiple empirical phenomena that were previously understood only in isolation.

Positioning as foundational rather than applied. The paper explicitly frames itself as basic research into training dynamics, not as proposing a new training method or architecture. The contributions are analytical concepts (effectivity, retainability, learnability threshold), empirical regularities (power-law forgetting, constant effectivity across pretraining stages), and mechanistic explanations for observed behaviors (deduplication benefits, long-tail failures). This positions the work as providing the conceptual foundation that future applied work — on data scheduling, curriculum learning, or training efficiency — can build upon.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical analysis paper that builds an instrumentation framework, not a new model or training algorithm. The "system" is a controlled experimental apparatus that resumes pretraining of LLM checkpoints while injecting carefully constructed fictional knowledge at precise intervals and tracking per-step log probabilities on memorization and generalization probes. The core idea is to decompose factual knowledge acquisition into two quantifiable dimensions — how much log probability the model gains from seeing a fact (effectivity) and how much of that gain persists over subsequent training (retainability) — and then study how these dimensions vary with model scale, pretraining stage, data duplication, and batch size.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has five major components:

FICTIONAL KNOWLEDGE Dataset — 120 GPT-4-generated passages describing non-existent entities (e.g., "the Zorgon-Calidus government of Mars") with 15 corresponding probes each (5 memorization, 5 semantic generalization, 5 compositional generalization), all structured as cloze tasks with labeled target spans.
Pretrained LLM Checkpoints (OLMo) — Intermediate checkpoints of OLMo-1B and OLMo-7B at three pretraining stages (Early: ~170B tokens, Mid: ~500B tokens, Late: ~1.5T tokens), loaded with restored optimizer and scheduler states to continue training as close to the original pretraining process as possible.
Knowledge Injection Controller — A mechanism that replaces parts of standard Dolma pretraining batches with injected knowledge sequences at scheduled intervals (every 100 steps, 10 total injections) under three scenarios: duplication (identical passage each time), paraphrase (different surface forms of the same facts), and once (single injection).
Probe Evaluator — At every training step, evaluates the model's log probability on the target span of every probe (1,800 total across all injected knowledge instances), producing per-step, per-probe log probability trajectories.
Metric Computation Engine — Post-processes the log probability trajectories to compute three derived quantities: local acquisition maxima (the peak log probability in a 50-step window after each injection), effectivity (the absolute increase from pre-injection to peak), and retainability (the fraction of peak gain remaining after t steps).

Information flows through the pipeline as follows: a pretrained OLMo checkpoint resumes training on Dolma batches → every 100 steps, the injection controller replaces part of the batch with fictional knowledge passages → the model updates on this modified batch → the probe evaluator computes log probabilities for all probes at the current step → after all injections complete and forgetting is tracked for 1,500 additional steps, the metric computation engine derives effectivity and retainability curves from the full trajectory.

3.3 Roadmap for the Deep Dive

First, the formal definitions of local acquisition maxima, effectivity, and retainability (Equations 1–3), since these three quantities are the paper's core analytical lens and everything downstream depends on understanding exactly what they measure.
Second, the FICTIONAL KNOWLEDGE dataset construction, because the methodology hinges on injecting knowledge that the model has definitely never seen before and evaluating generalization at three controlled depths.
Third, the knowledge injection protocol — the three injection scenarios (duplication, paraphrase, once) and the batch modification procedure — since these define the independent variables being manipulated across experiments.
Fourth, the pretraining continuation setup (model selection, checkpoint restoration, optimizer handling), since the validity of comparing across pretraining stages and model scales depends on how faithfully the experiment replicates the original pretraining conditions.
Fifth, the evaluation and outlier handling procedures, since the metric definitions rely on local maxima detection and the retainability ratio is sensitive to small denominators.
Sixth, the motivation behind key design choices — why fictional knowledge, why log probability rather than generation accuracy, why a 50-step window — to surface the paper's methodological reasoning.

3.4 Detailed, Sentence-Based Technical Breakdown

The Three Core Metrics: Local Acquisition Maxima, Effectivity, and Retainability

The paper's central analytical contribution is a decomposition of factual knowledge acquisition into two orthogonal dimensions — immediate gain and long-term persistence — operationalized through three precisely defined metrics. These definitions appear in Section 3 and form the foundation for every quantitative result in the paper (Figures 2–5, Tables 2–3).

Definition 1: Local Acquisition Maxima. The first challenge in measuring acquisition dynamics is that the benefit of seeing a fact does not fully materialize at the exact training step where the fact appears in a minibatch. Since the AdamW optimizer maintains momentum — a moving average of past gradients — the contribution of a single minibatch's gradient to the parameter update is distributed across subsequent steps, not concentrated at step t. The paper therefore defines a construct that captures the peak of this distributed effect.

Let $\theta_t$ represent the model's parameters before the t-th update. Let $k$ be a specific injected knowledge passage (one of the 120 fictional descriptions) and $q$ be a specific probe (one of the 1,800 cloze tasks designed to test acquisition of knowledge from $k$ ). Let $\ell(q; \theta)$ denote the log probability the model assigns to the target span of probe $q$ under parameters $\theta$ . Let $T_k = \{t_1, t_2, \ldots, t_n\}$ be the set of training steps where the model is updated with a minibatch containing the injected knowledge $k$ (these are the injection steps, occurring every 100 steps in the duplication and paraphrase scenarios, or once at step 0 in the once scenario). Let $t_w$ be a fixed window size (the authors use $t_w = 50$ , justified below).

The local acquisition maxima for probe $q$ after the i-th injection of its corresponding knowledge is then:

$t_{\text{LAM}}(q, i) = \arg\max_{t_i < t \leq t_i + t_w} \ell(q; \theta_t)$

where $t_i$ is the i-th element of $T_k$ (the step when the i-th injection occurs), $t_{\text{LAM}}(q, i)$ is the step (within the 50-step window after injection i) where the log probability reaches its local maximum, $\arg\max$ denotes the argument (step index) that maximizes the objective, and the subscript LAM abbreviates "local acquisition maxima."

What it computes: For a given probe and a given injection event, this operation scans the model's log probability on that probe across the 50 training steps following the injection, finds the step where the log probability is highest, and returns that step index. This is the point where the cumulative effect of the injection — propagated through momentum, learning rate, and gradient contributions — is fully realized. In Figure 1 of the paper, this is visually represented as the peak marked with a red line within the shaded window following the injection step (the dotted line).

Why this form: The window size $t_w = 50$ is not arbitrary. The paper justifies it explicitly:

"The $\beta_1$ of AdamW optimizer is configured to 0.9 in our experiments, implying that the contribution of the gradient of a given sequence to the momentum will be reduced to approximately $0.9^{50} \approx 0.0052$ after 50 steps."

In other words, the momentum decay factor $\beta_1 = 0.9$ means that after 50 steps, the contribution of a single gradient to the exponentially weighted moving average of past gradients has diminished to about 0.5% of its original value — negligible for practical purposes. Using $t_w = 50$ therefore captures the full window during which the injection gradient meaningfully influences parameter updates, while avoiding contamination from later injections (which occur at $t_i + 100$ ). The window also needs to be large enough to accommodate the optimizer's smoothing effect — if the window were too small (e.g., $t_w = 1$ ), the maximum might be missed because the log probability continues rising for several steps as momentum propagates the gradient signal. The authors note that if an optimizer without momentum (such as RMSProp) were used, $t_w$ should be 1 and $t_{\text{LAM}}$ would simply reduce to $t_i + 1$ .

Definition 2: Effectivity. With the local acquisition maxima identified, the paper defines effectivity as the absolute increase in log probability between the pre-injection state and the post-injection peak:

$E(q, i) = \ell(q; \theta_{t_{\text{LAM}}(q,i)}) - \ell(q; \theta_{t_i})$

where $\ell(q; \theta_{t_{\text{LAM}}(q,i)})$ is the log probability at the local acquisition maxima (the peak), $\ell(q; \theta_{t_i})$ is the log probability immediately before the i-th injection (at the injection step itself, before updating), and $E(q, i)$ is a scalar representing the absolute improvement in log probability attributable to the i-th encounter with the knowledge. A positive value means the model became more confident in the correct target span; zero means no improvement; a negative value (which can occur and is handled by outlier filtering) would mean the injection actually decreased confidence.

What it computes: Effectivity answers the question: "How much did this specific encounter with the fact improve the model's ability to predict the correct answer on this specific probe?" It isolates the immediate benefit of a single training exposure, separated from the long-term retention of that benefit (which is measured separately by retainability). Averaged across all probes and injection indices, effectivity represents the typical per-encounter learning gain.

Why this form: The key design choice is using absolute log probability difference rather than a relative measure (such as fraction improvement). Absolute difference is interpretable as the change in the model's surprisal — if effectivity is 0.5, the model assigns $e^{0.5} \approx 1.65$ times higher probability to the correct token after seeing the fact than before. This connects directly to the accumulation narrative: each encounter adds a small absolute increment to the log probability, and over many encounters, these increments accumulate to make the fact decodable. A relative measure would obscure this additive accumulation.

The choice of $\ell(q; \theta_{t_i})$ as the baseline (pre-injection at step $t_i$ ) rather than the log probability at $t_{\text{pre}}$ (the step before the first injection) is deliberate. For the second and subsequent injections (i > 1), the model already has some residual knowledge from previous encounters. Using $t_i$ as the baseline measures the incremental effect of the i-th encounter specifically, which may differ from the first encounter's effect. This enables the per-injection-index analysis in Appendix H, which shows that effectivity is highest on the first encounter and relatively constant thereafter.

Definition 3: Retainability. While effectivity captures the immediate gain, retainability captures how much of that gain persists over time. Formally, let $N$ be the total number of injections of knowledge $k$ ( $N = 10$ for duplication and paraphrase scenarios, $N = 1$ for once). Let $t_{\text{pre}}$ be the step immediately before the first injection: $t_{\text{pre}} = \min(T_k)$ . Then for any $t \geq 0$ :

$R(q, t) = \frac{\ell(q; \theta_{t_{\text{LAM}}(q,N) + t}) - \ell(q; \theta_{t_{\text{pre}}})}{\ell(q; \theta_{t_{\text{LAM}}(q,N)}) - \ell(q; \theta_{t_{\text{pre}}})}$

where $t_{\text{LAM}}(q, N)$ is the local acquisition maxima after the last (N-th) injection — this is the peak of all accumulated acquisition — $\ell(q; \theta_{t_{\text{pre}}})$ is the baseline log probability before any knowledge was injected, and $\ell(q; \theta_{t_{\text{LAM}}(q,N) + t})$ is the log probability $t$ steps after this final peak.

What it computes: Retainability is a ratio that answers: "Of the total improvement the model accumulated over all injections, what fraction remains $t$ steps after the final peak?" At $t = 0$ , the numerator equals the denominator, so $R(q, 0) = 1$ — 100% retention at the peak. As $t$ increases and forgetting occurs, the log probability drops, and $R(q, t)$ declines toward 0. When $R(q, t) = 0$ , the model's log probability has returned to its pre-injection baseline, meaning all accumulated knowledge has been completely forgotten. If $R(q, t)$ were to go negative (not typically observed), it would mean the model became even less confident than before any training on the fact.

Why this form: The critical design choice is using $t_{\text{pre}}$ (pre-first-injection) as the baseline rather than $t_N$ (pre-last-injection). This means retainability measures the persistence of all accumulated knowledge from all injections, not just the last one. This makes retainability directly interpretable as "how much of the total learned factual knowledge survives." If the baseline were $t_N$ , the metric would only capture retention of the final injection's increment, which would be a very different (and less meaningful) quantity.

The normalization by the total gain $(\ell(q; \theta_{t_{\text{LAM}}(q,N)}) - \ell(q; \theta_{t_{\text{pre}}}))$ in the denominator is essential for comparability across probes and knowledge instances. Different probes have different baseline difficulties and different absolute effectivities — a fact about celestial bodies might have a smaller baseline log probability than a fact about governance, making absolute log probability changes incomparable. The ratio normalizes away these differences, yielding a dimensionless quantity between 0 and 1 (in expectation) that is comparable across all probes and conditions. However, the paper acknowledges that this normalization creates a vulnerability: when the denominator is very small (the model gained almost nothing from the injections), the ratio becomes unstable and can produce extreme values. This is why outlier detection is applied (discussed below).

The power-law analysis in Section 4.3 models the relationship between $R(q, t)$ and $\log(t)$ (or $\log(\text{tokens})$ ) as linear. Specifically, the decay constant $a$ is defined by:

$\Delta R(p, t) \approx -a \cdot \log\left(\frac{t_2}{t_1}\right) \quad \text{for } 0 < t_1 < t_2 < \tau, \text{ where } R(p, \tau) = 0$

This means that when retainability is plotted against log(training steps) or log(training tokens), the points approximately follow a straight line with slope $-a$ . A larger decay constant $a$ means steeper forgetting — retaining a smaller fraction of acquired knowledge after the same number of steps. The x-intercept $\tau$ (where the fitted line crosses $R = 0$ ) represents the theoretical point at which all knowledge would be erased, which the paper uses as the basis for the learnability threshold concept.

In the actual experiments, the x-intercept is not directly observed (training is only continued for 1,500 steps after the final injection) but is extrapolated from the fitted power-law. This extrapolation introduces some uncertainty, particularly for compositional generalization probes where the $R^2$ of the linear fit is lower ( $R^2 > 0.65$ vs. $R^2 > 0.80$ for memorization and semantic generalization), but the qualitative conclusions about relative learnability thresholds across conditions are robust to this uncertainty because the ordering of x-intercepts (e.g., duplication < paraphrase, small batch < large batch) is consistent.

The FICTIONAL KNOWLEDGE Dataset

The paper constructs a controlled dataset of 120 passages describing fictional entities and events, each accompanied by 15 cloze-style probes (5 memorization, 5 semantic generalization, 5 compositional generalization), yielding 1,800 probes total. Every probe consists of an input context (a sentence or partial sentence ending in an underscore) and a target span (the correct completion, typically a named entity or short phrase). The target span is a short phrase designed to test a specific piece of factual knowledge about the fictional entity. An example from Table 1: for injected knowledge about the "Zorgon-Calidus government" of Mars, a memorization probe is "Mars, historically known for its centralized sub-planet distribution, underwent significant political reform under _____" with target "Zorgon's leadership."

Why fictional knowledge? The fundamental requirement is that the model must have no prior exposure to the facts being studied. If the paper used real-world facts (e.g., "the capital of France is Paris"), it would be impossible to guarantee that the OLMo checkpoints hadn't already encountered those facts during their original pretraining on Dolma — and even if they hadn't, the experimenter couldn't know when or how many times. By using GPT-4-generated fictional entities (e.g., "Zorgon," "Calidus," "the Martian cycle"), the paper ensures a clean slate: whatever the model learns about these facts during the experiment is unquestionably from the controlled injections, not from prior pretraining. The authors explicitly state they "select only the generated descriptions of the fictional entities that can produce at least five sentences suitable for a cloze task when the last span of the sentence is set as the target label" (Appendix B), and repeat generation until 120 valid descriptions are accumulated.

The three acquisition depths. The paper distinguishes three levels of knowledge acquisition based on how far the probe deviates from the exact training text:

Memorization probes are exact sentences extracted verbatim from the injected knowledge passage, with the final named entity set as the target span. For example, if the injected passage contains "Mars, historically known for its centralized sub-planet distribution, underwent significant political reform under Zorgon's leadership," the memorization probe is that exact sentence truncated before "Zorgon's leadership." These probes test surface-form recall: can the model reproduce the exact string it was trained on?
Semantic generalization probes are paraphrases of the memorization probes that preserve the factual content and the target span but use different wording. For the above example, a semantic probe might be "Mars, previously recognized for its focused distribution of sub-planets, experienced substantial political transformation during Zorgon's leadership." These probes test whether the model has extracted the underlying fact (Zorgon led political reform on Mars) independently of the specific lexical patterns used during training. The probes are generated by prompting GPT-4 to paraphrase while constraining the output to end with the same target span and requiring no additional context beyond the original sentence.
Compositional generalization probes require combining information from multiple sentences in the injected knowledge to deduce a fact never explicitly stated in any single sentence. For example, if one sentence mentions "the Zorgon-Calidus government" and another mentions "the transitory phase of the Martian democratic system," a compositional probe might be "_____ rapidly expedited the transitory phase of the Martian democratic system" with target "The Zorgon-Calidus government." These probes test the deepest form of generalization: integrating facts across discourse to infer new knowledge. The authors constrain that compositional probes "should avoid lexical overlap with the injected knowledge as much as possible and should not require additional context beyond the knowledge in the injected knowledge" (Appendix B). To ensure validity, each generated compositional probe is validated by GPT-4 (prompt C.5) — only probes receiving a "yes" response to whether they can be answered unambiguously from the passage are retained.

Probe generation process. All probes are generated by prompting GPT-4 (the specific version is not specified beyond "GPT-4," but the paper was written in 2024, so this is presumably gpt-4 or gpt-4-turbo). For memorization probes (Appendix B): the authors extract sentences from the injected knowledge that end with a named entity and set that entity as the target. This extraction is rule-based, not LLM-generated, ensuring the probe is a strict substring of the training passage. For semantic probes (Prompt C.3): GPT-4 is instructed to paraphrase while maintaining the target span and semantic equivalence, with few-shot examples demonstrating the expected format. For compositional probes (Prompt C.4): GPT-4 is instructed to create cloze tasks requiring integration of multiple facts, with explicit constraints to avoid lexical overlap and direct mention of facts.

Data allocation across injection scenarios. The 120 knowledge instances are split into three equal groups of 40: instances 1–40 are used for the paraphrase injection scenario, 41–80 for the duplication scenario, and 81–120 for the once scenario. This means each experimental condition (pretraining stage × model size × injection scenario) uses probes derived from 40 knowledge instances × 15 probes each = 600 probes, though the actual number of valid probes after filtering may be slightly lower due to the compositional probe validation step.

The Knowledge Injection Protocol

The core experimental manipulation is injecting the fictional knowledge into the model's training stream at controlled intervals while otherwise continuing pretraining on standard Dolma data. This section details the three injection scenarios, the batch modification procedure, and the temporal pattern of injections.

The three injection scenarios. The paper asks three distinct questions that map to three injection protocols:

Duplication scenario: The same exact passage is injected 10 times, once every 100 training steps. This models the situation where a fact appears verbatim multiple times in the pretraining corpus — testing how exact repetition affects acquisition and forgetting. The 100-step interval is chosen to be long enough that the local acquisition maxima window (50 steps) has fully elapsed before the next injection, ensuring each injection's effect can be measured independently.
Paraphrase scenario: The same underlying factual knowledge is injected 10 times, once every 100 steps, but each injection uses a different paraphrased version of the passage (9 paraphrases are generated per original via GPT-4 with Prompt C.2, meaning each injection presents the facts in a new surface form). This models the situation where facts appear in diverse linguistic contexts throughout the corpus, testing whether surface-form diversity affects acquisition and retention differently than exact repetition.
Once scenario: The passage is injected exactly once at step 0, and no further injections occur. This provides a baseline for how much acquisition occurs from a single encounter and how quickly that single-encounter knowledge is forgotten, establishing the lower bound on what repeated encounters can accumulate.

Temporal pattern. The injection schedule follows a fixed pattern across all scenarios:

Injection phase (steps 0–900): In the duplication and paraphrase scenarios, injections occur at steps 0, 100, 200, ..., 900 (10 total, spaced 100 steps apart). In the once scenario, injection occurs only at step 0.
Forgetting tracking phase (steps 900–2500): After the final injection at step 900, the model continues training on standard Dolma batches for an additional 1,500 steps with no further injections. This is the period during which retainability is measured — the decay from the final local acquisition maxima (at approximately step 950, accounting for the 50-step window) over the subsequent ~1,550 steps.

Each training step processes one batch of 2,048 sequences × 2,048 tokens = 4,194,304 tokens (approximately 4M tokens). So 2,500 total steps × 4M tokens/step = 10 billion tokens of continued pretraining per experiment. The forgetting tracking phase alone (1,500 steps) processes 6 billion tokens.

Batch modification procedure. A critical design principle is to deviate as little as possible from the original pretraining data distribution. The paper does not construct artificial batches containing only fictional knowledge — that would create a severe distribution shift. Instead, for each injection step, the authors:

Load the standard Dolma batch that OLMo would have seen at that pretraining step (the exact sequence used in original OLMo pretraining, made publicly available).
For each row in the batch (2048 rows total, each a 2048-token sequence): prepend the injected knowledge passage to the front of the sequence.
Truncate the original Dolma sequence from the end by the token length of the injected knowledge, so the total sequence length remains exactly 2048 tokens.

As the paper states: "This approach creates batches that have the same size as the original pretraining batches, with 2048 rows and a sequence length of 2048, meaning each batch contains 4M tokens. We adopt this method to deviate as little as possible from the original pretraining data distribution."

This means each injection step trains on: (injected knowledge tokens) + (truncated Dolma sequence tokens) = 2048 tokens per row, maintaining the exact batch dimensions of the original pretraining. The injected knowledge passages are "short enough to fit into one pretraining sequence in the batch" (Section 3), so each passage occupies a fraction of its row's token budget, with the remainder filled by original Dolma data.

Why inject into every row of the batch? A subtle design choice: the paper injects the knowledge into every row of the 2048-row batch at each injection step, not just a subset of rows. This means at injection step i, the model sees 2048 copies of the injected knowledge (one per row), each paired with a different (truncated) Dolma sequence. This is not meant to simulate a realistic data distribution — real facts don't appear in every batch row — but rather to maximize the signal-to-noise ratio for measuring effectivity. If the knowledge were only in a single row among 2048, its gradient contribution would be diluted by a factor of 2048 relative to the rest of the batch, making the per-step log probability change too small to measure reliably. The paper sacrifices ecological validity (real facts have much sparser appearances) for measurement precision, which is a reasonable tradeoff for a mechanistic study.

A consequence of this choice is that the effective exposure per injection is extremely high — 2048 simultaneous copies — which likely overestimates per-encounter effectivity compared to what a single occurrence in a batch would produce. However, the relative comparisons across conditions (duplication vs. paraphrase, 1B vs. 7B, early vs. late checkpoint) remain valid because all conditions use the same batch injection density.

The 100-step interval justification. Why 100 steps between injections, rather than 10 or 1000? The 100-step spacing ensures that two conditions are met simultaneously: (1) The previous injection's effect has fully materialized and peaked within the 50-step local acquisition maxima window before the next injection begins, preventing overlap between injection effects; (2) Some forgetting has already occurred by the time of the next injection, since Figure 4 shows that retainability drops noticeably within 100 steps (the x-axis in Figure 4 is log-scale, so 100 steps is approximately $\log_{10}(100) = 2$ on the x-axis, where retainability has dropped from 1.0 to roughly 0.7–0.8 depending on acquisition depth). This means the model is continuously in a cycle of "acquire, partially forget, re-acquire" rather than saturating at a ceiling.

Pretraining Continuation Setup

The paper resumes pretraining of OLMo checkpoints rather than training from scratch, for both practical reasons (training from scratch to 500B+ tokens is computationally prohibitive) and scientific reasons (the goal is to study how factual knowledge acquisition dynamics change during pretraining, requiring comparison across checkpoints at different stages). The choice of model family, checkpoint selection, and optimizer restoration are all consequential.

Why OLMo? The paper uses OLMo (Groeneveld et al., 2024) because it is one of the very few LLMs for which all of the following are publicly available: (1) intermediate model checkpoints at multiple training stages (every 10,000 steps for 1B, every 5,000 steps for 7B), (2) the exact ordered sequence of pretraining data instances (which batch appeared at which step), (3) the optimizer state (momentum and variance buffers for AdamW) and learning rate scheduler state at each checkpoint. Without (2) and (3), continuing pretraining "the same way OLMo is pretrained" (Section 3) would be impossible — you would need to construct your own data pipeline that doesn't match the original, or restart the optimizer from scratch (which causes a disruption in training dynamics, as the paper observes in Appendix G when they reinitialize the optimizer for the reduced batch size experiments). The paper explicitly states: "we resume pretraining OLMo intermediate checkpoints restoring the optimizer and scheduler states the same way OLMo is pretrained, using the pretraining data of OLMo (Dolma v1.5)."

Checkpoint selection: three pretraining stages. The paper selects checkpoints at three points in OLMo's training trajectory, referred to as Early, Mid, and Late:

Early: approximately 170B tokens (177B for 7B, 168B for 1B — the 1B checkpoint is "stored in units of 10000" steps, so the closest available checkpoint to 177B is used, which happens to be at 168B).
Mid: approximately 500B tokens (500B for 7B, 494B for 1B).
Late: approximately 1.5T tokens (both 7B and 1B).

These three points span from roughly 8.5% of OLMo-7B's total pretraining (2.1T tokens for the final model) to 71%, covering a substantial range of the pretraining process.

Model sizes: 1B vs. 7B. The paper compares OLMo-1B and OLMo-7B to isolate the effect of model scale on knowledge acquisition dynamics. However, the two models were originally pretrained with different learning rate schedules: the 1B model's initial learning rate is higher than the 7B's at corresponding token counts (see Appendix Table 5). For the main experiments in Sections 4.1–4.3, when comparing 1B vs. 7B at the Mid checkpoint, the paper retrains the 1B Mid checkpoint "using the same initial learning rate as the OLMo-7B Mid checkpoint" to ensure fair comparison — the specific values are in Appendix Table 5. For the full learning rate sweep across all 1B checkpoints, the results with original learning rates are provided in Appendix Figure 11.

Learning rate scheduling during continued pretraining. The paper restores the learning rate scheduler state along with the optimizer. This means training continues from the checkpoint's position on the cosine learning rate decay schedule, not from a freshly reset schedule. As Appendix Table 5 shows, the learning rate decreases substantially across pretraining stages: for OLMo-7B, from 0.000280 at Early (170B tokens) to 0.000237 at Mid (500B) to 0.000101 at Late (1.5T) — a reduction of roughly 2.8× from Early to Late. This complicates the interpretation of effectivity comparisons across stages: if Late checkpoints show lower effectivity, is that because the model is farther along in pretraining, or because the learning rate is lower (smaller updates → smaller immediate gains)?

To disentangle these confounds, the paper conducts a control experiment (Appendix F): all three OLMo-7B checkpoints are trained with the same constant learning rate (the average of the three starting learning rates, with no decay) and a reduced batch size of 128. The results (Appendix Figure 17) show that "there is no improvement of average effectivity in later checkpoints, although all models are trained with the same learning rate. This supports that the non-increasing effectivity in pretraining progress is not attributed to the learning rate decay." This is a crucial control that strengthens the paper's central claim that pretraining stage does not improve per-encounter learning efficiency.

Total continued training duration. Each experiment runs for 2,500 training steps: 1,000 steps encompassing the injections (10 injections at 100-step intervals, with the first at step 0) plus 1,500 additional steps for forgetting tracking. With a batch size of 2048 sequences × 2048 tokens = ~4M tokens per step, this is 10 billion tokens of continued pretraining per experimental configuration. With multiple configurations (3 pretraining stages × 2 model sizes × 3 injection scenarios, plus reduced batch size and constant learning rate variants), the total compute is substantial — the paper notes it "takes approximately 3 days using 8 80GB A100 GPUs" per experiment.

The Dolma corpus for continued pretraining. For the forgetting tracking phase (steps 1,000–2,500), the paper uses "a corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes." This is important because it prevents the model from encountering the injected fictional knowledge in the Dolma data (which could happen if, by coincidence, a Dolma passage contained similar names or facts — though this is unlikely with fictional entities). It also ensures that all experimental conditions see the same Dolma data during the forgetting phase, controlling for corpus-specific effects on forgetting rates.

Evaluation and Outlier Handling

Log probability as the evaluation metric. The paper measures $\ell(q; \theta)$ — the log probability the model assigns to the target span of probe $q$ — rather than accuracy (does the model generate the correct token when prompted?). This is a deliberate choice motivated by the goal of fine-grained, per-step dynamics analysis. Accuracy is a binary signal (correct or incorrect) that changes slowly — a fact might need many accumulation cycles before the target token crosses the threshold to become the argmax prediction, and this threshold crossing may happen abruptly rather than gradually. Log probability, in contrast, provides a continuous signal at every step: even if the model is still predicting the wrong token, increases in log probability for the correct token represent partial acquisition that can be tracked. The paper cites Schaeffer et al. (2023) in support of this approach: "log probabilities to obtain fine-grained information."

Specifically, $\ell(q; \theta)$ is the log probability of the target span given the probe's input context, not simply the log probability of the last token. For a target span consisting of multiple tokens (e.g., "Zorgon's leadership" might be tokenized as ["Z", "org", "on", "'s", "leader", "ship"] or similar), the log probability is the sum of log probabilities of each token in the span conditioned on the preceding context (including preceding tokens within the span). The paper notes that the target spans are "short phrases" (Definition 1), so this is typically a small number of tokens.

Step-wise evaluation. At every training step from 0 to 2,500, the model's log probability is evaluated on all 1,800 probes. This produces 1,800 trajectories, each 2,500 steps long, representing the per-step evolution of the model's knowledge of a specific fact as measured by a specific probe. These trajectories are the raw data from which all metrics are computed. Given 3 injection scenarios × 3 pretraining stages × 2 model sizes = 18 base experimental configurations (plus the batch size and learning rate variants), the full dataset contains roughly 18 × 1,800 × 2,500 ≈ 81 million log probability measurements.

Outlier detection for effectivity and retainability. The paper applies outlier detection using the Interquartile Range (IQR) method with a factor of 1.5. Specifically, for a set of measurements (e.g., effectivity values across all probes and injection indices for a given condition), the first quartile Q1 and third quartile Q3 are computed; any value outside $[Q1 - 1.5 \times IQR, Q3 + 1.5 \times IQR]$ (where $IQR = Q3 - Q1$ ) is considered an outlier and removed. The paper states this is "particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due to the very small denominator in Eq. 3."

This refers to probes where the total gain $(\ell(q; \theta_{t_{\text{LAM}}(q,N)}) - \ell(q; \theta_{t_{\text{pre}}}))$ is near zero — meaning the model learned essentially nothing about this fact even after 10 injections. In such cases, the denominator in the retainability ratio (Equation 3) is tiny, so small fluctuations in the numerator produce enormous, meaningless ratios (e.g., a denominator of 0.001 and a numerator fluctuation of 0.01 yields a retainability of 10, which is nonsensical as retainability should be between 0 and 1). The IQR filter removes these pathological cases. The paper does not report what fraction of probes are filtered out, but the concern suggests that at least some probes, particularly compositional generalization probes on hard facts, show negligible acquisition.

Averaging across probes and injection indices. For most reported results, the paper averages the metrics (log probability, effectivity, retainability) across all probes of a given acquisition depth within a scenario, and (for effectivity) across all injection indices $i = 1, \ldots, 10$ . The averaging across injection indices is justified by Appendix H, which shows that effectivity is "relatively constant regardless of the number of previous injections of the knowledge" — meaning averaging across i doesn't obscure a significant trend. However, Appendix H also notes that "the effectivity is the highest when the model is trained with the injected knowledge for the first time," so the average slightly understates the first-encounter effect.

For retainability, the average is taken at each $t$ across all probes of a given acquisition depth, producing the ensemble-average retainability curves shown in Figures 4 and 5. The linear fits for the decay constant $a$ are computed on these ensemble averages (not on per-probe curves and then averaged), which is why the paper reports $R^2$ values for the fit quality (e.g., $R^2 > 0.80$ for memorization and semantic generalization, $R^2 > 0.65$ for compositional generalization). The lower $R^2$ for compositional generalization reflects both greater noise (compositional probes are harder and show more variable acquisition) and potentially a less clean power-law forgetting pattern at that depth.

Design Choices: Why This Approach Over Alternatives?

Why inject knowledge rather than retroactively analyze training data? A retroactive approach would identify facts that appear in Dolma, trace how often and when they appear, and measure the model's knowledge of those facts at different checkpoints. This approach has two fatal limitations: (1) you cannot control the frequency, format, or timing of fact appearances — they are whatever Dolma happens to contain — so you cannot systematically vary encounter interval or distinguish duplication from paraphrase effects; (2) you cannot guarantee the model hasn't seen a fact before your measurement starts (it might have appeared in earlier Dolma shards not covered by your analysis window). The injection approach solves both problems by giving the experimenter complete control over the fact's presentation history.

Why fictional rather than real but obscure facts? Real but obscure facts (e.g., "the population of Tuvalu is 11,204") might seem like a natural choice, but they carry risk: the model might have some partial exposure from pretraining (Wikipedia contains articles on Tuvalu), or the fact might not be factually isolated (population numbers are related to other statistics like GDP, which could provide indirect learning signals). Fictional facts about non-existent entities (the "Zorgon-Calidus government") eliminate both concerns — there is zero chance of prior exposure, and there are no real-world correlations that could confound learning. The tradeoff is ecological validity: real factual knowledge in pretraining concerns real entities, and the model's behavior on fictional entities might differ (e.g., the model might have learned that "Mars" is a real planet and treat statements about "Martian government" as inherently fictional, affecting how it processes them). The paper implicitly accepts this tradeoff, prioritizing experimental control.

Why log probability rather than generation accuracy? As discussed above, log probability provides a continuous, per-step signal that generation accuracy (correct/incorrect binary) cannot. However, this choice means the paper cannot directly connect its findings to standard downstream evaluation metrics — it doesn't establish what magnitude of log probability increase corresponds to the fact becoming reliably extractable via prompting. The Discussion section (Section 5) acknowledges this limitation: "we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model's accumulation of probability of factual knowledge and the model's generation output." This is a significant gap: the learnability threshold concept is defined in terms of log probability retention, but its practical relevance depends on how log probability maps to generation behavior, which is not established.

Why a 50-step window for local acquisition maxima? The $t_w = 50$ choice is grounded in the AdamW momentum decay, as explained above. A smaller window (e.g., $t_w = 20$ ) would risk missing the true peak — with $\beta_1 = 0.9$ , after 20 steps the gradient contribution has only decayed to $0.9^{20} \approx 0.12$ , so substantial momentum-driven effects would still be unfolding. A larger window (e.g., $t_w = 80$ ) would risk capturing effects from the next injection (which occurs at $t_i + 100$ ), conflating the two. The 50-step window elegantly balances these concerns: at $t_i + 50$ , the momentum contribution is ~0.5%, and there are still 50 steps of buffer before the next injection.

Why not use exact memorization metrics like those in Carlini et al. (2022)? The paper's memorization probes test whether the model can complete a verbatim training sentence given its prefix — this is "exact memorization" in the sense of surface-form reproduction, but it's evaluated via log probability, not via exact match of generated output (which Carlini et al. would require). The log probability approach is more sensitive (can detect partial memorization) but less directly connected to extraction attacks or copyright concerns. This choice reflects the paper's focus on the dynamics of acquisition rather than on measuring the extent of memorization for practical implications.

Why 10 injections spaced 100 steps apart? The number 10 and spacing 100 are design parameters that affect what can be learned from the experiments. With 10 injections over 1,000 steps, the model sees each fact 10 times with a total budget of 10 × 2048 = 20,480 exposures (counting each batch row). This is a "medium-frequency" regime: much more frequent than rare long-tail facts (which might appear once or twice in the entire corpus) but much less frequent than extremely common facts (which might appear hundreds of times). The 100-step spacing (400M tokens between injections, since 100 steps × 4M tokens/step = 400M tokens) is chosen to be long enough that meaningful forgetting occurs between injections (visible in Figure 2 as the downward slopes between injection peaks), but short enough that some accumulation remains at the time of the next injection. Different spacings would reveal different regimes of the forgetting curve, but exploring this systematically would multiply the experimental cost.

Connecting the Metrics to the Research Questions

The three metrics map directly to the paper's three research questions:

RQ1 (How is factual knowledge acquired?): The step-wise log probability trajectories (Figure 2) and the effectivity metric show the micro-acquisition pattern — sharp increases at each injection followed by gradual decay — establishing the "accumulation with forgetting" mechanism.
RQ2 (How is effectivity affected by training conditions?): The effectivity comparisons across pretraining stages, model sizes, and injection scenarios (Figure 3) reveal that model scale improves per-encounter learning gain while pretraining stage does not.
RQ3 (How is knowledge forgotten, and how is this affected by training conditions?): The retainability curves, decay constants, and anticipated x-intercepts (Figures 4–5, Tables 2–3) quantify the power-law forgetting relationship and show how duplication accelerates forgetting while larger batch sizes slow it.

4. Key Insights and Innovations

Innovation 1: Decomposing Knowledge Acquisition into Two Orthogonal, Quantifiable Dimensions

The paper's most fundamental conceptual contribution is the decomposition of factual knowledge acquisition into effectivity (how much log probability the model gains from a single encounter with a fact) and retainability (what fraction of that gain persists over subsequent training). Prior to this work, the field lacked any vocabulary or measurement apparatus for distinguishing these two mechanisms. Studies of knowledge in pretrained models asked what the model knows (Petroni et al., 2019; Roberts et al., 2020) or where knowledge is stored (Dai et al., 2022; Geva et al., 2021; Meng et al., 2022), but never how knowledge accumulates at the level of individual parameter updates. Studies of memorization dynamics (Tirumala et al., 2022) tracked aggregate memorization scores over time but conflated acquisition and retention into a single observed quantity — you could see that memorization increased, but not whether that increase came from strong per-encounter learning (high effectivity) or slow forgetting between encounters (high retainability). This conflation means prior work could not diagnose why a particular fact failed to be learned: was each encounter too weak to move the needle, or was the learning strong but erased by forgetting before the next encounter?

The paper's metric trio — local acquisition maxima, effectivity, and retainability — provides the first operational vocabulary for separating these mechanisms. This is not an incremental refinement of existing metrics but a new diagnostic apparatus. The significance is that it transforms factual knowledge acquisition from a monolithic outcome into a dynamic process with two independently varying components, each potentially influenced by different training factors. The paper immediately demonstrates the utility of this decomposition by showing that model scale improves effectivity while pretraining stage does not (Section 4.2, Figure 3), and that data duplication increases effectivity but also accelerates forgetting, yielding limited net benefit (Section 4.3, Table 2). Without the effectivity/retainability split, both of these findings would be invisible — you would only see the net outcome and wrongly conclude that neither model scale nor duplication mattered much for the final state.

The distinction is analogous to decomposing a chemical reaction rate into activation energy and catalyst presence: knowing the net rate is insufficient for engineering the process; you need to know which factor is the bottleneck. This paper gives the field the conceptual tools to perform that bottleneck analysis for factual knowledge acquisition.

Innovation 2: The Learnability Threshold — A Formal Condition for Whether a Fact Can Be Acquired

Building on the retainability metric, the paper introduces the concept of a learnability threshold: if the interval between consecutive encounters of a fact in the pretraining data exceeds a critical duration (measured in training steps or tokens), the between-encounter forgetting will completely erase any accumulated probability before the next encounter, rendering the fact mathematically unlearnable regardless of how long pretraining continues. This threshold is not an asymptotic trend but a hard boundary — the x-intercept of the retainability curve where $R(q, t) = 0$ — estimated from the power-law fit to the observed forgetting dynamics (Figure 5, Appendix Tables 6 and 11).

Prior work documented that LLMs struggle with long-tail knowledge (Kandpal et al., 2022; Mallen et al., 2022), but characterized this as a statistical tendency — rare facts are harder to learn — rather than as a threshold phenomenon with a mechanistic explanation. The default assumption was that more training data and longer training would help, albeit with diminishing returns. The learnability threshold reframes this: for facts with encounter intervals above the threshold, no amount of additional training on other data will help, because the fact's own encounters are too sparse to overcome the between-encounter forgetting rate. This is a fundamentally different diagnosis than "the model needs more capacity" or "the data distribution is imbalanced." It identifies a temporal constraint on learnability that is independent of model size (within the scales studied — 1B to 7B, the threshold shifts but does not disappear) and independent of total training duration.

The practical implication is sharp: to make long-tail knowledge learnable, you must either increase the frequency of encounters (e.g., through oversampling or data augmentation) to push the encounter interval below the threshold, or increase the threshold itself (e.g., through larger batch sizes, which Section 4.3 shows lengthens the threshold). More pretraining data alone does not solve the problem if it doesn't change the relative frequency of the long-tail facts — a larger corpus might still present a rare fact at the same low rate, leaving it below the threshold. This insight provides a principled target for data engineering efforts that the prior "long-tail is hard" framing did not.

The estimated x-intercepts (Appendix Tables 6, 8, 11) quantify the threshold in log(tokens). For the main OLMo-7B Mid checkpoint with duplication injection, memorization has an x-intercept of 10^11.02 tokens, meaning the model would need to encounter the fact again within roughly 10 billion tokens to avoid complete forgetting. For the reduced batch size experiment, the x-intercepts drop to approximately 10^9.45 tokens (Table 11 vs. Table 6) — a reduction of roughly 40× in log space — meaning the safe interval between encounters shrinks dramatically, making far fewer facts learnable. These are not small effects; they represent order-of-magnitude shifts in what corpus statistics are viable for knowledge acquisition.

Innovation 3: Scaling Pretraining Tokens Improves Performance Without Improving Per-Encounter Learning Efficiency

Perhaps the paper's most counterintuitive finding is that the amount of pretraining data a model has seen does not improve its per-encounter ability to acquire new factual knowledge. Section 4.2 (Figure 3, left panel) shows that effectivity — the immediate log probability gain from seeing a fact — is essentially flat across Early (170B tokens), Mid (500B tokens), and Late (1.5T tokens) checkpoints. The constant learning rate control experiment (Appendix F, Figure 17) confirms this is not an artifact of learning rate decay. A model that has seen 1.5 trillion tokens gains no more from encountering a new fact than a model that has seen only 170 billion.

This challenges an implicit but widespread assumption in the scaling laws literature: that training on more data makes models "better learners" — more efficient at extracting knowledge from text. Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) show that loss improves with data quantity, and one natural interpretation is that the model's learning capability improves over the course of pretraining. The paper demonstrates this interpretation is incorrect for factual knowledge acquisition: longer pretraining does not increase the per-encounter learning rate. Instead, the benefit of more data must come from a statistical mechanism — more facts cross the learnability threshold because the corpus is larger and more diverse, and frequently-appearing facts accumulate more total probability through more encounters, not because each encounter becomes more impactful.

This is a fundamental reframing of what data scaling accomplishes. It suggests that pretraining performance improvements from larger corpora are primarily about coverage (more facts presented above the learnability threshold) and repetition (more accumulation cycles for facts already above threshold), not about an emergent improvement in the model's ability to learn per se. The model's "learning efficiency" saturates early in pretraining, and all subsequent gains come from the statistics of exposure rather than from an improved learning mechanism.

This finding also clarifies why scaling model size (1B to 7B) does improve effectivity (Figure 3, right panel), while scaling tokens does not — larger models have more capacity to absorb knowledge from each encounter, but that capacity does not grow with further training. The qualitative difference between model scaling and data scaling — that they operate through different mechanisms (capacity vs. exposure) — had been hypothesized but never directly demonstrated at the level of individual fact acquisition dynamics.

Innovation 4: Reconciling Data Deduplication Benefits with the Necessity of Repetition Through Differential Forgetting Rates

The paper resolves an apparent tension in the literature: deduplicating pretraining data improves model performance (Lee et al., 2021; Tirumala et al., 2023), but factual knowledge obviously requires multiple exposures to be learned (a single encounter would be fully forgotten according to the retainability curves). If repetition is necessary for learning, why does removing repetitions help?

The answer, revealed by the paper's metric decomposition, is that duplicated and paraphrased exposures have different forgetting dynamics. Section 4.3 (Table 2) shows that the decay constant $a$ — the rate at which acquired knowledge is forgotten — is consistently larger (steeper forgetting) under the duplication injection scenario than under the paraphrase scenario. For OLMo-7B Mid, the decay constants for memorization are 0.25 (duplication) vs. 0.21 (paraphrase); for semantic generalization, 0.25 vs. 0.23; for compositional generalization, 0.20 vs. 0.15. Duplication produces higher immediate effectivity (Figure 3 — the model gains more log probability from seeing an identical passage again), but the acquired knowledge decays faster, such that the net benefit at the end of training is comparable to paraphrase exposure (visible in Figure 2, where the initial gap between duplication and paraphrase curves narrows by t = 2000).

This finding provides a mechanistic explanation for why semantic deduplication (removing near-duplicates, as in SemDeDup; Abbas et al., 2023) might be beneficial above and beyond exact deduplication: presenting the same fact in varied surface forms maintains the repetition benefit for accumulation while avoiding the accelerated forgetting that identical repetitions induce. It also explains Allen-Zhu and Li's (2023) observation that diverse presentation formats aid knowledge extraction — the benefit is not only at retrieval time but at acquisition time, through slower forgetting of knowledge learned from diverse exposures.

This is not an incremental finding about deduplication; it reframes the entire discussion. Prior work framed deduplication as removing "wasted" training on redundant text that provides no new information. This paper shows that duplicated text is not neutral — it actively harms retention of generalized knowledge by increasing the forgetting rate. Deduplication is therefore not just about efficiency (avoiding wasted compute) but about learning quality (preventing accelerated forgetting). This distinction matters because it predicts that the optimal deduplication strategy should be aggressive for verbatim duplicates (which accelerate forgetting without proportional long-term benefit) but permissive for paraphrases (which provide repetition benefits without the forgetting penalty).

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper constructs the FICTIONAL KNOWLEDGE dataset — 120 GPT-4-generated passages describing non-existent entities (e.g., "the Zorgon-Calidus government of Mars"), each accompanied by 15 cloze-style probes: 5 memorization probes (exact sentences from the passage with the final named entity as target span), 5 semantic generalization probes (paraphrases of memorization probes preserving the target span), and 5 compositional generalization probes (requiring integration of facts from multiple sentences). This yields 1,800 probes total. The first 40 knowledge instances (probes 1–600) are allocated to the paraphrase injection scenario, instances 41–80 to the duplication scenario, and instances 81–120 to the once scenario (Section 3, Appendix B). All probes are generated by prompting GPT-4 with few-shot examples and validated (for compositional probes) by an additional GPT-4 check for ambiguity (Appendix C.5).

Base model(s). The paper uses OLMo-1B and OLMo-7B (Groeneveld et al., 2024), selected because OLMo uniquely provides public access to intermediate checkpoints, the exact ordered sequence of pretraining data instances (Dolma v1.5), and the optimizer and scheduler states at each checkpoint — all three are necessary for faithfully resuming pretraining under controlled conditions. Three pretraining stages are compared per model size: Early (177B tokens for 7B, 168B for 1B), Mid (500B for 7B, 494B for 1B), and Late (1.5T for both). The paper argues OLMo is "representative of the capabilities of many contemporary LLMs" (Section 4).

Metrics. The paper uses three derived metrics computed from per-step log probabilities of target spans on cloze probes (Section 3, Definitions 1–3):

Local acquisition maxima (t_LAM): The training step within a 50-step window after an injection where the model's log probability on a probe reaches its peak, defined as $t_{\text{LAM}}(q, i) = \arg\max_{t_i < t \leq t_i + t_w} \ell(q; \theta_t)$ with window size $t_w = 50$ (justified by AdamW's $\beta_1 = 0.9$ momentum decay).
Effectivity (E): The absolute increase in log probability from the pre-injection step to the local acquisition maxima: $E(q, i) = \ell(q; \theta_{t_{\text{LAM}}(q,i)}) - \ell(q; \theta_{t_i})$ , measuring the immediate gain from a single encounter.
Retainability (R): The fraction of total accumulated log probability gain (across all injections) that remains after $t$ steps past the final local acquisition maxima: $R(q, t) = \frac{\ell(q; \theta_{t_{\text{LAM}}(q,N) + t}) - \ell(q; \theta_{t_{\text{pre}}})}{\ell(q; \theta_{t_{\text{LAM}}(q,N)}) - \ell(q; \theta_{t_{\text{pre}}})}$ . The decay constant $a$ is extracted from the linear fit of $R(q, t)$ against $\log(t)$ or $\log(\text{tokens})$ , and the x-intercept of this fit (where $R = 0$ ) represents the point of complete forgetting.

All metrics use IQR-based outlier filtering (factor 1.5), which the paper notes is "particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due to the very small denominator" (Section 3).

Baselines. The paper does not compare against alternative training methods or prior approaches. The experimental design compares within the same model family across conditions: duplication vs. paraphrase vs. once injection scenarios; Early vs. Mid vs. Late pretraining stages; 1B vs. 7B model sizes; batch size 2048 vs. 128. The "once" injection scenario serves as a minimal-exposure baseline, and the duplication scenario represents the status quo of what happens with repeated identical text.

Generation budget / compute accounting. The paper measures training steps and tokens rather than FLOPs. Each training step processes one batch of 2,048 sequences × 2,048 tokens = ~4M tokens. The full experimental run per configuration is 2,500 steps — 1,000 steps of injection phase (10 injections at 100-step intervals, plus any initial injection at step 0) followed by 1,500 steps of forgetting tracking — totaling 10 billion tokens of continued pretraining. At each injection step, the injected knowledge is prepended to every row of the 2,048-row batch (maintaining 2,048 total sequence length by truncating the original Dolma data), meaning the model sees 2,048 simultaneous copies of each fact per injection. The paper reports experiments take "approximately 3 days using 8 80GB A100 GPUs" per configuration.

Cross-validation / statistical protocol. The paper does not employ train/test splits or cross-validation, as it is not training a model for downstream evaluation but measuring training dynamics directly. Statistical uncertainty is reported as standard errors on the decay constant $a$ (e.g., "0.25 ± 0.0019" in Table 2), derived from the linear fit to the ensemble-averaged retainability curve. The $R^2$ of these fits is reported ( $R^2 > 0.80$ for memorization and semantic generalization, $R^2 > 0.65$ for compositional generalization). However, no confidence intervals are reported for effectivity measurements or the extrapolated x-intercepts of retainability, which are point estimates from the fitted power-law parameters.

Main Quantitative Results

Factual Knowledge Acquisition Occurs Through Accumulation with Forgetting

Figure 2 (showing OLMo-7B Mid checkpoint across all three injection scenarios) establishes the paper's foundational empirical pattern. In the duplication scenario (top panel), the average log probability of target spans shows immediate and distinctive increases at each of the 10 injection steps (marked by dotted vertical lines), with memorization probes showing the largest jumps (reaching roughly 0.8–0.9 in average log probability by the final injection), followed by semantic generalization, then compositional generalization. Critically, after each injection peak, the log probability decays back downward before the next injection — the model forgets between encounters. The net acquisition at the end of training (step 2000) represents the accumulated residue after 10 cycles of micro-acquisition and partial forgetting, not a monotonic climb.

In the paraphrase injection scenario (center panel), the same sawtooth pattern appears but with two notable differences: (1) the gap between memorization and semantic generalization "almost disappears" — the two curves largely overlap, in contrast to duplication where memorization pulls clearly ahead; (2) the initial injection produces a smaller jump in memorization (roughly 0.3–0.4 vs. roughly 0.5 for duplication) but subsequent injections maintain comparable levels, and the gap between duplication and paraphrase performance narrows by step 2000.

In the once injection scenario (bottom panel), a single exposure produces a sharp spike in memorization (to roughly 0.5) followed by monotonic decay, with semantic and compositional generalization showing smaller initial spikes and similar decay patterns. By step 2000, all three curves have largely returned toward baseline, illustrating that a single encounter is insufficient for lasting acquisition.

The paper interprets this pattern as direct evidence for the mechanism: "LLMs acquire factual knowledge by accumulating micro-acquisitions with subsequent forgetting each time the model encounters the knowledge during pretraining" (Section 4.1). This pattern is "consistent across all pretraining stages of OLMo-7B we investigate" (Appendix Figures 6, 7). However, OLMo-1B Early checkpoint (Appendix Figure 8) exhibits "much more unstable dynamics" with larger fluctuations, suggesting "pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models."

Effectivity Does Not Improve with Pretraining Stage, But Does Improve with Model Scale

Figure 3 reports average effectivity (across all probes and injection indices) for OLMo-7B across three pretraining stages (left panel) and for 1B vs. 7B at the Mid checkpoint (right panel).

Pretraining stage effect (Figure 3, left): Across all three injection scenarios and all three acquisition depths, effectivity is essentially flat from Early (170B tokens) to Mid (500B) to Late (1.5T). For the duplication scenario, memorization effectivity stays around 0.6–0.7 across all three stages; semantic generalization around 0.4–0.5; compositional generalization around 0.15–0.25. The paraphrase scenario shows a similar flat pattern at slightly lower absolute values. The once scenario shows the highest effectivity (since the first encounter produces the largest gain, and in the once scenario there is only one encounter to average), but also flat across stages.

The paper explicitly states: "the effectivity of fact acquisition does not improve with checkpoints trained with more tokens" (Section 4.2). The constant learning rate control experiment (Appendix F, Figure 17) confirms this is not an artifact of decreasing learning rates — when all three 7B checkpoints are trained with the same fixed learning rate and batch size 128, effectivity remains flat across stages (values around 2.0–2.5 for duplication memorization, substantially higher than the main experiment due to the smaller batch size effect discussed below).

Model scale effect (Figure 3, right): Comparing 1B-Mid vs. 7B-Mid, the 7B model shows consistently higher effectivity across all conditions. For duplication, 7B memorization effectivity is roughly 0.65 vs. 1B at roughly 0.50; for semantic generalization, roughly 0.50 vs. 0.35; for compositional generalization, roughly 0.22 vs. 0.15. The pattern holds in paraphrase and once scenarios as well. The 1B model used the same initial learning rate as 7B for this comparison; Appendix Figure 11 shows results with the original 1B learning rates, where the effectivity gap between 1B and 7B is somewhat smaller but the qualitative direction remains.

Interpretation: The paper concludes that "the benefits of scaling the model size and pretraining tokens are qualitatively different" (Section 4.2) — larger models gain more from each encounter (higher effectivity), but models that have seen more tokens do not become more efficient learners per encounter. This implies that data scaling benefits come from a statistical mechanism (more encounters, wider coverage) rather than from an emergent improvement in per-encounter learning ability.

Forgetting Follows a Power-Law Relationship with Training Steps

Figure 4 plots average retainability $R(q, t)$ against $\log(t)$ (training steps past the final local acquisition maxima) for OLMo-7B Mid under duplication (left) and paraphrase (right) injection. The data points in log-linear space fall approximately along straight lines, with the paper reporting $R^2 > 0.80$ for memorization and semantic generalization and $R^2 > 0.65$ for compositional generalization.

The fitted decay constants $a$ (the absolute slopes of these lines, where $\Delta R \approx -a \cdot \log(t_2/t_1)$ ) reveal several patterns, quantified in Table 2:

Acquisition depth matters for forgetting rate: Compositional generalization consistently shows smaller decay constants (slower forgetting) than memorization and semantic generalization. For OLMo-7B Mid duplication: memorization $a = 0.25$ , semantic $a = 0.25$ , composition $a = 0.20$ . For paraphrase: $a = 0.21$ , $0.23$ , $0.15$ respectively. The paper interprets this as: "the acquisition of compositional generalization accumulates most slowly but is more robust to forgetting" (Section 4.3).
Paraphrase injections produce slower forgetting than duplication: Across all pretraining stages and acquisition depths, the decay constants for paraphrase are consistently smaller than for duplication. For OLMo-7B Early: duplication memorization $a = 0.26$ vs. paraphrase $a = 0.20$ ; Mid: $0.25$ vs. $0.21$ ; Late: $0.20$ vs. $0.18$ . The paper interprets this as explaining the benefits of data deduplication: "deduplication tends to slow the forgetting of generalizing acquired factual knowledge" (Section 4.3).
Late-stage forgetting appears slower, but this is due to learning rate decay: Table 2 shows smaller decay constants for the Late checkpoint in the duplication scenario (e.g., memorization $a = 0.20$ at Late vs. $0.25$ – $0.26$ at Early/Mid). However, the constant learning rate control experiment (Appendix Table 9) shows no such decrease — decay constants remain similar or even slightly increase at Late — confirming that the apparent slower forgetting in the main experiment is an artifact of the reduced learning rate at late pretraining stages, not an intrinsic improvement in retention with more tokens seen.

This power-law relationship holds across both model scales and all pretraining stages (Appendix Figures 12–16, Appendix Tables 7–8 for 1B). The 1B Early checkpoint (Appendix Figure 14) shows poor linear fit ( $R^2 < 0.4$ for the fit, per Appendix Table 7 note), attributed to the unstable training dynamics of that particular checkpoint, but the Mid and Late 1B checkpoints follow the power-law cleanly.

Larger Batch Sizes Enhance Robustness to Forgetting

Figure 5 compares forgetting dynamics between the standard batch size (2048, left panels) and a reduced batch size (128, right panels), both with the same total tokens on the x-axis. The key finding is that reducing batch size by a factor of 16 substantially increases the forgetting rate.

Under duplication with batch size 2048, memorization has decay constant $a = 0.25$ ; under batch size 128, this increases to $a = 0.31$ (comparing Table 2 Mid values with Appendix Table 10 Mid values). The x-intercept (theoretical point of complete forgetting) shifts dramatically: for OLMo-7B Mid duplication memorization under batch size 2048, the anticipated x-intercept is $10^{11.02}$ tokens (Appendix Table 6); under batch size 128, this drops to $10^{9.45}$ tokens (Appendix Table 11) — a decrease of roughly $10^{1.57} \approx 37\times$ in token space. The pattern is consistent across acquisition depths and injection scenarios.

The paper interprets this as: "pretraining with a small batch size reduces the set of learnable knowledge due to accelerated forgetting" (Section 4.3). The mechanism is not about effectivity — Appendix Figure 21 shows that the smaller batch size actually produces higher effectivity (roughly 2.0–2.5 vs. 0.6–0.7 for duplication memorization at batch size 2048), likely because each training step with fewer total tokens produces a less noisy gradient update — but the dramatically faster forgetting more than offsets this gain. The learnability threshold (x-intercept) shrinks by approximately 40×, meaning facts must appear much more frequently to be retained.

The effect is "more pronounced for compositional generalization compared to memorization and semantic generalization" (Section 4.3): under batch size 128, compositional generalization decay constants reach $a = 0.26$ – $0.27$ (Appendix Table 10), which are comparable to or exceed memorization rates at batch size 2048, suggesting that the deeper acquisition type is disproportionately harmed by small batch training.

Forgetting Dynamics Provide a Unified Explanation for Several Observed LLM Behaviors

Section 4.4 synthesizes the quantitative findings into explanations for prior empirical observations:

Why popularity matters for factual knowledge acquisition: The extrapolated x-intercepts of retainability curves represent a learnability threshold — the maximum interval between consecutive encounters of a fact before all accumulated probability is erased. If a fact's encounter interval in the pretraining corpus exceeds this threshold, the fact is "impossible to be decoded as the top-k generation of the model, or learned, regardless of the duration of the pretraining" (Section 4.4). Well-known facts have encounter intervals below this threshold, allowing log probability to accumulate across encounters until the fact becomes reliably decodable. Long-tail facts with intervals above the threshold are fundamentally unlearnable through pretraining alone.

The paper quantifies this threshold for the main experimental configuration (OLMo-7B Mid, batch size 2048): under duplication, the x-intercept is approximately $10^{11.02}$ tokens for memorization, $10^{10.98}$ for semantic generalization, and $10^{11.32}$ for compositional generalization (Appendix Table 6). Under paraphrase, the thresholds shift to $10^{11.37}$ , $10^{10.94}$ , and $10^{11.88}$ tokens respectively. These translate to encounter intervals in the range of tens to hundreds of billions of tokens — consistent with the observation that facts appearing only a handful of times in a trillion-token corpus are poorly learned.

Why deduplication enhances model performance: The smaller decay constants under paraphrase injection (Table 2: $a = 0.21$ – $0.23$ for paraphrase vs. $a = 0.25$ – $0.25$ for duplication in memorization/semantic) mean that presenting facts in diverse surface forms reduces the forgetting rate. Combined with the observation that duplicated text produces higher effectivity for memorization relative to generalization (Figure 3, left: duplication shows a wider gap between memorization and semantic generalization effectivity than paraphrase), the paper argues that duplication both accelerates forgetting and "drives the model to prefer generating memorized contexts compared to generalizing factual knowledge" (Section 4.4). Deduplication therefore improves performance by (1) slowing forgetting of generalized knowledge and (2) preventing memorization from being disproportionately reinforced relative to generalization.

Why larger pretraining corpora help despite flat per-encounter effectivity: Since effectivity does not improve with pretraining stage (Figure 3, left), the observed improvement from data scaling cannot be attributed to models becoming better learners. Instead, the paper hypothesizes that "larger and more diverse pretraining data helps the model performance [because] the model can acquire a broader range of factual knowledge (more knowledge will be presented with an interval shorter than the learnability threshold) since the skewness of the distribution of factual knowledge popularity is likely to be mitigated as the data becomes larger and more diverse" (Section 4.4). In other words, larger corpora shift more facts from the "below threshold" to the "above threshold" regime by increasing their effective frequency.

Ablation Studies and Robustness Checks

Constant learning rate control experiment (Appendix F): To rule out learning rate decay as the explanation for flat effectivity across pretraining stages, all three OLMo-7B checkpoints (Early, Mid, Late) were trained with the same fixed constant learning rate (average of the three original starting rates) and batch size 128. Figure 17 shows effectivity remains flat across stages despite equal learning rates, confirming that "the non-increasing effectivity in pretraining progress is not attributed to the learning rate decay." Appendix Table 9 shows that decay constants also do not decrease for later checkpoints under constant learning rate (unlike Table 2 where Late shows smaller $a$ ), confirming that the apparent slower forgetting at Late in the main experiment was a learning rate artifact.

Model scale comparison with matched learning rate (Figure 3, right vs. Appendix Figure 11): The main 1B vs. 7B comparison uses the 1B Mid checkpoint retrained with the 7B Mid's initial learning rate for fairness. Appendix Figure 11 shows the full 1B results with original learning rates — the qualitative pattern (7B shows higher effectivity) persists, though the absolute gap narrows somewhat.

Effect of injection count on effectivity (Appendix H): The paper measures effectivity separately for each injection index $i = 1, \ldots, 10$ . Figures 27–29 show that effectivity is "relatively constant regardless of the number of previous injections of the knowledge" but "the effectivity is the highest when the model is trained with the injected knowledge for the first time, both in the duplication and paraphrase injection scenarios." This justifies the paper's practice of averaging across all injection indices but also suggests a diminishing-returns dynamic: the first encounter produces the largest gain, and subsequent encounters add smaller, roughly equal increments.

OLMo-1B Early checkpoint instability (Appendix Figure 8, 14): The OLMo-1B Early checkpoint (168B tokens) exhibits "much more drastic fluctuations" in training dynamics compared to later 1B checkpoints and all 7B checkpoints. Appendix Figure 14 shows that the retainability curve for this checkpoint has very poor power-law fit ( $R^2 < 0.4$ ), and the decay constants and x-intercepts are omitted from Table 7 and Table 8. The paper interprets this as evidence that "pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models" (Section 4.1). This is a robustness concern: the paper's quantitative framework (power-law forgetting, decay constants) does not cleanly apply to all checkpoints, and the conditions under which it does apply (sufficient pretraining) are identified post-hoc rather than predicted a priori.

Reduced batch size with re-initialized optimizer (Appendix G): The batch size experiment reinitializes the optimizer state (unlike the main experiment which restores it), which "results in unstable dynamics in early steps, but the dynamics are stabilized soon, and do not harm the model's overall behavior" (Appendix G). Appendix Figures 24–26 show the training dynamics under this configuration. The early-step instability is visible but confined to the first ~200 steps, after which the expected sawtooth pattern emerges. This is a methodological robustness check — the paper demonstrates that the qualitative findings about batch size and forgetting are not driven by optimizer state continuity.

Forgetting dynamics consistency across model sizes (Appendix Figures 15–16): The power-law forgetting pattern observed in 7B (Figure 4) replicates in 1B Mid and Late checkpoints (Appendix Figures 15–16), with comparable $R^2$ values and qualitatively similar decay constants. Appendix Table 7 shows 1B Mid duplication memorization $a = 0.24$ (vs. 7B Mid $a = 0.25$ in Table 2), and paraphrase memorization $a = 0.26$ (vs. 7B $a = 0.21$ ). Notably, for 1B, the paraphrase scenario does not consistently show slower forgetting than duplication — the pattern reverses in some cases — suggesting the beneficial forgetting-slowing effect of diverse presentation may be model-size-dependent. The paper does not comment on or interpret this discrepancy.

Dolma corpus consistency across conditions (Section 3, Appendix D): The forgetting tracking phase (steps 1,000–2,500) uses "a corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes." This controls for potential confounds from the Dolma data distribution affecting forgetting rates differently across experimental configurations.

Critical Assessment

The experiments provide strong support for the paper's central descriptive finding about the mechanism of factual knowledge acquisition — that it occurs through accumulating micro-acquisitions with subsequent forgetting. Figure 2 shows this sawtooth pattern cleanly and consistently across injection scenarios, pretraining stages, and model sizes (Figures 2, 6–10). The qualitative pattern is unambiguous and requires no complex interpretation. However, the paper does not establish whether this sawtooth pattern is specific to the injection methodology (injecting 2,048 copies of a fact simultaneously into a batch) or would persist under more ecologically valid conditions where a fact appears in a single sequence within a batch. This is a meaningful gap because real pretraining facts do not appear in every batch row simultaneously.

The effectivity-result (flat across pretraining stages, scaling with model size) is well-supported by the main data (Figure 3) and the constant learning rate control (Appendix Figure 17). The absence of pretraining-stage effect on effectivity is robust to learning rate confounds. However, the paper only tests two model scales (1B and 7B) and three pretraining stages, which limits the generality of any scaling claims. The finding that effectivity does not improve with pretraining stage might hold only for the particular range of token budgets tested (170B to 1.5T) — it's possible that effectivity improves very early in pretraining (before 170B tokens) and then plateaus, a regime the paper does not study and explicitly acknowledges as a limitation (Appendix A: "we do not analyze the pretraining dynamics at very early stages"). Additionally, the flat effectivity across pretraining stages is measured for novel fictional facts the model has never seen, not for additional encounters of already-known facts, which might show different dynamics.

The power-law forgetting result is the paper's most quantitatively developed finding, but it has several important caveats. First, the power-law relationship is fitted to ensemble-averaged retainability curves, not to per-probe curves — this smooths out individual-level noise but may obscure heterogeneity in forgetting patterns across different types of facts. The $R^2 > 0.65$ for compositional generalization is notably lower than $R^2 > 0.80$ for memorization and semantic generalization, suggesting the power-law is a weaker model for the deepest acquisition type, but the paper treats all three depths under the same power-law framework without discussing why compositional generalization might deviate. Second, the x-intercepts (learnability thresholds) are extrapolated from fits to data covering only ~1,550 steps (the forgetting tracking phase), while the predicted x-intercepts correspond to $10^9$ – $10^{12}$ tokens — many orders of magnitude beyond the observed range. The paper does not discuss the uncertainty of these extrapolations, which could be enormous. Third, the paper does not validate that the log-linear relationship holds over the full range to the x-intercept — it assumes linearity beyond the observed data. If the forgetting curve bends (e.g., asymptotically approaches some non-zero baseline rather than crossing zero), the learnability threshold concept would need revision.

The batch size finding is clean and consistent across acquisition depths: smaller batches accelerate forgetting (Figure 5, Appendix Tables 10–11). However, only two batch sizes are compared (2048 and 128), and the 128-case also reinitializes the optimizer state, which the paper acknowledges causes "unstable dynamics in early steps" (Appendix G). The comparison is therefore not purely about batch size — it conflates batch size with optimizer state continuity. The paper argues the instability "does not harm the model's overall behavior in general," but a cleaner ablation would have kept optimizer state consistent (e.g., by using gradient accumulation to simulate smaller batches without changing the optimizer step frequency) or at least tested an intermediate batch size to establish monotonicity. Without intermediate values, the paper cannot claim that larger batches are monotonically better for retention — only that 2048 is better than 128 by a large margin.

The connection from log probability metrics to generation behavior is entirely absent from the experimental design. The paper defines the learnability threshold in terms of when retainability crosses zero (complete loss of log probability gain), but this definition does not establish what level of log probability is sufficient for a fact to become reliably generated by the model. A fact might have positive retained log probability (above the pre-exposure baseline) but still not be the argmax prediction at generation time. Without calibrating log probability to generation accuracy, the practical significance of the learnability threshold — and by extension, the entire effectivity/retainability framework — remains unclear. The paper acknowledges this gap explicitly in Appendix A: "we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model's accumulation of probability of factual knowledge and the model's generation output."

The single model family limitation is significant: all experiments use OLMo (1B and 7B). While the paper argues OLMo is "representative," the specific dynamics observed — particularly the power-law forgetting exponent and the effectivity saturation point — could depend on OLMo's architecture, training recipe (AdamW with specific hyperparameters), data mixture (Dolma), or learning rate schedule. The paper provides no evidence that these findings generalize to other model families (LLaMA, Gemma, etc.), other architectures (mixture-of-experts, non-transformer models), or other pretraining data distributions. The 1B Early checkpoint instability further suggests that the quantitative framework may only apply after some minimum pretraining threshold that is model-specific.

Missing experiments that would strengthen the paper:

Multiple encounter intervals: The paper uses a fixed 100-step interval between injections. Varying this interval (e.g., 50, 200, 500, 1000 steps) would directly test the learnability threshold concept by showing that facts with intervals above the x-intercept indeed fail to accumulate any net probability. Without this experiment, the learnability threshold remains an extrapolation rather than a verified prediction.
Intermediate batch sizes: Testing batch sizes of 512 or 1024 would establish whether the batch-size effect on forgetting is monotonic and where the diminishing returns set in.
Generation-based evaluation: Running even a small set of probes through greedy/beam-search decoding at different points in the acquisition/forgetting cycle would calibrate the log probability metrics against the behavior that ultimately matters for applications.
Multiple model families: At minimum, replicating a subset of conditions on a different model architecture (e.g., a LLaMA-derived model) would test whether the power-law forgetting pattern and the flat-effectivity result are universal or OLMo-specific.
Longer forgetting tracking: The current forgetting tracking phase (1,500 steps = 6B tokens) captures only the early part of the decay curve. Extending this to 10,000+ steps would reduce the extrapolation distance for the x-intercept estimate and test whether the power-law holds at longer timescales.
Varying the number of injections: The paper uses 10 injections throughout. Testing 2, 5, 20, or 50 injections would reveal whether accumulation saturates or continues linearly with encounter count.

The paper's central claims — accumulation with forgetting, flat effectivity across pretraining stages, power-law forgetting, deduplication slowing forgetting, larger batches improving retention — are supported by the experiments as far as they go, but they go less far than the paper's discussion implies. The learnability threshold framing (Section 4.4) is an insightful synthesis of the quantitative results, but it relies on extrapolated x-intercepts whose uncertainty is unquantified, and it lacks direct experimental verification that facts with above-threshold intervals truly cannot be learned. The explanations for why scaling helps (wider coverage, not better learning) and why deduplication helps (slower forgetting, not just avoiding wasted compute) are compelling hypotheses consistent with the data, but the paper does not run the experiments that would distinguish these explanations from alternatives — for instance, directly manipulating encounter frequency in a realistic data distribution and measuring learning outcomes. The paper's contribution is therefore strongest as a conceptual framework and weakest as a verified predictive model of real pretraining behavior.

6. Limitations and Trade-offs

6.1 The Learnability Threshold Is Extrapolated, Not Directly Measured

The assumption or constraint: The paper's central explanatory concept — the learnability threshold, defined as the x-intercept of the power-law fit to retainability where $R(q, t) = 0$ — is derived by extrapolating from only ~1,500 training steps (6 billion tokens) of forgetting data to predicted x-intercepts in the range of $10^9$ to $10^{12}$ tokens (Appendix Tables 6, 8, 11). This represents an extrapolation spanning 3–6 orders of magnitude beyond the observed data. The paper fits a linear function in log-log space and extends it to the zero-crossing without any validation that the power-law relationship holds at those timescales.

The consequence: If the forgetting curve bends — for example, if retainability asymptotically approaches some non-zero baseline rather than crossing zero, or if the decay rate changes at longer timescales due to learning rate annealing or data distribution shifts — then the extrapolated x-intercept is meaningless, and the learnability threshold concept collapses. The paper's entire explanatory framework for long-tail knowledge failures, deduplication benefits, and batch size effects (Section 4.4) rests on these extrapolated thresholds. Without direct measurement or validation, the quantitative claims about which facts are learnable (e.g., "facts with encounter intervals above $10^{11}$ tokens are unlearnable") are speculative rather than demonstrated. The $R^2$ values for the power-law fits — $R^2 > 0.80$ for memorization and semantic generalization but only $R^2 > 0.65$ for compositional generalization — already indicate that the power-law is a weaker model for the deepest acquisition type, yet the paper extrapolates it to the same extreme distances for all three depths.

What evidence exists in the paper: The x-intercept estimates are provided in Appendix Tables 6, 8, and 11 as point estimates without any confidence intervals or uncertainty quantification. The forgetting tracking phase is limited to 1,500 steps by experimental design; the paper does not discuss whether this duration was constrained by computational resources or was chosen as sufficient, and does not acknowledge the magnitude of the extrapolation gap. Figure 4 and Figure 5 plot the observed data (up to $t \approx 10^6$ steps) and show the linear fit extending to the x-intercept in log space, but the fitted lines beyond the data range are visual extrapolations without empirical support. The paper states in the limitations (Appendix A) that "future works exploring these would help us to further enhance our understanding of LLMs," but does not specifically call out the extrapolation issue as a limitation.

Mitigation status: The paper does not address this limitation. It does not run longer forgetting tracking experiments, does not test whether the power-law holds at longer intervals (e.g., by spacing injections further apart in the once scenario), and does not report confidence intervals or prediction intervals on the x-intercept estimates. A direct test of the learnability threshold would involve injecting facts at variable encounter intervals and demonstrating that those with above-threshold intervals indeed fail to accumulate net log probability, but this experiment is not performed.

6.2 The Injection Methodology Is Ecologically Invalid for Real Pretraining

The assumption or constraint: The paper's controlled injection protocol places the fictional knowledge into every row of a 2,048-sequence batch simultaneously at each injection step. This means each "encounter" consists of the model seeing 2,048 copies of the same fact in a single gradient step, paired with different (truncated) Dolma sequences. In real pretraining, a fact appears in a single sequence within a batch — or at most, a handful of sequences if the fact is extremely common — and its gradient contribution is diluted by the 2,047 other unrelated sequences in the batch. The paper's batch modification procedure (Appendix D) explicitly acknowledges this design choice: "Each injected knowledge is short enough to fit into one pretraining sequence in the batch, and we fill the rest of the sequence with the original sequence in the batch."

The consequence: The measured effectivity values are likely overestimates of what a real single-sequence encounter would produce, because the simultaneous injection across all batch rows creates a coordinated gradient signal 2,048× stronger than a realistic single occurrence. The sawtooth acquisition pattern in Figure 2 — sharp jumps of 0.3–0.5 in log probability per injection — may be much smoother and less pronounced under ecologically valid conditions where each fact encounter is diluted by the batch. More importantly, the relative comparisons between conditions (duplication vs. paraphrase, 1B vs. 7B) might not scale linearly from the high-exposure regime to the low-exposure regime — for example, the finding that paraphrase has lower effectivity but slower forgetting might behave differently when each encounter's gradient is 2,048× weaker, since the signal-to-noise ratio of the gradient update changes fundamentally. The paper cannot claim that its quantitative metrics (decay constants, effectivity values, learnability threshold positions) translate directly to realistic pretraining settings.

What evidence exists in the paper: The paper does not vary the injection density within the batch (e.g., injecting into 1, 10, 100, or 2048 rows) to test whether the acquisition dynamics scale with the number of simultaneous copies. The batch size experiment (Section 4.3, Appendix G) changes the total batch size (from 2048 to 128) but keeps the injection density at 100% of rows — the model still sees the injected fact in every row, just with fewer total rows. So even in the reduced batch size experiment, each "encounter" consists of 128 simultaneous copies, which is still 128× stronger than a realistic single occurrence. The paper does not acknowledge this ecological validity gap as a limitation.

Mitigation status: The paper does not address this limitation. It does not run an ablation where the fact is injected into a single row per batch, nor does it discuss how the simultaneous-injection design might affect the interpretation of effectivity and retainability metrics. The justification for the design choice — "to deviate as little as possible from the original pretraining data distribution" — addresses batch size and sequence length but not the per-fact exposure density within a batch.

6.3 Single Model Family and Single Dataset Limit Generality

The assumption or constraint: All experiments use OLMo-1B and OLMo-7B (Groeneveld et al., 2024), trained on the Dolma corpus (Soldaini et al., 2024), with the AdamW optimizer and a cosine learning rate schedule. The paper states in Appendix A that "future works exploring these would help us to further enhance our understanding of LLMs," but frames the OLMo models as representative: "we believe this model is representative of the capabilities of many contemporary LLMs" (Section 4).

The consequence: The quantitative findings — the power-law decay constants, the estimated learnability thresholds, the effectivity values — are likely sensitive to model architecture, training recipe, and data distribution. Different model families use different architectures (dense vs. mixture-of-experts), different optimizers (AdamW vs. Adafactor vs. Lion), different learning rate schedules (cosine vs. linear vs. constant-with-cooldown), and different data mixtures — all of which could affect the forgetting dynamics. The finding that paraphrased injection produces slower forgetting than duplication (Table 2) might depend on OLMo's specific training dynamics; a model with different in-context learning properties or different robustness to repeated tokens might show a different pattern. The 1B Early checkpoint instability (Appendix Figures 8, 14) — where the power-law fit fails entirely ( $R^2 < 0.4$ ) — demonstrates that even within the same model family, the paper's quantitative framework does not cleanly apply to all training stages, suggesting the framework's applicability conditions are narrower than the paper implies.

What evidence exists in the paper: The paper provides no cross-model-family comparisons. The finding that OLMo-1B Early shows unstable dynamics (Section 4.1, Appendix E.2) is interpreted as suggesting that "pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models," but this post-hoc interpretation is not tested across other model families. The difference in decay constants between duplication and paraphrase is consistent for OLMo-7B but not for OLMo-1B (compare Table 2 and Appendix Table 7: for 1B Mid, paraphrase memorization $a = 0.26$ vs. duplication $a = 0.24$ — the pattern reverses), but the paper does not discuss this discrepancy.

Mitigation status: The paper acknowledges in Appendix A that "we do not analyze the pretraining dynamics at very early stages, which can exhibit significantly different behaviors" and "we do not study the effect of training batch size and learning rate on the dynamics of factual knowledge acquisition across multiple values." However, it does not acknowledge the single-model-family limitation as a threat to generality, nor does it discuss how the findings might or might not transfer to other architectures, optimizers, or data distributions. The claim that OLMo is "representative" is asserted without evidence.

6.4 No Calibration Between Log Probability Metrics and Generation Behavior

The assumption or constraint: The paper's entire analytical framework — effectivity, retainability, decay constants, learnability thresholds — is defined in terms of the log probability the model assigns to the target span of cloze probes. The paper explicitly states in Appendix A: "we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model's accumulation of probability of factual knowledge and the model's generation output." The learnability threshold is defined as the point where retainability crosses zero (i.e., the log probability gain relative to baseline is completely lost), but there is no established mapping between a given log probability level and whether the fact becomes reliably decodable via greedy decoding, beam search, or any standard generation method.

The consequence: The practical significance of the entire quantitative framework is uncertain. A fact might have a positive retained log probability (above the pre-exposure baseline) but still rank below many incorrect alternatives at generation time, making it functionally "unknown" despite being partially "acquired" by the paper's metric. Conversely, a fact with zero retained log probability might still be generated correctly if the baseline (pre-exposure) probability was already high enough — for instance, if the target span is syntactically or semantically predictable from context without factual knowledge. The learnability threshold concept — "the point such that an LLM cannot learn the knowledge presented with intervals longer than that threshold" (Section 4.4) — is framed in terms of generation behavior ("impossible to be decoded as the top-k generation of the model"), but this connection is asserted without evidence. The paper cannot claim to have identified generation-relevant learnability thresholds without calibrating log probability against actual generation accuracy.

What evidence exists in the paper: The paper provides no analysis of generation outputs at any point. The probes are evaluated purely via log probability, and no experiment reports accuracy (exact match of generated text against target span) at any training step or checkpoint. The Discussion section (Section 5) acknowledges this gap in passing but treats it as a future direction rather than a limitation that affects the interpretation of the current results.

Mitigation status: The paper does not address this gap. It acknowledges it only in Appendix A as an area for future work, not in the main text as a limitation that affects the interpretation of the learnability threshold or the practical relevance of the effectivity/retainability decomposition. The conceptual edifice of the paper — particularly the claims about which facts are learnable and why deduplication and batch size matter — stands without validation against the behavior that ultimately matters for downstream applications.

6.5 The Experiments Capture Only the Initial Decay Regime and a Fixed Encounter Interval

The assumption or constraint: The paper's experimental design fixes two parameters that are likely critical to the forgetting dynamics: (1) the interval between consecutive injections is always 100 training steps (corresponding to 400 million tokens between encounters), and (2) the forgetting tracking phase extends only 1,500 steps (6 billion tokens) beyond the final injection. The number of injections is fixed at 10. These choices are not varied systematically — the paper studies three injection scenarios (duplication, paraphrase, once) but only one temporal schedule.

The consequence: Three important questions are left unanswered by this fixed design. First, does accumulation saturate? With 10 injections, the log probability peaks in Figure 2 appear to continue increasing with each subsequent injection, but it is unknown whether 20, 50, or 100 injections would continue to produce additive gains or would eventually plateau as the model's capacity to represent the fact saturates. If saturation occurs after some number of encounters, the benefit of further repetition is zero regardless of forgetting rates, which would fundamentally change the interpretation of what data scaling accomplishes. Second, does the encounter interval interact with the forgetting rate? The paper's learnability threshold concept implies that if the interval between encounters exceeds the x-intercept, nothing accumulates. But what happens at intermediate intervals — 200 steps, 500 steps, 1000 steps? The forgetting curve might be power-law in the short-term but transition to a different regime at longer intervals, or the accumulation might follow a qualitatively different pattern when encounters are sparse. Third, does the forgetting curve remain power-law at longer timescales? As discussed in Section 6.1, the extrapolation from 1,500 steps to billions of tokens is unvalidated. The fixed experimental design cannot distinguish between a power-law that continues indefinitely and one that bends or plateaus at timescales beyond the observation window.

What evidence exists in the paper: The paper provides no ablation over encounter intervals. The 100-step interval is justified implicitly by the need for each injection's local acquisition maxima window (50 steps) to close before the next injection begins, but no alternative intervals are tested. The forgetting tracking duration of 1,500 steps appears to be set by the total continued training budget (2,500 steps = 10 billion tokens), with no discussion of whether this is sufficient to characterize the long-term decay. The paper's own finding that the 1B Early checkpoint shows "unstable dynamics" (Appendix Figures 8, 14) and poor power-law fit suggests that the forgetting curve's shape may depend on pretraining stage in ways not captured by the fixed-schedule design.

Mitigation status: The paper does not discuss the fixed-interval limitation or the lack of saturation analysis. It does not suggest varying the encounter interval or the number of injections as future work. The Discussion section (Section 5) focuses on the implications of the findings rather than the experimental design constraints that limit those implications.

6.6 The Difficulty Estimation and Strategy Selection Overhead Is Not Quantified for This Methodology

The assumption or constraint: While this paper does not deploy a "strategy" in the sense of a production system, its methodology requires substantial per-configuration computation that is not amortized or discussed in terms of resource requirements. Each experimental configuration (one model scale × one pretraining stage × one injection scenario) requires 2,500 steps of continued pretraining at a batch size of 2,048 sequences × 2,048 tokens per step, totaling 10 billion tokens of training plus 1,800 probes evaluated at every step (producing ~81 million log probability measurements across all configurations). The paper reports this takes "approximately 3 days using 8 80GB A100 GPUs" per configuration, but with 18 base configurations (3 stages × 2 model sizes × 3 injection scenarios) plus the constant learning rate and reduced batch size variants, the total compute is substantial (on the order of 50–100 GPU-days minimum). This compute investment produces a set of measured metrics (decay constants, effectivity values, x-intercepts) that are specific to OLMo trained on Dolma and cannot be assumed to generalize without re-measurement.

The consequence: Any practitioner wanting to apply this paper's framework to their own model family, data distribution, or training recipe would need to replicate the entire experimental pipeline — constructing fictional knowledge datasets, resuming pretraining with controlled injections, running thousands of steps of per-step probe evaluation — at a compute cost comparable to the original study. The paper does not provide a lightweight proxy for estimating decay constants, effectivity, or learnability thresholds without full-scale experimentation. The framework is therefore an analytical contribution for academic understanding rather than a practical tool that can be deployed in training pipeline decisions. The paper's explanations for deduplication benefits, batch size effects, and long-tail failures are post-hoc interpretations of observed dynamics; they do not constitute a predictive model that could guide hyperparameter selection without re-running the entire analysis for each new setting.

What evidence exists in the paper: The paper reports the per-configuration compute requirements (3 days on 8 A100s) but does not discuss the total computational cost of the study or the cost of applying the framework to new settings. The methodological description in Section 3 and Appendix D provides enough detail for replication but makes clear that the pipeline is compute-intensive and requires specific infrastructure (access to intermediate checkpoints with optimizer states, the exact Dolma data sequence, and substantial GPU resources).

Mitigation status: The paper does not address this as a limitation. It does not propose simplified measurement protocols, proxy metrics, or extrapolation methods that could estimate the key quantities (decay constants, effectivity) without full-scale injection experiments. The Discussion section (Section 5) frames the work as an initial attempt to understand training dynamics, implicitly accepting that the methodology is research-grade rather than deployment-grade, but does not explicitly discuss the cost barrier to practical application of the framework.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around factual knowledge in LLMs from a static, post-hoc inventory of what models know to a dynamic, mechanistic understanding of how they come to know it. Prior work — from Petroni et al. (2019) through Kandpal et al. (2022) — treated factual knowledge acquisition as a black box: you put training data in, you get knowledge out, and the intervening process was opaque. This paper cracks that black box open by providing the first operational vocabulary and measurement apparatus for decomposing knowledge acquisition into two distinct, quantifiable dimensions: effectivity (the immediate gain from seeing a fact) and retainability (the persistence of that gain over subsequent training). This is not an incremental refinement of existing metrics — it is a new diagnostic lens that transforms factual knowledge from a monolithic outcome into a dynamic process with independently varying components.

The significance of this decomposition is that it explains previously puzzling empirical phenomena through a single mechanistic framework. Why do LLMs struggle with long-tail knowledge? Because rare facts have encounter intervals exceeding the learnability threshold — the point where between-encounter forgetting completely erases any accumulated probability. Why does deduplication improve model performance? Because duplicated text accelerates forgetting of generalized knowledge, while paraphrased presentations of the same facts produce slower decay. Why do larger batch sizes help? Because they extend the learnability threshold, allowing facts to survive longer gaps between encounters. These are not three separate explanations for three separate phenomena — they are three manifestations of the same underlying mechanism: the interaction between accumulation and forgetting as characterized by the effectivity/retainability decomposition. The paper reconciles what appeared to be contradictory findings (deduplication helps, but facts need repetition to be learned) by showing that the form of repetition matters — verbatim duplication and paraphrased re-exposure have qualitatively different effects on the forgetting rate.

The finding that scaling pretraining tokens does not improve per-encounter learning efficiency (Figure 3, left; Section 4.2) challenges an implicit assumption in the scaling laws community: that models become "better learners" as they see more data. The paper demonstrates that a model trained on 1.5T tokens gains no more from encountering a new fact than a model trained on only 170B tokens. This reframes what data scaling accomplishes — not an emergent improvement in learning capability, but a statistical benefit: larger corpora present more facts above the learnability threshold and provide more accumulation cycles for facts already above it. This distinction has practical implications for how we think about data engineering: expanding a pretraining corpus helps not because it makes the model smarter per token, but because it makes more facts learnable by shifting their effective frequency above the forgetting threshold. This suggests that data diversity and repetition management are more important than sheer data volume for factual knowledge coverage — a 10× larger corpus that keeps the same relative frequency distribution for rare facts provides no benefit for those facts.

The paper also establishes that model scaling and data scaling operate through qualitatively different mechanisms for factual knowledge acquisition. Larger models show higher effectivity (Figure 3, right) — they extract more knowledge from each encounter — but models trained on more tokens do not. This means that scaling laws for factual knowledge are not a single phenomenon but the combination of two distinct effects: capacity-driven improvement in per-encounter learning (model scaling) and coverage-driven improvement in the number of facts crossing the learnability threshold (data scaling). This decomposition provides a more nuanced framework for predicting when scaling model size versus scaling data quantity will improve factual knowledge performance — model scaling helps for all facts (higher per-encounter gain), while data scaling helps only for facts whose encounter frequency can be shifted above the threshold.

Methodologically, the paper demonstrates the power of controlled injection experiments for studying training dynamics at fine granularity. By injecting fictional knowledge that the model has definitely never seen before and tracking per-step log probabilities on memorization and generalization probes, the paper achieves a level of experimental control that retroactive analysis of real pretraining data cannot match. This methodology — which requires access to intermediate checkpoints, optimizer states, and the exact training data sequence — points toward a research paradigm where understanding training dynamics requires not just analyzing final models but actively intervening in the training process. The fact that OLMo provides all necessary artifacts (checkpoints, data order, optimizer states) is not incidental — it enabled this entire research program, and suggests that the field should demand similar transparency from other model releases to enable replication and extension of these findings across model families.

The paper identifies verifier over-optimization of a different kind: not over-optimization of a learned reward model as in RLHF, but over-optimization of memorization at the expense of generalization when training on repeated identical text. The observation that duplicated data accelerates forgetting of generalized knowledge while increasing immediate memorization effectivity (Table 2, Figure 3) implies that pretraining on verbatim repeated text is not just wasteful — it actively damages the model's ability to generalize facts. This reframes data deduplication from an efficiency concern (avoiding wasted compute on redundant data) to a learning quality concern (preventing the model from developing a preference for surface-form memorization over semantic generalization). This has implications for how aggressively we should deduplicate and whether semantic deduplication (removing near-duplicates, not just exact duplicates) may be necessary to preserve the beneficial effects of diverse re-exposure while avoiding the harmful effects of verbatim repetition.

Follow-Up Research This Work Enables

Directly testing the learnability threshold by varying injection interval. The paper's most important hypothesis — that there exists a critical encounter interval beyond which facts are unlearnable — is derived from extrapolation of power-law fits to observed forgetting curves. A direct test would vary the interval between injections systematically (e.g., 50, 100, 200, 500, 1000, 5000 steps, corresponding to 200M to 20B tokens between encounters in a batch size 2048 setting) and measure whether net acquisition after a fixed number of exposures drops to zero when the interval exceeds the predicted x-intercept. For OLMo-7B Mid, the predicted x-intercept for duplication memorization is approximately 10^11.02 tokens (Appendix Table 6), which corresponds to roughly 25,000 training steps at batch size 2048. An experiment that injects the same fact 10 times with intervals of 5,000, 10,000, 25,000, and 50,000 steps would directly validate or falsify the threshold hypothesis. If net acquisition drops sharply around the predicted threshold, the paper's extrapolation is validated; if accumulation persists even at intervals well beyond the predicted x-intercept, the power-law model must be revised (perhaps the decay curve bends and asymptotes above zero, or the extrapolation from short-term data is unreliable). This experiment would transform the learnability threshold from an extrapolated hypothesis to a measured phenomenon.

Calibrating log probability gains against generation accuracy. The paper acknowledges as a key limitation that it "do[es] not investigate the exact relationship between the model's accumulation of probability of factual knowledge and the model's generation output" (Section 5). A natural follow-up would measure, at each step of the acquisition/forgetting cycle, both the log probability of the target span and whether greedy decoding from the probe context actually produces the correct answer. This would establish the mapping between the paper's continuous metric space and the binary outcome that matters for applications. Specifically: at what log probability does the fact become reliably decodable? Is the threshold sharp (once log probability exceeds some value, accuracy jumps from near-zero to near-one) or gradual (accuracy improves smoothly with log probability)? This calibration would anchor the effectivity and retainability metrics in practical terms — for example, "an effectivity of 0.5 per encounter means roughly N encounters are needed before the fact is generated correctly 90% of the time." The experiment could be run on the existing experimental infrastructure by adding a generation pass at each evaluation step for a subset of probes, without requiring additional training runs.

Replication across model families and architectures. All results in this paper are specific to OLMo-1B and OLMo-7B trained on Dolma with AdamW. A high-priority replication would test whether the power-law forgetting relationship, the flat effectivity across pretraining stages, and the learnability threshold concept generalize to other model families (LLaMA-2, Gemma, Mistral) and other architectures (mixture-of-experts models, models trained with different optimizers such as Adafactor or Lion). The minimum viable replication would run the duplication and paraphrase injection scenarios on a Mid-stage checkpoint of a comparably-sized LLaMA-2-7B model (using a subset of the RedPajama or C4 data as the equivalent of Dolma for continued pretraining), measuring whether the decay constants and x-intercepts fall in the same range as OLMo-7B. If the power-law relationship holds across architectures with similar exponents, it suggests a universal property of gradient-based language model training. If the exponents vary substantially, it suggests the learnability threshold is architecture- or optimizer-dependent, which would have implications for hardware-aware training design (e.g., choosing an optimizer partly based on its effect on factual knowledge retention).

Testing whether the learnability threshold can be raised through intervention. The paper shows that larger batch sizes increase the learnability threshold (Appendix Tables 6 vs. 11: ~40× longer estimated threshold for batch size 2048 vs. 128). A natural extension would test other interventions hypothesized to slow forgetting: gradient clipping thresholds, weight decay settings, learning rate warmup and decay schedules, and data ordering strategies (e.g., spaced repetition scheduling, where facts are presented with increasing intervals following a Leitner-system-like pattern). For each intervention, measuring the decay constant and x-intercept under the duplication injection scenario would quantify its effect on the forgetting dynamics independently of its effect on effectivity. An intervention that slows forgetting without reducing effectivity would directly expand the set of learnable facts by pushing the learnability threshold outward. This research direction could lead to training recipes optimized not just for final loss but for the range of factual knowledge the model can retain — a metric that current pretraining recipes do not explicitly target.

Studying the interaction between knowledge relatedness and forgetting. The paper's fictional knowledge instances are independent — knowledge about the "Zorgon-Calidus government of Mars" does not relate to knowledge about whatever other fictional entities appear in the same injection set. Real factual knowledge is highly interconnected: learning that Paris is the capital of France is related to learning that France is a country in Europe, that the Seine flows through Paris, that the Eiffel Tower is in Paris, etc. An important extension would test whether related facts exhibit different forgetting dynamics than isolated facts. The experimental design could inject clusters of related fictional knowledge (e.g., a set of facts about a fictional country — its capital, its leader, its primary export, its founding date, its neighboring countries) and measure whether the decay constants for probes within a cluster are lower (slower forgetting) than for probes about isolated facts. This would test whether semantic relatedness provides a form of "mutual reinforcement" that slows forgetting — if so, it would suggest that pretraining data should be organized to present related facts in temporal proximity, and that the learnability threshold is not a fixed property of the model but depends on the knowledge graph structure of the training data.

Longer-horizon forgetting tracking to validate the power-law extrapolation. The paper's most significant quantitative extrapolation — estimating the x-intercept at which all acquired knowledge is lost — is based on only ~1,500 steps (6B tokens) of forgetting data. A critical stress-test would extend the forgetting tracking phase to 10,000–50,000 steps (40B–200B tokens) for a subset of configurations (e.g., OLMo-7B Mid duplication and paraphrase) and test whether the power-law relationship between log(tokens) and retainability holds over this 10–50× longer horizon. If the linear fit in log-log space holds over the extended range, the extrapolated x-intercepts gain credibility. If the curve bends — for example, transitioning to a slower decay rate at longer timescales, or asymptotically approaching a non-zero baseline — the learnability threshold concept would need revision to account for a "persistent residue" of knowledge that survives even very long intervals without reinforcement. This experiment is computationally expensive (50,000 steps × 4M tokens/step = 200B tokens of continued pretraining per configuration, roughly 20× the current experimental budget) but is the most direct way to validate or falsify the central quantitative claim.

Practical Applications and Downstream Use Cases

Guiding data deduplication strategy for pretraining corpora. The paper provides a mechanistic justification for aggressive deduplication that goes beyond the traditional "avoid wasting compute on redundant data" argument. Since duplicated text accelerates forgetting of generalized factual knowledge (Table 2: decay constants of 0.25–0.26 for duplication vs. 0.18–0.21 for paraphrase in OLMo-7B), pretraining on deduplicated data should produce models that not only avoid memorizing training strings but also retain generalized factual knowledge longer. The practical implication: when building a pretraining corpus, exact deduplication should be applied aggressively (remove all verbatim duplicates), while near-duplicate detection (identifying passages that convey the same factual content in different words) should be applied more judiciously — removing exact duplicates while preserving paraphrased re-expressions may be optimal, since paraphrases provide repetition benefits without the accelerated forgetting penalty. The estimated x-intercepts from Appendix Table 6 provide rough targets for minimum acceptable encounter intervals: with deduplicated data (modeled by paraphrase), facts need to appear within ~10^11–10^12 tokens to avoid complete forgetting. Corpus engineers can use this as a guideline for assessing whether important factual domains have sufficient coverage.

Batch size selection for maximizing factual knowledge retention in domain-specific continued pretraining. When continuing pretraining a general-purpose LLM on domain-specific data (e.g., medical literature, legal documents, internal enterprise knowledge), the choice of batch size is typically driven by hardware constraints. This paper suggests batch size is also a knowledge retention hyperparameter: the 16× reduction from batch size 2048 to 128 caused a ~40× reduction in the estimated learnability threshold (Appendix Tables 6 vs. 11), meaning that with small batch sizes, domain-specific facts must appear much more frequently to be retained. For a continued pretraining run where the goal is to teach the model new factual knowledge (not just adapt its style), the paper's results argue for using the largest batch size that fits in memory, even if this requires gradient accumulation across fewer steps — since what matters for retention is the interval between encounters in tokens, not in optimizer steps. Concretely: if you have a fixed 10B-token domain corpus and must decide between batch size 2048 (fewer steps, each covering more tokens) and batch size 128 (more steps, each covering fewer tokens), the paper's decay constants and x-intercepts strongly favor the larger batch size for long-term retention of the domain knowledge. A practitioner could estimate the expected encounter interval for key facts in their corpus and compare against the learnability threshold for their chosen batch size, selecting a batch size that keeps most facts above the threshold.

Diagnosing and mitigating long-tail knowledge failures in deployed models. The learnability threshold concept provides a diagnostic framework for understanding why a deployed LLM fails on specific factual queries about rare entities. Rather than treating failure as an opaque consequence of "not enough training data," a practitioner can estimate the encounter frequency of the relevant facts in the pretraining corpus and compare against the paper's estimated thresholds (~10^11 tokens for deduplicated data at batch size 2048). If the fact's encounter interval exceeds the threshold, the failure is predictable and cannot be fixed by further pretraining on the same data distribution — the model would need explicit upsampling of the rare facts (to push their encounter interval below the threshold) or retrieval augmentation (to bypass the parametric acquisition bottleneck entirely). This diagnostic capability is actionable: it distinguishes between facts that are "hard but learnable" (encounter interval slightly above threshold — solvable by moderate oversampling) and facts that are "fundamentally unlearnable through pretraining alone" (encounter interval far above threshold — requiring retrieval or structured knowledge injection). For a production system serving knowledge-intensive queries, this enables a principled division of labor between parametric memory (for facts above the threshold) and retrieval-based memory (for facts below the threshold), rather than the current heuristic approach.