Quantifying the Effect of Test Set Contamination on Generative Evaluations

ArXiv: 2601.04301

🎯 Pitch

Even a single leaked MATH test problem in pretraining data lets models beat the theoretical best possible loss on clean data, shattering the assumption that small contamination is harmless. High-temperature sampling acts as a 'truth serum' that collapses contamination gains by up to 40×, exposing the performance as brittle memorization rather than real reasoning. The authors also fixed a bug in a popular evaluation library that was silently deflating math scores.

1. Executive Summary

This paper studies how test set contamination affects generative evaluations—where models must produce long-form solutions rather than selecting from options—by pretraining 34M–344M parameter Qwen 3 models on web data contaminated with 0–3162 replicas of the MATH test set for math reasoning, and then analyzing performance through the full model lifecycle of pretraining, additional training, and inference. The authors characterize contamination-driven performance through three mechanisms: scaling law analysis of irreducible error (fitting $L(C, R) = E(R) + C_0(R) \cdot C^{-\alpha(R)}$ and finding a single replica achieves lower loss than the uncontaminated corpus's irreducible error), overtraining and supervised finetuning interactions (overtraining on fresh data diluting contamination effects while SFT on the training set improves performance at low contamination but degrades it at high contamination), and memorization survival regimes (identifying three distinct regimes—Exponentially Fast Decoherence, Brittle Memorization, and Deterministic Lock-In—governed by solution length and sampling temperature, where high temperature acts as a "truth serum" that collapses contamination gains by a factor of up to 40×). The paper further identifies and fixes a critical implementation error in the EleutherAI LM Evaluation Harness that previously underreported Math Verify scores, establishing that contamination-driven performance is brittle verbatim memorization rather than robust mathematical generalization and collapses under rephrased or perturbed problem variants.

2. Context and Motivation

The Core Problem: Contamination Effects on Generative Benchmarks Are Poorly Understood

Test set contamination—the accidental or intentional inclusion of benchmark evaluation data in a model's pretraining corpus—has emerged as one of the most consequential threats to trustworthy AI evaluation. When a model has already seen the questions it is being tested on, its performance ceases to measure generalization and instead reflects some mixture of reasoning ability and memorization. This breaks the fundamental contract of benchmarking: that test scores proxy for capabilities the model can deploy on genuinely novel problems.

The field has developed substantial understanding of how contamination affects discriminative evaluations—tasks where models select from provided answer choices, such as multiple-choice question answering (MMLU, Hendrycks et al., 2021a), classification (SST-2, Socher et al., 2013), or fill-in-the-blank comprehension (CNN/Daily Mail, Nallapati et al., 2016). These tasks share a structural property: the model need only place higher probability mass on the correct token from a small candidate set (often just 2–4 tokens), and the generative burden is minimal. Prior controlled contamination studies (Magar & Schwartz, 2022; Jiang et al., 2024; Oren et al., 2023; Yao et al., 2024; Bordt et al., 2025) have systematically quantified how many benchmark replicas produce what performance inflation, how overtraining interacts with leakage, and what detection methods can flag contaminated models—all within this discriminative paradigm.

However, the field is undergoing a fundamental shift in evaluation methodology. With the advent of reasoning models (OpenAI o1, Gemini 2.5, and others; OpenAI et al., 2024a; Google Gemini Team et al., 2025; Xu et al., 2025) and the increasing sophistication of language model capabilities, benchmarks increasingly demand that models generate answers rather than choose them. The MATH benchmark (Hendrycks et al., 2021b), which asks models to produce complete mathematical derivations ending in boxed answers, exemplifies this trend: solutions span tens to thousands of tokens, requiring the model to maintain a coherent reasoning chain without straying from the correct path. This generative setting introduces dynamics that are simply absent from discriminative tasks.

The central question this paper addresses is deceptively simple: do the contamination dynamics established for discriminative evaluations transfer to generative evaluations, or does the requirement of producing long-form solutions fundamentally alter how contamination manifests? As the authors state in Section 1:

"Whether test set contamination has the same effect on generative evaluation as on discriminative evaluations is unclear."

This is not a minor nuance. Generative evaluations introduce at least three new considerations that have no analog in multiple-choice settings:

• Sampling temperature (Ackley et al., 1985): At temperature 0 (greedy decoding), the model deterministically outputs the most probable next token; as temperature increases, the model increasingly samples from lower-probability alternatives. A memorized solution might survive greedy decoding but collapse under any stochasticity if the model's confidence in the memorized path is marginal.
• Solution length: Maintaining a coherent memorized chain over 1000+ tokens is fundamentally different from selecting among four candidate answers, each perhaps 2–5 tokens long. Each additional token introduces a risk of decoherence—the model might know the answer but fail to faithfully reproduce the solution.
• The survival process: Generative success is a product over tokens: the model must get every single token right (or at least mathematically equivalent) for the solution to verify. This makes generative evaluation a "weakest link" problem where failure at any step cascades, in contrast to discriminative tasks where the decision is made once at a single token position.

These dynamics mean that contamination on generative benchmarks involves a competition between two opposing forces: the model's increasing familiarity with the memorized sequence (which grows with contamination dose and context length) versus the accumulating risk of decoherence (which grows with sequence length and sampling temperature). Characterizing this competition mathematically and empirically is the paper's primary theoretical contribution.

Why This Problem Matters: Practical and Theoretical Significance

The practical stakes are immediate and high. MATH is one of the most widely used benchmarks for assessing mathematical reasoning in language models, and its scores are routinely cited in model cards, research papers, and industry reports as evidence of capability. If contamination inflates these scores in ways that are not understood, the entire edifice of claims about mathematical reasoning ability—including from frontier models—rests on potentially fragile ground. The authors' discovery of a bug in the EleutherAI LM Evaluation Harness (Appendix C) that caused the benchmark's own gold reference solutions to score only ~70% underscores that even the measurement infrastructure has been unreliable, compounding the contamination problem.

The theoretical significance is equally important. The paper identifies that contamination-driven performance is not generalization—it is brittle, verbatim memorization that collapses under rephrasing or numerical perturbation (Table 1). This finding has deep implications for how we interpret benchmark scores. A model scoring 90% on MATH after being contaminated with 1000 test set replicas has not learned mathematics; it has learned to reproduce specific token sequences. Distinguishing between these two sources of performance—memorization versus reasoning—is essential for understanding what capabilities models actually possess.

Furthermore, the paper's framework for analyzing memorization as a survival process governed by temperature and sequence length provides a theoretical language for reasoning about generative contamination that did not previously exist. The identification of three distinct memorization regimes (Exponentially Fast Decoherence, Brittle Memorization, Deterministic Lock-In) in Section 5 offers a principled way to predict when contamination will or will not translate to inflated benchmark scores.

Where Prior Approaches Fall Short

The paper identifies several gaps in the existing literature that motivate its investigation.

1. The discriminative bias in contamination research. Prior controlled contamination studies have overwhelmingly used discriminative benchmarks. Magar & Schwartz (2022) used SST-2 (sentiment classification, 2 classes). Jiang et al. (2024) used SST-2, MMLU (MCQA, 4 choices), SQuAD (extractive QA), and CNN/Daily Mail (fill-in-the-middle). Oren et al. (2023) used 7 MCQA benchmarks and only 1 generative benchmark (GSM8K). Yao et al. (2024) used 3 MCQA benchmarks. Bordt et al. (2025) used 7 MCQA benchmarks. As the authors note:

"While foundational, these investigations have predominantly focused on discriminative benchmarks like classification or multiple-choice question-answering (MCQA)."

GSM8K, the one generative benchmark in Oren et al. (2023), involves relatively short solutions (typically 50–150 tokens) compared to MATH (where solutions range from ~15 to ~2000 tokens), meaning the length-dependent dynamics the paper identifies would be largely invisible in that setting.

2. No framework for how inference-time parameters modulate contamination. Prior work has studied contamination as a property of the training process—how many replicas, what model size, what training duration—but has largely ignored that contamination manifests through the mechanics of generation at inference time. Sampling temperature, top-k, top-p, and solution length all interact with memorization in ways that are not captured by simply measuring test set cross-entropy or greedy decoding accuracy. The paper frames this as a fundamental gap:

"Generative evaluations require the model to produce solutions spanning tens-to-thousands of tokens without straying from the memorized path, and introduce new considerations such as the sampling temperature, the sampling algorithm (e.g., top-k, top-p) and the solution length."

3. No characterization of the survival process for memorized sequences. When a model generates a 1000-token solution by memorization, it must "survive" the risk of decoherence at each token. Prior work has not modeled this as a probabilistic process, nor identified the functional forms that govern how cumulative survival probability decays with sequence length under different contamination regimes. This gap matters because it determines when contamination will produce observable performance gains and when it will remain latent (the model knows the answer but cannot faithfully reproduce it).

4. Incomplete understanding of how additional training interacts with contamination. Bordt et al. (2025) studied overtraining with fresh data for discriminative benchmarks, finding that it reduces contamination effects. Kocyigit (2025) studied SFT after contamination, finding it generally helps. But neither study examined these interactions in the generative setting, nor did they identify the counterintuitive finding that SFT can hurt performance for highly contaminated models (Section 4, Finding #5).

5. No application of scaling laws to quantify the "value" of contamination. While neural scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) have been extensively used to predict model performance as a function of compute, they have not been applied to quantify how much pretraining compute contamination "buys" a model. The paper's scaling law analysis fills this gap, providing a formal answer to the question: how much uncontaminated pretraining compute would be required to match the test loss achieved by training on R replicas of the test set? The answer—that a single replica can achieve lower loss than the irreducible error of uncontaminated training—has profound implications for benchmark integrity.

How the Paper Positions Itself

The paper positions itself as the first comprehensive quantification of contamination mechanics in the generative regime, spanning the full model lifecycle: pretraining → additional training → inference. This lifecycle approach is deliberate and distinguishes the work from prior studies that examine contamination at a single stage.

In the pretraining stage, the paper establishes baselines and uses scaling laws to quantify the "economic" value of contamination—how much compute it effectively simulates. This connects contamination research to the scaling laws literature, providing a formal language for discussing the severity of leakage.

In the additional training stage, the paper examines how two common post-pretraining interventions—overtraining on fresh data and supervised finetuning—interact with contamination. This addresses practical deployment concerns: if a contaminated model undergoes standard post-training, do the contamination effects persist, diminish, or transform? The finding that SFT has opposite effects depending on contamination level (helping at low contamination, hurting at high contamination) reveals a previously unrecognized interaction that has direct implications for model development pipelines.

In the inference stage, the paper introduces the novel framework of memorization as a survival process governed by temperature and sequence length, identifying three distinct regimes. This is the paper's most theoretically novel contribution, providing a predictive framework for when contamination-driven performance will or will not survive the stochasticity of generation.

The paper does not claim to solve contamination—it does not propose new detection methods, decontamination procedures, or benchmark designs. Instead, it aims to provide the foundational understanding that such solutions will require. By characterizing the mechanics of generative contamination with the same rigor that prior work applied to discriminative settings, the paper enables future research to develop targeted interventions (e.g., temperature-based auditing, length-stratified evaluation, training protocols that minimize memorization while preserving generalization).

The paper also explicitly connects to—and in some cases contradicts—findings from related domains. It notes that its scaling law result (a single replica breaking the irreducible error barrier) "potentially contradicts Huang et al. (2024)'s claim that single-shot verbatim memorization is an 'illusion' and Hayes et al. (2025)'s claim that membership inference attacks are limited on pre-trained LLMs, with AUC asymptoting to ~0.689" (Section 3). This suggests that MATH problems are distributionally distinct from the data used in those studies (FineWeb-Edu-Dedup), highlighting that contamination dynamics may be dataset-dependent—a nuance that future work must address.

3. Technical Approach

3.1 Reader Orientation

This paper builds a controlled experimental pipeline for measuring how test set contamination affects generative evaluations by pretraining dozens of language models from scratch on deliberately contaminated web data, then systematically analyzing performance through scaling laws, additional training interventions, and inference-time parameter sweeps. The core idea is that to understand contamination in generative settings—where models must produce long, coherent solutions rather than selecting from options—you need to control the contamination dose precisely, track performance across the full model lifecycle (pretraining → additional training → inference), and develop mathematical frameworks that characterize memorization as a probabilistic survival process governed by sequence length and sampling temperature, since generative success requires maintaining a coherent trajectory over potentially thousands of tokens.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental system has five major components, arranged as a pipeline through the model lifecycle:

1. Contaminated Pretraining Corpus Construction: For each experimental condition (model size × number of test set replicas), the authors construct a pretraining corpus by mixing a fixed token budget of clean web data (FineWeb-Edu-Dedup, Penedo et al., 2024) with $R$ replicated copies of the MATH test set. Total training tokens are held constant across contamination levels, isolating the effect of contamination dose from the effect of additional data.

2. Model Pretraining: Transformer-based causal language models (Qwen 3 architecture, Yang et al., 2025a) are pretrained from random initialization at sizes ranging from 34M to 344M parameters using compute-optimal token budgets (20 tokens per parameter, following Hoffmann et al., 2022). One model is trained for each (model size, number of replicas) pair.

3. Additional Training (Overtraining & SFT): For a subset of configurations, models undergo further training: either overtraining on fresh data (training beyond the compute-optimal token budget with only new, non-repeated tokens) or supervised finetuning on the MATH training set to study how standard post-training interventions interact with contamination.

4. Evaluation Metrics: Two metrics are computed on the MATH test set: Math Verify score (fraction of generated solutions whose final answer is mathematically equivalent to the ground-truth boxed answer, using a corrected implementation that fixes a previously undetected bug in the EleutherAI LM Evaluation Harness) and cross-entropy loss on the gold reference solutions under teacher forcing (measuring how well the model predicts the correct solution token by token).

5. Inference-Time Analysis Framework: For the memorization survival analysis, the authors evaluate models under temperature-only sampling across multiple temperatures (0 to 1.5) and bin problems by solution length, then fit per-token negative log-likelihood trajectories to characterize how memorization probability decays with sequence position under different contamination regimes.

Information flows as follows: pretraining corpus (clean + contaminant) → model training → checkpoint evaluation → (optionally) additional training → evaluation → scaling law fitting and survival analysis. The key design principle is controlled variation: every factor that might influence contamination effects (model size, contamination dose, training duration, temperature, solution length) is systematically varied so that its isolated contribution can be measured.

3.3 Roadmap for the Deep Dive

• First, the contaminated pretraining corpus construction, because the contamination dose is the fundamental independent variable and the method for ensuring fair comparison across conditions (constant total tokens) is critical to interpreting all subsequent results.
• Second, the model architecture and training configuration, since the scaling law analysis (Section 3) depends on the compute budget and model size being well-specified.
• Third, the evaluation metrics (Math Verify and cross-entropy), including the critical bug fix in the evaluation harness, because measurement validity is a prerequisite for all downstream analysis.
• Fourth, the scaling law fitting procedure (Equation 1 in the paper) that quantifies how contamination reduces irreducible error, since this is the paper's primary tool for assessing contamination severity.
• Fifth, the additional training protocols (overtraining and SFT), which introduce interactions between contamination and standard post-pretraining interventions.
• Sixth, the inference-time analysis framework (temperature sweeps, length binning, per-token NLL fitting, and the survival probability derivation), which is the most mathematically involved component and requires building on the pretraining baselines established earlier.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a controlled experimental analysis paper whose core technical approach is to pretrain models on deliberately contaminated corpora, systematically vary experimental conditions, and fit mathematical models (scaling laws, power-law NLL trajectories, survival probability functions) to characterize how contamination manifests in generative evaluations across the model lifecycle.

Contaminated Pretraining Corpus Construction

Objective: Create pretraining corpora where the only difference between conditions is the number of replicated copies of the MATH test set, with all other factors (total tokens, data distribution, training procedure) held constant.

Data sources. The clean web data comes from FineWeb-Edu-Dedup (Penedo et al., 2024), described as "a high quality web crawl corpus." This serves as the base pretraining distribution—representative of the kind of web-scale data used to train frontier models, but filtered for educational quality. The contaminant is the MATH test set (Hendrycks et al., 2021b), consisting of 5,000 competition mathematics problems with solutions. Under the Qwen 3 tokenizer, the MATH test set contains approximately 1.4 million tokens.

Formatting. The MATH test set is formatted using the template from EleutherAI's LM Evaluation Harness:

"Problem: {problem}\n\nSolution: {solution}"

This template matches the format that the evaluation harness expects, meaning the model sees test problems in exactly the same format during both pretraining (if contaminated) and evaluation. This is a deliberate choice: it maximizes the chance that contamination will be effective, providing an upper bound on contamination severity. Real-world contamination might involve different formatting, which could reduce memorization effectiveness—the authors are studying the worst case.

Token budget control. The critical design choice is holding total training tokens constant across contamination levels. For a given model size $N$ and compute-optimal token budget $D_{\text{total}} = 20 \times N$ (following Hoffmann et al., 2022), the authors:

1. Replicate the tokenized MATH test set $R$ times (where $R \in \{0, 1, 3, 10, 32, 100, 316, 1000, 3162\}$, logarithmically spaced).
2. Compute the remaining token budget as $D_{\text{corpus}} = D_{\text{total}} - R \times |\text{MATH test set}|$.
3. Sample documents from FineWeb-Edu-Dedup to fill $D_{\text{corpus}}$ tokens.
4. Shuffle the combined dataset.

This procedure ensures that every condition receives exactly the same number of total training tokens. Without this control, increasing contamination would also increase total tokens, making it impossible to separate the effect of contamination dose (more exposure to the test set) from the effect of more training data in general. By holding $D_{\text{total}}$ constant, the clean portion of the corpus shrinks as contamination increases—the model sees fewer unique web documents but more copies of the test set. This tradeoff is essential for isolating contamination effects.

The contamination dose spectrum. The choice of $R$ values is logarithmically uniform from 1 to 3162 replicas. This range spans several orders of magnitude, from essentially undetectable contamination (a single copy in billions of tokens of web data) to extreme over-representation where the test set makes up a substantial fraction of training tokens. At $R = 3162$ with a 34M parameter model ($D_{\text{total}} = 680$M tokens), the MATH test set replicas would consume $3162 \times 1.4\text{M} \approx 4.4$B tokens, but the total budget is only 680M—meaning the corpus is constructed by first adding the replicas and then truncating to the total budget. At high $R$, the corpus becomes dominated by MATH test set tokens, which is an extreme but informative condition for studying memorization saturation.

Shuffling. Documents are shuffled after mixing, meaning MATH test set replicas are randomly interspersed throughout the pretraining data rather than appearing in contiguous blocks. This matches how contamination would typically occur in web-scale pretraining (benchmark data scattered across many web pages) rather than the continued pretraining setup used by Kocyigit (2025), where contaminated data is added in a distinct phase.

Truncation. Each sequence is truncated to a maximum length of 2048 tokens and terminated with an EOS token. This is a standard pretraining practice and ensures that model context windows are exercised up to 2048 positions.

Why this approach over alternatives. An alternative would be to continue-pretrain (Jin et al., 2022; Jang et al., 2022) an already-trained base model on contaminated data, as Kocyigit (2025) did. The authors instead pretrain from scratch with contamination uniformly distributed throughout training. This choice probably reflects a desire to study contamination as an inherent property of the pretraining process rather than as a post-hoc injection. When contamination is present from the start, the model's optimization trajectory and learned representations are shaped by the contaminated distribution throughout training, which may produce different memorization patterns than adding contamination at the end. The paper does not directly compare these approaches, but the distinction is methodologically important.

Model Architecture and Pretraining Configuration

Objective: Pretrain transformer language models at multiple scales to enable scaling law analysis (how contamination effects grow with model size) and to provide models for all subsequent experiments.

Architecture. All models use the Qwen 3 architecture (Yang et al., 2025a), a decoder-only causal language model based on the standard Transformer (Vaswani et al., 2017). The specific configurations are:

The intermediate size for feed-forward layers follows Qwen 3's formula:

$\text{intermediate size} = 256 \cdot \left\lfloor \frac{255 + \lfloor 8 \cdot \text{hidden size} / 3 \rfloor}{256} \right\rfloor$

where $\lfloor \cdot \rfloor$ denotes floor division (rounding down to the nearest integer). This formula ensures the feed-forward dimension is a multiple of 256 while approximating the standard 8/3 expansion ratio.

Why this architecture. Qwen 3 is a contemporary architecture representative of modern open-weight models. Using it ensures results are applicable to current practice. The specific scaling pattern (relationship between depth, width, and parameter count) follows Qwen 3's established scaling curve rather than an arbitrary grid.

Compute budget. Each model is pretrained with 20 tokens per parameter, following the Chinchilla compute-optimal scaling law (Hoffmann et al., 2022). For a model with $N$ parameters, the total training tokens are:

$D = 20 \times N$

For example, the 34M model trains on $20 \times 34\text{M} = 680$M tokens, while the 344M model trains on $20 \times 344\text{M} = 6.88$B tokens.

Why 20 tokens per parameter. Hoffmann et al. (2022) found that, for compute-optimal training, the number of training tokens should scale approximately linearly with model size, with a ratio of roughly 20 tokens per parameter. Training with fewer tokens per parameter would be compute-suboptimal (leaving model capacity unused); training with more would be compute-superoptimal (the "overtrained" regime studied separately in Section 4). Using compute-optimal budgets establishes a standardized baseline where model capacity and data quantity are balanced.

Pretraining compute measurement. The authors approximate pretraining compute (floating point operations, FLOP) using the standard formula from Kaplan et al. (2020):

$C \approx 6 N D$

where $N$ is the number of model parameters and $D$ is the number of training tokens. The factor of 6 comes from the forward pass requiring approximately $2ND$ FLOPs (one multiply and one add per weight per token) and the backward pass requiring approximately $4ND$ FLOPs (roughly twice the forward pass cost due to gradient computations for both activations and weights). This is an approximation—it ignores embedding layers, attention computation details, and efficiency optimizations like Flash Attention—but provides a consistent relative measure across model sizes.

Optimizer and learning rate. The authors use AdamW (Loshchilov & Hutter, 2019) with HuggingFace defaults: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$, weight decay = 0. Weight decay being set to 0 is notable—AdamW is typically used with weight decay (the "W" stands for weight decay regularization), but here it is used purely for its adaptive learning rate properties.

The learning rate schedule uses linear warmup for 250 steps followed by cosine annealing to zero. This is a standard schedule that provides stable early training (warmup) and smooth convergence (cosine decay).

Batch size scaling. Following Shuai et al. (2024), the batch size scales with total training tokens:

$B = 3.24 \times 10^3 \times D^{0.264}$

where $B$ is the number of tokens per optimizer step and $D$ is the total training tokens. This means larger training runs use proportionally larger batch sizes, which is both computationally efficient (better GPU utilization) and theoretically motivated (larger models benefit from larger batches due to reduced gradient noise).

The learning rate scales with the square root of the batch size:

$\eta = 10^{-6} \times \sqrt{B}$

This linear scaling rule is a common heuristic: when batch size increases by a factor of $k$, the learning rate should increase by roughly $\sqrt{k}$ to maintain similar optimization dynamics, since larger batches provide more accurate gradient estimates that can tolerate larger steps.

Gradient clipping. Gradients are clipped to a maximum norm of 1.0, preventing individual batches from causing destructively large parameter updates.

Precision and efficiency. All models use bfloat16 precision (a 16-bit floating point format with the same exponent range as 32-bit float but reduced mantissa precision, providing numerical stability for training without the overhead of full precision) and Flash Attention 2 (Dao, 2023), an optimized attention implementation that reduces memory usage and computation time.

Distributed training. Training uses PyTorch's DistributedDataParallel (DDP) with the NCCL backend for multi-GPU parallelism. All experiments are logged to Weights & Biases (a cloud-based experiment tracking platform) and trained models are uploaded to HuggingFace Hub (a model repository) for reproducibility.

Scale rationale. The model size range (34M to 344M) spans roughly an order of magnitude. This is smaller than frontier models (which range from 7B to 400B+ parameters) due to computational constraints—each experimental condition requires training a model from scratch, and the full sweep involves 5 model sizes × 9 contamination levels = 45 pretraining runs just for the base experiment, plus additional runs for overtraining and SFT. The authors argue that results at this scale provide insights into contamination mechanics that may extrapolate to larger models, but they acknowledge this as a limitation (Section 6).

Evaluation Metrics

Objective: Measure model performance on the MATH test set using two complementary metrics that capture different aspects of contamination-driven memorization.

Metric 1: Math Verify Score. Math Verify (Kydlicek et al., 2025) is defined as:

$\text{Math Verify} = \frac{1}{N_{\text{problems}}} \sum_{i=1}^{N_{\text{problems}}} \mathbf{1}\left[\text{verify}\left(\text{extract}(s_i), a_i^*\right) = \text{True}\right]$

where $N_{\text{problems}} = 5000$ is the number of test problems, $s_i$ is the model's generated solution for problem $i$, $\text{extract}(s_i)$ parses the $\boxed{}$ notation from the solution to extract the model's claimed answer, $a_i^*$ is the ground-truth answer from the MATH dataset, and $\text{verify}(\cdot, \cdot)$ is a mathematical equivalence checker (handling symbolic expressions, numerical tolerances, and common formatting variations). The indicator function returns 1 if the extracted answer is mathematically equivalent to the ground truth and 0 otherwise.

What it measures: the fraction of problems for which the model generates a solution whose boxed final answer is mathematically correct. This is a "harsh" metric that requires the model to not only know the answer but to faithfully reproduce the solution path (or at least arrive at the correct boxed expression). A model that knows the answer but can't generate the correct derivation will fail.

Why this metric for contamination studies: Math Verify captures the end-to-end generative success—did the model produce a verifiably correct solution? This is exactly what contamination threatens to inflate. However, it has a critical limitation: it does not distinguish between a model that genuinely reasons through the problem and one that reproduces a memorized solution verbatim. Both would score 1.0 under Math Verify, which is why the paper supplements it with cross-entropy and with the rephrased/perturbed evaluations (Table 1) that expose memorization.

Greedy decoding baseline. Initial evaluations (Figures 1, 2, 3, 4) use greedy decoding (temperature $T = 0$, also called "temperature-only sampling" with $T \to 0$). At $T = 0$, the model deterministically selects the highest-probability next token at each step. This maximizes the chance of reproducing memorized sequences, since any stochasticity might cause the model to deviate from the memorized path. Using greedy decoding establishes an upper bound on contamination-driven performance.

Bug fix: the Math Verify scoring correction. During experiments, the authors discovered a critical bug in the EleutherAI LM Evaluation Harness (Gao et al., 2024) that caused Math Verify scores to be systematically underreported. The bug is described in Appendix C.

The original implementation in the minerva_math task called remove_boxed() on extracted answers before passing them to math_verify.parse():

res = verify(parse(doc["answer"]), parse(candidates))

where doc["answer"] had already been processed to strip the $\boxed{}$ wrapper. The math_verify.parse() function relies on the $\boxed{}$ notation to properly identify and extract mathematical expressions. Without it:

• parse("\dfrac{9}{7}") returns empty (parsing fails because there is no $\boxed{}$ marker to identify the expression boundaries)
• parse("\boxed{\dfrac{9}{7}}") correctly returns the parsed expression

This bug caused Math Verify to return scores of 0 even for mathematically correct answers when the solution used $\boxed{}$ notation (which is the standard format). As a diagnostic, the authors evaluated the benchmark's own gold reference solutions—solutions that are, by definition, correct—and observed Math Verify scores of only approximately 70%. A correct scoring function on correct solutions should return 100%. This 30% failure rate is direct evidence that the scoring mechanism was flawed independently of model quality.

The fix changes the evaluation to pass the full solution text (with $\boxed{}$ intact) to math_verify.parse():

res = verify(gold=parse(doc["solution"]), target=parse(candidates))

By using doc["solution"] (the complete solution including $\boxed{}$) rather than the pre-processed doc["answer"] (with $\boxed{}$ stripped), the parser can correctly identify math expressions and verify equivalence. This bug was present in versions of the evaluation harness prior to v0.4.8 (August 2025). The authors state:

"This suggests that research reporting minerva_math scores with the Eval Harness prior to task version 3.0 may have reported incorrect scores."

Implications for contamination research. This bug is particularly significant for contamination studies because it could mask contamination effects—if correct memorized solutions are being scored as incorrect due to a parsing bug, then contamination-driven performance gains would be underestimated. The correction is essential for the paper's results to be reliable. Models that memorized the correct $\boxed{}$-formatted answers would have been penalized by the bug, making it harder to detect contamination. The fact that the paper still finds strong contamination effects despite this indicates that the true effects (had the bug been absent) would be even larger.

Metric 2: Cross-Entropy Loss on Gold Reference Solutions. The second metric is the cross-entropy loss of the gold reference solutions under the model, computed under teacher forcing:

$L_{\text{test}} = -\frac{1}{T} \sum_{t=1}^{T} \log P(s_t^* | s_{<t}^*, p)$

where $T$ is the length of the reference solution in tokens, $s_t^*$ is the $t$-th token of the gold reference solution, $s_{<t}^*$ is the prefix of the reference solution up to token $t-1$, $p$ is the problem text, and $P(\cdot | \cdot)$ is the model's predicted token probability. In teacher forcing, at each step $t$, the model receives the true prefix $s_{<t}^*$ as context (even though its own generation might have diverged) and is evaluated on how much probability mass it assigns to the correct next token $s_t^*$. The loss is averaged over all tokens in all test problems.

What it measures: how well the model predicts the correct solution tokens given the correct prefix. This is a "local" measure of memorization—it asks "does the model know what the next token should be?"—rather than the "global" measure of whether the model can produce the entire solution from scratch under its own generation. A model with perfect local memorization ($L_{\text{test}} \approx 0$) would assign probability 1 to every correct token, but might still fail to generate the full solution at $T > 0$ if small errors compound.

Why cross-entropy for contamination studies. Cross-entropy under teacher forcing is useful because it factors out the "generation decoherence" problem and isolates memorization strength. If a model has strongly memorized a solution, it should achieve very low cross-entropy on that solution even if it cannot reliably generate the whole thing due to stochasticity. The paper uses cross-entropy as the primary metric for pretraining scaling laws (Section 3, Finding #3) because it provides a smooth, continuous signal even at very low contamination levels where Math Verify scores might still be near zero. Schaeffer et al. (2025b) previously demonstrated that cross-entropy on generative evaluations follows predictable scaling laws during pretraining, making it suitable for the scaling law analysis.

Relationship between the two metrics. Math Verify and cross-entropy capture different aspects of contamination. Math Verify measures end-to-end generative success (did the model produce a verifiably correct solution?), while cross-entropy measures local prediction accuracy (does the model know what comes next given the correct prefix?). In Appendix Figure 11, the authors show that Math Verify score correlates strongly with pretraining test cross-entropy at low temperatures, but the correlation weakens significantly at higher temperatures and falls to near zero for temperatures above 1.0. This makes sense: at high temperature, even a model that knows the solution well (low cross-entropy) may fail to generate it faithfully (low Math Verify) due to stochastic deviations.

Rephrased and perturbed evaluations. To test whether contamination-driven performance is generalization or memorization, the authors evaluate contaminated models on modified versions of the MATH test set:

1. Rephrased: Problems are rewritten to preserve the numerical values and logical structure but alter the linguistic surface form. For example, "Find the value of $x$ if $2x + 3 = 7$" might become "Determine $x$ given that $2x + 3 = 7$."
2. Perturbed: Problems retain the original wording but change the numerical values and correct answers. For example, "$2x + 3 = 7$" might become "$2x + 3 = 11$."

If contamination-driven performance derives from genuine mathematical reasoning, the model should perform similarly on rephrased and perturbed problems (since the underlying math is the same). If it derives from verbatim memorization, performance should collapse, because the specific token sequences have changed. Table 1 shows that performance collapses to near zero across all contamination levels for both conditions, supporting the memorization interpretation.

Scaling Law Fitting Procedure

Objective: Quantify how test set contamination reduces the irreducible cross-entropy loss on the MATH test set, and estimate how much pretraining compute contamination effectively "simulates."

The scaling law functional form. For each contamination level $R \in \{0, 1, 3, 10, 32, 100, 316\}$, the authors fit a parametric function relating cross-entropy loss $L$ on the MATH test set to pretraining compute $C$:

$L(C, R) = E(R) + C_0(R) \cdot C^{-\alpha(R)}$

where $C = 6 N D$ is the pretraining compute measured in floating point operations (FLOP), $E(R)$ is the irreducible error for contamination level $R$ (the minimum achievable loss as compute goes to infinity), $C_0(R)$ is the compute prefactor (controlling the initial loss at low compute, with units of loss × FLOP$^{\alpha}$), and $\alpha(R)$ is the compute exponent (controlling how quickly loss decreases with additional compute, with units of 1/log(FLOP)). Each of these three parameters is fit separately for each contamination level $R$, yielding three functions $E(R)$, $C_0(R)$, $\alpha(R)$.

What it computes: given a contamination level $R$ and a pretraining compute budget $C$, the function predicts the expected cross-entropy loss on the MATH test set. The three terms decompose the loss into a component that cannot be reduced by more compute (the irreducible error $E(R)$), a component that decays with compute (the reducible error $C_0(R) \cdot C^{-\alpha(R)}$), and the rate of that decay ($\alpha(R)$ controls how many orders of magnitude of loss reduction you get per order of magnitude of compute increase).

Why this form. This is the standard neural scaling law functional form from Kaplan et al. (2020) and Hoffmann et al. (2022), who found that language model loss follows a power-law relationship with compute: $L(C) = E + A \cdot C^{-\alpha}$. The power law captures the empirical observation that each doubling of compute reduces loss by a roughly constant factor (the "diminishing returns" pattern). Alternative functional forms (exponential decay, logarithmic decay) would not capture this behavior. The additive decomposition into irreducible and reducible error follows from the observation that even with infinite compute, some loss remains due to inherent uncertainty in the data (the entropy of natural language), while the reducible portion reflects the model's capacity to learn the training distribution.

Fitting procedure. The authors fit the scaling law separately for each contamination level $R$ using the cross-entropy losses of their pretrained models at different compute budgets (different model sizes). For each $R$, they have 5 data points (one per model size: 34M, 62M, 93M, 153M, 344M), each corresponding to a different compute budget $C = 6 N \cdot (20N)$. The three parameters $E(R)$, $C_0(R)$, $\alpha(R)$ are fit to minimize the squared error between predicted and observed losses.

The irreducible error of the uncontaminated corpus. For $R = 0$ (no contamination), the fitted irreducible error is:

$E(0) = 3.594$

This means that even with infinite compute on the FineWeb-Edu-Dedup corpus, the model would achieve a cross-entropy of approximately 3.594 nats per token on MATH test set solutions. This reflects the fundamental difficulty of predicting mathematical text from general web training—the model never sees math-specific patterns and cannot reduce uncertainty below this level.

How contamination reduces irreducible error. As contamination increases, the irreducible error $E(R)$ drops dramatically:

$E(316) = 0.0347$

This is nearly a 100× reduction from the uncontaminated case. At $E \approx 0.035$, the model achieves near-perfect prediction of MATH solution tokens—it has essentially memorized the test set.

The key finding: a single replica breaks the irreducible error barrier. Figure 2 shows that for almost all model sizes and contamination levels $R \geq 1$, the achieved cross-entropy loss is lower than $E(0) = 3.594$. This is the paper's most dramatic scaling law result. Formally:

$L(C, R=1) < E(R=0) \quad \text{for sufficiently large } C$

where "sufficiently large" means any compute budget above roughly $10^{17}$ FLOP (achieved by the 93M+ models). In operational terms: a model trained on a corpus containing just a single copy of the MATH test set achieves lower test loss than a model could ever achieve with infinite compute on the uncontaminated corpus.

Why this is a "fundamental breach." The irreducible error $E(0)$ represents the asymptotic limit—the best possible performance under the assumption that the model perfectly learns the training distribution but the training distribution simply does not contain the test patterns. A single replica of the test set adds those patterns to the training distribution, allowing the model to cross a barrier that no amount of scale on clean data could overcome. As the authors state:

"Under standard scaling law assumptions—specifically, that the scaling law extrapolates infinitely—a contaminated pretraining corpus can buy more than an 'infinite' amount of pretraining compute relative to pretraining on an uncontaminated pretraining corpus."

This is a precise quantitative statement about the severity of even minimal contamination. If a benchmark maintainer accidentally includes a single copy of a test problem in a web crawl used for pretraining, a model trained on that crawl can achieve performance on that problem that would be mathematically impossible for any model trained on clean data alone, regardless of scale.

What the scaling law parameters reveal. The bottom panel of Figure 2 shows how $E(R)$, $C_0(R)$, and $\alpha(R)$ vary with $R$:

• $E(R)$ drops sharply from 3.594 at $R=0$ to 0.0347 at $R=316$, with most of the reduction occurring between $R=10$ and $R=100$. This is the primary driver of contamination-driven performance gains.
• $C_0(R)$ increases with $R$, meaning that contaminated models benefit more from additional compute—the prefactor amplifies the reducible component. This makes intuitive sense: when the test set is in the training data, spending more compute to better learn that data yields larger returns on the test set.
• $\alpha(R)$ is roughly constant around 0.3–0.5, meaning that the rate of learning (how quickly loss decays with compute) is similar across contamination levels. Contamination mainly shifts the asymptotic floor rather than changing the learning dynamics.

The average fitting error is under $10^{-2}$ for all $R$. This indicates that the three-parameter power-law form fits the data well, supporting the use of scaling laws to extrapolate. However, the extrapolation to infinite compute is inherently speculative—the functional form might break down at scales far beyond the fitted range.

Comparison to Huang et al. (2024) and Hayes et al. (2025). The paper explicitly flags that this result potentially contradicts prior work:

"This conclusion potentially contradicts Huang et al. (2024)'s claim that single-shot verbatim memorization is an 'illusion' and Hayes et al. (2025)'s claim that membership inference attacks are limited on pre-trained LLMs, with AUC asymptoting to ~0.689."

The authors hypothesize that MATH may be distributionally different from FineWeb-Edu-Dedup in a way that makes test set contamination easier to detect. MATH consists of structured mathematical text with distinctive formatting ($\boxed{}$ notation, LaTeX math, consistent problem-solution structure), which may be more salient to the model than the natural language text studied in prior memorization work. This possible explanation is left to future work.

Additional Training: Overtraining Protocol

Objective: Study how training beyond the compute-optimal token budget with fresh (non-repeated) data interacts with pretraining contamination.

Overtraining definition. Following Sardana et al. (2024) and Gadre et al. (2024), the authors define an overtraining multiplier $m$ that scales the number of training tokens beyond the compute-optimal budget:

$D(m, N) = m \times 20 \times N$

where $m = 1$ is the compute-optimal training (20 tokens per parameter, the baseline used in Section 3), and $m > 1$ is the overtrained regime. The additional tokens for $m > 1$ are "new, fresh, non-repeated tokens"—they come from FineWeb-Edu-Dedup but are additional documents not seen during the baseline $m = 1$ training. This is distinct from training for multiple epochs on the same data, where documents are repeated.

Swept values. $m \in \{1, 2, 4, 8, 16\}$, representing a geometric progression from compute-optimal to 16× overtrained (160 tokens per parameter). For example, a 34M parameter model at $m=16$ trains on $34\text{M} \times 320 = 10.88$B tokens.

Why this matters for contamination. When a model is overtrained on fresh data, two competing effects occur:

1. General performance improvement: The model becomes better at language modeling in general, which might incidentally improve performance on MATH even without contamination (since better language models are better at predicting mathematical text).
2. Contamination dilution: As the total token budget grows while the number of contaminated tokens stays fixed, the proportion of contaminated tokens in the corpus decreases. If contamination dose matters, this dilution should reduce contamination effects.

The overtraining experiment separates these effects by comparing performance changes across contamination levels. If overtraining helps all models equally (regardless of contamination), the effect is general performance improvement. If overtraining hurts contaminated models more than uncontaminated ones, the effect is contamination dilution.

Crucial design detail: no repeated data. The authors emphasize that the additional tokens are fresh, distinguishing this from practical settings where models may see documents repeated multiple times (Hernandez et al., 2022) or train for multiple epochs (Muennighoff et al., 2023; Fang et al., 2025). In multi-epoch training, the model sees the same contaminated tokens repeatedly, which might reinforce memorization rather than dilute it. The overtraining studied here specifically examines what happens when you add more unique data to dilute the contamination dose.

Crossing point measurement. The authors identify a "crossover point" for each model size: the number of test set replicas at which overtraining shifts from decreasing to increasing test loss. For example, at 34M parameters, the crossover is at 32 replicas—models with ≤32 replicas benefit from overtraining, while models with >32 replicas are hurt. At 93M, the crossover drops to just 1 replica. This suggests that larger models lose their contamination advantage more readily when overtrained with fresh data. The authors interpret this through a "dose-response" framework (Schaeffer et al., 2025c):

"Our interpretation is that while more fresh data is generally useful for improving model performance generally, it dilutes the 'dose' of the contaminated data, weakening how the model 'responds' (as measured by task performance)."

Additional Training: Supervised Finetuning Protocol

Objective: Study how standard supervised finetuning on the MATH training set interacts with pretraining contamination, following the observation that SFT is typically the first post-pretraining step in modern LLM pipelines (Wei et al., 2022b; Ouyang et al., 2022).

Training data. The MATH training set (Hendrycks et al., 2021b), containing 7,500 problems with solutions, is used for SFT. The format is preserved from pretraining: "Problem: {problem}\n\nSolution: {solution}". This means the model sees training problems in the same format as the test problems (and the contaminated pretraining data), maximizing transfer.

Why SFT after contamination is interesting. Kocyigit (2025) found that SFT on the training set generally improves performance on the test set, even for contaminated models. However, Kocyigit (2025) induced contamination via continued pretraining (adding contaminated data in a separate phase), while this paper introduces contamination uniformly throughout pretraining. This methodological difference might produce different SFT interactions.

The counterintuitive result. The paper finds that SFT has opposing effects depending on the pretraining contamination level (Figure 4):

• For models with low contamination (< 10 replicas), SFT on the training set decreases test loss (improves performance). This is expected—SFT provides task-specific training that helps the model learn mathematical reasoning.
• For models with high contamination (> 10 replicas), SFT on the training set increases test loss (degrades performance). This is counterintuitive—why would task-specific training hurt performance on the same type of task?

The authors' conjectured mechanism. They propose that SFT causes the model to both learn generalization (from the training set examples) and forget its memorized test set data (through gradient updates that overwrite the specific weights encoding the memorized solutions). For highly contaminated models, the contribution of memorization to test performance dwarfs the contribution of generalization, so even though SFT improves generalization, the net effect is a performance decrease because memorization loss outweighs generalization gain. This is an instance of catastrophic forgetting (McCloskey & Cohen, 1989) applied to contamination—the model "forgets" the test set solutions it memorized during pretraining when it is trained on the training set during SFT.

Why this matters practically. This finding has counterintuitive implications for model development: if you suspect a model is contaminated, you might expect SFT to improve it (by providing legitimate task training). But the paper shows that for highly contaminated models, SFT can reduce test performance—a drop that would normally be interpreted as a negative outcome (degradation, alignment tax) might actually be a positive signal of decontamination. The authors explicitly note this implication in Section 6:

"This implies that a drop in test accuracy after SFT, usually a sign of alignment tax or forgetting, may actually be a positive signal of decontamination."

Inference-Time Analysis Framework: Temperature and Solution Length

Objective: Characterize how memorization-driven performance depends on inference-time parameters (sampling temperature and solution length) that have no analog in discriminative evaluation, and develop a mathematical framework for understanding memorization as a probabilistic survival process.

Temperature-only sampling. The authors use temperature-only sampling (Schaeffer et al., 2025a), where the next-token distribution is given by:

$p(x_t | x_{<t}) = \text{softmax}\left(\frac{\text{logit}(x_t | x_{<t})}{\tau}\right)$

where $\tau$ is the temperature parameter. At $\tau = 0$, the distribution becomes a point mass on the highest-logit token (greedy decoding). As $\tau \to \infty$, the distribution approaches uniform over the vocabulary. The temperature sweep spans $\tau \in \{0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.5\}$, though the paper's main figures focus on a subset (0, 0.2, 0.4, 0.56, 0.8, 1.0). Note that temperature 0.56 is an unusual value—most papers use round numbers like 0.5 or 0.6. This specific value may have been chosen because it is a threshold where some qualitative change occurs in the temperature-performance relationship.

Solution length binning. To study how solution length modulates memorization, problems are binned into 10 logarithmically spaced intervals based on the token length of the gold reference solution, ranging from 15 tokens (shortest) to 1949 tokens (longest). Log-spacing ensures that each bin covers a roughly equal multiplicative factor of length (e.g., 15–25, 25–45, 45–80, etc.), which is appropriate when effects might scale multiplicatively with length.

Per-token negative log-likelihood analysis. For each model and contamination level, the authors compute the negative log-likelihood (NLL, another name for cross-entropy at the token level) of the gold reference solution tokens at each position $t$ in the sequence:

$\text{NLL}(t) = -\log P(s_t^* | s_{<t}^*, p)$

where $s_t^*$ is the $t$-th token of the reference solution and $s_{<t}^*$ is the true prefix. This is computed under teacher forcing: the model always receives the correct prefix regardless of what it would have generated. The per-token NLL trajectory shows how prediction confidence changes as the solution progresses—does the model become more confident (lower NLL) as it accumulates context, or less confident (higher NLL)?

The in-context scaling law. For most model sizes and contamination levels, the authors find that $\text{NLL}(t)$ follows a power law plus irreducible error as a function of token index $t$:

$L_t(N, R) \approx E(N, R) + A(N, R) \cdot t^{-\alpha(N, R)}$

where $L_t$ is the negative log-likelihood at token position $t$, $E(N, R)$ is the irreducible per-token error (the minimum NLL achievable even with infinite context), $A(N, R)$ is the scaling coefficient (controlling the initial NLL at early tokens), and $\alpha(N, R)$ is the scaling exponent (controlling how quickly NLL decays with token position). This equation is fit separately for each model size $N$ and contamination level $R$.

What this equation means physically. As the model processes more tokens of a solution, its prediction of the next token becomes more confident (NLL decreases). This is because each token provides additional context that helps constrain what comes next. The power-law form means that the marginal benefit of additional context diminishes—the first few tokens provide a lot of information, while later tokens provide progressively less. The irreducible error $E$ is the floor: even with infinite context, some uncertainty remains due to the inherent unpredictability of language.

Why a power law for in-context scaling. This functional form mirrors findings from prior work on in-context scaling laws (Anthropic, 2023; Team et al., 2024; Xiong et al., 2024; Anil et al., 2024), where model performance on long-context tasks improves with context length following a power law. In the memorization setting, the context is the solution prefix, and the "task" is predicting the next token of the memorized solution. As the model accumulates more of the memorized prefix, it becomes increasingly certain about what comes next—a form of "in-context retrieval" where the prefix acts as a cue for the memorized continuation.

Exceptions to the power law. The paper identifies two cases where the power law breaks down (Appendix D):

1. Uncontaminated small models: For the 34M model at $R = 0$, NLL increases with token position rather than decreasing. The authors attribute this to limited effective context windows—small models without memorization can't effectively utilize the accumulating context and their predictions degrade as the required context grows beyond their capacity.

2. Highly contaminated large models: For the 344M model at $R = 3162$, NLL is extremely low (~10^{-3}) but rises at late token indices. The authors attribute this to selection bias: only the longest solutions reach late tokens, and these long solutions are systematically harder to memorize. The uptick reflects that the surviving sequences are harder, not that memorization weakens with position.

Fitting parameters by model size and contamination (Figure 10). The fitted parameters $E$, $A$, and $\alpha$ show systematic variation:

• $E$ (irreducible per-token NLL) decreases with both model size and contamination level, approaching near-zero for large, highly contaminated models.
• $A$ (initial NLL) also decreases with contamination, reflecting that contaminated models start with higher confidence even at the first token.
• $\alpha$ (decay rate) varies around 0.5–1.0, with some dependence on model size and contamination. Values of $\alpha > 1$ are particularly significant, as they lead to the Deterministic Lock-In regime described below.

Memorization as a Survival Process: The Three Regimes

Objective: Derive the cumulative probability of successfully generating a complete memorized solution and characterize how this probability behaves under different contamination and temperature conditions.

The survival probability derivation. Starting from the per-token NLL $L_t$, the probability that the model correctly predicts token $t$ (under greedy decoding) can be approximated as $P_t \approx e^{-L_t}$. Under the assumption that token-level errors are independent (which is an approximation—in reality, errors at early tokens affect the context for later tokens), the probability of successfully generating the entire solution of length $T$ is the product of per-token success probabilities:

$P(\text{success} | T) \approx \prod_{t=1}^{T} e^{-L_t} = \exp\left(-\sum_{t=1}^{T} L_t\right)$

Substituting the power-law form for $L_t$ and approximating the sum with an integral, the authors derive:

$P(T) \approx \exp\left(-E(T-1) - \frac{A}{1-\alpha}(T^{1-\alpha} - 1)\right)$

where $E$ is the irreducible per-token error, $A$ is the scaling coefficient, $\alpha$ is the scaling exponent, and $T$ is the solution length in tokens.

What this equation computes. Given a model's memorization parameters ($E$, $A$, $\alpha$) and a solution length $T$, it estimates the probability that the model can generate the entire solution correctly (or at least arrive at the correct final answer). The two terms in the exponent represent competing forces:

• The $E(T-1)$ term: a linear penalty that grows proportionally with sequence length. This represents the irreducible uncertainty—even with perfect memorization, there is some residual probability of error at each token, and these errors accumulate linearly with length.
• The $\frac{A}{1-\alpha}(T^{1-\alpha} - 1)$ term: a sublinear penalty that grows more slowly than linearly when $\alpha < 1$, and actually decreases with $T$ when $\alpha > 1$. This represents the beneficial effect of accumulating context—as the model sees more of the solution, it becomes more confident, and this confidence growth can partially or fully offset the linear penalty from error accumulation.

Why this form. The survival probability decomposes memorization success into two components: (1) how uncertain the model is even with perfect memorization (the $E$ term), and (2) how quickly the model's confidence grows with accumulated context (the $\alpha$ term, through the scaling of $T^{1-\alpha}$). This decomposition explains why contamination effects are not simply proportional to memorization strength—they depend on the shape of the per-token confidence trajectory, not just its average level.

The three regimes (Figure 8). The behavior of $P(T)$ as $T \to \infty$ depends on the values of $E$ and $\alpha$, defining three qualitatively distinct regimes:

Regime I: Exponentially Fast Decoherence ($E > 0$). When the irreducible error $E$ is non-negligible (as in uncontaminated or lightly contaminated models), the $E(T-1)$ term dominates the exponent for large $T$. The survival probability simplifies to:

$P(T) \sim e^{-E \cdot T}$

This is exponential decay: each additional token multiplies the survival probability by a constant factor $e^{-E} < 1$. As $T \to \infty$, $P(T) \to 0$. Long solutions are statistically impossible because the irreducible per-token error accumulates without bound.

When this occurs: uncontaminated models, or models with very low contamination where $E$ remains substantial (e.g., $E \approx 3.6$ for $R=0$). Even if the model has some memorization ($\alpha > 0$), the linear penalty from $E$ overwhelms any context benefit.

Regime II: Brittle Memorization ($E \approx 0$, $\alpha \leq 1$). When contamination has driven the irreducible error near zero, but the scaling exponent $\alpha$ is at most 1, the survival probability follows a stretched exponential (Weibull-like) form:

$P(T) \sim e^{-k T^{1-\alpha}}$

where $k = A/(1-\alpha)$ is a constant. Because $1 - \alpha \geq 0$, the exponent $T^{1-\alpha}$ grows with $T$ (either linearly if $\alpha = 0$, or sublinearly if $0 < \alpha < 1$), so $P(T) \to 0$ as $T \to \infty$, but more slowly than pure exponential decay.

What "brittle" means. The model has memorized the solution and can reproduce it with high per-token confidence, but the cumulative probability of success still eventually vanishes for long enough sequences. The memorization is "brittle" because while it works for short-to-medium solutions, it eventually breaks for sufficiently long ones. The threshold length at which failure becomes likely depends on $\alpha$—higher $\alpha$ means slower decay and longer sustainable solutions.

Regime III: Deterministic Lock-In ($E \approx 0$, $\alpha > 1$). This is the qualitatively distinct regime. When $\alpha > 1$, the exponent $1 - \alpha$ is negative, so $T^{1-\alpha} = 1 / T^{\alpha-1}$ decays to zero as $T \to \infty$. The penalty term becomes:

$\frac{A}{1-\alpha}(T^{1-\alpha} - 1) \to \frac{A}{\alpha-1} \quad \text{as } T \to \infty$

because $T^{1-\alpha} \to 0$ (since $1-\alpha < 0$) and the $-1$ remains. The survival probability converges to a non-zero constant:

$P(T) \to \exp\left(-\frac{A}{\alpha-1}\right) > 0 \quad \text{as } T \to \infty$

What "deterministic lock-in" means. When $\alpha > 1$, the model's per-token confidence grows faster than linearly with context—each additional token of context provides more than proportional benefit for predicting the next token. This rapid confidence growth eventually "locks in" the correct trajectory, making errors vanishingly unlikely even for arbitrarily long solutions. The model doesn't just have better-than-chance memorization; it has asymptotically perfect memorization where the probability of successfully generating the solution plateaus at a finite value regardless of length.

The phase transition. The transition at $\alpha = 1$ is a genuine phase transition in the survival probability: for $\alpha < 1$, $P(T) \to 0$ as $T \to \infty$ (failure is certain for long solutions); for $\alpha > 1$, $P(T) \to \text{constant} > 0$ (success is possible even for arbitrarily long solutions). This makes $\alpha = 1$ a critical threshold separating memorization that eventually fails from memorization that is self-sustaining.

How temperature modulates the effective exponent. The paper derives a relationship between the sampling temperature $\tau$ and the effective scaling exponent $\alpha_{\text{eff}}$ that governs generation:

$\alpha_{\text{eff}}(\tau) \approx \alpha / \tau$

where $\alpha$ is the "native" exponent of the model (the one measured under teacher forcing, which is equivalent to $\tau = 1$ inference with correct prefixes), and $\tau$ is the sampling temperature.

Derivation (implicit in the paper). The native exponent $\alpha$ describes how the model's logit confidence in the correct token grows with context length. Under teacher forcing, the logit gap $\Delta z_t$ (the difference between the logit of the correct token and the logit of the highest incorrect token) grows as $\Delta z_t \propto \alpha \ln t$. The temperature $\tau$ divides all logits before the softmax: $z \to z/\tau$. This means the effective logit gap becomes $\Delta z_t / \tau \propto (\alpha/\tau) \ln t$, yielding an effective exponent $\alpha_{\text{eff}} = \alpha/\tau$.

What this implies for generation. The temperature can artificially shift a model between regimes:

• At $\tau < 1$ (low temperature), $\alpha_{\text{eff}} > \alpha$. A model with native $\alpha = 0.8$ would have $\alpha_{\text{eff}} > 0.8 / 0.2 = 4$ at $\tau = 0.2$, pushing it deep into the Deterministic Lock-In regime. Greedy decoding ($\tau \to 0$) makes $\alpha_{\text{eff}} \to \infty$, ensuring lock-in for any model with $\alpha > 0$.
• At $\tau = 1$, $\alpha_{\text{eff}} = \alpha$, revealing the native memorization strength.
• At $\tau > 1$ (high temperature), $\alpha_{\text{eff}} < \alpha$. A model with native $\alpha = 0.8$ would have $\alpha_{\text{eff}} = 0.8 / 1.5 \approx 0.53$ at $\tau = 1.5$, pushing it from Brittle Memorization into near-exponential decoherence.

This is why high temperature acts as a "truth serum." The paper's provocative phrase "truth serum" captures the idea that high-temperature sampling reveals the true nature of the model's knowledge. A model might appear to have robust memorization (Deterministic Lock-In) at $\tau = 0$ but collapse to Brittle Memorization or Exponential Decoherence at $\tau \geq 1$. This is because low-temperature sampling amplifies the native exponent, artificially creating the appearance of lock-in. High temperature reveals the native exponent and thus whether the model's knowledge is genuinely robust.

The fragility of contamination. This temperature modulation explains why contamination-driven performance is so brittle. A model contaminated with 1000 replicas might achieve near-perfect Math Verify at $\tau = 0$ (deterministic lock-in due to $\alpha_{\text{eff}} \to \infty$) but collapse by a factor of 40× at $\tau = 1$ (Figure 5). The collapse occurs because the native $\alpha$ is actually less than 1, so at $\tau = 1$ the model is in the Brittle Memorization regime where long solutions eventually fail. The impressive greedy-decoding performance was an artifact of temperature amplification, not genuine memorization strength.

Why the three-regime framework is a key contribution. Prior work on contamination has focused on whether contamination occurs and how much it inflates scores. This framework goes further by characterizing when contamination will produce observable performance gains and what factors control the boundary between successful memorization and decoherence. It provides a predictive theory: given a model's per-token NLL trajectory (from which $E$, $A$, $\alpha$ can be estimated), the survival probability equation predicts how the model will perform at any temperature and any solution length. This is the paper's most theoretically novel contribution and the one most likely to influence future work on contamination detection and mitigation.

Figure 8: Phase diagrams. The paper visualizes the three regimes using phase diagrams that plot the maximum sustainable solution length $T$ (for a fixed survival probability $P(T) = 0.01$) as a function of the irreducible error $E$ and scaling coefficient $A$, for two values of $\alpha_{\text{eff}}$. These diagrams show the sharp transition at $\alpha = 1$: for $\alpha_{\text{eff}} = 0.5$, sustainable lengths are limited to a few hundred tokens even for favorable $(E, A)$ values; for $\alpha_{\text{eff}} = 1.5$, there is a large region (low $E$, moderate $A$) where arbitrarily long solutions are sustainable (the "Lock-In, $T = \infty$" region). This visualization makes concrete the qualitative difference between the three regimes and shows how temperature (through $\alpha_{\text{eff}}$) controls which regime a model occupies.

4. Key Insights and Innovations

Innovation 1: Generative Contamination Is Not Discriminative Contamination Plus Length — It Introduces a Qualitatively Distinct Survival Process

The paper's most fundamental intellectual contribution is recognizing that generative evaluation is not simply discriminative evaluation with longer outputs. Prior work on test set contamination — spanning Magar & Schwartz (2022), Jiang et al. (2024), Oren et al. (2023), Yao et al. (2024), and Bordt et al. (2025) — treated contamination as a uniform phenomenon: add benchmark data to pretraining, observe score inflation, characterize the dose-response curve. The implicit assumption was that the same dynamics would govern any evaluation format, because memorization of the correct answer is memorization regardless of how that answer is elicited.

This paper demonstrates that assumption is wrong, and the reason it is wrong is fundamental rather than superficial. In a discriminative task, the model makes a single decision — which of K candidate answers to select — and that decision typically spans 1–5 tokens. Contamination helps by increasing the probability mass on the correct choice. The generative setting introduces a compound probability structure: success requires the model to make not one correct decision but T correct decisions in sequence, where T ranges from 15 to 1949 tokens for MATH solutions. The probability of overall success is the product of per-token success probabilities, which means that even strong per-token memorization can fail to produce correct final answers if the sequence is long enough.

The authors crystallize this insight mathematically through the survival probability framework in Section 5, but the conceptual move precedes the math. The key recognition is that memorization in generative settings involves a race between two opposing forces: (1) the model's growing confidence as it accumulates context (each token of the memorized prefix makes the next token more predictable), and (2) the accumulating risk of decoherence (each token position is an independent opportunity for error). Whether contamination produces observable performance gains depends on which force wins, and that depends on three parameters — the irreducible per-token error E, the confidence growth rate α, and the sampling temperature τ — that have no discriminative analog.

This is a fundamental reframing, not an incremental extension. It converts contamination from a scalar "how much score inflation" question into a structural question about survival dynamics. A model contaminated with R replicas does not simply gain Δ score points on MATH; it acquires a particular (E, A, α) trajectory that governs when and under what conditions that contamination translates to correct answers. Two models with the same average per-token memorization strength can exhibit qualitatively different generative behavior depending on whether α is above or below the critical threshold of 1. This explains why prior work might have reached apparently contradictory conclusions about contamination severity — some studies evaluated under conditions (short solutions, low temperature) where even modest memorization survives, while others evaluated under conditions (long solutions, high temperature) where only the strongest memorization persists.

The three-regime taxonomy — Exponentially Fast Decoherence, Brittle Memorization, Deterministic Lock-In — is therefore not just a classification scheme; it provides a predictive theory for when contamination matters. A model in Exponentially Fast Decoherence (E > 0) gains essentially nothing from contamination regardless of dose, because the irreducible per-token error ensures failure on any non-trivial sequence length. A model in Brittle Memorization (E ≈ 0, α ≤ 1) gains on short solutions but eventually fails on long ones — its contamination is real but fragile. A model in Deterministic Lock-In (E ≈ 0, α > 1) achieves asymptotic perfection: once contaminated enough to push α above 1, its memorization becomes self-sustaining and succeeds on arbitrarily long solutions. The phase diagram in Figure 8 visualizes this sharply, showing the qualitative boundary between "contamination that helps on some problems" and "contamination that helps on every problem regardless of length."

This framework has immediate diagnostic value. A benchmark designer concerned about contamination can stratify results by solution length and temperature: if a model's performance degrades exponentially with solution length, its high overall score is likely contamination-driven Brittle Memorization rather than robust reasoning. If performance remains stable with length (Deterministic Lock-In), the contamination is deep enough to be indistinguishable from genuine capability by length-based tests alone — and temperature-based auditing (the "truth serum" effect) becomes the discriminating lever. This is an intellectual contribution that transforms how the field should think about contamination, not just how it should measure it.

Innovation 2: Temperature as a "Truth Serum" — The Inference-Time Lever That Decouples Memorization from Generalization

The observation that temperature affects generation quality is not new — temperature has been a standard hyperparameter since the Boltzmann machine era (Ackley et al., 1985). What is novel is the paper's recognition that temperature plays a unique and previously unrecognized role in the contamination-memorization-generative evaluation pipeline: because temperature divides logits before the softmax, it directly modulates the effective confidence growth rate α_eff = α/τ, and thus can artificially shift a model between the three survival regimes entirely independently of its actual knowledge.

The conceptual contribution is identifying temperature as a diagnostic tool for distinguishing memorization from generalization. The paper's empirical demonstration is stark: at high contamination (1000 replicas), raising temperature from 0 to 1.0 reduces Math Verify performance by a factor of up to 40×, collapsing it to near-baseline (Figure 5). At low contamination, the same temperature increase produces only a ~2× reduction. The differential sensitivity is the diagnostic signal — contamination-driven performance is disproportionately fragile to temperature because it relies on the amplification effect of low temperature (τ << 1 making α_eff >> α) to create the false appearance of robust memorization. When temperature is raised to τ = 1, the native α is exposed, and if α < 1, the performance collapses.

This is a significant reframing of what temperature means in evaluation. In standard practice, temperature is treated as a generation quality knob — lower for deterministic tasks, higher for creative ones — and evaluation protocols typically report whichever setting yields the best score. The paper argues instead that temperature is a veracity diagnostic: the temperature-sensitivity of a model's performance contains information about whether that performance derives from robust knowledge or brittle memorization. A model whose MATH score is stable across τ ∈ [0, 1] has different properties than one whose score plummets by 40× over the same range, even if both achieve the same greedy-decoding score. This insight did not exist in the contamination literature prior to this work, and it provides a practical, zero-cost auditing mechanism — simply evaluate at multiple temperatures and examine the degradation pattern.

The connection to the scaling exponent α is what elevates this from an empirical observation to a theoretical contribution. Without the α framework, one might say "performance drops at high temperature" and leave it at that. The paper shows why the drop occurs and what governs its magnitude: the native exponent α of the model's per-token confidence trajectory. This connects inference-time behavior directly to the pretraining memorization dynamics studied in Section 3, creating a unified picture. The same α that appears in the per-token NLL scaling law (Equation 3) controls the temperature-sensitivity at inference time (α_eff = α/τ). This means α is the fundamental quantity linking contamination dose, model size, and inference behavior — it is the "memorization quality" parameter that all three lifecycle stages express in different ways.

The "truth serum" framing is memorable but the intellectual contribution is deeper: it provides a principled criterion for evaluation protocol design. If a benchmark is meant to measure generalization, evaluations should be conducted at sufficiently high temperature that contamination-driven lock-in cannot masquerade as understanding. The paper does not specify what temperature is "sufficient," but the theory implies that τ ≥ 1/α_native is the threshold above which lock-in becomes impossible, and since α_native is bounded above by roughly 1 for realistic contamination levels (the paper finds α ≤ 1 for most conditions in Figure 10), τ ≥ 1 should be sufficient to expose most brittle memorization. This is a concrete, actionable recommendation that emerges from the theoretical framework.

Innovation 3: Scaling Laws Reveal That Even Minimal Contamination Bypasses an Asymptotic Barrier — Reframing Contamination Severity in Compute-Equivalent Terms

Prior work on contamination has quantified severity in terms of score inflation: X replicas produces Y percentage-point gain on benchmark Z. This is useful but narrow — it tells you how much a specific contamination dose inflates a specific metric, but does not provide a principled answer to "how bad is contamination, really?" that is independent of the particular benchmark and model scale.

The paper's scaling law analysis (Section 3, Finding #3) reframes contamination severity in a fundamentally different way: by asking how much pretraining compute contamination substitutes for. The functional form L(C, R) = E(R) + C₀(R) · C^(-α(R)) allows the authors to estimate the irreducible error E(R) for each contamination level R. The finding that E(R=1) < E(R=0) — that even a single replica of the test set enables achieving a lower loss than the best possible performance (as C → ∞) on the uncontaminated corpus — is not just a large effect size. It is a qualitative breach: contamination does not merely accelerate progress toward the same asymptote; it shifts the asymptote itself. No amount of scale on clean data can reach the loss achievable with one contaminated replica.

This reframing matters because it provides a compute-equivalent measure of contamination damage. When a benchmark maintainer asks "how serious is it that one copy of my test set leaked into Common Crawl?", the traditional answer would be some estimate of score inflation for a representative model. The scaling-law answer is more severe: "that single leak enables models to achieve performance on your benchmark that would require infinite compute to reach on clean data." This is a statement about the benchmark's integrity, not about any particular model's score. It means the benchmark's measurement of generalization capability is fundamentally compromised — not by a margin of a few percentage points, but in its asymptotic validity as a generalization measure.

The paper explicitly connects this finding to potential contradictions with prior work (Huang et al., 2024; Hayes et al., 2025), which claimed that single-shot verbatim memorization is limited. The authors hypothesize that MATH's distributional distinctiveness — structured LaTeX, \boxed{} notation, consistent problem-solution formatting — makes it more memorizable than the natural language text studied in prior memorization research. This hypothesis is testable and suggests that contamination severity may be benchmark-dependent in ways not captured by generic memorization studies. A benchmark consisting of unique natural language paragraphs might resist single-shot memorization where MATH succumbs to it. This is a significant nuance: contamination research cannot assume uniform memorizability across benchmarks, and the scaling law framework provides the tool for measuring benchmark-specific susceptibility.

The compute-equivalent reframing also connects naturally to the paper's findings about overtraining (Finding #4) and SFT (Finding #5). Overtraining on fresh data partially restores the uncontaminated asymptote by diluting the contamination dose, but at a cost in total compute that must be weighed against the contamination's compute-equivalent benefit. SFT has the opposite sign for highly contaminated models — it degrades performance because it overwrites memorized solutions, effectively undoing some of the contamination's compute-equivalent gain. These interactions make sense only when contamination is understood in compute-equivalent terms: overtraining and SFT are interventions that shift the effective contamination dose and thus the effective irreducible error.

This is an incremental advance in methodology (scaling laws were already a standard tool) but a fundamental advance in conceptual framing. It converts contamination from a nuisance variable to be controlled into a quantity with well-defined scaling properties that can be compared directly to other resource investments (compute, data, model size). This puts contamination on the same analytical footing as the quantities that scaling laws were originally developed to study, opening the door to joint optimization frameworks that trade off contamination risk against other desiderata.

Innovation 4: The Interaction Between SFT and Contamination Is Not Monotonic — Reframing Post-Training As a Contamination Modulator

The standard mental model of the pretraining → SFT pipeline assumes that SFT is purely beneficial: it adapts the pretrained model to the target task distribution, and any performance change on in-distribution test data is positive. Kocyigit (2025) studied SFT after contamination and found it generally improved performance, reinforcing this assumption.

This paper's Finding #5 — that SFT on the MATH training set helps low-contamination models but actively hurts high-contamination models — breaks this assumption. The mechanism the authors propose (SFT improves generalization but causes catastrophic forgetting of memorized test solutions, and the memorization contribution dominates for highly contaminated models) is plausible but not definitively established. The conceptual contribution, however, does not depend on the specific mechanism being correct.

The conceptual contribution is recognizing that SFT is a contamination modulator, not just a task adapter. When a contaminated model undergoes SFT, two things happen simultaneously: the model learns legitimate task structure from the training set (which should improve generalization to the test set) and it undergoes gradient updates that may overwrite the specific weights encoding memorized test examples (which should degrade contamination-driven performance). The net effect depends on the relative magnitude of these two forces, which in turn depends on the contamination dose. This means SFT can serve as an unintentional decontamination procedure — for highly contaminated models, the net performance drop from SFT is actually evidence that the model is becoming less reliant on memorization and more reliant on generalization, which is a desirable outcome even though the headline accuracy number decreases.

This reframing has immediate practical implications for model development and evaluation. A model developer who observes a performance drop after SFT might interpret it as a failure of the SFT procedure and attempt to mitigate it. The paper suggests that for contaminated models, this drop may be a positive signal — it indicates that the model is unlearning its memorized shortcuts and acquiring more robust capabilities. The authors make this point explicitly in the Discussion: "a drop in test accuracy after SFT, usually a sign of alignment tax or forgetting, may actually be a positive signal of decontamination."

The finding also highlights a tension in current training paradigms. If the goal is to maximize reported benchmark scores (which is often the implicit incentive in competitive model development), then for highly contaminated models, the optimal strategy is to skip SFT or to apply it in a way that minimizes forgetting (e.g., replay of contaminated data, elastic weight consolidation). But this produces a model whose benchmark scores are artificially inflated by memorization rather than reflecting genuine capability — precisely the "illusion of competence" the paper warns against. The tension between score maximization and faithful evaluation is structural, and the paper's identification of SFT as a point where this tension surfaces is a valuable diagnostic contribution.

This is an incremental empirical finding (it extends the scope of contamination-interaction studies from pretraining to the full lifecycle) but a potentially significant conceptual reframing of what post-training means for contaminated models. It suggests that every stage of the model lifecycle is a point where contamination effects can be amplified, preserved, or attenuated, and that understanding these interactions is necessary for trustworthy evaluation — a theme the paper's lifecycle-spanning experimental design embodies throughout.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021b), consisting of 5,000 competition mathematics test problems with solutions. The test set contains approximately 1.4 million tokens under the Qwen 3 tokenizer. The authors chose MATH because it is comparatively large, has both problems and full solutions, and represents the kind of generative evaluation — where models must produce long-form derivations rather than selecting from options — that reasoning models increasingly demand.

• Base model(s). The authors pretrain transformer-based causal language models from random initialization using the Qwen 3 architecture (Yang et al., 2025a) at five scales: 34M, 62M, 93M, 153M, and 344M parameters. Qwen 3 is chosen as a representative contemporary architecture, and the scale range (roughly one order of magnitude) spans from very small to moderately sized models. All models are trained with compute-optimal token budgets of 20 tokens per parameter (Hoffmann et al., 2022), using Flash Attention 2, bfloat16 precision, and the AdamW optimizer.

• Metrics. Two complementary metrics are reported. Math Verify score (Kydlicek et al., 2025) is the fraction of test problems for which the model's generated solution contains a final boxed answer that is mathematically equivalent to the ground truth, as determined by a symbolic verification function. This captures end-to-end generative success. Cross-entropy loss on the gold reference solutions is computed under teacher forcing: at each token position, the model receives the true prefix as context and is evaluated on its predicted probability of the correct next token. Cross-entropy isolates memorization strength from generation decoherence, providing a continuous signal even when Math Verify scores are near zero. A critical implementation detail: the authors discovered and fixed a bug in the EleutherAI LM Evaluation Harness (Gao et al., 2024) that previously underreported Math Verify scores by stripping \boxed{} notation before parsing, causing even gold reference solutions to score only ~70% (Appendix C).

• Baselines. The primary baseline is the uncontaminated model (R = 0 replicas), which establishes the performance floor from training on clean web data alone. For the rephrased/perturbed evaluations (Table 1), the baseline is the same model evaluated on unmodified test problems, with the rephrased and perturbed variants testing whether performance gains survive changes to surface form or numerical values.

• Generation budget / compute accounting. For pretraining, compute is measured as C ≈ 6ND FLOP (Kaplan et al., 2020), where N is model parameters and D is training tokens. For inference, generation is performed using temperature-only sampling (Schaeffer et al., 2025a) with a single generation per problem — the budget is not varied at inference time for the main pretraining results (Figures 1–4). For the temperature sweep (Section 5), temperatures range from 0 (greedy) to 1.5. For the overtraining experiments (Section 4), the budget is measured in terms of the overtraining multiplier m, where total training tokens = m × 20 × N, with m ∈ {1, 2, 4, 8, 16}. All models within a contamination level receive identical total training tokens to isolate contamination dose from data quantity.

• Cross-validation / statistical protocol. No cross-validation is used — the experimental design is a full factorial sweep over model sizes and contamination levels, with one model trained per condition. Statistical uncertainty is not reported (no error bars, confidence intervals, or multiple seeds). The scaling law fits in Figure 2 are evaluated by average fitting error (< 10⁻² for all R), but this measures goodness-of-fit to the parametric form, not generalization to held-out data. This is a limitation: with a single model per condition, the variance due to random initialization and data ordering is uncharacterized.

Main Quantitative Results

Pretraining: Contamination Inflates Math Verify Scores and Reduces Cross-Entropy Across All Model Scales

Figure 1 (left) establishes the fundamental dose-response relationship. As the number of MATH test set replicas in the pretraining corpus increases, Math Verify scores rise monotonically for all model sizes. The relationship is non-linear: at low contamination (≤ 10 replicas), Math Verify scores remain close to the uncontaminated baseline; at around 100 replicas, performance sharply increases; and at the highest contamination levels (1000–3162 replicas), models approach ceiling performance near 100% Math Verify. Cross-entropy on the test set shows the inverse pattern, falling from approximately 4–5 nats at R = 0 to near zero at R = 3162.

The effect is amplified by model scale. Figure 1 (right) shows that the loss ratio between R replicas and 0 replicas grows with model size — larger models benefit more from the same number of replicas. For example, at R = 100, the 344M model achieves a lower loss ratio than the 34M model. The authors interpret this as evidence that larger models more efficiently exploit contaminated data, consistent with prior findings that memorization increases with model scale (Carlini et al., 2023; Tirumala et al., 2022).

Table 1 provides the critical falsification test: when MATH test problems are rephrased (same numerical values and logic, different wording) or perturbed (same wording, different numerical values), model performance collapses to near-zero across all contamination levels and model sizes. For the largest model (344M) at the highest contamination (3162 replicas), rephrased accuracy is 0.04% and perturbed accuracy is 0.00% — effectively at the uncontaminated baseline of 0.04% for rephrased and 0.00% for perturbed at R = 0. This confirms that contamination-driven performance gains derive from verbatim memorization of specific token sequences, not from acquisition of generalizable mathematical reasoning.

Scaling Laws: A Single Test Set Replica Breaks the Irreducible Error Barrier

Figure 2 presents the scaling law analysis. For each contamination level R ∈ {0, 1, 3, 10, 32, 100, 316}, the authors fit the functional form L(C, R) = E(R) + C₀(R) · C^(-α(R)) relating cross-entropy loss on the MATH test set to pretraining compute C = 6ND.

The top panel shows the fitted curves alongside the data points (one per model size per contamination level). The horizontal purple line marks E(0) = 3.594 — the irreducible error of the uncontaminated corpus. The critical observation is that for every contamination level R ≥ 1, the fitted scaling law passes below this line, meaning that contaminated models achieve lower loss than any model could ever achieve on clean data, regardless of scale. At R = 1, models above ~10¹⁷ FLOP (corresponding to the 93M parameter model and larger) fall below E(0). At R = 316, all models except the smallest are below E(0) by a wide margin.

The bottom panel decomposes the fitted parameters:

• E(R) drops from 3.594 at R = 0 to 0.0347 at R = 316 — a reduction of more than 100×. Most of the reduction occurs between R = 10 and R = 100, consistent with the non-linear dose-response observed in Math Verify scores (Figure 1).
• C₀(R) increases with R, from approximately 10⁹ at R = 0 to above 10²⁰ at R = 316. A larger prefactor means that contaminated models benefit more from additional compute — the reducible loss component starts higher and decays over a larger range.
• α(R) is roughly constant at 0.3–0.5 across contamination levels, meaning that the rate at which loss decays with compute (the scaling efficiency) is similar regardless of contamination. Contamination primarily shifts the asymptotic floor E(R) rather than changing the learning dynamics.

The average fitting error is under 10⁻² for all R, indicating that the three-parameter power-law form fits the observed data well within the fitted range (10¹⁷–10¹⁹ FLOP). However, the extrapolation to C → ∞ is inherently speculative — the functional form might change at scales far beyond those tested.

Additional Training: Overtraining with Fresh Data Dilutes Contamination Effects

Figure 3 examines the interaction between contamination and overtraining. For three model sizes (34M, 63M, 93M), the authors plot cross-entropy on the MATH test set as a function of the number of test set replicas, with separate curves for overtraining multipliers m ∈ {1, 2, 4, 8, 16}.

The key pattern is a crossover interaction: for models with low contamination, cross-entropy decreases with increasing overtraining (more fresh data improves performance on the test set, likely through general language modeling improvements); for models with high contamination, cross-entropy increases with overtraining (the contamination benefit is diluted). The crossover point — the number of replicas at which overtraining shifts from helpful to harmful — moves leftward as model size increases: 32 replicas for 34M, 10 replicas for 63M, and 1 replica for 93M. This means larger models lose their contamination advantage more readily when overtrained with fresh data.

The right panel of Figure 3 reformats the same data to show cross-entropy as a function of overtraining multiplier, with separate curves for each contamination level. The divergence is clear: at R = 0 (uncontaminated), loss decreases monotonically with m; at R = 3162 (heavily contaminated), loss increases monotonically with m. Intermediate contamination levels show the crossover behavior.

The authors interpret this through a dose-response framework: as fresh data is added, the proportion of contaminated tokens in the corpus shrinks, reducing the effective contamination "dose" and its impact on test performance. The fact that larger models are more sensitive to dilution (their crossover occurs at lower R) suggests that larger models may require a higher concentration of contaminated data to maintain memorization, possibly because their greater capacity enables them to learn the clean data distribution more thoroughly, competing with memorization.

Additional Training: SFT on the Training Set Has Opposite Effects Depending on Contamination Level

Figure 4 examines the effect of supervised finetuning on the MATH training set. The left panel plots test loss after SFT against test loss before SFT, with points colored by the number of pretraining replicas. Points below the diagonal (purple) indicate SFT helped; points on the diagonal (aqua) indicate no effect; points above the diagonal (yellow/green) indicate SFT hurt.

The pattern is stark: models with low contamination (< 10 replicas) fall below the diagonal — SFT on the training set significantly reduces test loss. At exactly 10 replicas, points cluster near the diagonal — SFT has negligible effect. Above 10 replicas, points rise above the diagonal — SFT significantly increases test loss. The right panel quantifies this as the ratio of pre-SFT test loss to post-SFT test loss, showing a clear transition: SFT helps when contamination is low (ratio > 1), has no effect at intermediate contamination (ratio ≈ 1), and hurts when contamination is high (ratio < 1). This pattern holds across model sizes (34M–344M).

The authors conjecture that SFT simultaneously improves generalization (by training on legitimate MATH examples) and induces forgetting of memorized test solutions (through gradient updates that overwrite the specific weights encoding the contaminated sequences). For highly contaminated models, the memorization contribution to test performance dwarfs the generalization contribution, so even though SFT improves the latter, the net effect is a performance decrease because memorization loss outweighs generalization gain. This is effectively catastrophic forgetting of contamination — the model "unlearns" the test set it memorized during pretraining when trained on the training set during SFT.

Inference: High Temperature Collapses Contamination-Driven Performance

Figure 5 shows Math Verify scores as a function of test set replicas, model size, and sampling temperature. At greedy decoding (τ = 0), the pattern matches Figure 1: performance rises with contamination and model size, reaching near-100% for large, heavily contaminated models. As temperature increases, two things happen:

First, overall performance degrades — even uncontaminated models see their Math Verify scores drop as stochasticity is introduced. Second, and more dramatically, the degradation is disproportionately larger for contaminated models. The authors quantify this as a differential sensitivity: at low contamination (≤ 10 replicas), increasing temperature from 0 to 1.0 reduces performance by a factor of approximately 2×; at high contamination (1000 replicas), the same temperature increase reduces performance by a factor of up to 40×. At τ = 1.0, the performance of highly contaminated models has collapsed to near the uncontaminated baseline, meaning that contamination-driven gains are almost entirely eliminated.

This disproportionate sensitivity is the empirical basis for the "truth serum" claim: high-temperature sampling selectively penalizes memorized solutions while leaving generalization-based performance relatively intact. The mechanism, formalized in Section 5's survival analysis, is that temperature divides logits (z → z/τ), reducing the effective confidence growth rate α_eff = α/τ. Contaminated models that appeared to have Deterministic Lock-In at τ → 0 (because α_eff → ∞) are revealed to have native α < 1 when τ increases, causing them to revert to Brittle Memorization or Exponential Decoherence.

Inference: Longer Solutions Reduce Contamination-Driven Performance, with Functional Form Shifting by Contamination Level

Figure 6 stratifies Math Verify scores by solution length, binning problems into logarithmically spaced intervals from 15 to 1949 tokens. Three contamination levels are shown (R = 0, 10, 316) at two temperatures (τ = 0, τ = 1.0).

At high contamination (R = 316) and low temperature (τ = 0), Math Verify scores are near-100% regardless of solution length — the model is in the Deterministic Lock-In regime, where length has almost no effect. As temperature increases to 1.0, a length-dependent degradation emerges: scores remain high for short solutions but decay for longer ones, indicating that the model is now in the Brittle Memorization regime (or Exponential Decoherence, depending on the exact per-token parameters).

At lower contamination, the decay pattern is qualitatively different. For R = 10, performance decays with solution length but the functional form appears to be a slower, concave-up decay rather than the exponential drop-off seen at high contamination and high temperature. The authors characterize this as a shift in the decay's functional form: at high contamination, the decay is approximately exponential (or power-law with α < 1) with rapid drop-off; at lower contamination, the decay is "concave up and occurs much more slowly."

The interaction between length and temperature is also notable: for short solutions (≤ 100 tokens) at R = 316 on the largest model, raising τ from 0 to 1.0 drops accuracy by approximately 45%. For solutions of 400 tokens, the same temperature increase causes accuracy to drop by nearly 100%. This confirms that temperature and length multiply each other's effects — longer solutions accumulate more stochastic decoherence events, and higher temperature makes each event more likely.

Inference: Per-Token NLL Follows Power-Law Decay, Revealing Three Memorization Regimes

Figure 7 (top) shows the negative log-likelihood (NLL) of the gold reference solution tokens as a function of token index, for selected model sizes (34M, 93M, 344M) across contamination levels. For most conditions, NLL decays with token index — as the model accumulates context, it becomes more confident in predicting the next token. The decays approximately follow the power-law form L_t(N, R) ≈ E(N, R) + A(N, R) · t^(-α(N, R)).

Two exceptions are noted (Appendix D): the uncontaminated 34M model (R = 0) shows NLL increasing with token index (attributed to limited effective context windows in small architectures without memorization), and the heavily contaminated 344M model (R = 3162) shows an uptick at late tokens (attributed to selection bias — only the longest, hardest-to-memorize solutions survive to late token positions).

Figure 7 (bottom) translates the per-token NLLs into cumulative survival probabilities: the probability that the model correctly generates the entire solution from scratch as a function of solution length. The three regimes are visible:

• R = 0, 1: Exponential decay — survival probability plummets to near zero within the first few hundred tokens.
• R = 10–100: Gradual decay that reaches zero for sufficiently long solutions — Brittle Memorization. The decay rate decreases with contamination.
• R = 316–3162: Near-constant survival probability (approaching 1.0 for the largest model) even for the longest solutions — Deterministic Lock-In.

Figure 8 provides phase diagrams that map the three regimes as a function of the irreducible error E and scaling coefficient A, for two values of the effective exponent α_eff. At α_eff = 0.5 (Brittle Memorization), sustainable solution lengths (for P(T) = 0.01) range from tens to thousands of tokens depending on (E, A), but no parameter combination achieves infinite length. At α_eff = 1.5 (Deterministic Lock-In), a large region of the (E, A) space — characterized by low E and moderate A — achieves T = ∞, meaning solutions of any length are sustainable. The sharp qualitative difference between these two phase diagrams illustrates the critical nature of α = 1 as a phase transition in memorization dynamics.

Ablation Studies and Robustness Checks

The paper's experimental design is a full factorial sweep rather than an ablation study in the traditional sense — the independent variables (model size, contamination level, temperature, solution length, overtraining multiplier) are systematically varied, and the effects of each are examined through the main results. However, several analyses serve the function of ablations or robustness checks:

Rephrased and perturbed evaluations (Table 1): This is the critical ablation establishing that contamination-driven performance is memorization, not generalization. By modifying the test problems while preserving the underlying mathematical structure (rephrasing) or modifying the structure while preserving the surface form (perturbation), the authors isolate the contribution of verbatim token sequence matching. The collapse to near-zero in both conditions — across all model sizes and contamination levels — demonstrates that the performance gains observed in Figures 1 and 2 are entirely attributable to memorization of specific token sequences, with no measurable transfer to rephrased or perturbed variants. This is a negative result that is essential for interpreting all other findings: contamination does not teach the model mathematics; it teaches the model to reproduce the MATH test set.

Overtraining interaction (Figure 3): This serves as a robustness check on the dose-response interpretation of contamination. If contamination effects were simply additive (more contamination → better performance, regardless of other training factors), overtraining would either help all models equally or have no interaction with contamination level. The observed crossover interaction — that overtraining helps low-contamination models but hurts high-contamination models — confirms that the contamination dose depends on its proportion in the training corpus, not its absolute quantity. The dose-response framework (Schaeffer et al., 2025c) is supported by the dilution effect.

SFT interaction (Figure 4): This is a robustness check on the memorization interpretation. If contamination-driven performance were robust generalization, SFT on the training set — which provides additional legitimate mathematical training — should improve or at least not degrade performance on the test set. The finding that SFT degrades performance for highly contaminated models is inconsistent with the generalization interpretation but consistent with the memorization interpretation (via catastrophic forgetting of memorized sequences). The non-monotonicity — SFT helps at low contamination and hurts at high contamination — would be difficult to explain if contamination effects were not memorization-based.

Per-token NLL exceptions (Appendix D, Figure 9): The authors identify and analyze systematic deviations from the power-law NLL form. For the 34M uncontaminated model, the increasing NLL with token index is attributed to limited effective context windows — a capacity limitation that is a sanity check on the measurement (if NLL never decayed for uncontaminated models, the power-law interpretation would be suspect). For the 344M highly contaminated model, the rising NLL at late token indices is attributed to selection bias from the dataset's length distribution — only the longest solutions survive to late tokens, and these are systematically harder to memorize. Both exceptions are explained by mechanisms orthogonal to the main memorization dynamics, supporting the validity of the power-law interpretation for the remaining conditions.

Scaling law fitting quality (Figure 2): The reported average fitting error of < 10⁻² for all contamination levels is a quantitative check that the functional form L(C, R) = E(R) + C₀(R) · C^(-α(R)) adequately describes the data within the fitted range. However, this is a goodness-of-fit measure on the training data, not a cross-validation metric, so it should be interpreted as a consistency check rather than evidence of predictive accuracy.

Math Verify vs. cross-entropy correlation (Appendix, Figure 11): The authors examine how Math Verify scores correlate with pretraining cross-entropy as a function of temperature. At low temperatures, the correlation is strong — models with lower test cross-entropy achieve higher Math Verify scores. As temperature increases, the correlation weakens, falling to near zero above τ = 1.0. This ablation confirms that cross-entropy and Math Verify capture related but distinct aspects of contamination: cross-entropy measures memorization strength under teacher forcing, while Math Verify measures end-to-end generation success, and the relationship between them is mediated by temperature. This is consistent with the survival process framework and validates the use of both metrics for complementary analysis.

Missing ablations: The paper does not include several analyses that would strengthen its claims. It does not vary the tokenizer — all results use the Qwen 3 tokenizer, and it is unknown whether tokenization artifacts affect the memorization thresholds. It does not compare contamination introduced uniformly throughout pretraining (the paper's approach) to contamination introduced via continued pretraining (Kocyigit, 2025), which limits the generalizability of the SFT interaction findings. It does not test whether the rephrased/perturbed problem results (Table 1) hold at finer granularity (e.g., by difficulty level or problem type within MATH). And critically, the scaling law extrapolation to infinite compute is not validated by any held-out test — the claim that a single replica enables bypassing the irreducible error depends entirely on the assumed functional form extrapolating correctly beyond the fitted range.

Critical Assessment

Claim 1: Contamination-Driven Performance Is Fragile, Verbatim Memorization Rather Than Robust Generalization

Does the evidence support this? The evidence is strong and multi-pronged. Table 1 directly demonstrates that rephrasing or perturbing test problems collapses performance to baseline across all contamination levels and model sizes. If contamination were producing genuine mathematical reasoning, the model should transfer to structurally identical but reworded problems — it does not, at any scale or contamination dose. The temperature sensitivity analysis (Figure 5) provides converging evidence: robust generalization should survive stochastic sampling (the model understands the math and can produce it even when sampling from a broader distribution over tokens), while brittle memorization should collapse. The 40× performance drop for highly contaminated models at τ = 1.0 compared to τ = 0 is dramatic and supports the brittleness interpretation.

However, the generalization claim is established only for MATH under the specific contamination protocol used. The authors do not test whether contamination of a different form (e.g., paraphrased test problems in the training data, or contamination of MATH-like problems that share structure with the test set) would transfer. The rephrasing and perturbation in Table 1 use the same underlying transformations applied uniformly — it is possible that contamination with some variation in the training data (data augmentation of the test set) would produce more robust memorization that survives rephrasing, which would blur the boundary between memorization and generalization that the paper draws sharply. This is an important caveat: the paper establishes that verbatim contamination does not generalize, but does not explore the continuum between verbatim memorization and robust generalization that might be induced by intermediate forms of contamination.

Claim 2: A Single Test Set Replica Achieves Lower Loss Than the Uncontaminated Irreducible Error

Does the evidence support this? The scaling law analysis in Figure 2 provides compelling evidence within the fitted range. At R = 1, models with compute budgets above approximately 10¹⁷ FLOP achieve cross-entropy below E(0) = 3.594. This is observed, not extrapolated — the 93M, 153M, and 344M models at R = 1 all fall below the E(0) line. For R = 3, all models except the 34M are below. The claim that this represents a "breach" of the irreducible error barrier is therefore empirically grounded in the data.

The extrapolation to "infinite compute" is a different claim — that under standard scaling law assumptions, E(R=1) < E(R=0) means contaminated training can achieve what unlimited uncontaminated compute cannot. This extrapolation depends on the scaling law functional form continuing to hold at compute scales far beyond those tested. The fitted curves span roughly two orders of magnitude in FLOP (10¹⁷–10¹⁹). Extrapolating to 10²² or beyond is a standard practice in scaling law research (Kaplan et al., 2020; Hoffmann et al., 2022), but it is an assumption, not a measurement. The paper could strengthen this claim by testing whether the functional form holds at a wider range of compute budgets (e.g., training models at 10²⁰ FLOP) or by using cross-validation across model sizes to assess extrapolation quality.

The potential contradiction with Huang et al. (2024) and Hayes et al. (2025) flagged by the authors is significant and underexplored. If a single replica genuinely enables achieving sub-E(0) loss, why did prior work find single-shot memorization to be limited? The authors' hypothesis — that MATH is distributionally distinct from FineWeb-Edu-Dedup in a way that makes contamination easier to detect — is plausible but untested. The paper does not quantify the distribution shift between MATH and FineWeb-Edu-Dedup, nor does it test whether the same phenomenon holds for other benchmarks or other clean corpora. This is a genuine gap: the generalizability of the "single replica breaks irreducible error" result is unknown.

Claim 3: Overtraining with Fresh Data and SFT Have Opposite, Contamination-Dependent Effects

Does the evidence support this? The overtraining interaction (Figure 3) is demonstrated for three model sizes (34M, 63M, 93M). The crossover point shifting leftward with model size is a consistent pattern. However, the larger models (153M, 344M) are not shown in the overtraining sweep — Figure 3 only displays 34M, 63M, and 93M. The authors do not explain this omission. If the claim is that overtraining dilutes contamination effects generally, it should be verified across the full model scale range. The absence of the 153M and 344M models from the overtraining analysis is a notable gap.

The SFT interaction (Figure 4) is demonstrated across all model sizes. The non-monotonic pattern (SFT helps at low contamination, hurts at high contamination) is clear and consistent. However, the proposed mechanism — catastrophic forgetting of memorized test solutions — is conjectural, not directly tested. The paper does not measure whether the model's per-token NLL on test solutions increases after SFT (which would indicate forgetting), nor does it compare the gradient updates during SFT to the weights encoding memorized solutions. The mechanism is plausible but unverified. Alternative explanations are possible: for example, SFT might change the model's generation dynamics (e.g., shifting from memorized reproduction to more exploratory generation) without necessarily "forgetting" the memorized solutions in the sense of weight-space overwriting. Distinguishing between these hypotheses would require mechanistic analysis that the paper does not perform.

Additionally, the SFT result depends on the specific relationship between the training and test sets. MATH training and test problems are drawn from the same distribution (competition math problems with solutions in \boxed{} format). If the training set were less similar to the test set, SFT might not produce the same forgetting effect. The paper's finding should be interpreted as specific to the MATH benchmark under the standard train/test split, not as a universal property of SFT after contamination.

Claim 4: Temperature and Solution Length Govern Three Distinct Memorization Survival Regimes

Does the evidence support this? This is the paper's most theoretically ambitious claim, and the evidence is mixed regarding the empirical basis versus the mathematical derivation.

The empirical evidence for temperature effects is strong. Figure 5 clearly shows that temperature degrades performance, and that the degradation is disproportionately large for contaminated models. The 40× collapse at τ = 1.0 is a striking quantitative result. Figure 6 shows the length-dependence convincingly, with the interaction between length and temperature clearly visible. The per-token NLL trajectories in Figure 7 (top) follow the power-law form for most conditions, providing empirical support for the assumed functional form in Equation 3.

However, the derivation of the three regimes (Equation 4 and the subsequent analysis) involves several approximations that are not empirically validated:

1. Independence assumption. The survival probability P(T) ≈ exp(-Σ L_t) assumes that token-level errors are independent — that the probability of correctly predicting token t given correct prefix is approximately e^(-L_t), and that these probabilities multiply. In reality, the model's generation trajectory diverges from the reference solution after the first error, meaning the probabilities are not independent. The product formula is an approximation valid only when errors are rare. For long solutions or high temperatures where errors are not rare, the approximation may be poor.

2. Sum-to-integral conversion. The derivation replaces the discrete sum Σ L_t with a continuous integral, which is a standard approximation for large T but may introduce errors for shorter solutions (where the paper's distinctions between regimes are being made).

3. α_eff = α/τ. The relationship between native exponent α and effective exponent α_eff under temperature is derived from the argument that logit gaps scale as Δz_t ∝ α ln t, and temperature divides logits. This argument assumes that the logit gap grows logarithmically with t — an assumption that follows from the power-law NLL form but is not directly verified by measuring logit gaps at different temperatures. The paper does not measure α_eff directly (e.g., by fitting generation trajectories rather than teacher-forced NLLs) to confirm the α/τ relationship.

4. Regime classification depends on fitted parameters. The phase diagrams in Figure 8 are computed from the fitted (E, A, α) parameters, which come from the teacher-forced NLL fits in Figure 7. Any error in those fits propagates to the regime classification. The paper does not report uncertainty in the fitted parameters, so the reliability of regime assignments for individual model-contamination pairs is unknown.

These qualifications do not invalidate the three-regime framework — it is a valuable conceptual contribution regardless of the precision of the mathematical derivation. But they do mean that the exact boundaries between regimes, the quantitative predictions of the survival probability formula, and the claim that temperature shifts models between regimes via the specific α/τ relationship should be treated as theoretical predictions awaiting direct empirical validation, not as established facts. Experiments that measure actual generation trajectories (not teacher-forced NLLs) and directly fit survival curves as a function of length and temperature would strengthen these claims considerably.

Overall Assessment

The paper's central empirical findings — that contamination on generative benchmarks produces brittle, verbatim memorization that collapses under rephrasing, perturbation, and high-temperature sampling, and that temperature and solution length modulate how memorization manifests — are well-supported by the experimental evidence. The scaling law analysis demonstrating that even a single replica enables bypassing the uncontaminated irreducible error is empirically grounded within the fitted compute range.

The paper's theoretical framework — the three survival regimes and the temperature-as-truth-serum mechanism — is conceptually valuable but has not been directly validated. The derivations rely on approximations (independence, sum-to-integral, α_eff = α/τ) that are not empirically tested. The framework should be understood as a productive hypothesis rather than an experimentally confirmed theory.

The most significant limitations from an experimental standpoint are: (1) the absence of statistical uncertainty quantification (no error bars, confidence intervals, or multiple seeds), meaning the reliability of individual data points is unknown; (2) the restriction to a single benchmark (MATH) and single model architecture (Qwen 3), limiting generalizability; (3) the reliance on untested extrapolation for the "infinite compute" scaling law claim; (4) the missing overtraining data for the two largest model sizes; and (5) the untested mechanisms for the SFT and temperature effects. These limitations do not undermine the paper's qualitative conclusions, but they do mean that the precise quantitative thresholds, the exact functional forms, and the generalizability to other settings remain open questions.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Main Results

The assumption or constraint. The scaling law analysis establishes that a single test set replica enables achieving lower loss than the irreducible error of the uncontaminated corpus, and quantifies how much compute contamination "simulates" in equivalent terms. However, this analysis treats the benchmark itself as a known quantity — the test set is available for repeated evaluation during experimentation. In practical deployment or benchmark auditing, knowing that contamination has occurred and quantifying its severity requires detecting the contamination first, which is an unsolved problem that the paper does not address.

The authors acknowledge this implicitly in their treatment of the EleutherAI Evaluation Harness bug (Appendix C): the bug caused gold reference solutions to score only ~70%, and fixing it was essential for the paper's measurements to be valid. But this means that even the benchmark's own maintainers, with full access to the test set and reference solutions, were using a broken evaluation pipeline without realizing it. If the community cannot reliably measure Math Verify scores on models they control and evaluate directly, then detecting contamination in black-box or API-only models — where one cannot inspect training data, compute per-token NLLs under teacher forcing, or sweep hyperparameters — is substantially harder.

The paper does not propose or evaluate any contamination detection method. The analyses in Sections 3–5 are retrospective: given a known contaminated model, how does contamination manifest? The forward problem — given an unknown model, determine whether and how much contamination is present — is not addressed.

The consequence. The paper's scientific contributions (the three regimes, the temperature-truth-serum effect, the scaling law breach) require controlled access to both the model and the benchmark to be applied. A practitioner who suspects a model is contaminated cannot directly apply the paper's framework to quantify the contamination dose without having trained the model themselves. They could, in principle, evaluate the model at multiple temperatures and solution lengths and compare the degradation pattern to the paper's phase diagrams (Figure 8), but this requires assuming that the paper's fitted parameters (E, A, α) transfer to different model architectures, training distributions, and contamination protocols — an assumption that is untested.

Conversely, a benchmark designer cannot use the paper's results to certify that a particular benchmark is contamination-resistant because the severity of contamination depends on the benchmark's distributional properties (the authors' hypothesis that MATH is distinctively memorizable relative to FineWeb-Edu-Dedup). Without a predictive theory of which benchmarks are resistant to single-shot memorization, each benchmark would need its own scaling law analysis to assess vulnerability.

What evidence exists in the paper. The paper does not measure the difficulty of contamination detection — it neither proposes nor evaluates detection methods. The only indirect evidence is the rephrased/perturbed evaluation (Table 1), which shows that contamination effects are detectable if one has the ability to create and evaluate on modified test sets. But this requires creating rephrased or perturbed variants, which itself requires access to the test set and a method for generating valid modifications — not a trivial capability, and one that itself risks further contamination if the modifications are not carefully isolated.

The authors flag the contradiction with Huang et al. (2024) and Hayes et al. (2025) regarding single-shot memorization detectability, but frame it as an open question:

"Future work should aim to understand this difference; one possible explanation is that MATH is distributionally different from FineWeb-Edu-Dedup in a way that makes identifying test set contamination easier."

This is an acknowledgment that the paper's scaling law result may not generalize, not a resolution.

Mitigation status. Not addressed. The paper does not propose detection methods, does not evaluate existing detection methods on its controlled contamination models, and does not provide practical guidance for practitioners who need to determine whether a model they encounter is contaminated. The Discussion notes that "future work should investigate what the explanatory differences are" for the memorization threshold question, but does not frame detection as a specific next step. This is a significant gap: the paper characterizes what contamination does once it has occurred, but provides no tools for determining whether it has occurred in the first place.

Single Benchmark, Single Model Architecture, Single Task Domain

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021b) with the Qwen 3 architecture (Yang et al., 2025a). The authors justify the choice of MATH by its properties — large test set (5,000 problems), availability of full solutions, presence of both problems and answers enabling automatic verification, and the benchmark's status as a canonical generative evaluation — but do not test any other benchmark, any other architecture, or any other task domain (e.g., code generation, formal proof, scientific reasoning). The authors acknowledge this as a limitation:

"We focused on a single generative benchmark, MATH, to enable automatic verification and controlled contamination. Consequently, our findings may not fully capture how contamination behaves for other tasks such as coding or creative writing."

They also note the model scale limitation:

"Additionally, our experiments utilized decoder-only dense transformer models (Qwen 3) up to 344M parameters; results at this scale may not extrapolate to larger models or other architectures."

The consequence. The paper's headline results — the three memorization regimes, the temperature-truth-serum effect, the single-replica irreducible error breach — may be specific to mathematical text as a domain, and potentially to MATH's specific formatting conventions (LaTeX math, \boxed{} notation, consistent problem-solution structure). Mathematical text has properties that may make it distinctively memorizable: it is highly structured, uses a restricted vocabulary relative to natural language, contains recurring patterns (equation formatting, proof structures), and has a strong sequential dependency structure (each step follows from the previous one). These properties may amplify both the memorizability (easy to learn the patterns) and the brittleness (easy to detect deviations) of contamination.

For other generative tasks — code generation (where solutions have different structural properties and correctness can be verified by execution), creative writing (where there is no single "correct" answer), or scientific reasoning (where solutions may involve natural language arguments rather than formal derivations) — the memorization dynamics could differ qualitatively. Code, for instance, has a more rigid syntax than mathematical LaTeX but also more local dependencies (a bug in one line may not affect subsequent lines if they are syntactically independent), which could produce different α values and different regime boundaries. The paper's framework might still apply — the three-regime taxonomy is abstract enough to be domain-general — but the specific thresholds (what contamination dose produces α > 1, what temperature produces the differential collapse) would almost certainly change.

The architecture limitation is similarly consequential. The Qwen 3 architecture uses standard dense attention with Flash Attention 2 and a specific feed-forward expansion ratio. Different architectures — mixture-of-experts (MoE), state-space models (SSMs), or models with different attention mechanisms — might exhibit different memorization dynamics due to differences in capacity allocation, routing, or context processing. The paper acknowledges this in the Discussion:

"We studied dense models, but it remains an open question whether Mixture-of-Experts (MoEs) or State Space Models (SSMs) exhibit different memorization patterns."

What evidence exists in the paper. The paper contains no cross-benchmark, cross-architecture, or cross-domain experiments. All 45+ pretraining runs, all overtraining experiments, all SFT experiments, and all temperature/length analyses are on MATH with Qwen 3. The generalizability of the results is an entirely open question. The authors' hypothesis about MATH's distributional distinctiveness — that it is easier to detect contamination in MATH than in natural language benchmarks — is a speculation about why their results differ from prior memorization work, not a measured property.

Mitigation status. The authors explicitly flag both limitations in the "Limitations" subsection of the Discussion:

"Future Directions: Several avenues for future research: ... Architectural Susceptibility ... Harder Benchmarks: We hypothesize that contamination efficacy is inversely correlated with problem compressibility."

This is an acknowledgment and a suggestion for future work, not a mitigation. The paper does not run even a single experiment on a second benchmark or a second architecture to provide evidence about transferability. This is understandable given computational constraints — adding a second benchmark would double the experimental matrix — but it means the paper's claims should be interpreted as findings about MATH under Qwen 3, with unknown generalizability.

The 344M Parameter Cap Limits the Scaling Law Extrapolation and the Generalizability to Frontier Models

The assumption or constraint. The largest model pretrained is 344M parameters, trained on approximately 6.88B tokens. The scaling law analysis (Figure 2) fits curves to five data points spanning compute budgets from approximately 10¹⁷ to 10¹⁹ FLOP, then extrapolates to infinite compute. The temperature and survival analyses (Figures 5–8) are performed on these same models.

Frontier language models are 3–4 orders of magnitude larger (7B to 400B+ parameters) and are trained on 2–4 orders of magnitude more data (trillions of tokens). The paper does not test whether the functional forms, parameter values, or regime boundaries observed at the 34M–344M scale persist at the 7B–70B+ scale. The authors acknowledge this:

"Additionally, our experiments utilized decoder-only dense transformer models (Qwen 3) up to 344M parameters; results at this scale may not extrapolate to larger models or other architectures."

The consequence. The scaling law claim that a single replica enables bypassing the uncontaminated irreducible error depends on an extrapolation over approximately 5+ orders of magnitude in compute (from the fitted range of 10¹⁷–10¹⁹ FLOP to frontier training runs at 10²³–10²⁵ FLOP). If the functional form changes at larger scales — for instance, if the irreducible error E(R) itself depends on model scale in ways not captured by the three-parameter power law (e.g., if E(R) itself has a power-law dependence on model size) — then the extrapolation is invalid.

Similarly, the temperature-truth-serum effect is demonstrated at the 34M–344M scale. At frontier scale, models exhibit qualitatively different behaviors — emergent capabilities, different calibration properties, possibly different memorization dynamics — that could alter the relationship between temperature, solution length, and contamination-driven performance. For example, larger models might have higher native α (faster confidence growth with context) due to greater capacity for in-context pattern matching, potentially making Deterministic Lock-In achievable at lower contamination doses or making native α > 1 even under uncontaminated conditions (which would mean the temperature-truth-serum effect would not discriminate memorization from generalization, since both would exhibit lock-in).

The overtraining crossover points (Finding #4) are shown to depend on model size — the crossover shifts from 32 replicas at 34M to 1 replica at 93M. Extrapolating this trend, a 7B model might lose its contamination advantage at less than 1 replica, meaning that even minimal contamination would be diluted by moderate overtraining. But this extrapolation is speculative without data at larger scales. The linear trend might not continue; the crossover might saturate at some minimum value above zero.

What evidence exists in the paper. The scaling law fits (Figure 2) show average fitting error < 10⁻² within the fitted range, but this measures interpolation quality, not extrapolation quality. The fitted curves match the five data points well, but this is expected for a three-parameter fit to five points — the fit has enough degrees of freedom to match the training data without constraining the asymptotic behavior. There is no held-out test of the extrapolation (e.g., training a 500M or 1B parameter model and checking whether its loss matches the extrapolated scaling law).

The model-size trends that are observed (Figure 1, right: loss ratio increases with model size; Figure 3: overtraining crossover shifts leftward with model size) are consistent across the 34M–93M range (or 34M–344M for the loss ratio), but the 93M–344M range is less than one order of magnitude of parameters. Whether the trends continue monotonically over two additional orders of magnitude (to 7B+) is unknown.

Mitigation status. The authors explicitly flag the scale limitation and suggest future work, but do not perform any scaling analysis to assess whether the observed trends are likely to continue. The paper could have included a "scaling of the scaling law parameters" analysis — fitting how E, C₀, and α change with model size itself, and using that to predict behavior at larger scales — but does not. The Discussion notes:

"Our experiments utilized decoder-only dense transformer models (Qwen 3) up to 344M parameters; results at this scale may not extrapolate to larger models or other architectures."

This is an honest acknowledgment but not a mitigation.

No Statistical Uncertainty Quantification — Single Models per Condition, No Error Bars

The assumption or constraint. The paper trains exactly one model per experimental condition (model size × contamination level). There is no replication across random seeds, no ensemble averaging, and no reporting of confidence intervals, standard errors, or statistical tests anywhere in the paper. The authors do not discuss this as a methodological choice or acknowledge it as a limitation explicitly, though it is visible from the experimental design: "We trained one model for each pair of model size and number of test set replicas" (Section 2).

This is a standard practice in scaling law research where computational constraints make replication expensive — training multiple seeds per condition would multiply an already large experimental matrix by a factor of 2–5×. But it means that every data point, every fitted curve, and every conclusion is conditional on the specific random initialization, data ordering, and optimization trajectory of a single training run.

The consequence. The scaling law fits (Figure 2) might be sensitive to outliers — a single model that happened to train unusually well or poorly at a particular (N, R) condition could bias the fitted parameters. With five points per fit, a single outlier can exert substantial leverage. The clean monotonic trends in the fitted parameters (E(R) decreasing smoothly with R, C₀(R) and α(R) roughly constant) suggest that the data is not dominated by noise, but the magnitude of run-to-run variance is unknown.

The temperature and survival analyses (Figures 5–8) compound this issue. The per-token NLL trajectories are derived from a single model per condition; the fitted (E, A, α) parameters inherit any noise from that single training run. The regime classifications (Exponentially Fast Decoherence, Brittle Memorization, Deterministic Lock-In) for individual (N, R) pairs are therefore point estimates without associated uncertainty. A model that is classified as Deterministic Lock-In because its fitted α > 1 might have α = 1.2 ± 0.4, where the uncertainty interval crosses the α = 1 threshold, making the classification unreliable.

The SFT result (Figure 4), which shows a non-monotonic effect, is the most vulnerable to seed noise. The pre-SFT and post-SFT loss values for each model are single observations; the crossover point where SFT shifts from helpful to harmful (around 10 replicas) might shift by ±5 replicas with different random seeds, potentially changing the qualitative interpretation.

What evidence exists in the paper. The paper provides no evidence about run-to-run variance. There are no error bars on any figure, no tables of standard deviations across seeds, and no discussion of statistical reliability. The only quantitative measure of fit quality is the average fitting error for the scaling laws (< 10⁻²), which measures consistency with the parametric form, not uncertainty in the parameters.

The smooth trends across model sizes and contamination levels provide some informal reassurance — if the results were dominated by noise, the systematic patterns (e.g., E(R) decreasing monotonically with R in Figure 2; the loss ratio increasing with model size in Figure 1, right; the crossover point shifting monotonically with model size in Figure 3) would be unlikely. However, these trends could still be present but with enough noise to make individual thresholds unreliable. The lack of quantitative uncertainty makes it impossible to distinguish between a genuine effect and a chance fluctuation, particularly for threshold-dependent claims (the α = 1 phase transition, the 10-replica SFT crossover).

Mitigation status. Not addressed. The paper does not mention seed variance, does not train multiple seeds for any condition, and does not report any form of uncertainty quantification. This is likely a computational constraint rather than an oversight — training 45+ models from scratch for the base sweep, plus additional runs for overtraining and SFT, represents a substantial compute investment. Adding even 2 seeds per condition would at least double this cost. However, the paper could have trained multiple seeds for a subset of conditions (e.g., just the 93M model at 3–5 contamination levels) to provide an estimate of seed variance and characterize how it scales with model size and contamination. The absence of even a minimal uncertainty analysis means the reliability of individual data points and the precision of threshold estimates are entirely unknown.

The Temperature-to-Regime Derivation Relies on Untested Approximations

The assumption or constraint. Section 5 derives the three memorization regimes and the temperature-truth-serum mechanism through a sequence of mathematical steps that have not been empirically validated:

1. The per-token negative log-likelihoods $L_t$ are fit to a power-law form $L_t \approx E + A \cdot t^{-\alpha}$ under teacher forcing — the model receives the correct prefix at each step.
2. The survival probability $P(T)$ for autoregressive generation is approximated as $\exp(-\sum L_t)$, assuming token-level errors are independent and that the probability of correctly predicting token $t$ given the correct prefix is approximately $e^{-L_t}$.
3. The discrete sum is converted to a continuous integral.
4. The effective exponent under temperature $\tau$ is approximated as $\alpha_{\text{eff}} \approx \alpha / \tau$, based on the argument that logit gaps grow as $\Delta z_t \propto \alpha \ln t$.

None of these steps is directly validated. The paper does not compare the predicted survival probabilities $P(T)$ from Equation 4 to actual generation success rates (i.e., the observed Math Verify scores stratified by length). It does not measure logit gaps directly to confirm $\Delta z_t \propto \alpha \ln t$. It does not test whether $\alpha_{\text{eff}} = \alpha / \tau$ by fitting $\alpha_{\text{eff}}$ from generation trajectories at different temperatures and comparing to the predicted relationship. The survival probability framework is a mathematical model derived from teacher-forced measurements, but its predictions for autoregressive generation are not tested against empirical generation data.

The consequence. The three-regime classification — and particularly the claim that temperature shifts models between regimes via $\alpha_{\text{eff}} = \alpha / \tau$ — is a hypothesis derived from a model, not an empirically established fact. The framework might correctly predict qualitative behavior (high temperature degrades performance, longer solutions are harder to generate) while failing quantitatively or in edge cases. Specifically:

• The independence assumption is almost certainly violated in practice: after the first token error, the model enters an off-distribution state that may make subsequent errors more (or less) likely. The magnitude of this violation is unknown and could substantially affect $P(T)$ for long solutions.
• The sum-to-integral conversion introduces approximation error for short-to-medium solution lengths, which is where the regime distinctions are most consequential (short solutions may succeed even in Exponentially Fast Decoherence if $E$ is small).
• The $\alpha_{\text{eff}} = \alpha / \tau$ relationship assumes that temperature affects all logits uniformly and that the logit gap is the primary determinant of sampling probability, but in practice, the softmax temperature also affects the relative probabilities of competing tokens in ways that may not reduce to a simple rescaling of the exponent.

What evidence exists in the paper. The paper provides indirect support for the framework through the qualitative consistency between the model's predictions and observed behavior: high temperature degrades contaminated performance dramatically (Figure 5, consistent with $\alpha_{\text{eff}} < \alpha$), and longer solutions are harder (Figure 6, consistent with the survival probability accumulating with length). However, these observations are also consistent with simpler explanations (e.g., higher temperature increases the probability of sampling any particular incorrect token, longer solutions provide more opportunities for error) that do not require the specific power-law NLL or $\alpha / \tau$ mechanism.

The per-token NLL trajectories (Figure 7, top) follow the power-law form for most conditions, providing support for step 1. But steps 2–4 — connecting teacher-forced NLLs to autoregressive generation survival probabilities — are entirely untested. The cumulative probability curves in Figure 7 (bottom) are computed from the teacher-forced NLL fits using Equation 4, not measured from actual model generations. This is a crucial distinction: the bottom panel of Figure 7 does not show empirical survival probabilities; it shows the survival probabilities predicted by the model based on teacher-forced measurements. Whether these predictions match actual generation behavior is unknown.

Mitigation status. Not addressed. The paper presents the survival probability derivation as part of the analysis without noting the assumptions or the lack of direct validation. The Discussion does not flag this as a limitation. This is the paper's most significant methodological gap: the theoretical framework that distinguishes generative from discriminative contamination, and that forms the basis for the "truth serum" claim and the three-regime taxonomy, is an untested model rather than a validated theory. Future work that directly compares predicted $P(T)$ to observed generation success rates, measures logit gaps during generation, and tests the $\alpha_{\text{eff}} = \alpha / \tau$ relationship would be necessary to validate the framework.

The Contamination Protocol May Not Reflect Real-World Contamination Scenarios

The assumption or constraint. The paper's experimental design assumes a specific contamination protocol: the MATH test set is replicated $R$ times and uniformly shuffled into the pretraining corpus, formatted using the exact template from the EleutherAI LM Evaluation Harness ("Problem: {problem}\n\nSolution: {solution}"), and training proceeds from random initialization with contamination present throughout. This is a worst-case controlled experiment designed to maximize the detectability and severity of contamination effects.

Real-world contamination is likely to be messier. Benchmark problems might appear on the web in different formats (HTML pages, PDFs, forum discussions with extraneous text), might be partially corrupted (OCR errors, missing parts), might appear alongside other content that dilutes their signal, or might appear only in specific epochs of a web crawl rather than uniformly throughout training. Additionally, real contamination might involve only the problems without solutions, or solutions without the final boxed answers, or paraphrased versions of problems that are semantically equivalent but tokenically distinct. The paper's rephrasing/perturbation experiment (Table 1) shows that contamination benefits collapse when the token sequence changes, suggesting that real-world contamination that does not perfectly match the evaluation format may be substantially less effective.

The authors acknowledge the specificity of the contamination format implicitly (they describe their formatting choice as using the exact evaluation harness template) but do not discuss how results might change with more realistic contamination protocols beyond the rephrasing/perturbation test.

The consequence. The paper's quantitative results — the dose-response curves, the irreducible error breach, the regime boundaries — are upper bounds on contamination severity. In practice, contamination may be less effective because:

• Web-crawled benchmark instances may use different formatting (reducing verbatim overlap with the evaluation prompt).
• Problems may appear without solutions, providing only partial contamination.
• Solutions may appear without the final \boxed{} answer, making the memorized sequence incomplete for end-to-end generation.
• Contamination may be concentrated in specific documents rather than uniformly distributed, potentially interacting with the model's training dynamics (e.g., if contamination appears late in training when the learning rate is low, memorization may be less effective).

The finding that even a single replica breaks the irreducible error barrier is particularly sensitive to the contamination protocol. If the single replica in a real web crawl is an HTML page with extraneous navigation text, advertisements, and user comments — rather than a clean "Problem: ... Solution: ..." format — its memorizability might be substantially reduced. The paper shows that contamination works via verbatim token sequence matching (Table 1), so any deviation from the exact evaluation format reduces the effective contamination dose.

What evidence exists in the paper. The rephrased/perturbed evaluation (Table 1) provides the only direct evidence about robustness to contamination format variation. It shows that when the token sequence changes — even while preserving meaning — contamination benefits vanish entirely. This suggests strong brittleness to format, which would imply that real-world contamination would need to match the evaluation format closely to be effective. However, the rephrasing/perturbation experiment changes the test problems, not the contamination format — it tests whether the model transfers to modified problems, not whether modified contamination produces different effects. These are distinct questions, and the paper does not vary the contamination format itself.

Mitigation status. Partially addressed through the rephrasing/perturbation evaluation, but the contamination format is not directly varied. The Discussion does not identify contamination protocol variation as a specific limitation. Future work could systematically vary the contamination format (matching vs. mismatched templates, problems with vs. without solutions, clean vs. web-scraped formatting) to characterize how contamination effectiveness degrades with format mismatch.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that generative evaluation is not discriminative evaluation plus length—it is a fundamentally different regime for contamination, governed by a probabilistic survival process that has no analog in multiple-choice or classification tasks. The implications of this shift span how the field should design benchmarks, audit models, and interpret reported scores.

A new diagnostic axis: the temperature-sensitivity of performance. The paper's most immediately actionable insight is that sampling temperature is not just a generation quality knob but a veracity diagnostic. The finding that contaminated models suffer a 40× performance collapse when moving from greedy decoding (τ = 0) to τ = 1.0, while uncontaminated models see only a ~2× drop (Figure 5), provides a concrete, zero-cost auditing mechanism: evaluate at multiple temperatures and examine the degradation ratio. If a model's MATH score falls by 20× or more between τ = 0 and τ = 1.0, the paper's results suggest that most of its greedy-decoding performance derives from brittle memorization rather than robust mathematical reasoning. This does not require access to training data, model weights, or the benchmark itself—only the ability to query the model at different temperatures, which is available for most API-based models. The temperature-truth-serum effect thus converts a standard hyperparameter into a contamination litmus test that any evaluator can apply today.

This is a methodological shift, not a paradigm shift—temperature sweeps are already common in evaluation. But the paper provides the mechanistic understanding (α_eff = α/τ and the phase transition at α = 1) that transforms temperature from a sensitivity analysis into a principled diagnostic. Prior to this work, an evaluator who observed a large temperature-dependent performance drop might attribute it to poor calibration or sensitivity to stochasticity. The paper shows that the differential sensitivity—contaminated performance being disproportionately more fragile than uncontaminated performance—has a specific, theoretically grounded interpretation as exposure of the native α exponent. This is a contribution to evaluation methodology that can be adopted immediately without new infrastructure.

Reframing what a benchmark measures. The scaling law analysis (Finding #3) delivers a sobering message: on MATH under Qwen 3, a single replica of the test set in the pretraining corpus enables achieving lower cross-entropy loss than the irreducible error of training on the uncontaminated corpus. Under standard extrapolation assumptions, this means that even minimal contamination can simulate "infinite" clean compute. This reframes contamination not as a percentage-point inflation of an otherwise valid measurement, but as a qualitative breach of the benchmark's asymptotic validity. The benchmark no longer measures generalization in the limit—it measures some mixture of generalization and memorization, and the memorization component can dominate even at contamination doses that would be undetectable by n-gram overlap with web-scale data (a single replica in billions of tokens).

This has immediate implications for how benchmark designers think about their work. A benchmark whose test set has ever appeared anywhere on the public web—in a forum post, a PDF, a GitHub repository—is at risk of having its asymptotic measurement properties compromised. The paper does not claim this applies universally (the authors hypothesize MATH's structured format makes it distinctively memorizable), but the framework for assessing vulnerability—fit scaling laws for L(C, R) and compare E(R) to E(0)—provides a concrete methodology that benchmark designers can apply to their own tasks. A benchmark for which E(1) ≈ E(0) is asymptotically robust to single-copy contamination; a benchmark for which E(1) ≪ E(0) (as with MATH) is asymptotically compromised.

Reconciling contradictory prior findings on memorization and SFT. The paper provides a unifying framework for several tensions in the literature:

1. Huang et al. (2024) claimed single-shot verbatim memorization is an "illusion"; Hayes et al. (2025) claimed membership inference AUC asymptotes at ~0.689. The paper's scaling law result that E(1) < E(0) contradicts both claims for MATH specifically. The authors hypothesize that MATH's distributional distinctiveness (structured LaTeX, \boxed{} notation) makes it more memorizable than the natural language corpora studied in prior work. This does not invalidate the prior findings—it contextualizes them. Memorization severity is benchmark-dependent, and studying contamination on one distribution (e.g., FineWeb-Edu-Dedup natural language) does not necessarily predict behavior on another (e.g., MATH). This reframes the question from "how severe is single-shot memorization?" to "for which types of data is single-shot memorization severe?"—a more nuanced and productive framing.

2. Kocyigit (2025) found SFT generally improves performance after contamination; this paper finds SFT harms highly contaminated models. The discrepancy is likely methodological: Kocyigit (2025) induced contamination via continued pretraining (a distinct phase after initial training), while this paper introduces contamination uniformly throughout pretraining. The paper's proposed mechanism—SFT causes forgetting of memorized test examples—may operate differently (or not at all) when memorization is acquired in a separate continued-pretraining phase where representations are less deeply integrated. This does not resolve the contradiction but illuminates it: SFT-contamination interactions depend on when and how contamination occurs, not just on the contamination dose.

The three-regime framework as a conceptual contribution. The paper's taxonomy—Exponentially Fast Decoherence (E > 0), Brittle Memorization (E ≈ 0, α ≤ 1), Deterministic Lock-In (E ≈ 0, α > 1)—provides a language for discussing memorization quality that did not previously exist. Prior work treated memorization as a scalar (how much probability mass on the correct token), but the paper shows that the shape of the per-token confidence trajectory—captured by α—determines whether memorization translates to successful generation. Two models with identical average per-token memorization strength (same E, same mean NLL) can have qualitatively different generative behavior if one has α < 1 (eventual decoherence) and the other has α > 1 (asymptotic lock-in). This is a genuinely new dimension for characterizing contamination effects, and it opens research directions that were previously invisible.

Which research directions become more attractive, and which less so.

More attractive: (1) Benchmark-specific memorization susceptibility measurement—the scaling law framework provides a concrete methodology for assessing which benchmarks are most vulnerable to contamination. (2) Temperature-stratified evaluation protocols—the differential temperature sensitivity is a diagnostic that should become standard practice. (3) Understanding the architectural and distributional determinants of α—why do some models/data achieve α > 1 while others do not? (4) Adversarial robustness of verifiers—the paper identifies verifier over-optimization as a bottleneck, redirecting effort from better search to better verifiers. (5) Joint optimization of pretraining and inference compute allocation, with contamination as a variable to be minimized rather than an accident to be detected.

Less attractive: (1) Contamination detection methods that rely solely on average probability metrics (e.g., perplexity-based membership inference) without considering the generative survival dynamics—the paper shows that two models with similar average memorization can have radically different generation behavior. (2) Uniform evaluation protocols that report only greedy-decoding scores or a single temperature—the paper demonstrates that this can mask contamination by creating false lock-in. (3) Contamination mitigation strategies that focus exclusively on pretraining data filtering without considering how inference-time parameters (temperature, length) modulate contamination expression—the paper shows that inference is a point of leverage for both exposure and mitigation.

Follow-Up Research This Work Enables

Direct measurement of autoregressive survival curves to validate the three-regime framework. The paper derives the survival probability P(T) from teacher-forced NLL measurements using Equation 4, but never directly measures P(T) from actual autoregressive generations. A direct validation would: (1) for each (N, R) condition, generate K solutions at various temperatures using the model's own autoregressive sampling (not teacher forcing); (2) compute the empirical survival rate as a function of solution length—the fraction of generated solutions that are correct among those whose reference solutions have length T; (3) compare the empirical survival curves to the predictions from Equation 4 using the fitted (E, A, α) from teacher-forced measurements; (4) quantify the approximation error from the independence assumption and the sum-to-integral conversion. If the predicted and empirical curves match, the framework is validated; if they diverge systematically (e.g., the model achieves higher empirical survival than predicted, suggesting error correction), the framework needs refinement. This experiment is computationally feasible—it requires only inference on the already-trained models, not new pretraining.

Benchmark susceptibility catalog: which generative benchmarks are asymptotically compromised by a single replica? The paper's scaling law methodology—pretrain models on corpora with R ∈ {0, 1, 3, ...} replicas, fit L(C, R) = E(R) + C₀(R)·C^(-α(R)), and test whether E(1) < E(0)—is directly portable to other benchmarks. A systematic study would apply this protocol to: (1) GSM8K (grade-school math, shorter solutions ~50–150 tokens), (2) HumanEval (code generation, structured but different domain), (3) a natural language reasoning benchmark (e.g., a subset of BIG-Bench), and (4) a benchmark with deliberately low structure (free-form creative writing evaluation). The hypothesis is that the E(1) < E(0) breach depends on benchmark structure—highly structured benchmarks (math, code) may be more vulnerable than unstructured ones. If confirmed, this would provide evidence-based guidance for benchmark design: benchmarks that are inherently resistant to single-shot memorization should be prioritized for high-stakes evaluation. If all benchmarks show E(1) < E(0), the problem is universal and more fundamental interventions (dynamic benchmarks, private test sets) are necessary.

Determining the architectural and capacity determinants of α > 1. The paper observes Deterministic Lock-In (α > 1) only for highly contaminated models, but does not explore what architectural properties or training conditions produce high native α. A systematic study would: (1) measure α for a range of architectures at matched parameter counts (dense transformers, MoE transformers, state-space models, and potentially retrieval-augmented models) trained on identical contaminated corpora; (2) within the dense transformer family, vary depth, width, and attention mechanism while measuring α; (3) test whether α correlates with standard capacity metrics (total parameters, FLOPs, effective context length) or with emergent phenomena (in-context learning ability). The practical question is whether certain architectures are inherently more resistant to memorization lock-in (lower α for the same contamination dose), which would inform architecture selection for models deployed in settings where benchmark contamination risk is high (e.g., models whose training data cannot be fully audited).

SFT as a deliberate decontamination intervention: characterizing the forgetting dynamics. The paper's Finding #5—that SFT on the MATH training set degrades test performance for highly contaminated models—suggests SFT could be used intentionally as a decontamination procedure, but the dynamics are uncharacterized. A follow-up study would: (1) measure the per-token NLL on test solutions before, during, and after SFT to track whether SFT causes uniform forgetting (all tokens equally affected) or selective forgetting (early tokens, which provide cues for the rest of the solution, are more affected); (2) vary the SFT dataset to determine what properties maximize test-set forgetting while preserving generalization—does the training set need to be from the same distribution as the test set, or can any task-specific data cause forgetting? (3) test whether SFT followed by selective replay of the contaminated data (a small amount of retraining on the original pretraining mixture) can restore generalization without restoring memorization; (4) quantify the tradeoff between decontamination (reducing memorization-driven performance) and legitimate capability loss (reducing generalization-driven performance) as a function of SFT duration and learning rate. If SFT can be tuned to selectively erase memorized test sequences while preserving task-relevant generalization, it would be a practical tool for model developers who discover contamination after pretraining.

Temperature-based contamination auditing for black-box models: operationalizing the truth serum. The paper demonstrates that the ratio of performance at τ = 0 to performance at τ = 1.0 is a strong signal of contamination (40× for R = 1000 vs. 2× for R = 0), but does not provide a decision rule or characterize the detection power. A practical follow-up would: (1) establish a null distribution for the temperature-degradation ratio on models known to be uncontaminated (trained on verified clean data), measuring how the ratio varies with model size, architecture, and benchmark; (2) for a set of models with known contamination doses (from controlled experiments like this paper's), establish the sensitivity and specificity of a decision rule based on the degradation ratio; (3) determine the minimum contamination dose detectable at a given false-positive rate; (4) test whether the diagnostic works for models where contamination is introduced through realistic mechanisms (web-scraped benchmark instances in varied formats) rather than the clean-template contamination used in this paper. The goal is a practical auditing protocol: for a new model claiming a MATH score of X at greedy decoding, evaluate at τ = 1.0; if the score drops by more than a calibrated threshold (e.g., 5× compared to an uncontaminated reference of similar scale), flag the model for contamination investigation. This requires no access to training data and can be performed through an API.

The dose-response curve for realistic contamination formats. The paper uses a single, clean contamination format (the exact evaluation harness template) to establish an upper bound on contamination effects. A critical extension would systematically degrade the contamination format toward realism: (1) vary the template from exact match to "Problem: ... Solution: ..." to "Q: ... A: ..." to raw LaTeX without wrappers to HTML-formatted forum posts; (2) introduce noise (random character substitutions, OCR errors, missing tokens) into the contaminated replicas; (3) provide only problems without solutions, or solutions without the final boxed answer; (4) surround MATH problems with unrelated web text (mimicking a forum page where the problem is embedded in discussion). For each format, measure the effective contamination dose—how many replicas of degraded contamination produce the same performance as one replica of clean-template contamination. The hypothesis is that format mismatch acts as a dose reduction factor, and quantifying this factor would allow translating between controlled-experiment contamination doses and estimated real-world contamination exposure. If the reduction factor is large (e.g., 100×—it takes 100 web-scraped replicas to match 1 clean replica), then real-world contamination may be much less severe than the paper's upper bounds suggest, providing partial reassurance while also quantifying the residual risk.

Practical Applications and Downstream Use Cases

Temperature-stratified evaluation for model cards and leaderboards. The paper's finding that contamination-driven performance collapses at τ = 1.0 while uncontaminated performance remains relatively stable provides an immediately deployable evaluation protocol. A model card reporting MATH performance should include not just a single greedy-decoding score, but a table of scores at τ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1.0}. The ratio of the τ = 0 score to the τ = 1.0 score is a diagnostic: ratios above ~5× warrant investigation (the paper shows ~2× for uncontaminated models vs. up to 40× for heavily contaminated models). This adds minimal evaluation cost—a temperature sweep requires no additional model training or infrastructure, and the inference cost is linear in the number of temperatures. Leaderboards (e.g., the Open LLM Leaderboard) could incorporate this as a "contamination robustness" metric alongside raw scores, disincentivizing the inclusion of benchmark data in pretraining by making the resulting fragility visible. A model that achieves 90% at τ = 0 but 15% at τ = 1.0 would be transparently relying on brittle memorization, and evaluators could discount its scores accordingly.

Post-training decontamination monitoring. The paper's Finding #5—that SFT on the training set increases test loss for highly contaminated models while decreasing it for uncontaminated models—can be used as a diagnostic during model development. A model development team that runs standard SFT and observes a counterintuitive increase in test loss should consider pretraining contamination as a possible cause. Conversely, if a team suspects their pretraining data may contain benchmark contamination (e.g., because they crawled Common Crawl without exhaustive decontamination), they can use the SFT response as a probe: measure test performance before and after SFT on the training set. A performance drop of more than ~10% relative suggests high contamination (based on Figure 4, where the ratio of pre-SFT to post-SFT test loss exceeds 1.0 for R ≥ 10 replicas). This is not a definitive test—SFT can cause performance drops for other reasons—but it is a low-cost diagnostic that adds information to the contamination assessment toolkit. Teams could also deliberately run a small SFT experiment on a held-out contaminated model to calibrate the expected drop for their specific architecture and training setup.

Scaling law-based benchmark integrity assessment for benchmark designers. The paper's scaling law methodology provides a concrete protocol for benchmark designers to assess the asymptotic vulnerability of their benchmarks to contamination. The procedure is: (1) pretrain a small-to-medium model family (e.g., 50M–500M parameters) on clean data contaminated with R ∈ {0, 1, 10, 100} replicas of the benchmark test set; (2) fit L(C, R) = E(R) + C₀(R)·C^(-α(R)) for each R; (3) test whether E(1) is statistically distinguishable from E(0). If E(1) ≈ E(0), the benchmark is asymptotically robust to single-copy contamination—even if a test problem leaks onto the web, models trained on that leak cannot achieve better asymptotic performance than models trained on clean data. If E(1) ≪ E(0) (as with MATH), the benchmark is asymptotically compromised, and the only robust solutions are dynamic benchmarks (periodically refreshed test sets), private test sets, or benchmarks of genuinely unsolved problems (Nie et al., 2025). This assessment can be performed once per benchmark and published alongside the benchmark to inform users about its contamination vulnerability. The computational cost is modest—training ~20 small models (4 contamination levels × 5 model sizes) is within the budget of most academic labs, and the assessment only needs to be done once for each benchmark (not for each model evaluated on it).

Inference-time contamination mitigation via forced stochastic decoding. The paper's temperature-truth-serum effect can be operationalized as a deliberate mitigation: for high-stakes evaluations where contamination is suspected, evaluators can require models to be assessed at τ ≥ 1.0 rather than the standard greedy decoding. This does not eliminate contamination effects—a model with native α > 1 would still exhibit Deterministic Lock-In even at τ = 1.0—but it eliminates the artificial lock-in created by temperature amplification (α_eff → ∞ at τ → 0). Based on Figure 5, this would reduce or eliminate the performance advantage of models with R ≤ 1000 replicas, bringing their scores closer to the uncontaminated baseline. The cost is a general reduction in scores for all models (both contaminated and uncontaminated) due to increased stochasticity, which evaluators would need to account for by recalibrating expectations or by reporting normalized scores relative to uncontaminated reference models evaluated at the same temperature. This is not a perfect solution—it penalizes legitimate performance alongside contamination—but for applications where false confidence from contamination is more costly than reduced sensitivity to genuine capability (e.g., safety evaluations, capability forecasting for policy decisions), forced high-temperature evaluation provides a conservative measurement that is harder to game through data contamination.