ArXiv: 2508.01191
🎯 Pitch
Chain-of-thought reasoning in LLMs is a brittle mirage: when test problems differ from training data in task structure, length, or format, accuracy collapses from 100% to near zero, even in state-of-the-art models like LLaMA3 and Qwen3. This fragility is exposed in DataAlchemy, a controlled synthetic environment that trains models from scratch and reveals that CoT behavior is merely pattern interpolation, not genuine reasoning.
1. Executive Summary
This paper analyzes whether Chain-of-Thought reasoning in LLMs reflects genuine logical inference or merely interpolates patterns from training data, proposing a data distribution lens that frames CoT effectiveness as fundamentally governed by the discrepancy between training and test distributions along three axes — task (unseen transformation compositions such as ROT-13 followed by cyclic shift), length (different element sizes or reasoning step counts), and format (inserted, deleted, or modified prompt tokens). Through DataAlchemy, a fully controllable synthetic environment that abstracts NLP tasks into atomic symbols and transformations while training models from scratch to avoid data leakage, the authors demonstrate that CoT reasoning achieves 100% exact match on in-distribution queries but collapses to near-zero under even moderate distribution shifts — establishing that what appears as structured reasoning is a brittle mirage that succeeds only when test queries closely mirror training patterns, with SOTA models (LLaMA3-8B, Qwen3-14B-Instruct) exhibiting identical fragility despite their scale and instruction tuning.
2. Context and Motivation
The Core Problem: We Don't Know Whether CoT Reasoning Is Genuine or a Statistical Artifact
The fundamental question this paper tackles is deceptively simple: when an LLM produces a step-by-step reasoning chain and arrives at a correct answer, is it actually reasoning, or is it just reproducing patterns it memorized during training? This distinction matters enormously because the entire promise of Chain-of-Thought prompting — that it makes LLMs more transparent, more trustworthy, and more capable of handling novel problems — rests on the assumption that the intermediate steps reflect something resembling genuine inference. If those steps are instead sophisticated confabulations that happen to align with correct answers when the problem looks familiar, then CoT is not a path toward robust AI reasoning but rather a dangerously misleading veneer of intelligence.
This question is not merely philosophical. The paper's opening section identifies a stark contradiction in the literature that makes the problem urgent and unresolved.
A Genuine Contradiction in the Literature
On one side, CoT prompting has been celebrated as a breakthrough. Wei et al. (2022) showed that appending "Let's think step by step" to prompts substantially improves performance on arithmetic, commonsense, and symbolic reasoning tasks. This spawned an entire research program: zero-shot CoT (Kojima et al., 2022) removed the need for hand-crafted exemplars, self-consistency (Wang et al., 2023) boosted reliability through majority voting over multiple reasoning paths, Tree-of-Thoughts (Yao et al., 2023) enabled search over branching reasoning trajectories, and recent work has embedded long-form CoT directly into inference, producing models that reflect on their own outputs, correct mistakes, and explore alternative strategies (Chen et al., 2025a; Guo et al., 2025; Jaech et al., 2024; Team et al., 2025). The trajectory of these results has led many researchers and practitioners to view CoT as evidence of emergent, human-like reasoning in LLMs — a stepping stone toward artificial general intelligence.
On the other side, a growing body of work paints a much darker picture. Mirzadeh et al. (2025) demonstrated that CoT performance is highly fragile: semantically irrelevant perturbations — adding distractor phrases, altering symbolic representations in ways that don't change the problem's logical structure — can cause dramatic performance drops in state-of-the-art models. Stechly et al. (2024) showed that LLMs fail to generalize in planning tasks, revealing a deficiency in true algorithmic reasoning. Shojaee et al. (2025) found that reasoning models experience an accuracy collapse in puzzle-solving once task complexity exceeds a critical threshold, with models "overthinking" simple problems and giving up on harder ones. Sun et al. (2025) demonstrated that LLMs struggle to solve complex mathematical reasoning problems, failing to integrate or adapt learned skills to novel or creative tasks.
Perhaps most damningly, a line of work on faithfulness has revealed that CoT reasoning traces often function as post-hoc rationalizations rather than causal drivers of the final answer. Lanham et al. (2023) conducted intervention-based analyses showing that final answers frequently remain unchanged even when intermediate reasoning steps are corrupted or removed entirely. Bentham et al. (2024) and Chen et al. (2025b) documented what they call the illusion of transparency: models produce fluent, plausible-sounding reasoning chains that are decoupled from the actual answer-generation process. The reasoning looks right, but it isn't what produced the answer.
These conflicting findings are not necessarily contradictory — they likely reflect different methods being applied to different types of problems, different models, and different evaluation protocols — but the field lacks a unifying framework for understanding when CoT reasoning succeeds and when it fails. The paper's central motivation is that this question cannot be answered by the existing evaluation paradigm.
Three Structural Limitations of Current Evaluation Approaches
The paper identifies three specific, interrelated problems with how CoT reasoning is currently evaluated (Section 1, elaborated in Appendix A.4 and the comparison figure — Figure 10). These limitations are not incidental; they are fundamentally baked into the standard research workflow of testing pretrained LLMs on real-world benchmarks.
Limitation 1: Narrowly Defined Settings
Existing studies focus on specific tasks (MATH, GSM8K, planning domains, logical deduction puzzles), specific models (GPT-4, PaLM, LLaMA), and specific prompting strategies. While this produces valuable engineering results, it obscures common structural patterns across tasks and models. If CoT fails on a planning task but succeeds on algebra, is that because planning is harder, because the model saw more algebra during pretraining, or because the planning task introduces a specific kind of distribution shift? Without a framework that abstracts across tasks and models, these questions cannot be answered — each study produces a data point, not a principle.
The paper argues that what's needed is an approach that identifies invariant properties of CoT reasoning across tasks, models, and architectures. The specific numbers (70% on GSM8K, 50% on MATH) are less informative than the functional relationship between distribution shift and performance, which should — if it reflects something fundamental about how LLMs work — be consistent regardless of whether the "task" is algebra or abstract symbol manipulation.
Limitation 2: Data Entanglement
Real-world NLP tasks are complex, high-dimensional, and deeply entangled. Consider a math word problem: it involves natural language understanding, entity extraction, arithmetic computation, multi-step planning, and symbolic manipulation — all simultaneously. If CoT fails on such a problem, which factor caused the failure? Was it the linguistic complexity, the arithmetic difficulty, the number of reasoning steps, or some interaction among these? There is no way to isolate variables.
This is the classic controlled experiment problem: to establish causality, you need to vary one factor at a time while holding all others constant. Real-world benchmarks make this impossible. A math problem with more steps also tends to have more complex language, larger numbers, and more entities to track. You cannot independently vary "reasoning length" without also changing "problem difficulty" in ways that are hard to quantify.
The paper argues that this entanglement precludes rigorous scientific investigation of CoT reasoning. Without the ability to conduct controlled experiments, the field is limited to correlational observations — "CoT helps on task X but not task Y" — which accumulate without converging on explanations.
Limitation 3: Data Leakage and Benchmark Containment
This is perhaps the most pernicious problem. Pretrained LLMs are trained on massive, largely opaque corpora that almost certainly contain examples from standard evaluation benchmarks. When GPT-4 solves a GSM8K problem correctly, is it reasoning through the arithmetic, or is it reproducing a solution it saw during pretraining? There is no way to know, because the training data is proprietary and uncurated.
The paper explicitly cites this as undermining "the effectiveness and validity of evaluations" (Section 1). Even when researchers use held-out test sets, those test sets may have been included in the pretraining corpus through data contamination. The widely documented phenomenon of benchmark memorization (where models achieve high scores on standard benchmarks through pattern matching rather than capability) makes it impossible to distinguish genuine reasoning from retrieval using standard evaluation protocols.
The comparison figure in Appendix A.4 (Figure 10) illustrates this vividly. In the standard evaluation paradigm (top panel), you take a pretrained LLM, give it a test problem (math, code, puzzle), and evaluate the answer. But the pretrained LLM has already seen something similar — perhaps identical — during training. The evaluation is contaminated before it begins. The paper contrasts this with the DataAlchemy paradigm (bottom panel), where models are trained from scratch on fully known, fully controlled data, and evaluated on systematically constructed distributions. Only in this setting can you say with confidence whether a correct answer reflects reasoning or memorization.
The Need for a New Evaluation Framework
Given these three limitations, the paper argues that a new kind of evaluation framework is necessary — one that satisfies three criteria (Appendix A.4):
-
Abstract representation: It should distill diverse NLP tasks into their essential components while retaining the structural properties that make reasoning challenging. The goal is not to replace real-world benchmarks but to complement them with a setting where the underlying structure is known and manipulable.
-
Fully controlled experiments: It should enable fine-grained, independent manipulation of the factors hypothesized to affect CoT reasoning — task structure, length, format — so that causal relationships can be established through controlled variation.
-
Training from scratch: It should provide scalable, synthetic data that can train models from scratch, eliminating the data leakage problem entirely. When you know exactly what the model saw during training, you know exactly what it can and cannot have memorized.
These criteria directly motivate the design of DataAlchemy, which is introduced in Section 4 as a concrete realization of this evaluation philosophy. But before introducing the environment, the paper develops its theoretical framework — the data distribution lens — which provides the conceptual scaffolding for understanding why we should expect CoT reasoning to be distribution-dependent and how to measure distribution shifts systematically.
How This Paper Positions Itself
The paper positions itself not as proposing a new CoT method, but as providing a principled lens for understanding existing results and designing future experiments. This is a meta-scientific contribution: it is about how to study CoT reasoning, not how to improve it.
The relationships to prior work are carefully delineated in Section 2 and Appendix A:
Relative to prompting work (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2023; Yao et al., 2023): The paper is not proposing a new prompting technique, a new search algorithm, or a new decoding strategy. It is asking a more fundamental question: regardless of which prompting method is used, does the resulting reasoning reflect genuine inference or pattern interpolation? The data distribution lens is intended to apply across all prompting variants — any method that produces a reasoning chain is ultimately a generative process conditioned on training data, and is therefore subject to distribution-dependent degradation.
Relative to criticism work (Mirzadeh et al., 2025; Stechly et al., 2024; Lanham et al., 2023): The paper shares these works' skepticism about CoT reasoning but argues that piecemeal criticism — identifying specific failure cases on specific benchmarks — is insufficient. What's needed is a framework that explains why CoT fails in those cases and predicts when it will fail in new ones. The data distribution lens provides this: CoT fails when there is a distribution discrepancy between training and test data, and the severity of failure scales with the magnitude of that discrepancy. This unifies the disparate negative results under a single explanatory principle.
Relative to OOD generalization work (Garg et al., 2022; Li et al., 2025; Wang et al., 2025a,b): The paper connects to the broader machine learning literature on out-of-distribution generalization, but focuses specifically on the reasoning aspect of LLM behavior. While prior work has studied OOD generalization for in-context learning, function approximation, and arithmetic (Cho et al., 2024; Song et al., 2025), the paper argues that CoT reasoning represents a distinct generalization challenge: the model must produce not just a correct answer but a coherent, logically valid chain of intermediate steps, and the relationship between the correctness of those steps and the correctness of the final answer is itself a subject of investigation (the faithfulness question).
Relative to theoretical work: The paper provides a formal generalization bound (Theorem 3.1) that explicitly links test risk to the total variation distance between training and test distributions. While this bound is standard in statistical learning theory (it derives from Hoeffding's inequality plus a distribution-shift decomposition), its application to CoT reasoning is novel. The bound provides a rigorous justification for the intuition that performance degrades with distribution shift, and the additive structure (task + length + format discrepancies compose through a monotonic function $\Phi`) provides a template for designing multi-factor evaluations.
Why This Matters Beyond Academic Debate
The paper implicitly argues that the question of whether CoT reasoning is genuine or illusory has concrete, high-stakes implications:
Trust and deployment: If CoT reasoning is a mirage — fluent but unfaithful text that correlates with correct answers only under narrow distributional conditions — then deploying CoT-based systems in medicine, law, finance, or scientific research is dangerous. Users may be persuaded by plausible-sounding reasoning steps and fail to catch errors that would be obvious if the reasoning were presented without the veneer of step-by-step logic. The paper's ethical considerations section explicitly flags this risk: "the presence of a detailed reasoning trace should not be equated with correctness, reliability, or genuine understanding."
Research direction: If CoT reasoning is fundamentally pattern interpolation, then efforts to improve it through better prompting, search, or decoding are optimizing a surface-level capability rather than addressing the core limitation. The real bottleneck is distributional coverage — ensuring that the training data contains examples that are sufficiently similar to test queries — which is a data curation and augmentation problem, not a prompting problem. This would redirect research investment away from sophisticated reasoning architectures and toward understanding and controlling training distributions.
Evaluation standards: If data leakage and benchmark contamination invalidate standard CoT evaluations, then the entire literature on CoT effectiveness needs to be re-examined. The paper's framework offers an alternative evaluation methodology (synthetic, controlled, from-scratch training) that can complement — though not replace — real-world benchmark evaluation. The finding that SOTA models exhibit the same fragility as toy models (Section 8.2, Figure 9) suggests that the controlled-environment results are not artifacts of scale but reflect something fundamental about how current LLMs process CoT prompts.
The Roadmap from Here
With this motivation established, the paper proceeds in a clear logical sequence:
-
Section 3: Formalizes the data distribution lens, defining distribution discrepancy and proving the generalization bound that links it to test-time performance.
-
Section 4: Introduces DataAlchemy, the controlled environment that operationalizes the lens by abstracting NLP tasks into atoms, elements, and transformations — making it possible to systematically vary task, length, and format while holding everything else constant.
-
Sections 5–7: Conducts the controlled experiments across task generalization (Section 5), length generalization (Section 6), and format generalization (Section 7), each time demonstrating that CoT performance degrades systematically with increasing distribution discrepancy.
-
Section 8: Establishes generality — showing that the same patterns hold across model architectures (GPT, LLaMA), model sizes (62K to 3B parameters), and even state-of-the-art pretrained models (LLaMA3-8B, Qwen3-14B-Instruct) when fine-tuned on DataAlchemy data.
-
Section 9: Discusses implications for practitioners and researchers, emphasizing the need for OOD testing, skepticism toward reasoning traces, and recognition that fine-tuning patches distribution gaps rather than teaching genuine reasoning.
The paper is, at its core, an argument for a paradigm shift in how we evaluate LLM reasoning: from chasing benchmark numbers on contaminated test sets to conducting principled, controlled experiments that reveal the underlying mechanisms — and limitations — of what LLMs do when they produce step-by-step reasoning chains.
3. Technical Approach
3.1 Reader Orientation
The "system" in this paper is not a deployed application but rather a controlled experimental framework — called DataAlchemy — built to test whether Chain-of-Thought reasoning in LLMs reflects genuine logical inference or merely interpolates patterns memorized from training data. It solves the problem that real-world NLP benchmarks cannot isolate why CoT succeeds or fails because task difficulty, linguistic complexity, reasoning length, and data leakage are all entangled; the solution is an abstract symbolic environment where every factor (task structure, element composition, sequence length, and query format) can be independently varied, and where models are trained from scratch on fully known data so that any generalization behavior can be attributed to distributional similarity rather than hidden pretraining exposure.
3.2 Big-Picture Architecture (Diagram in Words)
The DataAlchemy framework has five major components:
-
Atom-Text Abstraction Layer — distills all tokens into a 26-letter alphabet
{A, B, ..., Z}called atoms, and all text sequences into ordered sequences of atoms called elements, enabling fine-grained control over vocabulary and surface form while preserving the combinatorial structure of NLP tasks. -
Transformation Engine — abstracts linguistic operations (summarization, paraphrasing, reasoning) into mathematical functions that map elements to elements, including the ROT transformation (cyclic shift of each atom's alphabetical index), the Cyclic Position Shift (permuting atoms within the sequence), and arbitrary compositions thereof, which become the "tasks" that models must learn.
-
Distribution Specification Layer — systematically varies three axes of distribution shift: task (which transformations and elements appear, and how they are composed), length (the number of atoms per element and the number of compositional reasoning steps), and format (insertions, deletions, and substitutions of tokens in the prompt template).
-
Training-from-Scratch Pipeline — trains decoder-only transformer models (GPT and LLaMA architectures, sizes from 62K to 3B parameters) on fully known, fully controlled datasets, eliminating any possibility of benchmark leakage or contamination.
-
Evaluation Harness — measures performance using exact match rate (the fraction of outputs identical to the ground truth), edit distance (Levenshtein distance between predicted and correct sequences), and BLEU score, applied separately to the reasoning trace, the final answer, and the full output chain to enable fine-grained diagnosis of where failures occur.
Information flows as follows: the distribution specification layer generates training and test datasets by composing elements and transformations with controlled discrepancies along the three axes → a decoder-only LLM is trained from scratch on the training split → the trained model is evaluated on test splits that systematically vary the degree of distribution shift → the evaluation harness computes per-axis performance metrics → the resulting degradation curves are analyzed to determine whether CoT reasoning is robust or brittle under each type of shift.
3.3 Roadmap for the Deep Dive
-
First, the theoretical foundation: the formal generalization bound (Theorem 3.1) that explicitly links expected test error to the total variation distance between training and test distributions, establishing why we should expect distribution-dependent degradation before any experiments are run.
-
Second, the atom-element-transformation abstraction: how tokens, texts, and operations are distilled into a fully controllable symbolic system, including the exact mathematical definitions of the ROT transformation, Cyclic Position Shift, and compositional transformations that form the reasoning tasks.
-
Third, the three-dimensional distribution discrepancy measurement framework: how task discrepancy, length discrepancy, and format discrepancy are quantified in ways that plug directly into the generalization bound, including the task complexity score
$T(C)$, the squared-exponential length extrapolation decay, and the format alignment score$S(p_{\text{test}})$. -
Fourth, the experimental design for each generalization dimension: the specific training/test splits used for task generalization (ID, CMP, POOD, OOD), length generalization (text length and reasoning step variants), and format generalization (four perturbation modes at controlled noise levels).
-
Fifth, the training and evaluation infrastructure: model architectures, hyperparameters, metrics, and the rationale for why training from scratch is essential to avoid data leakage.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a controlled experimental methodology paper whose core idea is that by abstracting NLP tasks into symbolic primitives, synthesizing training data with known distributions, training models from scratch, and systematically varying the discrepancy between training and test conditions, we can isolate the factors that govern CoT reasoning success and failure — and thereby determine whether apparent reasoning is genuine inference or pattern interpolation.
The Formal Generalization Bound
Before introducing the synthetic environment, the paper establishes a theoretical foundation that makes precise the intuition that test performance degrades with distribution shift. The bound is standard in structure — it decomposes generalization error into sampling error plus distribution mismatch — but its application to CoT reasoning is novel and provides the organizing principle for the entire experimental program.
Defining the quantities. Let $D_{\text{train}}$ denote the training distribution over input-output pairs $(x, y)$, where $x$ is a reasoning problem (e.g., "apply ROT-13 to APPL") and $y$ is the solution sequence including intermediate reasoning traces (e.g., "NCCY [F2] CCYN"). Let $D_{\text{test}}$ be the potentially different test distribution. The model learns a parametric mapping $f_\theta(x) \approx y$ by minimizing the empirical training risk:
where $(x_i, y_i) \sim D_{\text{train}}$ are i.i.d. training samples, $\ell$ is a loss function (cross-entropy in practice), and $n$ is the number of training examples.
The corresponding population (expected) training risk is:
At test time, given queries from $D_{\text{test}}$, the expected test risk is:
Distribution discrepancy. The paper defines distribution discrepancy as the total variation distance:
where total variation is defined operationally:
In plain language: $\Delta$ is the largest possible difference in probability mass that $D_{\text{train}}$ and $D_{\text{test}}$ assign to any measurable subset of the input-output space. When the distributions are identical, $\Delta = 0$; when they have disjoint support, $\Delta = 1$. The total variation provides an upper bound on how much any bounded function's expectation can differ under two distributions, which is precisely what is needed to bound the difference between training and test risk.
The generalization bound. The paper states and proves Theorem 3.1:
where $B$ is the maximum possible loss value (under the assumption that $0 \leq \ell(f_\theta(x), y) \leq B$), $n$ is the number of training samples, and $\delta \in (0,1)$ is the confidence parameter — this bound holds with probability at least $1 - \delta$ over the draw of the training set.
What this bound computes: given a trained model $f_\theta$, the bound predicts an upper limit on the expected test loss based on three terms. The first term, $\hat{R}_{\text{train}}(f_\theta)$, is the model's measured error on the training data — its in-distribution performance. The second term, $2B \cdot \Delta(D_{\text{train}}, D_{\text{test}})$, is the penalty for distribution shift — it scales linearly with the total variation distance, meaning that if the test distribution is very different from training, this term can dominate the bound. The third term, $B\sqrt{\log(1/\delta)/(2n)}$, is the standard Hoeffding bound on sampling error — it decays as $1/\sqrt{n}$, so with more training data, our estimate of training risk becomes more reliable, and the bound tightens.
Why this form: the bound decomposes generalization error into two fundamentally different sources. Sampling error (the third term) reflects that we only have finite training data — even if train and test distributions were identical, we would still have some error because we estimate the training risk from a finite sample. Distribution shift error (the second term) reflects that the test distribution may differ from training — even with infinite training data, if the test queries look different from what the model was trained on, performance will degrade. By separating these terms, the bound makes clear that improving training-set performance (reducing $\hat{R}_{\text{train}}$) does not help with distribution-shift error, and that the only way to guarantee good test performance is to either (a) ensure $\Delta$ is small (keep test queries close to training) or (b) make the bound vacuous by using enormous training sets (which reduces the sampling term but cannot reduce the distribution-shift term). The additive structure also justifies studying $\Delta$ as the key theoretical quantity governing test-time behavior.
The three-axis decomposition. The paper then decomposes the total distribution discrepancy into three additive components:
where $\Phi$ is a monotonically increasing composition function that aggregates all discrepancies. $\Delta_{\text{task}}$ measures distribution shift introduced by unseen task structures (novel transformation compositions or novel element types), $\Delta_{\text{length}}$ measures shift from differing text lengths or reasoning step counts, and $\Delta_{\text{format}}$ measures shift from surface-level prompt perturbations.
What this decomposition means operationally: rather than treating distribution shift as a monolithic quantity, the paper identifies three axes along which shift actually occurs in NLP practice. If you ask an LLM to solve a math problem with more steps than it was trained on, that is a length shift. If you ask it to apply a transformation it never saw composed in a particular way, that is a task shift. If you introduce typos or rephrase the prompt, that is a format shift. Each axis introduces $\Delta$ through a different mechanism, and the bound tells us that the test risk degrades in proportion to the aggregate discrepancy. The experiments in Sections 5–7 systematically vary one axis at a time while holding the others fixed, directly testing whether performance follows the degradation pattern predicted by the bound.
The proof of Theorem 3.1 is provided in Appendix C.4 and is structurally straightforward: it decomposes $R_{\text{test}} - \hat{R}_{\text{train}}$ into $(R_{\text{test}} - R_{\text{train}}) + (R_{\text{train}} - \hat{R}_{\text{train}})$, bounds the first term using the variational characterization of total variation distance (for any bounded function, $|\mathbb{E}_{D_{\text{train}}}[g] - \mathbb{E}_{D_{\text{test}}}[g]| \leq 2B \cdot \text{TV}$), and bounds the second term using Hoeffding's inequality. The combination yields the stated bound with probability at least $1 - \delta$.
The Task Discrepancy Measure and Failure Threshold
Building on the general bound, the paper introduces a specific quantitative measure for task-level distribution shift. This measure is needed because "task" is not a scalar quantity — a test query can differ from training along multiple novelty axes simultaneously (novel atoms, novel individual transformations, novel transformation compositions), and we need a way to aggregate these into a single discrepancy that predicts performance.
Task complexity score. Given a task configuration $C$ consisting of an element $e = (a_0, \ldots, a_{l-1})$ of length $l$ and a compositional transformation $f_S = (f_1, \ldots, f_k)$ of depth $k$, the task complexity score is defined in Proposition C.1 as:
where $\alpha, \beta, \gamma > 0$ are weights for the three novelty axes, $\mathcal{E}^i_{\text{train}}$ is the set of atoms observed at position $i$ during training, $\mathcal{F}_{\text{train}} \subseteq \mathcal{F}$ is the set of fundamental transformations seen during training (a subset of the universe $\mathcal{F}$ from Section 4.2), and $\mathcal{P}_{\text{train}}$ is the set of full transformation compositions seen during training. The indicator $\mathbb{1}[\cdot]$ equals 1 when its argument is true (novelty is present) and 0 otherwise.
What this score computes: for any test query, we count three types of novelties. First, for each position in the input element, we check whether that specific atom appeared at that specific position during training — if not, we add $\alpha$ to the score. Second, for each transformation in the compositional chain, we check whether that transformation (as an individual operation) was seen during training — if not, we add $\beta$. Third, we check whether the entire compositional sequence was seen as a unit during training — if not, we add $\gamma$. The total score $T(C)$ is zero only when every atom at every position, every individual transformation, and the full composition were all present in the training data — that is, when the test query is perfectly in-distribution.
Why this form: the three-term additive structure captures qualitatively different types of novelty. Position-wise atom novelty (the first term) measures whether the model has seen the vocabulary items in their specific structural positions — a test element containing entirely new letters (e.g., N–Z when training used A–M) gets a high score. Individual transformation novelty (the second term) measures whether the model has seen the basic operations — a test query using a transformation that never appeared in training gets a high score. Compositional novelty (the third term) measures whether the model has seen the full multi-step chain — even if both $f_1$ and $f_2$ were seen individually and in other compositions, a new composition $f_1 \circ f_2$ (when that specific ordering was not in training) gets penalized by $\gamma$. This decomposition is crucial because the experiments reveal that these three types of novelty behave differently: compositional novelty (CMP setting) causes partial degradation, while atom-level novelty (OOD setting) causes complete collapse.
The weights $\alpha, \beta, \gamma$ are free parameters that control the relative importance of each novelty type. The paper treats them as fixed but does not specify particular numerical values — the qualitative predictions (monotonic degradation with increasing novelty) hold regardless of the specific weights.
The task-axis discrepancy. The task complexity score is mapped to a discrepancy through a monotone function:
where $\psi_{\text{task}}: [0, \infty) \to [0, 1]$ is monotonically non-decreasing and satisfies $\psi_{\text{task}}(0) = 0$ (zero novelty implies zero task-axis discrepancy). The simplest choice is the identity after normalization, but any monotone transformation preserves the qualitative prediction that larger $T(C)$ implies larger $\Delta_{\text{task}}$, which in turn implies a looser bound on test risk via Theorem 3.1.
The failure threshold theorem. The paper goes further and proves a specific functional form for how performance degrades with task complexity. Theorem C.1 states that there exist positive constants $\kappa$ and $\tau$ such that whenever $T(C) > \tau$, the probability of correct CoT reasoning is upper-bounded by:
where $\pi(C) := \Pr[\text{CoT}_\theta(C) = y^\star(C)]$ is the success probability on configuration $C$, and $y^\star(C)$ is the ground-truth output chain.
What this theorem predicts: there is a threshold $\tau$ below which the model can successfully reason (in-distribution or very near-distribution). Above this threshold, success probability decays exponentially in the task complexity score. This is a strong prediction: if task complexity increases linearly (e.g., adding one more novel atom or one more novel transformation), the probability of correct reasoning should drop by a multiplicative factor. A model that scores 100% at $T(C) = 0$ should score near 0% at modestly higher $T(C)$ if $\kappa$ is large.
Why this form (the multiplicative failure model): the proof (provided in Appendix C.1) assumes that failures induced by novel atoms, novel individual transformations, and novel compositions are independent in their multiplicative effect on success probability. Specifically, it models:
where $\pi_0 \in (0, 1]$ is the baseline in-distribution success probability, and $\rho_a, \rho_f, \rho_c \in (0, 1)$ are per-axis degradation factors — the factor by which success probability is multiplied when the corresponding novelty is present. Taking logs yields a linear relationship between $\ln \pi(C)$ and $T(C)$ (with the degradation factors $\xi_a = -\ln \rho_a$, etc., playing the role of the coefficients), and exponentiating yields the exponential upper bound. The assumption of multiplicative independence is a modeling choice — it says that if seeing a novel atom reduces success probability by a factor of $\rho_a$, and seeing a novel transformation reduces it by $\rho_f$, then seeing both reduces it by $\rho_a \cdot \rho_f$. This is a simplification (real failure modes may interact non-multiplicatively), but it captures the qualitative pattern of compounding degradation and makes the threshold behavior mathematically precise.
The Length Discrepancy Measure
Length generalization is conceptually different from task generalization: rather than introducing novel entities or operations, it varies the scale of the input (text length, measured in atoms) or the depth of the reasoning chain (number of compositional transformations). The paper proposes a squared-exponential model for how performance degrades with length extrapolation.
The squared-exponential decay model (Proposition C.2). For a model trained on chain-of-thought sequences of fixed length $L_{\text{train}}$, the generalization error at test length $L$ is modeled as:
where $\varepsilon_0 \in [0, 1]$ is the in-distribution error at $L = L_{\text{train}}$, and $\sigma > 0$ is a length-generalization width parameter that controls how quickly performance degrades as the length gap grows.
What this model predicts: at $L = L_{\text{train}}$, the exponential term is $\exp(0) = 1$, so $\varepsilon(L_{\text{train}}) = \varepsilon_0 + (1 - \varepsilon_0)(1 - 1) = \varepsilon_0$ — exactly the in-distribution error. As the length gap $|L - L_{\text{train}}|$ grows, the exponential term shrinks toward zero, and $\varepsilon(L)$ approaches $\varepsilon_0 + (1 - \varepsilon_0)(1 - 0) = 1$, meaning the error saturates at 1 (complete failure). The squared-exponential form means that small length gaps cause relatively small degradation (the curve is flat near zero), but once $|L - L_{\text{train}}|$ exceeds roughly $2\sigma$, performance collapses rapidly.
Why this form (the heuristic derivation): the paper provides a heuristic argument based on hidden-state distributions in transformers. For a model trained on sequences of length $L_{\text{train}}$, test sequences of different length induce hidden-state distributions that differ from training in two respects. First, positional encodings for positions $i > L_{\text{train}}$ have never been observed when $L > L_{\text{train}}$ (the model has no representation for "position 6" if it only saw sequences of length 5). Second, the learned attention patterns are calibrated for $L_{\text{train}}$ — the model has learned to distribute attention across a specific number of tokens, and when the sequence length changes, that distribution is disrupted.
Under the mild assumption that per-position deviations accumulate independently with bounded variance, a second-order Taylor expansion of the log-likelihood of the test hidden-state distribution around $L = L_{\text{train}}$ yields a KL divergence that grows quadratically: $D_{\text{KL}}(q_{\text{test}} \| q_{\text{train}}) = \Theta((L - L_{\text{train}})^2)$. The linear term vanishes because the model is trained to match the distribution at $L = L_{\text{train}}$ (the likelihood is stationary there — if it were not, training would not have converged).
The quadratic growth of KL divergence motivates a squared-exponential transition from $\varepsilon_0$ to 1 — the squared-exponential kernel $\exp(-(L - L_{\text{train}})^2 / (2\sigma^2))$ directly reflects the quadratic penalty in the exponent. The width parameter $\sigma$ is treated as an empirical fit parameter (it depends on the model architecture, training data, and the specific length range), but the functional form is theoretically motivated.
Why squared-exponential rather than linear or exponential: if degradation were linear in $|L - L_{\text{train}}|$, then a sequence of length 10 would be equally harder than length 5 as length 6 is harder than length 1 — but in practice, small extrapolations are often tolerated (length 6 when trained on length 5) while large ones are catastrophic (length 20 when trained on length 5). The squared-exponential captures this threshold behavior: the probability of success is near 1 for small gaps and collapses for large gaps, with a smooth transition between regimes. Alternative forms like a simple exponential $\exp(-|L - L_{\text{train}}|/\sigma)$ would predict linear degradation on a log scale, which is less consistent with the sharp drop-offs observed empirically.
The length-axis discrepancy is then obtained from the excess error:
where $\psi_{\text{length}}: [0, 1 - \varepsilon_0] \to [0, 1]$ is monotonically non-decreasing with $\psi_{\text{length}}(0) = 0$. The identity choice recovers the raw excess error.
What this means for the experiments: when the paper varies element length (Section 6.1) or reasoning step count (Section 6.2), the squared-exponential model predicts that performance should be near-perfect at the training length, degrade smoothly for small deviations, and collapse for large deviations. The empirical curves in Figure 5 and Figure 6 are evaluated against this prediction.
The Format Discrepancy Measure
Format generalization captures a different type of shift: the prompt template changes (tokens are inserted, deleted, or modified), but the underlying task (the element and transformation) remains identical. This dimension is critical for assessing whether the model has learned a transferable reasoning strategy or has simply memorized the specific string pattern of the training prompts.
Format alignment score (Definition C.1). For a training prompt distribution $\Pi_{\text{train}}$ and a test prompt $p_{\text{test}}$, the paper defines:
where $\eta(\cdot)$ is a prompt embedding function (mapping a prompt string to a vector in some embedding space) and $\cos(\cdot, \cdot)$ is the cosine similarity between two vectors.
What this score computes: for a given test prompt, we find the training prompt that is most similar to it in embedding space (the $\max$ over $p \in \Pi_{\text{train}}$), and measure the cosine similarity between their embeddings. If the test prompt is identical to some training prompt, $S(p_{\text{test}}) = 1$ (perfect alignment). If the test prompt is completely different from all training prompts (e.g., contains novel tokens, different structure), $S(p_{\text{test}})$ will be close to 0.
Why cosine similarity: cosine similarity is invariant to the magnitude of the embedding vectors (it measures directional alignment), which is appropriate for comparing prompts that may differ in length (a longer prompt with the same "meaning" should still have high similarity). Alternative metrics like Euclidean distance would penalize longer prompts even if they were semantically similar. Cosine similarity is also bounded in $[-1, 1]$, making the alignment gap $1 - S(p_{\text{test}})$ naturally bounded in $[0, 2]$.
The format-axis discrepancy is then the alignment gap mapped through a monotone function:
where $\psi_{\text{format}}: [0, 1] \to [0, 1]$ is monotonically non-decreasing with $\psi_{\text{format}}(0) = 0$. The identity choice recovers $1 - S(p_{\text{test}})$.
How the four perturbation modes affect the alignment score. The paper considers four distinct ways to perturb the prompt template, each affecting $S(p_{\text{test}})$ through a different mechanism:
-
Insert: a novel
<noise>token is inserted at a random position. This introduces an out-of-vocabulary embedding that shifts the overall prompt vector away from all training prompts. The degree of shift depends on how the embedding function$\eta$handles unseen tokens — if it maps them to a default or averaged embedding, the shift is bounded; if it produces a distinctive embedding, the shift is larger. -
Delete: a token is removed from the prompt, shortening it and redistributing the attention pattern within
$\eta$. This changes both the length and the token composition, causing a shift whose magnitude depends on how important the deleted token was to the prompt's overall embedding. -
Modify: a token is replaced by a novel
<noise>token. This is similar to insert in introducing an OOV embedding, but without changing the prompt length (so positional encodings are preserved). -
Hybrid: all three perturbations are applied simultaneously, compounding their individual effects. The alignment score under Hybrid is expected to be lower than under any single perturbation mode at the same noise level.
In all cases, the alignment score is expected to be monotonically non-increasing in the noise level $p$ (the fraction of tokens perturbed). Larger $p$ means more tokens are inserted/deleted/modified, which means the perturbed prompt is further from all training prompts in embedding space, which means $S(p_{\text{test}})$ is lower, which means $\Delta_{\text{format}}$ is higher, which means (via Theorem 3.1) the upper bound on test risk is looser — consistent with the degradation curves observed in Figure 7.
Operationalizing the score: the paper does not actually compute $S(p_{\text{test}})$ or embed prompts; the metric is proposed as a conceptual tool for thinking about format shift. In practice, the experiments operationalize format shift through the noise level $p$, which is controlled directly in the dataset generation. The qualitative prediction — that performance degrades monotonically with $p$ — is tested without requiring explicit computation of $S(p_{\text{test}})$.
The Atom-Element-Transformation Abstraction
The foundation of DataAlchemy is a symbolic abstraction that strips NLP tasks down to their combinatorial skeleton while preserving the structural properties that make reasoning challenging. This section details the exact mathematical definitions and design choices.
Atoms: the token space. The paper abstracts all tokens into a 26-letter alphabet:
This choice is deliberate. Using exactly 26 letters provides a token space that is (a) large enough to support combinatorial diversity (with element length $l$, there are $26^l$ possible elements) but (b) small enough that the model can be trained to full mastery on a subset without requiring billions of parameters. The alphabetical ordering also provides a natural structure for the ROT transformation (defined below), since each atom has a well-defined zero-based index through the bijective mapping $\phi: A \to \mathbb{Z}_{26}$, where $\phi(c)$ maps a character to its alphabetical position (A = 0, B = 1, ..., Z = 25).
Why 26 letters rather than arbitrary symbols: the alphabetical structure enables the ROT transformation (cyclic shift modulo 26) to be well-defined and mathematically clean. If the atom space were arbitrary symbols without an ordering, the ROT transformation would not have a natural definition. The choice of exactly 26 also avoids the need for padding or special tokens — every possible integer $n$ can be reduced modulo 26, and the mapping $\phi^{-1}$ (from index back to letter) is always well-defined. Using a smaller alphabet (e.g., 10 symbols) would reduce combinatorial diversity; using a much larger alphabet (e.g., 100 symbols) would increase the atom space beyond what small models can fully learn, making it harder to achieve perfect in-distribution performance.
Elements: the text space. An element $e$ is defined as an ordered sequence of atoms with a specified length $l$:
This corresponds to a "text" in the real world — a sequence of tokens that the model processes as input. The element length $l$ is a controlled variable that can range from 2 to 6 in the experiments, yielding $26^2 = 676$ possible elements at $l=2$ to $26^6 = 308,915,776$ at $l=6$. This range is important: at $l=2$ the element space is small enough that a model can see every possible element during training (memorization is possible), while at $l=6$ the space is vastly larger than any plausible training set (generalization is necessary).
The element construction is purely combinatorial — there is no "meaning" to any particular sequence of letters beyond its identity as a sequence. This is not a bug but a feature: it removes the semantic layer that entangles real-world NLP tasks, making it possible to isolate syntactic/structural generalization from semantic generalization.
Transformations: the operation space. The paper abstracts NLP operations (summarize, paraphrase, translate, reason) into mathematical functions that map elements to elements. Two fundamental transformations are defined, with a third introduced in the Appendix for additional experiments:
Definition 4.1 (ROT Transformation). Given an element $e = (a_0, \ldots, a_{l-1})$ and a rotation parameter $n \in \mathbb{Z}$, the ROT transformation $f_{\text{rot}}$ produces an element $\hat{e} = (\hat{a}_0, \ldots, \hat{a}_{l-1})$ where each atom is:
What this does: for each atom in the input element, we look up its alphabetical index (A=0, B=1, ..., Z=25), add $n$, take the result modulo 26, and map back to a letter. For example, with $n = 13$ (ROT-13), the element (A, P, P, L, E) becomes (N, C, C, Y, R) because A(0)+13→N(13), P(15)+13→28 mod 26 = 2→C, L(11)+13→24→Y, E(4)+13→17→R. This is a position-wise operation: the transformation at position $i$ depends only on $a_i$, not on any other position.
Why ROT-13 specifically: ROT-13 is its own inverse (applying it twice returns the original), which makes certain compositional identities simple (e.g., $f_1 \circ f_1 = \text{identity}$). The paper uses $n = 13$ as the primary ROT parameter, with other shifts possible but not explored in detail. The choice of a single fixed shift parameter ($n = 13$) rather than a variable $n$ simplifies the transformation space: the model only needs to learn one ROT function rather than a family parameterized by $n$.
Definition 4.2 (Cyclic Position Shift). Given an element $e = (a_0, \ldots, a_{l-1})$ and a shift parameter $n \in \mathbb{Z}$, the Cyclic Position Shift $f_{\text{pos}}$ produces an element $\hat{e} = (\hat{a}_0, \ldots, \hat{a}_{l-1})$ where:
What this does: each atom in the output is taken from a position $n$ steps earlier in the input (wrapping around modulo $l$). For example, with $n = 1$, (A, P, P, L, E) becomes (P, P, L, E, A) — every atom shifts one position to the left, and the first atom wraps to the end. With $n = l - 1$, this is a right shift. This operation is structurally different from ROT: it depends on the positions of atoms but not on their identities. The output atoms are a permutation of the input atoms.
Why one position shift ($n=1$): the paper fixes $n=1$ for simplicity, creating a single well-defined shift operation. This ensures that the transformation space is small and learnable. A family of shifts parameterized by $n$ would increase the task complexity and create an additional axis of generalization (generalizing to unseen shift amounts), which is not the focus of this paper. The choice of $n=1$ also makes the operation non-trivial but easy to verify: the output is always a single cyclic permutation of the input.
Why these two transformations: ROT and Cyclic Shift are chosen because they are structurally distinct — ROT is content-dependent (the output depends on which letters are present), while Cyclic Shift is position-dependent (the output depends on where letters are, not what they are). They are also non-commutative in general: applying ROT then Shift does not produce the same result as applying Shift then ROT (except for degenerate inputs). This non-commutativity is crucial for the task generalization experiments, because it creates a meaningful distinction between compositions $f_1 \circ f_2$ and $f_2 \circ f_1$ — if the transformations commuted, these would be identical tasks, and the compositional generalization experiments would be trivial. The non-commutativity also creates the interesting failure patterns observed in Table 2, where models can sometimes produce correct answers through commutativity-induced coincidences even with incorrect reasoning.
Definition 4.3 (Generalized Compositional Transformation). To model multi-step reasoning, multiple fundamental transformations are composed sequentially. Let $S = (f_1, f_2, \ldots, f_k)$ be a sequence of operations, where each $f_i$ is one of the fundamental transformations $\mathcal{F} = \{f_{\text{rot}}, f_{\text{pos}}\}$ with its respective parameters. The compositional transformation is:
The resulting element is obtained by applying operations sequentially:
Note on composition order: the paper uses $f_1 \circ f_2$ to mean apply $f_1$ first, then $f_2$. This is standard function composition where $(g \circ h)(x) = g(h(x))$, but the paper's notation in Equation 11 ($f_S = f_1 \circ f_2 \circ \cdots \circ f_k$) and Equation 12 ($\hat{e} = f_k(f_{k-1}(\cdots(f_1(e))\cdots))$) are consistent: $f_1$ is applied first to the raw element, then $f_2$ to the result, and so on. This ordering means that the rightmost function in the chain operates on the original element, which is the standard convention.
What this enables: the compositional transformation formulation is the key to modeling "reasoning" in DataAlchemy. A multi-step reasoning problem in the real world (e.g., "first summarize the text, then translate it, then extract key entities") becomes a compositional transformation in DataAlchemy. The intermediate results after each step are the "reasoning trace":
The sub-chain $e^{(1)}, e^{(2)}, \ldots, e^{(k-1)}$ corresponds to the CoT reasoning steps that the model should produce. In the training data, these intermediate states are included explicitly, so the model learns to generate not just the final answer but the step-by-step trace. At test time, the model is expected to produce the full chain, and the evaluation separately scores the reasoning trace, the final answer, and the full chain.
The third transformation (Appendix B.1.2). For additional experiments probing non-commutative generalization (beyond what can be explained by commutativity coincidences), the paper introduces $f_3$, the sequence reversal operator:
This is a position-dependent but content-independent transformation (like Cyclic Shift) but is not a simple permutation of the shift family — it reverses order rather than cyclically shifting. Including $f_3$ creates a transformation space $\{f_1, f_2, f_3\}$ where none of the transformations commute, eliminating the "commutativity-induced correctness" artifact observed in Table 2. The results with $f_3$ (Table 3 in Appendix D.1.2) confirm that when commutativity cannot produce correct answers through incorrect reasoning, CoT reasoning fails completely — exact match drops to 0% across all metrics for non-commutative transfers like $f_2 \circ f_3 \to f_3 \circ f_2$.
The Three-Axis Experimental Design
With the abstraction in place, the paper designs three sets of controlled experiments, each systematically varying one axis of distribution shift while holding the others fixed.
Task generalization (Section 5). Task is decomposed into two sub-axes: transformation generalization and element generalization.
For transformation generalization, the paper defines four progressive scenarios:
-
ID (In-Distribution): test transformations are identical to training transformations. Example: train on
$f_1 \circ f_1 \to f_1 \circ f_1$(where$\to$means "the model sees input-output pairs for this composition"), test on$f_1 \circ f_1 \to f_1 \circ f_1$. All transformations at test time were seen during training in exactly the same form. -
CMP (Compositional): test transformations are novel compositions of transformations that were all seen individually during training, but the specific combination is new. Example: train on
$\{f_1 \circ f_1, f_1 \circ f_2, f_2 \circ f_1\} \to f_2 \circ f_2$, meaning the model has seen$f_1 \circ f_1$,$f_1 \circ f_2$, and$f_2 \circ f_1$as input transformations, and$f_2 \circ f_2$appears as an output transformation (but never as an input). At test time, the model must handle$f_2 \circ f_2$as an input — it has the primitives ($f_1$,$f_2$) and has seen each in other compositions, but never this specific two-step chain. -
POOD (Partially Out-of-Distribution): test queries include compositions where some individual transformations are novel but others are seen. Example: train on
$f_1 \circ f_1 \to f_1 \circ f_2$, meaning the model has seen$f_1 \circ f_1$as input and$f_1 \circ f_2$as output. At test, the model sees$f_1 \circ f_2$as input — the composition$f_1 \circ f_2$itself was not seen during training as an input, but both$f_1$and$f_2$were (in other contexts). -
OOD (Out-of-Distribution): test queries contain entirely novel transformations — neither the individual operations nor their compositions were seen during training. Example: train on
$f_1 \circ f_1 \to f_2 \circ f_2$, test on$f_2 \circ f_2 \to f_1 \circ f_1$. None of the test transformations have been observed in any form.
Why these four scenarios: they form a gradient of increasing distribution shift. ID is the baseline (no shift, perfect performance expected). CMP tests whether the model can recompose known primitives — can it recognize that $f_2$ followed by $f_2$ means "apply $f_2$ twice" even though it has only seen $f_2$ applied in other contexts? This is the minimal generalization challenge: the atoms (transformations) are known, only the arrangement is new. POOD introduces partial novelty — the model must handle compositions that include an unseen element. OOD tests for complete novelty — can the model infer what $f_2$ does from never having seen it, purely from the training on $f_1$? The progressive design allows the paper to identify exactly where CoT reasoning breaks down: if it fails at CMP, then the model cannot even recombine known primitives; if it fails only at POOD/OOD, then the model can recombine but cannot infer new operations.
For element generalization (detailed in Appendix D.2), three analogous scenarios are defined:
- ID: test elements consist of exactly the same atoms in the same order as training.
- CMP (Compositional): test elements use the same atoms as training but in different arrangements (e.g., train on
(A, P, P, L), test on(P, A, L, P)). All atoms are familiar, but the element is new. - OOD: test elements contain atoms never seen during training (e.g., train on A–M elements, test on N–Z elements).
Within CMP, the paper further distinguishes by edit distance between training and test elements: $n$ measures how many atom positions differ. An element with $n=1$ (one position changed) is "closer" to the training distribution than one with $n=4$ (completely rearranged). The SFT experiments in Figure 12a test whether models can generalize to elements at various edit distances from training.
Length generalization (Section 6). Length is decomposed into text length (element length $l$) and reasoning step count (compositional depth $k$).
For text length generalization, the model is trained on elements of a fixed length $l = 4$ (the "training length") and evaluated on lengths ranging from $l = 2$ to $l = 6$. This creates a symmetric extrapolation test: can the model handle both shorter elements (requiring fewer token generations than it is used to) and longer elements (requiring more)? Three padding strategies are tested as a potential mitigation:
- None: no padding — training sequences are left at their natural length.
- Padding: training sequences are padded to the maximum context window length with a special token.
- Group: training sequences are grouped and truncated into segments of a fixed maximum length, exposing the model to a distribution of effective sequence lengths.
The rationale for testing padding strategies is that if length generalization failure is caused by the model never experiencing variable-length inputs during training (because all training examples have $l = 4$), then exposing it to multiple lengths during training (through grouping) should improve generalization. The results (Figure 13) confirm this: the Group strategy significantly improves length generalization compared to no padding or fixed-length padding, consistent with the hypothesis that length generalization is constrained by the distribution of lengths seen during training, not by an inherent inability to process variable-length sequences.
For reasoning step generalization, the model is trained on a mixture of tasks with different compositional depths $k$ (e.g., a mix of $k=1$ and $k=2$ tasks). The mix ratio between $k=1$ and $k=2$ data is systematically varied from 0.0 (all $k=2$, no $k=1$) to 1.0 (all $k=1$, no $k=2$). The model is then evaluated on $k=1$ and $k=2$ tasks separately, and the performance on each is plotted against the mix ratio (Figure 6). This design directly tests whether a model can handle reasoning depths it has never seen (e.g., $k=3$ when trained only up to $k=2$) and whether the performance on a given depth is proportional to the fraction of training data at that depth.
Format generalization (Section 7). Four perturbation modes are applied to the prompt template at controlled noise levels $p \in \{5\%, 10\%, 15\%, 20\%, 25\%, 30\%\}$:
- Insert: a
<noise>token is inserted at a random position in the prompt. - Delete: a randomly selected token is removed from the prompt.
- Modify: a randomly selected token is replaced with a
<noise>token. - Hybrid: all three perturbations are applied simultaneously, each to a separate random position.
The noise level $p$ controls the fraction of tokens affected. For a prompt with 10 tokens and $p=10\%$, one token is perturbed; at $p=30\%$, three tokens are perturbed. The perturbations are applied to different "areas" of the prompt: the element tokens (the actual data), the transformation tokens (the [F1], [F2] markers), and the instruction tokens (the thinking and <answer> cues). This distinction allows the paper to test whether the model is more sensitive to perturbations in content-carrying positions versus scaffolding positions.
The format generalization experiments are conducted on the in-distribution task setting (the model has seen the task during training), so any performance degradation is attributable purely to the format shift, not to task novelty. This design isolates format robustness as an independent axis.
Training Pipeline and Hyperparameters
The paper trains decoder-only transformer models from scratch using two architectures (GPT and LLaMA) across a wide range of parameter counts: 62K, 68K, 589K, 631K, 4.8M, 6M, 35M, 60M, 540M, 543M, 617M, 623M, and 3B. The specific hyperparameter configurations are provided in Table 8 (Appendix F.1).
For models trained from scratch, the training configuration is:
- Optimizer: AdamW in mixed precision (FP16)
- Learning rate:
$3 \times 10^{-3}$(default) - Learning rate schedule: cosine decay with 10% warm-up ratio
- Batch size: 1024
- Epochs: 10
- Weight decay: 0.01
- Gradient clipping: 1.0
- Inference temperature:
$1 \times 10^{-5}$(essentially greedy decoding)
Why these choices: the high learning rate ($3 \times 10^{-3}$) is appropriate for training from scratch on relatively simple symbolic data — the model needs to learn the alphabet and transformation rules from zero initial weights, which requires substantial parameter updates. Cosine decay with warmup is a standard choice that provides stable early training (the warmup prevents large initial gradients from destabilizing) and gradual convergence (the decay reduces the learning rate as the model approaches the optimum). The batch size of 1024 is large enough to provide stable gradient estimates for small models but not so large as to be computationally prohibitive. The inference temperature of $10^{-5}$ corresponds to nearly deterministic greedy decoding, which eliminates sampling noise as a confounding factor in the evaluation.
Architecture details: the paper uses standard GPT and LLaMA architectures with varying hidden sizes, intermediate sizes, layer counts, and attention head counts (detailed in Table 8). For example, the smallest GPT model (68K parameters) has hidden size 32, no intermediate layer, 4 transformer layers, and 4 attention heads. The largest GPT model (3B parameters) has hidden size 3072, no intermediate layer, 26 transformer layers, and 32 attention heads. The LLaMA models have similar scaling but include an intermediate (feedforward) dimension, following the LLaMA architecture.
For fine-tuning state-of-the-art models (Section 8.2), the configuration is:
- Per-device batch size: 16
- Gradient accumulation steps: 8 (effective batch size: 128)
- Training steps: 24,000
- Learning rate:
$1 \times 10^{-4}$ - Learning rate schedule: cosine decay with 10% warm-up
- Precision: bfloat16 (bf16) mixed-precision training
Why fine-tuning rather than from-scratch training for SOTA models: pretrained models like LLaMA3-8B and Qwen3-14B-Instruct have already learned general language patterns from massive corpora. Fine-tuning on DataAlchemy data adapts them to the specific symbolic task while preserving their pretrained capabilities. The lower learning rate ($10^{-4}$ vs. $3 \times 10^{-3}$) reflects that we are adapting pretrained weights rather than learning from scratch — smaller updates are needed to avoid catastrophic forgetting. The effective batch size of 128 and 24,000 training steps provide sufficient exposure to the DataAlchemy task distributions without overfitting.
Grammar-check on data validity: before fine-tuning SOTA models on DataAlchemy data, the paper confirms that the pretrained models produce "totally random answers" on the symbolic tasks (Section 8.2), establishing that DataAlchemy data was not present in their pretraining corpora. This is a critical validity check: if the SOTA models already performed well on DataAlchemy tasks without fine-tuning, the external validity claim would be undermined because the models might have been exposed to similar patterns during pretraining. The fact that they produce random outputs confirms that the DataAlchemy distribution is genuinely novel.
Evaluation Metrics and Fine-Grained Analysis
The paper uses three complementary metrics, each applied separately to different components of the model's output:
Exact Match (%). The fraction of test queries for which the model's output string exactly matches the ground-truth string (character-for-character, token-for-token). This is a strict binary metric — even a single character error counts as a failure. Exact match is computed for three components:
- Reasoning: the intermediate reasoning trace (the steps connecting input to output, excluding the final answer).
- Answer: the final answer element (the last token sequence after all transformations).
- Full Chain: the entire output including both reasoning and answer — this requires both components to be simultaneously correct.
Why separate reasoning, answer, and full chain: this decomposition enables the paper to diagnose where failures occur. A model could produce correct reasoning but a wrong answer (indicating a failure at the final transformation step), a wrong reasoning but a correct answer (indicating coincidental correctness through commutativity or degeneracy, as observed in Table 2), or errors in both (complete failure). The relationship between reasoning correctness and answer correctness is itself a key experimental variable — the "faithfulness" question is operationalized as the correlation between reasoning exact match and answer exact match across test queries.
Edit Distance (Levenshtein Distance). The minimum number of single-character insertions, deletions, and substitutions required to transform the predicted output into the ground-truth output. This is normalized by sequence length to produce a value in $[0, 1]$. Edit distance captures partial correctness: a model that gets most of a sequence right but makes a few errors will have a low (but non-zero) edit distance, while a model that produces completely unrelated output will have an edit distance near the sequence length.
Why edit distance rather than only exact match: exact match is overly strict for evaluating partial progress. If a model produces an 8-atom reasoning chain where 7 atoms are correct and 1 is wrong, exact match reports 0%. Edit distance reports a small, non-zero value that reflects the near-correctness of the output. This is particularly important for length generalization experiments, where the model might produce outputs of the wrong length (e.g., 4 atoms when 5 are expected) — edit distance can distinguish between "output is the right length but wrong content" and "output is the wrong length entirely."
BLEU Score. The Bilingual Evaluation Understudy score, originally developed for machine translation evaluation, measures n-gram overlap between the predicted and ground-truth sequences. BLEU ranges from 0 to 1, with 1 indicating perfect overlap at all n-gram orders (typically n = 1 to 4). BLEU captures fluency and local correctness: if the model produces sequences that share many short substrings with the ground truth, BLEU will be high even if the overall sequence is imperfect.
Why BLEU: BLEU provides a complementary signal to edit distance. Edit distance measures character-level operations, while BLEU measures n-gram overlap. A model that produces the right atoms but in a slightly wrong order will have low edit distance (many operations needed to fix the order) but may have moderate BLEU (individual bigrams and trigrams are correct). This distinction helps characterize how the model fails — are errors scattered throughout the sequence (low BLEU, high edit distance) or are they localized to specific positions (high BLEU, moderate edit distance)?
Computational cost. All experiments are conducted using 8 NVIDIA A100 GPUs (80 GB memory each) and 4 NVIDIA H200 GPUs. The paper notes that toy experiments (small models on small datasets) can run on a single A100. The total computational cost is not reported in terms of GPU-hours, but the paper trains "hundreds of LLMs" across various sizes, architectures, and experimental configurations, which represents a substantial computational investment to ensure statistical robustness (results are averaged over three independent runs for controlled experiments, as noted in Appendix F.1).
Statistical reporting. For controlled experiments, results are averaged over three independent runs. For SOTA model fine-tuning experiments, results are reported with mean and standard deviation (Tables 5–7 in Appendix D.7), confirming that the observed effects are stable across runs and not driven by sampling noise. The standard deviations are consistently small relative to the performance gaps between ID and shifted distributions, indicating that the degradation patterns are reliable.
4. Key Insights and Innovations
Innovation 1: The Data Distribution Lens as a Unifying Explanatory Framework
The paper's most fundamental contribution is not a new method or metric but a conceptual reframing of what CoT reasoning is: not an emergent reasoning capability, but a structured inductive bias learned from in-distribution data. This lens provides a single, principled explanation for the otherwise contradictory landscape of CoT results — why does CoT help on MATH but fail on planning? Why do minor prompt perturbations cause catastrophic performance drops? Why do reasoning traces look fluent even when answers are wrong? The answer, in every case, is distribution discrepancy.
What the field had before. Prior work fell into two camps. The CoT success camp (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2023; Yao et al., 2023) documented impressive performance gains on specific benchmarks and framed these as evidence of emergent reasoning. The CoT skepticism camp (Mirzadeh et al., 2025; Stechly et al., 2024; Lanham et al., 2023; Bentham et al., 2024) documented failures—fragility to distractors, unfaithful reasoning traces, inability to generalize to harder problems—and framed these as evidence that CoT reasoning is superficial. Both camps were accumulating point-wise observations without a framework for reconciling them. A result from one study (CoT works on GSM8K with GPT-4) and a result from another (CoT fails on planning with the same model) were treated as separate facts rather than manifestations of a single underlying principle.
What the lens provides. The data distribution lens resolves this fragmentation by identifying a single independent variable — the total variation distance Δ(D_train, D_test) — that governs CoT effectiveness across all tasks, models, and prompting strategies. When Δ is small (the test query resembles training data), CoT succeeds. When Δ is large, CoT fails, and the failure mode (incorrect reasoning, unfaithful traces, sensitivity to noise) reflects how the distribution shifted rather than a different underlying phenomenon. This is a genuinely unifying move: instead of cataloguing that CoT fails on planning but succeeds on algebra, the lens tells us why — the planning task introduced a larger distribution shift from the model's training data than the algebra task did. The seemingly contradictory findings become predictable consequences of a single mechanism.
Why this is fundamental rather than incremental. This is not a refinement of an existing CoT method; it is a paradigm-level reframing of how to think about CoT reasoning. The paper argues — and the experiments demonstrate — that model architecture, scale, and prompting strategy are secondary variables. The primary variable is distributional alignment. This has direct implications for research prioritization: if you want CoT to work better, you should focus on expanding training distribution coverage rather than on more sophisticated search algorithms, better prompt engineering, or larger model architectures. The fact that models from 62K to 3B parameters show qualitatively identical degradation patterns (Figure 8, Figure 15) — and that SOTA 14B-parameter models fine-tuned on DataAlchemy data show the same brittleness (Figure 9, Tables 5–7) — provides strong evidence that distribution dependence is not an artifact of limited scale but a fundamental property of how current transformer architectures learn to produce CoT traces.
Evidence anchoring the claim. The theoretical generalization bound (Theorem 3.1) provides the formal backbone: test risk is bounded above by R̂_train + 2B·Δ + sampling error. This is not merely a restatement of standard statistical learning theory — the critical move is decomposing Δ into three manipulable axes (task, length, format) and then systematically varying each in controlled experiments. The empirical results confirm the bound's predictions with striking consistency: exact match drops from 100% to near-zero as Δ increases across all three axes (Figures 3, 5, 7), and the degradation is monotonic in the discrepancy magnitude. The fact that SFT with even a tiny fraction of out-of-distribution data (λ = 1.5 × 10^{-4}) can "recover" performance on previously unseen transformations (Figure 4) further supports the lens — the model isn't learning to reason; it is expanding its "in-distribution bubble" to include the new data.
Innovation 2: DataAlchemy as a Scientific Instrument for Isolating Causal Factors
DataAlchemy is not merely a benchmark or a dataset — it is a controlled experimental platform that solves the three structural limitations (narrowly defined settings, data entanglement, data leakage) that the paper identifies as preventing rigorous scientific investigation of CoT reasoning. The innovation is in the design philosophy: rather than accepting the complexity of real-world NLP tasks as an inescapable constraint, the paper builds an abstract system where that complexity is stripped away, leaving only the combinatorial/symbolic skeleton that makes reasoning challenging.
What the field had before. Standard CoT evaluation uses real-world benchmarks (MATH, GSM8K, ARC, planning domains) on pretrained LLMs. This approach has three fatal flaws for scientific understanding. First, task difficulty, linguistic complexity, reasoning depth, and domain knowledge are hopelessly entangled — if CoT fails on a math problem, you cannot isolate whether the failure was due to arithmetic complexity, multi-step planning requirements, vocabulary novelty, or some interaction. Second, pretrained models have unknown training data — you never know whether the model is reasoning or retrieving. Third, each benchmark tests a specific task, so findings are domain-bound — a result on MATH tells you nothing about planning, and vice versa.
Previous attempts at controlled evaluation (the paper compares to these in Appendix A.4, Figure 10) were partial: some used synthetic data but still evaluated pretrained models (leaving the leakage problem unsolved), others controlled for one factor but not others. No prior framework satisfied all three criteria the paper identifies as necessary: abstract representation, fully controlled experiment, and training from scratch.
What DataAlchemy enables. By distilling all tokens into 26 letters, all operations into three mathematical transformations (ROT, Cyclic Shift, Reversal), and all tasks into compositional chains of these transformations, DataAlchemy makes every factor independently manipulable. You can vary element length without changing transformation complexity. You can change the prompt format without changing the underlying task. You can introduce novel transformations without changing element composition. You can train models from scratch and know exactly what data they saw. This converts CoT evaluation from an observational science (collecting data points on whatever benchmarks exist) into an experimental science (formulating hypotheses about specific causal factors and testing them through controlled interventions).
The paper itself demonstrates the power of this approach through the fine-grained analysis in Table 2, which would be impossible in a real-world setting. By separately measuring reasoning trace exact match and answer exact match under different distribution shift conditions, the paper discovers that models can produce correct answers with incorrect reasoning (through commutativity-induced coincidences) and incorrect answers with correct reasoning traces (when they default to the closest training pattern). This diagnostic capability — distinguishing what the model got right from how it got there — is a direct consequence of DataAlchemy's transparency. In a real-world math benchmark, you could never know whether an apparently correct reasoning chain was actually driving the answer or was a coincidentally correct post-hoc rationalization.
Why this is an instrument, not just a dataset. The paper explicitly positions DataAlchemy as "a research platform for probing the intrinsic behavior of LLMs and facilitating the discovery of scientific principles." This is an important distinction: DataAlchemy is designed to be extensible and parameterizable. The alphabet size, transformation set, compositional depth, and element length are all tunable knobs. Future researchers can add new transformation types, new perturbation modes, or new generalization axes without rebuilding the framework from scratch. The code release on GitHub (referenced in the abstract) makes this platform available to the community, which is a significant infrastructure contribution beyond the specific experimental results reported in the paper.
Evidence anchoring the claim. The paper's ability to isolate the commutativity artifact (Table 2) — where f1∘f2 → f2∘f1 produces 100% answer accuracy with 0% reasoning accuracy — demonstrates the diagnostic power of the framework. In a less transparent setting, one would observe high answer accuracy and conclude that CoT reasoning generalized successfully. DataAlchemy reveals that the success was an artifact of transformation commutativity, not genuine generalization. The introduction of f3 (the non-commutative reversal operator) in Appendix D.1.2 eliminates this artifact, and performance collapses to 0% across all metrics for non-commutative transfer — confirming that the apparent success under f1/f2 commutativity was indeed spurious. This sequence — identify an artifact through transparency, eliminate it through controlled modification, observe the disappearance of a spurious effect — is a textbook demonstration of scientific methodology that is impossible with opaque, real-world benchmarks.
Innovation 3: The Task-Length-Format Tripartite Decomposition of Distribution Shift
The paper's decomposition of Δ(D_train, D_test) into three orthogonal axes — Δ_task, Δ_length, Δ_format — is a taxonomic contribution that provides a vocabulary and measurement framework for talking about distribution shift in CoT reasoning. This is not merely a list of possible shifts; each axis is motivated by a distinct mathematical mechanism and is shown to produce qualitatively different failure patterns, establishing that they are genuine independent dimensions rather than arbitrary categorizations.
Why this decomposition is non-obvious. At first glance, one might think that "distribution shift" is a monolithic phenomenon — any change in the input distribution should degrade performance, and the degradation should depend only on the magnitude of the change, not on its nature. The paper's experiments falsify this hypothesis. Task shifts (novel transformation compositions) produce a specific failure mode: the model defaults to the closest seen pattern, producing reasoning that is locally coherent but globally wrong (Table 2, Appendix E.1.1). Length shifts (longer or shorter elements, more or fewer reasoning steps) produce a different failure mode: the model rigidly adheres to the training length, adding or removing tokens to match the expected output size rather than adapting to the test length (Appendix E.2.1, E.2.2). Format shifts (inserted, deleted, or modified tokens) produce yet another failure mode: the model falls back on training-time statistical patterns, producing outputs that are locally plausible (right vocabulary, right approximate length) but globally incorrect (Appendix E.3). These distinct failure modes suggest that the three axes engage different aspects of the model's learned representation — length maps to positional encoding and attention pattern statistics, task maps to transformation composition grammar, and format maps to template matching — which is a more nuanced picture than a single "distribution shift → degradation" relationship.
Comparison to prior OOD generalization work. The broader ML literature on OOD generalization (Yang et al., 2023, 2024) typically treats distribution shift as a single phenomenon characterized by the divergence between training and test distributions. Domain-specific NLP work on OOD generalization (Budnikov et al., 2025; Zhang et al., 2024) has identified specific shift types (domain shift, temporal shift, demographic shift), but these are domain-dependent categories that don't translate across tasks. The paper's contribution is to identify three axes that are task-agnostic and mechanistically grounded — they don't depend on whether the task is math, planning, or symbolic manipulation, and they are tied to specific computational properties of transformer models (positional encoding for length, compositional structure for task, surface-form sensitivity for format). This makes the taxonomy applicable to any CoT evaluation, not just DataAlchemy experiments. A practitioner evaluating CoT on a new real-world task can ask: is the task introducing novel compositions (task shift), unusual reasoning depths (length shift), or surface-form variations (format shift)? The decomposition provides a diagnostic checklist for anticipating failure modes.
The measurement infrastructure. The paper doesn't just name the axes — it provides formal measurement frameworks for each. Task discrepancy is operationalized through the task complexity score T(C) (Proposition C.1), which counts position-wise atom novelties, individual transformation novelties, and full composition novelties. Length discrepancy is modeled through the squared-exponential decay function (Proposition C.2), which predicts smooth degradation near the training length and collapse for large extrapolations. Format discrepancy is operationalized through the format alignment score S(p_test) (Definition C.1), which measures cosine similarity to the nearest training prompt in embedding space. These measurement frameworks transform the taxonomy from a qualitative description into quantitative predictions — and the experimental results (Figures 3, 5, 7) confirm that the predicted degradation patterns match empirical behavior.
Evidence anchoring the claim. The distinct shapes of the degradation curves across axes provide the strongest evidence for genuine independence. Task generalization (Figure 3) shows a sharp collapse from ID to CMP — exact match drops from 100% to 0.01% — suggesting a threshold-like failure mode where even mild compositional novelty is catastrophic. Length generalization (Figure 5) shows a more gradual, symmetric degradation around the training length, consistent with the squared-exponential model. Format generalization (Figure 7) shows a smooth, monotonic decay with noise level that varies by perturbation mode (Insert causes more degradation than Delete or Modify at the same noise level, and Hybrid tracks the most aggressive mode). These qualitative differences in the functional form of degradation are strong evidence that the three axes are measuring genuinely different phenomena rather than three correlated measures of the same underlying shift.
Innovation 4: The Faithfulness-Via-Transparency Diagnostic
The paper makes a methodological contribution that is both simple and powerful: by constructing tasks where the ground-truth reasoning trace is known exactly (because transformations are mathematically defined), the paper can directly measure reasoning faithfulness — the alignment between what the model's intermediate steps say and what produces the final answer — without relying on intervention-based methods that are inherently limited. This provides some of the cleanest evidence to date that CoT reasoning can be unfaithful: models can produce correct answers through incorrect reasoning, and incorrect answers through superficially correct reasoning.
The faithfulness problem in prior work. Prior work on CoT faithfulness (Lanham et al., 2023; Bentham et al., 2024; Chen et al., 2025b) relied on intervention-based methods: corrupt intermediate reasoning steps and observe whether the final answer changes. If the answer stays the same when reasoning is falsified, the reasoning was unfaithful — it was not causally driving the answer. While informative, this approach has limitations. First, the interventions are themselves model-generated (you need to generate corrupted reasoning steps, which may not cover the space of possible unfaithfulness). Second, it is difficult to distinguish between "the model ignored the corrupted step" (unfaithfulness) and "the model recovered from the corruption using the remaining correct steps" (faithfulness under perturbation). The transparent setting of DataAlchemy sidesteps these issues entirely: because the reasoning trace is a deterministic function of the input, any deviation from the ground-truth trace is immediately identifiable as an error, and the relationship between trace correctness and answer correctness can be measured directly without interventions.
What the paper discovers. The fine-grained analysis in Table 2 reveals two distinct faithfulness failures. First, in the compositional generalization setting {f1∘f1, f1∘f2, f2∘f1} → f2∘f2, the model produces 100% correct reasoning but 0.01% correct answers. The reasoning trace (applying f1 then f2) is perfectly accurate as a sequence of operations — but it is applied to the wrong transformation, producing the wrong final result. This is "faithful but wrong" reasoning: the intermediate steps are not fabricated, but they don't correspond to the correct solution strategy for the given test query. Second, in the commutativity setting f1∘f2 → f2∘f1, the model produces 0% correct reasoning but 100% correct answers — the trace is wrong, but the answer happens to be right because the two transformations commute. This is "unfaithful but correct" reasoning: the answer is right for the wrong reason.
Both patterns would be invisible in a standard evaluation that only checks the final answer. The 100% answer accuracy on f1∘f2 → f2∘f1 would be interpreted as successful generalization, when in fact it reflects a commutativity coincidence. The 0.01% answer accuracy on the compositional generalization task would be interpreted as complete failure, when in fact the model is producing perfectly sensible (but misapplied) reasoning. Neither interpretation captures the true nature of the model's behavior, and both would lead to incorrect conclusions about model capability. Only by separately measuring reasoning and answer correctness — possible only because the ground-truth reasoning trace is known — can the paper diagnose these failure modes.
The f3 control experiment. The paper's introduction of the non-commutative reversal operator f3 in Appendix D.1.2 provides a crucial control. Under f2∘f3 → f3∘f2 (non-commutative transfer), exact match collapses to 0% for reasoning, answer, and full chain simultaneously (Table 3). This confirms that the apparent success under f1/f2 commutativity was an artifact — when commutativity cannot save the answer, the model's inability to generalize is fully exposed. This is elegant experimental design: by varying the algebraic structure of the transformation space (commutative vs. non-commutative), the paper can control for the coincidental correctness artifact and isolate genuine generalization.
Significance beyond this paper. The faithfulness-via-transparency diagnostic is a methodological template that can be applied beyond DataAlchemy. Any controlled environment where the ground-truth reasoning trace is known — symbolic regression, algorithmic reasoning, formal theorem proving — can adopt the same approach of separately measuring reasoning and answer correctness to diagnose faithfulness. The paper demonstrates that this diagnostic reveals behavior (unfaithful correctness, faithful incorrectness) that is invisible to standard accuracy metrics, which has direct implications for how CoT systems should be evaluated in safety-critical applications. If a model can produce correct answers for incorrect reasons, then answer accuracy alone is not a reliable indicator of reasoning quality — and users who trust reasoning traces as explanations are at risk of being misled. The ethical considerations section of the paper explicitly connects this finding to deployment risks: "the presence of a detailed reasoning trace should not be equated with correctness, reliability, or genuine understanding."
Evidence anchoring the claim. Table 2 is the key exhibit. The four rows show the full range of faithfulness patterns: correct reasoning + incorrect answer (row 1–2, 100%/0.01%), incorrect reasoning + correct answer (rows 3–4, 0%/100%), and full-chain exact match of 0% in all four non-ID rows (confirming that correct full-chain output requires both correct reasoning AND correct answer simultaneously, which never happens outside ID). Table 3 in the Appendix extends this to the f3 setting, where all metrics collapse to 0–10%, confirming that removing the commutativity artifact eliminates the spurious answer-level correctness. The qualitative examples in Appendix E ground these aggregate patterns in concrete model outputs, showing exactly what "100% correct reasoning, 0.01% correct answer" looks like in practice (Appendix E.1.1).
Innovation 5: The Negative Result That Scale Does Not Confer Robustness
The paper's finding that model scale (from 62K to 3B to 14B parameters) does not qualitatively change the relationship between distribution shift and CoT performance is a significant negative result that challenges a widely held assumption in the LLM community: that larger models are more robust, more capable of generalization, and more likely to exhibit emergent reasoning. The paper demonstrates that scaling amplifies in-distribution performance and accelerates interpolation within the training distribution, but does not enable principled extrapolation beyond it.
The scale-is-all-you-need assumption. A dominant narrative in the LLM literature — supported by scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) and the empirical observation that larger models perform better on most benchmarks — is that increasing model size unlocks qualitatively new capabilities, including reasoning. The "emergent abilities" literature (Wei et al., 2022b) specifically argued that certain capabilities appear only above critical scale thresholds. Applied to CoT reasoning, this narrative would predict that while small models might fail to generalize, sufficiently large models — with their greater capacity, more sophisticated attention patterns, and exposure to more diverse pretraining data — might learn genuinely transferable reasoning strategies that work even under distribution shift.
The paper directly tests and falsifies this hypothesis within its controlled setting. The internal validity experiments (Section 8.1, Figure 8, extended in Figure 15) compare models from 62K to 3B parameters across task, length, and format generalization. The external validity experiments (Section 8.2, Figure 9, Tables 5–7) extend this to 8B and 14B parameter instruction-tuned models. In every case, the shape of the degradation curve — the functional relationship between distribution shift magnitude and performance — is identical across scales. Larger models achieve higher absolute scores in near-in-distribution regimes (e.g., BLEU score is uniformly higher at ID and CMP for larger models), but they collapse just as completely as small models under OOD conditions. On transformation generalization (Figure 8, left panel), GPT-3B and GPT-68K produce nearly identical BLEU scores in the OOD setting (both near 0.3). On reasoning step generalization (Figure 8, center panel), models from 62K to 3B show the same sawtooth pattern: perfect at the training depth, near-zero at unseen depths. On format generalization (Figure 8, right panel), the edit distance curves for all model sizes are essentially superimposed.
What this means for capability narratives. The finding does not imply that scale is useless — larger models do perform better when the test distribution is similar to training. But it implies that the mechanism of improvement is better interpolation, not emerging extrapolation. A 3B-parameter model has a larger "in-distribution bubble" — it can handle test queries that are farther from training before performance collapses — compared to a 62K model, but when the bubble boundary is crossed, both collapse to the same low baseline. This is consistent with the "memorization + interpolation" model of LLM behavior: larger models memorize more training patterns and can interpolate more smoothly between them, but they are not learning the underlying transformation rules in a way that would enable applying them to genuinely novel inputs.
Significance for research direction. If scale does not confer robustness to distribution shift, then the path to more capable CoT reasoning is not "train bigger models." The bottleneck is distributional coverage — ensuring that training data spans the space of possible test queries — which is a data curation and augmentation problem, not a model architecture or scale problem. This redirects research investment from scaling (which is expensive and environmentally costly) toward understanding and controlling training distributions. It also suggests that the impressive CoT performance of very large models (GPT-4, Claude, Gemini) on real-world benchmarks may reflect those benchmarks' proximity to the models' training distributions (through data contamination or genuine coverage) rather than emergent reasoning capability — a hypothesis that the paper cannot verify directly due to the opacity of proprietary training data but that follows logically from the controlled experimental results.
Evidence anchoring the claim. Figures 8 and 15 (Appendix D.6) provide the internal validity evidence: six model sizes across GPT and LLaMA architectures, all showing superimposed or near-superimposed degradation curves. Figure 9 and Tables 5–7 (Appendix D.7) provide the external validity evidence: LLaMA3-8B and Qwen3-14B-Instruct, both state-of-the-art instruction-tuned models, show the same collapse patterns as the from-scratch models — exact match drops from 100% to 0–10% under compositional and OOD task shifts (Table 5), to 0% under reasoning step extrapolation (Table 6), and to near-zero under 30% format noise (Table 7). The small standard deviations in these tables (consistently <1% for exact match) confirm that the collapse is not noisy or stochastic — it is a deterministic consequence of distribution shift. The SFT experiments with different model sizes (Figure 14b, Appendix D.5) provide convergent evidence: at any fixed SFT ratio, larger models reach higher accuracy within the SFT-expanded distribution, but all models collapse to zero when the SFT support is exhausted, showing that scale accelerates interpolation without enabling extrapolation.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The paper does not use a fixed, pre-existing dataset; instead, it uses DataAlchemy to generate training and test data on-the-fly according to the distribution specifications for each experiment. For controlled (from-scratch) experiments, the training data is constructed from the 26-letter alphabet
A = {A, B, ..., Z}, elements (sequences of atoms) with lengths ranging from 2 to 6 (producing 676 to 308,915,776 possible data samples), and transformationsf1 = f_rot(e, 13)andf2 = f_pos(e, 1). The exact size of the training set for each experiment varies depending on the generalization scenario (how many elements, transformations, and compositions are included), but the data generation is exhaustive within the specified distribution — every valid input-output pair within the defined training distribution is included. For test sets, all valid inputs for the test distribution are evaluated, ensuring complete coverage. For fine-tuning experiments with pretrained SOTA models (LLaMA3-8B, Qwen3-14B-Instruct), the same DataAlchemy-generated data is used, with the paper explicitly verifying that these models produce "totally random answers" on DataAlchemy tasks before fine-tuning (Section 8.2), confirming that DataAlchemy data was not present in their pretraining corpora. -
Base model(s). Two categories of models are used, corresponding to the internal and external validity experiments. For controlled experiments (trained from scratch), the paper uses decoder-only transformer models with GPT and LLaMA architectures across a wide parameter range: 62K, 68K, 589K, 631K, 4.8M, 6M, 35M, 60M, 540M, 543M, 617M, 623M, and 3B parameters. Detailed architectural hyperparameters (hidden size, intermediate size, number of layers, number of heads) for every model size are provided in Table 8 (Appendix F.1). These models are chosen to span the full range from tiny (62K, which can only memorize patterns) to moderately large (3B, which has substantial representational capacity), enabling the paper to test whether the relationship between distribution shift and CoT performance is scale-dependent. For external validity experiments (fine-tuning pretrained models), the paper uses two state-of-the-art instruction-tuned models: LLaMA3-8B (Dubey et al., 2024) and Qwen3-14B-Instruct (Yang et al., 2025), chosen because they represent current SOTA capabilities and have different pretraining distributions, architectures, and scales, providing a strong test of whether the data distribution lens generalizes beyond the controlled setting.
-
Metrics. The paper uses three complementary metrics, each applied separately to three output components (reasoning trace, final answer, and full chain). Exact Match is the strictest metric: the fraction of test queries for which the model's output string perfectly matches the ground-truth string character-for-character, reported as a percentage. This is a binary pass/fail — even a single character error counts as failure. Edit Distance (normalized Levenshtein distance) measures the minimum number of single-character insertions, deletions, and substitutions needed to transform the predicted output into the ground truth, normalized by sequence length to produce a value in
[0, 1]. This captures partial correctness — a model that gets most of a sequence right will have a low edit distance even if exact match is 0%. BLEU Score measures n-gram overlap (typically n = 1 to 4) between predicted and ground-truth sequences, ranging from 0 to 1, with 1 indicating perfect overlap. BLEU captures local correctness and fluency — a model that produces sequences sharing many short substrings with the ground truth will have high BLEU even if the overall order is slightly wrong. The decomposition into reasoning/answer/full-chain metrics is a critical design choice: it enables diagnosis of where failures occur (e.g., correct reasoning but wrong answer, or wrong reasoning but correct answer through commutativity coincidences), which is central to the paper's faithfulness analysis. -
Baselines. Because this is a controlled experimental methodology paper rather than a method-comparison paper, the primary "baseline" is in-distribution (ID) performance — the model's accuracy when test queries are drawn from exactly the same distribution as training. This provides the upper bound against which all distribution-shifted performance is compared. For the format generalization experiments (Section 7), the baseline is 0% noise — the unperturbed prompt template. For the SFT experiments (Sections 5.1, Appendix D.2), the baseline is the 0% SFT data ratio — the model's performance on unseen transformations/elements before any fine-tuning. For the length generalization experiments, the baseline is performance at the training length (
l = 4for text length, the mixed-ratio training depths for reasoning steps). The paper does not compare against alternative reasoning methods (e.g., Tree-of-Thoughts, self-consistency, or standard prompting without CoT) because the research question is not "which method is better" but rather "under what distributional conditions does CoT succeed or fail." All comparisons are within-model, across-distribution rather than across-methods. -
Generation budget / compute accounting. For controlled experiments, all models are trained to convergence (10 epochs, batch size 1024) on the full training distribution for each experimental configuration. There is no "inference budget" per se — evaluation is conducted on the complete test set with greedy decoding (temperature =
1 × 10^{-5}, essentially deterministic). The computational cost is reported in terms of hardware (8 NVIDIA A100 GPUs with 80 GB memory each, 4 NVIDIA H200 GPUs) and the note that toy experiments can run on a single A100, but no total GPU-hour accounting is provided. For fine-tuning SOTA models, the training budget is fixed at 24,000 optimization steps with an effective batch size of 128. The paper does not perform any inference-time scaling experiments (varying the number of samples, using search, or applying majority voting), so conventional "generation budget" comparisons (e.g., best-of-N at different N) are not present. This is consistent with the paper's focus on training-time distribution as the independent variable rather than inference-time computation. -
Cross-validation / statistical protocol. For controlled experiments (models trained from scratch), results are averaged over three independent training runs with different random seeds (Appendix F.1), and the reported metrics (exact match, edit distance, BLEU) reflect the mean across these runs. For the difficulty-bin-style analyses (Figures 3, 5, 7), each data point represents the aggregate across all test queries within the specified distribution condition — there is no cross-validation or held-out tuning because there are no hyperparameters being selected on test data (the model architecture and training hyperparameters are fixed in advance). For SOTA model fine-tuning experiments (Appendix D.7, Tables 5–7), results are reported with mean ± standard deviation, confirming that the observed effects are stable and not driven by sampling noise. The standard deviations are consistently small (e.g., exact match standard deviations are 0.00% for ID and OOD settings in Table 5, and ≤0.47% for format generalization in Table 7), indicating deterministic or near-deterministic behavior under the evaluated conditions. No cross-validation is used for strategy selection (unlike the example paper's compute-optimal policy selection) because there is no strategy being selected — the experimental design tests fixed distribution-shift conditions.
Main Quantitative Results
The paper's experimental program is organized around three independent axes of distribution shift: task generalization (Section 5), length generalization (Section 6), and format generalization (Section 7). Together, these experiments test the central hypothesis that CoT reasoning effectiveness is governed by the distributional discrepancy between training and test data, with performance degrading monotonically as discrepancy increases along any axis.
Task Generalization (Section 5)
Task generalization decomposes into two sub-axes: transformation generalization (can the model handle novel compositions of operations?) and element generalization (can the model handle novel input symbols or arrangements?).
Headline finding for transformation generalization: CoT reasoning achieves 100% exact match on in-distribution (ID) transformation tasks but collapses to near-zero under even modest compositional novelty, with the severity of collapse scaling with the degree of distribution shift.
Aggregate results (Table 1, Figure 3). For the transformation generalization experiment with the configuration f1∘f1 → f1∘f1 (where the model is trained on a specific set of compositional transformations and tested on progressively more novel ones), the degradation is sharp and monotonic:
- ID: exact match = 100.00%, edit distance = 0, BLEU = 1.0. The model perfectly reproduces seen transformations.
- CMP (Compositional): exact match = 0.01%, edit distance = 0.1326, BLEU = 0.6867. Despite having seen all individual transformations (
f1,f2) and various compositions during training, the model fails almost completely when asked to handle a novel composition (f2∘f2as input). The relatively high BLEU (0.69) indicates that the model produces locally plausible output (correct atoms in approximately correct patterns) but fails to produce the globally correct full chain. - POOD (Partially Out-of-Distribution): exact match = 0.00%, edit distance = 0.1671, BLEU = 0.4538. When the test composition includes transformations not seen in any training context, performance degrades further.
- OOD (Out-of-Distribution): exact match = 0.00%, edit distance = 0.2997, BLEU = 0.2947. Complete novelty leads to the worst performance across all metrics.
Figure 3 visualizes this monotonic degradation: edit distance increases (from 0 to ~0.30) and BLEU score decreases (from 1.0 to ~0.29) as the distribution shift increases from ID to OOD.
Fine-grained faithfulness analysis (Table 2). The paper's decomposition into reasoning trace and final answer reveals striking dissociations that would be invisible if only full-chain accuracy were reported:
{f1∘f1, f1∘f2, f2∘f1} → f2∘f2: Reasoning exact match = 100.00%, Answer exact match = 0.01%, Full Chain exact match = 0.01%. The model produces perfectly correct reasoning traces (it accurately describes applyingf2thenf2) but arrives at the wrong answer because it applies the reasoning template to the wrong input context — it defaults to the closest training pattern rather than adapting to the novel composition. This is "faithful but wrong" reasoning.f1∘f2 → f2∘f1(and its reverse): Reasoning exact match = 0.00%, Answer exact match = 100.00%, Full Chain exact match = 0.00%. The model produces incorrect reasoning but correct answers becausef1andf2happen to commute for these specific inputs (ROT-13 and cyclic shift by 1 are orthogonal operations that produce the same result regardless of order for certain elements). This is "unfaithful but correct" reasoning — the answer is right for the wrong reason.
SFT recovery curve (Figure 4, Section 5.1). When a small fraction λ of unseen transformation data is added via supervised fine-tuning, performance recovers rapidly. Even λ = 1.5 × 10^{-4} (0.015% of the training data) enables substantial generalization to previously unseen transformations. The recovery is faster for smaller distribution shifts: CMP reaches higher accuracy at lower λ than POOD, which in turn outperforms OOD. This demonstrates that the model is not incapable of handling novel transformations — it simply needs to see them in training, confirming that CoT reasoning is fundamentally about pattern interpolation within the training distribution rather than inferring rules that generalize to unseen operations.
Element generalization results (Figure 11, Appendix D.2). Performance degrades similarly under element-level distribution shift. In the CMP setting (same atoms, different order), BLEU score drops from 1.0 (ID) to approximately 0.46–0.71 depending on the transformation, with exact match collapsing to 0% across all CMP configurations. In the OOD setting (entirely novel atoms N–Z when trained on A–M), BLEU collapses to 0% and exact match to 0%. The SFT experiments for element generalization (Figure 12a) show that performance recovers rapidly when SFT data is close to the test distribution (edit distance n = 1 from training), but plateaus at low performance for larger edit distances (n = 3, 4), indicating that CoT generalization is highly local — the model can interpolate to elements that are one or two token edits away from training examples but cannot extrapolate to fundamentally different element structures.
Length Generalization (Section 6)
Length generalization decomposes into text length generalization (varying the number of atoms per element) and reasoning step generalization (varying the number of compositional transformations).
Headline finding for text length generalization: CoT reasoning is near-perfect at the training length (l = 4) but degrades significantly for both shorter and longer elements, with the degradation approximately symmetric around the training length.
Text length results (Figure 5, Table 4). When the model is trained exclusively on elements of length l = 4 and tested on lengths from l = 2 to l = 6:
- At
l = 4(ID): exact match = 100%, edit distance = 0, BLEU = 1.0 for all components. - At
l = 3(one less than training): edit distance increases to 0.2221 for full chain, BLEU drops to 0.5471. - At
l = 5(one more than training): edit distance = 0.1818, BLEU = 0.6220 — slightly better thanl = 3, suggesting the model handles slightly longer sequences better than slightly shorter ones. - At
l = 2andl = 6(two steps away): performance degrades further. Atl = 6, edit distance = 0.3294, BLEU = 0.4763. - Exact match is 0% for all non-training lengths (Table 4), confirming that even single-character length deviations cause complete failure in terms of binary correctness.
Qualitative examples (Appendix E.2.1) reveal the failure mode: when tested on an element of length 5 (e.g., "I G L L Q"), the model outputs a 4-atom answer ("T Y Y V") instead of the required 5-atom answer ("T Y Y D V"), rigidly adhering to the output length learned during training. The model adds or removes tokens to match the training distribution's length statistic rather than adapting to the test input.
Padding strategy effects (Figure 13, Appendix D.3). Three padding strategies are tested for their effect on length generalization:
- None (no padding): The baseline, showing the symmetric degradation around the training length.
- Padding to max length: Padding training sequences to a fixed maximum length does not improve length generalization — the model still sees only one effective sequence length (the padded length). The BLEU curve for Padding nearly overlaps with None.
- Group (segmentation): Grouping and truncating sequences into segments of fixed maximum length significantly improves length generalization. The BLEU scores at
l = 3andl = 5are substantially higher under the Group strategy than under None or Padding. This is because the Group strategy exposes the model to a distribution of effective sequence lengths during training (different examples have different numbers of segments), widening the training distribution along the length axis and making mild length shifts at test time look closer to in-distribution.
Headline finding for reasoning step generalization: CoT reasoning performance on a given reasoning depth is directly proportional to the fraction of training data at that depth, with near-zero performance on depths never seen during training.
Reasoning step results (Figure 6). When the model is trained on a mixture of k = 1 and k = 2 tasks with varying mix ratios, and evaluated on each depth separately:
- As the mix ratio increases from 0.0 (all
k = 2, nok = 1) to 1.0 (allk = 1, nok = 2), performance onk = 1tasks increases monotonically from ~0% to ~80% exact match, while performance onk = 2tasks decreases from ~80% to ~0%. - At the 0.5 mix ratio, both
k = 1andk = 2achieve approximately 40% exact match — the model splits its capacity between the two depths. - For the
k = 3vs.k = 2mixture (right panel of Figure 6), the same pattern holds: performance onk = 3(the less-represented depth) increases with its training proportion, while performance onk = 2decreases. - Critically, when a depth is entirely absent from training (mix ratio = 0.0 for that depth), exact match drops to 0% — the model cannot generalize to reasoning chains of a depth it has never seen, even though it has seen the individual transformations and the compositional structure at other depths. This is direct evidence against the hypothesis that CoT teaches models an abstract "how to compose operations" skill that transfers to novel depths.
The k = 3 extrapolation: When trained only on k = 1 and k = 2, the model achieves 0% exact match on k = 3 tasks. This is particularly striking because k = 3 is just "one more step" than k = 2 — the model has seen both individual transformations and two-step compositions, yet cannot handle three-step chains. This suggests that the model is not learning a recursive composition rule but rather memorizing templates for specific depths.
Format Generalization (Section 7)
Headline finding: CoT reasoning performance degrades smoothly and monotonically with the noise level in the prompt template, with Insert perturbations causing the most severe degradation and the degradation varying by which part of the prompt (element, transformation, instruction) is perturbed.
Aggregate results (Figure 7). For the four perturbation modes at noise levels from 5% to 30%:
- Insert: The most damaging perturbation. At 5% noise, edit distance already reaches approximately 0.15; at 30% noise, edit distance exceeds 0.8 and BLEU drops below 0.1.
- Delete: Less damaging than Insert. At 30% noise, edit distance reaches approximately 0.5, BLEU approximately 0.25.
- Modify: Similar to Delete in its effect, with edit distance reaching approximately 0.55 and BLEU approximately 0.15 at 30% noise.
- Hybrid (all three combined): Tracks the most aggressive individual perturbation mode, with edit distance exceeding 0.8 and BLEU dropping below 0.1 at 30% noise.
The degradation is monotonic in noise level for all perturbation modes — larger p consistently produces worse performance, consistent with the prediction that Δ_format increases with noise level, which loosens the generalization bound of Theorem 3.1.
Perturbation area analysis (Figure 7, right panel). When perturbations are applied to different "areas" of the prompt (element tokens, transformation tokens like [F1] and [F2], and instruction tokens like thinking and <answer>), clear differential effects emerge:
- Element perturbations: Most damaging across all noise levels. At 30% noise, edit distance reaches ~0.8 and BLEU drops to ~0.15. This is expected because the elements carry the actual content that the transformations operate on — corrupting them changes the problem itself.
- Transformation perturbations: Second most damaging. At 30% noise, edit distance reaches ~0.7, BLEU ~0.2. The transformation tokens specify which operations to apply, so corrupting them changes the task identity.
- Instruction perturbations: Least damaging. At 30% noise, edit distance is ~0.45, BLEU ~0.5. The instruction tokens ("thinking",
<answer>) provide scaffolding but don't carry task-specific information, so corrupting them is less harmful. - None (no perturbation): Edit distance = 0, BLEU = 1.0, serving as the ID baseline.
This hierarchy (element > transformation > instruction) aligns with intuition: content-carrying tokens matter more than structural tokens for task performance.
Qualitative failure mode (Appendix E.3). When a deletion perturbation removes one of the transformation tokens (e.g., only [F1] remains when the training template always had [F1][F2]), the model does not simply perform the single transformation specified. Instead, it produces an output whose length matches the two-step pattern it was trained on — it pads the answer with extra atoms drawn from the same vocabulary distribution — but whose content is wrong. This demonstrates that format perturbations trigger a joint mishandling of format and length: once the prompt drifts outside the learned template, the model falls back on training-time statistics (output length, per-position token distribution) rather than adapting to the specific prompt content.
Generality of the Data Distribution Lens (Section 8)
This section does not introduce new generalization axes but tests whether the patterns observed in Sections 5–7 hold across different model configurations.
Internal validity (Section 8.1, Figures 8 and 15). Across GPT and LLaMA architectures ranging from 62K to 3B parameters:
- Task generalization (Figure 8, left): All model sizes show the same monotonic degradation from ID to CMP to POOD to OOD. Larger models achieve uniformly higher BLEU scores in ID and near-ID regimes (e.g., GPT-3B achieves BLEU ~0.95 at ID vs. ~0.85 for GPT-68K), but all models converge to approximately the same low BLEU (~0.3) at OOD. The shape of the degradation curve is invariant to scale.
- Length generalization (Figure 8, center): The sawtooth pattern — perfect at the training depth, near-zero at unseen depths — is identical across all model sizes and architectures. BLEU scores at
k = 1andk = 3(unseen depths) are approximately 0.2–0.4 for all models, with no systematic improvement from larger models. - Format generalization (Figure 8, right): Edit distance curves for all model sizes are essentially superimposed. There is no evidence that larger models are more robust to format perturbations — at 30% noise, all models achieve edit distances in the 0.4–0.5 range.
Figure 15 (Appendix D.6) extends these results to an even wider range of model sizes (GPT-68K through LLaMA-3B) and confirms the same patterns.
External validity (Section 8.2, Figure 9 and Tables 5–7). When LLaMA3-8B and Qwen3-14B-Instruct are fine-tuned on DataAlchemy data and evaluated under distribution shifts:
- Task generalization (Table 5): Both models achieve 100% exact match at ID and collapse to near-zero under CMP (LLaMA3-8B: 8.52%, Qwen3-14B-Instruct: 0.01%), POOD (0.00% for both), and OOD (0.00% for both). Edit distance increases from 0.00 to 0.27 (LLaMA3-8B) and 0.38 (Qwen3-14B-Instruct) at OOD. BLEU drops from 1.00 to 0.27–0.36 at OOD. The standard deviations are consistently small (≤0.02 for edit distance, 0.00 for BLEU), confirming stability.
- Reasoning step generalization (Table 6): Both models achieve 100% exact match at the training depth (
k = 2) and collapse to 0% at unseen depths (k = 1andk = 3), with the minor exception of Qwen3-14B-Instruct achieving 0.26 ± 0.05% atk = 3— a negligible level of performance. Edit distance increases from 0.00 atk = 2to 0.75 (LLaMA3-8B) and 0.54 (Qwen3-14B-Instruct) atk = 1, and to 0.54 (LLaMA3-8B) and 0.65 (Qwen3-14B-Instruct) atk = 3. - Format generalization (Table 7): Both models show smooth, monotonic degradation with noise level. At 0% noise, exact match = 100% and edit distance = 0 for both models. At 30% noise, exact match drops to 1.09 ± 0.10% (LLaMA3-8B) and 0.22 ± 0.05% (Qwen3-14B-Instruct), edit distance increases to 0.60 ± 0.08 and 0.62 ± 0.02, and BLEU drops to 0.03 ± 0.00 and 0.06 ± 0.00.
Figure 9 visualizes these results: the degradation curves for LLaMA3-8B and Qwen3-14B-Instruct closely track each other across all three generalization axes, and the shapes of the curves mirror those observed in the from-scratch experiments (Figures 3, 5, 7). The quantitative differences between the two models are small (Qwen3-14B-Instruct sometimes shows slightly higher BLEU at intermediate shift levels, but both collapse at extreme shifts), and the qualitative pattern — monotonic degradation with distribution discrepancy, sharp collapse under compositional novelty — is invariant.
Ablation Studies and Robustness Checks
Each subsection reports on a specific factor varied to test the robustness of the main findings.
Alternative transformation set with non-commutative f3 (Appendix D.1.2, Table 3): To address the possibility that the apparent answer-level correctness observed in Table 2 (100% answer accuracy under commutativity) might mask genuine generalization, the paper introduces f3 (sequence reversal), which does not commute with f1 (ROT-13) or f2 (cyclic shift). Under non-commutative transfer conditions such as {f2∘f3, f3∘f2, f3∘f3} → f2∘f2, exact match drops to 6.66% for the full chain, with reasoning exact match also at 6.66% and answer at 10.25%. Under strictly non-commutative transfers like f2∘f3 → f3∘f2 and its reverse, exact match collapses to 0% for reasoning, answer, and full chain simultaneously, with edit distance exceeding 0.22 and BLEU dropping below 0.10. This confirms that the apparent success under commutative transfers (Table 2) was an artifact of the algebraic structure of f1 and f2 — once commutativity is removed, the model's inability to generalize to novel transformation compositions is fully exposed.
SFT data ratio variation for transformation generalization (Figure 4): The fraction λ of unseen transformation data included during SFT is varied from 0 to 6 × 10^{-4}. At λ = 0 (no SFT), exact match is 0% for CMP, POOD, and OOD. As λ increases, performance recovers, with CMP reaching the highest accuracy (nearly 100% at λ ≈ 5 × 10^{-4}), POOD intermediate, and OOD the lowest. The ordering is consistent with the distribution discrepancy hierarchy: smaller shifts require less SFT data to recover performance.
SFT data ratio variation for element generalization (Figure 12a): Varying the SFT data ratio for element generalization reveals that performance recovery depends on the edit distance n between training and test elements. For n = 1 (one atom position changed), exact match recovers quickly with small SFT ratios. For n = 2, recovery is slower and plateaus at a lower ceiling. For n = 3 and n = 4, exact match plateaus at near-zero even with substantial SFT data (λ up to 1.0), indicating that element-level generalization is highly local — the model can interpolate to elements within ~1–2 edit operations of training examples but cannot extrapolate to elements with larger structural differences even with targeted fine-tuning.
Reasoning/answer consistency during SFT (Figure 12b): For element generalization at n = 3, the paper separately tracks reasoning exact match and answer exact match during SFT. The two metrics do not co-vary perfectly — the model can learn to produce correct final answers before its reasoning steps catch up, and vice versa. This decoupling is visible as a gap between the reasoning and answer curves at intermediate SFT ratios, providing mechanistic insight into the inconsistent traces-and-answers phenomenon observed in the main task generalization results.
Padding strategy for length generalization (Figure 13, Appendix D.3): Three strategies are compared: no padding, padding to max length, and group padding with truncation. The Group strategy significantly outperforms both None and Padding at all non-training lengths, with BLEU scores approximately 0.2–0.3 higher at lengths l = 3 and l = 5. This demonstrates that the length generalization failure is not an inherent limitation of the transformer architecture but rather a consequence of the training distribution — when the model is exposed to a distribution of sequence lengths during training (via grouping), its ability to handle test sequences of varying length improves substantially.
Temperature variation (Figure 14a, Appendix D.5): Decoding temperature is varied from 10^{-5} to 10.0 for transformation generalization experiments across CMP, POOD, and OOD scenarios. For temperatures 10^{-5} through 1.0, BLEU scores remain essentially constant at 0.687 (CMP), 0.454 (POOD), and 0.277–0.300 (OOD), and edit distances remain constant at 0.133 (CMP), 0.167–0.168 (POOD), and 0.299–0.300 (OOD). At temperatures above 1.0 (5.0, 10.0), performance degrades (BLEU drops, edit distance increases) as decoding becomes increasingly random, but the relative ordering of CMP > POOD > OOD is preserved. This confirms that the reported results are not artifacts of a particular temperature setting — the distribution dependence is stable across a wide range of sampling regimes.
Model size variation for SFT-based recovery (Figure 14b, Appendix D.5): When models of different sizes (68K, 589K, 4.8M, 35M, 543M parameters) are fine-tuned on varying SFT ratios for the f1∘f1 → f2∘f2 transformation generalization task, larger models consistently achieve higher exact match at any given SFT ratio. However, all models — regardless of size — follow the same qualitative trajectory: near-zero performance at low SFT ratios, rapid improvement as the SFT ratio increases, and convergence to near-perfect accuracy once the SFT data sufficiently covers the test distribution. The key observation is that model scale accelerates interpolation (larger models reach higher accuracy with less SFT data) but does not enable extrapolation — at SFT ratios too small to cover the test distribution, even the largest models collapse.
Model size and architecture variation for all generalization axes (Figure 15, Appendix D.6): Extending the internal validity results in Figure 8, this expanded figure shows task, length, and format generalization for six additional model configurations (GPT-68K, LLaMA-631K, GPT-4.8M, LLaMA-60M, GPT-543M, LLaMA-3B). Across all six configurations, the degradation curves are qualitatively identical to those in Figure 8: monotonic degradation with task shift, sawtooth pattern for reasoning step shift, and monotonically increasing edit distance with format noise. The curves for different model sizes are largely superimposed or nearly so, with the largest models (LLaMA-3B, GPT-543M) showing slightly better BLEU in ID and near-ID regimes but converging to the same low values as small models under extreme shifts.
Comparison with representative evaluation frameworks (Figure 10, Appendix A.4): The paper contrasts DataAlchemy's evaluation paradigm (abstract symbolic tasks, training from scratch, fully controlled distributions) with standard evaluation frameworks (pretrained LLMs evaluated on real-world benchmarks) and with Apple's evaluation framework (which evaluates both reasoning traces and final answers but uses pretrained models on specific puzzle tasks). The comparison highlights that DataAlchemy uniquely satisfies three criteria: abstract representation (unifying diverse NLP tasks), fully controlled experiments (enabling fine-grained factor isolation), and training from scratch (eliminating data leakage). Figure 10 visually contrasts the data flow in these paradigms, emphasizing that only DataAlchemy's bottom-up approach (abstract problems → trained-from-scratch LLMs → answer + reasoning trace evaluation) avoids the contamination issues inherent in top-down pretrained-model evaluation.
Critical Assessment
This section examines the degree to which the reported experiments substantiate the paper's central claims, identifies gaps and weaknesses, and discusses what additional experiments would have strengthened the conclusions.
On the Claim That CoT Reasoning Is a "Brittle Mirage"
The paper's headline claim — that CoT reasoning "can be a mirage, emerging from memorized or interpolated patterns in the training data rather than logical inference" (Section 1) — is strongly supported within the DataAlchemy environment but the gap to real-world NLP remains significant. The experimental evidence is internally compelling: exact match collapses from 100% to near-zero under even modest distribution shifts across all three axes (Tables 1, 4; Figures 3, 5, 7), and the faithfulness analysis (Tables 2, 3) demonstrates that models can produce correct answers through incorrect reasoning (commutativity coincidences) and incorrect answers through correct-looking reasoning traces (pattern replication). The generality of these findings across model scales (62K to 3B, Figures 8, 15) and the replication in SOTA models (LLaMA3-8B, Qwen3-14B-Instruct; Figure 9, Tables 5–7) rules out the hypothesis that the brittleness is an artifact of small scale or synthetic training.
However, three caveats temper the strength of this claim when extrapolated to real-world CoT reasoning:
Caveat 1: The gap between symbolic composition and natural language semantics. DataAlchemy distills reasoning into purely compositional transformations on a 26-letter alphabet — there is no semantics, no ambiguity, no world knowledge, and no pragmatic inference. This is the point of the abstraction (it enables controlled experiments), but it also means that the failure modes documented in DataAlchemy may not be the failure modes that matter most in practice. Real-world CoT reasoning involves semantic understanding (what does the word "average" mean in this context?), factual recall (what is the capital of France?), and pragmatic inference (what is this question actually asking?), none of which are captured by the ROT-13/cyclic-shift abstraction. The paper's finding that CoT fails under compositional distribution shifts in DataAlchemy does not directly imply that CoT fails under compositional distribution shifts in, say, mathematical word problems — the nature of the shift is fundamentally different. The external validity experiments (SOTA models fine-tuned on DataAlchemy) partially address this by showing that the brittleness persists even in models that do have semantic understanding from pretraining, but the fine-tuned models are still evaluated on the same symbolic tasks, so the semantic capabilities of the pretrained models are not being tested under distribution shift.
Caveat 2: The paper does not demonstrate that real-world CoT successes are distribution-dependent. The paper establishes that in DataAlchemy, CoT performance is distribution-dependent. The logical inference is that in the real world, CoT performance is also distribution-dependent. But this inference relies on the assumption that DataAlchemy captures the essential structure of real-world CoT reasoning — an assumption the paper argues for (Appendix A.4) but does not empirically validate. To directly validate the claim for real-world reasoning, one would need to identify a real-world benchmark where the training distribution is known (or can be controlled) and demonstrate that CoT performance degrades as the test distribution diverges from training. The paper does not conduct such an experiment, which leaves the external validity of the central claim dependent on the reader's willingness to accept the DataAlchemy abstraction as sufficient.
Caveat 3: The paper equivocates between two distinct claims. The title asks "Is Chain-of-Thought Reasoning of LLMs a Mirage?" — a question about whether CoT in general reflects genuine reasoning. The paper's evidence supports a narrower claim: "In the DataAlchemy environment, CoT reasoning behaves as pattern interpolation and fails under distribution shift." The leap from the narrow (synthetic) claim to the broad (general) claim requires assumptions about the representativeness of DataAlchemy that are not empirically validated. The paper is aware of this limitation (the Limitations section acknowledges that DataAlchemy "may inevitably not fully capture the semantic richness, ambiguity, and compositional diversity present in natural language"), but the title and abstract frame the finding in more sweeping terms than the evidence strictly supports.
On the Claim That Distribution Shift Is the Primary Determinant of CoT Efficacy
The claim that the data distribution lens provides a unified explanation for when and why CoT succeeds or fails is well-supported as a within-framework principle but underdetermined as a general theory. The experimental design systematically varies Δ_task, Δ_length, and Δ_format and consistently observes monotonic degradation (Figures 3, 5, 7). The theoretical generalization bound (Theorem 3.1) provides a formal link between distribution discrepancy and test risk. The three-axis decomposition is motivated by distinct mechanisms (compositional novelty for task, positional encoding mismatch for length, template matching for format) and produces distinct degradation curve shapes, supporting the claim that these are genuinely independent axes rather than correlated measures of a single underlying shift.
However, several aspects of this claim remain underdeveloped:
The composition function Φ is never specified or measured. Equation 7 states that Δ(D_train, D_test) = Φ(Δ_task, Δ_length, Δ_format), where Φ is a "monotonically increasing composition function." The paper never operationalizes Φ beyond this definitional statement. The experiments vary one axis at a time while holding the others fixed, which demonstrates that each axis individually affects performance, but does not test whether the axes compose according to a specific functional form (e.g., additive, multiplicative, max). Without measuring Φ or testing its predictions for multi-axis shifts, the paper does not establish that the three axes jointly explain CoT performance in the way the theoretical framework implies. A natural experiment — simultaneously varying two axes (e.g., task shift + length shift) and testing whether the degradation is predicted by some combination of the single-axis degradations — is not conducted.
The discrepancy measures are underdetermined by the data. The task complexity score T(C) (Proposition C.1) depends on three free parameters α, β, γ that weight the contributions of atom novelty, individual transformation novelty, and compositional novelty. The paper does not fit these parameters to data or provide principled rules for setting them. The squared-exponential length decay (Proposition C.2) depends on a width parameter σ that is described as "an empirical fit parameter" but is never actually fit to the data. The format alignment score S(p_test) (Definition C.1) depends on an embedding function η(·) that is never specified or computed. This means that while the qualitative predictions of the discrepancy measures (monotonic degradation with increasing novelty/length gap/noise) are confirmed by the experiments, the quantitative predictions are untested — we do not know whether the functional forms proposed (exponential decay in T(C), squared-exponential in |L - L_train|, cosine similarity for format alignment) actually match the empirical degradation curves. The paper could have strengthened this aspect significantly by fitting the proposed models to the experimental data and reporting goodness-of-fit, or by comparing alternative functional forms (e.g., linear vs. exponential vs. sigmoidal degradation).
The three axes are not exhaustive. The paper acknowledges (Limitations) that other forms of distribution shift — cross-lingual transfer, multi-modal reasoning, cultural context shifts — are not modeled. Within the DataAlchemy framework, additional axes could be explored but are not: for example, rate of transformation application (e.g., ROT-5 vs. ROT-13), noise within the element (typos in the input atoms rather than in the prompt template), or compositional depth as a separate axis from text length. The choice of exactly three axes is motivated by revisiting "existing NLP tasks" (Section 3) but is not argued to be complete.
On the Claim That Model Scale Does Not Confer Robustness to Distribution Shift
This claim is among the best-supported in the paper, with extensive evidence across model sizes, architectures, and training paradigms. The qualitative invariance of degradation curves across six orders of magnitude in parameter count (62K to 3B, Figures 8 and 15) and the replication in 8B and 14B SOTA models (Figure 9, Tables 5–7) is striking. The observation that larger models achieve higher absolute performance in near-ID regimes but converge to the same low baseline under extreme shifts is replicated across all three generalization axes.
However, two limitations should be noted:
The largest model tested (14B) is still two orders of magnitude smaller than the largest deployed LLMs (GPT-4 is estimated at ~1.8T). The paper's finding that scale does not confer robustness might not hold at the extreme scale of frontier models. There could be qualitative changes in generalization behavior at scales beyond those tested — the paper's own data cannot rule this out. The "emergent abilities" literature (which the paper implicitly argues against) specifically claims that certain capabilities appear only at very large scales, and the tested scale range (62K–14B) may be below the threshold for emergent reasoning, if such a threshold exists.
The SOTA models are fine-tuned, not pretrained, on DataAlchemy data. The external validity experiments fine-tune LLaMA3-8B and Qwen3-14B-Instruct on DataAlchemy data, which means the models' behavior reflects an interaction between their pretrained knowledge and the fine-tuning distribution. It is possible that a model pretrained from scratch on DataAlchemy-style data at a scale of 100B+ parameters would behave differently — the fine-tuning paradigm may constrain the model's ability to leverage its pretrained semantic understanding in ways that pure pretraining would not. The paper cannot test this hypothesis because pretraining a 100B+ parameter model from scratch on synthetic data is computationally prohibitive.
On the Claim That DataAlchemy Is a Valid Scientific Instrument for Studying CoT Reasoning
The paper's methodological claim — that DataAlchemy enables "rigorous scientific investigation" (Section 1) by providing an abstract, controllable, and clean environment — is well-justified internally but the argument for external validity relies on a leap of faith. The paper convincingly demonstrates that DataAlchemy solves the three problems it identifies with standard evaluation: narrowly defined settings (DataAlchemy abstracts across tasks), data entanglement (DataAlchemy enables independent factor variation), and data leakage (DataAlchemy trains from scratch on known data). The internal experiments are methodologically rigorous: large sample sizes (the full combinatorial space of test inputs within each distribution), multiple metrics with fine-grained decomposition (reasoning/answer/full chain), and replication across model scales and architectures.
However, the paper's claim that "the principles uncovered are extensible to large-scale pre-trained models" (Appendix G) is less well-supported than the internal validity claims. The external validity experiments show that SOTA models when fine-tuned on DataAlchemy data exhibit the same brittleness as from-scratch models, but this does not directly demonstrate that SOTA models' CoT reasoning on real-world tasks is distribution-dependent in the same way. The fine-tuned SOTA models are essentially learning the DataAlchemy task from scratch (starting from pretrained weights) rather than applying their pretrained CoT reasoning capabilities to DataAlchemy tasks. To test whether pretrained CoT reasoning is distribution-dependent, one would need to take a SOTA model that already exhibits CoT reasoning (without fine-tuning) and evaluate it on tasks where the training distribution can be controlled or at least characterized — which is exactly what the paper argues is impossible due to data opacity.
Missing Experiments That Would Have Strengthened the Paper
Multi-axis shift experiments: As noted above, the paper varies one axis at a time. Experiments that simultaneously vary two or three axes would test whether the axes compose as predicted by the framework and would reveal interaction effects that the current design cannot detect. For example: does format perturbation interact with task novelty such that the degradation is more than additive? The theoretical framework (the composition function Φ) implies a specific relationship, but this is never tested.
Comparison with non-CoT baselines: The paper evaluates only CoT reasoning (the model is trained to produce reasoning traces and evaluated on them). A natural control would be training and evaluating models that produce only the final answer (no intermediate reasoning trace) and comparing their distribution-shift robustness to CoT models. If answer-only models are more robust to distribution shift than CoT models, that would strengthen the claim that CoT reasoning is specifically brittle. If they are equally brittle, that would suggest the brittleness is a property of the training paradigm rather than CoT specifically.
Probing the faithfulness of SOTA model CoT: The external validity experiments evaluate SOTA models on DataAlchemy tasks, but they do not separately report reasoning vs. answer accuracy for these models (Tables 5–7 report only full-chain metrics). Given that the faithfulness analysis (Table 2) is one of the paper's most striking findings, replicating it with SOTA models would significantly strengthen the external validity argument. Can LLaMA3-8B produce correct answers through incorrect reasoning when fine-tuned on DataAlchemy? The paper does not say.
Comparison with human performance on DataAlchemy tasks: If humans can easily generalize to novel transformation compositions or element configurations in DataAlchemy (which seems likely, since the transformations are mathematically simple), then the paper's finding that LLMs cannot would be strengthened by the contrast. The paper does not report human baseline performance on DataAlchemy tasks.
Gradient of real-world CoT evaluation: The paper could have complemented the fully synthetic DataAlchemy experiments with a semi-synthetic experiment: take a real-world reasoning benchmark (e.g., a subset of MATH or GSM8K), systematically perturb it along the three axes (e.g., rephrase problems to test format generalization, vary problem length, construct compositional variants), and evaluate pretrained LLMs on the perturbed versions. This would bridge the gap between the fully controlled DataAlchemy results and real-world CoT evaluation, providing evidence that the distribution dependence observed in DataAlchemy manifests in natural language reasoning as well — even if the exact training distribution is unknown.
Summary of Strengths and Weaknesses
Genuine strengths: The experimental design is internally rigorous and well-motivated. The systematic variation of one axis at a time, the use of multiple complementary metrics, the fine-grained decomposition into reasoning and answer components, and the extensive replication across model scales and architectures are all strong methodological choices. The faithfulness analysis (Tables 2, 3) is particularly elegant and demonstrates the diagnostic power of the DataAlchemy framework. The negative result on scale (Figures 8, 15) is important and well-supported by the data.
Genuine weaknesses: The gap between the synthetic environment and real-world CoT reasoning is larger than the paper's framing suggests, and the external validity experiments (fine-tuning SOTA models on synthetic data) do not directly bridge this gap — they test whether SOTA models can learn the synthetic task and exhibit the same brittleness, not whether their pretrained CoT reasoning on real-world tasks is distribution-dependent. The theoretical framework (Theorem 3.1, the three-axis decomposition, the discrepancy measures) is under-tested — the qualitative predictions are confirmed, but the quantitative models are never fit to data or compared against alternatives. The absence of multi-axis shift experiments, non-CoT baselines, and real-world perturbation experiments leaves several natural questions unanswered. The paper's strongest claims ("CoT reasoning is a brittle mirage") outrun its strongest evidence (CoT reasoning is distribution-dependent in the DataAlchemy environment), and the difference between these claims is significant for the practical implications the paper draws.
6. Limitations and Trade-offs
The Gap Between Symbolic Abstraction and Natural Language Complexity
The assumption or constraint. DataAlchemy distills all NLP tasks into a 26-letter alphabet and three mathematical transformations (ROT, Cyclic Shift, Reversal). There is no semantic meaning, no ambiguity, no world knowledge, and no pragmatic inference — the reasoning tasks are purely compositional symbol manipulation. The paper explicitly acknowledges this in its Limitations section:
"While this abstraction enables full and fine-grained control over distribution factors and avoids data leakage, it may inevitably not fully capture the semantic richness, ambiguity, and compositional diversity present in natural language."
The consequence. The failure modes documented in DataAlchemy — pattern replication under compositional novelty, length rigidity under text length shift, template matching under format perturbation — may not be the failure modes that matter most in real-world CoT reasoning. A model that cannot generalize ROT-13 from f1∘f2 to f2∘f2 in a symbolic setting might nevertheless generalize from "solve two-step algebra problems" to "solve three-step algebra problems" in natural language, because semantic understanding provides additional structure (e.g., the meaning of "solve for x") that the symbolic abstraction strips away. Conversely, real-world CoT failures may involve factors — factual recall errors, pragmatic misinterpretation, ambiguity resolution — that DataAlchemy cannot model. The paper's central inference — that real-world CoT reasoning is distribution-dependent in the same way — assumes that DataAlchemy captures the essential structure of reasoning, an assumption the paper does not empirically validate.
What evidence exists in the paper. The external validity experiments (Section 8.2, Tables 5–7, Figure 9) partially address this gap by showing that SOTA models fine-tuned on DataAlchemy data exhibit the same brittleness as from-scratch models. However, this demonstrates that SOTA models can learn the symbolic task and exhibit brittleness on it — not that their pretrained CoT reasoning on natural language tasks is distribution-dependent. The paper does not conduct any experiment where a pretrained model's existing CoT capability (without fine-tuning) is tested under controlled distribution shifts on a real-world benchmark. The external validity transfer is therefore one-directional: DataAlchemy brittleness → SOTA brittleness on DataAlchemy tasks, but not DataAlchemy brittleness → SOTA brittleness on natural language tasks. The paper's own argument about data opacity (Section 1: "the training data distributions of commercial or large proprietary models remain uncovered") undermines the possibility of directly validating this inference.
Mitigation status. The paper acknowledges this as a limitation and suggests future work on more sophisticated abstractions, but does not provide a concrete path toward bridging the gap. The Limitations section frames this as the observation that "the observed brittleness of CoT reasoning under distribution shifts may manifest more stealthily, sophistically, and task-dependently in more complex real-world settings" — which concedes that the failure modes may be qualitatively different, not just quantitatively more subtle, in natural language. This is a fundamental limitation of the experimental paradigm rather than a fixable methodological issue: any abstraction that enables full control necessarily sacrifices some of the complexity that makes real-world reasoning challenging.
The Computational Cost of Difficulty Assessment Is Not Addressed (and Not the Point)
The assumption or constraint. The paper does not face the difficulty-estimation problem that affects deployment-oriented frameworks (like the compute-optimal test-time scaling paper), because DataAlchemy is an evaluation environment, not a deployment methodology. However, the paper draws practical implications for practitioners (Appendix G: "Guard against over-reliance and false confidence," "Prioritize OOD testing," "Recognize fine-tuning as a patch, not a panacea") that implicitly assume practitioners can determine the distribution discrepancy between their test queries and the model's training data. In the paper's controlled setting, this discrepancy is known by construction — you generated the training and test data, so you know exactly how they differ. In the real world, this discrepancy is unknown and, for proprietary models, unknowable.
The consequence. A practitioner reading this paper might conclude: "I should evaluate my CoT-based system under distribution shifts and only deploy it when the test distribution is sufficiently close to training." But the practitioner has no way to measure how close the test distribution is to the model's training distribution — the training data is proprietary and opaque. The data distribution lens provides a conceptual framework for understanding why CoT fails, but not a practical tool for predicting when it will fail on a specific deployment query. This is the diagnostic-vs-prognostic gap: the lens explains failures after they occur (by identifying which axis of distribution shift caused the degradation), but does not enable predicting failures before they occur (because the necessary distributional information is unavailable). The paper's practical recommendations (Appendix G) are therefore more aspirational than actionable — they tell practitioners what they should do (test for distribution shifts) without providing the tools to do it.
What evidence exists in the paper. The paper's entire experimental program relies on the fact that distribution discrepancy is known in DataAlchemy: the training and test distributions are constructed by the experimenter, so Δ(D_train, D_test) can be controlled and measured. The paper does not attempt to estimate distribution discrepancy from model outputs or from query features in a black-box setting. The theoretical framework (Theorem 3.1) requires Δ(D_train, D_test) as an input — it provides a bound on test risk given the discrepancy, not a method for estimating the discrepancy from data. The external validity experiments (Section 8.2) use the same DataAlchemy distributions with known discrepancies; they do not attempt to measure discrepancy for real-world queries.
Mitigation status. The paper does not acknowledge this as a limitation, likely because the research goal is understanding rather than deployment. The Limitations section focuses on the synthetic-vs-natural gap, distribution discrepancy measurement methodology (the mathematical definitions in Appendix C), and the scope of generalization dimensions — not on the practicality of applying the lens to black-box models. However, for a paper whose Discussion and Implication section offers advice to practitioners ("Sufficient auditing from domain experts is indispensable," "Practitioners must implement rigorous adversarial and OOD testing"), the lack of guidance on how to conduct such testing without known training distributions is a significant gap between the paper's framing and its actionable content.
Hard Problems Are Essentially Unsolved — and the Framework Offers No Path Forward
The assumption or constraint. The paper demonstrates that CoT reasoning fails under distribution shift, but it does not investigate whether there exists any training or prompting strategy that could make CoT reasoning robust to shift. The experimental design tests only one condition: standard supervised training on in-distribution data, evaluated on shifted data. The SFT experiments (Figures 4, 12) show that adding a small amount of shifted data "patches" performance, but this simply expands the in-distribution bubble — it does not teach the model to generalize to genuinely novel transformations.
The consequence. A practitioner who accepts the paper's conclusion that CoT reasoning is a "brittle mirage" is left without guidance on what to do instead. If CoT cannot handle distribution shift, should we abandon CoT entirely? Use it only in tightly controlled settings? Invest in fundamentally different architectures? The paper does not evaluate alternative training strategies (e.g., data augmentation, meta-learning, multi-task training across diverse distributions) that might produce more robust CoT reasoning. The finding that Group padding improves length generalization (Figure 13) hints that training distribution design can improve robustness, but this is a single data point rather than a systematic investigation. The paper's negative framing — "what appears to be structured reasoning can be a mirage" — is supported by the data, but the paper does not provide evidence that the mirage is inescapable for current architectures.
Specifically, the hardest problems in DataAlchemy — OOD transformation generalization (novel compositions of unseen transformations) and OOD element generalization (entirely novel atoms) — show near-zero performance regardless of model scale (Figures 3, 11; Tables 1, 5). The SFT experiments show that even targeted fine-tuning on OOD data requires substantial data to recover performance (Figure 4: λ ≈ 5 × 10^{-4} needed for CMP recovery, and OOD recovery is never achieved). This establishes a capability ceiling for the specific training paradigm tested, but does not establish that the ceiling is inherent to transformer architectures or to CoT reasoning as a concept.
What evidence exists in the paper. The SFT recovery curves (Figures 4, 12a) show that performance can be recovered by adding shifted data, but the recovery is distribution-specific: SFT on f2∘f2 enables generalization to f2∘f2 but not to f3∘f3. The element generalization SFT experiments (Figure 12a) show that recovery plateaus at low accuracy for edit distances n ≥ 3 even with substantial SFT data — the model cannot learn to handle elements structurally distant from training even with targeted fine-tuning. The reasoning step generalization experiments (Figure 6) show that the model can handle multiple depths when they are all present in training, but cannot extrapolate to unseen depths — again, the model interpolates but does not extrapolate. These results consistently demonstrate that the training paradigm (standard supervised learning on a fixed distribution) fails to produce transferable reasoning. But the paper does not test whether different training paradigms — curriculum learning, multi-task training across diverse distributions, meta-learning, or RL-based approaches — might succeed where standard supervised learning fails.
Mitigation status. Not addressed. The paper's Discussion section (Appendix G) recommends "OOD testing" and "auditing," but does not propose methods for improving OOD robustness. The paper's framing — CoT reasoning is a mirage — implicitly suggests that the problem is inherent to CoT rather than to the specific training setup, but this causal claim is not tested. The absence of any positive results on OOD generalization (all metrics collapse to near-zero for truly novel tasks) leaves open the question of whether the brittleness is a property of CoT reasoning or a property of the particular training distribution + architecture combination tested.
The Three-Axis Decomposition Is Not Shown to Be Complete, Additive, or Predictive
The assumption or constraint. The paper decomposes distribution discrepancy into three axes — task, length, and format — and states that Δ(D_train, D_test) = Φ(Δ_task, Δ_length, Δ_format) for some monotonically increasing composition function Φ (Equation 7). The experiments systematically vary one axis at a time while holding the others fixed (Sections 5–7), demonstrating that each axis individually affects performance. However, the paper never varies multiple axes simultaneously, never estimates Φ from data, and never tests whether the three-axis decomposition explains a sufficient fraction of the variance in CoT performance to serve as a predictive model.
The consequence. The paper's central theoretical claim — that the effectiveness of CoT reasoning is "fundamentally governed" by the three-axis distribution discrepancy — is supported only in a qualitative, within-experiment sense. We know that increasing task discrepancy degrades performance (holding length and format fixed), that increasing length discrepancy degrades performance (holding task and format fixed), and that increasing format discrepancy degrades performance (holding task and length fixed). We do not know whether these three axes jointly account for CoT performance in a multi-axis shift scenario, whether there are interaction effects (e.g., format perturbation might be more damaging when combined with task novelty than either alone would predict), or whether additional axes (e.g., transformation rate, element vocabulary diversity, compositional depth as distinct from text length) would capture significant additional variance.
The theoretical framework is therefore more of a taxonomy (three useful categories for thinking about distribution shift) than a model (a quantitative relationship between measured discrepancies and expected performance). This is a significant gap because the paper's practical recommendations (Appendix G: "Practitioners must implement rigorous adversarial and OOD testing that systematically probes for vulnerabilities across task, length, and format variations") imply that testing along these three axes is sufficient to characterize a system's robustness. If the three axes are incomplete — if there are important shift types not captured by task, length, or format — then testing along only these axes could give a false sense of security.
What evidence exists in the paper. The paper provides extensive single-axis evidence: Figures 3, 11, and Table 1 (task), Figures 5–6 and Table 4 (length), Figure 7 (format). The distinct shapes of the degradation curves across axes (sharp collapse for task, symmetric decay for length, smooth monotonic decay for format) are interpreted as evidence that the axes are genuinely independent. However, no experiment tests multi-axis shifts. The discrepancy measures themselves (T(C) in Proposition C.1, ε(L) in Proposition C.2, S(p_test) in Definition C.1) are proposed but never fit to data — the paper does not report, for example, how well the exponential failure model of Theorem C.1 fits the observed degradation in Figure 3, or how well the squared-exponential model of Proposition C.2 fits the length generalization curves in Figure 5. Without such validation, the discrepancy measures remain conceptual rather than operational.
Mitigation status. The paper treats Φ as a defined but unspecified monotonic function. There is no attempt to estimate it, to compare alternative functional forms, or to test its predictions. The Limitations section acknowledges that the paper "did not explicitly model other forms of distribution shift, such as cross-lingual transfer, multi-modal reasoning, or shifts in cultural context," but does not acknowledge the more fundamental limitation that the composition of the three modeled axes is untested. This is a methodological gap that could be addressed in future work by conducting factorial experiments (varying two or three axes orthogonally) and fitting multivariate models to the resulting performance surface.
The Revision Model Is Trained on Offline, Post-Hoc Pairs — Not Genuine Multi-Turn Rollouts
What the reader needs to know: This limitation applies to the reference example paper (the compute-optimal test-time scaling work), not to the DataAlchemy paper being analyzed. The DataAlchemy paper does not train a revision model — it trains models from scratch on fixed CoT traces and evaluates them without any iterative refinement. Therefore, this limitation should be omitted.
However, since the instructions require 4–6 limitations total and the DataAlchemy paper has been thoroughly analyzed above (with four substantial limitations identified), a fifth limitation drawn from the DataAlchemy paper's own design choices completes the set.
Training on Fixed CoT Traces May Induce Spurious Length and Format Correlations
The assumption or constraint. The DataAlchemy training data includes CoT reasoning traces as part of the target output: the model is trained to produce not just the final answer ê but the full chain e → e^{(1)} → ... → e^{(k-1)} → ê. These traces have a fixed structure determined by the training distribution — a specific number of reasoning steps, a specific output length, and a specific template format (e.g., the thinking and <answer> tokens). The model is never trained on incorrect reasoning traces or on traces of varying lengths/formats for the same task. The paper does not experiment with alternative trace formats or with training on mixed-format data.
The consequence. The observed brittleness under length and format shifts may be partly an artifact of the training setup rather than a fundamental property of CoT reasoning. The model learns that CoT traces have a specific length (equal to the number of compositional steps) and a specific format (the templated thinking ... <answer> structure), and when test queries violate these expectations, the model falls back on training-time statistics rather than adapting. This is evident in the qualitative failure modes: under text length shift, the model pads or truncates its output to match the training length (Appendix E.2.1); under format shift, it produces outputs whose length matches the two-step training pattern even when the test query specifies a single-step task (Appendix E.3). These failures suggest that the model has learned a template for CoT responses rather than a procedure for composing transformations. A model trained on more diverse CoT traces — with varying lengths, varying formats, and explicit demonstrations of length/format adaptation — might exhibit greater robustness. The paper does not test this hypothesis.
What evidence exists in the paper. The Group padding experiment (Figure 13) provides indirect evidence: when the model is exposed to a distribution of effective sequence lengths during training (via grouping and truncation), its length generalization improves substantially. This suggests that the brittleness observed under the None and Padding strategies is not an inherent limitation but a consequence of the narrow training distribution. Extending this logic, training on multiple CoT formats, multiple reasoning depths, and multiple element lengths might similarly improve robustness along the other axes. The paper's SFT experiments (Figures 4, 12) also support this interpretation: adding even a tiny fraction of shifted data enables generalization to that specific shift, confirming that the model can learn to handle distribution shifts when it sees relevant examples — it simply fails when those examples are absent from training.
Mitigation status. The paper does not directly address this limitation. The experimental design intentionally varies only one axis at a time during evaluation while keeping the training distribution fixed and narrow, which is necessary to isolate the causal effect of distribution shift. But this design choice also means that the paper cannot distinguish between "CoT reasoning is inherently brittle" and "CoT reasoning trained on a narrow, fixed-format distribution is brittle." The Discussion section (Appendix G) acknowledges that fine-tuning "simply expands the model's 'in-distribution' bubble slightly" rather than teaching genuine reasoning, but does not consider whether training on multiple distributions from the start would produce a model with a larger effective bubble — or whether a sufficiently diverse training distribution would enable genuine extrapolation. This is a distinct limitation from the four above because it concerns the training methodology rather than the evaluation framework, the generalization gap, or the theoretical model.
Single Model Family (When Controlling for Scale) and Single Task Paradigm
The assumption or constraint. All controlled experiments use decoder-only transformer architectures — specifically, GPT and LLaMA variants (Table 8). While the paper varies model size and architecture within this family, it does not test encoder-decoder models (T5, BART), mixture-of-experts architectures, retrieval-augmented models, or models with different positional encoding schemes (e.g., ALiBi, RoPE, learned absolute). The task paradigm is also singular: all experiments involve compositional transformations on a 26-letter alphabet. The paper does not test whether the same brittleness patterns emerge with different symbolic systems (e.g., arithmetic on digits, graph transformations, formal logic) or with non-symbolic reasoning (e.g., visual reasoning, spatial reasoning).
The consequence. The paper's claim that "the data distribution lens is invariant across LLMs with different sizes and architectures" (Section 8.1, Figure 8 caption) is supported for the architectures and sizes tested but may not generalize to fundamentally different model classes. Encoder-decoder models, for instance, process input and output through different mechanisms, and their length generalization behavior might differ from decoder-only models (which must handle variable-length input and output through the same autoregressive mechanism). Models with relative positional encodings (RoPE, ALiBi) are known to exhibit better length generalization than models with absolute positional encodings, and the paper's finding that length generalization is highly brittle (Figure 5) might be specific to the absolute positional encoding used in the tested architectures. Similarly, the finding that CoT fails under compositional task shifts (Table 1) might be specific to the ROT/Cyclic-Shift transformation space — other symbolic systems with different algebraic structures (e.g., group operations, arithmetic, string rewriting) might produce different generalization patterns.
What evidence exists in the paper. The paper tests GPT and LLaMA architectures across a wide size range (62K to 3B), which is more architectural diversity than most controlled studies provide. The external validity experiments (Section 8.2) extend this to LLaMA3-8B and Qwen3-14B-Instruct, which use RoPE (LLaMA3) and likely a variant of RoPE (Qwen3). However, these are all decoder-only transformer architectures. The paper does not test encoder-decoder models, and it does not systematically vary positional encoding schemes — the GPT and LLaMA variants tested use absolute and possibly RoPE-style encodings, but this is not a controlled variable in the experimental design. The impact of positional encoding on length generalization is particularly relevant because the paper's own heuristic derivation of the squared-exponential length decay (Appendix C.2) explicitly references positional encodings as a mechanism: "positional encodings for positions i > L_train have never been observed." This implies that alternative encoding schemes might produce different length generalization curves — a hypothesis the paper does not test.
Mitigation status. The Limitations section does not mention architecture or task paradigm diversity as a limitation. The paper frames the GPT/LLaMA variation as demonstrating "internal validity" and the SOTA model experiments as demonstrating "external validity," which implicitly claims sufficient architectural coverage. However, the absence of encoder-decoder models and alternative symbolic domains means that the scope of the invariance claim is narrower than the paper's language suggests. Future work could test the data distribution lens on a broader set of architectures and symbolic systems to determine whether the observed brittleness is a property of autoregressive transformers, of the specific symbolic system used, or of CoT reasoning in general.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper does not propose a new prompting method, a new architecture, or a new training objective. It offers something more foundational: a scientific instrument and a falsifiable lens for studying whether LLMs actually reason. That distinction matters because the CoT literature has been trapped in an unresolvable cycle — one paper shows CoT succeeds on benchmark X, another shows it fails on benchmark Y, and neither can explain why because the training data, task structure, and evaluation conditions are all entangled. DataAlchemy breaks that cycle by making every variable controllable and every distribution known, converting CoT evaluation from observational data collection into experimental science.
The shift is methodological rather than technical, but its implications are practical. Before this work, a researcher who wanted to ask "does my model genuinely reason or just pattern-match?" had no rigorous way to answer. They could run benchmarks, but contamination was always a confound. They could do intervention studies (corrupt reasoning steps, see if answers change), but those interventions were themselves model-generated and hard to control. DataAlchemy provides a template: abstract the task to its combinatorial skeleton, train from scratch on known data, systematically vary the distribution, and measure not just whether the model succeeds but how — separately scoring reasoning traces and final answers to diagnose faithfulness.
The paper also resolves a surface-level contradiction that has fractured the CoT literature. Why do Stechly et al. (2024) find that CoT fails catastrophically on planning while Wei et al. (2022) find it succeeds on arithmetic? The data distribution lens provides a unified answer: the planning tasks introduced a larger distribution discrepancy between training and test than the arithmetic tasks did. Both results are manifestations of the same underlying mechanism — CoT performance is bounded by distributional alignment — rather than evidence of fundamentally different phenomena. The apparent conflict dissolves when you recognize that "task difficulty" is not a scalar but a vector of discrepancies along multiple axes. What Mirzadeh et al. (2025) call "fragility to irrelevant perturbations" and what Lanham et al. (2023) call "unfaithful reasoning" are both predicted by Theorem 3.1: as Δ_format increases (perturbations shift the prompt away from training templates), the bound on test risk loosens, and performance degrades — with the degradation manifesting as plausible-looking but logically decoupled reasoning traces because the model falls back on training-time statistical patterns.
The most uncomfortable implication is for scale-as-solution narratives. The paper's evidence that model size (from 62K to 14B parameters) does not qualitatively change the relationship between distribution shift and CoT performance (Figures 8, 15; Tables 5–7) directly challenges the assumption that larger models will naturally develop robust reasoning. Scale amplifies in-distribution performance — larger models have bigger "in-distribution bubbles" and can interpolate across wider gaps — but it does not create extrapolation. A 14B-parameter model collapses to 0% exact match on OOD transformation tasks just as completely as a 62K model (Table 5). If this pattern holds at the 100B+ scale of frontier models (an open question the paper cannot answer), then the path to genuinely robust reasoning is not "train bigger models on more internet text" but rather "train on distributions that cover the space of possible queries" — a data engineering challenge, not a scaling challenge.
This reframes the research landscape in concrete ways. More attractive now: controlled synthetic experiments that isolate causal factors (following the DataAlchemy template), work on training distribution design and coverage (what data mix produces the widest generalization envelope?), and verifier/faithfulness diagnostics that separately measure reasoning quality and answer correctness. Less attractive: pure benchmark-chasing on contaminated test sets (the paper argues this is scientifically uninformative regardless of the scores achieved), prompting tricks that improve in-distribution performance without addressing distribution-shift robustness, and capability claims based solely on final-answer accuracy without faithfulness analysis. The paper does not argue that CoT prompting should be abandoned — it argues that CoT prompting's effectiveness should be understood as distribution-dependent pattern interpolation rather than emergent logical inference, and that evaluations should reflect this understanding.
Follow-Up Research This Work Enables
Directly testing whether frontier-model CoT reasoning is distribution-dependent on real-world tasks. The paper's central limitation is the gap between DataAlchemy's symbolic abstractions and natural language reasoning. A direct extension would construct a semi-synthetic benchmark: take a subset of GSM8K or MATH problems, systematically perturb them along the three axes (rephrase to vary format, add/remove reasoning steps to vary length, construct compositional variants that combine operations in novel ways), and evaluate frontier models (GPT-4, Claude, Gemini) without fine-tuning on the perturbed versions. The paper predicts monotonic degradation with perturbation magnitude, but this has never been tested. A positive result (frontier models are robust to these perturbations) would bound the scale at which extrapolation emerges; a negative result (frontier models collapse similarly to DataAlchemy models) would directly validate the paper's central claim in a natural-language setting and have immediate implications for deployment safety.
Multi-axis shift experiments to test whether Φ is additive, multiplicative, or something else. The paper decomposes distribution discrepancy into three axes and states that they compose through a monotonic function Φ, but never varies two axes simultaneously. A natural factorial experiment would orthogonally vary task novelty (ID, CMP, POOD, OOD) and format noise level (0%, 10%, 20%, 30%) for a fixed model and measure whether the combined degradation is predicted by the sum, product, or maximum of the single-axis degradations. If the degradation is super-additive (format perturbation hurts more under task novelty than under ID), that would reveal an interaction effect — the model's template-matching failure compounds with its composition failure — which would have practical implications for robustness testing (you cannot assess axes independently). If the degradation is sub-additive (the model saturates at poor performance and additional axes don't make things worse), that would suggest a floor effect rather than a true interaction.
Systematic comparison of positional encoding schemes for length generalization. The paper's heuristic derivation of the squared-exponential length decay (Appendix C.2) explicitly references positional encodings as the mechanism: "positional encodings for positions i > L_train have never been observed." This makes a testable prediction: models with relative positional encodings (RoPE, ALiBi) should exhibit better length generalization than models with absolute positional encodings, and the squared-exponential width parameter σ should be larger for relative encoding schemes. A controlled experiment would train GPT-style models with identical architectures except for their positional encoding scheme (absolute learned, sinusoidal, RoPE, ALiBi, NoPE) on DataAlchemy tasks at a fixed element length, evaluate on a range of lengths, and fit the squared-exponential model to each curve to estimate σ. If RoPE/ALiBi models show substantially larger σ (slower degradation with length extrapolation), that would validate the mechanistic role of positional encodings and provide concrete guidance for practitioners (use RoPE if you expect variable-length queries). If all schemes show similar σ, the bottleneck lies elsewhere (e.g., attention pattern calibration rather than encoding novelty).
Can training on diverse CoT formats produce a model that genuinely composes transformations rather than matching templates? The paper shows that narrow-format training produces brittle template matching (Appendix E.3), and that Group padding (which exposes models to a distribution of lengths) improves length generalization (Figure 13). This suggests a more general hypothesis: training on diverse CoT traces — with varying numbers of reasoning steps, varying output formats, varying prompt templates, and explicit demonstrations of length/format adaptation — might produce a model that learns the underlying transformation composition rules rather than surface-level templates. A strong test would train models on DataAlchemy tasks where every transformation composition appears in multiple formats, multiple reasoning depths, and multiple element configurations, then evaluate on completely held-out compositions (e.g., f3∘f3 when only f1 and f2 compositions were trained). If the model succeeds on held-out compositions — something the current paper's narrow-training models fail at completely (Table 3) — that would demonstrate that diversity, not just coverage, is the key to generalization. Negative results (even diverse-format models fail on novel compositions) would strengthen the paper's claim that current architectures fundamentally cannot extrapolate.
Faithfulness diagnosis for SOTA models on real-world CoT traces. The paper's faithfulness analysis — separately measuring reasoning and answer correctness and identifying commutativity-induced coincidences (Table 2) — is one of its most elegant contributions, but it is only demonstrated in the DataAlchemy setting where ground-truth reasoning traces are known. Extending this diagnostic to real-world tasks requires constructing problems where the reasoning trace is verifiably and uniquely correct, independent of the final answer. Mathematical proof problems (where the proof steps can be checked by a symbolic verifier), algorithmic reasoning (where intermediate states are uniquely determined), and code execution traces (where intermediate variable values can be checked) are natural candidates. A follow-up study would take a SOTA model, prompt it for CoT reasoning on problems with verifiable intermediate steps, and apply the DataAlchemy faithfulness analysis: separately score reasoning vs. answer, identify cases of correct-answer-with-incorrect-reasoning and incorrect-answer-with-correct-reasoning, and report the frequency of each failure mode as a function of problem difficulty (measured by, e.g., proof step count or algorithmic depth). If the commutativity-coincidence pattern (correct answers from wrong reasoning) occurs at non-trivial rates in real-world settings, this would directly validate the paper's practical warning that "the presence of a detailed reasoning trace should not be equated with correctness."
The DataAlchemy platform as a benchmark suite. The paper releases DataAlchemy as open-source code, but it is not packaged as a standardized benchmark with fixed train/test splits, leaderboard metrics, and difficulty tiers. A natural follow-up would develop DataAlchemy-Bench: a versioned benchmark with predefined distribution-shift scenarios (ID, CMP-length-2, CMP-length-3, POOD-format-10%, OOD-task-f3, etc.), standardized evaluation protocols (train from scratch vs. fine-tune pretrained model), and leaderboard metrics that report not just accuracy but faithfulness scores (reasoning/answer correlation) and degradation slopes (how quickly performance drops with increasing Δ along each axis). This would enable the community to compare architectures, training recipes, and prompting strategies on a single, controlled, contamination-free platform — something that currently does not exist for reasoning evaluation. The benchmark could also track the "generalization frontier": as models improve, harder distribution-shift tiers can be added by increasing alphabet size, transformation complexity, or compositional depth, making it a moving-target benchmark that resists saturation.
Practical Applications and Downstream Use Cases
Safety auditing for deployed CoT systems. A team deploying an LLM with CoT prompting in a high-stakes domain (medical diagnosis support, legal document analysis, financial forecasting) can use the paper's three-axis decomposition as a pre-deployment audit checklist. For each axis, construct a perturbation suite: task shifts (rephrase the query to use novel compositions of the same concepts), length shifts (add or remove contextual detail to vary input length), and format shifts (introduce typos, change template wording, swap instruction order). Measure the degradation curve — does accuracy hold stable, degrade smoothly, or collapse at a threshold? The paper's results predict that systems trained on narrow distributions will show threshold collapse, and the specific perturbation level at which collapse occurs provides a quantitative safety margin. If a medical CoT system maintains >95% accuracy up to 20% format noise but collapses to 60% at 25%, that tells the deployment team exactly how much input variability the system can tolerate before human review becomes mandatory. This is actionable today using the paper's methodology, applied to the specific domain and model at hand.
Training data curation for fine-tuned reasoning models. Organizations that fine-tune LLMs for domain-specific reasoning (e.g., a legal tech company training a model to produce case law analysis with CoT reasoning) can use the data distribution lens to design their fine-tuning datasets. The paper's finding that model performance on a given reasoning depth is directly proportional to the fraction of training data at that depth (Figure 6) implies that if your deployment will involve variable-length reasoning chains, you must explicitly include all expected depths in the training mix — the model will not extrapolate from 2-step training to 3-step test queries. Similarly, the finding that element generalization is highly local (Figure 12a: recovery plateaus at low accuracy for edit distances n ≥ 3) implies that if your domain vocabulary has combinatorial structure, you need training examples that explicitly span the vocabulary space rather than expecting the model to generalize from a few seed examples. The SFT recovery curves (Figures 4, 12) provide concrete guidance on how much OOD data is needed: even λ = 1.5 × 10^{-4} (0.015% of training data) substantially improves generalization to nearby distributions, suggesting that a small investment in diverse annotation can yield disproportionate robustness gains.
Synthetic data generation for self-improvement pipelines. In self-improvement loops where LLMs generate their own training data (e.g., STaR, ReST^EM, rejection sampling fine-tuning), the quality and diversity of generated data determine the model's eventual generalization envelope. The paper's framework suggests a concrete diversity metric: measure the task complexity score T(C) (Proposition C.1), the length distribution, and the format variation of the generated data, and compare against the expected deployment distribution. If the generated data's T(C) distribution has zero mass beyond a certain novelty threshold, the model trained on that data will fail on test queries exceeding that threshold — the SFT experiments (Figure 4) show that performance collapses when the test distribution contains transformations not represented in training. This provides a principled stopping criterion for self-improvement loops: continue generating and training until the synthetic data's distribution covers the target deployment distribution along all three axes. The paper's finding that format perturbations to instruction tokens are less damaging than perturbations to content tokens (Figure 7, right panel) also suggests a data augmentation strategy: deliberately vary instruction phrasing during generation to improve format robustness without distorting task content.
