Textbooks Are All You Need

ArXiv: 2306.11644

🎯 Pitch

A 1.3B-parameter model trained on just 7B tokens of textbook-quality code—not the billion-token web dumps used by models 10× its size—matches GPT-3.5-level coding and beats StarCoder by 17 points on HumanEval, proving that data quality can shatter conventional scaling laws. The secret is a tiny finetuning stage on synthetic exercises that unlocks emergent abilities like library use and reasoning, which the base model never explicitly learned.

1. Executive Summary

This paper studies how training data quality—rather than model or dataset scale—shapes the coding proficiency of large language models, introducing phi-1, a 1.3B-parameter Transformer trained on 7B tokens of "textbook quality" data. The central mechanism is the deliberate curation of a CodeTextbook dataset (filtered web code plus synthetically generated Python textbooks from GPT-3.5, emphasizing clear, self-contained, instructive examples) followed by finetuning on a small CodeExercises dataset (180M tokens of synthetic function-completion exercises), which together yield pass@1 accuracy of 50.6% on HumanEval and 55.5% on MBPP—outperforming models trained on orders of magnitude more data, including StarCoder (15.5B parameters, 1T tokens) and matching GPT-3.5's reported 47% on HumanEval. The finetuning stage unlocks emergent capabilities absent in the base model—including library usage (PyGame, Tkinter) and chat-like reasoning—establishing that textbook-quality data can dramatically reshape scaling laws, but only when the finetuning corpus, despite its small size, forces the model to reorganize knowledge already acquired during pretraining.

2. Context and Motivation

The Core Problem: Scaling Laws Told an Incomplete Story

The paper enters the conversation at a moment when the dominant narrative in large language model research was organized around scale. Beginning with the empirical observation that neural network performance improves predictably with increases in compute and model size (Hestness et al., 2017), and crystallized by the formal scaling laws of Kaplan et al. (2020), the field had largely converged on a formula: invest more compute, train a bigger model on more data, and performance will follow. The subsequent landmark models—GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), Chinchilla (Hoffmann et al., 2022)—all validated this principle, with each generation pushing parameter counts into the hundreds of billions and training datasets into the trillions of tokens.

This scaling-centric view produced a set of beliefs that the paper directly challenges. The conventional wisdom held that to reach competitive performance on reasoning-heavy tasks like code generation, a model needed to be large—not merely for memorization capacity, but because the implicit assumption was that the signal-to-noise ratio in web-scale training data was fundamentally low, and that massive scale was necessary to extract that signal. The field's attention was therefore focused on scaling the compute budget and the model architecture, with data treated as a relatively uniform resource to be gathered in ever-larger quantities from the web.

The paper identifies a fundamental gap in this picture: the scaling laws literature had systematically varied model size, dataset size, and compute, but had not systematically investigated what happens when data quality itself is treated as an independent variable. The authors state this explicitly in their opening:

"In this work, following the footsteps of Eldan and Li [EL23], we explore the improvement that can be obtained along a different axis: the quality of the data."

This is not merely an observation that "clean data helps." The paper's claim is stronger: that data quality improvements can change the shape of the scaling laws themselves, allowing small models trained on small, high-quality datasets to match or exceed large models trained on orders of magnitude more data. If true, this would mean that the scaling laws the field had been operating under were not fundamental properties of learning from language data, but rather artifacts of the mediocre quality of the web-scale datasets on which they were empirically measured.

Why This Problem Matters: Environmental, Economic, and Scientific Stakes

The dominance of the scaling paradigm carries concrete costs that the paper explicitly invokes. Training large language models is extraordinarily resource-intensive: GPT-3's training was estimated to emit over 500 tons of CO₂, and subsequent models have only grown larger. The environmental impact of this trajectory—what Bender et al. (2021) termed the "stochastic parrots" problem—creates an urgent need for more efficient training paradigms. The paper positions itself directly in this conversation:

"Importantly, smaller models requiring less training can significantly reduce the environmental cost of LLMs."

Beyond environmental concerns, the scaling paradigm imposes economic barriers to entry. When state-of-the-art performance requires hundreds of billions of parameters and trillions of training tokens, only organizations with massive compute budgets can participate. A 1.3B-parameter model trained for 4 days on 8 A100s (as phi-1 is) represents a qualitatively different accessibility threshold than a 540B-parameter model trained on thousands of TPUs for months. If data quality can substitute for scale, it democratizes the ability to build capable models, enabling smaller research labs, startups, and academic groups to contribute meaningfully.

There is also a scientific motivation that transcends practical concerns. The paper's core hypothesis—that high-quality, textbook-like data enables fundamentally more efficient learning—touches on a deep question about how neural networks acquire reasoning capabilities. The dominant pretraining paradigm (scraping vast quantities of web text and hoping that reasoning emerges from scale) is, from a pedagogical standpoint, bizarre: no human educator would teach programming by having students read millions of random, decontextualized code snippets from GitHub repositories of wildly varying quality. The paper asks, in effect: what if we trained models the way we teach humans—with clear, self-contained, progressively structured examples? The fact that this approach yields such dramatic efficiency gains suggests that the field's default training data pipelines may be profoundly suboptimal from a learning-theoretic perspective, and that better understanding how data organization affects learning is a fundamental research question.

Where Prior Approaches Fall Short

The paper identifies specific, concrete limitations in standard code training datasets—specifically The Stack (Kocetkov et al., 2022) and StackOverflow—that motivate the need for a fundamentally different approach to data curation. These are not vague complaints about "noise"; the authors enumerate four specific failure modes based on manual inspection of random samples (Section 2):

1. Non-self-contained snippets. Many code files in The Stack depend on external modules, imports, or files not present in the training sample. For a human learner, encountering a function that calls an undefined helper or imports a project-specific module is frustrating and uninformative. For a language model trained with next-token prediction, such samples provide ambiguous signal: the model cannot learn the relationship between the function's docstring and its implementation if critical dependencies are missing. The model may learn surface-level patterns (function definition syntax, common variable names) but cannot learn the deeper mapping from intent to implementation that constitutes genuine coding proficiency.

2. Trivial or boilerplate code. A substantial fraction of code in web repositories consists of configuration files, constant definitions, GUI setup code, or other boilerplate that performs no meaningful computation. These samples are syntactically valid but algorithmically vacuous. When they dominate the training distribution, the model spends the majority of its learning budget on patterns that contribute nothing to reasoning ability—it learns to generate import statements and class constructor boilerplate flawlessly while never internalizing the logical structures (loops, conditionals, algorithmic transformations) that distinguish code generation from text generation.

3. Algorithmic logic buried in complexity. The samples that do contain meaningful computation are often embedded in large, poorly documented functions within complex codebases. Extracting the algorithmic insight requires understanding extensive surrounding context. For a language model processing fixed-length context windows, the relevant signal is diluted by surrounding noise. The model may see hundreds of examples of list manipulation without ever encountering a clean, self-contained demonstration of, say, finding the closest pair of elements in a sorted list.

4. Skewed topic distribution. Web code repositories are not a balanced curriculum. Certain domains (web development frameworks, data science utilities) are massively overrepresented, while others (algorithmic problem-solving, mathematical reasoning through code) are underrepresented. The resulting training distribution teaches the model to be an expert in the most common coding patterns while leaving it weak on precisely the kinds of reasoning that benchmarks like HumanEval test.

These limitations are not incidental—they are structural features of how code exists on the web. Code repositories are built for software engineering, not pedagogy. They optimize for functionality and maintainability, not for teaching fundamental concepts to a learner. The paper's central insight is that treating such repositories as a training corpus for language models conflates two very different goals: building working software versus learning to reason about programs.

The Prior Art That This Paper Builds On

The paper is not operating in a vacuum. It draws on several strands of prior work that, together, set the stage for its contribution.

TinyStories and the data quality axis. The most direct intellectual precursor is Eldan and Li (2023)'s TinyStories, which demonstrated that a synthetically generated dataset of simple, structured children's stories—designed to teach English in a progressive, textbook-like manner—enabled very small language models (on the order of 10M parameters) to generate coherent, grammatically correct English. This was surprising because conventional wisdom held that coherent language generation required models at the scale of hundreds of millions of parameters trained on web-scale data. TinyStories showed that data quality could dramatically reshape the scaling curve, allowing small models to achieve capabilities previously thought to require much larger scale. The paper explicitly positions itself as extending this program from natural language storytelling to code generation, a domain where the gap between "textbook-quality" instruction and web-scale data is arguably even wider.

LLMs for program synthesis. The paper enters a well-established literature on using large language models for code generation. Codex (Chen et al., 2021) established the HumanEval benchmark and demonstrated that scaling up language models on GitHub code could yield impressive program synthesis capabilities, achieving 28.8% pass@1 with a 12B-parameter model. Subsequent models—CodeGen (Nijkamp et al., 2022, 2023), PaLM-Coder (Chowdhery et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023)—all followed the same basic formula: collect more code from the web, train larger models, and watch performance improve. Table 1 in the paper makes the contrast stark: CodeGen-Mono-16.1B was trained on 577B tokens and achieved 29.3% on HumanEval; StarCoder at 15.5B parameters trained on over 1T tokens reached 33.6%. These models represent the scaling paradigm at work: each generation invested more parameters and more data to achieve incremental improvements.

Synthetic data generation and recursive training. The paper's methodology—using GPT-3.5 to generate training data for phi-1—is part of an emerging trend of using large language models to train other language models. Self-Instruct (Wang et al., 2022), Alpaca (Taori et al., 2023), and Orca (Mukherjee et al., 2023) all used GPT-3.5 or GPT-4 to generate instruction-following data for fine-tuning smaller models. The paper acknowledges an ongoing debate in this literature about whether "recursive training" leads to model collapse or narrowing (Shumailov et al., 2023; Gudibande et al., 2023), but positions itself on the side of the debate that sees synthetic data as potentially enabling better performance than the teacher model on specific, narrow tasks—a phenomenon that Jung et al. (2023) call "impossible distillation." The key distinction the paper makes is focus: by targeting a narrow domain (Python function completion) with carefully constrained generation (topic constraints, function name constraints to elicit diversity), the synthetic data can be higher quality for the specific task than the teacher model's general-purpose training data.

Data quality as a known but underexplored lever. The paper acknowledges that data quality's importance is not a new discovery. Data cleaning is standard practice in modern dataset creation (Raffel et al., 2020), and prior work had shown that higher-quality data could enable training on smaller datasets (Longpre et al., 2023; Yu et al., 2023) or more passes over the same data without overfitting (Muennighoff et al., 2023). The paper's contribution is not the observation that quality matters, but rather the demonstration that quality improvements can produce a qualitative regime change—not just a constant-factor improvement but a reshaping of the performance-scale relationship that allows small models to compete with much larger ones.

How This Paper Positions Itself

The paper's positioning can be understood along three dimensions:

Against the scaling laws paradigm. The paper does not claim that scaling laws are wrong, but rather that they are incomplete. The axis of "data quality" was not systematically explored in the foundational scaling laws work because those studies used fixed, web-scale datasets and varied model size and training duration. The paper's experiments in Figure 2.1 are designed to directly illustrate the contrast: within a single model size (350M or 1.3B parameters), switching from the standard Stack dataset to the CodeTextbook dataset produces gains comparable to or exceeding those from scaling model size. The implicit argument is that the field's resource allocation—investing in larger models and more data rather than better data—may be suboptimal, and that future scaling laws must incorporate data quality as a first-class variable alongside model size and token count.

Within the synthetic data literature. The paper distinguishes itself from approaches like Alpaca that use synthetic data to mimic a teacher model's general capabilities. Instead, phi-1 uses synthetic data to create a curriculum—structured, textbook-like content that teaches fundamental concepts more effectively than any naturally occurring data source. This is closer in spirit to TinyStories than to instruction-tuning approaches: the goal is not to distill GPT-3.5's behavior but to create a pedagogical environment that enables efficient learning. The paper's emphasis on diversity through constrained generation—using topic constraints and function name constraints to force the teacher model to explore different regions of concept space—is a concrete methodological contribution to how synthetic data should be generated for curriculum learning.

On the specialization-vs-generality spectrum. The paper is notably candid about phi-1's narrowness. It is "specialized in Python coding" (Section 6), and the authors explicitly acknowledge limitations in domain-specific knowledge, robustness to stylistic variations, and performance on tasks requiring broader world knowledge. Rather than positioning phi-1 as a general-purpose model, the paper argues that specialization enabled by high-quality data is a legitimate and underexplored point in the design space. This is significant because much of the LLM literature has pursued generality as the primary objective; the paper suggests that for many practical applications, a small, specialized model trained on high-quality data may be more useful than a large, general model.

The Unanswered Question Driving the Paper

The paper is ultimately motivated by a question that, after years of scaling-focused research, remained unresolved: can data quality substitute for scale? The scaling laws literature had established that more data and more parameters predictably improve performance, but it had not established whether those improvements were due to increased quantity of information or increased quality of the signal extracted from that information. If the latter—if web-scale data is inherently noisy and models spend most of their capacity learning to ignore irrelevant patterns—then improving data quality could yield efficiency gains far beyond what scaling laws would predict. The paper's experiments are designed to test exactly this hypothesis, and its central result—that a 1.3B-parameter model trained on 7B tokens can match or exceed 15B-parameter models trained on trillions of tokens—suggests that the hypothesis is correct, and that the field's understanding of scaling has been fundamentally limited by its focus on quantity over quality.

3. Technical Approach

3.1 Reader Orientation

The system being built is a data curation and training pipeline that produces a code-generation language model from a small, deliberately constructed dataset rather than from web-scale code repositories. It solves the problem that existing code LLMs require enormous scale—hundreds of billions of parameters and trillions of training tokens—by demonstrating that when training data is structured like a textbook (clear, self-contained, instructive, and balanced), a 1.3B-parameter model can match or exceed 15B-parameter models trained on 100× more data. The shape of the solution is a two-stage process: first, pretraining on a mixture of filtered high-quality web code and synthetically generated Python textbooks (CodeTextbook, ~7B tokens) to build foundational coding knowledge; second, finetuning on a small set of synthetic coding exercises (CodeExercises, ~180M tokens) that reorganizes and consolidates this knowledge, unlocking capabilities—library usage, algorithmic reasoning, instruction following—that the pretrained model does not reliably exhibit.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a sequential pipeline:

1. Web Code Filter — takes the raw Stack and StackOverflow datasets (~35M files, ~35B tokens) and produces a filtered subset (~6B tokens) by scoring each file's educational value using a GPT-4-annotated random forest classifier. This removes non-self-contained, trivial, or poorly documented code.

2. Synthetic Textbook Generator — uses GPT-3.5 with diversity-inducing topic and audience constraints to produce <1B tokens of Python textbook content: natural language explanations interleaved with relevant, self-contained code examples covering reasoning and algorithmic concepts.

3. Synthetic Exercise Generator — uses GPT-3.5 with function-name constraints to produce ~180M tokens of Python function-completion exercises (docstring → implementation pairs), designed to align the model with the HumanEval-style task format.

4. Pretraining Engine — trains a 1.3B-parameter decoder-only Transformer (phi-1-base) on the combined CodeTextbook dataset (filtered web code + synthetic textbooks) using standard next-token prediction for ~50B total tokens seen (~8 epochs), producing a model that achieves 29% on HumanEval before any task-specific finetuning.

5. Finetuning Engine — further trains phi-1-base on the CodeExercises dataset using the same next-token prediction objective but with different hyperparameters, producing phi-1, which exhibits substantially improved HumanEval performance (50.6%) and emergent capabilities not present in the base model.

Information flows strictly left to right: raw web data → filter → CodeTextbook → pretraining → phi-1-base → finetuning on CodeExercises → phi-1. The synthetic data generators (components 2 and 3) operate in parallel to the filter, feeding into the same training pipeline.

3.3 Roadmap for the Deep Dive

• First, the web code filtering mechanism (Section 2.1), because it establishes the baseline philosophy—"would this help a student learn?"—and provides the largest component (6B of 7B tokens) of the pretraining corpus. Understanding how the filter works is prerequisite to understanding what kinds of code patterns phi-1 sees during pretraining.

• Second, the synthetic textbook generation process (Section 2.2), because it introduces the core methodological contribution: using diversity-inducing constraints to make a teacher LLM produce varied, curriculum-like content. The textbook data provides the natural-language-heavy, concept-structured counterpart to the filtered code.

• Third, the synthetic exercise generation process (also Section 2.2), because it is the smallest dataset (180M tokens) yet produces the largest performance jump (29% → 50.6% on HumanEval). Understanding the exercise format and diversity mechanism explains why finetuning generalizes beyond the finetuning distribution.

• Fourth, the model architecture and pretraining procedure (Section 2.3), because these are deliberately conventional—the paper's thesis is that data quality, not architectural innovation, drives the results. Knowing the architecture establishes a baseline for interpreting the ablation experiments in Figure 2.1.

• Fifth, the finetuning procedure (also Section 2.3), because the hyperparameter changes and the relationship between pretraining and finetuning data volumes (~50B tokens seen vs. ~1.5B tokens seen) are critical to understanding the emergence phenomenon described in Section 3.

• Sixth, the decontamination and evaluation methodology (Sections 4–5), because the paper anticipates and addresses the natural skepticism that a small model trained on synthetic data might be memorizing benchmark problems rather than learning generalizable coding skills.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering and empirical analysis paper whose core idea is that deliberately structuring training data to mimic pedagogical materials—self-contained, progressively organized, concept-focused—produces qualitatively more efficient learning than training on larger volumes of uncurated web data.

Web Code Filtering via GPT-4-Annotated Classifier

The first stage of the data pipeline addresses a concrete problem: the standard Python code datasets used for training code LLMs—specifically the deduplicated Python subset of The Stack (Kocetkov et al., 2022) and StackOverflow—contain over 35 million files totaling over 35B tokens, but manual inspection reveals that most of these files are not instructive for learning programming fundamentals. The authors identify four specific failure modes (Section 2): non-self-contained snippets that depend on external modules not present in the context, trivial or boilerplate code (configuration constants, GUI setup) that contains no algorithmic reasoning, algorithmic logic buried inside large undocumented functions that dilute the signal, and skewed topic distributions that overrepresent certain domains while underrepresenting fundamental concepts.

The filtering solution proceeds in three steps: annotation, feature extraction, and classifier training.

Annotation with GPT-4. The authors sample approximately 100,000 code files from the combined Stack + StackOverflow corpus and prompt GPT-4 to "determine its educational value for a student whose goal is to learn basic coding concepts." This produces a labeled dataset where each sample has a quality judgment grounded in a specific pedagogical criterion: whether the code would help a novice programmer understand fundamental concepts. The authors explicitly note that GPT-4 is used only for this annotation step on a small subset, not for the full filtering, making the approach scalable:

"We note that unlike GPT-3.5, which we use extensively to generate synthetic content (discussed below), we use GPT-4 minimally only for annotations on the quality of a small subset of The Stack and StackOverflow samples. We thus view our usage of GPT-4 as merely a way to avoid tedious human-annotation efforts."

This design choice—using a stronger model (GPT-4) for annotation but a weaker model (GPT-3.5) for generation—is deliberate: annotation quality directly impacts the classifier's ability to distinguish educational from non-educational code, while generation volume matters more for synthetic data diversity.

Feature extraction via pretrained CodeGen embeddings. For each code file, the system computes an embedding vector using a pretrained CodeGen-Mono 350M model (Nijkamp et al., 2023). The embedding captures semantic properties of the code—what the code does, how it is structured, what patterns it uses—in a fixed-dimensional vector space. This is the standard approach of using a pretrained model as a feature extractor: rather than hand-engineering features (which would be infeasible for code), the system leverages a model already trained to understand code semantics. The CodeGen-Mono 350M model was trained on 577B tokens of code, giving it substantial knowledge of code structure that can be transferred to the classification task.

Random forest classifier training. The authors train a random forest classifier that maps each CodeGen embedding vector to a binary (or quality-scored) prediction of educational value. A random forest is an ensemble of decision trees, each trained on a random subset of features and samples; the ensemble aggregates votes to produce a final prediction. The choice of a random forest rather than a neural classifier is pragmatic: random forests are fast to train on 100K examples, require minimal hyperparameter tuning, and produce well-calibrated outputs without the risk of overfitting that a small neural network might exhibit on limited annotation data. The trained classifier is then applied to the entire 35M-file corpus, retaining only those files predicted to have high educational value, resulting in approximately 6B tokens of filtered code.

What the filter selects for. The paper provides illustrative examples (Section 2.1) of code with "high educational value" versus "low educational value." The high-value example shows self-contained utility functions (normalize, euclidean_dist, cosine_dist) that implement clear mathematical operations with docstrings, explicit imports, and no external dependencies. The low-value example shows a large, complex class definition for a Vim plugin with extensive internal state, relying on project-specific modules (Nvim, SyncParent) and performing no clearly teachable algorithmic operation. These examples operationalize the "textbook quality" criterion: the filter selects code that a human learner could read in isolation and understand both what it does and how it does it.

Empirical validation of filtering. The paper reports a concrete ablation (Section 2.1, within the text describing the filter): a 350M-parameter model trained on unfiltered Stack + StackOverflow achieves 12.19% on HumanEval after 96K steps (~200B tokens), while the same architecture trained on the filtered subset achieves 17.68% after only 36K steps. This is a 45% relative improvement from filtering alone, before any synthetic data is added. The filtered model sees fewer total tokens (6B vs. 35B+ in the raw corpus, though training steps differ) yet learns substantially more per token, providing direct evidence for the paper's central hypothesis that data quality—not just quantity—determines learning efficiency.

Synthetic Textbook Generation

The second major data source is a synthetically generated corpus of Python textbooks, totaling <1B tokens. The motivation is that even filtered web code lacks an essential ingredient for learning: natural language exposition that explains concepts, provides context, and demonstrates reasoning patterns. Human programmers learn not just by reading code examples but by reading explanations of why code works, how algorithms are structured, and what design patterns to apply in different situations. The web filter can select instructive code snippets, but it cannot manufacture the surrounding pedagogical narrative.

The diversity challenge. The central technical problem in generating synthetic textbook data is ensuring diversity. The authors state this explicitly:

"Simply prompting the model to produce a coding textbook or a set of exercises, even with some variation in the instructions or the parameters, will likely result in a very homogeneous and redundant dataset, where the same concepts and solutions are repeated over and over with minor changes. This is because language models tend to follow the most probable or common paths given their training data and their priors, and they lack the creativity or the incentive to explore alternative or novel ways of generating code."

In other words, a naive prompt like "write a Python textbook chapter about lists" produces content that converges to GPT-3.5's modal output: the most typical, least surprising version of that chapter. Repeated across many topics, this produces a dataset where surface features vary but the underlying structure, vocabulary, and pedagogical approach are nearly identical. Models trained on such data would overfit to these surface patterns rather than learning the underlying concepts.

The diversity mechanism: topic and audience constraints. The solution, inspired by Eldan and Li's (2023) TinyStories approach of injecting random word constraints, is to constrain the topic and target audience for each generated textbook segment. By specifying that a segment should cover, for example, "matrix operations for a high school student" versus "graph algorithms for a college sophomore," the prompt forces GPT-3.5 to adapt its vocabulary, examples, and level of detail to different contexts. This creates variation along multiple dimensions: the mathematical domain (linear algebra vs. graph theory), the complexity level (high school vs. college), and the types of examples that are natural for that domain and audience.

The paper does not specify the exact number of distinct topic-audience combinations or the generation prompt in full detail, noting that they "omit some details of the synthetic data generation, for proprietary reasons" (Section 1). However, the provided example (Section 2.2) illustrates the output format: natural language paragraphs explaining concepts (singular vs. nonsingular matrices, determinants) interleaved with self-contained code examples that demonstrate the concept in Python. The code examples include imports, function definitions, docstrings, and test invocations, making them executable in isolation—precisely the "textbook quality" property that the web filter selects for.

Coverage of reasoning and algorithmic concepts. The paper explicitly states that the textbook content was "targeted to cover topics that promote reasoning and basic algorithmic skills" (Section 2.2). This is a crucial design choice: rather than covering all of Python programming (which would require far more than 1B tokens and would duplicate much of what the filtered web data already provides), the synthetic textbooks focus on the reasoning-intensive parts of programming that are most underrepresented in typical web code. Topics likely include (based on the paper's examples and the math benchmark performance): matrix operations, sorting and searching, graph algorithms, numerical methods, probability and statistics through code, and basic data structure manipulations. These are precisely the kinds of tasks that HumanEval tests and that standard web code datasets—dominated by web frameworks, data science pipelines, and configuration files—do not adequately represent.

Token count and scaling context. The synthetic textbook dataset contains <1B tokens, which is remarkably small: it is less than 0.1% of StarCoder's 1T-token training corpus. Yet when combined with the filtered web code (forming the ~7B-token CodeTextbook dataset), it produces a pretrained model (phi-1-base) that achieves 29% on HumanEval—competitive with models trained on orders of magnitude more data (CodeGen-Mono-16.1B at 29.3% on 577B tokens; Replit-Finetuned at 30.5% on 525B tokens). This disproportionate impact of a small, high-quality corpus on reasoning performance is the paper's central empirical result.

Synthetic Exercise Generation (CodeExercises)

The third dataset, CodeExercises, is a collection of approximately 880,000 Python function-completion exercises totaling ~180M tokens. Each exercise consists of a function signature with type annotations, a docstring describing the function's behavior, and the completed function body. This dataset serves a fundamentally different purpose from the textbooks: while the textbooks teach concepts and explanations, the exercises train task execution—the ability to translate a natural language specification into correct Python code.

Format alignment with evaluation. The exercise format is deliberately aligned with the HumanEval and MBPP evaluation format: a function signature with a docstring that describes inputs, outputs, and behavior, followed by an implementation. This means the finetuning stage directly trains the model on the exact format it will encounter at evaluation time, reducing the gap between training and inference distributions. However, the paper argues (and Section 5 demonstrates) that the performance gains are not primarily due to format memorization, because pruning exercises similar to HumanEval problems does not eliminate the gains.

The diversity mechanism: function name constraints. For the exercise dataset, the primary method of eliciting diversity is "constraining the function names" (Section 2.2). The intuition, though not fully spelled out in the paper, is that function names encode the semantic category of the task. Constraining the generation to use specific function names (e.g., valid_guessing_letters, find_closest_two_holes, frequency_ranges_plot) forces GPT-3.5 to invent tasks that match those names, naturally producing variation in the types of operations, data structures, and algorithms covered. This is more effective than simply asking for "diverse exercises" because the language model's default behavior is to generate prototypical examples; the name constraint overrides this default by anchoring each generation to a different semantic starting point.

Example exercise structure. The paper provides a representative example (Section 2.2):

def valid_guessing_letters(word: str, guesses: List[str]) -> List[str]:
    """
    Returns a list of valid guessing letters, which are letters that have not
    been guessed yet and are present in the word.
    Parameters:
    word (str): The word to guess.
    guesses (List[str]): A list of letters that have already been guessed.
    Returns:
    List[str]: A list of valid guessing letters.
    """
    valid_letters = []
    for letter in word:
        if letter not in guesses and letter not in valid_letters:
            valid_letters.append(letter)
    return valid_letters

Several properties of this example are notable. First, it is self-contained: all imports (in this case, List from typing would need to be imported, though this detail is omitted in the snippet) and the function body use only standard Python. Second, it is algorithmically non-trivial but conceptually simple: the logic involves iterating over a string, tracking state across iterations, and filtering based on membership tests—exactly the kinds of basic algorithmic patterns that HumanEval tests. Third, the docstring is thorough: it describes the purpose, parameters, and return value in structured natural language, which teaches the model to attend to and follow natural language specifications.

Size of CodeExercises relative to pretraining. The CodeExercises dataset is tiny compared to even the already-small CodeTextbook: 180M tokens versus ~7B tokens in pretraining. During finetuning, the model sees each exercise roughly 8 times (6,000 steps × batch size 256 ÷ 880,000 exercises ≈ 1.75 passes, but since the effective number of tokens per step is higher due to sequence packing, the paper reports this as ~1.5B total tokens seen during finetuning versus 180M unique tokens, or roughly 8 effective epochs). The fact that such a small dataset produces a 21-percentage-point improvement on HumanEval (29% → 50.6%) is the paper's most striking result and motivates the "emergent capabilities" analysis in Section 3.

Decontamination. Because the exercises are synthetically generated by GPT-3.5, there is a risk that the teacher model inadvertently reproduces HumanEval-like problems, creating a contamination issue where phi-1 performs well because it memorized benchmark problems during training rather than learning generalizable skills. The paper addresses this through both standard n-gram overlap analysis (Section 5.1) and a more aggressive embedding-and-AST-based pruning approach (Section 5.2), which are detailed later in the evaluation sections.

Model Architecture

The paper's thesis is that data quality, not architectural innovation, drives the results. Accordingly, the model architecture is deliberately conventional, drawing from established design patterns in the code generation literature.

Base architecture. phi-1 is a decoder-only Transformer (Vaswani et al., 2017) using the FlashAttention implementation (Dao et al., 2022) for efficient multi-head attention computation. Decoder-only means the model processes tokens left-to-right with causal masking—each token can only attend to previous tokens, not future ones—which is the standard architecture for autoregressive language models since GPT (Radford et al., 2018). The choice of FlashAttention is a computational optimization, not a modeling change: it reduces the memory footprint of attention from O(n²) to O(n) in practice by tiling the attention computation, enabling longer sequences and larger batch sizes on the same hardware.

Parallel MHA and MLP configuration. The model uses multi-head attention (MHA) and MLP layers in parallel rather than the standard sequential arrangement. In a standard Transformer block, the input passes through attention, then through a feedforward network (MLP), with residual connections and layer normalization at each step. In the parallel configuration, following CodeGen (Nijkamp et al., 2022), PaLM (Chowdhery et al., 2022), and GPT-NeoX (Black et al., 2022), the attention and MLP computations are applied independently to the same input, and their outputs are summed:

$\text{output} = \text{MHA}(\text{LayerNorm}(x)) + \text{MLP}(\text{LayerNorm}(x)) + x$

Where MHA is the multi-head attention function, MLP is the feedforward network, LayerNorm is layer normalization, and $x$ is the input to the block.

What it computes: For each Transformer block, the input $x$ is first normalized via LayerNorm, then processed independently through two parallel branches—attention (capturing token-to-token relationships) and MLP (applying per-token nonlinear transformations). The outputs are summed with the original input via a residual connection, producing the block output.

Why this form: The parallel configuration allows the attention and feedforward pathways to specialize independently on different aspects of the representation—attention handles long-range dependencies and token interactions, while the MLP handles position-wise transformations and knowledge storage—without the sequential dependency that forces one pathway to wait for the other's output. Empirically, this configuration has been found to train more stably and efficiently at scale in the models cited above.

Size configurations. The paper trains two model sizes:

• phi-1 (1.3B parameters): 24 layers, hidden dimension $d_{\text{model}} = 2048$, MLP inner dimension $d_{\text{ff}} = 8192$, 32 attention heads of dimension 64 each. The MLP inner dimension is 4× the hidden dimension, following the standard Transformer expansion ratio.

• phi-1-small (350M parameters): 20 layers, hidden dimension $d_{\text{model}} = 1024$, MLP inner dimension $d_{\text{ff}} = 4096$, 16 attention heads of dimension 64 each.

Both configurations maintain an attention head dimension of 64, which is a standard choice (the original Transformer used 64). The number of heads scales with the hidden dimension (2048/64 = 32 heads for phi-1; 1024/64 = 16 for phi-1-small), so the total attention capacity per layer is proportional to $d_{\text{model}}$.

Positional encoding. The model uses rotary position embeddings (RoPE; Su et al., 2021) with rotary dimension 32. Rotary embeddings encode position information by applying a rotation to the query and key vectors in attention based on their relative positions, rather than adding absolute position encodings to the input embeddings. This has the advantage that the dot-product attention score between two tokens depends naturally on their relative distance, and the encoding decays smoothly with distance, providing a useful inductive bias for sequence modeling. The rotary dimension of 32 means the rotation is applied to the first 32 dimensions of each attention head (each head has dimension 64, so half the dimensions encode position via rotation, half are position-agnostic).

Tokenizer. phi-1 uses the same tokenizer as codegen-350M-mono (Nijkamp et al., 2022), which is a Byte-Pair Encoding (BPE) tokenizer trained specifically on code. Using an existing tokenizer avoids the need to train a new one and ensures compatibility with the CodeGen ecosystem. Code-specific tokenizers typically handle whitespace, indentation, and common code tokens (parentheses, brackets, operators) differently from natural language tokenizers, which matters for accurately representing code structure.

What the model does NOT use. The paper explicitly notes architectural features that phi-1 does NOT employ: Fill-In-the-Middle (FIM) training (Bavarian et al., 2022), which trains models to complete code given both prefix and suffix context; and Multi-Query Attention (MQA; Shazeer, 2019), which reduces the memory footprint of attention by sharing key and value projections across heads. Both techniques are used in StarCoder (Li et al., 2023) and have been shown to improve code generation performance. The paper's choice to omit them reinforces the thesis that data quality, not architectural sophistication, accounts for phi-1's performance: even without these enhancements, the model matches or exceeds StarCoder on HumanEval.

Pretraining Procedure

Pretraining follows a standard autoregressive language modeling setup with next-token prediction loss, but with careful attention to the data composition and training hyperparameters.

Data preparation. The CodeTextbook dataset (filtered web code + synthetic textbooks) is concatenated into a single one-dimensional array of tokens, with the special token <∣endoftext∣> used as a separator between files. This is the standard approach for causal language model pretraining: by treating the entire corpus as one long sequence and relying on the document separator to prevent cross-document attention (though in practice, attention can cross document boundaries during training, which the model learns to handle), the model sees documents in random order and learns to generate the next token given all previous tokens in the concatenated stream.

Training objective: next-token prediction. The model is trained to minimize the cross-entropy loss between its predicted token distribution and the actual next token. For a sequence of tokens $x_1, x_2, ..., x_T$, the loss at position $t$ is:

$\mathcal{L}_t = -\log p_\theta(x_t | x_{<t})$

where $p_\theta(x_t | x_{<t})$ is the model's predicted probability for the true token $x_t$ given all previous tokens $x_{<t}$, and $\theta$ represents all model parameters.

The total loss is the average over all positions in the training data:

$\mathcal{L} = \frac{1}{T} \sum_{t=1}^T \mathcal{L}_t$

What it computes: For every token position in the training data, the model produces a probability distribution over its vocabulary (a vector of size equal to the tokenizer's vocabulary size, where each entry is the model's estimated probability that the corresponding token comes next). The loss compares this predicted distribution to the ground-truth next token by taking the negative log of the probability assigned to the correct token. Lower loss means the model assigns higher probability to the correct continuation. The average across all positions gives a single scalar measuring the model's overall predictive accuracy.

Why this form: Next-token prediction is the standard objective for autoregressive language models because it is a self-supervised task—no human labels are needed, since the ground-truth next token is simply the token that actually appears next in the training data. It forces the model to learn all aspects of the data distribution: syntax (what token sequences are grammatically valid), semantics (what tokens are meaningful given the context), and reasoning (what tokens logically follow from the problem description). For code, next-token prediction on function bodies given docstrings implicitly teaches the model to translate natural language specifications into implementations.

Context length and batching. The model is trained on sequences of length 2048 tokens, sliced from the concatenated dataset array. With an effective batch size of 1024 (achieved through data parallelism across 8 GPUs and gradient accumulation), each optimizer step processes 1024 × 2048 ≈ 2.1M tokens. This batch size is moderate by modern standards—large enough to provide stable gradient estimates without requiring excessive memory.

Optimizer and learning rate schedule. Training uses the AdamW optimizer (Loshchilov and Hutter, 2019), which is Adam with decoupled weight decay. The specific hyperparameters are:

• Maximum learning rate: $1 \times 10^{-3}$
• Warmup: 750 steps of linear warmup from 0 to the maximum learning rate
• Decay schedule: linear decay from the maximum learning rate to 0 over the remaining steps
• Weight decay: 0.1 (applied independently of the learning rate, per AdamW)
• Betas: Not explicitly specified for pretraining, but standard AdamW defaults are $\beta_1 = 0.9, \beta_2 = 0.999$
• Attention and residual dropout: 0.1

The linear-warmup-linear-decay schedule (sometimes called a "triangular" schedule) is a common choice that avoids the instability of high learning rates at initialization (warmup phase) while allowing the model to converge to a precise minimum (decay phase).

Training duration and checkpoint selection. Pretraining runs for a total of 36,000 steps. The model checkpoint at 24,000 steps is selected as phi-1-base. This corresponds to approximately 24,000 × 2.1M = 50.4B total tokens seen during training. Given that the CodeTextbook dataset contains ~7B tokens, this represents roughly 7–8 epochs (passes through the dataset). The authors select the 24K-step checkpoint rather than the final 36K-step checkpoint, suggesting that validation performance (likely on a held-out set, though validation details are not specified for pretraining) peaked before the end of training. Training takes "under 4 days" on 8 Nvidia A100 GPUs, which at ~770 GPU-hours for the 24K checkpoint represents a tiny fraction of the compute used by models like StarCoder (trained on 512 TPUv4 chips for days).

Numerical precision. Training uses fp16 (16-bit floating point) mixed precision, which stores model weights and activations in half-precision to reduce memory usage and increase throughput, while maintaining a master copy of weights in fp32 for numerical stability during updates. DeepSpeed is used for distributed training across the 8 GPUs, handling data parallelism, gradient communication, and memory optimization.

Finetuning Procedure

Finetuning phi-1-base on CodeExercises to produce phi-1 uses the same basic training setup (next-token prediction, same architecture, same hardware) but with different hyperparameters and a crucial change in data composition.

Data composition shift. The finetuning corpus (CodeExercises) consists exclusively of short Python function-completion tasks—function signatures with docstrings followed by implementations. This is a narrower distribution than the pretraining data, which included long-form textbook explanations, multi-function code files, and diverse code styles. The deliberate narrowing serves to align the model with the task format it will encounter at evaluation: given a function signature and docstring, complete the function body.

Hyperparameter changes for finetuning. The finetuning hyperparameters differ from pretraining in ways that reflect the smaller dataset size and the different goal (adaptation rather than knowledge acquisition):

• Effective batch size: 256 (reduced from 1024). Smaller batch size provides more frequent updates per epoch, which is appropriate for a smaller dataset where each example should influence the model more strongly.
• Maximum learning rate: $1 \times 10^{-4}$ (reduced from $1 \times 10^{-3}$). Lower learning rate prevents catastrophic forgetting of pretrained knowledge—if the learning rate were too high, the model would overfit to the finetuning distribution and lose the general coding capabilities acquired during pretraining.
• Warmup: 50 steps (reduced from 750), reflecting the shorter total training duration.
• Weight decay: 0.01 (reduced from 0.1). Lower weight decay allows the model to make larger parameter changes during finetuning without excessive regularization.
• Total steps: 6,000, with checkpoints saved every 1,000 steps and the best checkpoint selected.

Sequence length and token packing. Finetuning uses the same 2048-token sequence length as pretraining. Since individual exercises are typically much shorter than 2048 tokens (the example exercise is ~70 tokens including docstring and implementation), multiple exercises are packed into each training sequence, separated by the <∣endoftext∣> token. This is standard practice for efficient training: rather than padding each sequence to 2048 tokens (which would waste computation), the dataloader concatenates exercises until the sequence is full.

Training duration and total tokens. With batch size 256, sequence length 2048, and 6,000 steps, the model sees approximately 256 × 2048 × 6,000 ≈ 3.1B tokens during finetuning. However, this counts packed tokens; the unique data is only ~180M tokens. The effective number of epochs is therefore roughly 3.1B / 180M ≈ 17 epochs, though the paper states 6,000 steps at this batch size corresponds to seeing ~1.5B total tokens (which would be ~8 epochs if sequence packing is accounted for differently). Finetuning takes "an additional 7 hours" on the same 8-A100 hardware, bringing the total training time for phi-1 to approximately 4 days and 7 hours.

Checkpoint selection. Unlike pretraining, where the 24K-step checkpoint was selected (before the end of training), finetuning selects the best checkpoint based on validation performance from checkpoints saved every 1,000 steps. This is critical because with a small finetuning dataset, overfitting can occur rapidly: after some number of epochs, the model begins memorizing specific exercises rather than learning generalizable code-completion skills, and validation performance (on whatever held-out set the authors use, though this is not explicitly specified) will degrade.

The Role of Finetuning: Knowledge Reorganization, Not Just Task Adaptation

The paper argues that finetuning on CodeExercises does more than simply teach the model the HumanEval format. Section 3 presents evidence that the finetuning stage unlocks emergent capabilities—abilities not explicitly present in the finetuning data and not reliably exhibited by phi-1-base. These include:

• Using external libraries (PyGame, Tkinter, PyTorch, Matplotlib) that do not appear in CodeExercises (Figure 3.1 shows that the exercise dataset imports are dominated by typing, math, collections, itertools, functools, and other standard-library modules—no PyGame, no Tkinter, no PyTorch).

• Following complex multi-step instructions with nested logical conditions (the Alice/Bob/Charles game example in Section 3.1).

• Engaging in chat-like interactions (the TA example in Section 3.2).

The paper's interpretation is that finetuning causes the model to reorganize and consolidate knowledge already present from pretraining:

"This suggests that our finetuning process might have helped the model in reorganizing and consolidating the knowledge acquired during pretraining, even if such knowledge is not explicitly present in our CodeExercises dataset."

This is a non-obvious claim with significant implications. If true, it means that the pretraining corpus contained sufficient information about, say, PyGame API calls, but the base model could not reliably access that information when prompted because its internal representations were not organized around the task of "follow an instruction and produce the corresponding code." Finetuning on simple function-completion tasks, even without any PyGame examples, teaches the model the meta-skill of instruction-following: given a natural language specification, attend to the relevant parts of the specification, retrieve the relevant knowledge from pretraining, and compose it into a coherent implementation. This meta-skill then transfers to domains where the specific knowledge (e.g., PyGame functions) was acquired during pretraining but previously inaccessible due to poor instruction-following ability.

This interpretation is consistent with the "emergent abilities" literature (Wei et al., 2022), which observes that certain capabilities appear only above a threshold of scale or training. Here, the "scale" is not parameter count but the combination of pretraining knowledge breadth and finetuning-induced instruction-following precision. The phi-1-base model (1.3B parameters after pretraining) has the knowledge but not the precision; phi-1 (after finetuning) has both. The smaller phi-1-small model (350M parameters) shows partial understanding—it grasps the logic of tasks (Section 3.1 example) but fails to produce correct API calls because its smaller capacity limits knowledge acquisition during pretraining. This three-way comparison (phi-1-base knows but can't execute; phi-1-small can reason but has limited knowledge; phi-1 can both reason and access knowledge) provides a nuanced picture of what finetuning accomplishes.

4. Key Insights and Innovations

Innovation 1: Data Quality as a Scaling Law Axis — Not Just a Constant-Factor Improvement

The paper's most fundamental intellectual contribution is demonstrating that data quality is not merely a constant-factor optimization but an independent axis of the scaling law that can produce regime changes in the performance-vs-compute relationship. Prior to this work, the dominant framework—established by Kaplan et al. (2020) and refined by Hoffmann et al. (2022)—treated scaling as a function of three variables: model parameters, training tokens, and compute budget. Data was treated as a fixed, interchangeable resource: more data was better, but "data" was implicitly assumed to be drawn from the same web-scale distribution regardless of curation effort. Quality improvements (deduplication, filtering) were viewed as variance-reduction techniques that shifted the intercept of the scaling curve but not its fundamental shape.

phi-1 breaks this assumption. The model achieves 50.6% on HumanEval with 1.3B parameters trained on 7B tokens, while StarCoder—trained on 1T tokens with 15.5B parameters—reaches only 33.6%. This is not a 2× or 3× efficiency gain; it is roughly a 100× reduction in training tokens and a 10× reduction in parameters to achieve superior performance. If data quality were merely a constant-factor improvement, one would expect phi-1's performance to fall somewhere between a 1.3B and 15.5B model trained on standard data—perhaps 20–25% on HumanEval. The fact that it exceeds the larger model means the relationship between data quality and capability is super-linear: high-quality data doesn't just teach faster, it teaches differently, enabling the model to learn conceptual structures that noisy web data obscures.

The evidence for this claim is not a single ablation but the entire architecture of Figure 2.1. The bars show that at fixed model size (350M or 1.3B), switching from standard Stack data (orange) to CodeTextbook (light green) produces a larger performance jump than what would be expected from simply training longer on standard data. For the 350M model, standard data saturates at 12.19% even after 96K steps (~200B tokens), while the same architecture on filtered data reaches 17.68% after only 36K steps—improving efficiency by roughly 6× in tokens while simultaneously improving absolute performance. The synthetic textbook data (not captured in that specific ablation but present in the final CodeTextbook) pushes this further to 20.12%. These are not marginal gains; they represent a fundamental shift in what a given parameter budget can accomplish.

The intellectual significance extends beyond code generation. If data quality can reshape scaling curves for reasoning tasks, then the entire edifice of scaling laws—which the field has used to justify ever-larger models and ever-larger datasets—may be contingent on the assumption that web-scale data is the best available training signal. The paper doesn't disprove scaling laws; it demonstrates that they are incomplete without a data quality dimension, and that future scaling law research must treat data curation strategy as a first-class variable alongside parameter count and token count.

This is a fundamental shift in framing, not an incremental refinement. It changes the question from "how much bigger must we scale?" to "how much better can we curate?"—a question that is arguably more scientifically interesting because it engages with what makes data informative for learning rather than simply how much data is available.

Innovation 2: Finetuning as Knowledge Reorganization, Not Just Task Specialization

The paper's second major conceptual contribution is reframing what finetuning accomplishes. The standard view in the transfer learning literature—from ULMFiT (Howard and Ruder, 2018) through T5 (Raffel et al., 2020) and instruction tuning (Wei et al., 2022)—treats finetuning as task adaptation: pretraining builds general capabilities from broad data, and finetuning narrows those capabilities to a specific task distribution. Under this view, finetuning on CodeExercises should improve HumanEval performance because the exercise format matches the evaluation format, but it should not improve performance on unrelated tasks like PyGame programming or chat interactions, which lie outside the finetuning distribution.

The paper's Section 3 directly contradicts this expectation. phi-1, after finetuning exclusively on short Python function-completion exercises (which contain no PyGame, no Tkinter, no PyTorch, and no multi-turn dialogue—confirmed by Figure 3.1's import distribution), demonstrates substantially improved capabilities on all these tasks compared to phi-1-base. The PyGame example shows phi-1 correctly implementing a ball-bouncing game with proper API calls and boundary logic, while phi-1-base produces syntactically valid but semantically confused code (defining unused variables, ignoring the movement logic). The Tkinter example shows phi-1 correctly wiring button callbacks to textfield operations, while phi-1-base hallucinates nonsensical API calls. The chat example shows phi-1 giving coherent instructional responses, while phi-1-base produces a single unhelpful sentence.

The paper's interpretation—that finetuning causes knowledge reorganization and consolidation—is a genuinely novel claim about the mechanism of transfer learning in language models. It suggests that pretraining stores knowledge in a format that is not readily accessible for instruction-following tasks: the model has the PyGame API knowledge (acquired during pretraining from the filtered web code), but it cannot reliably retrieve and compose that knowledge when prompted because its internal routing mechanisms—the attention patterns and feedforward pathways that map from a prompt to the relevant stored knowledge—are not optimized for instruction-following. Finetuning on simple exercises, even without PyGame examples, teaches the model the meta-capability of attending to instructions and retrieving relevant knowledge, which then transfers to any domain where the underlying knowledge exists from pretraining.

The three-way comparison between phi-1, phi-1-base, and phi-1-small provides a clean diagnostic for this claim. phi-1-base (1.3B parameters, pretrained but not finetuned) has the knowledge but cannot deploy it effectively. phi-1-small (350M parameters, both pretrained and finetuned) shows partial instruction-following ability—it understands the logic of tasks (the Alice/Bob/Charles example in Section 3.1 shows it correctly identifies the need for random sampling and point tracking) but makes API errors because its smaller parameter budget limited knowledge acquisition during pretraining. phi-1 (1.3B parameters, both pretrained and finetuned) combines sufficient capacity for knowledge storage with the instruction-following meta-skill, achieving both correct logic and correct API usage. This double dissociation—knowledge without instruction-following (phi-1-base) vs. instruction-following with limited knowledge (phi-1-small)—provides strong evidence that finetuning is doing something qualitatively different from task specialization.

This is a fundamental reframing with implications beyond this paper. If finetuning on narrow tasks can unlock broad capabilities by reorganizing existing knowledge, then the relationship between pretraining data diversity and finetuning data specificity is more complex than previously understood. It suggests that the optimal training strategy may involve a two-stage curriculum: broad pretraining to acquire diverse knowledge, followed by narrow but carefully structured finetuning to build the meta-skills needed to access that knowledge. This is conceptually distinct from both standard transfer learning (where finetuning is assumed to narrow the model) and instruction tuning (where finetuning data is typically broad and diverse). The paper doesn't develop this into a full theory, but it provides the empirical phenomenon—emergent cross-domain capabilities from narrow finetuning—that any such theory must explain.

Innovation 3: The "Textbook" as a Training Data Design Principle

The paper operationalizes a specific, previously vague intuition—"high-quality data"—into a concrete design principle with testable properties. Rather than treating data quality as an abstract notion to be optimized post hoc through filtering, the paper proposes that training data should be constructed to satisfy the properties of a good textbook: clear, self-contained, instructive, and balanced. Each of these four properties represents a departure from how training data is typically sourced.

Clarity means that the mapping from input (natural language specification or surrounding context) to output (code) is unambiguous, with explicit connections between explanation and implementation. This contrasts with standard web code, where docstrings may be missing, misleading, or in a different language, and where the relationship between comments and code is inconsistent.

Self-contained means that each training example can be understood without external context—no missing imports, no references to project-specific modules, no dependencies on files not present in the training sample. This contrasts with The Stack, where the authors identify non-self-contained snippets as a primary failure mode: code that calls functions defined elsewhere in a repository teaches the model nothing about the function's implementation, only that such a function exists.

Instructive means that examples are selected or generated to teach specific concepts, with explicit pedagogical intent. This contrasts with web code, where the majority of examples are "trivial or boilerplate code" that teaches nothing about algorithmic reasoning.

Balanced means that the distribution of concepts, difficulty levels, and coding patterns is deliberately constructed rather than left to the natural skew of web repositories. This contrasts with standard datasets where certain domains (web frameworks, data science utilities) are massively overrepresented while algorithmic problem-solving is underrepresented.

The innovation is not the observation that these properties are desirable—they are obvious from any pedagogical perspective—but rather the demonstration that operationalizing them through a combination of classifier-based filtering and constrained synthetic generation produces a qualitative regime change in learning efficiency. The paper provides a concrete recipe: use a strong model (GPT-4) to annotate a small sample for educational value, train a classifier to scale that annotation to a large corpus, and use a weaker model (GPT-3.5) with diversity-inducing constraints to generate textbook-style content that fills gaps in the filtered corpus. This recipe is specific enough to be replicated and general enough to apply beyond code generation to any domain where "educational value" can be defined.

The contrast with prior data curation approaches is instructive. Data cleaning (Raffel et al., 2020; Longpre et al., 2023) focuses on removing harmful or low-quality content—a negative criterion (what to exclude). The textbook principle provides a positive criterion (what to include and how to structure it). Deduplication (Lee et al., 2022) removes redundancy—the textbook principle actively seeks coverage of concepts. Filtering by heuristic rules (minimum length, presence of imports) captures some textbook properties but misses the deeper requirement that code should be instructive, which requires semantic understanding of what the code does and whether it teaches a transferable concept.

This is an incremental-to-fundamental contribution. The idea that data quality matters is old; the paper's contribution is in providing a sufficiently precise operationalization that it can be systematically varied and shown to produce outsized effects. The "textbook" framing also connects the machine learning problem to the human learning literature in a productive way, suggesting that curriculum design principles from education research—scaffolding, progressive complexity, balanced coverage—may translate directly to neural network training.

The evidence that this principle drives results, rather than merely correlating with them, comes from the ablation structure in Figure 2.1. The 350M model trained on filtered data (which implements clarity, self-containedness, and some instructiveness) substantially outperforms the same model on unfiltered data. Adding synthetic textbooks (which add instructiveness and balance) further improves performance. The finetuning stage (which adds task-specific clarity in the CodeExercises format) produces the largest jump. Each component of the textbook principle, when added, produces measurable gains, suggesting that the principle is causally relevant rather than merely descriptive.

Innovation 4: Data Pruning as a More Rigorous Alternative to Standard Decontamination

The paper introduces a methodological innovation in how to evaluate whether a model's performance on a benchmark reflects genuine capability versus memorization of training data. Standard decontamination studies in the LLM literature (e.g., Brown et al., 2020, Section 4.8; Chowdhery et al., 2022) typically measure n-gram overlap between training and test data: if test examples share long substrings with training examples, they are flagged as potentially contaminated, and performance on the remaining "clean" examples is reported separately.

The paper demonstrates that n-gram overlap is insufficient for code. Section 5.1 finds only 4 HumanEval problems with 13-gram overlap with CodeExercises, and all four are false positives—cases where the shared phrase (e.g., "the frequency of an integer is the number of times it appears in the list") describes a general concept rather than indicating that the exercise is a copy of the benchmark problem. N-gram analysis misses semantically similar problems that use different variable names, different function signatures, or different natural language phrasing while testing identical algorithmic reasoning.

The paper's alternative—embedding and AST-based similarity pruning—is more sophisticated. Embedding distance (L2 distance between CodeGen-Mono 350M embeddings) captures semantic similarity: two code snippets that implement the same algorithm with different surface forms will have similar embeddings even if their n-gram overlap is zero. Abstract Syntax Tree (AST) edit distance captures structural similarity: two functions that use the same control flow patterns will have similar ASTs even if their variable names and docstrings differ. By combining both measures, the paper identifies code pairs that are functionally similar even when superficially different—precisely the kind of "contamination" that matters for evaluating whether a model has learned generalizable skills versus memorized specific solutions.

The pruning experiment (Table 3) demonstrates the value of this approach. As the AST match rate threshold τ is lowered from 0.95 to 0.8, increasingly aggressive pruning removes more CodeExercises entries that are structurally similar to HumanEval problems—from 42.5K to 354K of the 879.5K total exercises (5% to 40% of the dataset). The retrained phi-1 models on these pruned datasets still achieve 45.1–50.6% on HumanEval, consistently outperforming StarCoder-Prompted (41.5%). Even at τ = 0.8, where over 40% of the finetuning data has been removed—including not just exact copies but even vaguely similar problems—phi-1 achieves 45.1%, well above StarCoder's 41.5%.

The conceptual contribution here is methodological rather than empirical: the paper demonstrates that similarity-based pruning is a more rigorous and informative decontamination procedure than n-gram overlap, and that it can serve as a general tool for evaluating whether model performance reflects memorization or generalization. The approach is scalable (embedding computation and AST parsing are automated), tunable (τ can be adjusted to vary the strictness of decontamination), and interpretable (the examples in Appendix C show exactly what kinds of similarity are captured at each τ level).

This is an incremental but practically significant innovation. The idea of using embeddings for similarity detection is not new in itself, but applying it systematically as a decontamination methodology—with explicit thresholds, retraining on pruned data, and comparison to n-gram baselines—is a contribution that other researchers can adopt when training on synthetic data that risks inadvertently reproducing benchmark problems. Given the growing use of LLM-generated training data in the field (Alpaca, Orca, Self-Instruct and their descendants), the need for rigorous decontamination methods is increasing, and the paper provides a template.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluations are conducted on two standard code-generation benchmarks: HumanEval (Chen et al., 2021), consisting of 164 handwritten Python programming problems where the model must complete a function given its signature and docstring, evaluated using the pass@1 metric; and MBPP (Mostly Basic Python Programs; Austin et al., 2021), a dataset of crowd-sourced Python problems, evaluated at pass@1. The paper also introduces a custom evaluation set of 50 new unconventional coding problems created by a dedicated team with explicit instructions to design problems unlikely to appear in real-world code bases or standard coding exercises, used for LLM-graded evaluation in Section 4. The MATH benchmark split from Lightman et al. (2022) of 12,000 training and 500 test questions is not used in this paper—the paper focuses exclusively on code generation, not math reasoning (despite the paper's title appearing to borrow from the MATH domain, the evaluations here are entirely on code).

• Base model(s). The primary model is phi-1, a 1.3B-parameter decoder-only Transformer trained from scratch by the authors. Two additional variants are used for controlled comparisons: phi-1-base, which is the same architecture trained only on the CodeTextbook pretraining dataset without the CodeExercises finetuning stage; and phi-1-small, a 350M-parameter model trained with the identical pipeline (CodeTextbook pretraining followed by CodeExercises finetuning) but with fewer layers (20 vs. 24), smaller hidden dimension (1024 vs. 2048), and fewer attention heads (16 vs. 32). External models used for comparison include CodeGen-Mono-350M and CodeGen-Mono-16.1B (Nijkamp et al., 2023), Replit and Replit-Finetuned (Replit, 2023), StarCoder and StarCoder-Prompted (Li et al., 2023), CodeGeeX (Zheng et al., 2023), SantaCoder (Allal et al., 2023), CodeT5+ and InstructCodeT5+ (Wang et al., 2023), PaLM-Coder (Chowdhery et al., 2022), PaLM 2-S (Anil et al., 2023), GPT-3.5 and GPT-4 (OpenAI, 2023), Codex-300M and Codex-12B (Chen et al., 2021), CodeGen2-1B and CodeGen2-7B (Nijkamp et al., 2023), and WizardCoder (Luo et al., 2023).

• Metrics. The primary metric is pass@1 accuracy, defined as the fraction of problems for which the single generated answer (greedy or top-1 sampled) passes all provided unit tests for HumanEval and MBPP. For the unconventional problems in Section 4, pass@1 is not directly usable because test suites were not developed for this custom set; instead, the paper uses LLM-graded scores, where GPT-4 evaluates each candidate solution against a reference solution on a scale of 0–10, combining a short verbal evaluation with a numerical grade. The grading prompt instructs GPT-4 to evaluate the student's solution holistically, considering correctness, efficiency, and code quality. Scores are averaged across the 50 problems to produce a single "Understanding score" per model. For HumanEval, all reported pass@1 numbers use the standard grading function released by Chen et al. (2021) unless otherwise noted, and the paper reports self-reported scores from prior work whenever available (Table 1 note).

• Baselines. The paper compares phi-1 against a comprehensive set of prior code generation models at various scales: Codex-300M and Codex-12B (the original HumanEval baselines), CodeGen-Mono-350M and CodeGen-Mono-16.1B (representing the scaling approach on filtered GitHub code), PaLM-Coder-540B (the largest model in the comparison, demonstrating the scaling extreme), GPT-3.5 and GPT-4 (representing the frontier of proprietary models), SantaCoder-1.1B, StarCoder-15.5B, StarCoder-Prompted-15.5B (the BigCode project's models trained on The Stack), CodeGeeX-13B (a multilingual code model), Replit-2.7B and Replit-Finetuned-2.7B (trained on 525B tokens, representing the previous smallest model achieving near-30% HumanEval), CodeT5+-2B, CodeT5+-16B, InstructCodeT5+-16B (encoder-decoder code models), CodeGen2-1B and CodeGen2-7B (the updated CodeGen series), PaLM 2-S (Google's state-of-the-art model), and WizardCoder-16B (which achieves the highest HumanEval score in the table at 57.3%). For the decontamination analysis in Section 5, StarCoder-Prompted serves as the primary external baseline, consistently achieving 41.5% on the HumanEval subsets used in that analysis.

• Generation budget / compute accounting. For the pretraining phase, compute is measured in GPU-hours (770 GPU-hours on 8 Nvidia A100s for phi-1-base at the 24K-step checkpoint) and total training tokens seen (approximately 50B tokens). For finetuning, an additional 7 GPU-hours are consumed. The paper uses model size (parameter count) and dataset size (tokens) as the primary axes for comparing training efficiency across models (Table 1). Unlike the compute-optimal test-time scaling paper from the prior example, this paper does not measure inference-time compute budgets—all evaluations use single greedy generations (pass@1) with no beam search, majority voting, or verifier-guided sampling. The comparison is therefore strictly about training efficiency (how much pretraining and finetuning compute is needed to achieve a given benchmark performance), not about trading off pretraining compute against inference-time compute. The relevant scaling dimensions in Figure 2.1 are compute time (measured by total tokens seen, from 26B to 76B) and number of parameters (350M vs. 1.3B), with the third dimension being dataset composition (The Stack vs. CodeTextbook vs. CodeTextbook + CodeExercises).

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation for its main results. For the decontamination analysis in Section 5, the retrained models on pruned datasets are trained once each (at different τ thresholds: 0.95, 0.9, 0.85, 0.8) and evaluated on the full HumanEval set, with results reported separately for "similar" and "non-similar" subsets defined by whether each HumanEval problem has at least one close match in the original (unpruned) CodeExercises dataset at the given τ. The 50 new unconventional problems in Section 4 are evaluated once per model with GPT-4 grading. No statistical significance tests, confidence intervals, or multiple-seed training runs are reported, which is a notable methodological limitation relative to standard practice in the LLM evaluation literature. The checkpoint selection for both pretraining (24K steps out of 36K total) and finetuning (best checkpoint saved every 1,000 steps out of 6,000 total) implies some form of held-out validation, but the validation procedure and metric are not described.

Main Quantitative Results

Overall Benchmark Performance (HumanEval and MBPP)

The headline result appears in Table 1: phi-1 achieves 50.6% pass@1 on HumanEval and 55.5% pass@1 on MBPP, with 1.3B parameters trained on 7B tokens. This places it above every model in the comparison table except GPT-4 (67% HumanEval, no MBPP reported) and WizardCoder (57.3% HumanEval, 51.8% MBPP). Specifically:

• Against StarCoder-15.5B (trained on 1T tokens): phi-1 outperforms by 17.0 percentage points on HumanEval (50.6% vs. 33.6%) and by 2.8 points on MBPP (55.5% vs. 52.7%). StarCoder-Prompted achieves 40.8% on HumanEval—phi-1 leads by 9.8 points despite using <1% of the training data.

• Against CodeGen-Mono-16.1B (trained on 577B tokens): phi-1 outperforms by 21.3 points on HumanEval (50.6% vs. 29.3%).

• Against PaLM-Coder-540B (trained on 780B tokens): phi-1 outperforms by 14.7 points on HumanEval (50.6% vs. 35.9%) and by 8.5 points on MBPP (55.5% vs. 47.0%).

• Against GPT-3.5 (175B parameters, dataset size not available): phi-1 exceeds the reported 47% HumanEval score by 3.6 points.

• Against Replit-Finetuned-2.7B (the previous smallest model near 30% HumanEval, trained on 525B tokens): phi-1 outperforms by 20.1 points on HumanEval (50.6% vs. 30.5%).

The MBPP result (55.5%) is particularly notable because it exceeds all models in the table except GPT-4 (no score reported) and PaLM 2-S (50.0%), and it demonstrates that phi-1's performance generalizes beyond the HumanEval distribution to a different benchmark with different problem styles.

The Effect of Data Quality: Ablation in Figure 2.1

Figure 2.1 provides the paper's central causal evidence for the data quality hypothesis by comparing models along three dimensions simultaneously: model size, compute (tokens seen), and dataset composition.

350M-parameter models, unfiltered vs. filtered data. The unfiltered Stack + StackOverflow model (orange, leftmost bar in the 350M group) achieves 12.19% on HumanEval after 96K steps (~200B tokens seen). The filtered-only model (not shown as a separate bar in Figure 2.1, but described in the text of Section 2.1) achieves 17.68% after only 36K steps—a 45% relative improvement from filtering alone, using fewer training tokens. When the filtered data is combined with synthetic textbooks to form the CodeTextbook dataset (light green, second bar in the 350M group), performance reaches 20.12%, representing a 65% total improvement over the unfiltered baseline. This establishes the first rung of the data quality ladder: filtering improves efficiency, and adding synthetic textbook data further improves it.

1.3B-parameter models, CodeTextbook pretraining. The 1.3B-parameter model trained on CodeTextbook alone—phi-1-base (light green, fifth bar)—achieves 29% on HumanEval after 51B tokens seen (770 GPU-hours). This is the paper's key intermediate result: even without any task-specific finetuning, the CodeTextbook-trained model matches or exceeds models trained on orders of magnitude more data. Specifically, the 1.3B phi-1-base at 29% equals CodeGen-Mono-16.1B at 29.3% (577B tokens, ~20× more compute) and approaches Replit-Finetuned-2.7B at 30.5% (525B tokens, ~100× more training tokens per the paper's claim). The paper highlights this:

"The previous smallest model that achieves close to 30% performance on HumanEval was Replit-Finetuned at 2.7B parameters, which was trained with 100 times more training tokens than us."

The finetuning jump. Finetuning phi-1-base on CodeExercises (dark green bars vs. light green bars) produces the largest absolute improvement: from 29% to 50.6% for the 1.3B model (+21.6 percentage points), and from 20.12% to 45% for the 350M model (+24.9 percentage points, based on the phi-1-small result reported in the abstract and Table 2 but not explicitly shown in Figure 2.1's bar chart—the abstract reports 45% on HumanEval for phi-1-small). The finetuning dataset is only 180M tokens (compared to 7B for pretraining), yet it accounts for nearly half of phi-1's final HumanEval performance.

Comparison of the three model versions. The improvement from finetuning is not attributable to simply training longer on more data. The paper shows a control: the 1.3B model trained only on CodeTextbook for 76B tokens (orange bar, "The Stack+" trained for 1090 GPU-hours)—which is more total training than phi-1-base (51B tokens)—achieves lower performance than phi-1-base at 51B tokens, demonstrating that the CodeTextbook data composition matters more than raw token count. Similarly, phi-1 (finetuned on CodeExercises) substantially outperforms phi-1-base despite seeing fewer total pretraining-equivalent tokens during finetuning, demonstrating that the finetuning data's format and quality, not its volume, drives the improvement.

LLM-Graded Evaluation on Unconventional Problems (Table 2)

To address concerns that HumanEval performance might reflect memorization or format-specific overfitting, Section 4 evaluates models on 50 newly created unconventional coding problems using GPT-4 as a grader. The problems are designed to be outside the training distribution—unlikely to appear in real-world code bases or standard coding exercise datasets.

Headline result from Table 2. phi-1 achieves a GPT-4-graded score of 52%, compared to 51% for StarCoder-15.5B, 45% for phi-1-small, 37% for phi-1-base, 37% for Replit-2.7B, 38% for CodeGen-Mono-16.1B, and 19% for CodeGen-Mono-350M. The ranking on these unconventional problems closely matches the HumanEval ranking: phi-1 > StarCoder > phi-1-small > phi-1-base ≈ Replit ≈ CodeGen-Mono-16.1B > CodeGen-Mono-350M.

Interpretation. The fact that phi-1 (52%) exceeds StarCoder (51%) on problems explicitly designed to be unlike training data strongly supports the claim that phi-1's performance reflects generalizable coding proficiency rather than benchmark memorization. The 15-point gap between phi-1 (52%) and phi-1-base (37%) on these unconventional problems mirrors the HumanEval gap (50.6% vs. 29%) and confirms that the finetuning benefit extends well beyond the specific format or content of the CodeExercises dataset. The phi-1-small result (45%) is also consistent with its HumanEval performance (45%), providing convergent validity for the LLM-grading methodology.

Methodological note on LLM grading. The paper explicitly justifies using GPT-4 as a grader by noting that it "obviates the need for tests" and provides "more fine-grained and meaningful signal." This is a practical compromise—writing rigorous unit tests for 50 hand-crafted problems would be labor-intensive—but it introduces a potential confound: GPT-4 is the teacher model (or a stronger variant thereof) used in data generation, and its grading preferences may align with the synthetic data distribution that phi-1 was trained on. The paper does not address this potential circularity, though the consistent ranking with HumanEval (which uses automated unit tests) provides some reassurance.

Decontamination Analysis via Data Pruning (Table 3, Section 5)

Section 5 conducts the most rigorous test of whether phi-1's HumanEval performance reflects memorization. The methodology is described in detail in the prior sections; here we focus on the quantitative results.

Table 3 structure. The table reports HumanEval accuracy for three models—original phi-1, phi-1 retrained on pruned CodeExercises data, and StarCoder-Prompted—at four pruning thresholds τ (0.95, 0.9, 0.85, 0.8), with problems split into "similar" and "non-similar" subsets based on whether the HumanEval problem has at least one close match in the original (unpruned) CodeExercises dataset at that τ.

Key results at τ = 0.95 (most conservative pruning, removing 42.5K of 879.5K exercises). On the 71 "similar" problems, both original phi-1 and retrained phi-1 achieve 81.7%, substantially above StarCoder-Prompted's 57.7%. On the 93 "non-similar" problems, the retrained model achieves 32.3%, above StarCoder-Prompted's 29.0%. The retrained model's overall accuracy (50.6%) exactly matches the original phi-1, indicating that removing exercises flagged as similar at this strict threshold has no effect on performance.

Results at lower τ values (more aggressive pruning). As τ decreases, the retrained model's performance gradually declines but remains well above StarCoder-Prompted's 41.5% at all thresholds:

• τ = 0.9 (93 similar, 71 non-similar): retrained phi-1 achieves 45.1% overall, StarCoder gets 41.5%.
• τ = 0.85 (106 similar, 58 non-similar): retrained phi-1 achieves 46.3%.
• τ = 0.8 (116 similar, 48 non-similar): retrained phi-1 achieves 45.1%.

At τ = 0.8, over 40% of the CodeExercises dataset (354K of 879.5K problems) has been removed, yet phi-1 still outperforms StarCoder-Prompted by 3.6 percentage points (45.1% vs. 41.5%). This is the paper's strongest evidence against memorization: even after aggressively removing any exercise that shares structural similarity with any HumanEval problem, the model retains most of its performance advantage.

Differential performance on similar vs. non-similar subsets. Across all τ values, the retrained model performs substantially better on the "similar" subset than on the "non-similar" subset. At τ = 0.8, the split is 52.6% (similar) vs. 27.1% (non-similar). This could be interpreted in two ways: (1) the similar problems are genuinely easier, and the model's knowledge transfers better to them even without memorization; or (2) the pruning did not fully remove all contamination, and residual memorization contributes to the higher similar-subset performance. The paper does not resolve this ambiguity, but the consistent outperformance of StarCoder on both subsets (and the qualitative examples in Appendix C showing that "similar" at low τ can mean very different problems) favors the first interpretation.

The StarCoder baseline in Table 3. StarCoder-Prompted consistently achieves 41.5% on the total 164 problems across all τ values (by definition, its score doesn't change with pruning since it wasn't trained on CodeExercises). On the "similar" subsets, StarCoder performs meaningfully above its overall average (57.7% at τ = 0.95, 48.4% at τ = 0.9), suggesting that problems flagged as similar to some synthetic exercise are—independent of contamination—easier for code models in general. This provides an important baseline: even a model with no exposure to CodeExercises finds the "similar" problems more solvable.

Emergent Capabilities: Qualitative Evidence (Section 3)

Section 3 presents a series of qualitative comparisons between phi-1, phi-1-base, and phi-1-small on tasks that lie outside the CodeExercises finetuning distribution. These are not quantitative benchmarks but side-by-side code generation examples that the paper uses to support the "knowledge reorganization" hypothesis.

Library usage (PyGame, Tkinter, PyTorch, Pyplot). In the PyGame example (Section 3.2), phi-1 correctly implements the full game loop with proper API calls (pygame.display.set_mode, pygame.draw.circle, pygame.display.update), boundary checking, and spacebar event handling. phi-1-base produces a semantically confused implementation with unused variables and incorrect logic. phi-1-small produces code that understands the logic (moving a ball, checking bounds) but fails on API correctness (using pygame.draw.rect instead of pygame.draw.circle, missing the event loop structure). This three-way comparison cleanly demonstrates the double dissociation: knowledge without instruction-following (phi-1-base) vs. instruction-following with insufficient knowledge capacity (phi-1-small) vs. both (phi-1).

Algorithmic reasoning with complex logic. In the Alice/Bob/Charles game example (Section 3.1), phi-1 correctly implements the nested random number generation with proper ranges, the point-scoring condition involving a floor-of-square-root divisibility check, and the iteration loop. phi-1-base ignores the prompt's logical structure entirely, instead defining class attributes that don't correspond to the problem. phi-1-small shows partial understanding—it generates random numbers and tracks points—but misinterprets "Bob then pick a number starting from Alice's number to 888" and applies the wrong condition for scoring. This illustrates the gradient of capability: phi-1-small has the meta-skill of instruction-following but lacks the capacity to handle all the logical relationships simultaneously; phi-1-base has the capacity but lacks the meta-skill.

Chat capability. The TA example (Section 3.2) shows phi-1 providing a helpful, structured response (numbered steps, code example) to a student question about Pyplot resolution and rotation. phi-1-base produces a one-sentence response with incorrect function names. phi-1-small enters a repetitive loop. The chat capability is particularly informative because CodeExercises contains no dialogue data—this capability must have been acquired during pretraining (from the synthetic textbooks or filtered web data) and made accessible through finetuning-induced reorganization.

Coverage of API imports in CodeExercises. Figure 3.1, generated by phi-1 itself, shows the distribution of imports across the ~880K exercises in CodeExercises. The dominant imports are typing, math, collections, itertools, functools, random, re, string, datetime, heapq, bisect, fractions, decimal, statistics, and itertools—all standard-library modules focused on data structures and basic algorithms. No PyGame, Tkinter, PyTorch, or Matplotlib imports appear (the authors state they "ignore libraries imported less than 10 times," and the plot shows no external library imports above this threshold). This figure serves as quantitative evidence that the emergent library-usage capabilities cannot be explained by direct exposure during finetuning.

Overall assessment of emergent capabilities evidence. The qualitative examples are compelling as demonstrations, but they are not systematic evaluations. The paper does not report success rates across many examples of each capability type, does not compare to external baselines on these tasks, and does not quantify the frequency with which phi-1 succeeds versus fails on library-usage prompts. The examples are selected to illustrate the phenomenon, but the paper provides no information about how representative they are. This is acknowledged implicitly by the framing as "spikes of model capability" (Section 3 title) and the reliance on qualitative comparison rather than benchmark scores.

Ablation Studies and Robustness Checks

Filtering ablation (Section 2.1, described in text): Training a 350M model on unfiltered Stack + StackOverflow achieves 12.19% HumanEval after 96K steps (~200B tokens), while the same architecture on the filtered subset achieves 17.68% after 36K steps. This is the most direct evidence that data quality alone—independent of synthetic data or model scale—produces substantial improvements. However, this ablation is not a controlled experiment: the filtered model was trained for fewer steps, so token counts differ, and the unfiltered model may have been undertrained relative to its data volume.

CodeTextbook composition ablation (Figure 2.1): The 350M filtered-only model (described in text but not explicitly labeled as a separate bar) improves to 20.12% when synthetic textbooks are added, demonstrating the additive value of the textbook data beyond filtering alone. The 1.3B CodeTextbook model (phi-1-base) achieves 29%, demonstrating that scaling the model on the same high-quality data further improves performance without hitting a saturation point at 350M parameters.

Finetuning on CodeExercises (Figure 2.1, dark green vs. light green bars): Both model sizes show large improvements from finetuning: 350M improves from 20.12% to (implied) 45% (phi-1-small's final performance), and 1.3B improves from 29% to 50.6%. This is an ablation in the sense that the only difference between phi-1-base and phi-1 is the additional 7 hours of finetuning on 180M tokens of exercises. The magnitude of the jump suggests that the exercise format and content provide a uniquely strong training signal, but the paper cannot disentangle whether the benefit comes from (a) format alignment with HumanEval, (b) the specific algorithmic content of the exercises, or (c) the knowledge-reorganization effect hypothesized in Section 3.

Pruning threshold sweep (Table 3, τ ∈ {0.95, 0.9, 0.85, 0.8}): The retrained phi-1 models at different pruning levels provide a robustness check on the decontamination claim. The performance remains consistently above StarCoder-Prompted (45.1–50.6% vs. 41.5%) even as up to 40% of the finetuning data is removed. This is a genuine robustness finding: the CodeExercises benefit is not fragile to the removal of specific problems and does not depend on a small number of HumanEval-similar exercises. However, the gradual decline from 50.6% (no pruning) to 45.1% (τ = 0.8) suggests that some of the performance does come from exercises that share structure with HumanEval problems, even if the benefit persists after their removal.

Model size scaling on high-quality data (Figure 2.1, comparing 350M and 1.3B models within each dataset condition): Within the CodeTextbook condition, scaling from 350M to 1.3B improves HumanEval from 20.12% to 29% (+8.9 points). Within the finetuned condition, scaling from 350M (phi-1-small) to 1.3B (phi-1) improves from 45% to 50.6% (+5.6 points). The diminishing returns at the finetuned stage could indicate that finetuning is saturating the available knowledge in the 350M model (consistent with the phi-1-small API-usage failures), or that the finetuning dataset is too small to benefit substantially from additional capacity. The paper does not explore this further.

N-gram overlap as a negative result for standard decontamination (Section 5.1): The paper finds only 4 HumanEval problems with 13-gram overlap with CodeExercises, and all are false positives—cases where the shared phrase describes a generic concept rather than indicating problem duplication. This is presented as evidence that n-gram overlap is an inadequate decontamination method for code, motivating the embedding+AST approach. It also indirectly supports the paper's claim that the synthetic data generation did not simply reproduce HumanEval problems.

The Stack+ baseline trained longer (Figure 2.1, 1.3B orange bar at 76B tokens): Training the 1.3B model on the standard Stack dataset for 1,090 GPU-hours (76B tokens seen) produces lower performance than phi-1-base at 770 GPU-hours (51B tokens seen on CodeTextbook). This controlled comparison—same architecture, same broad compute budget, different data composition—is the cleanest evidence in the paper that the CodeTextbook data drives the performance difference rather than training duration or model architecture. Unfortunately, the exact HumanEval score for this Stack+ baseline is not reported as a number in the text or figure; it appears only as a bar in Figure 2.1, making precise comparison difficult.

Sensitivity to prompt variations (Appendix B): The paper provides qualitative examples of phi-1's brittleness: increasing the number of layers from 3 to 4 in a PyTorch network prompt causes the model to fail; adding import torch to the prompt fixes it. Changing the phrasing "x stays unchanged" to "x stays at x" alters behavior. These are not controlled ablation experiments but they serve as robustness checks in the negative direction: they demonstrate that phi-1's capabilities, while impressive at their peak, are fragile to prompt perturbations. The paper attributes this to the structured nature of the training data:

"Due to the structured nature of the datasets and the lack of diversity in terms of language and style, phi-1 is less robust to stylistic variations or errors in the prompt."

Limitation demonstrations (Appendix B): The paper documents specific failure modes: counting and spatial reasoning (the Tkinter example with buttons and textfields), sensitivity to prompt length, and difficulty with ambiguous natural language. Again, these are qualitative demonstrations rather than quantitative ablations, but they serve the important function of bounding the claims: phi-1 is strong on clean, short, well-specified Python tasks but degrades when prompts become longer, more ambiguous, or require spatial layout reasoning.

Critical Assessment

The experiments in this paper demonstrate a genuinely striking empirical phenomenon: a 1.3B-parameter model trained on 7B tokens of carefully curated data can match or exceed 15B-parameter models trained on 1T+ tokens on standard Python code generation benchmarks. The evidence for this core claim is strong and multi-faceted—it appears consistently across HumanEval (Table 1), MBPP (Table 1), the unconventional problem evaluation (Table 2), and the decontamination-pruned evaluations (Table 3). The ranking phi-1 > StarCoder > smaller models holds across all these evaluation settings, making it unlikely that the result is an artifact of any single benchmark's peculiarities.

However, the paper makes several stronger claims that the experiments support only partially or conditionally.

Claim: "Data quality can dramatically change the shape of the scaling laws." The evidence for this claim is Figure 2.1, which shows that at fixed model size, switching data composition produces larger performance gains than scaling model size at the same data composition. This is suggestive but not conclusive for "changing the shape of scaling laws" because the paper varies data quality at only two points (unfiltered vs. filtered+textbooks) and model size at only two points (350M vs. 1.3B). A genuine demonstration of scaling law reshaping would require showing that the slope of the performance-vs-compute curve changes under different data quality regimes across a wider range of model sizes (e.g., 100M, 350M, 1.3B, 3B, 7B). With only two sizes and two data conditions, the paper has four data points for this claim—insufficient to characterize a curve, let alone demonstrate that its shape has changed. The paper's language on this point ("dramatically change the shape of the scaling laws") is stronger than what the experiments directly support; a more accurate characterization would be that data quality appears to shift the intercept and possibly the slope of the performance-vs-scale relationship, but the slope change is not rigorously measured.

Claim: "Finetuning reorganizes and consolidates knowledge, unlocking emergent capabilities." The evidence consists of qualitative side-by-side examples in Section 3 and Appendices A-B. These examples are impressive but suffer from selection bias—the paper presents examples where phi-1 succeeds and phi-1-base fails, without quantifying how often this pattern holds across a representative sample. A reader cannot determine from the presented evidence whether the PyGame capability emerges reliably (e.g., phi-1 succeeds on 70% of such prompts vs. phi-1-base on 10%) or only on cherry-picked examples. The lack of any quantitative benchmark for these "emergent" capabilities—even a small one of 20–30 prompts per capability with pass/fail grading—is the most significant methodological gap in the paper. The claim of knowledge reorganization is an interpretation of the data, not a direct empirical finding; alternative interpretations (e.g., finetuning simply teaches better instruction-following heuristics that happen to transfer) are not ruled out.

Claim: "Data pruning shows performance is not due to memorization." Table 3 shows that retrained phi-1 on pruned data (45.1% at τ = 0.8) still outperforms StarCoder (41.5%). This is strong evidence that performance does not depend on memorization of HumanEval-similar exercises. However, it does not rule out that some memorization occurs and contributes to the higher 50.6% score of the unpruned model. The 5.5 percentage point drop from unpruned (50.6%) to τ = 0.8 pruned (45.1%) could represent the removal of memorized content, or it could represent the removal of genuinely instructive exercises that happen to share structure with HumanEval problems. The paper cannot distinguish these interpretations because it does not measure whether the pruned model's lower performance on the "similar" subset specifically (52.6% at τ = 0.8 vs. 59.5% unpruned) reflects loss of memorized solutions or loss of transferable skills.

Weakness: single model family and training pipeline. All results are for the specific architecture, tokenizer, and training recipe described in Section 2.3. The paper cannot distinguish whether the data quality benefit generalizes across architectures or is specific to this particular combination of FlashAttention, parallel MHA/MLP, rotary embeddings, and the codegen-350M-mono tokenizer. Replicating with a different architecture (e.g., standard sequential Transformer blocks, different positional encodings) would strengthen the claim that data quality, not architectural synergy, drives the results.

Weakness: no comparison to models trained on CodeTextbook at larger scale. The paper demonstrates that 1.3B parameters on 7B tokens outperforms 15.5B parameters on 1T tokens, but it does not train a larger model on the same high-quality data. The natural follow-up—what would a 7B or 15B model achieve if trained on CodeTextbook-scale curated data with CodeExercises finetuning?—is not explored. This makes it impossible to determine whether the data quality benefit plateaus or whether further scaling on high-quality data would yield even more dramatic gains. The paper's implicit message ("textbooks are all you need") suggests that scale is unnecessary if data quality is sufficient, but the experiments only show this for the specific scale point of 1.3B parameters—not that larger models on the same data wouldn't do even better.

Weakness: the 100× token efficiency claim conflates unique tokens with effective compute. The paper states that phi-1 was trained on "100 times less [sic] training tokens" than Replit-Finetuned (7B vs. 525B). However, phi-1-base was trained for ~8 passes over its 7B tokens (~50B tokens seen), and phi-1 saw additional finetuning tokens. The "100×" figure compares unique dataset size, not total training compute. The actual compute ratio is closer to 10–20× when total tokens seen are accounted for (50B + 1.5B for phi-1 vs. 525B for Replit), which is still impressive but less dramatic than the abstract's framing.

Missing experiment: the role of GPT-3.5 as teacher. All synthetic data in this paper was generated by GPT-3.5. The paper does not train a model on the same quantity of web data + CodeExercises generated by a weaker model, or by human authors, or by heuristic templates. It is therefore impossible to determine how much of the data quality benefit comes from the "textbook" structure versus from the fact that the synthetic data was generated by a model already capable of strong code generation. If GPT-3.5's synthetic exercises implicitly encode the teacher model's reasoning patterns in ways that human-written exercises wouldn't, then "data quality" may be better described as "distillation from a stronger model into a structured curriculum"—a still-interesting finding but one that constrains the generality of the "textbooks are all you need" thesis.

Missing evaluation: human judgment of code quality beyond correctness. All evaluations in the paper use pass@1 (binary correct/incorrect based on unit tests) or GPT-4 grading on a 0–10 scale. Neither captures dimensions of code quality that matter for practical use: readability, efficiency, adherence to conventions, robustness to edge cases not covered by the provided tests, or modularity. A model that achieves high pass@1 by generating code that barely passes tests through brittle pattern-matching would score identically to a model that generates clean, efficient, well-structured solutions. The paper's qualitative examples suggest phi-1's code quality is reasonable, but no systematic evaluation of this dimension is provided.

Missing ablation: the effect of exercise dataset size. The CodeExercises dataset contains 880K problems (180M tokens). The paper does not vary this size to determine whether the finetuning benefit saturates (would 440K exercises achieve nearly the same performance?) or whether the full 880K is necessary. Given the paper's thesis about data efficiency, understanding the relationship between finetuning data volume and performance would be particularly informative.

The unconventional problems evaluation (Section 4) has methodological strengths and weaknesses. The strength is that it directly addresses the contamination concern by using completely new problems designed outside the training distribution. The weakness is the reliance on GPT-4 grading, which introduces a potential confound (GPT-4 may favor the style of code that GPT-3.5-generated training data teaches) and is not validated against human judgments. The paper does not report inter-rater reliability between GPT-4 grades and any other standard, making it difficult to assess whether the 0–10 scores are meaningful beyond their rank correlation with HumanEval performance.

In summary, the paper convincingly demonstrates that a specific recipe—filtering web code for educational value, supplementing with synthetically generated textbook-style content, and finetuning on synthetic exercises—produces a 1.3B model with remarkable HumanEval and MBPP performance that generalizes beyond the finetuning distribution and is not explained by memorization. The strength of the evidence varies across the paper's claims: the benchmark performance claim is very well-supported; the decontamination claim is well-supported; the emergence claim is qualitatively demonstrated but not quantitatively validated; the scaling law reshaping claim is suggested by the data but not rigorously established given the limited number of scale points tested. The most significant unknown is how much of the benefit comes from the specific data curation principles (textbook structure, self-containedness, balance) versus from using a strong teacher model (GPT-3.5) to generate synthetic training data—a confound that the paper does not disentangle.

6. Limitations and Trade-offs

6.1 Specialization to Python Limits Generality to a Single Language and Task Domain

The assumption or constraint. phi-1 is trained and evaluated exclusively on Python code, with a specific focus on completing short functions from docstrings. The paper explicitly acknowledges this as the primary limitation of the approach:

"phi-1 is specialized in Python coding, which restricts its versatility compared to multi-language models."

The entire data pipeline—filtering The Stack's Python subset, generating Python textbooks, creating Python function-completion exercises—is language-specific. The synthetic textbook generation targets "topics that promote reasoning and basic algorithmic skills" in Python specifically, and the CodeExercises dataset consists of "Python exercises and solutions" exclusively. The evaluation benchmarks (HumanEval, MBPP, the 50 unconventional problems) are all Python-only.

The consequence. A practitioner considering deploying phi-1 faces a stark domain constraint: the model has no demonstrated ability to generate code in any language other than Python, nor to handle non-function-completion code tasks (e.g., writing full programs, refactoring existing code, explaining code behavior, translating between languages). This is not merely a "hasn't been tested" limitation—the training data contains no non-Python code (the filtering step explicitly selects the Python subset of The Stack), so the model likely has zero knowledge of other programming languages' syntax, idioms, or standard libraries. Even within Python, the model's competence may be narrow: it was finetuned on short function-completion tasks and may not generalize to writing multi-file projects, using uncommon standard library modules not represented in the filtered web data or synthetic textbooks, or handling code that depends on project-specific context.

Additionally, the paper notes that phi-1 "lacks the domain-specific knowledge of larger models such as programming with specific APIs or using less common packages." This is a direct consequence of the 7B-token training corpus: while models trained on 1T tokens of diverse code encounter long-tail APIs and domain-specific patterns, phi-1's compact, textbook-oriented corpus necessarily omits vast swaths of practical programming knowledge that a working developer needs—web frameworks, database interfaces, cloud SDKs, machine learning pipelines beyond basic PyTorch, testing frameworks, and so on.

What evidence exists in the paper. The paper provides no quantitative evidence on how phi-1 performs on non-Python tasks because it does not evaluate on any multi-language benchmark (e.g., HumanEval-X, MBXP, or MultiPL-E). The limitation is stated qualitatively in Section 6 and Appendix B, but its severity is unmeasured. A practitioner cannot determine from the paper whether phi-1 retains any latent multi-language capability from the pretraining data (the filtered web code is Python-only, so there is none) or whether the synthetic textbook approach could be replicated for other languages.

Mitigation status. The paper does not attempt to address this limitation and offers only a forward-looking statement:

"None of these limitations seem fundamental, and with more work our approach could be used to tackle each one of them, although it is unclear what scaling might be necessary to overcome them (both for the model size and the dataset size)."

This is an honest acknowledgment that the paper does not know whether the textbook-quality approach transfers to multi-language settings, nor what resources would be required to make it do so. The data generation methodology (filter web code, generate synthetic textbooks and exercises) is in principle language-agnostic, but the paper provides no evidence that it works for languages with less abundant web data, different syntactic structures, or different pedagogical traditions.

6.2 Sensitivity to Prompt Perturbations Prevents Robust Deployment

The assumption or constraint. The paper assumes that prompts presented to the model at inference time will be "textbook quality"—clear, well-structured, grammatically correct, and semantically unambiguous. This assumption is implicit in the training data design (all synthetic exercises use clean, formally structured docstrings) but is contradicted by real-world usage patterns, where users provide messy, ambiguous, or stylistically varied prompts.

The paper is candid about the resulting fragility:

"Due to the structured nature of the datasets and the lack of diversity in terms of language and style, phi-1 is less robust to stylistic variations or errors in the prompt (for instance, its performance substantially degrades when there are grammatical mistakes in the prompt)."

The consequence. phi-1 exhibits brittle behavior that makes it unreliable for user-facing applications where prompt quality cannot be guaranteed. Appendix B documents specific failure modes: changing phrasing from "x stays unchanged" to "x stays at x" alters model behavior; increasing the number of neural network layers from 3 to 4 in a prompt (while keeping the logical structure identical) causes the model to fail entirely; adding import torch to the beginning of a prompt that previously failed can "fix" the model's output. These are not minor variations—they represent the kind of natural prompt diversity that any deployed system must handle. A developer integrating phi-1 into a coding assistant would need to carefully control input formatting, potentially requiring a separate prompt-normalization step that the paper does not provide.

The sensitivity also manifests as difficulty with longer prompts: "its performance drops significantly as the length of the prompt increases, as it tends to ignore, forget or misinterpret parts of the prompt when it is too long." The paper hypothesizes that this is because "our exercises predominantly consist of short prompts," meaning the model was never trained to maintain coherence across lengthy specifications. Real-world programming tasks often require extensive specifications, making this a practical deployment barrier.

What evidence exists in the paper. The evidence is entirely qualitative—side-by-side examples in Appendix B showing prompt variations and model outputs. No quantitative benchmark measures the degradation as a function of prompt perturbation type or magnitude. A practitioner cannot determine from the paper whether the failure rate on grammatically imperfect prompts is 10% or 90%, nor which types of perturbations are most damaging. The paper does not systematically vary prompt properties (length, grammaticality, synonym substitution, reordering of clauses) and measure performance.

Mitigation status. The paper does not attempt to mitigate this limitation. It identifies the root cause (structured training data lacking stylistic diversity) but does not propose solutions such as data augmentation with perturbed prompts, adversarial training, or post-hoc prompt normalization. The limitation is presented as inherent to the textbook-quality approach:

"This may be because we filter out certain types of data from the training process to guarantee textbook-level quality."

This creates an unresolved tension: the very filtering that enables efficient learning also removes the diversity needed for robustness. The paper does not explore whether a middle ground exists—for instance, augmenting the high-quality core data with a small amount of diverse, lower-quality data to improve robustness without sacrificing the learning efficiency gains.

6.3 The Cost of Difficulty Estimation Is Not Amortized into the Efficiency Claims

The assumption or constraint. The difficulties in this section stem from comparing unique dataset tokens rather than total training compute. The paper's central efficiency narrative—that phi-1 achieves its results with "100× less training tokens" than competing models—is based on comparing unique dataset size (7B tokens for phi-1 vs. 525B tokens for Replit, 577B for CodeGen, 1T for StarCoder) rather than total training compute.

However, phi-1-base was trained for approximately 8 passes over its 7B-token pretraining corpus (24,000 steps × 2.1M tokens/step ≈ 50.4B tokens seen), and phi-1 saw additional tokens during finetuning (approximately 1.5–3.1B tokens seen, depending on how sequence packing is counted). The total tokens processed during training is therefore approximately 52–54B, not 7B. This is still substantially less than competing models (525B–1T), but the ratio is roughly 10–20× rather than the 100× claimed in the paper's framing:

"the previous smallest model that achieves close to 30% performance on HumanEval was Replit-Finetuned at 2.7B parameters, which was trained with 100 times more training tokens than us."

The "100 times more training tokens" claim compares 7B (unique) to 525B (unique or total?—the paper uses Replit's reported dataset size, which likely refers to unique tokens as well). If Replit also trained for multiple epochs, the total-tokens-seen ratio would differ from the unique-tokens ratio. The paper does not clarify whether competing models' token counts refer to unique dataset size or total training tokens seen, making cross-model efficiency comparisons ambiguous.

The consequence. The efficiency gains, while genuinely large, are overstated by approximately 5–10× when measured in terms of total compute rather than dataset size. This matters for practical decision-making: an organization choosing between "train a 1.3B model on 7B tokens of curated data" vs. "train a 15B model on 1T tokens of web data" needs accurate estimates of the total compute required for each approach, including the cost of data curation itself. The curation costs are substantial: generating 1B tokens of synthetic textbooks and 180M tokens of synthetic exercises requires paying for GPT-3.5 API calls (the paper does not report the number of calls or cost); training the web-code quality classifier requires GPT-4 annotations on 100K samples (again, cost unreported); and the filtering pipeline requires embedding 35M code files with CodeGen-Mono-350M before classification. None of these curation costs appear in the 4-days-on-8-A100s training budget.

What evidence exists in the paper. The paper reports training compute transparently (770 GPU-hours for pretraining, 7 GPU-hours for finetuning) but does not report or amortize the data generation and curation costs into any efficiency metric. The total-token-seen vs. unique-token distinction can be inferred from the reported training steps and batch size (Section 2.3), but the paper itself never makes this distinction or acknowledges the resulting overstatement of the efficiency ratio. The data generation cost is mentioned only in passing ("we omit some details of the synthetic data generation, for proprietary reasons") with no quantification.

Mitigation status. Not addressed. The paper does not report total wall-clock time or dollar cost for data curation, does not compare the FLOPs or GPU-hours of data generation across approaches, and does not acknowledge the unique-vs-total token distinction. A fair efficiency comparison would account for: (1) GPT-3.5 generation compute (tokens generated × cost per token), (2) GPT-4 annotation compute for classifier training, (3) embedding computation for the 35M-file corpus, (4) classifier training and inference, and (5) the standard training compute. Until these are quantified, the headline efficiency claims (100× fewer tokens, 4 days of training) represent an incomplete picture of the resources required to replicate the approach.

6.4 No Quantitative Evidence for Emergent Capabilities Leaves the Central Claim Unvalidated

The assumption or constraint. The paper's most theoretically significant claim—that finetuning on narrow CodeExercises unlocks broad emergent capabilities not present in the base model—is supported entirely by qualitative, cherry-picked examples in Section 3 and Appendices A–B. There is no benchmark, no success-rate measurement, and no statistical comparison for any of the claimed emergent capabilities: library usage (PyGame, Tkinter, PyTorch, Pyplot), complex algorithmic reasoning, or chat-like interaction.

The paper acknowledges this implicitly through its methodological framing as "reminiscent of the Sparks of AGI paper [BCE+23] that argued for moving away from static benchmarks," but this is a justification for using qualitative examples, not a substitute for quantitative validation of the specific claims being made.

The consequence. A reader cannot determine from the paper whether the emergent capabilities are reliable (does phi-1 succeed on 80% of PyGame prompts or 5%?), representative (were the shown examples the best of many attempts, or typical?), or causally attributable to finetuning (could a model trained longer on CodeTextbook alone eventually develop these capabilities?). The three-way comparison between phi-1, phi-1-base, and phi-1-small is suggestive—phi-1-base fails qualitatively on the shown examples—but without quantitative pass/fail rates across a representative sample of prompts, the evidence does not distinguish between "finetuning causes a general improvement in library-usage ability" and "finetuning occasionally produces a correct library-usage output, and those cases were selected for presentation."

This is not merely a presentation weakness; it is an evidentiary gap for the paper's central theoretical claim about knowledge reorganization. The paper argues that finetuning causes "reorganizing and consolidating the knowledge acquired during pretraining," which is a mechanistic hypothesis about what happens inside the model. The evidence presented—a handful of qualitative examples—is consistent with this hypothesis but also consistent with weaker explanations: finetuning simply teaches better instruction-following heuristics that incidentally transfer to some library-usage prompts; the base model occasionally succeeds on library tasks too, but those cases weren't shown; or the finetuned model benefits from format alignment (all CodeExercises are function-completion tasks) that partially transfers to library-using tasks that can be expressed in function-completion format.

What evidence exists in the paper. Section 3 provides approximately 6 qualitative comparisons (Alice/Bob/Charles game, PyGame, Tkinter, Chat, plus Appendix A's PyTorch and Pyplot examples). Figure 3.1 provides quantitative evidence that the claimed capabilities cannot come from direct exposure during finetuning (no PyGame/Tkinter/PyTorch imports appear in CodeExercises), which rules out the simplest alternative explanation (direct memorization of finetuning examples) but does not validate the positive claim of reliable emergent capability. Appendix B provides additional qualitative examples of failure modes, establishing that phi-1 is not universally capable on these tasks, but without quantifying the success rate.

Mitigation status. The paper does not attempt to quantify emergent capabilities. It explicitly chooses qualitative demonstration over benchmark evaluation, citing the Sparks of AGI methodology. The limitation is acknowledged implicitly in the section title ("Spikes of model capability") and the framing as qualitative comparison, but the gap between "spikes of capability" (suggesting occasional successes) and "reliable emergent capability" (suggesting a generalizable skill) is never clarified. Future work would need to construct even a modest benchmark—e.g., 50 prompts each for PyGame, Tkinter, PyTorch, and chat tasks, with pass/fail grading by a human or a strong LLM—to determine whether these capabilities are reliable enough for practical use.

6.5 The Teacher Model Confound: Data Quality vs. Distillation from GPT-3.5

The assumption or constraint. All synthetic data in phi-1's training pipeline—both the <1B tokens of textbooks and the ~180M tokens of exercises—was generated by GPT-3.5. The paper's central thesis attributes phi-1's performance to data quality (textbook structure, self-containedness, instructive value, balance), but a competing explanation is that phi-1 benefits from distillation: GPT-3.5, a model already capable of strong Python code generation (47% on HumanEval per Table 1), implicitly encodes its own reasoning patterns, coding style, and solution strategies into the synthetic data, and phi-1 learns to mimic these patterns.

Under the distillation hypothesis, the textbook-quality structure of the data may be less important than the textbook-quality source: GPT-3.5 generates code that is cleaner, more idiomatic, and more likely to pass unit tests than the average web code snippet, regardless of how it is formatted or organized. The filtering of web data and the synthetic textbook generation might both be approximating a simpler operation: "select or generate code that looks like what a strong model would produce."

The paper does not disentangle these hypotheses. It provides no ablation where synthetic data is generated by a weaker model, by human authors, or by heuristic templates (e.g., generating exercises by instantiating parameterized templates for common algorithmic patterns). Without such a control, the observed benefit of "high-quality data" is confounded with the benefit of "data generated by a model that already knows how to code."

The consequence. A practitioner seeking to replicate the approach faces a fundamental ambiguity: must they use a capable teacher model (GPT-3.5 or better) to generate synthetic data, or can they achieve similar results with careful human authoring? If the distillation hypothesis is correct, then "textbooks are all you need" is misleading—what you actually need is access to a strong teacher model to generate the textbooks, which may not be available (GPT-3.5 is a proprietary model with usage costs and rate limits) or may become unavailable in the future. If the data-quality hypothesis is correct, then human-authored textbooks of comparable quality should produce similar results, and the approach is genuinely democratizing (anyone can write a textbook, even if it is labor-intensive).

The confound also affects the paper's claims about scaling. If phi-1's performance is largely attributable to distilling GPT-3.5's knowledge, then the approach may hit a ceiling at or slightly above GPT-3.5's performance level—you cannot distill capabilities the teacher model doesn't have. The paper speculates that using GPT-4 as the teacher would yield further gains, which supports the distillation interpretation more than the data-quality interpretation: if the benefit comes from textbook structure, a weaker teacher should work nearly as well as long as the structure is maintained.

What evidence exists in the paper. The paper provides no evidence that directly addresses this confound. There is no ablation varying the teacher model, no comparison of GPT-3.5-generated exercises to human-written exercises of comparable size and format, and no analysis of whether GPT-3.5's synthetic data contains reasoning patterns that are absent from the filtered web code. The only indirect evidence is the filtering ablation (Section 2.1): training on filtered web code alone (no synthetic data) achieves 17.68% on HumanEval for 350M parameters, which is substantially above the unfiltered baseline (12.19%) but still well below the full CodeTextbook result (20.12%) and far below the finetuned result (45%). This shows that filtering alone—which does not involve a teacher model—provides a meaningful benefit, but it does not separate the textbook-structure benefit from the teacher-model benefit in the synthetic data.

The paper acknowledges the teacher model's quality as a potential confound in passing:

"We also believe that significant gains could be achieved by using GPT-4 to generate the synthetic data instead of GPT-3.5, as we noticed that GPT-3.5 data has a high error rate."

This statement implies that better teacher models would produce better training data, which is consistent with the distillation hypothesis. The observation that phi-1 "is able to achieve such high coding proficiency despite those errors" (the GPT-3.5 data's errors) is interesting but doesn't resolve whether the benefit comes from structure or from distilling the correct portions of GPT-3.5's outputs.

Mitigation status. The paper does not address this confound. It frames the contribution around data quality and the textbook principle without acknowledging the alternative distillation explanation. The paper's title—"Textbooks Are All You Need"—implies that the structure and quality of the data, not its source, is sufficient, but the experiments do not support this strong interpretation because the source (GPT-3.5) is never varied. Resolving this confound would require at minimum an ablation where synthetic data is generated by a significantly weaker model (e.g., an open-source 350M code model) following the same diversity-inducing constraints, to determine whether the textbook structure alone provides benefits above and beyond the teacher model's competence.

6.6 Hard Problems and Out-of-Distribution Tasks Are Effectively Unsolved

The assumption or constraint. The paper's approach assumes that the target task distribution is well-represented by the curated training data. For phi-1, this means short, well-specified Python function-completion tasks that can be solved with basic algorithmic reasoning and standard library usage. The model shows no ability to handle harder tasks that require more complex reasoning, longer-horizon planning, or knowledge not present in the 7B-token training corpus.

The paper acknowledges this limitation in Section 6:

"our model has only 1.3B parameters trained with only 7B tokens, this restricts our model's capacity to manage more complex tasks such as developing an intricate Flask application, in comparison to other models like Starcoder."

This is not merely an observation about scale—it reflects a fundamental design tension in the textbook-quality approach. The deliberate curation that makes the training data efficient (removing non-self-contained code, filtering out complex multi-file projects, focusing on instructive examples) also removes exactly the kind of complex, messy, real-world code that would teach the model to handle larger-scale software engineering tasks. A model trained exclusively on textbook examples and synthetic exercises has never seen a multi-file project with interdependent modules, never encountered build systems or configuration management, and never practiced the kind of architectural reasoning needed for non-trivial applications.

The consequence. phi-1's capabilities have a hard ceiling determined by what can be learned from 7B tokens of curated, Python-focused, function-level data. The model is not merely "not as good" at complex tasks—it is structurally incapable of performing them because the necessary knowledge (of specific APIs, of multi-file project patterns, of domain-specific conventions) was never present in its training data. This is qualitatively different from larger models trained on web-scale data, which may perform poorly on complex tasks but at least have encountered the relevant patterns and can occasionally produce reasonable outputs. phi-1 has zero exposure to, for example, Django request handling, Flask application structure, or multi-module Python package organization, and therefore cannot even attempt these tasks.

The limitation extends to problem difficulty within the function-completion domain. The paper's difficulty analysis (Section 5, Table 3) shows that phi-1 performs substantially worse on the "non-similar" HumanEval subset (27.1–34.5%) compared to the "similar" subset (52.6–74.6%), even after aggressive pruning. This suggests that phi-1's performance is strongest on problems that share some structural or conceptual similarity with its training data, and degrades on problems that require genuinely novel reasoning patterns. The paper does not provide a difficulty breakdown analogous to the five-quintile analysis in the compute-optimal test-time scaling paper, so it is impossible to determine whether phi-1's performance on the hardest HumanEval problems is near zero (as with the hardest MATH problems in that paper) or merely reduced.

What evidence exists in the paper. Appendix B provides qualitative examples of failure modes: counting and spatial reasoning (the Tkinter button layout example where phi-1 generates an extra textfield and misplaces buttons), difficulty with ambiguous natural language, and sensitivity to prompt length. Table 3 shows a consistent performance gap between similar and non-similar HumanEval subsets across all pruning thresholds (e.g., 52.6% vs. 27.1% at τ = 0.8), indicating that problem difficulty or distance from training distribution substantially affects performance. Section 6 qualitatively acknowledges the limitation on complex tasks but provides no benchmark measuring the performance gap. Unlike the compute-optimal scaling paper, which explicitly showed pass@1 ≈ 0–5% on the hardest difficulty bin, this paper provides no quantitative characterization of the difficulty ceiling.

Mitigation status. The paper does not attempt to mitigate this limitation and treats it as inherent to the small-scale, specialized approach. The discussion in Section 6 frames it as a tradeoff:

"phi-1 is specialized in Python coding, which restricts its versatility compared to multi-language models."

The implication is that specialization is a feature, not a bug—phi-1 is designed to excel at a narrow task and should not be expected to generalize beyond it. However, the paper's title and framing ("Textbooks Are All You Need") suggest a stronger claim: that the textbook-quality approach is sufficient for coding proficiency. The hard ceiling on complex tasks contradicts this strong interpretation—textbooks are sufficient for basic function-completion tasks but not for the full scope of software engineering. The paper does not explore whether scaling the textbook approach (more topics, larger textbooks, multi-file examples, project-level exercises) could extend the capability ceiling, or whether a fundamentally different approach is needed for complex software engineering tasks.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a methodological reframing rather than a paradigm shift. It does not overturn scaling laws—the fundamental relationship between compute, data, and performance still holds—but it demonstrates that data quality is an independent axis of the scaling law with sufficient leverage to produce regime changes in efficiency. Prior to this work, data quality was understood as a constant-factor optimization: deduplication, filtering, and cleaning could improve performance by 10–30%, but the dominant strategy for reaching state-of-the-art remained "collect more data and train a bigger model." phi-1 shows that curated data can compress the required training budget by roughly two orders of magnitude (7B curated tokens vs. 1T web tokens for comparable or superior HumanEval performance), which qualitatively changes what is possible with a given compute budget.

The reframing can be stated precisely: the unit of progress in language model training is not the token, but the instructive token—a token that teaches a transferable concept or skill. Most tokens in web-scale datasets are not instructive in this sense (they are boilerplate, context-dependent, or redundant), meaning that scaling dataset size by 100× might only increase instructive content by 10× or less. The textbook-quality approach increases the density of instructive content, achieving with 7B tokens what would require ~700B–1T tokens of web data by the paper's own efficiency calculations. This reframes the optimization problem from "how do we gather more data?" to "how do we identify or generate data that teaches specific capabilities?"

The paper reconciles an apparent contradiction in the literature between the scaling hypothesis (more data → better performance, Kaplan et al., 2020; Hoffmann et al., 2022) and the observation that human experts can learn programming from a few well-chosen textbooks while models trained on orders of magnitude more web code still struggle. The resolution is that scaling laws were measured on fixed data distributions—specifically, web-scraped text and code with all their attendant noise, redundancy, and pedagogical deficiencies. Those laws describe the learning dynamics for that distribution, not a fundamental property of language or code learning. By changing the data distribution to one optimized for instruction, the paper shows that the scaling curve shifts dramatically—not just in intercept but in the relationship between tokens and capability. This does not invalidate scaling laws; it demonstrates that they are distribution-dependent and that the field's default training distribution (unfiltered web data) is far from optimal.

The work also redirects attention from architecture to data engineering as the primary lever for improving model capability at a given scale. phi-1 uses a deliberately conventional architecture (decoder-only Transformer with FlashAttention, parallel MHA/MLP, rotary embeddings—all standard components in 2023) and achieves its results through data curation alone. This suggests that, at least for the current generation of architectures, the returns to architectural innovation may be smaller than the returns to data quality improvement for reasoning-heavy tasks like code generation. This is not a claim that architecture doesn't matter—StarCoder's Fill-In-the-Middle training and Multi-Query Attention likely provide additional gains—but rather that the variance explained by data quality dominates the variance explained by architectural choices within the class of standard Transformer designs.

A more subtle shift concerns how we think about finetuning. The standard transfer learning paradigm treats pretraining as knowledge acquisition and finetuning as task specialization—the model learns general capabilities from broad data and then narrows them to a specific task. The paper's emergent capabilities evidence (Section 3) suggests a different model: pretraining stores knowledge in a format that is not readily accessible for instruction-following tasks, and finetuning reorganizes that knowledge into an accessible form, even on tasks not represented in the finetuning data. If this interpretation is correct, then finetuning is better understood as knowledge retrieval training than as task specialization—teaching the model how to find and deploy knowledge it already possesses rather than teaching it new knowledge. This would shift how practitioners design finetuning datasets: the goal is not to cover every task the model should perform, but to provide enough diverse examples of instruction-following that the model learns the meta-skill of mapping any instruction to the relevant pretrained knowledge.

Research directions that become more attractive:

• Data quality measurement and optimization as a first-class research area, analogous to how architecture search was a major focus in 2018–2021. Questions include: how do we quantify "instructiveness" of a training example? Can we train models to automatically identify high-quality training data without human annotation or expensive teacher model calls? What are the scaling laws when data quality is systematically varied?

• Synthetic data generation for curriculum learning, where the goal is not to distill a specific teacher model but to create a structured pedagogical sequence that efficiently teaches target capabilities. The paper's diversity-inducing constraints (topic constraints for textbooks, function-name constraints for exercises) are a first step; more sophisticated curriculum generation—where later examples build on concepts introduced earlier, and difficulty is progressively increased—could yield further gains.

• Small, specialized models for narrow domains, challenging the assumption that good performance requires general-purpose models. If a 1.3B model can match a 15B model on Python function completion, what other narrow domains (SQL query generation, data visualization code, shell scripting, LaTeX formatting) could be served by small, textbook-trained models?

• Understanding the mechanism of finetuning-induced emergence, since the paper provides a phenomenon (narrow finetuning unlocks broad capabilities) but no mechanistic explanation. Does finetuning change attention patterns to better attend to instruction text? Does it reorganize the feedforward layers to create more direct pathways from natural language specifications to code snippets stored from pretraining? Probing experiments (à la mechanistic interpretability) could open a new understanding of how knowledge is stored and retrieved in language models.

Research directions that become less attractive:

• Brute-force scaling of web data collection as the primary strategy for improving code models. The paper demonstrates that a 1.3B model on 7B curated tokens outperforms a 15.5B model on 1T web tokens, making the ROI of scraping ever-larger code repositories questionable if comparable (or better) results can be achieved with a fraction of the data through curation.

• Architecture search as a substitute for data quality work. phi-1's conventional architecture matching or exceeding StarCoder (which uses FIM and MQA) suggests that architectural improvements, while real, may offer smaller absolute gains than data quality improvements at current scales.

Follow-Up Research This Work Enables

Quantifying the emergent capabilities with a targeted benchmark. The paper's most provocative claim—that finetuning on narrow exercises unlocks broad capabilities like library usage and chat—is supported only by cherry-picked qualitative examples. A direct follow-up would construct a phi-1 Emergent Capabilities Benchmark with, say, 50 prompts each for five capability categories not present in CodeExercises: PyGame programming, Tkinter GUI construction, PyTorch model definition, Matplotlib visualization, and multi-turn chat about Python. Each prompt would be evaluated with a pass/fail rubric (for code: does it execute without error and produce the correct output? for chat: does it provide a helpful, accurate response?). The benchmark would be run on phi-1, phi-1-base, phi-1-small, and a baseline web-trained model at comparable scale (e.g., CodeGen-Mono-350M or a 1.3B model trained on The Stack for equivalent compute), producing the first quantitative measurement of how reliable the emergent capabilities actually are. A strong result would show phi-1 succeeding on, say, >60% of PyGame prompts vs. <20% for phi-1-base; a weak result would show sporadic successes (10–20%) that still exceed the base model but are insufficient for practical use. Either outcome advances understanding by replacing qualitative demonstration with quantitative characterization.

Varying the teacher model to disentangle data quality from distillation. The paper's central confound is that all synthetic data was generated by GPT-3.5, a model already capable of strong code generation. A critical follow-up would replicate the synthetic data pipeline using three different teacher models: (a) a weak open-source model (e.g., CodeGen-Mono-350M at 12.8% HumanEval), (b) GPT-3.5 (the original teacher, 47% HumanEval), and (c) GPT-4 (67% HumanEval, as a stronger-than-original baseline). For each teacher, generate the same quantity of textbook and exercise data using the same diversity-inducing constraints, train phi-1-equivalent models, and measure HumanEval performance. If the textbook structure drives the benefit, all three should produce models substantially above their teacher's performance (with differences attributable to the teacher's data quality). If distillation drives the benefit, model performance should track teacher performance closely, and the weak-teacher model should perform poorly regardless of textbook structure. This experiment cleanly separates the two hypotheses and determines whether the approach is accessible to practitioners without access to frontier proprietary models. The paper already notes that GPT-3.5 data "has a high error rate," making the comparison to a cleaner GPT-4 teacher particularly informative for understanding how data errors interact with the textbook-quality principle.

Training a model on human-authored textbooks to test the synthetic data necessity. The paper's title claims "Textbooks Are All You Need," but all textbooks used were synthetic. A strong test of the thesis would be to curate a dataset of genuinely human-authored Python textbooks and tutorials (from sources like "Automate the Boring Stuff with Python," official Python documentation tutorials, university course materials, and high-quality blog posts), filter them for self-containedness and instructive value using the same classifier approach, and train a model on this human-authored corpus (matched for token count to CodeTextbook) followed by human-authored exercises (e.g., from coding challenge platforms like Exercism or Codewars, decontaminated against HumanEval). If human-authored data achieves comparable performance to synthetic data, it validates the textbook-quality principle independent of teacher model quality and opens a path forward that doesn't depend on API access to proprietary models. If human-authored data substantially underperforms synthetic data, it suggests that GPT-3.5's synthetic data encodes something beyond textbook structure—perhaps a more systematic coverage of Python features, or a coding style particularly well-suited to unit-test evaluation.

Difficulty-stratified evaluation to characterize the capability ceiling. The paper provides only aggregate HumanEval scores and a similar/non-similar split based on training data proximity (Table 3). A more informative evaluation would stratify HumanEval problems by difficulty—either using the existing HumanEval difficulty annotations (if available) or by measuring pass rates across all models in Table 1 and clustering problems by how many models solve them. This would reveal whether phi-1's performance advantage is concentrated on easy-to-medium problems (where the textbook approach teaches fundamental patterns effectively) and whether it falls below larger models on the hardest problems (where broader knowledge or more complex reasoning is required). The analogous analysis in the compute-optimal test-time scaling paper's Figure 3 (right) revealed that test-time compute was most effective on medium problems and nearly useless on the hardest bin—a similarly informative pattern likely exists here. If phi-1 matches StarCoder on hard problems as well as easy ones, it strengthens the textbook hypothesis. If phi-1 excels on easy problems but falls behind on hard ones, it suggests that the textbook approach efficiently teaches fundamentals but that some capabilities genuinely require larger scale.

Measuring robustness degradation quantitatively across perturbation types. Appendix B provides qualitative examples of phi-1's brittleness but no quantification. A systematic robustness evaluation would define perturbation types—grammatical errors (subject-verb disagreement, missing articles), synonym substitution ("unchanged" → "stays the same"), prompt length (varying the number of specification clauses), reordering of instructions, and addition of irrelevant context—and measure the degradation in pass@1 on a fixed set of prompts (either HumanEval problems or the unconventional problems from Section 4) as a function of perturbation type and magnitude. This would produce a robustness surface showing which perturbations are most damaging and at what magnitude performance drops below a usability threshold. The result would inform whether phi-1 can be deployed without prompt normalization (if degradation is modest) or requires a preprocessing step (if degradation is severe). It would also reveal whether the brittleness is a necessary consequence of the textbook approach (because clean training data creates fragile representations) or an incidental consequence of limited data diversity that could be addressed by augmenting the training set with perturbed examples.

Extending the textbook approach to a second domain to test generality. The paper's results are confined to Python code generation. A crucial test of generality would be to replicate the entire pipeline—filter existing data for educational value, generate synthetic textbooks with diversity constraints, generate synthetic exercises with function-name constraints, pretrain and finetune—for a different reasoning-heavy domain with existing benchmarks. Obvious candidates include: (a) SQL query generation (using the Spider benchmark), where "textbook quality" would mean self-contained database schemas with clear natural language questions and correct SQL solutions; (b) mathematical proof generation (using NaturalProofs or miniF2F), where textbooks would teach proof techniques with interleaved natural language and formal statements; (c) data visualization code (using a benchmark of Vega-Lite or Matplotlib specifications), where textbooks would explain visualization principles alongside code examples. Success in a second domain—achieving competitive performance with a small model trained on curated data—would establish the textbook approach as a general methodology rather than a one-off success in Python code. Failure would bound the approach's applicability to domains where a strong teacher model (GPT-3.5) already excels and can generate high-quality synthetic data.

Training a larger model on the same curated data to probe scaling limits. The paper shows that a 1.3B model on curated data outperforms a 15B model on web data, but it does not train a 7B or 15B model on the curated data to determine whether the benefit continues to scale or plateaus. A natural follow-up would train phi-1-equivalent models at 350M (already done), 1.3B (already done), 3B, 7B, and possibly 13B parameters—all on the same CodeTextbook + CodeExercises pipeline—and measure HumanEval at each scale. This would produce a data-quality-conditioned scaling curve that could be directly compared to the web-data scaling curve implicitly traced by models in Table 1 (CodeGen-Mono-350M at 12.8%, CodeGen-Mono-16.1B at 29.3%, StarCoder-15.5B at 33.6%, GPT-3.5 at 47%, GPT-4 at 67%). If the curated-data curve shows a steeper slope (larger gains per parameter doubling) than the web-data curve, it would confirm that data quality changes the shape of scaling laws, not just their intercept. If the curves are parallel (same slope, higher intercept), it would suggest that data quality provides a constant-factor benefit that can be overcome by sufficient scale on web data—a still-valuable finding that would bound the textbook approach's ultimate advantage.

Practical Applications and Downstream Use Cases

On-device or edge-deployed coding assistants. The most direct application of phi-1's efficiency is deploying a capable code generation model on consumer hardware where a 15B-parameter model cannot run. A 1.3B-parameter model can be quantized to 4-bit precision (~700MB) and run on a laptop CPU or a mid-range smartphone with acceptable latency (hundreds of milliseconds per token). For an IDE plugin that suggests function completions based on docstrings, phi-1's 50.6% HumanEval means it can correctly complete roughly half of well-specified Python functions on the first attempt—a useful level of assistance. The alternative (calling a cloud API for GPT-3.5 or StarCoder) introduces latency, requires network connectivity, and incurs per-query costs. A local phi-1-derived model could provide instant, free, offline suggestions for Python developers, with the understanding that it works best on short, clearly specified functions—exactly the kind of completions that benefit most from low latency. The environmental benefit the paper claims (reducing the carbon cost of LLM inference) is realized most directly in this scenario, where a small efficient model displaces cloud inference entirely.

Synthetic training data generation for specialized coding tasks. Organizations that need to train code models for domain-specific tasks (e.g., generating data processing pipelines for a particular industry, writing test cases for an internal API, producing configuration files for a proprietary system) can adopt the paper's pipeline directly: use a strong general-purpose model (GPT-3.5 or GPT-4) to generate domain-specific textbooks and exercises following the diversity-inducing constraint approach, then train a small phi-1-scale model on this synthetic data. The key insight from the paper is that the synthetic data volume needed is small (7B tokens total, 180M tokens for finetuning), making the approach economically feasible even for narrow enterprise applications. For example, a financial services company could generate 100M tokens of synthetic exercises for writing regulatory compliance checks in Python, train a 350M-parameter model in hours on a single GPU, and deploy a specialized model that outperforms general-purpose coding assistants on that specific task. The decontamination methodology (Section 5) provides a template for ensuring the synthetic data doesn't simply reproduce evaluation examples.

Curriculum design for programming education. The paper's methodology—identifying what makes code instructive, filtering existing repositories for educational value, generating synthetic exercises with controlled diversity—has direct applications to human education, not just model training. A platform teaching introductory Python could use the same GPT-4-based quality classifier to filter open-source code repositories for examples suitable for students, and could use the same diversity-inducing constraints to generate problem sets that systematically cover Python concepts without repetition. The paper's finding that finetuning on 180M tokens of exercises produces the largest performance jump suggests that a human curriculum might similarly benefit from a large, diverse set of practice problems even if the explanatory content (textbooks) is relatively compact. The specific constraint mechanisms—varying topics and target audience for textbooks, constraining function names for exercises—translate directly to educational content generation prompts.

Cost-efficient benchmarking and model comparison. The paper's data pruning methodology (embedding + AST similarity for decontamination, Section 5) provides a practical tool for any team training models on synthetic data. As LLM-generated training data becomes more common (Alpaca, Orca, Self-Instruct derivatives), the risk of inadvertently including evaluation-like examples increases. The paper demonstrates that n-gram overlap (the standard decontamination method) fails for code—only 4 HumanEval problems showed 13-gram overlap, and all were false positives. The embedding + AST approach is more expensive than n-gram matching but much cheaper than training and evaluating a model to detect contamination effects, and it can be run once per synthetic dataset. Teams generating synthetic training data can adopt this methodology as a standard preprocessing step, reporting performance both on the full dataset and on aggressively pruned versions (as in Table 3) to give users confidence that results reflect generalization rather than memorization.