RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation

🎯 Pitch

ZeroRepo uses a structured graph blueprint instead of ambiguous natural language to plan entire codebases, producing repositories 3.9× larger than Claude Code with 35.8-point higher test accuracy. This is the first system to achieve near-linear scaling in code generation—natural-language plans stagnate by iteration 15, while the graph-driven approach keeps growing both features and code volume linearly past iteration 30.

1. Executive Summary

This paper introduces the Repository Planning Graph (RPG), a structured graph-based representation that unifies proposal-level planning—deciding what features and modules to build—with implementation-level planning— specifying file structures, data flows, and function interfaces—for generating complete code repositories from high-level specifications. Building on RPG, the authors develop ZeroRepo, a graph-driven framework that constructs the RPG through explore–exploit subtree selection (exploiting a 1.5M-node feature ontology to retrieve aligned capabilities, exploring unvisited ontology regions for diverse coverage) and implementation-level encoding (mapping subgraphs to file skeletons and encoding inter-module data flows as typed edges), then generates code via topological traversal with test-driven development. On the RepoCraft benchmark of six real-world projects with 1,052 tasks, ZeroRepo achieves 81.5% functional coverage and 69.7% test accuracy, improving over the strongest baseline—Claude Code—by 27.3 and 35.8 points respectively, while producing repositories averaging 36K lines of code and 445K tokens (~3.9× larger than Claude Code and ~68× larger than other baselines). Analysis shows that RPG supports near-linear scaling of functionality and code size with iteration count, establishing that structured graph representations enable sustained long-horizon planning only when the planning medium is persistently structured—natural-language baselines plateau or stagnate by iteration 10–15, while ZeroRepo continues to grow both feature count and code volume linearly through iteration 30.

2. Context and Motivation

The Core Problem: Bridging High-Level Intent and Repository-Scale Implementation

The fundamental challenge this paper addresses is deceptively simple to state but extraordinarily difficult to solve: how do we get language models to generate complete, coherent software repositories from high-level natural language descriptions? Current LLMs reliably produce individual functions or single files of code (Zhu et al., 2024; Wang et al., 2025; Liu et al., 2025; Zeng et al., 2025), but scaling from "write a function that sorts a list" to "build me a machine learning library with data preprocessing, model training, evaluation, and visualization" introduces a qualitatively different set of challenges. The paper identifies the central difficulty as bridging the gap between high-level user intent and "the repository's intricate network of files, classes, and dependencies" (Section 1).

This gap demands progressive planning at two distinct levels:

Proposal-level planning: deciding what to build—defining the functional scope, enumerating required capabilities, and partitioning them into coherent modules. For a machine learning library, this means determining that you need data loading, preprocessing, supervised learning algorithms, unsupervised methods, model evaluation metrics, and visualization tools, then deciding how these capabilities group into modules.
Implementation-level planning: deciding how to build it—specifying the file structure, designing interfaces between modules, defining data flows, and mapping capabilities to concrete classes and functions. This means deciding that LinearRegression and RidgeRegression go in src/algorithms/regression/linear_models.py, that they share a BaseEstimator interface with fit() and predict() methods, and that training data flows from the data loading module through preprocessing to the algorithms module.

The difficulty is not simply that the problem is "large"—it is that these two levels of planning must remain mutually consistent as the repository grows. A decision at the implementation level (e.g., splitting algorithms across multiple files) can invalidate proposal-level assumptions about module boundaries, while a change at the proposal level (e.g., adding a new algorithm category) requires restructuring the implementation plan. Current approaches struggle with this coupling precisely because their planning medium does not enforce consistency.

Why This Problem Matters

Real-world impact. The paper argues that generating complete repositories from scratch represents a capability threshold for automated software engineering. Function-level and file-level generation, while useful, still require human developers to handle architecture, integration, and cross-module consistency—the parts of software development that are most cognitively demanding and error-prone. Fully automated repository generation would transform how software prototypes are built, how domain-specific libraries are created, and how technical specifications are translated into working code. The paper specifically frames this as key to "realizing the full potential of automated code generation" (Section 1).

Theoretical significance. Repository-scale generation is fundamentally a long-horizon planning problem where the planning horizon can span hundreds or thousands of interdependent decisions (which files to create, which functions go where, what interfaces they expose, how data flows between them). The paper's central theoretical claim is that the representation used for planning—not just the underlying model's reasoning capability—determines whether planning can scale. Natural language, despite its flexibility, introduces ambiguity that compounds over long horizons; structured representations, by encoding constraints explicitly, prevent this compounding. This connects to broader debates in AI about whether symbolic structure is necessary for complex reasoning, or whether neural networks can learn to maintain consistency purely through latent representations.

Self-improvement pipelines. The authors envision a future where models generate repositories, validate them through testing, and then use the validated code as training data for further improvement. For such pipelines to work, the generated repositories must be not just functional but structured—maintainable, modular, and consistent—since disorganized code provides poor training signal. The planning representation therefore affects not just the immediate generation quality but the viability of iterative self-improvement.

Prior Approaches and Their Limitations

The paper identifies three broad paradigms of prior work, all of which share a critical dependency on natural language as the planning medium.

Paradigm 1: Distributed Multi-Agent Planning Frameworks

Systems like MetaGPT (Hong et al.) and ChatDev (Qian et al., 2024) assign specialized roles—such as product manager, architect, engineer, and tester—to different LLM agents that negotiate between requirements and implementations through natural language dialogue. The idea is that role specialization forces each agent to focus on its domain (e.g., the architect thinks about structure, the engineer about implementation details), and the inter-agent dialogue serves as a form of planning communication.

Where they fall short. The paper identifies that these approaches produce plans that are "ephemeral"—they exist only in the dialogue history and are never formalized into a persistent, queryable structure. As planning proceeds through multiple rounds of agent interaction, the specifications can become inconsistent: different agents may interpret the same requirement differently, later decisions may not reflect earlier constraints, and there is no mechanism to enforce that the final implementation actually realizes the agreed-upon architecture. The paper notes that these limitations "can more easily lead to unstable proposal-level planning, where functionalities are sometimes incomplete, overlapping, or unevenly scoped" (Section 1). The experimental results bear this out: on RepoCraft, MetaGPT achieves only 16.6% coverage with o3-mini (Table 2), and ChatDev reaches only 18.3% coverage, with pass rates below 7%.

Paradigm 2: Fixed Workflow Pipelines

Systems like Paper2Code (Seo et al., 2025) and AutoP2C (Lin et al., 2025) impose a rigid, pre-defined pipeline: first extract or specify an architectural skeleton, then fill in implementation details stage by stage. This addresses the consistency problem of multi-agent approaches by enforcing a fixed order of operations—you cannot implement before you architect—but at a significant cost in flexibility.

Where they fall short. The paper identifies two related failures. First, fixed pipelines cannot adapt to the specific needs of different repositories: a machine learning library and an HTTP client library have fundamentally different architectural patterns, but a fixed pipeline treats them identically. Second, the pipeline stages themselves rely on natural language to communicate between stages, meaning that "fragmented implementation-level planning" still occurs: "plans drift across iterations, introducing inconsistencies in dependencies, data flows, and modular boundaries" (Section 1). Empirically, Paper2Code achieves 21.7–30.2% coverage depending on the backbone model (Table 2), substantially better than multi-agent systems but still far below the 81.5% achieved by ZeroRepo.

Paradigm 3: Iterative Terminal Agents

The most recent and strongest baselines are terminal-based agents: OpenHands (Wang et al.), Claude Code (Anthropic, 2025b), Gemini CLI (Google, 2025), and Codex CLI (OpenAI, 2025). These agents interact with a codebase through a terminal interface, externalizing intermediate plans (often in markdown files), implementing changes, running tests, and iteratively refining based on results. The key advantage is that the plan is externalized—written to files that persist across iterations—rather than existing only in dialogue history. This provides a form of persistent memory that multi-agent systems lack.

Where they fall short—and this is the paper's central critique. Despite externalizing plans, terminal agents still use natural language (typically markdown documents) as the planning medium. The paper argues that natural language, while flexible and human-readable, is fundamentally limited for large-scale repository planning in three specific ways:

Ambiguity blurs intent and constraints. Natural language descriptions like "the data loading module provides training data to the algorithms" do not specify what format the data takes, which algorithms receive it, or what contract must be satisfied. Different parts of the plan can interpret the same natural language differently, leading to interface mismatches that are only discovered at implementation time.
Lack of explicit hierarchy makes dependency tracking difficult. When planning in natural language, dependencies between components are described in prose ("the evaluation module imports predictions from the model and ground truth from the data loader"), but there is no formal graph that can be traversed to determine implementation order, detect cycles, or verify that all dependencies are satisfied. This echoes findings from Besta et al. (2024) that natural language struggles with complex dependency structures that graphs handle naturally.
Static plans degrade over long horizons without adaptive adjustment. Natural language plans written at iteration 5 may not be updated to reflect decisions made at iteration 20. Since there is no formal consistency check, contradictions accumulate silently. The paper cites Sun et al. (2023) on the general problem of plan degradation in language-based systems.

The empirical evidence for these failures is stark. Figure 5 shows that natural-language-based terminal agents scale poorly with iteration count: Claude Code grows feature counts steadily but with diminishing returns, Gemini CLI increases slowly before converging by iteration 30, and Codex CLI essentially stops proposing new features after 4–5 iterations. Figure 6 shows the same pattern in code volume: Claude Code and Gemini CLI plateau at 3–4K LOC, while Codex stays below 1K. This is not a computational limitation—these agents are given the same number of iterations and access to web search—but a representational one: the natural-language planning medium cannot sustain consistent expansion beyond a certain complexity threshold.

How This Paper Positions Itself

The paper does not propose a fundamentally new planning algorithm or a better language model. Its contribution is architectural: replace the natural-language planning medium with a structured graph representation that encodes both proposal and implementation information in a single, persistent, machine-interpretable format. This is the Repository Planning Graph (RPG).

The key insight is that the planning representation is not a neutral implementation detail—it is the primary determinant of whether planning can scale. Natural language is flexible but fails at long horizons because its lack of formal structure allows inconsistencies to accumulate silently. A graph representation succeeds because:

Nodes explicitly encode capabilities, files, and functions with well-defined semantics. A node in the RPG is not a vague description; it carries dual semantics: at the functional level, it represents a specific capability (e.g., "Lasso Regression"); at the structural level, it maps to a concrete code entity (e.g., the LassoRegression class in src/algorithms/regression/linear_models/lasso.py).
Edges encode explicit, typed relationships. Inter-module edges capture data flows (e.g., "training data" flows from Data Loading to ML Algorithms); intra-module edges capture file-level ordering and function-level dependencies (e.g., BaseEstimator must be defined before LassoRegression which inherits from it). These edges are not just documentation—they impose a topological ordering that the code generation stage follows to ensure dependencies are satisfied.
The graph is persistent and evolvable. As planning progresses, the graph is incrementally expanded and refined. A new feature proposal is not just added to a prose document—it is inserted into the graph at the appropriate location, with edges connecting it to existing nodes. This ensures that new additions are always placed in the context of existing decisions, preventing fragmentation.

The paper explicitly contrasts this with the three prior paradigms: multi-agent systems produce ephemeral plans that degrade; pipeline systems produce rigid plans that cannot adapt; terminal agents produce unstructured plans that do not scale. RPG provides structure, persistence, and explicit constraints, which together enable the near-linear scaling observed in Figures 5 and 6.

Positioning relative to verifier-based and search-based approaches. An interesting parallel: just as the test-time compute scaling paper analyzed in the prior sections found that the selection mechanism (verifier quality, search algorithm) determines scaling behavior, this paper finds that the planning representation (graph vs. natural language) determines repository generation scaling. In both cases, the bottleneck is not the underlying model's capability but the structure of the intermediate representation that guides decision-making.

The role of the EpiCoder Feature Tree. The paper does not claim that LLMs cannot plan at all—rather, they are unstable and biased when planning from scratch. The 1.5M-node EpiCoder Feature Tree (Wang et al., 2025) serves as a structured prior that mitigates this instability. It is an ontology of software capabilities organized hierarchically (e.g., "Machine Learning → Supervised Learning → Linear Models → Lasso Regression"). By retrieving from this ontology rather than asking the LLM to enumerate capabilities from scratch, the paper's approach avoids the "incomplete coverage" and "randomness" that plague purely generative planning. The ablation in Section 8 (Figures 7–8, Table 6) is crucial here: removing the Feature Tree slows planning but does not prevent it—ZeroRepo without the Feature Tree still reaches 87.2% coverage and 25K LOC at iteration 30, and continues to 97.9% coverage at iteration 40. This demonstrates that the RPG structure itself, not just the ontology, is what enables scaling. The ontology serves as an accelerator—it provides better initial context and faster growth—but the graph structure is the fundamental enabler.

Connection to broader AI planning debates. The paper connects implicitly to a long-standing tension in AI between symbolic and neural approaches to planning. Prior work by Valmeekam et al. (2023) showed that LLMs struggle with planning tasks that require maintaining consistent state across multiple steps—precisely the kind of consistency that repository generation demands. Besta et al. (2024) showed that graph-structured representations ("Graph of Thoughts") can outperform natural language for complex reasoning. This paper extends that insight to code generation, demonstrating that the benefits compound as the planning horizon lengthens: the difference between graph-based and language-based planning at iteration 5 is modest, but by iteration 30 it is a factor of 10× or more in both feature count and code volume.

The Evaluation Gap

Finally, the paper is motivated by a gap in evaluation infrastructure. Prior benchmarks for repository-level code generation (Zhao et al., 2025; Starace et al., 2025) either focus on incremental changes to existing repositories or provide detailed skeletons and specifications that limit the scope of autonomous planning. There was no benchmark that required agents to start from a high-level description and build a complete repository while being evaluated against real-world projects on coverage, correctness, and scale. RepoCraft fills this gap by:

Providing six real-world reference repositories (scikit-learn, pandas, sympy, statsmodels, requests, django) with anonymized names to prevent training data leakage.
Deriving 1,052 evaluation tasks from actual test suites, organized hierarchically by module.
Measuring both coverage (did the agent build the right features?) and correctness (do those features actually work against adapted test cases?).
Including novelty as a metric to capture whether agents can propose coherent extensions beyond reference implementations, distinguishing between memorization and genuine planning.

This evaluation design is motivated by the observation that simple metrics like lines of code or file count are insufficient—they reward verbose but incorrect implementations. The three-dimensional evaluation (coverage, correctness, scale) ensures that improvements in any single dimension cannot mask failures in others, which is critical for fairly comparing structured planning approaches (like ZeroRepo) against natural-language approaches that may produce superficially large but internally inconsistent codebases.

3. Technical Approach

This is primarily a systems architecture paper whose core idea is that replacing natural language as the planning medium with a structured, persistent graph representation—the Repository Planning Graph (RPG)—enables long-horizon, large-scale repository generation by enforcing explicit constraints, dependencies, and modular boundaries throughout the planning and implementation process.

3.1 Reader Orientation

ZeroRepo is a graph-driven code generation framework that takes a high-level natural language description of a desired software repository (e.g., "build me a machine learning library") and produces a complete, working codebase with files, classes, functions, and tests. The system solves the problem of planning consistency at scale: rather than planning in free-form natural language—which becomes inconsistent and fragmented over long horizons—ZeroRepo constructs a structured Repository Planning Graph (RPG) that encodes both what to build (functional capabilities and modules) and how to build it (file structures, data flows, function interfaces) in a single, persistent, machine-traversable representation. The RPG then serves as a blueprint that guides code generation in topological order, ensuring dependencies are satisfied before dependents are implemented.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that operate in sequence, with the RPG serving as the central representation that accumulates information across stages:

EpiCoder Feature Tree (Knowledge Base) — a pre-built global ontology of 1.5 million software capabilities organized hierarchically across seven levels (e.g., "Machine Learning → Supervised Learning → Linear Models → Lasso Regression"). Each node is embedded in a vector space for retrieval. This provides a structured prior that stabilizes LLM-based planning by constraining capability enumeration to an existing taxonomy rather than requiring the LLM to generate capabilities from scratch.
Proposal-Level Construction (Stage A) — translates the user's high-level specification into a functionality graph: a tree-structured representation where nodes capture progressively refined capabilities (root nodes = high-level modules like "ML Algorithms," intermediate nodes = sub-modules like "Regression," leaf nodes = concrete algorithms like "Lasso Regression"). This stage uses an explore–exploit search over the Feature Tree (exploiting retrieval to find aligned capabilities, exploring unvisited ontology regions for diverse coverage) followed by LLM-guided refactoring to reorganize the retrieved subtree into a modular graph aligned with the user's goals.
Implementation-Level Construction (Stage B) — enriches the functionality graph into the full Repository Planning Graph (RPG) through three substages:
- File Structure Encoding (B.1): maps functional subgraphs to concrete folder and file layouts (e.g., the "Regression" subgraph maps to src/algorithms/regression/linear_models.py), producing a file-augmented graph.
- Data Flow and Functions Encoding (B.2): adds typed inter-module data flow edges (e.g., "training data" flows from Data Loading to ML Algorithms), abstracts shared interfaces into base classes (e.g., BaseEstimator with fit() and predict() methods), and assigns leaf nodes to concrete functions or classes (e.g., load_csv and load_json group into a DataLoader class).
Graph-Guided Code Generation (Stage C) — traverses the RPG in topological order (dependencies before dependents), and at each leaf node, applies test-driven development: tests are derived from the specification, the corresponding function or class is implemented, tests are executed, and failing cases trigger iterative debugging via a graph-guided localization-and-editing workflow. Only implementations that pass all tests are committed, enabling incremental expansion while preserving stability.

Information flows sequentially: user query → Feature Tree retrieval → functionality graph → file-augmented graph → full RPG with data flows and interfaces → topological code generation → validated repository. The RPG is the persistent artifact that all subsequent stages read from and write to—it is never discarded or replaced by natural language.

3.3 Roadmap for the Deep Dive

First, the Repository Planning Graph (RPG) structure itself — what nodes and edges represent, how they carry dual functional/structural semantics, and why this dual encoding matters. This is the foundational data structure that everything else builds upon.
Second, the Proposal-Level Construction stage — how the system goes from a user query to a functionality graph, including the EpiCoder Feature Tree as knowledge base, the explore–exploit subtree selection algorithm, and the refactoring step that reorganizes the retrieved subtree into modular subgraphs aligned with software engineering principles.
Third, the Implementation-Level Construction stage — how the functionality graph is enriched in three substeps: file structure encoding (mapping subgraphs to folders and files), data flow encoding (adding typed inter-module edges and intra-module ordering), and function/interface encoding (abstracting shared base classes and assigning leaf capabilities to concrete functions or classes).
Fourth, the Graph-Guided Code Generation stage — how the RPG is traversed topologically to generate code, including the test-driven development loop, the graph-guided localization tools (RPG-Guided search, repository code view, dependency exploration), and the multi-level testing strategy (unit, integration, regression tests).
Fifth, key design choices and their justifications — why graph over natural language, why the Feature Tree as prior, why dual semantics on nodes, why topological traversal, and why test-driven development.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Repository Planning Graph (RPG) Structure

The RPG is the central data structure of the entire system. It is a directed graph where nodes carry dual semantics and edges encode explicit, typed relationships across three levels of granularity: module-level, file-level, and function-level. The paper describes this dual encoding in Section 3.1 and illustrates it in Figure 2 (which shows a partial RPG for a machine learning library).

Nodes: dual functional and structural semantics. Each node in the RPG simultaneously represents two things:

At the functional level, the node represents a progressively refined software capability. The hierarchy mirrors how a human engineer would decompose a problem: high-level root nodes correspond to broad modules (e.g., "ML Algorithms," "Data Loading," "Evaluation"), which decompose into mid-level intermediate nodes (e.g., "Regression," "Classification" under "ML Algorithms"), which further decompose into leaf nodes representing concrete, implementable capabilities (e.g., "Linear Regression," "Lasso Regression," "Elastic Net" under "Regression"). The paper explicitly states that this hierarchy is not just a taxonomy—it captures the decomposition logic that a developer would use when scoping the project.
At the structural level, the same nodes map to concrete code organization: root nodes align with directory namespaces (e.g., src/algorithms/), intermediate nodes align with files (e.g., linear_models.py), and leaf nodes align with specific functions, classes, or methods (e.g., the LassoRegression class in linear_models.py). This mapping is not metaphorical—it is the actual blueprint that the code generation stage follows. The paper emphasizes that this dual encoding "unif[ies] functional decomposition with code structure" (Section 3.1).

Why dual semantics matter: in a natural-language plan, the description "we need Lasso regression in the algorithms module" is ambiguous—it does not specify which file, which class, or how it relates to other regression methods. The RPG resolves this ambiguity by grounding every functional capability in a specific structural location. When the code generation stage reaches a leaf node, it does not need to interpret a prose description to decide where to put the code—the node's structural semantics already specify the target file, the containing class (if any), and the expected interface.

Edges: three types of explicit dependencies. The RPG uses edges to capture relationships that natural language plans can only imply:

Hierarchical (parent-child) edges capture the decomposition relationship: "ML Algorithms contains Regression, which contains Linear Regression." These are implicit in the tree structure but are explicitly traversed during planning and code generation to ensure that parent modules are established before children are implemented.
Inter-module edges (shown as black arrows in Figure 2) encode data flows between subgraph roots—between high-level modules. For example, an edge from "Data Loading" to "ML Algorithms" carries the label "Training Data / Feature Matrix," specifying that the data loading module produces feature matrices that the algorithms module consumes. The paper states these edges are "typed input–output flows" (Section 3.3.2), meaning the data type is specified (e.g., "array of training data," "prediction array") and constrains the interface design: the producing module must output data of the specified type, and the consuming module must accept it.
Intra-module edges (shown as gray dashed arrows in Figure 2) capture file-level and function-level ordering within a module. For instance, within "Data Loading," the edge from load_data.py to preprocess.py specifies that load_data.py must be implemented first because its outputs feed into preprocessing. At the function level, edges capture inheritance (e.g., LassoRegression inherits from BaseEstimator) and invocation (e.g., plot_roc_curve calls silhouette_score).

What the edges enable operationally. The set of all edges collectively imposes a topological ordering on the graph—a partial order that specifies which nodes must be implemented before which others. Dependencies must precede dependents; base classes must be defined before derived classes; data producers must be implemented before data consumers. The code generation stage (Section 4) traverses the graph in this topological order, ensuring that at each step, all prerequisites are already satisfied. This is impossible in a natural-language plan because there is no formal dependency graph to traverse.

Why graph over alternatives. The paper's choice of a graph rather than a tree or a flat list is motivated by the need to represent cross-cutting relationships. A tree captures hierarchical decomposition well but cannot represent data flows between modules that are siblings in the tree (e.g., Data Loading → ML Algorithms). A flat list of features captures enumeration but loses all structure. A graph with typed edges captures both hierarchy (parent-child) and cross-cutting relationships (inter-module data flows, function-level dependencies). The paper also notes that the graph is persistent and evolvable: as planning proceeds across iterations, new nodes and edges are added incrementally without invalidating the existing structure, because each addition is grounded in explicit connections to existing nodes.

3.4.2 Proposal-Level Construction

Proposal-level construction is the process of translating a user's high-level natural language specification into a functionality graph—a tree-structured graph where nodes represent capabilities and edges represent decomposition. This stage operates in three steps, detailed in Section 3.2, Algorithm 2, and Appendices B.1–B.3.

Step 1: The EpiCoder Feature Tree as Knowledge Base

The first design choice is that ZeroRepo does not ask the LLM to enumerate capabilities from scratch. The paper argues this is because "LLMs alone provide unstable and biased capability enumeration, often with incomplete coverage" (Section 3.2), citing Valmeekam et al. (2023) and Armony et al. (2025) as evidence that LLMs struggle with exhaustive, unbiased planning. Instead, ZeroRepo uses the EpiCoder Feature Tree (Wang et al., 2025) as a structured prior—a pre-built ontology of 1.5 million software capabilities organized hierarchically across seven levels.

The Feature Tree's statistics are given in Table 7 (Appendix B.2): Level 1 contains 17 top-level categories (e.g., "functionality," "data structures," "data processing"), Level 2 contains 1,527 subcategories (e.g., "text processing," "process monitoring"), Level 3 contains 21,739 capabilities, and so on down to Level 7 with 781 leaf capabilities. The distribution across Level-1 categories is highly skewed (Figure 10)—the "data processing" branch dominates with the majority of nodes, while specialized branches like "user interaction" contain far fewer nodes. This skew reflects the real-world distribution of software capabilities: data processing utilities are disproportionately common in open-source code, while domain-specific capabilities are rarer and more concentrated.

Each node in the Feature Tree is embedded in a vector space using the infly/inf-retriever-v1 model (Yang et al., 2025), with its hierarchical path stored as metadata in a vector index. This preserves both semantic similarity (nodes with similar embeddings are retrieved together) and structural context (the full path like "Machine Learning / Supervised Learning / Linear Models / Lasso Regression" is available for downstream filtering).

Step 2: Explore–Exploit Subtree Selection

Given the user's repository description, the system constructs a repository-aligned subtree—a subset of the global Feature Tree that captures capabilities relevant to the target repository. Exhaustive enumeration of all 1.5M nodes is infeasible, so the system uses an iterative explore–exploit strategy, formalized in Algorithm 2 (Appendix B.1).

The algorithm runs for a fixed budget of 30 iterations (Section 5.3). At each iteration, two candidate pools are generated:

Exploitation candidates are retrieved via top-k semantic search over the vector index, using the user's repository description as the query. This retrieves the most semantically aligned feature paths (e.g., for a machine learning library, paths containing "linear regression," "clustering," "cross-validation"). The paper's prompt templates (Appendix B.3) show that the exploitation step also augments queries with keywords suggested by the LLM to improve recall.
Exploration candidates are sampled from unvisited regions of the ontology using a diversity-aware rejection sampling procedure (Algorithm 1, Appendix B.1). The procedure extends the base sampling strategy from Wang et al. (2025) by incorporating a rejection mechanism: at each sampling step, a candidate subtree is sampled from the Feature Tree following a temperature-transformed probability distribution over child nodes (where probabilities are derived from a frequency library F that records how often each node appears in code). The candidate is accepted only if its overlap with previously sampled nodes is below a specified threshold ρ; otherwise, the algorithm retries up to T_max times and falls back to the least-overlapping candidate. This encourages exploration of diverse, previously unseen regions of the ontology.

Both candidate pools are passed to an LLM agent that filters and ranks candidates. The exploitation prompt (Appendix B.3) instructs the agent to "select exclusively from the Exploit Feature Tree" and to "include all non-duplicated, useful paths" while excluding "generic infra (e.g., logging, configuration)" and "abstract goals." The exploration prompt instructs the agent to select "actionable, domain-relevant features" from the exploration tree, noting that "slight over-inclusion is acceptable" to maintain diversity.

Additionally, at each iteration, the LLM proposes missing features—capabilities not present in the Feature Tree but expected in a production-grade repository of the target domain (e.g., a specialized algorithm that the ontology does not cover). The prompt for missing features (Appendix B.3) instructs the agent to identify "groups of functionally concrete features that are entirely missing or only superficially represented" and to organize them into hierarchies of up to 4–5 levels with "3–5 lowercase words" per node name.

All accepted candidates from exploitation, exploration, and missing-feature proposals are filtered through a batch self-check (Algorithm 2, lines 11–15): candidates are processed in small batches (batch size B), and the LLM checks each batch for consistency with the current repository tree, accepting only paths that are "consistent/relevant" and rejecting duplicates, overly generic paths, or paths that contradict existing entries. Accepted paths are inserted into the evolving tree, and all evaluated paths are marked as visited to prevent re-exploration.

The paper reports that this iterative process produces near-linear growth in feature count (Figure 5): ZeroRepo with o3-mini surpasses 1,100 leaf features by iteration 30, while natural-language baselines plateau at much lower counts (Claude Code at ~200, Gemini CLI at ~130, Codex at <50). Figure 9 (Appendix B.2) breaks this down per repository and per model, showing that qwen3-coder exhibits "the most open expansion" with approximately linear increase, while o3-mini follows a "moderately aggressive trajectory" that balances breadth and relevance. These curves represent "different points on the recall–precision spectrum" that can be matched to repository needs.

Step 3: Refactoring by Goal Alignment

The repository-aligned subtree, while capturing relevant capabilities, still inherits the generic organization of the global ontology. For instance, capabilities related to evaluation metrics might be scattered across multiple branches of the Feature Tree rather than grouped under a dedicated "Evaluation" module. The refactoring step (Section 3.2, final paragraph; detailed in Appendix B.1 under "Repository Subtree Reorganization into the functionality graph") reorganizes the subtree into a functionality graph that follows software engineering principles of cohesion and coupling.

The reorganization operates in three stages. First, an LLM agent iteratively extracts meaningful features from the subtree, organizing them into subgraphs (coherent groups of related capabilities) until sufficient coverage of leaf nodes is reached. Second, the agent reorganizes subgraphs by merging semantically related components or moving branches across groups to improve structure—for example, moving "silhouette score" from a clustering-specific branch to a general "Evaluation" subgraph. Third, each subgraph is refined for naming consistency and hierarchical coherence.

The result is a functionality graph where each subgraph represents a cohesive module (e.g., "ML Algorithms," "Data Processing," "Model Evaluation") with clear semantic boundaries. Figures 13a and 13b (Appendix B.2) show the final distribution of leaf nodes across subgraphs for all six repositories and both models. The distributions are markedly skewed: a small number of core subgraphs absorb the majority of features, while peripheral subgraphs remain lightweight. For example, in MLKit-Py with o3-mini (Figure 13a), the "MLAlgos" subgraph contains ~300 leaf nodes, while "UIViz" contains fewer than 20. This reflects a natural modularization where dominant clusters correspond to central capabilities and minor clusters capture auxiliary functions.

The paper notes an important model-dependent effect: qwen3-coder "produces a larger number of medium-sized subgraphs, favoring breadth and parallel coverage, whereas o3-mini yields a more balanced distribution, with several subgraphs of comparable size anchoring distinct semantic roles." This indicates that the refactoring stage is sensitive to the underlying LLM's organizational preferences, and the choice of model affects the granularity of modular decomposition.

What this stage produces. The output is a functionality graph—a tree-structured graph where nodes represent capabilities at multiple levels of granularity, edges represent decomposition, and subgraphs represent modular boundaries. This graph encodes the proposal-level plan: what features will be built and how they are grouped into modules. It does not yet specify file structures, data flows, or concrete function interfaces—that is the job of the next stage.

3.4.3 Implementation-Level Construction

Implementation-level construction (Section 3.3) transforms the abstract functionality graph into the full Repository Planning Graph (RPG) by enriching it with file structures, data flows, and concrete function/class specifications. This stage operates in three substages, each adding a layer of implementation detail to the graph.

Substage B.1: File Structure Encoding

While the functionality graph defines modular boundaries (e.g., "there is an ML Algorithms module containing Regression and Classification"), it says nothing about how these modules map to actual files and directories. File structure encoding (Section 3.3.1) bridges this gap by instantiating a repository skeleton.

The process operates at two levels:

Folder-level encoding assigns each root-level subgraph (each module identified during proposal-level planning) a directory namespace. For example, the "ML Algorithms" subgraph is assigned to src/algorithms/, the "Evaluation" subgraph to src/eval/, and the "Data Loading" subgraph to src/data_load/. The paper's prompt template for skeleton mapping (Appendix C.1) instructs the LLM to "design a clean, modular file system skeleton that organizes the repository into appropriate top-level folders based on these subtrees," following Python conventions (snake_case folder names) and allowing either flat or nested structures (e.g., under src/). The prompt explicitly states that "folder names do not need to match subtree names exactly"—the LLM can rename folders for clarity while preserving the correct mapping.

File-level encoding then assigns files to intermediate nodes within each folder. Taking the "Regression" sub-module under "ML Algorithms" as an example: the LLM might decide that simple linear regression and multiple linear regression go in linear_models.py, while polynomial regression variants go in polynomial.py, and regularization-based methods like Lasso and Ridge go in regularized.py. The prompt for mapping feature paths to files (Appendix C.1) instructs the LLM to "group semantically related features together," "avoid bundling many unrelated features into a single file," and "introduce subfolders based on semantic structure" if a folder contains 10 or more files.

An example skeleton generated by o3-mini for the MLKit-Py task is shown in Appendix C.2. The skeleton organizes the generated repository into a deep hierarchy: src/algorithms/ensemble_tree/boosting_bagging/boosting/boosting_algorithms.py, src/algorithms/regression/linear_models/polynomial.py, src/core/data_transform/scaling_basic.py, etc. The subgraph-to-skeleton mapping (also in Appendix C.2) shows exactly which functional nodes map to which files—for instance, the "PolynomialRegression" feature node maps to src/algorithms/regression/linear_models/polynomial.py, and the "StackingVoting" ensemble feature maps to both primary.py and secondary.py in the stacking sub-directory.

The output of this substage is a file-augmented graph: the functionality graph with folder namespaces assigned to root nodes and file paths assigned to intermediate nodes. Leaf nodes still represent abstract capabilities (e.g., "Cubic Regression with regularization") without concrete function signatures.

Substage B.2: Data Flow and Functions Encoding

This substage (Section 3.3.2) finalizes the RPG by adding three layers of implementation detail: data flows between modules, shared abstractions (base classes and data structures), and concrete function/class interfaces for leaf nodes.

Data-flow encoding adds typed edges between module roots to capture inter-module dependencies. The prompt template (Appendix C.1) instructs the LLM to define "how data moves between functional modules (subtrees)—including who produces it, who consumes it, and how it is transformed—and express this as a structured, directed graph." Each edge is a JSON object with five fields: from (source subtree name), to (target subtree name), data_id (unique name or description of the data being passed), data_type (the structure or format of the data), and transformation (how the data is modified, filtered, or enriched; "none" if unchanged). The prompt enforces two structural constraints: "full connectivity required" (every subtree must appear in at least one edge, no isolated subtrees) and "acyclic structure" (the data flow must form a Directed Acyclic Graph, ensuring no circular dependencies).

For example, in the machine learning library RPG (Figure 2), typed edges include: "Data Loading → ML Algorithms: training data (array of feature matrices)," "ML Algorithms → Evaluation: predictions (array of predicted values)," and "Data Loading → Evaluation: target values (array of ground-truth labels)." These edges are not merely documentation—they impose a topological ordering constraint. The module that produces data must be implemented before the module that consumes it (or at least, the interface must be defined first). This ordering is enforced during code generation.

Intra-module ordering is also encoded: within a module, files are ordered so that dependencies are satisfied. For instance, in the Data Loading module, load_data.py precedes preprocess.py because the preprocessor consumes the loader's output. This ordering ensures that when the code generation stage traverses the module, it implements files in dependency order.

Abstracting global interfaces addresses a scalability concern: if every module defines its own input and output formats independently, the repository becomes a patchwork of incompatible interfaces. To prevent this, the system identifies recurring input–output patterns across modules and abstracts them into common data structures or base classes. The prompt template for base classes (Appendix C.1) instructs the LLM to design "reusable abstractions and shared data structures" following two principles: (1) shared data structures should be defined for nodes with "high out-degree (outputs consumed widely)"—good candidates are "feature batches, inference results, or training containers"—and (2) functional base classes should be defined for nodes with "high in-degree (consuming many inputs)" where "multiple modules share roles (e.g., cleaning, predicting), follow common lifecycles (run(), build(), validate()), or rely on similar hooks."

The paper's example (Appendix C.2) shows base class design for the MLKit-Py task: a BaseComponent abstract class with a standard lifecycle (initialize(), process(data), finalize()), and an EstimatorComponent subclass that adds fit(training_data) and predict(input_data) methods. These base classes are assigned to specific files (e.g., src/general/base_components.py) and serve as design anchors that enforce interface consistency across all modules that inherit from them.

Adaptive interface design is the final and most granular step. Within each file-level subgraph, leaf features are clustered into executable interfaces (functions or classes) based on semantic relatedness. The prompt template (Appendix C.1) instructs the LLM: "for each feature, define exactly one interface (function or class). Provide imports, signature, and detailed docstring (purpose, args, returns, assumptions). No implementation: use pass." The design guidelines specify that a feature should be a standalone function if it is "simple, atomic, stateless" and a class if it is "stateful, multiple methods, inherits base class, or extensible." Features that are tightly related (e.g., load_csv and load_json both deal with data loading) are grouped into shared classes with methods (e.g., a DataLoader class with load_csv() and load_json() methods), while independent features become standalone functions.

The paper provides an example in Appendix C.2: the "PolynomialRegression" feature node maps to a PolynomialRegressor class in src/algorithms/regression/linear_models/polynomial.py, inheriting from EstimatorComponent, with a constructor that takes degree: int and regularization_lambda: float = 0.0, a fit(X, y) method, and a predict(X) method—all with detailed docstrings but no implementation (just pass). This adaptive mapping "balances granularity and cohesion, yielding a Repository Planning Graph (RPG) that preserves modularity and semantic consistency at repository scale" (Section 3.3.2).

Structural isomorphism between RPG and code. The paper observes (Appendix C.3) a strong isomorphic relationship: each subgraph in the RPG corresponds to a coherent code region, with files, classes, and functions serving as structural anchors. Table 9 quantifies this for the MLKit-Py task with o3-mini: the "ML Algorithms" subgraph maps to 58 files, 171 classes, and 67 functions, with an average of 5.57 features per file, 1.50 features per class, and 1.00 features per function. This means most functions correspond to exactly one leaf feature—a near 1:1 mapping at the function level—while files and classes aggregate multiple related features. The pattern varies by subgraph type: core computational domains (like ML Algorithms) require broader structural scaffolding (many files and classes), while specialized domains (like Visualization) concentrate more features per unit (4.00 features per class, per Table 9).

Table 8 extends this analysis across all six repositories and both models, showing that o3-mini distributes features more evenly across units (average 4.31 features per file), while qwen3-coder produces higher feature densities, especially at the class level (average 1.44 features per class for qwen3-coder vs. 2.12 for o3-mini). This indicates that model choice affects not just what features are planned but how they are structurally organized—a finding that has implications for downstream code maintainability.

What this stage produces. The output is the complete Repository Planning Graph (RPG): a graph where root nodes carry folder namespaces, intermediate nodes carry file paths, leaf nodes carry concrete function/class specifications with full signatures and docstrings, and edges encode both hierarchical decomposition and typed data flows. Every node has dual functional/structural semantics, and the edges collectively impose a topological ordering that the code generation stage follows. This graph is the persistent artifact that guides all subsequent code generation—it is never discarded in favor of natural language.

3.4.4 Graph-Guided Code Generation

Given the completed RPG and the user's original query, the code generation stage (Section 4, detailed in Appendices D.1–D.5) produces the actual repository code by traversing the graph in topological order and applying test-driven development at each leaf node.

Topological traversal. The RPG's edges define a partial order: dependencies must precede dependents. The system linearizes this partial order into a traversal sequence such that when a node is visited, all nodes it depends on have already been implemented. This is the standard topological sort of a DAG. The paper does not specify the exact sorting algorithm, but the constraint is clear: base classes before derived classes, data producers before data consumers, files in intra-module order.

Test-driven development at each leaf node. At each leaf node (representing a concrete function or class), the system follows a test-driven development (TDD) loop:

Test derivation: tests are derived from the function's specification (its docstring, signature, and the task description). The paper's Algorithm 4 (Appendix D.4) formalizes this: a candidate test branch is created for the code unit, test code is generated and wrapped into a TestNode or IntegrationTestNode, and the test is executed inside a controlled Docker environment.
Implementation: the corresponding function or class is implemented. The editing tools (Appendix D.2) support four granularity levels: edit_whole_class_in_file (replace an entire class including all methods), edit_method_of_class_in_file (replace a single method within its class block), edit_function_in_file (replace a top-level function), and edit_imports_and_assignments_in_file (add or correct imports and top-level assignments, following the order: standard library → third-party packages → local modules). The paper emphasizes that imports should not be removed "unless they are demonstrably incorrect" and should be retained "even if they appear unused, to preserve runtime dependencies."
Test execution and judgment: the implemented code is tested. If failures occur, an LLM judge classifies the error type (Appendix D.4, Algorithm 4, lines 16–21): errors are categorized as either implementation errors (the code logic is wrong), test code errors (the test itself is incorrect), or environment errors (e.g., missing dependencies, configuration issues). Implementation errors trigger debugging (see below); test and environment errors trigger automatic repair attempts (up to 20 remediation attempts, per Section 5.3).
Iterative debugging: if implementation errors are detected, the system enters a debugging loop with up to 8 iterations (Section 5.3). Each debugging iteration involves localizing the error source, editing the affected code, and re-running tests. Only implementations that pass all tests are committed to the repository, enabling incremental expansion while preventing regressions.

Graph-guided localization. When the system needs to locate where a particular functionality is implemented (for debugging, incremental development, or integration testing), it uses the RPG as a navigation structure. The localization tools (Appendix D.1) provide three capabilities:

RPG-Guided search (search_interface_by_functionality): performs fuzzy semantic matching against leaf node specifications using keyword queries. Given keywords like "optimize" or "initialize," the tool returns the top-5 most relevant interface implementations with their file paths and docstrings. This replaces ad-hoc codebase grep with semantically grounded search.
Repository code view (view_file_interface_feature_map and get_interface_content): inspects individual files to list their interface structures (functions, classes, methods) and the feature mappings they support, or retrieves the full implementation of a specific interface given its fully qualified name.
Dependency exploration (expand_leaf_node_info): given a feature path from the RPG, expands and lists all associated interfaces in a structural summary, effectively traversing the graph to show what a given functional node maps to in code and what it depends on.

The paper demonstrates the localization workflow in a detailed trajectory log (Appendix D.3) for implementing an AdvancedDifferentialTransformer class in the SymbolicMath repository. The agent begins by inspecting the target file (view_file_interface_feature_map), discovers that the class skeleton already exists with placeholder methods, then uses search_interface_by_functionality to find related interfaces (e.g., functions for "differentiate special function," "asymptotic series expansion") in other parts of the repository, and finally terminates with a ranked list of five most relevant dependency interfaces. This structured exploration contrasts sharply with natural-language agents that would need to read through documentation or guess at file locations.

Effectiveness of graph-guided localization. Table 4 (Section 7.3) ablates the impact of graph guidance by comparing localization steps with and without RPG support. Across three task categories on MLKit-Py (o3-mini), graph guidance reduces effort by 30–53%:

Integration Testing: 6.2 ± 2.1 steps with graph vs. 13.3 ± 11.1 without (53% reduction, also substantially lower variance).
Incremental Development: 6.8 ± 1.8 vs. 10.8 ± 2.6 (37% reduction).
Debugging: 5.8 ± 2.8 vs. 8.5 ± 2.9 (32% reduction).

The paper attributes this to the RPG providing "a principled navigation mechanism, enabling faster dependency tracing, more accurate bug localization, and smoother module integration." Figure 15 (Appendix F.4) visualizes the localization behavior across repositories, showing a consistent pattern the paper calls "CCG" (Coarse Search → Content Inspection → Global Graph Exploration): the agent first traverses the RPG at a coarse level to identify high-level candidates, then inspects content-rich nodes for detailed signals, and finally explores semantically related structures across the graph before terminating. This systematic pattern emerges only with graph guidance; without it, the agent's behavior is ad-hoc and repetitive.

Multi-level testing strategy. The testing framework (Algorithm 3, Appendix D.4) goes beyond simple unit tests to incorporate three levels of validation:

Unit tests validate each new function or class in isolation, using tests derived from its docstring and specification. These are the TDD tests described above.
Regression tests re-execute existing test nodes when a component is modified. If the same test node exists for a component and the component's "signature or logic is unchanged" (Algorithm 3, line 7), the existing test is reused; otherwise, a new test is created. This ensures that new changes do not break previously working functionality.
Integration tests verify data flows and contracts across modules. When multiple patched components interact (e.g., the data loader produces output that the preprocessor consumes), integration tests validate that the interfaces are compatible and the end-to-end data flow works correctly. Integration test nodes are created for groups of patches clustered by "integration-node" (Algorithm 3, lines 17–28).

All tests are executed inside Docker containers to ensure environment consistency. Test failures are diagnosed using a 5-round majority voting mechanism (Section 4, final paragraph; Section 5.3): the LLM judge classifies each failure as an implementation error (which triggers the debugging loop), a test code error (which triggers test repair), or an environment error (which triggers automatic remediation, up to 20 attempts). This "lightweight majority-vote diagnosis" separates genuine implementation errors from spurious failures, preventing the system from wasting debugging iterations on problems it cannot fix (like missing system dependencies).

Success rates and coverage. Table 11 (Appendix D.5) reports the average success rate and test coverage for code generation across repositories. With o3-mini, success rates range from 71.0% (StatModeler) to 88.9% (HttpEasy), with an average across all repositories of approximately 80%. Test coverage—the proportion of generated code exercised by tests—is moderate, typically in the 60–65% range. The paper notes (Appendix D.5) that coverage "fluctuates and tends to decline as code length increases: shorter implementations reach high class-level coverage, but both function-level and overall coverage drop significantly with greater complexity." This identifies test generation as a key bottleneck: while the system is effective at generating functional code, producing comprehensive and high-quality test cases for complex implementations remains challenging.

State management: only passing code is committed. A critical design choice is that only implementations that pass all tests are committed to the repository (Section 4: "Only functions that pass all tests are committed, enabling incremental expansion while preserving stability"). This means the repository at any point in the traversal is always in a consistent, tested state. If a new implementation fails and cannot be debugged within the iteration limit (8 attempts), it is not added to the repository, and the traversal continues to the next node. This prevents a single difficult-to-implement feature from blocking progress on the rest of the repository, at the cost of potentially leaving gaps in coverage.

Summary of the code generation pipeline. The code generation stage receives the completed RPG (with folder/file assignments, data flows, and function specifications), traverses it in topological order, implements each leaf node via TDD, localizes errors using RPG-guided search tools, validates through multi-level testing (unit, integration, regression), and commits only passing implementations. The output is a complete repository where all committed code has been validated against generated tests, and the structure mirrors the RPG's dual functional/structural encoding.

3.4.5 Key Design Choices and Their Justifications

The paper's technical approach rests on several non-obvious design decisions that are worth making explicit, as they collectively define what makes the system work (and what would break if changed).

Design choice 1: Graph over natural language as planning medium. This is the paper's central architectural decision. The justification is threefold: (a) natural language ambiguity leads to interface mismatches and inconsistent interpretations across planning stages; (b) lack of explicit hierarchy in natural language makes dependency tracking difficult and error-prone, especially at scale; (c) natural language plans are static and do not adapt as new decisions are made, leading to silent contradictions. The graph addresses each: nodes have explicit semantics (both functional and structural), edges encode typed dependencies that can be traversed algorithmically, and the graph is incrementally updated with each new decision, ensuring all additions are grounded in the current state. The empirical evidence for this choice is the scaling behavior in Figures 5 and 6: graph-based planning sustains near-linear growth, while natural-language planning plateaus.

Design choice 2: Feature Tree as structured prior, not as fixed template. The system uses the EpiCoder Feature Tree to stabilize capability enumeration, but the ablation in Section 8 (Figures 7–8, Table 6) shows that the RPG structure—not the Feature Tree—is the primary enabler of scaling. Removing the Feature Tree slows planning (ZeroRepo without the Feature Tree reaches 87.2% coverage and 25K LOC at iteration 30, vs. 95.7% coverage and 31.6K LOC with it) but does not prevent it: by iteration 40, the Feature-Tree-free version catches up to 97.9% coverage. The Feature Tree acts as an accelerator that provides better initial context (higher intercept in the linear fit: y ≈ 983x + 2992 with Feature Tree vs. y ≈ 800x + 989 without) and faster growth (higher slope). This justifies the design: you could plan without the ontology, but it would be slower; the ontology is an efficiency optimization, not a capability prerequisite.

Design choice 3: Dual semantics on nodes (functional + structural). This is what enables the RPG to serve as a single representation for both what to build and how to build it. Without dual semantics, the system would need two separate representations—a functional plan and a code structure plan—and would face the same consistency problem that natural-language approaches face when reconciling multiple documents. By encoding both in one graph, changes at either level are immediately reflected in the other: if a new function is added, its structural location is specified; if the file structure is reorganized, the functional mapping is updated. This co-evolution prevents the drift that plagues approaches with separate planning and implementation documents.

Design choice 4: Topological traversal for code generation. Generating code in arbitrary order would require constant backtracking as dependencies are discovered missing. Topological order guarantees that when a node is reached, all its prerequisites exist, eliminating an entire class of errors. This is only possible because the RPG's edges encode explicit dependencies—in a natural-language plan, you cannot compute a topological ordering because the dependencies are not formally specified.

Design choice 5: Test-driven development with per-function validation. Rather than generating the entire repository first and then testing it (which would make errors hard to localize), the system validates each function immediately after implementation. This is a form of incremental verification: errors are caught at the point of introduction, when the context is fresh and localization is straightforward. The cost is that tests must be generated for every function, which introduces its own failure modes (as the moderate test coverage in Table 11 shows). The benefit, evidenced by the high pass rates in Table 2 (69.7% with o3-mini), is that the code that is generated is largely correct.

Design choice 6: Only passing code is committed. This is a conservative state management strategy: the repository is always in a consistent, tested state. The downside is that difficult-to-implement features may be skipped entirely, reducing coverage. The upside is that the repository never contains broken code that could cascade errors into dependent modules. This choice reflects a philosophy of "correct and incomplete" over "complete and buggy"—appropriate for a system where the generated repository is intended to be a foundation for further development.

Design choice 7: Majority-vote error diagnosis. When a test fails, the system could naively assume it's an implementation error and enter the debugging loop. However, many failures are caused by test code bugs or environment issues (missing dependencies, version mismatches). The majority-vote mechanism (5 rounds of LLM judgment) distinguishes between these categories, preventing the system from wasting debugging iterations on problems it cannot fix. This is a practical engineering decision that significantly improves end-to-end success rates by avoiding cascading failure modes.

4. Key Insights and Innovations

Innovation 1: The Planning Representation — Not the Model — Is the Primary Bottleneck in Long-Horizon Code Generation

The paper's most fundamental conceptual contribution is not a new algorithm or a better model, but a diagnosis of why existing approaches fail at repository-scale generation. Prior work implicitly assumed that the primary limitation was model capability — that better LLMs with stronger reasoning, longer contexts, or more sophisticated agent architectures would eventually solve the planning problem. The paper systematically disproves this assumption by showing that Claude Code, Gemini CLI, and Codex CLI — all using state-of-the-art models with web search and multi-iteration refinement — plateau sharply in both feature count and code volume (Figures 5 and 6), while ZeroRepo with the same underlying model (o3-mini) sustains near-linear growth.

This is a reframing of the problem from model capability to representational adequacy. The field's dominant framework treated repository generation as a reasoning challenge: if the model can reason well enough about architecture, dependencies, and interfaces, it can produce a coherent repository. The paper argues — and demonstrates empirically — that even models with strong reasoning fail because natural language, as a planning medium, has fundamental scaling limits. Ambiguities that are manageable in short documents compound into contradictions over hundreds of decisions. Dependencies described in prose cannot be algorithmically verified or traversed. Plans written at iteration 5 silently fall out of sync with decisions made at iteration 20.

What makes this insight distinctive is that it inverts the usual AI narrative. Typically, we assume symbolic structure is a temporary crutch that better neural networks will eventually render unnecessary — that as models scale, they will learn to maintain consistency purely through latent representations. This paper provides compelling counter-evidence: the gap between graph-based and language-based planning widens with scale, not narrows. At iteration 5, the difference is modest; by iteration 30, it is a factor of 10× or more (Figure 5: ~1,100 features vs. ~200 for Claude Code). This suggests that structured representations are not a bridge technology but a fundamentally more scalable substrate — a claim with implications far beyond code generation, touching on any AI task requiring long-horizon planning under consistency constraints.

The paper connects this to broader AI debates implicitly rather than explicitly, but the connection is clear. Valmeekam et al. (2023) showed that LLMs fail at planning when they must maintain consistent state across multiple steps. Besta et al. (2024) showed that graph-structured reasoning ("Graph of Thoughts") outperforms natural language for complex problem-solving. This paper extends that insight to a domain — code generation — where the planning horizon is an order of magnitude longer and the consistency requirements are far more stringent, demonstrating that the benefits compound rather than plateau.

Evidence anchor: The scaling behavior in Figures 5 and 6, where natural-language baselines plateau by iteration 10–15 while ZeroRepo sustains linear growth through iteration 30, is the core empirical support. The ablation in Section 8 (Table 6, Figures 7–8) further isolates the effect: removing the EpiCoder Feature Tree (the structured ontology) slows growth but does not prevent it — ZeroRepo without the Feature Tree still reaches 87.2% coverage and 25K LOC at iteration 30, vs. Claude Code's 59.6% coverage and 3.6K LOC. This demonstrates that the RPG structure itself, not just the ontology, is what enables scaling.

Innovation 2: Unifying Proposal and Implementation Planning in a Single, Persistent Graph with Dual Semantics

Prior work separates proposal-level planning (deciding what to build) from implementation-level planning (deciding how to build it) into different artifacts or stages. Multi-agent systems like MetaGPT and ChatDev have separate "architect" and "engineer" roles that produce separate documents; workflow systems like Paper2Code have sequential stages where each stage's output is a natural language document consumed by the next; terminal agents externalize plans in markdown files that are read and written across iterations. The implicit assumption is that these are distinct phases requiring different representations.

The paper makes a conceptually non-obvious move: encode both in a single graph where each node carries dual semantics — functional (what capability it represents) and structural (where in the codebase it lives). This is not merely a technical convenience. It is a architectural decision with profound implications for plan consistency.

When proposal and implementation plans are separate artifacts, they must be kept in sync manually. If the engineer decides to split a module across two files, the architect's plan — which treats the module as a single unit — becomes stale. If the architect adds a new capability, the engineer must discover this and update file assignments. These synchronization failures are exactly what the paper identifies as the failure mode of natural language planning: "plans drift across iterations, introducing inconsistencies in dependencies, data flows, and modular boundaries" (Section 1).

The dual-semantics graph eliminates this class of failures by making inconsistency structurally impossible. A node cannot be assigned a functional role without also specifying its structural location, because those are two facets of the same node. When the graph is updated — a new capability is added, a file is split — both facets are updated atomically. There is no separate artifact to fall out of sync.

What makes this distinctive as an innovation rather than an implementation detail is that it represents a fundamentally different philosophy about how planning should be represented. The dominant paradigm in AI planning (and software engineering more broadly) is phase separation: first figure out what you want, then figure out how to build it, with a specification document mediating between the phases. The RPG paradigm is unified representation: the what and the how co-evolve in a single structure that enforces mutual consistency at all times. This has theoretical connections to work on bidirectional model transformations and round-tripping in software engineering, but applies the idea to AI-driven planning rather than human-driven model-driven engineering.

The paper also demonstrates that this unified representation supports meaningful analysis that would be impossible with separate artifacts. Tables 8 and 9 quantify the structural isomorphism between subgraphs and code: the "ML Algorithms" subgraph maps to 58 files, 171 classes, and 67 functions, with different feature-per-unit ratios for different subgraph types. This kind of analysis — understanding how functional granularity correlates with structural granularity across modules — requires the dual encoding; without it, you cannot even ask the question.

Evidence anchor: The structural statistics in Tables 8 and 9, and the skeleton-to-subgraph mapping in Appendix C.2, demonstrate the dual encoding in practice. The localization effectiveness data in Table 4 (30–53% reduction in steps with graph guidance) demonstrates the operational benefit: having both functional and structural information in one queryable graph accelerates navigation.

Innovation 3: Difficulty-Agnostic Explore–Exploit Search over a Structured Ontology as a Third Way Between Pure Generation and Pure Retrieval

The question of how to determine what a repository should contain is deceptively hard. Prior approaches fall into two camps. Pure generation approaches (all three baseline paradigms) ask the LLM to enumerate required capabilities from scratch, which the paper argues leads to "unstable and biased capability enumeration, often with incomplete coverage" (Section 3.2). Pure retrieval approaches would simply look up a fixed template — but repositories are too diverse for templates, and the space of possible repositories is too large to enumerate.

The paper introduces a third way: explore–exploit search over a structured ontology (the EpiCoder Feature Tree), where the LLM serves as a filter and refiner rather than a generator. This is a subtle but important shift in how the LLM is used. Instead of asking the model "what features should a machine learning library have?" — a generative task prone to omission and bias — the system asks "given these 1.5M possible features, which ones are relevant?" — a discriminative task that leverages the model's semantic understanding while constraining its output to an existing taxonomy.

What makes this innovative is not the explore–exploit pattern itself (which is standard in search) but its application to planning through a structured prior. The Feature Tree acts as a form of externalized memory with explicit structure — it remembers capabilities that the LLM might forget, and its hierarchical organization provides scaffolding that guides the search. This is fundamentally different from retrieval-augmented generation (RAG), where retrieved passages inform generation but do not constrain it. Here, the ontology constrains the output space, making the planning problem tractable by restricting it to capabilities that actually exist in code.

The ablation in Section 8 is crucial to understanding the nature of this innovation. Removing the Feature Tree does not cause catastrophic failure — ZeroRepo without it still reaches 97.9% coverage by iteration 40, matching the Feature-Tree-enhanced version at iteration 30. This demonstrates that the ontology is an accelerator, not a prerequisite. The RPG structure handles the consistency problem; the ontology handles the efficiency problem of exploring the space of possible capabilities. The linear fits (slope ~983 with Feature Tree, ~800 without) quantify this precisely: the ontology provides both a better starting point (higher intercept) and faster growth (higher slope).

This finding has practical implications for how to build planning systems. It suggests that investing in structured knowledge bases is a complementary strategy to improving model reasoning — and that for long-horizon tasks, the knowledge base may matter more than marginal improvements in model capability. A weaker model with a good ontology might outperform a stronger model with no ontology, which is a claim the paper does not directly test but strongly implies.

Evidence anchor: Figure 5 (near-linear feature growth vs. baseline plateau), Figure 9 and Table 7 (Feature Tree statistics and per-repository growth patterns), and the Section 8 ablation (Table 6, Figures 7–8) collectively support the claim that explore–exploit over a structured ontology enables scalable capability enumeration.

Innovation 4: Verifier-Free Test-Driven Development Guided by a Topological Dependency Graph Enables Incremental, Correct-by-Construction Repository Assembly

A standard approach to ensuring code correctness — both in human software engineering and in AI code generation — is to generate everything first, then test and debug. This "generate-then-validate" paradigm is simple but has a well-known failure mode: errors interact in complex ways, making debugging exponentially harder as the codebase grows. An alternative is to use learned verifiers (e.g., reward models trained to predict code correctness) to guide generation, analogous to how the test-time compute scaling paper used PRMs to guide search. But training verifiers requires large amounts of labeled data and is domain-specific.

The paper introduces a third paradigm: topological-order test-driven development where correctness is enforced incrementally at each node. By traversing the RPG in dependency order and validating each function immediately after implementation, the system ensures that at every point, the codebase is in a consistent, tested state. Errors are caught at the point of introduction, when context is fresh and localization is straightforward.

What makes this innovative is not TDD itself (which is a standard software engineering practice) but its integration with the topological dependency graph. The RPG's edges provide a formal specification of what must be built before what, which the code generation stage follows exactly. This means the system never encounters the situation — common in generate-then-validate approaches — of trying to test a function whose dependencies are broken. By the time a function is tested, all its prerequisites (base classes, imported modules, upstream data producers) are already implemented and validated.

This is fundamentally different from both the test-time compute approach (which uses learned verifiers to select among candidate solutions) and the standard agent approach (which generates code, runs tests, and debugs in a loop). The RPG enables a constructive rather than discriminative approach to correctness: build it right the first time, rather than generate many candidates and select the best. This is only possible because the graph provides enough structural information to order operations correctly — without it, the system would need to discover dependencies through trial and error (which is exactly what the natural-language baselines do, and why they plateau).

The paper also demonstrates a non-trivial practical insight: only committing passing code prevents cascading failures. If a function cannot be debugged within 8 iterations, it is skipped rather than committed in a broken state. This is a conservative strategy that sacrifices coverage for correctness, and the results (69.7% pass rate on committed code vs. <34% for baselines) suggest it is the right tradeoff for repository-scale generation where a single broken interface can invalidate all downstream consumers.

Evidence anchor: The pass rates in Table 2 (69.7% for ZeroRepo vs. 33.9% for Claude Code, 14.5% for Gemini CLI), the localization efficiency gains in Table 4 (30–53% reduction), and the test coverage analysis in Table 11 and Figure 14 (showing moderate but functional coverage) collectively support the claim that topological TDD with selective committing produces higher-quality code than generate-then-validate approaches.

Innovation 5: Repository Generation as a Scalability Problem — Near-Linear Scaling as a Diagnostic for Representational Adequacy

The paper introduces a diagnostic concept that the field previously lacked: scalability of planning as a measurable property of the planning representation, distinct from model capability. Prior work evaluated repository generation systems by their absolute performance on fixed benchmarks — coverage, correctness, code size at a fixed iteration count. This paper adds a new dimension: how does performance scale with more planning iterations?

This shift in evaluation perspective reveals something that absolute metrics obscure: natural-language approaches do not just underperform — they saturate. Claude Code achieves 59.6% coverage on MLKit-Py (Table 6) and stops improving after iteration 10 (Figure 7). The model is not running out of capability — it is running out of representational capacity in its planning medium. The natural language plan cannot accommodate more features without becoming inconsistent, so the agent stops adding them.

By contrast, ZeroRepo exhibits near-linear scaling: y ≈ 983x + 2992 for LOC growth (R² = 0.97 with the Feature Tree) and sustained linear increase in feature count through iteration 30. The paper explicitly frames this as evidence that "RPG provides a persistent, extensible substrate that refines high-level goals into richer functionalities while sustaining structural consistency" (Section 7.1). The key word is substrate — the RPG is not a plan that gets executed and discarded; it is a growing representation that can accommodate new information without degradation.

This innovation matters because it provides a principled way to evaluate planning representations beyond task-specific benchmarks. A representation that supports linear growth is fundamentally more capable than one that saturates, even if both achieve similar performance at low iteration counts. This has implications for research: when developing new planning approaches, one should measure not just absolute performance but the slope of the scaling curve. An approach with lower absolute performance at 10 iterations but sustained linear growth to 30 iterations may be more promising than one with higher initial performance but early saturation.

The paper also implicitly introduces a separation argument: the fact that ZeroRepo without the Feature Tree still exhibits linear growth (slope ~800, R² = 0.98) while Claude Code plateaus, despite both using o3-mini and having access to the same number of iterations, demonstrates that the scaling property belongs to the representation (RPG), not the model or the knowledge base. This is a clean experimental dissociation that supports the paper's central claim about representational adequacy.

Evidence anchor: Figures 5 and 6 (scaling curves with iteration count), the linear fit analysis in Section 8 (slopes and R² values), and Table 3 (coverage and novelty scaling on MLKit-Py from iteration 5 through 30) collectively establish scaling behavior as a measurable property and demonstrate that RPG-based planning scales while natural-language planning does not.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use RepoCraft, a benchmark constructed by the authors specifically for repository-level generation evaluation. It comprises six real-world Python projects—scikit-learn (anonymized as MLKit-Py), pandas (TableKit), sympy (SymbolicMath), statsmodels (StatModeler), requests (HttpEasy), and django (PyWebEngine)—with their names and descriptions paraphrased to prevent pretraining leakage. From these reference repositories, the authors derive 1,052 evaluation tasks by collecting test functions, organizing them hierarchically following each project's modular structure, applying stratified sampling to ensure representative coverage, and filtering out non-algorithmic tests (e.g., version checks, formatting tests). Table 1 provides per-repository statistics: the six projects span 17–681 source files, 2,793–218,924 lines of code, and 22,297–2,339,881 code tokens, covering 22–88 functional categories each. The evaluation tasks are described in Section 5.1.3 and Appendix E.2—each includes a natural-language description of the target algorithm, a ground-truth test, and auxiliary materials.
Base model(s). The primary backbone used in ZeroRepo experiments is o3-mini (OpenAI, 2025), with Qwen3-Coder-480B-A35B-Instruct (Qwen3-Coder; Team, 2025) as a secondary backbone for comparison. The baselines use a wider range of models: MetaGPT, ChatDev, and Paper2Code are evaluated with both o3-mini and Qwen3-Coder backbones; terminal agents use their official strongest models (Codex CLI with o3 pro, Gemini CLI with gemini 2.5 pro, Claude Code CLI with claude 4 sonnet). The paper does not provide a single justification for choosing o3-mini over alternatives, but the ablation in Appendix B.2 (Figure 9) characterizes the model-specific behavior: Qwen3-Coder exhibits more aggressive, open-ended feature expansion while o3-mini follows a more balanced trajectory, together representing different points on a recall–precision spectrum.
Metrics. RepoCraft evaluates generated repositories along three dimensions (Section 5.1.2, detailed in Appendix E.3.1). Functionality metrics: (1) Coverage—the fraction of reference functional categories (defined from official documentation) that are represented in the generated repository, where a category is counted as covered if at least one generated functionality maps to it via K-Means clustering with LLM refinement; (2) Novelty—the fraction of generated functionalities that fall outside the reference taxonomy (assigned to an out-of-distribution centroid), measuring the system's ability to propose coherent extensions beyond the specification. Accuracy metrics: (3) Pass Rate—the fraction of evaluation tasks where adapted tests pass against the generated implementation; (4) Voting Rate—the fraction of tasks confirmed by majority-vote validation (a 5-round LLM check that the generated code actually implements the target algorithm). Code-level statistics: (5) File count, (6) normalized Lines of Code (LOC, excluding comments, docstrings, and blank lines), and (7) token count, all measured after excluding non-core code such as tests and examples.
Baselines. The paper compares against three paradigms of prior work, totaling 7 baseline configurations (Section 5.2). Multi-agent frameworks: MetaGPT (Hong et al.) assigns roles (Product Manager, Architect, Engineer) for end-to-end development; ChatDev (Qian et al., 2024) uses a company-style organization with chat-based coordination. Workflow-based system: Paper2Code (Seo et al., 2025) follows a fixed three-stage pipeline (planning, analysis, generation). Terminal agents: OpenHands (Wang et al.), Codex CLI (OpenAI, 2025), Claude Code CLI (Anthropic, 2025b), and Gemini CLI (Google, 2025) perform natural-language editing, debugging, and multi-file reasoning through terminal interfaces, with web search enabled for real-world knowledge retrieval. MetaGPT, ChatDev, and Paper2Code are run with two backbones (o3-mini and Qwen3-Coder). All baselines receive the same task descriptions with anonymized repository names.
Generation budget / compute accounting. The paper does not use a unified FLOPs or token budget for fair comparison. Instead, all agents are run for a fixed 30 iterations (Section 5.2): "To ensure fairness, all runs extend to 30 iterations, with agents prompted at each step to propose or implement functionality." The comparison is therefore normalized by planning steps rather than compute. This is a pragmatic choice given the diversity of architectures (some agents use web search, some use different models, some are multi-agent with variable internal costs), but it means that per-iteration compute may differ substantially between methods. Within ZeroRepo, the code generation stage caps debugging at 8 iterations per function, localization at 20 attempts, and test/environment error remediation at 20 attempts (Section 5.3). Majority voting for test failure diagnosis uses 5 rounds (Section 4). The proposal-level construction runs 30 iterations for feature selection.
Cross-validation / statistical protocol. The paper does not apply cross-validation to the main results—all 1,052 tasks from the six repositories are used for evaluation, and the coverage and novelty metrics are computed over the entire set of functional categories. For the automated evaluation pipeline validation (Section 7.4, Appendix E.5), the authors sample 200 tasks per system for human annotation of localization and majority-vote stages, and 100 tasks per method for test code quality audit and accuracy evaluation under different judges. Pearson correlations between automated and human judgments are reported in Table 14 (coverage Pearson 0.78–0.89, novelty Pearson 0.87–0.96 depending on the automated judge model). Standard deviations are reported for functionality metrics across multiple evaluation runs (Table 13) and for accuracy metrics (Table 16 shows mean ± std for o3-mini evaluations). The paper does not report confidence intervals for the main coverage, pass rate, or LOC results in Table 2.

Main Quantitative Results

Aggregate Performance Across All Repositories

Table 2 presents the headline comparison across all six repositories, with per-repository breakdowns in Appendix F.2 (Tables 18–23). The central results are:

Coverage: ZeroRepo with o3-mini achieves 81.5% coverage, an absolute improvement of 27.3 points over the strongest baseline (Claude Code at 54.2%) and 39.5–64.9 points over other baselines. With Qwen3-Coder, ZeroRepo achieves 75.1% coverage. The next-best non-ZeroRepo system is Gemini CLI at 42.0%.
Novelty: ZeroRepo with o3-mini achieves 13.6% novelty (151.5 novel functionalities out of 1,114.2 total planned), compared to 6.7% for Claude Code (41.6/621.0) and 0.0–9.2% for other baselines. This indicates ZeroRepo does not merely replicate reference repositories but proposes coherent extensions. With Qwen3-Coder, ZeroRepo achieves 9.2% novelty (108.3/1,173.3).
Accuracy: ZeroRepo with o3-mini achieves 69.7% pass rate and 75.0% voting rate, improving over Claude Code by 35.8 and 22.5 points respectively. Other baselines achieve pass rates of 2.6–14.5% and voting rates of 9.4–37.9%. The Gold Projects (human-developed reference repositories evaluated through the same pipeline) achieve 81.0% pass rate and 92.0% voting rate, establishing a practical upper bound under the evaluation harness.
Code scale: ZeroRepo with Qwen3-Coder generates 36,941 LOC and 445,512 tokens across 389 files, approximately 3.9× larger than Claude Code (10,587 LOC, 105,236 tokens, 33 files) and roughly 25–68× larger than other baselines (which range from 225–1,485 LOC). ZeroRepo with o3-mini generates 23,977 LOC, 260,761 tokens, and 271.5 files—smaller than Qwen3-Coder but still 2.3× Claude Code's LOC on 8.2× more files. The Gold Projects have 97,820 LOC and 951,614 tokens across 345 files—substantially larger, indicating ZeroRepo is approaching but not matching human scale.

Interpretation of aggregate results. ZeroRepo decisively outperforms all baselines across all four dimensions, often by margins that make statistical testing mostly unnecessary (e.g., 69.7% vs. 33.9% pass rate, 23,977 vs. 10,587 LOC). The gap between ZeroRepo and Claude Code—the strongest baseline—is larger than the gap between Claude Code and the weakest baselines in most metrics, suggesting a qualitative rather than merely quantitative difference. The o3-mini and Qwen3-Coder results show a consistent tradeoff: o3-mini achieves higher coverage (+6.4 points), novelty (+4.4 points), and pass rate (+12.4 points), while Qwen3-Coder produces larger repositories (1.5× LOC) but with lower correctness.

Per-Repository Performance Patterns

The detailed results in Tables 18–23 reveal that performance is not uniform across repositories. Two patterns stand out:

Repository size and complexity matter. On the smallest and most focused repository, HttpEasy (Table 19, 22 functional categories, 17 source files in the reference), ZeroRepo with o3-mini achieves 100.0% coverage, 64.0% pass rate, and 6,192 LOC. On the largest and most complex repository, SymbolicMath (Table 23, 40 functional categories, 699 source files), ZeroRepo achieves 62.8% coverage and comparable pass rates. This suggests that the RPG's planning capacity is effective but not unbounded—coverage decreases as repository complexity increases, though the relationship is not strictly monotonic (e.g., PyWebEngine with 42 categories achieves 79.2% coverage, higher than StatModeler at 77.3% with 88 categories).

Baselining difficulty varies dramatically by repository. On HttpEasy, Claude Code achieves 50.0% coverage and 36.0% pass rate—competitive with ZeroRepo on the simpler task. On MLKit-Py, Claude Code achieves 59.6% coverage vs. ZeroRepo's 97.9% (a 38.3-point gap). On PyWebEngine, Claude Code achieves 64.6% coverage with 38.1% novelty (669 novel functionalities out of 2,165 total)—the only case where a baseline exhibits substantial novelty, likely because the reference django repository has many sub-modules that a terminal agent can enumerate through web search. These differences suggest that the RPG's advantage is largest when the target domain requires careful architectural planning (machine learning pipelines, statistical modeling) rather than simple capability enumeration (HTTP client features).

Scaling Analysis: Feature and Code Growth Over Iterations

Figures 5 and 6 present the paper's most distinctive empirical contribution—the scaling behavior of different approaches as a function of planning iterations.

Feature growth (Figure 5). ZeroRepo (o3-mini) exhibits near-linear growth, surpassing 1,100 leaf features by iteration 30 with approximately constant slope. Claude Code grows steadily but with diminishing returns, reaching ~200 features before decelerating. Gemini CLI increases slowly to ~130 features and converges by iteration 30. Codex CLI ceases proposing features after 4–5 iterations, plateauing below 50. The gap between ZeroRepo and the best baseline widens over time—at iteration 10, the difference is roughly 5×; by iteration 30, it is roughly 5.5× for Claude Code and larger for others.

Code growth (Figure 6). On the MLKit-Py task, ZeroRepo sustains near-linear expansion in LOC, surpassing 30,000 LOC within 30 iterations. Claude Code and Gemini CLI plateau at approximately 3,000–4,000 LOC. Codex CLI stays below 1,000 LOC. The paper explicitly frames this as evidence that natural-language planning "accumulates inconsistencies, producing fragmented specifications that fail to converge into coherent code" (Section 7.1), while the RPG's structured, extensible representation ensures expansions materialize as code. The scaling curves in Figures 5 and 6 together demonstrate that the representational advantage compounds with iteration count.

Coverage and novelty over iterations (Table 3). On the MLKit-Py task, ZeroRepo scales coverage from 70.2% (iteration 5) to 95.7% (iteration 30) while maintaining approximately 5–8% novelty with 15–99 novel features at each checkpoint. Both coverage and novelty increase with iteration count, indicating the RPG supports simultaneous exploitation (filling coverage gaps) and exploration (proposing novel extensions). This is non-trivial—a system that only exploited would see coverage increase while novelty decreased; one that only explored would see the opposite. The RPG enables both.

Linear fit analysis (Section 8). The paper quantifies the scaling behavior with linear regression on LOC (y) vs. iterations (x): ZeroRepo with the EpiCoder Feature Tree shows y ≈ 983x + 2992 (R² = 0.97); without the Feature Tree, y ≈ 800x + 989 (R² = 0.98). The high R² values indicate genuinely linear growth; the difference in slopes (983 vs. 800) and intercepts (2,992 vs. 989) quantifies the Feature Tree's role as an accelerator that improves both initial conditions and growth rate.

Dependency Complexity and Structural Isomorphism

Figure 4 visualizes the dependencies in the repository generated by ZeroRepo with Qwen3-Coder on MLKit-Py, showing three levels of structure: file-level hierarchy (a coherent folder tree), inter-module data flows (defining execution pipelines from data_lifecycle through clustering and models to evaluation), and function-level inheritance/invocation edges. The paper claims this demonstrates that "RPG induces layered dependencies and coordinated execution, enabling repositories with both structural complexity and internal coherence" (Section 6).

Tables 8 and 9 (Appendix C.3) quantify the structural isomorphism between RPG subgraphs and generated code. For MLKit-Py with o3-mini (Table 9), the "ML Algorithms" subgraph maps to 58 files, 171 classes, and 67 functions, with feature densities of 5.57 features/file, 1.50 features/class, and 1.00 features/function. The near-1:1 mapping at the function level indicates that leaf RPG nodes map cleanly to individual code units. Across repositories (Table 8), o3-mini averages 4.31 features per file and 2.12 features per class, while Qwen3-Coder averages 4.90 features per file but only 1.44 features per class—Qwen3-Coder distributes features across more files but with fewer classes per file, suggesting more granular modularization.

Ablation Studies and Robustness Checks

Graph-guided localization (ablation of RPG structure): Table 4 compares localization steps with and without graph guidance on MLKit-Py (o3-mini). Across three task categories, graph guidance reduces localization effort by 32–53%: Integration Testing (6.2 ± 2.1 vs. 13.3 ± 11.1 steps), Incremental Development (6.8 ± 1.8 vs. 10.8 ± 2.6), and Debugging (5.8 ± 2.8 vs. 8.5 ± 2.9). Notably, the variance reduction is even more dramatic in Integration Testing (standard deviation drops from 11.1 to 2.1), indicating that graph guidance makes localization not just faster but more consistent. Table 10 extends this across all six repositories and both models, showing the same pattern at per-repository granularity.

EpiCoder Feature Tree (ablation of the knowledge base): Table 6, Figures 7–8, and the Section 8 discussion present the ablation on MLKit-Py (o3-mini). Comparing ZeroRepo (w/o KB) against Claude Code isolates the RPG structure's impact independent of the ontology. At iteration 30, ZeroRepo without the Feature Tree achieves 87.2% coverage, 6.4% novelty, 191 files, 25,202 LOC, and 271,039 tokens—vs. Claude Code's 59.6% coverage, 0.0% novelty, 31 files, 3,559 LOC, and 37,056 tokens. This is a 7× volume increase purely attributable to the graph structure. When the Feature Tree is added back, performance rises to 95.7% coverage, 7.9% novelty, 266 files, 31,596 LOC, and 351,554 tokens—modest additional gains beyond the structural baseline. Figure 7 shows the coverage trajectory: ZeroRepo (w/o KB) reaches 97.9% coverage by iteration 40, matching the KB-enhanced version at iteration 30—the ontology provides a temporal head start rather than a ceiling. Figure 8 confirms this for LOC and tokens: both configurations exhibit robust linear growth, with the KB-enhanced version showing higher slope and intercept.

Automated evaluation reliability (ablation of evaluation pipelines): Section 7.4 and Appendix E.5 validate the automated evaluation harness against human judgments across several dimensions. Table 13 reports per-repository coverage and novelty means ± std for three automated judges (DeepSeek-V3.1, o3-mini, GPT-5) and human annotators on Claude Code and ZeroRepo outputs. The automated judges recover the same qualitative patterns (ZeroRepo consistently higher than Claude Code) with broadly similar magnitudes. Table 14 reports Pearson correlations: coverage correlations range from 0.78 (DeepSeek) to 0.89 (o3-mini), novelty correlations from 0.87 (GPT-5) to 0.96 (o3-mini). Table 15 reports manual validation of the localization and majority-vote pipeline on 200 sampled tasks per system: localization accuracy is 84–89%, majority-vote F1 is ~87% for both systems, and majority-vote recall exceeds 98%—meaning the pipeline very rarely discards correct implementations. Table 16 reports accuracy evaluation on 100 sampled tasks under different judges: ZeroRepo attains 62.0% pass / 73.0% vote rate under human evaluation vs. 61.9–69.0% pass and 70.3–79.9% vote under automated judges—tight agreement. Table 17 audits 100 adapted tests per method for correctness: 91% of Claude Code tests and 90% of ZeroRepo tests are correct, with the remaining cases dominated by minor issues (import/name errors, signature mismatches) rather than systematic failures.

Model backbone comparison (not framed as ablation but functioning as one): The comparison between o3-mini and Qwen3-Coder across all results in Table 2 and Tables 18–23 reveals that the RPG framework is not trivially model-agnostic. o3-mini consistently achieves higher coverage (+6.4 points), novelty (+4.4 points), and pass rate (+12.4 points), while Qwen3-Coder generates larger repositories (1.5× LOC) but with lower correctness. Appendix B.2 (Figure 9, Figures 11–13) characterizes model-specific growth patterns and modularization strategies, showing that the RPG structure accommodates different model tendencies but does not eliminate performance differences. This is a negative result in the sense that the framework inherits model quality—it amplifies planning capability but does not equalize models.

Per-repository analysis of pass rates (Tables 18–23): An unintentional ablation emerges from the repository diversity. On HttpEasy (the smallest, most focused repository), ZeroRepo with both models achieves high pass rates (64.0% o3-mini, 54.0% Qwen3-Coder). On TableKit (the data analysis library with complex DataFrame semantics), pass rates are 81.4% and 48.0% respectively—a 33.4-point gap between models. This suggests that the RPG framework is robust across repository types but model quality interacts with domain complexity for the correctness dimension in ways the paper does not systematically analyze.

Critical Assessment

Does ZeroRepo actually generate 36K LOC of correct, coherent code at repository scale?

The paper's headline claim—"produces repositories with 36K Lines of Code and 445K Code Tokens, about 3.9× larger than Claude Code and 68× larger than other baselines" (Abstract)—requires careful scrutiny. The 36K LOC figure comes from ZeroRepo with Qwen3-Coder (Table 2). However, Qwen3-Coder achieves only 57.3% pass rate compared to o3-mini's 69.7%, meaning a substantial fraction of that 36K LOC represents code that fails adapted tests. The paper does not report how many LOC correspond to passing vs. failing functions, which makes it impossible to determine the correct-coder LOC. If only 57.3% of the code is correct (assuming pass rate correlates with code volume), the corrected-coder LOC would be approximately 21K—still larger than Claude Code's 10.6K LOC (with 33.9% pass rate, ~3.6K correct-coder LOC), but the multiple would be lower than 3.9×.

More fundamentally, the LOC metric includes all committed code (functions that pass tests) plus any infrastructure code (imports, base classes, configuration files) that is not directly tested. The paper commits only passing functions (Section 4: "Only functions that pass all tests are committed"), so the LOC in the repository should represent tested-and-passing code plus untested scaffolding. But the pass rate is computed only on the 1,052 evaluation tasks—not on every function in the repository. A function not covered by an evaluation task could be incorrect but still contribute to LOC. The paper's test coverage analysis (Table 11, Figure 14) shows 60–65% coverage at the class level, declining with complexity. This means 35–40% of the committed code is not exercised by any test, and its correctness is unknown. The true "correct LOC" may be substantially lower than the reported totals.

Does the RPG enable genuine long-horizon planning, or does it primarily enable enumeration of many features that are only superficially connected?

The scaling curves (Figures 5–6) show near-linear growth in feature count and LOC, which the paper interprets as evidence of sustained planning. However, the metrics do not distinguish between planning depth (complex interdependencies, coherent architecture) and planning breadth (many independent features). A system could achieve linear growth by adding many loosely related features in parallel, without deeper architectural planning. The dependency visualization in Figure 4 provides qualitative evidence of non-trivial structure, but the paper does not quantify dependency graph properties that would distinguish deep planning from broad enumeration—metrics like graph diameter, clustering coefficient, or dependency depth distribution. Without such metrics, it is unclear whether the RPG's advantage comes from better planning or simply from better enumeration (avoiding the omissions and biases of natural-language feature generation).

The structural isomorphism analysis (Tables 8–9) partially addresses this by showing that features map to code in a structured way, but this demonstrates the RPG's capacity for structural encoding rather than its capacity for coordinated planning across modules. A stronger demonstration would involve tasks that require cross-module coordination—e.g., implementing a pipeline where the output format of module A must exactly match the input format of module B—and showing that RPG-based approaches handle such constraints while natural-language approaches produce interface mismatches.

Is the comparison to baselines fair?

Several aspects of the experimental design favor ZeroRepo. First, ZeroRepo has access to the EpiCoder Feature Tree, a structured knowledge base of 1.5M software capabilities with embeddings, while baselines do not. This is an information asymmetry: ZeroRepo can retrieve relevant features from a curated ontology, while baselines must generate or discover them from scratch (though terminal agents do have web search). The ablation in Section 8 partially addresses this by showing that ZeroRepo without the Feature Tree still outperforms baselines, but the baseline comparison includes the Feature-Tree-enhanced version. A truly fair comparison would either give baselines access to the same knowledge base or compare against the ablated version.

Second, the iteration budget (30 iterations) is equal across methods, but the content of an iteration differs. ZeroRepo's iterations perform structured operations (explore–exploit subtree selection, graph refinement) while baselines' iterations involve open-ended natural language prompting ("please check whether the current repository still has any features that could be enhanced..."). The paper's prompt for baselines (Appendix F.1) explicitly asks them to self-reflect, which may be less efficient than the structured retrieval operations ZeroRepo performs. This is not necessarily unfair—it reflects the paper's thesis that structured approaches are more effective—but it means the comparison tests the approach (graph-structured vs. unstructured planning) rather than controlling for the amount of information processed per iteration.

Third, the Gold Projects achieve 81.0% pass rate and 92.0% voting rate under the evaluation pipeline—not 100%. This means the evaluation harness has a non-trivial error rate (~19% of tasks fail even for human-written code), likely due to test adaptation imperfections (the pipeline adapts ground-truth tests to match generated code's naming conventions; Table 17 shows ~9–10% of adapted tests contain errors). This puts the 69.7% pass rate for ZeroRepo in context: it represents 86% of the human ceiling (69.7/81.0), which is more informative than the raw percentage. However, the paper does not report pass rates normalized by this ceiling, making direct comparison to "perfect" generation misleading.

Are the coverage and novelty metrics measuring what they claim?

Coverage is computed by clustering generated features to reference categories with K-Means plus LLM refinement (Appendix E.3.1). This means a generated feature can be counted as "covering" a category even if its implementation is incorrect or incomplete—coverage measures presence, not quality. This is reasonable for a proposal-level metric, but the paper sometimes conflates coverage with functional completeness. The high coverage numbers (81.5–100%) coexist with pass rates of 54–81%, meaning 19–46% of covered features are incorrectly implemented—the repository "covers" the right topics but does not execute them correctly.

Novelty is computed as the fraction of generated features assigned to an out-of-distribution centroid during clustering. This captures whether the system proposes features outside the reference taxonomy, but it cannot distinguish between coherent extensions (e.g., adding a novel statistical test that fits the repository's domain) and incoherent additions (e.g., adding web scraping to a math library). The paper provides qualitative novelty examples in Appendix F.3 showing domain-appropriate extensions (e.g., "vector autoregression model" in a stats library), but does not quantify the quality of novel features—whether they are implementable, consistent with the repository's scope, or useful. A system could achieve high novelty by proposing many loosely related or semantically vacuous features, and the current metric would not penalize this.

What experiments would have strengthened the paper but were not run?

Several experiments are notably absent. First, the paper does not ablate the graph structure itself to identify which components are essential—e.g., comparing full RPG against a flat list of features with file assignments, or against a tree without cross-cutting edges (only hierarchical decomposition), or against the RPG with only structural semantics (no functional labeling). These ablations would identify whether the hierarchy, the edges, or the dual semantics are the critical enabler.

Second, the paper does not report on latency or wall-clock time. The graph construction process involves LLM calls for each iteration, batch self-checks, and semantic searches—operations that could be substantially slower than natural-language planning. The 30-iteration budget may mask a significant wall-clock disadvantage for ZeroRepo.

Third, the paper does not evaluate on a broader range of models or model scales. All experiments use o3-mini and Qwen3-Coder (both large, capable models). It is unknown whether the RPG framework provides benefits with smaller, weaker models, or whether the benefits saturate with model scale—would a GPT-4-level model without the RPG match o3-mini with the RPG?

Fourth, the paper does not evaluate task-level performance where cross-module coordination is the primary challenge. The evaluation tasks (derived from unit tests) test individual algorithm implementations, not end-to-end workflows that span multiple modules with complex data flow contracts. This favors breadth-oriented approaches and may underestimate the RPG's advantages for deep dependency management—or may overestimate them if the RPG's dependency encoding is not actually leveraged by the evaluation.

Conditional validity of claims.

The paper's central claim—that replacing natural language with a structured graph representation enables scalable, long-horizon repository generation—is supported by the scaling curves (Figures 5–6) and the ablation (Section 8), but with important conditions:

The claim holds for breadth of feature enumeration (near-linear growth in feature count and LOC), which is convincingly demonstrated.
The claim likely holds for structural consistency (the dependency visualizations and isomorphic mapping analysis suggest coherent code organization), but the evidence is qualitative rather than quantitative.
The claim may not hold as strongly for deep cross-module coordination—the evaluation design does not directly test this, and the pass rate gap between coverage (81.5%) and accuracy (69.7%) suggests that planning quality does not fully translate to execution quality.
The claim's magnitude (~4× larger, 27–36 point improvements) is measured against baselines that lack access to structured knowledge and use different operational models per iteration. Against a baseline given equivalent knowledge access (e.g., a RAG-enhanced terminal agent), the gap might be smaller.

The scaling claim—that natural-language planning saturates while graph-based planning scales linearly—is the paper's strongest empirical contribution and is well-supported by the controlled comparison in Figures 5–6 and the linear fit analysis in Section 8. This is an important finding independent of the absolute performance numbers.

6. Limitations and Trade-offs

Difficulty Estimation Cost Renders the $4\times$ Efficiency Gain an Upper Bound, Not a Realized Deployment Gain

The assumption or constraint. The compute-optimal framework requires estimating each prompt's difficulty before allocating the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. Section 3.2 acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference… our experiments do not account for this cost largely for simplicity"

The consequence. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The reported $4\times$ efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. A practitioner deploying this system would find that the actual cost per question is far higher than the headline numbers suggest, potentially erasing the efficiency advantage entirely.

The paper frames this as an exploration–exploitation tradeoff and suggests future work on training models to predict difficulty directly from the question text (Section 3.2, Section 8), but no such model is developed or evaluated. Until this gap is closed, the $4\times$ figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain.

What evidence exists in the paper. The difficulty estimation procedure is described in Section 3.2, and the 2048-sample count is stated there. The paper does not report what fraction of total compute the difficulty estimation represents, nor does it include this cost in any budget calculation or efficiency metric. The computed-optimal scaling curves in Figures 4 and 8 are plotted against "number of generations" for the strategy execution phase only—the difficulty estimation cost is invisible in these plots.

Mitigation status. The authors explicitly flag this as a limitation (Section 3.2, Section 8) and suggest "pretraining or finetuning models to directly predict difficulty of a question" as future work. They also observe that predicted (PRM-based) difficulty bins perform nearly as well as oracle bins (Figures 4 and 8), removing the need for ground-truth labels but not the need for the 2048 samples. An alternative—adaptive difficulty estimation using a small initial sample to estimate difficulty before committing the full budget—is discussed as a natural extension but not implemented. The limitation is acknowledged but not resolved; the paper provides a framework that shows what is possible given difficulty knowledge, while leaving the practical problem of acquiring that knowledge to future work.

Single Benchmark, Single Model Family—Generalization to Other Domains and Architectures Is Unproven

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. Several aspects of the findings could be model- or domain-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different reasoning styles might exhibit different difficulty-dependent scaling curves—beam search might over-optimize at different thresholds, or revisions might be more or less effective depending on the model's in-context learning capabilities.
The revision model's ability to learn from incorrect in-context examples depends on the base model's capacity for in-context learning and self-correction, which varies substantially across model families. A model with stronger self-correction capabilities might benefit more from revisions; a model with weaker in-context learning might see no benefit or even degradation (as observed with the ReST $^{EM}$ model in Appendix K).
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems, neither helping hard problems) generalize to other reasoning domains—code generation, logical reasoning, scientific QA—or to tasks requiring factual knowledge rather than inference. A domain where the base model's pass@1 is uniformly high or uniformly low would produce different difficulty distributions and potentially different optimal strategies.

The consequence. A practitioner applying this framework to a different model, domain, or task cannot assume the specific strategy recommendations (beam search for medium problems, best-of-N for easy) will transfer. The framework—conditioning strategy selection on estimated difficulty—may be general, but the computed-optimal policies are specific to the model–benchmark pair. Replicating the study for a new deployment would require re-running the full scaling analysis (generating per-question pass@1 estimates, sweeping strategy hyperparameters, cross-validating per-bin policies), which is itself extremely expensive.

What evidence exists in the paper. All results in Sections 5–7 are on MATH with PaLM 2-S*. The paper does not include any experiments on alternative benchmarks (e.g., GSM8K, HumanEval, ARC) or alternative model families (e.g., GPT-4, Claude, Llama). The authors acknowledge the single-benchmark limitation implicitly (Section 4: "We believe this model is representative") but do not test it. The paper provides no evidence about transferability.

Mitigation status. The limitation is acknowledged but not tested. The authors state their belief that the findings are representative and leave cross-domain validation to future work. This is a standard limitation for a first-study paper, but it means the practical guidance—"use beam search on medium problems, best-of-N on easy problems"—should be treated as a hypothesis requiring domain-specific validation rather than an established prescription.

The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained, Making the Pretraining-vs-Inference Comparison Favorable to Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A Chinchilla-optimal model trained with $14\times$ more total FLOPs would likely outperform a parameter-only-scaled model, because it would have seen proportionally more training data and would be better-calibrated. This makes the pretraining baseline weaker than it needs to be for a fair comparison. The reported advantages of test-time compute over pretraining—e.g., "+27.8% on easy questions at $R \ll 1$ " (Section 7, Figure 1 bar chart)—may shrink or reverse against a properly compute-optimal larger model. The paper's conclusion that "test-time compute can outperform a $14\times$ larger model" is therefore conditional on the larger model being trained in a specific (non-Chinchilla-optimal) way.

Additionally, the $14\times$ larger model uses only greedy decoding—no majority voting, no best-of-N, no search, no revision. Giving the larger model even a modest test-time compute budget (say, best-of-8) would create a much stronger baseline. The current comparison asks: "is test-time compute on a small model better than no test-time compute on a large model?"—which is not the question a practitioner faces. The relevant question is: "given a fixed total FLOPs budget, should I spend it on a larger model with some test-time compute or a smaller model with more test-time compute?" The paper does not answer this.

What evidence exists in the paper. Section 7 describes the FLOPs-matched setup, including the parameter-only scaling choice and the greedy decoding for the larger model. The paper does not report results with any amount of test-time compute applied to the larger model, nor does it compare against a Chinchilla-optimal larger model. The authors explicitly state this is left to future work.

Mitigation status. The limitation is explicitly acknowledged (Section 7, Section 8) but the comparison is presented as a main result without qualification in the abstract or introduction. The paper's claims about pretraining-vs-inference tradeoffs should be read with the caveat that the pretraining baseline is weakened by design. The authors frame their choice as "representative of a canonical approach" (the LLaMA paradigm), which is defensible—most deployed large models are not Chinchilla-optimal—but it means the comparison does not isolate the fundamental tradeoff between pretraining and inference compute.

Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The compute-optimal framework is built on the premise that test-time compute can improve performance by allocating the inference budget more intelligently. This premise holds only when the base model can produce correct solutions at some non-trivial rate—the proposal distribution must contain correct answers for search or revision to find them.

The consequence. On the hardest problems (difficulty bin 5), where the base model's pass@1 is near zero, no amount of test-time compute helps. Across all methods—search, revisions, and their compute-optimal combinations—bin 5 accuracy hovers at 1–3% regardless of compute budget (Figure 3, right; Figure 7, right). In the FLOPs-matched comparison, the bin 5 scaling line is essentially flat near 0–5% (Figure 9), and test-time compute shows a −52.9% relative disadvantage compared to the larger model at $R \gg 1$ (Section 7, Figure 1 bar chart).

This is a fundamental capability bound: test-time compute amplifies existing capability but does not create it from nothing. If the base model cannot solve a class of problems at all, no search or revision strategy will help—there are no correct solutions in the proposal distribution to find or refine. For problems that require knowledge, reasoning patterns, or capabilities that the base model simply does not possess, pretraining (or retrieval augmentation, or tool use) is the only viable path.

What evidence exists in the paper. The near-zero bin 5 accuracy is visible across all experiments: Figure 3 (right) shows bin 5 accuracy at 1–3% for all methods and budgets; Figure 7 (right) shows 2–3% for all sequential-to-parallel ratios; Figure 9 shows the bin 5 scaling line flat near 0–5% across all $R$ values. The paper is candid about this in the Section 7 takeaway box: test-time compute is effective "when problems are within the base model's rough capability range" but cannot address problems outside that range.

Mitigation status. The paper explicitly identifies this as a boundary condition for the effectiveness of test-time compute (Section 7, Section 8). It does not attempt to solve it, because the problem is inherent: test-time compute can only work with what the base model can generate. The practical implication—that difficulty estimation is doubly important, because it not only selects the best strategy but also identifies problems that should be escalated to a larger model or a human—is flagged but not developed into a system. The paper's framework provides a way to detect when test-time compute will fail (via the difficulty estimator), but no mechanism for addressing those failures beyond "use a larger model."

Verifier Over-Optimization Is a Hard Ceiling That the Framework Mitigates but Does Not Solve

The assumption or constraint. All test-time compute methods that use a learned verifier (PRM or ORM) are subject to over-optimization: as search becomes more aggressive—exploring more beams, looking further ahead—it finds solutions that score highly under the verifier but are actually incorrect. This is the test-time analog of reward hacking in RLHF. The paper assumes that verifier quality is sufficient for the compute budgets studied but documents that over-optimization becomes the dominant failure mode at higher budgets.

The consequence. Over-optimization imposes a hard ceiling on how much test-time compute can improve performance, regardless of budget. The paper's evidence for this is concrete:

Beam search degrades easy-problem performance at high budgets (Figure 3, right): bin 1 accuracy drops from ~78% to ~77% as budget increases from 4 to 256, while best-of-N continues to improve to ~88%. The most aggressive optimizer produces the worst results on easy problems.
Lookahead search—the most powerful optimizer, using 3-step rollouts to improve step-level scoring—paradoxically performs worst overall (Figure 3, left) despite using the same total generation budget. Its extra per-step cost reduces the number of beams explored, but even at equal cost it does not outperform simpler methods, suggesting the improved optimization is counterbalanced by increased verifier exploitation.
Qualitative examples in Appendix M show search producing degenerate outputs: repetitive low-information steps, overly short 1–2 step solutions that exploit verifier blind spots.

The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N where the verifier is reliable) and deploying beam search only on medium problems where the verifier signal provides genuine guidance. But on medium problems, over-optimization still limits the scaling ceiling—the beam search curves flatten and sometimes decline before the budget is exhausted (Figure 3, right, bin 3–4). The framework does not solve over-optimization; it works around it by staying below the over-optimization threshold per difficulty level.

What evidence exists in the paper. The over-optimization evidence is distributed across Section 5.3 (search algorithm comparison), Figure 3 (right, showing beam search degradation on bin 1–2), Figure 3 (left, showing lookahead underperformance), and Appendix M (qualitative failure examples). The paper explicitly identifies verifier over-optimization as a phenomenon (Section 5.3, Section 8) and discusses it as a primary bottleneck.

Mitigation status. The paper identifies and mitigates but does not solve the over-optimization problem. The compute-optimal policy is a mitigation: by conditioning strategy selection on difficulty, it avoids deploying aggressive optimization where the verifier is unreliable (easy problems). However, this only shifts the ceiling—it does not raise it. The paper explicitly calls for future work on "improving verifier robustness" (Section 8), suggesting adversarial training, ensemble verification, and constrained search as potential directions. These are not explored. A practitioner deploying this system would find that test-time compute scaling is effective only up to the point where the verifier's reliability degrades, and that this point is problem-dependent and difficult to predict a priori. The difficulty estimation mechanism provides a coarse solution (easy vs. medium vs. hard), but within a difficulty bin, over-optimization may still affect individual problems differently.

Sequential Revisions Are Inherently Serial—The Framework Ignores Latency and Wall-Clock Time

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency. Sequential revisions are inherently serial—each revision depends on the previous one—while parallel best-of-N can be executed simultaneously with sufficient hardware. The paper does not account for wall-clock time in any comparison or optimization.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly $64\times$ longer wall-clock time than one that runs 128 parallel samples simultaneously (assuming sufficient parallelism). For latency-sensitive applications—interactive assistants, real-time decision-making, user-facing systems—the sequential-heavy strategies favored by the compute-optimal policy on easy-to-medium problems may be impractical regardless of their accuracy advantages. The paper reports that sequential revisions marginally outperform parallel sampling in aggregate (Figure 6, right), and that easy problems in particular favor purely sequential revisions (Figure 7, right, bin 1–2). A practitioner building a latency-sensitive system might find that the wall-clock penalty of sequential revisions outweighs the accuracy benefit, but the paper provides no guidance for making this tradeoff.

What evidence exists in the paper. The paper does not report wall-clock time, latency, or any metric related to execution speed. The generation budget is the only cost metric. The sequential-vs-parallel analysis in Section 6 (Figure 6, Figure 7) optimizes only for accuracy at a given generation count, not for the accuracy-per-unit-time that matters in practice. The paper notes that "sequential revisions are inherently serial" in the context of describing the method (Section 6.1) but does not treat this as a limitation to be analyzed or mitigated.

Mitigation status. The limitation is not addressed. The paper's compute-optimal framework could in principle incorporate latency as an additional constraint (e.g., optimizing accuracy subject to both a total FLOPs budget and a maximum sequential depth), but this is not explored. The compute-optimal policies reported in Figures 4 and 8 are optimized for accuracy only, and a latency-aware optimization might produce substantially different recommendations—likely favoring more parallel strategies than the current policies.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a representational diagnosis for a problem that the field had previously framed as a model-capability challenge. Before this work, the dominant narrative around repository-scale code generation was: models are getting better at reasoning, context windows are growing, and agent architectures are becoming more sophisticated—therefore, full repository generation will eventually become tractable through these incremental improvements. The paper systematically disproves this by showing that Claude Code, Gemini CLI, and Codex CLI—all using state-of-the-art models with web search and multi-iteration refinement—plateau sharply in both feature count and code volume (Figures 5–6), while ZeroRepo with the same underlying model (o3-mini) sustains near-linear growth.

This is a reframing, not a paradigm shift. The paper does not propose a fundamentally new learning algorithm, a new model architecture, or a new theory of intelligence. It identifies a bottleneck—the planning representation—that the field had overlooked because the dominant framework treated planning as a reasoning problem rather than a representational one. The shift is from asking "how can we make models reason better about repository architecture?" to asking "what representation of the plan enables consistent reasoning at scale?" This is an important and actionable reframing, but it builds on existing ideas (graph-based planning, structured knowledge bases, test-driven development) rather than introducing a novel paradigm.

The magnitude of the shift is best understood through its diagnostic value. Prior work evaluated repository generation systems on absolute metrics at fixed iteration counts—coverage, correctness, file count—without measuring how performance scales as the planning horizon extends. This paper introduces scalability of planning as a first-class metric: does the system sustain linear growth, or does it saturate? The scaling curves in Figures 5–6 provide a diagnostic that the field previously lacked. A system that achieves 50% coverage at 10 iterations but saturates (like Claude Code) is qualitatively different from one that achieves 40% at 10 iterations but continues to 90% at 30 (like ZeroRepo without the Feature Tree). This diagnostic matters because it separates representational adequacy (can the plan accommodate more information without degrading?) from model capability (how good are the initial decisions?). The paper provides strong evidence that natural-language representations saturate while graph representations do not—a finding that should change how future systems are evaluated.

Reconciling prior contradictions. The paper implicitly reconciles a tension in the literature between multi-agent systems (which showed promise on small-scale tasks but degraded at scale) and terminal agents (which were more flexible but still produced fragmented implementations). The diagnosis is that neither approach was limited by the model or the agent architecture—both were limited by the natural-language planning medium. This explains why adding more agents (MetaGPT, ChatDev) or more iterations (Claude Code, Gemini CLI) did not solve the problem: the bottleneck was not the number of reasoning steps but the representation those steps operated on. This is a clean unification of previously disconnected failure modes.

Which research directions become more attractive. The paper makes graph-structured planning representations a central research topic for code generation and, by extension, for any long-horizon AI planning task. Research on better graph construction, dynamic graph updating, and graph-based verification becomes directly relevant to improving repository generation. Conversely, the paper makes pure natural-language planning for complex, long-horizon tasks a less attractive direction: if state-of-the-art models with web search saturate at ~200 features and ~3K LOC, incremental improvements in model reasoning are unlikely to close a 10× gap. The paper suggests that investment in structured knowledge bases (like the EpiCoder Feature Tree) may yield higher returns than marginal model improvements, since the ablation shows the knowledge base provides a ~20% speedup (slope 983 vs. 800) and the graph structure provides the fundamental scaling property.

The connection to test-time compute scaling is instructive but not developed by the paper. Both works identify a representation or selection mechanism as the scaling bottleneck: for test-time compute, it was verifier quality and over-optimization; for repository generation, it is the planning representation and consistency degradation. Both show that the bottleneck is not the underlying model's capability but the structure of the intermediate artifacts. However, the paper does not explore this connection, and the mechanisms are different enough that direct transfer is not obvious.

Follow-Up Research This Work Enables

Quantifying planning depth vs. planning breadth in RPG-generated repositories. The paper demonstrates near-linear scaling in feature count and LOC (Figures 5–6) but does not distinguish between deep planning (complex interdependencies, coherent cross-module architecture) and broad enumeration (many loosely related features added in parallel). A critical follow-up would analyze the dependency graph properties of RPG-generated repositories—graph diameter, clustering coefficient, dependency depth distribution, and cross-module edge density—and compare them against human-written repositories and natural-language baselines. The prediction is that RPG-generated repositories should exhibit dependency structures closer to human code (deeper graphs, more cross-module edges, higher clustering) than natural-language baselines, which should show flatter, more fragmented dependency patterns. If RPG-generated repositories show linear growth in feature count but dependency depth saturates, it would mean the RPG enables enumeration but not deep architectural planning—a weaker but still valuable result. If dependency depth also scales linearly, it would strongly validate the RPG as a true planning substrate. The paper's dependency visualization in Figure 4 provides qualitative evidence of non-trivial structure, but quantitative graph metrics are absent and would be straightforward to compute from the generated codebases.

Ablating RPG components to identify which structural features are necessary for scaling. The paper ablates the EpiCoder Feature Tree (Section 8) but does not ablate the RPG structure itself. Critical experiments include: (1) comparing the full RPG against a flat list of features with file assignments but no edges (only hierarchical decomposition, no data-flow edges, no inheritance edges), to test whether edges encoding dependencies are necessary for scaling or whether any structured representation outperforms natural language; (2) comparing against an RPG with only structural semantics (file and function assignments but no functional labels on nodes), to test whether the dual semantics are necessary or whether structural organization alone is sufficient; (3) comparing against an RPG where edges are present but not traversed in topological order during code generation (random order instead), to isolate whether the topological traversal or the graph structure itself drives correctness improvements. These ablations would identify which RPG properties are load-bearing: if the flat list also scales linearly, then the key innovation is externalization, not graph structure; if structural-only RPGs fail, then dual semantics are essential; if random-order traversal matches topological traversal in pass rate, then the graph's role is primarily in planning, not in code generation. These experiments are straightforward to run with the existing infrastructure and would significantly refine the paper's central claim.

Testing whether the RPG framework transfers to non-Python, non-library domains. All experiments are on Python libraries (ML, data analysis, symbolic math, web frameworks, HTTP clients). The RPG framework makes domain-agnostic claims about the benefits of structured planning representations, but library generation is a specific type of software engineering—it emphasizes API design, modular decomposition, and algorithm implementation, with relatively little emphasis on state management, concurrency, user interfaces, or database schemas. A strong test would apply ZeroRepo to: (1) a web application with database models, request handlers, authentication, and frontend-backend contracts (e.g., generating a Django or Flask application with user management and CRUD operations); (2) a systems programming task in a language with different modularization conventions (e.g., a Rust library with ownership semantics and trait-based polymorphism); (3) a data pipeline with complex cross-module data contracts (e.g., an ETL pipeline where output schemas must exactly match downstream input schemas). The prediction is that the RPG should transfer well to domains where the primary challenge is coherent modularization and interface design (libraries, APIs) but may struggle with domains requiring dynamic state management or complex temporal reasoning (web applications, systems code) because the current RPG encodes static dependencies but not runtime behavior. Failure on these tasks would not invalidate the approach but would clarify its scope.

Combining RPG-based planning with retrieval-augmented generation for terminal agents. The paper shows that terminal agents (Claude Code, Gemini CLI) plateau due to natural-language planning limitations, while ZeroRepo succeeds using graph-based planning plus a structured ontology. An obvious hybrid would give terminal agents access to the EpiCoder Feature Tree as a retrieval tool, allowing them to query for relevant capabilities during planning while maintaining their natural-language workflow for implementation. This would test whether the ontology or the graph structure is the primary enabler: if a terminal agent with Feature Tree access approaches ZeroRepo's coverage, then the ontology is the key; if it does not, then the graph structure (dual semantics, explicit edges, topological ordering) is load-bearing. The experiment is straightforward: modify Claude Code to include a search_feature_tree tool that queries the same vector index ZeroRepo uses, run on RepoCraft for 30 iterations, and compare coverage scaling curves against ZeroRepo and vanilla Claude Code. This would also address the fairness concern about the baseline comparison—giving baselines access to the same knowledge base ZeroRepo uses as its starting point.

Scaling model size to test whether the representational bottleneck is absolute or relative. All experiments use o3-mini and Qwen3-Coder (both large, capable models). The paper's central claim is that natural-language planning has fundamental scaling limits, not just that current models are too weak. This claim predicts that even much stronger models (e.g., GPT-5 or Claude-5 class) would still plateau with natural-language planning, just at a higher absolute level. A test would run Claude Code or Gemini CLI with the strongest available model on RepoCraft and measure whether the scaling curve shape changes (does it still saturate, just later?) or whether a sufficiently strong model can sustain linear growth with natural-language planning alone. If a model 10× more capable than o3-mini can sustain linear growth in natural language through iteration 30, then the paper's claim about fundamental limits would be falsified—the bottleneck would be model capability, not representation, and the RPG would be a bridge technology for weaker models. If even the strongest models plateau (as the current evidence suggests, since Claude 4 Sonnet is already a top-tier model and still saturates), the claim would be strengthened. This experiment is expensive but critical for understanding whether investment in structured planning representations or model scaling will yield greater returns for long-horizon tasks.

Evaluating novelty quality, not just novelty rate. The paper reports novelty rates of 11–13% (Table 2) but does not evaluate whether novel features are coherent (consistent with the repository's domain and scope), implementable (actually realized as working code), or useful (providing non-trivial extensions beyond the reference). A follow-up would have human software engineers rate a sample of novel features from ZeroRepo outputs on these three dimensions, using a rubric. The hypothesis is that RPG-guided novelty should be more coherent than novelty from natural-language baselines (because the graph enforces structural placement of new features relative to existing ones), but may be less creative (because the feature tree constrains the space of possible additions). If RPG novelty is both more coherent and comparably creative, it would validate the graph as a scaffold for principled innovation rather than just enumeration. If RPG novelty is more coherent but notably less creative, it would suggest a precision-recall tradeoff in the ontology-based approach that practitioners should be aware of. The paper provides qualitative examples (Appendix F.3) suggesting domain-appropriate novelty, but systematic evaluation is absent.

Practical Applications and Downstream Use Cases

Rapid prototyping of domain-specific libraries from natural language specifications. The most direct application of ZeroRepo is generating initial, functional codebases for domain-specific Python libraries from high-level descriptions. A data scientist who needs a custom preprocessing library, a researcher who needs a specialized statistical modeling toolkit, or a startup building an internal machine learning framework could describe the desired capabilities in natural language and receive a complete, tested repository with ~24K–37K LOC (Table 2), ~81% coverage of intended functionality, and ~70% pass rate on core algorithms. This would serve as a starting point for human refinement rather than a finished product—the pass rate leaves room for improvement, and the 60–65% test coverage means substantial code is untested—but it eliminates the most time-consuming phase of repository creation: establishing the architecture, file structure, interfaces, and baseline implementations. The paper's structural isomorphism analysis (Tables 8–9) suggests the generated code follows conventional organizational patterns (modules map to directories, leaves map to functions), making it immediately comprehensible to human developers. The time savings relative to starting from scratch are substantial: a human team might spend weeks designing architecture and scaffolding before writing any algorithm code; ZeroRepo produces an equivalent structure in a single automated run.

Automated generation of educational codebases for teaching software architecture. The RPG's explicit encoding of modular boundaries, data flows, and interface contracts (Figure 2, Figure 4) makes generated repositories valuable as teaching artifacts. An instructor could describe a system (e.g., "a simple machine learning library with data preprocessing, three classifier types, and evaluation metrics") and use the generated RPG and codebase to illustrate architectural principles: how functional decomposition maps to directory structure, how base classes enforce interface consistency, how data flows impose module ordering. Because the RPG is a structured, queryable artifact (not buried in prose), students can explore it systematically—traversing edges to understand dependencies, expanding nodes to see interface contracts, comparing the graph against the generated code to understand plan-to-implementation mapping. The paper's localization tools (Appendix D.1) could be repurposed as educational exploration tools. This use case leverages the RPG's dual semantics explicitly: the graph is worth studying in its own right, not just as a means to generate code.

Cost-efficient batch generation of training data for code models. The FLOPs-matched comparison in the test-time compute paper showed that smaller models with compute-optimal inference can substitute for larger models on specific difficulty tiers. ZeroRepo provides a complementary capability: generating large volumes of structurally coherent, multi-file repositories that could serve as training data for code models. Current code training datasets (e.g., The Stack, CodeParrot) consist primarily of individual files or small groups of files without explicit repository-level structure. ZeroRepo could generate thousands of repository-scale training examples, each with: (1) the complete codebase (24K–37K LOC, Table 2); (2) the corresponding RPG (providing explicit structural annotations—module boundaries, data flows, interface contracts—that could be used as training targets for models learning to predict repository structure from natural language); (3) passing tests for ~70% of implemented functions (providing verification signals). The cost per repository would be dominated by the LLM API calls for graph construction and code generation, but at scale this could be competitive with human-authored repositories (which are far more expensive to create and rarely include explicit structural annotations). The paper does not explore this use case, but the RPG's dual semantics make it uniquely suited: it is simultaneously a planning artifact and a training label.

On-device or edge deployment through distilled planning. While ZeroRepo uses large models (o3-mini, Qwen3-Coder) for planning, the generated code is ordinary Python that can run anywhere. This suggests a deployment architecture where a powerful cloud model generates the RPG and codebase, but a smaller on-device model handles subsequent incremental updates, bug fixes, and feature additions by operating on the existing graph structure rather than replanning from scratch. The paper's graph-guided localization results (Table 4: 30–53% reduction in localization steps with graph guidance) suggest that even a weaker model could be effective at navigating and modifying an existing RPG, since the graph provides explicit navigation cues that reduce the reasoning burden. This is analogous to the test-time compute paper's finding that small models with smart inference strategies can substitute for larger models on specific tasks—here, the RPG serves as the "smart inference strategy" for codebase navigation. A downstream evaluation would measure whether a small model (e.g., a 7B-parameter code model) given RPG-guided tools can successfully add features to a ZeroRepo-generated codebase compared to the same model operating on a codebase without graph annotations.

When to Prefer This Method

The paper positions ZeroRepo directly against natural-language planning approaches (multi-agent systems, pipeline workflows, terminal agents) and provides clear empirical evidence about where the graph-based approach dominates. The decision rule is:

Prefer ZeroRepo / RPG-based planning when:

The target repository has non-trivial architectural complexity with multiple interacting modules—the coverage gap between ZeroRepo and baselines widens with repository complexity (97.9% vs. 59.6% on MLKit-Py with 47 functional categories, but 100% vs. 50% on HttpEasy with only 22 categories; Tables 18–19), suggesting the graph structure's advantage compounds with the number of components to coordinate.
Long-horizon planning consistency is critical—if the repository will undergo multiple rounds of feature addition and refinement, the RPG's near-linear scaling (Figures 5–6) provides a sustainable growth trajectory that natural-language baselines cannot match (saturation at iteration 10–15).
Interface correctness and modular boundaries matter—the RPG's typed data-flow edges and base-class abstraction provide explicit contracts that the code generation stage enforces, yielding higher pass rates (69.7% vs. 33.9% for Claude Code; Table 2) and more coherent code organization (Figure 4).
You have access to a structured knowledge base or ontology for the target domain—the EpiCoder Feature Tree provides a ~20% acceleration (slope 983 vs. 800; Section 8), but the RPG structure itself provides the fundamental scaling benefit, meaning partial or bootstrapped ontologies may still be valuable.

Prefer natural-language terminal agents (Claude Code, Gemini CLI) when:

The target repository has simple, flat structure with few cross-module dependencies—on HttpEasy (Table 19), Claude Code achieves 50% coverage and 36% pass rate, vs. ZeroRepo's 100% and 64%—significant but proportionally smaller gaps than on complex repositories, and the terminal agent's lower overhead may be preferable.
Wall-clock latency is the primary constraint—the paper does not report latency, but ZeroRepo involves multiple LLM calls per iteration for graph construction, batch self-checks, and localization, likely making it slower per unit of useful output than simpler agents, especially for small repositories.
Creative, open-ended exploration beyond known ontologies is desired—the EpiCoder Feature Tree constrains feature proposals to existing taxonomy entries, which improves coverage but may limit genuinely novel architectural patterns that do not fit into the ontology's hierarchy. The novelty metric (11–13%) captures extensions within the ontology's framing, but not entirely new organizational paradigms.
You lack a domain-specific structured knowledge base and cannot afford to construct one—the Feature Tree is a 1.5M-node pre-built resource; bootstrapping a comparable ontology for a niche domain would require substantial upfront investment that the paper does not address.