Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining

ArXiv: 2603.11103

🎯 Pitch

Standard code pretraining only sees the final snapshot, not the reasoning that built it. This paper shows you can mine the missing development process from static repos—synthesizing planning, debugging, and iterative refinement trajectories—and that pretraining on this reconstructed history boosts Llama-3-8B’s code generation, long-context understanding, and agentic reasoning with no extra data.

1. Executive Summary

This paper introduces a novel data-centric pretraining paradigm—understanding via reconstruction—that reverse-engineers the latent agentic trajectories behind static code repositories to provide richer supervision signals than raw code alone. The authors develop a multi-agent simulation framework (a main agent generates high-level requirements and implementation plans while sub-agents handle per-file code generation using Read and Write tools) grounded in structural repository information (file hierarchies, dependency graphs, AST-extracted elements), then apply a search-based CoT optimization technique (iteratively refining Chain-of-Thought reasoning steps to minimize the perplexity of the ground-truth target code) to ensure logical rigor. When continuously pretrained on these reconstructed trajectories, Llama-3-8B achieves substantial gains over a raw-code baseline: 37.20 vs. 34.76 on HumanEval for coding, 61.80 vs. 57.10 on Ruler at 64k context for long-context understanding, and 30.10 vs. 29.02 overall on APTBench for agentic capability, establishing that learning from the reconstructed development process enhances downstream performance across reasoning, coding, and agentic benchmarks while requiring no increase in total training data volume—the 12% experimental data slot is held constant across all comparisons.

2. Context and Motivation

The Core Problem: Static Code Is a Compressed Artifact Missing Its Generative History

The fundamental problem this paper addresses is that standard pretraining data for code LLMs—static software repositories—represents only the terminal state of an intricate intellectual process, not the process itself. The authors articulate this in their introduction with a precise formulation:

"A software repository, in its final form, is the terminal state of an intricate intellectual process. It is a highly compressed artifact where the 'computational steps' of human reasoning—the requirement analysis, architectural planning, trial-and-error debugging, and iterative refinement—have been abstracted away."

This compression creates a supervision gap: when a model is trained solely on the final code, it learns to predict what code looks like (surface-level structural patterns, common idioms, syntactic regularities) but not why that code was written that way or how a developer arrived at it. The training objective—next-token prediction—becomes a memorization task rather than a reasoning task. The model sees only the destination, never the map.

The authors explicitly connect this to the broader "understanding via generation" paradigm that underpins modern LLM pretraining (citing Floridi & Chiriatti, 2020 and Ouyang et al., 2022). By learning to generate text token-by-token, models internalize semantics and world knowledge. But this paradigm faces a fundamental limit when applied to long-horizon artifacts like software repositories: the generation process being learned is the wrong one. The model learns to generate code as it appears in repositories—complete, polished, and final—rather than learning to generate code as a developer would—incrementally, through planning, reading dependencies, testing, and debugging.

This gap manifests in a well-documented failure mode: models that excel at generating short, self-contained code snippets (e.g., completing a function body, solving a LeetCode problem) often struggle with the deep, causal logic required to construct and maintain complex software systems. The authors cite SWE-Synth (Pham et al., 2025) as evidence of this gap. The specific abilities that suffer include:

• Long-horizon planning: understanding how changes to one file propagate through a dependency graph and affect distant components.
• Debugging and iterative refinement: recognizing why an initial implementation failed and how to adjust it, as opposed to generating a correct solution in one shot.
• Tool use and information gathering: knowing when to inspect other parts of the codebase (via reading files, checking signatures, examining documentation) before writing code that depends on them.
• Context management across files: maintaining coherence when a repository spans dozens or hundreds of files with complex inter-relationships.

These are precisely the skills required for real-world software engineering, and they are systematically underrepresented in standard pretraining data because they are invisible in the final code artifact.

Why This Problem Matters: Practical and Theoretical Significance

The problem has significant practical implications for how we build and deploy code LLMs. The dominant paradigm for improving model capabilities is scaling pretraining compute—larger models, more data, longer training. But if the pretraining data itself is structurally impoverished, scaling may yield diminishing returns: a model can memorize more repositories more accurately without fundamentally improving its ability to reason about software construction. The authors' approach proposes an alternative path: rather than scaling the quantity of pretraining data, improve its informational density by restoring the missing generative context. This is a data-centric rather than model-centric approach to capability improvement.

The theoretical significance lies in the paper's reconceptualization of what constitutes good supervision for learning complex generative processes. The dominant view in language model training is that next-token prediction on naturally occurring text provides sufficient inductive bias for models to learn whatever capabilities are embedded in the data distribution. This paper challenges that view by arguing that for certain capabilities—specifically, the ability to construct complex artifacts through multi-step reasoning and tool use—the naturally occurring data distribution is causally incomplete: the relationship between requirements (what needs to be built) and code (what gets written) cannot be learned from observing code alone because the intermediate reasoning steps that connect requirements to code are not present in the data. The paper's proposed solution—explicitly reconstructing those intermediate steps—is a concrete instantiation of the broader idea that generative models need generative supervision.

Prior Approaches and Their Limitations

The paper identifies three broad categories of prior work, each with distinct shortcomings that motivate the proposed approach.

1. Reasoning Recovery from Pretraining Data

Several prior works have attempted to extract or amplify reasoning signals from existing pretraining corpora. The paper cites three specific approaches:

Quiet-STaR (Zelikman et al., 2024) operates at the token level by training models to generate implicit rationales—internal monologue tokens—that minimize future-token uncertainty. The model learns to "think before speaking" by inserting latent reasoning steps between observed tokens, with the rationale tokens optimized to reduce perplexity on subsequent text. This approach recovers local reasoning at the token or sentence level but does not capture the structured, multi-step, tool-mediated reasoning characteristic of software development. A developer's thought process involves file-level planning, reading dependencies, writing entire functions, and debugging—not just inter-token reasoning. Quiet-STaR's token-level granularity is mismatched to the scale and structure of repository-level reasoning.

BOLT (Pang et al., 2025) learns latent reasoning for pre-training documents via an expectation-maximization (EM) framework, systematically bridging the gap between raw text and logical derivation at the document level. This is a step toward structural reasoning recovery, but it still operates on individual documents rather than multi-document artifacts like repositories, and it does not model the interactive, tool-using aspects of software development—reading files, checking interfaces, writing code incrementally.

Thinking Augmented Pre-training (TPT; Wang et al., 2025) prepends synthetic thinking trajectories to pre-training corpora, reallocating computational budget toward "logic-dense" segments. This approach shares the paper's intuition that augmenting raw text with reasoning traces improves data efficiency, but TPT's thinking trajectories are not grounded in any ground-truth generative process. They are synthetic additions that may or may not reflect the actual reasoning behind the target text. The paper's approach differs fundamentally: the reconstructed trajectories are causally connected to the target code through the search-based optimization (maximizing log p(x|z)), ensuring that the reasoning actually explains and predicts the code, rather than being a loosely associated commentary.

REER (Wang et al., 2025b) is the closest prior work, introducing a reverse-engineering approach for open-ended generation that uses perplexity-driven path searching to reconstruct the logical scaffolding behind high-quality reference answers. The paper explicitly builds on REER's methodology (the perplexity-minimization search over reasoning steps). However, REER focuses on isolated reasoning tasks (open-ended text generation), not on the holistic agentic trajectory of repository construction. The paper extends REER's insight from single-step reasoning to multi-agent, multi-file, tool-mediated development processes.

The key limitation across all these approaches is that they recover isolated reasoning steps—the reasoning behind a single answer, a single document, or a single code snippet—rather than reconstructing the full generative process of a complex artifact. Software development is not just reasoning about code; it is reasoning about architecture, planning file creation order, reading dependencies, handling errors, and revising based on feedback. These prior works capture fragments of this process but not its structure.

2. Synthetic Agent Trajectories

A separate line of work constructs agent trajectories for training, but the paper identifies two sub-categories, each with fundamental tradeoffs:

Environment-based trajectory generation (Fang et al., 2025; Pahuja et al., 2025; Team, 2025; Wang et al., 2025a) places agents in real or simulated interactive environments and records their trajectories. The advantage is authenticity: the trajectories reflect genuine agent-environment interaction, including realistic failures, retries, and feedback loops. The disadvantages, which the paper highlights, are severe:

"This approach has significant drawbacks, including potentially expensive tool invocation costs and substantial engineering efforts required for environment setup and maintenance."

Real-world environments (e.g., actual code execution, file system operations, web browsing) require infrastructure that is costly to maintain at scale, introduces latency, and may produce non-reproducible results. For generating millions of trajectories across hundreds of thousands of repositories, this approach is computationally and logistically prohibitive.

LLM-simulated trajectory generation (Chen et al., 2025; Li et al., 2025; Sengupta et al., 2024) uses LLMs to simulate both the agent's actions and the environment's responses, avoiding the cost and complexity of real environments. But this introduces a reliability problem: the simulated environment may not reflect reality, and the trajectories may contain extensive hallucinations—the LLM invents file contents, tool responses, or dependency relationships that don't exist. The resulting data can train models to exhibit behaviors that don't work in real environments.

The paper's approach synthesizes trajectories using LLM simulation (inheriting its low cost) but grounds the simulation in extracted repository structure to address the hallucination problem. Critically, the terminal state of every trajectory is guaranteed to be a real, complete repository—not a hallucinated one—because the Write tool outputs are replaced with ground-truth code and Read tool responses are replaced with actual file contents. This hybrid approach aims to capture the best of both paradigms: the cost-effectiveness of LLM simulation with the fidelity of ground-truth grounding.

3. Synthetic Data for Code Generation

The paper situates itself within the broader landscape of synthetic data for code, but identifies two dimensions where prior work falls short:

Scope: isolated snippets vs. entire repositories. Most prior work focuses on generating instruction-following datasets for isolated code snippets. Magicoder (Wei et al., 2023) synthesizes user instructions for open-source code snippets. Code Alpaca (Chaudhary, 2023) generates 20,000 code instructions via self-instruct (Wang et al., 2023). WizardCoder (Luo et al., 2023) uses an evolutionary pipeline to increase instruction complexity. Case2Code (Shao et al., 2025) collects input-output test cases from existing programs and generates new programs that satisfy them. SWE-Synth (Pham et al., 2025) generates synthetic data for bug-fixing. These approaches all operate on isolated functions or single files, not on the multi-file, dependency-rich structure of real repositories. The authors argue that this misses the cross-file reasoning, architectural planning, and dependency management that characterize real software development.

Content: final code vs. the development process. Even when prior work generates code for entire repositories (as in the pretraining corpora for models like Qwen2.5-Coder (Hui et al., 2024) and DeepSeek-Coder (Guo et al., 2024), cited in the paper), the data consists of the final code state—not the process that created it. The paper argues that what is missing is the agentic process:

"Rather than merely capturing the final code or the associated chain-of-thought, we reconstruct the entire agentic process of developing a repository. This involves synthesizing a sequence of actions, tool interactions, and evolving states, thereby providing a more comprehensive and realistic representation of the software development lifecycle."

This is a qualitative difference in the nature of the supervision signal. Standard pretraining teaches the model: "given this code context, predict the next token." The reconstructed trajectories teach the model: "given this requirement, plan the architecture; given this plan, read dependencies; given those dependencies, write this file; given that file, check its interface before writing the next one." The model learns not just what code looks like, but how and why it gets written in a particular order with particular dependencies.

How This Paper Positions Itself

The paper's positioning can be understood along four axes:

1. Data-centric rather than model-centric. The paper explicitly frames its contribution as a data paradigm, not a new architecture or training algorithm. The pretraining procedure is standard (continual pretraining with a fixed data mixture), and all gains are attributed to the informational quality of the reconstructed trajectories compared to raw code. This positions the work as complementary to model-scaling approaches: rather than competing with larger models, the method aims to extract more value from existing models by improving the data they are trained on.

2. Generative supervision for generative capabilities. The paper's core philosophical stance—articulated in its title, "Understanding by Reconstruction"—is that to learn to generate complex artifacts, models need to observe the generative process, not just the final product. This is a specific, operationalized claim about what constitutes good supervision for long-horizon reasoning tasks. It connects to broader debates in AI about whether next-token prediction on static data is sufficient for acquiring planning and reasoning capabilities, or whether explicit process supervision is necessary.

3. Grounded simulation as a practical middle ground. The paper positions its trajectory generation method as resolving the tension between authenticity (real environments) and cost (LLM simulation) in prior work on synthetic agent trajectories. By grounding the simulation in extracted repository structure and guaranteeing the terminal state is real code, the method achieves fidelity without infrastructure complexity. The authors acknowledge that the resulting trajectories may contain some noise (LLM-generated reasoning may not perfectly reflect actual developer reasoning), but they argue that this is acceptable for pretraining—which is inherently robust to noisy data—and that the search-based CoT optimization further refines the reasoning quality.

4. A unified reconstruction of the full development lifecycle. The paper distinguishes itself from prior reasoning-recovery work (Quiet-STaR, BOLT, TPT, REER) by reconstructing not just isolated reasoning steps but a holistic agentic trajectory that integrates:

• High-level architectural planning (main agent decomposing requirements into a file creation sequence)
• File-level action sequencing (sub-agents reading dependencies before writing code)
• Iterative tool use (Read and Write tools with explicit observations)
• Cross-file dependency reasoning (the order of file creation reflecting the dependency graph)

This unified trajectory captures the multi-dimensional generative process of software development in a way that single-level reasoning traces cannot. The paper's key hypothesis is that this structural richness—the explicit modeling of the planning→reading→writing→reviewing loop—provides a qualitatively better supervision signal than raw code for teaching models to reason about software construction.

The paper does not claim that its reconstructed trajectories perfectly mirror actual developer cognition. Rather, it claims that even an approximate reconstruction of the development process—one that captures the causal structure of requirement→plan→read→write→next file—is substantially more informative than the static final state alone. The search-based CoT optimization further ensures that the reconstructed reasoning is not just plausible but actually predictive of the target code, closing the loop between reasoning and outcome.

3. Technical Approach

3.1 Reader Orientation

This paper builds a data synthesis pipeline — not a new model architecture or training algorithm. The system takes a static code repository as input and produces a long, sequential text document that narrates the entire imagined process of building that repository: from high-level requirements and architectural planning, through per-file implementation with tool use (reading dependencies, writing code), to the final completed project. This synthetic narrative — called an agentic trajectory — becomes pretraining data for a language model.

The problem it solves is that standard pretraining on raw code teaches models only what finished code looks like, not how or why it was written. The "shape" of the solution is therefore: convert each repository into a causally rich, step-by-step development story, then train the model on these stories instead of (or alongside) raw code files. The core bet is that the additional reasoning tokens — the plans, the dependency checks, the implementation decisions — provide a qualitatively denser supervision signal than the final code alone, without increasing the total number of training tokens (the experimental data slot is held at a fixed 12% of the training budget).

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged as a sequential pipeline:

1. Repository Analyzer — ingests a raw GitHub repository and extracts structural ground truth: the complete file/directory tree, a directed inter-file dependency graph (built from import statements), and per-file intra-file structure (class and function signatures extracted via Abstract Syntax Tree parsing). This component produces the "facts" that will ground the simulation.

2. Multi-Agent Trajectory Simulator — uses a prompted LLM (Qwen3-30B-A3B-Instruct-2507, a Mixture-of-Experts model with 30B total parameters but only 3B active) to simulate two types of agents: a Main Agent that generates high-level requirements and a file-by-file implementation plan, and multiple Sub-Agents (one per file) that implement individual files by first reasoning about dependencies and using simulated Read/Write tools. The simulator outputs a complete trajectory — a chronological sequence of thoughts, tool calls, and tool responses.

3. Trajectory Grounding Module — overwrites the LLM-generated tool responses with actual repository data. Read tool outputs are replaced with the real content of the target file. Write tool outputs are replaced with the ground-truth code of the file being created. This ensures the trajectory's "observations" are factually correct, even though the "thoughts" are synthetic.

4. CoT Optimizer — applies a search-based refinement procedure to the reasoning (Chain-of-Thought) steps within the trajectory. For each reasoning step, it samples alternative "refinements" from an LLM, evaluates which refinement maximises the conditional probability (minimises the perplexity) of the ground-truth code that follows, and replaces the original step with the best refinement if it improves the score. This runs for 3 rounds with 2 candidate refinements per step.

5. Pretraining Data Flattener — converts the hierarchical multi-agent interaction (main agent calls sub-agent; sub-agent executes and returns) into a single flat text document by recursively inlining each sub-agent's trajectory into the call point in the main agent's trajectory. It applies a targeted loss mask that prevents the model from learning to predict Observation tokens (tool responses), forcing it to learn only the Think and Action tokens.

The flow is: Repository → Analyzer (extract structure) → Simulator (generate trajectory) → Grounding Module (replace hallucinations with facts) → CoT Optimizer (refine reasoning quality) → Flattener (produce pretraining token sequence) → Pre-training run.

3.3 Roadmap for the Deep Dive

I will explain the components in the order data flows through the pipeline, which is also roughly the order of increasing conceptual complexity:

• First, the Repository Analyzer — what structural information is extracted and why each piece matters for grounding the simulation. This is the foundation; without it, the simulation would drift into hallucination.
• Second, the Multi-Agent Trajectory Simulation — the design of the two agent roles, the prompt structure that induces their behaviour, the tool definitions (Read/Write), and how the simulation produces a complete chronological trajectory.
• Third, the Trajectory Grounding Mechanism — exactly which parts of the LLM-generated trajectory are overwritten with ground-truth data, how Read and Write responses are replaced, and why this partial grounding is sufficient.
• Fourth, the CoT Optimization via Search — the formal objective (maximising log p(x|z)), the decomposition into per-step refinement, the sampling and evaluation procedure, the perplexity metric, and the iterative update rule.
• Fifth, the Training Data Preparation — trajectory flattening (how the hierarchical multi-agent structure becomes a linear sequence), targeted loss masking (which tokens are masked and why), and the pretraining configuration (data mixture, context window, model choice, training budget).

This ordering follows the pipeline from raw repository to training-ready token sequence, with each step building on the output of the previous one.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data curation paper whose core idea is that the process of constructing a repository — planning, reading dependencies, writing code incrementally — provides richer supervision for code LLMs than the final static code alone. The technical contribution is the pipeline that synthesises these process trajectories at scale.

Repository Analysis: Extracting Structural Ground Truth

Before any simulation begins, the system analyses each source repository to extract three kinds of structural information. This extraction is fully deterministic — it uses standard software engineering tools (file system traversal, import parsing, AST parsing) rather than LLMs — and serves as the "source of truth" that constrains the subsequent simulation.

File Structure Tree. The system enumerates every file and directory in the repository and constructs a complete hierarchical tree representation. This is provided directly to the Main Agent during simulation as part of its context (see the prompt template in Appendix D of the paper):

The tree structure of repo: $file_tree.

The tree serves two purposes. First, it gives the Main Agent a complete inventory of what needs to be built, enabling it to decompose the project into a logical sequence of file creation steps. Second, it provides the Sub-Agents with awareness of the full project scope, so that when a Sub-Agent needs to read a dependency, it knows what files exist and can reference them by name.

Inter-File Dependency Graph. The system parses all import statements (e.g., from module import function, import package) across all files in the repository and constructs a directed graph where an edge A → B means "file A depends on file B" (A imports from B). The authors describe this as:

"We analyze import statements to build a graph representing how files depend on one another. This is important for the LLM to simulate the tool call and tool response of Read Tool."

The dependency graph is critical for two aspects of the simulation. First, it determines a valid implementation order: files with no dependencies (leaves in the dependency graph, or files that only import from external libraries) should be created first, and files that depend on them should be created later, so that when a Sub-Agent reads a dependency, that dependency has already been "implemented" in the simulation. Second, it informs the Sub-Agent's decision of which files to read: if file main.py imports from operations.py, the Sub-Agent for main.py should simulate reading operations.py to check its interface before writing main.py. The dependency graph provides the ground truth for which Read calls are plausible and which files actually need to be inspected.

Intra-File Structure via AST Parsing. For each file, the system parses its Abstract Syntax Tree to extract key structural elements — primarily class definitions, function signatures, and their associated docstrings or type annotations. The paper states:

"For each file, we parse its Abstract Syntax Tree (AST) to extract key structural elements like class and function definitions. This information is provided to the LLM to simulate the Sub-Agent trajectory."

This intra-file information is included in the prompt that generates the Sub-Agent's trajectory. It provides the simulator with the "end goal" — the functions and classes that the Sub-Agent's implementation must produce — without revealing the full implementation. The Sub-Agent must then reason about how to implement those signatures, which is where the interesting reasoning (algorithm choices, edge case handling, internal logic) is generated. The full source code is not provided to the Sub-Agent during reasoning; it is only used to overwrite the final Write tool output (see the grounding mechanism below).

Why extract all three, and why in this combination? The file tree provides scope (what needs to be built). The dependency graph provides order (what must be built first, what needs to be read before writing). The AST signatures provide targets (what each file must ultimately contain). Together, they constrain the simulation so that it produces trajectories that are structurally faithful to the real repository: the implementation order respects actual dependencies, the Read calls reference files that actually exist and are actually imported, and the generated code ultimately matches the ground-truth file contents. Without these constraints, an ungrounded LLM simulation would produce trajectories that sound plausible but diverge from any real repository — files would be read in arbitrary order, dependencies would be invented, and the final code would not match any real codebase.

Multi-Agent Trajectory Simulation

The core of the data synthesis pipeline is a multi-agent workflow simulation, implemented entirely through LLM prompting. The authors use Qwen3-30B-A3B-Instruct-2507 (a 30B-parameter Mixture-of-Experts model with 3B active parameters) as the simulation engine. The design mirrors a human software development process with two distinct agent roles, a task specification, and two simulated tools.

The Task Specification (User Prompt). Every trajectory begins with a user-provided task description that states, at a high level, what the project should accomplish. The prompt template for this is given in Appendix D:

A detailed requirement document for repo, but DO NOT mention implementation details of repo.

The instruction explicitly forbids mentioning implementation details — the user prompt must describe what the project does, not how it is implemented. This creates the correct information asymmetry for the subsequent simulation: the Main Agent receives a high-level goal and must reason about how to decompose it, just as a developer receives a product requirement and must design the architecture.

Main Agent: Project Planning and Orchestration. The Main Agent is responsible for the "architect" role. Given the full repository code and the file structure tree as context, it performs two sequential tasks:

1. Generate project requirements: The Main Agent synthesises a high-level description of the project's purpose and functionality. This is not a real user requirement but is generated by the LLM based on its understanding of what the repository (which it can see in full) accomplishes. The generated requirement serves as the "seed" that motivates the subsequent development plan.

2. Formulate an implementation plan: The Main Agent decomposes the project into a logical, dependency-aware sequence of file creation steps. The plan specifies which files should be created and in what order. The paper describes this as:

"Decompose the project into a logical sequence of file creation steps. This plan outlines which files should be created and in what order, establishing a dependency-aware development path."

The Main Agent's thinking is recorded as Think steps in the trajectory, and its orchestration actions — invoking Sub-Agents for each file — are recorded as Action steps using a CallSubAgent tool, with arguments specifying the file name, path, and implementation requirements.

Sub-Agent: Per-File Implementation with Tool Use. For each file in the implementation plan, the Main Agent spawns a Sub-Agent. The Sub-Agent is responsible for generating the code for exactly one file. Its workflow involves three phases, each recorded as trajectory steps:

1. Plan file implementation (Think): The Sub-Agent reasons about the file's structure — what classes or functions it needs, what logic it must contain, what edge cases to handle. This reasoning is informed by the requirement for that specific file (passed from the Main Agent), the overall project context (the tree structure, also passed from the Main Agent), and any dependency information the Sub-Agent discovers by reading other files.

2. Information gathering (Read Tool): Before writing code that depends on other modules, the Sub-Agent may need to inspect those modules to understand their interfaces. It simulates this by emitting a Read tool call specifying which file to read. The Read tool is defined to return "the definition/signature of a file" (from the system prompt in Appendix D). The Sub-Agent emits a Think step explaining what it is looking for, a Read(file="...") action, and then an Observation step containing the file's content. The paper gives the example in Table 1:

"I need to create main.py. This file must import the add function from operations.py. To ensure correctness, I should first read the content of operations.py."

This mirrors a developer checking the API of a dependency before writing code that calls it — a key aspect of real software development that is invisible in static code.

3. Code generation (Write Tool): After gathering necessary context, the Sub-Agent reasons about the implementation (another Think step) and then emits a Write(file="...", content="...") tool call containing the full source code for the file. The Write tool is defined to "write the code to the file system" (system prompt, Appendix D).

The entire sequence of thoughts, Read calls, observations, and Write calls for one file constitutes the Sub-Agent's trajectory. When the Main Agent calls a Sub-Agent, the simulation engine recursively injects the Sub-Agent's trajectory into the call point.

Table 1 Example. The paper provides a concrete miniature example (reproduced in Section 3.1 of the paper) showing the full structure for a trivial two-file calculator project. The trajectory begins with a user task ("Create a simple calculator project..."), proceeds through the Main Agent's planning (Step 1: "The plan is: 1. Create operations.py. 2. Create main.py."), the Main Agent's Sub-Agent call for operations.py (Step 2), and then the Sub-Agent's implementation of main.py (Steps 8-13), which includes reading operations.py before writing main.py to check the add function's signature. This 14-step trajectory demonstrates the full simulation loop for a minimal project.

System Prompt and Agent Skeleton. The simulation is driven by detailed prompt templates (reproduced in full in Appendix D). The system prompt for the Main Agent defines the overall workflow and the expected JSON output format — a list of role-content pairs representing the agent's "memory" of the development process. The system prompt for the Sub-Agent provides the agent's identity ("you are 'code_generator', an expert software engineer"), workflow steps (Analyse, Identify dependencies, Plan, Write), tool definitions, and critical instructions about thought process diversity:

"The 'content' fields in the 'gpt' turns must contain highly intelligent, specific, and varied thought processes. STRICTLY AVOID using the same template for every file."

The prompt also includes instructions for context-driven reasoning (complex algorithms → focus on algorithmic efficiency; simple config files → focus on correctness) and dependency logic (explain specifically what the agent is looking for when reading a file, and react to the content found). These instructions are designed to produce trajectories with realistic variability rather than repetitive, template-based reasoning.

Why multi-agent rather than single-agent? The two-level hierarchy (Main Agent planning, Sub-Agents implementing) decomposes the complex task of repository construction into two cognitively distinct sub-tasks: architectural decomposition (planning what to build and in what order) and implementation (writing code for individual components). This mirrors how real software teams operate — architects plan, developers implement — and produces trajectories where the planning and implementation reasoning are separated, making both more interpretable and creating clearer causal links between high-level decisions (file creation order) and low-level implementation details (checking a dependency before writing code that uses it). A single-agent simulation would conflate these levels, likely producing reasoning that jumps between architectural and implementation concerns without the explicit handoff and delegation structure that characterises real development.

Why simulated tools rather than real execution? The authors considered two alternatives. Real execution (actually reading and writing files, executing code) would provide authentic observations but is "potentially expensive" in tool invocation costs and requires "substantial engineering efforts" for environment setup at scale — for 300k repositories, managing thousands of file systems would be infeasible. Unconstrained LLM generation (hallucinating tool responses without grounding) would be cheap but would produce unreliable trajectories — the Read tool might return fabricated file contents that don't match the actual repository, defeating the purpose of learning from real code. The simulated-tools-with-grounding approach (where LLM generates the tool calls, but responses are overwritten with real data — see next section) captures the efficiency of LLM simulation while maintaining fidelity to the actual code.

Trajectory Grounding Mechanism

The LLM-simulated trajectory is, by itself, vulnerable to hallucination: the Sub-Agent might invent a Read response that doesn't match the actual file, or the Write tool output might not exactly match the ground-truth code. The grounding mechanism corrects these errors by replacing LLM-generated content with extracted ground-truth data at specific injection points.

Read Tool Response Replacement. When a Sub-Agent emits a Read tool call, the LLM generates a simulated Observation containing what it "thinks" the file looks like. The grounding module overwrites this Observation with the actual content of the target file extracted from the repository. This is a direct, exact-string replacement. The paper states:

"The response to a Read tool call is replaced with the actual content of the file from the repository."

The consequence is that the Sub-Agent's subsequent reasoning — its Think step that reacts to the observation — is now conditioned on the correct file content, even if the LLM originally hallucinated something different. This creates a potential "mismatch": the Sub-Agent's Think step before the Read call was generated expecting some interface, but the actual Read response might reveal a different interface. The Sub-Agent then must adapt its plan (in the Think step after the observation) to match reality. This adaptation reasoning is precisely the kind of realistic engineering judgment the trajectories aim to capture.

Write Tool Response Replacement. Similarly, when a Sub-Agent emits a Write tool call, the grounding module replaces the LLM-generated code in the content argument with the ground-truth code of the file from the repository. The paper states:

"The final output of the Write tool call is replaced with the ground-truth code of the file."

This is the most critical grounding step because it guarantees that the terminal state of the trajectory matches the actual repository. Even if the Sub-Agent's reasoning (the Think steps) discusses a slightly different implementation, the Write output that appears in the trajectory is the exact, verified, real code. The Observation following the Write call (Successfully wrote N bytes to file.py.) is also replaced to reflect the actual bytes written.

What is NOT replaced. Critically, the Think steps — the CoT reasoning — are never replaced by the grounding module. The LLM's generated reasoning about why it chose a particular approach, what it inferred from reading a dependency, or how it planned to structure the implementation is left as-is. This means the grounding module creates a hybrid document: the reasoning (Think tokens) is synthetic and may contain imperfections, but the facts (Observations and Write outputs) are ground truth. The CoT Optimization step (next section) then refines the reasoning to better align with the ground-truth code, closing the gap that grounding does not address.

Information Provided to Guide Generation. In addition to post-hoc replacement, the system also provides extracted information in the prompt to guide the LLM's generation before replacement occurs. The file structure tree (provided to both Main Agent and Sub-Agent) helps the LLM generate a realistic implementation plan and Read calls that reference actual files. The intra-file AST information (function/class signatures) helps the Sub-Agent's reasoning target the correct interface. The inter-file dependency graph (used implicitly by providing related file source code as context for the Sub-Agent prompt — see Appendix D: $related_source_code) helps the Sub-Agent understand which files are relevant and what their interfaces look like. This prompt-level guidance reduces (but does not eliminate) the frequency and magnitude of hallucinations that need to be corrected by post-hoc replacement.

Why partial grounding is sufficient. The grounding is asymmetric — facts are verified, reasoning is not — because the paper's hypothesis is that even imperfect reasoning about a correct codebase provides better supervision than no reasoning at all. The model observing the trajectory learns: "when the agent thought about X, the code Y was (actually) the result." If the reasoning is slightly misaligned (the agent plans to use a loop but the ground-truth code uses a list comprehension), the model still learns the general pattern that reasoning precedes coding, that reading dependencies is important, and that plans guide implementation. The CoT Optimization further reduces the misalignment, producing reasoning that is not just plausible but predictive of the actual code.

CoT Optimization via Search

The initial trajectory from the multi-agent simulation contains reasoning (CoT steps) that is plausible but may not be optimal — it might not provide the most informative context for predicting the ground-truth code. The CoT Optimizer addresses this by iteratively refining the reasoning to better "explain" the code that follows.

Formal Objective. The optimisation goal is stated formally in Equation 1 of the paper:

$z^* = \arg\max_z \log p(x|z)$

where $z$ is the Chain-of-Thought reasoning (the sequence of Think steps in the trajectory) and $x$ is the ground-truth code (the actual Write tool output, which has been replaced with real repository code by the grounding module).

What this equation computes: the search space is the set of all possible reasoning sequences $z$ (all possible ways the agent could have thought about the implementation task). The objective function $\log p(x|z)$ is the log-probability assigned by a language model to the ground-truth code $x$ when conditioned on the reasoning $z$. The operation is an $\arg\max$: find the reasoning sequence that maximises this conditional log-probability. The output $z^*$ is the "optimal" reasoning — the thought process that makes the subsequent code most predictable, i.e., most causally consistent with the code.

Why this form: this objective operationalises the intuition that good reasoning minimises surprise about the outcome. If the reasoning genuinely captures the causal structure behind the code, then knowing the reasoning should make the code seem obvious — the language model should assign it high probability. Conversely, if the reasoning is irrelevant or misleading, the code will remain surprising (low probability) even after conditioning on the reasoning. The log-probability provides a scalar metric for this "goodness of explanation." This is a departure from typical CoT optimisation that might focus on coherence or plausibility; here, the metric is predictive power — does the reasoning help a model anticipate what code comes next?

Decomposition into Per-Step Refinement. Optimising the entire reasoning sequence jointly would be combinatorially intractable — each Think step could take thousands of possible values, and the number of sequences grows exponentially with length. Following the REER methodology (Wang et al., 2025b, cited in the paper), the optimisation is decomposed into iterative per-step refinement. The CoT is treated as a sequence of steps $z = (z_1, z_2, ..., z_n)$. Each step $z_i$ is optimised independently while holding all other steps fixed. The procedure for a single step $z_i$ is:

1. Sample alternatives: An LLM (the same Qwen3-30B-A3B model used for simulation) is prompted to generate $k$ alternative "refinements" for the thought step $z_i$. The prompt (reproduced in Appendix D) provides the reference source code (the ground-truth code $x$, for the LLM's understanding only — it is instructed not to mention it), the full reasoning context (the surrounding CoT steps), and the target block to rewrite. The prompt includes strict instructions: the refinement must use first-person singular, present tense, and must not announce itself as a refinement (no meta-commentary like "Here is a better reasoning"). The original step $z_i$ is enclosed in <replace> tags.

2. Evaluate candidates: For each candidate refinement $z'_i$, construct a full candidate reasoning sequence $z_{\text{cand}} = (z_1, ..., z_{i-1}, z'_i, z_{i+1}, ..., z_n)$ by substituting the candidate into the original sequence. Then measure the perplexity (PPL) of the ground-truth code $x$ under this candidate reasoning:

   $\text{PPL}(x|z_{\text{cand}}) = \exp\left(-\frac{1}{|x|}\sum_{t=1}^{|x|} \log p(x_t|x_{<t}, z_{\text{cand}})\right)$

   where $|x|$ is the number of tokens in the code, $x_t$ is the $t$-th token, $x_{<t}$ are the preceding tokens, and $p(x_t|x_{<t}, z_{\text{cand}})$ is the probability assigned by the (frozen, reference) language model to token $x_t$ given the reasoning context $z_{\text{cand}}$ and the previously generated code tokens. Perplexity is the exponential of the average negative log-likelihood per token — lower PPL means the reasoning makes the code more predictable.

3. Update rule: Let $z'^*_i$ be the candidate refinement that achieves the lowest perplexity among the $k$ candidates. If $\text{PPL}(x|(z_1, ..., z'^*_i, ..., z_n)) < \text{PPL}(x|(z_1, ..., z_i, ..., z_n))$ — that is, the best refinement strictly improves over the original — then $z_i$ is permanently replaced by $z'^*_i$ in the trajectory. Otherwise, the original step is retained. This is a greedy, hill-climbing update: only accept a change if it locally improves the code perplexity.

Why evaluate with perplexity rather than generating code? A natural alternative would be to have the LLM generate code from each candidate reasoning and compare the output to the ground truth. This would require a full generation for each candidate, which at scale (thousands of steps across 300k repositories) would be computationally prohibitive. Perplexity evaluation is feed-forward only — it requires a single forward pass through the model, computing token-level probabilities for the ground-truth code without autoregressive sampling. This is dramatically cheaper and can be batched efficiently. The implicit assumption is that lower perplexity on the ground-truth code corresponds to higher-quality reasoning — reasoning that makes the correct answer "obvious" to the model.

Why greedy rather than exhaustive search? The search space at each step is potentially vast (the LLM could generate arbitrarily different refinements). Even with $k=2$ candidates per step, searching over all combinations of refinements across $n$ steps would be $\mathcal{O}(k^n)$ — exponential and infeasible. The greedy per-step approach (optimise each step independently assuming others are fixed, then iterate) reduces the complexity to $\mathcal{O}(k \cdot n \cdot T)$ where $T$ is the number of refinement rounds. This is a standard coordinate-ascent optimisation: it converges to a local optimum of the perplexity surface but is not guaranteed to find the global optimum. The paper's experiments (Figure 3b) show that even this greedy procedure produces meaningful improvements: code perplexity decreases steadily across 10 iterations on a sample of 100 trajectories.

Configuration. The paper specifies two numbers for the optimisation procedure: $k = 2$ candidates are generated per step, and the search-and-replace process iterates for 3 rounds. This means each reasoning step can be refined at most 3 times, and at each refinement opportunity, 2 alternatives are explored. The total computational cost per step is therefore $k \cdot T = 6$ perplexity evaluations (plus the LLM calls to generate the candidate refinements). The paper reports that applying this procedure to all 300k repositories produces the Repo2Agent-Search variant of the dataset.

What the optimisation achieves. The case study in Appendix C provides a concrete illustration. For a task requiring database interaction in a Streamlit application, the Original CoT is a functional but generic checklist ("1. Set page title. 2. Load config. 3. Check login. 4. Initialize database..."). After the first round, refinements become technically specific ("5-minute cache TTL for @st.cache_data"). After the second round, reasoning includes detailed rationale ("to align with the user's requirement"), edge-case handling ("if either is missing, display a warning and prevent further execution"), and broader project awareness ("ensuring environment variable loading, caching, and user access control... follows best practices for Streamlit app development"). These refinements are not merely verbose — they provide the model with causal explanations that connect requirements to implementation choices, which is precisely the supervision signal that static code lacks.

Why RL is not used. The paper briefly notes that the objective $\arg\max_z \log p(x|z)$ could be optimised using reinforcement learning (with $\log p(x|z)$ as the reward), but the authors opt against it:

"RL training is often complex, expensive, and unstable."

The inference-time search strategy is simpler to implement, does not require training a separate reward model or policy, and produces trajectories that can be directly verified (via the perplexity metric) before inclusion in the training set. The tradeoff is that search-based optimisation only explores refinements the base LLM can generate; it cannot discover reasoning patterns outside the LLM's existing capabilities, whereas RL could theoretically shape the model's reasoning distribution more fundamentally.

Training Data and Pretraining Configuration

Once the trajectories are generated, grounded, and optimised, they must be converted into a format suitable for continuous pretraining.

Trajectory Flattening. The multi-agent simulation produces a hierarchical structure: the Main Agent's trajectory contains Sub-Agent calls, and each Sub-Agent call triggers a separate sub-trajectory. The flattening step converts this into a single linear text document by recursively inlining each Sub-Agent's trajectory into the point where it was called:

"When the Main Agent calls a Sub-Agent, we recursively inject that Sub-Agent's entire trajectory (thoughts, tool calls, and observations) directly into the call point."

The resulting document is a chronological sequence of <role>: <content> entries that mirrors the complete development lifecycle from initial plan to final file, structurally similar to the example in Table 1. The flattening preserves the temporal order: first the Main Agent plans, then it calls a Sub-Agent for the first file, then the Sub-Agent's reasoning and tool interactions for that file proceed, then the Main Agent resumes and calls the next Sub-Agent, and so on. Each full trajectory corresponds to one repository and produces a single long-context document (average length: ~12k tokens for Repo2Agent-Search, as shown in Figure 2b).

Targeted Loss Masking. During pretraining, the language model is trained with a next-token prediction objective — it predicts each token given all preceding tokens. For the flattened trajectory data, the loss is computed only on a subset of tokens:

"To ensure the model learns the causal link between reasoning and action rather than memorizing feedback, we mask the tokens corresponding to Observations (tool responses). The model is thus trained exclusively to predict Think and Action tokens."

This design choice is critical and non-obvious. The Observation tokens — the Read tool responses (actual file contents) and the Write tool responses (success messages like "Successfully wrote 89 bytes to main.py") — are not something the model should learn to generate. They are the environment's feedback, not the agent's actions. If the model were trained to predict Observation tokens, it would learn to produce file contents divorced from any reasoning context (just "hallucinate" file contents because they frequently appear in training), which is exactly the surface-level pattern matching that the paper argues is the flaw in raw-code pretraining. By masking the loss on Observation tokens, the model is forced to focus on the generative process: predicting what the agent should think and do next, not what the environment will return.

The model still sees the Observation tokens as context (they are part of the input sequence), but it is not penalised for failing to predict them. This means the Observations provide informative context for predicting subsequent Think and Action tokens (e.g., after reading a file, the agent must adapt its plan based on what it read), but the model does not waste capacity learning to reproduce file contents.

Data Curation Scale. The paper curates approximately 300,000 GitHub repositories by applying filters to remove repositories that are too short (insufficient content for meaningful trajectories) or too long (potentially exceeding the context window or containing non-code assets). Using the Qwen3-30B-A3B-Instruct-2507 model, these 300k repositories are converted into 4 billion tokens of synthetic agent trajectories.

Continuous Pretraining Configuration. The experiments use continual pretraining rather than supervised fine-tuning (SFT) or post-training. The authors explicitly justify this choice:

"This choice is motivated by the inherent nature of our synthetic data. The trajectories inevitably contain noise and biases stemming from the LLM's potential hallucinations and our agent workflow. Continuous pre-training, which typically involves larger and more diverse datasets than SFT, is inherently more robust to such imperfections."

This is an important technical decision. SFT is typically used for clean, high-quality instruction data where the model is expected to memorise the exact patterns. Pretraining, by contrast, operates with larger data volumes and processes the data in a way that is more robust to label noise — the model learns broad statistical regularities rather than memorising individual examples. The synthetic trajectories contain imperfect reasoning (LLM-generated plans may not reflect optimal strategies, some Read calls may be unnecessary, some Think steps may contain minor errors), and pretraining is better suited to handle this noise than SFT.

The base model is Llama-3-8B-Instruct (the instruction-tuned variant, not the base pretrained model). This choice is notable: the paper does not start from a raw pretrained checkpoint but from an already instruction-tuned model, then applies further pretraining on the trajectory data. This means the gains are additive to existing instruction-following capabilities, and the trajectories are being used to augment an already-capable model rather than to teach coding from scratch.

The training configuration follows Gao et al. (2025), a reference on effective long-context training:

• Context window: 64,000 tokens (64k). This is substantially larger than the 4k–8k context windows typical of standard pretraining and is necessary because the flattened trajectory for a single repository is often 12k tokens on average (Figure 2b), and some trajectories will be longer.
• Total training budget: 20 billion tokens of continual pretraining.
• Data mixture: 70% general-domain data and 30% repository-related data. This split is shared across all compared variants (Prolong baseline, Raw-Repos, Repo2Agent, Repo2Agent-Search) to ensure a fair comparison.
• Within the 30% repository slot, the composition is: 18% fixed (Prolong repositories — a set of existing repository data used for long-context pretraining) and 12% experimental data (the variable being tested: raw code from the curated 300k repos, unoptimised trajectories, or search-optimised trajectories).

The 12% fixed experimental slot is the key control: all model variants differ only in what fills this 12% of the total training budget. The remaining 88% (70% general + 18% Prolong repos) is identical. This means any observed performance differences can be attributed to the informational quality of the experimental data, not to differences in data volume, domain composition, or training hyperparameters.

Why this split ratio? The paper does not ablate the 12% fraction, but the choice likely reflects a practical constraint: the trajectory generation pipeline produces 4B tokens of data, and 12% of a 20B training budget is 2.4B tokens — well within the 4B generated. A larger fraction might require generating more data (costly) or repeating the same trajectories (reducing diversity). A smaller fraction might dilute the signal below detectable levels. The 12% fraction is sufficient to observe consistent improvements across benchmarks.

Model Variants Compared. The experiments compare four variants, differing only in the contents of the 12% experimental data slot:

• Prolong: The official external baseline, which uses whatever data occupies all 30% of the repository slot in the Prolong training recipe. This serves as a state-of-the-art reference point.
• Raw-Repos: The 12% experimental slot is filled with raw source code from the 300k curated repositories — the same repositories used for trajectory generation, but with no trajectory simulation, just the static final files concatenated as training text. This is the controlled ablation that isolates the effect of adding static code versus adding trajectories.
• Repo2Agent: The 12% slot is filled with unoptimised synthetic trajectories — the output of the multi-agent simulation with grounding, but without the CoT search optimisation.
• Repo2Agent-Search: The 12% slot is filled with search-optimised synthetic trajectories — the output after applying the CoT Optimizer for 3 rounds with $k=2$ candidates per step.

The primary comparison is Raw-Repos vs. Repo2Agent vs. Repo2Agent-Search, which directly measures the impact of converting raw code into agentic trajectories and the additional value of search-based reasoning optimisation.

Evaluation Protocol. The paper evaluates on a diverse set of benchmarks chosen to measure the capabilities that the reconstructed trajectories are hypothesised to improve. These are described more fully in the Experiments section, but for the technical approach, it is sufficient to note that the evaluations cover: long-context understanding (Ruler, Helmet), code generation (HumanEval, LongCodeBench), general reasoning (BBH, AGIEval, GSM-8k, MATH, MMLU-Pro), and agentic capability (APTBench). All evaluations use the pretrained model directly (greedy decoding or equivalent) without further fine-tuning, to measure the inherent capability instilled by the pretraining data.

4. Key Insights and Innovations

Innovation 1: Reframing Static Code as a Compressed Artifact That Must Be Decompressed for Effective Supervision

The paper's most fundamental conceptual move is not a new architecture or algorithm, but a diagnostic reframing of what makes pretraining data for code LLMs inadequate. Prior to this work, the dominant assumption in the field was that pretraining on large corpora of real code — the bigger and more diverse, the better — was the path to improving code generation capabilities. The papers building Code LLMs (DeepSeek-Coder, Qwen2.5-Coder, StarCoder, CodeLlama) all operated on variants of this assumption: more repositories, more files, longer context windows would naturally lead to better models.

This paper identifies a specific structural deficiency in that assumption that had not been articulated before at this level of clarity. The diagnosis is that static repositories are not just "code" — they are the terminal states of generative processes whose intermediate steps have been compressed away. The missing content is not random noise or trivia; it is precisely the causal structure that connects requirements to architecture, architecture to file-level plans, and file-level plans to individual lines of code. When a model is trained only on the terminal state, it learns to predict tokens given preceding tokens, but the preceding tokens contain no information about why this particular token follows — because the reasoning that produced that ordering was never recorded.

This reframing matters because it changes what "better data" means. The field had largely conceived of data improvement in terms of: (a) larger volume, (b) higher quality (filtering out low-quality repos, removing duplicates, selecting permissive licenses), (c) broader domain coverage, and (d) longer context. The paper argues that none of these address the fundamental problem: even the highest-quality, longest-context, broadest-domain repository dataset is still causally incomplete. You cannot learn the mapping from requirements to code if you never observe requirements paired with code in a structured way. You cannot learn dependency-aware implementation ordering if you only see files in alphabetical or filesystem order. You cannot learn the information-gathering step (reading a dependency before calling it) if you only see the call site with no trace of how the developer knew the interface.

The paper operationalises this diagnosis through the concept of generative supervision: training data that makes the generative process itself visible. This is a conceptual advance that applies beyond code — any domain where human-produced artifacts are the compressed output of a complex reasoning process (legal documents, mathematical proofs, architectural designs, scientific papers) might benefit from reconstructing the generative trajectory. The paper does not claim that it has solved this for all domains, but it has named the problem in a sufficiently precise way that future work can operationalise it elsewhere.

The evidence for this reframing is structural, not empirical: it comes from the internal logic of what static code omits. But the downstream experiments validate the reframing by showing that restoring even an approximate generative trajectory produces consistent gains across diverse benchmarks (Tables 2–4). If the diagnosis were wrong — if static code already contained sufficient generative signal — then replacing 12% of the training data with synthetic trajectories would not produce measurable improvements. The fact that it does, across coding, reasoning, long-context, and agentic benchmarks, supports the claim that the missing signal was genuinely absent from the raw code, not just redundantly represented in a different format.

Innovation 2: Grounding LLM-Simulated Trajectories via Terminal-State Verification as a Practical Middle Ground Between Real Environments and Unconstrained Generation

The paper advances a pragmatic solution to a known tradeoff in synthetic agent trajectory generation that has broader implications for how the field should think about simulation fidelity. Prior work had established two poles: real-environment execution (authentic but expensive and fragile; Fang et al., 2025; Pahuja et al., 2025) and unconstrained LLM simulation (cheap but hallucination-prone; Chen et al., 2025; Li et al., 2025; Sengupta et al., 2024). The dominant assumption was that these two poles represented an inescapable tradeoff — you could have cheap or authentic, but not both.

The paper's key insight is that for the specific purpose of generating pretraining data, full trajectory authenticity is overkill. What matters is not whether every intermediate step perfectly mirrors real developer behaviour (the reasoning might be suboptimal, some Read calls might be unnecessary), but whether the trajectory's terminal state is correct and its causal structure (plan → read → write → next file) is preserved. By guaranteeing the terminal state through grounding (Write outputs replaced with ground-truth code) and providing structural constraints through extracted repository metadata (file trees, dependency graphs, AST signatures), the paper achieves a form of bounded simulation: the trajectory can be "wrong" in its reasoning details but is guaranteed to end at the correct artifact.

This is a conceptual advance because it identifies which aspects of a trajectory must be authentic for effective learning and which can be noisy. The claim is implicit but testable: the terminal state must be authentic (otherwise the model learns incorrect code), and the causal ordering must respect real dependencies (otherwise the model learns to ignore dependency structure), but the reasoning content can be approximate because pretraining is robust to noise in the supervision signal. This insight, if validated across other domains, suggests that synthetic data generation for pretraining should prioritise endpoint verification and structural constraints over per-step authenticity, which dramatically reduces the cost and complexity of data generation.

The evidence supporting this innovation is the performance of Repo2Agent (which has grounded terminals but unoptimised reasoning) relative to Raw-Repos (Table 2: 84.00 vs. 83.20 on Ruler at 32k; Table 3: 36.59 vs. 34.76 on HumanEval). The gains from trajectory structure survive even without search-based CoT refinement, indicating that the grounding and causal structure alone provide value. This is reinforced by the fact that Repo2Agent-Search (which additionally refines reasoning) outperforms Repo2Agent, but the gap is often modest (e.g., 37.20 vs. 36.59 on HumanEval), suggesting that the structural grounding provides the bulk of the benefit and reasoning refinement provides incremental polish.

The approach also represents a methodological contribution to the synthetic data literature: the idea that real artifacts can serve as "verification keys" for simulated processes. Rather than asking an LLM to simulate a process from scratch and hoping it produces accurate tool interactions, the paper flips the paradigm: use the LLM to simulate the process, then overwrite its outputs with verified data at the points where correctness matters most. This partial-overwrite pattern — generating structure and reasoning with LLMs but inserting ground-truth values at verification points — could generalise to other domains where ground-truth artifacts exist but their generative processes do not.

Innovation 3: Predictive Power as an Intrinsic Quality Metric for Chain-of-Thought Reasoning

The paper introduces a specific, operationalisable criterion for evaluating the quality of Chain-of-Thought reasoning that departs from the standard approaches in the literature. In most prior work, reasoning quality is assessed either through human evaluation (coherence, helpfulness, correctness), through downstream task performance (does the model with this CoT solve the problem?), or through process-based reward models trained on human labels (Lightman et al., 2023). These approaches are either expensive (human evaluation), circular (downstream performance on the same task), or require additional supervised training (reward models).

The paper's alternative — formalised as maximising log p(x|z), where z is the reasoning and x is the ground-truth code — is a self-supervised quality metric that requires no human labels and no downstream task execution. The criterion is purely internal to the data generation process: good reasoning is reasoning that makes the target output predictable. If knowing the reasoning reduces the language model's uncertainty about the code (lower perplexity), the reasoning is capturing genuine causal structure. If the reasoning is irrelevant or misleading, it will not reduce perplexity — the model will still be "surprised" by the code even after seeing the reasoning.

This metric is conceptually distinct from prior CoT optimisation approaches in two ways. First, it is causal rather than associative: it measures whether the reasoning enables prediction of the outcome, not whether the reasoning is correlated with or semantically similar to the outcome. A reasoning trace that says "I will use a hashmap for O(1) lookup" and is followed by code that uses a hashmap is coherent, but if the reasoning is generic enough that it would be equally coherent with many different implementations (using a list, using a tree, using a database), it is not actually predictive. The perplexity metric penalises generic reasoning because generic reasoning doesn't reduce uncertainty about the specific code that follows. Second, it is locally optimisable: because the reasoning is decomposed into steps and each step can be independently evaluated (holding other steps fixed), the search procedure is tractable even for long trajectories — a key practical consideration that makes the approach scalable to 300k repositories.

The evidence for this innovation's effectiveness is Figure 3: as CoT optimisation iterations increase, the average CoT length grows (Figure 3a — reasoning becomes more elaborated) and the perplexity of the target code decreases (Figure 3b — reasoning becomes more predictive). The case study in Appendix C provides a qualitative demonstration: the original CoT is a generic checklist, the first optimisation round adds technical specificity, and the second round adds causal rationale linking implementation choices to requirements. This progression from generic → specific → causal is exactly what the perplexity metric should incentivise.

This innovation's broader significance is that it provides a principled answer to the question "what makes reasoning good?" that is independent of any downstream task. In the ongoing debate about whether CoT reasoning in LLMs is faithful (reflecting the model's actual computation) or merely plausible (sounding reasonable but not causally connected to the answer), this paper offers a concrete, measurable standard: good reasoning is reasoning that predictively constrains the output. This could become a standard evaluation criterion for synthetic reasoning data generation beyond the code domain.

Innovation 4: The Symbiotic Relationship Between Process Reconstruction and Model Capability as a Self-Reinforcing Loop

The paper's experimental design reveals a dynamic that is more subtle than "adding process data improves performance." The observed improvements are not uniform across capability categories; they show qualitative patterns that suggest the reconstructed trajectories are teaching the model something structurally different from what raw code teaches, rather than simply providing more tokens of the same kind.

Consider the long-context results on Ruler (Table 5 in Appendix A). At 64k context, Repo2Agent-Search achieves 61.80 average score versus 57.10 for the Prolong baseline and 61.00 for Raw-Repos. But the gains are not evenly distributed: in the NIAH-Multi task (retrieving multiple pieces of scattered information), Repo2Agent-Search achieves 80.40 versus 66.20 for Prolong — a 14-point gap. In RULER-CWE (common word extraction), the gap is smaller and even reverses at some context lengths. This pattern suggests the trajectories are not just improving "general long-context ability" but specifically improving the model's capacity for structured dependency retrieval — the ability to locate and integrate multiple scattered pieces of information that have a logical relationship to each other, which is exactly the skill exercised by the Read-then-Write pattern in the trajectories.

On APTBench (Table 4), the pattern is similarly non-uniform. Repo2Agent excels at planning-centric categories (Issue-Fix Plan: 40.74 vs. 37.04 for Raw-Repos), while Repo2Agent-Search excels at error diagnosis (Env-Setup Error: 24.49 vs. 22.45 for Raw-Repos). The unoptimised trajectories provide better supervision for holistic, high-level planning; the search-optimised trajectories, with their more rigorous step-by-step logic, provide better supervision for meticulous debugging. This is not a uniform "trajectories are better" result — it is evidence that different aspects of the reconstructed process teach different sub-skills, and the data generation pipeline can be tuned (in this case, via search optimisation) to emphasise different aspects.

The implication — which the paper does not fully develop but which is latent in the results — is that the relationship between process reconstruction and model capability may be self-reinforcing. A model trained on reconstructed trajectories becomes better at reasoning about software construction, which could make it a better simulator for generating higher-quality trajectories, which could produce better training data, and so on. The paper touches on this self-improvement dynamic briefly in Section 5.1 (noting that Repo2Agent-Search trajectories have more than double the Sub-Agent Think tokens of Repo2Agent, indicating that the search process produces richer reasoning that could further improve training), but the experimental design — using a fixed external LLM (Qwen3-30B-A3B) for simulation rather than the model being trained — doesn't close the loop.

This innovation is significant because it suggests that the paper's contribution is not just a one-time data augmentation technique but potentially a bootstrapping mechanism: as models improve through process-aware training, they become better process reconstructors, enabling more sophisticated trajectory generation, and so on. This aligns with the broader vision articulated in prior work on self-improvement (STaR, ReST^EM; Zelikman et al., 2022; Singh et al., 2024) but operationalises it through data synthesis rather than model fine-tuning — the improvement loop is in the data quality, not the model parameters.

The evidence for this pattern is primarily in the differential benchmark results (Tables 2–4, 5–6) and the trajectory composition analysis (Figure 2a: doubling of Sub-Agent Think tokens from Repo2Agent to Repo2Agent-Search), but the experimental setup does not directly test the bootstrapping hypothesis. This makes it more of a diagnostic insight (revealing a pattern that warrants further investigation) than a fully validated contribution, but it changes how one should think about the potential of process-aware pretraining — not as a static data improvement but as the first step in a compounding cycle.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experimental data is generated from approximately 300,000 curated GitHub repositories (filtered to remove repositories that are too short or too long), processed through the multi-agent simulation pipeline described in Section 3. The simulation uses Qwen3-30B-A3B-Instruct-2507 (a 30B-parameter Mixture-of-Experts model with 3B active parameters) to produce 4 billion tokens of synthetic agent trajectories. This generated data fills a 12% slot within a larger 20B-token continual pretraining mixture, where the remaining 88% consists of 70% general-domain data and 18% Prolong repositories (a fixed set of repository data used for long-context pretraining).

• Base model. All experiments use Llama-3-8B-Instruct (the instruction-tuned variant of Meta's Llama 3 with 8 billion parameters; Dubey et al., 2024). The choice to start from an instruction-tuned rather than base pretrained model means the reported gains are additive to existing instruction-following capabilities. The model undergoes continuous pretraining (not supervised fine-tuning) for 20 billion tokens with a 64,000-token context window, following the training configuration of Gao et al. (2025). The authors justify continual pretraining rather than SFT by noting that the synthetic trajectories "inevitably contain noise and biases stemming from the LLM's potential hallucinations and our agent workflow," and pretraining "is inherently more robust to such imperfections" (Section 3.3).

• Metrics. All benchmarks report accuracy as the primary metric, computed as the fraction of test instances where the model's output matches the ground-truth answer (exact match for coding tasks, answer extraction for reasoning tasks). For APTBench specifically, the metric is a capability score on a per-subtask basis, aggregated into category averages. The CoT optimisation procedure during data generation uses perplexity (PPL) as an intrinsic quality metric: $\text{PPL}(x|z_{\text{cand}}) = \exp\left(-\frac{1}{|x|}\sum_{t=1}^{|x|} \log p(x_t|x_{<t}, z_{\text{cand}})\right)$, where lower perplexity indicates the reasoning $z$ better predicts the ground-truth code $x$. For the long-context benchmarks Ruler and Helmet, scores are reported as average accuracy across all subtasks at each context length (16k, 32k, 64k tokens).

• Baselines. The paper compares four model variants, differing exclusively in the contents of the 12% experimental data slot during pretraining. The Prolong baseline is the official externally-trained model that uses whatever data occupies the full 30% repository slot in the Prolong long-context training recipe (Gao et al., 2025) — it serves as a state-of-the-art reference point but is not a controlled ablation since its exact data composition may differ from the internal variants. The Raw-Repos baseline fills the 12% slot with raw source code from the same 300k curated repositories used for trajectory generation, presented as static files without any agentic simulation — this is the primary controlled ablation that isolates the effect of trajectory structure versus simply adding more code data. The Repo2Agent variant fills the 12% slot with unoptimised synthetic trajectories from the multi-agent simulation with grounding but without CoT search optimisation. The Repo2Agent-Search variant fills the 12% slot with search-optimised trajectories after applying 3 rounds of CoT refinement with $k=2$ candidates per step.

• Generation budget / compute accounting. All model variants are trained on exactly 20 billion tokens total with an identical 70/30 general-to-repository data split, and within the 30% repository slot, the 18% Prolong component is held constant. The 12% experimental data slot is the only variable — all variants process the same total token count, ensuring that performance differences reflect data quality (informational density of trajectories versus raw code) rather than training data volume. The trajectory generation itself uses a fixed inference budget: 2 candidate refinements per CoT step, 3 refinement rounds, producing 4B total trajectory tokens. For evaluation, generation is typically greedy decoding (deterministic, single-sample) unless otherwise specified, so test-time compute is not a variable.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. Results are presented as single accuracy scores on standard benchmark test sets. The experimental design uses a controlled ablation structure (Raw-Repos vs. Repo2Agent vs. Repo2Agent-Search all sharing the same base model, training configuration, and 88% of training data) to isolate causal effects, but no confidence intervals, standard errors, or significance tests are reported. The test sets for the benchmarks are standard, publicly available splits (e.g., HumanEval's 164 problems, MATH's 5,000 test problems, Ruler's synthetic tasks) and are not further partitioned for cross-validation.

Main Quantitative Results

Long-Context Understanding

The headline result is that training on structured agent trajectories yields consistent improvements over raw code on both Ruler and Helmet benchmarks, with the search-optimised variant (Repo2Agent-Search) frequently matching or exceeding the strong Prolong baseline, particularly at extreme context lengths and on tasks requiring structured information retrieval.

Ruler benchmark (Table 2, main text; Table 5, Appendix A). At 16k context length, Repo2Agent (87.50) and Repo2Agent-Search (87.10) both outperform Raw-Repos (86.90) and the Prolong baseline (83.61). The advantage persists at 32k: Repo2Agent-Search achieves 84.40 versus 83.20 for Raw-Repos and 81.77 for Prolong. At the maximum 64k context length — where all models degrade substantially — Repo2Agent-Search achieves the highest robustness with 61.80, compared to 61.00 for Raw-Repos, 58.10 for Repo2Agent, and 57.10 for Prolong. The unoptimised Repo2Agent variant underperforms Raw-Repos at 64k (58.10 vs. 61.00), suggesting that unrefined reasoning may introduce noise that hurts performance at extreme lengths, while search optimisation recovers and surpasses the raw code baseline.

The detailed subtask breakdown in Table 5 reveals that the gains are not uniform across task types. In the NIAH-Multi task (retrieving multiple pieces of scattered information — the subtask most analogous to locating dependencies across a codebase), Repo2Agent-Search achieves 80.40 at 64k compared to 76.30 for Raw-Repos and 66.20 for Prolong — a 14.2-point advantage over the external baseline. In RULER-CWE (common word extraction) at 32k, Repo2Agent-Search scores 42.30 versus 34.60 for Raw-Repos — a 7.7-point gap. However, in RULER-VT (variable tracking) at 64k, Raw-Repos (14.40) underperforms Prolong (20.50), and Repo2Agent-Search (16.60) does not close this gap. This task-level heterogeneity suggests the trajectories improve specific retrieval patterns (multi-key lookup, dependency-style tracking) rather than uniformly boosting all long-context abilities.

Helmet benchmark (Table 2, main text; Table 6, Appendix A). The pattern is similar but shows Repo2Agent-Search in a stronger position. At 16k, Repo2Agent-Search achieves 61.99 versus 60.41 for Raw-Repos and 60.17 for Prolong. At 32k, Repo2Agent-Search reaches 62.65 — the highest score across all variants at any context length — compared to 60.98 for Raw-Repos and 61.57 for Prolong. At 64k, the gap narrows: Repo2Agent-Search (57.84) still leads Raw-Repos (57.13) but trails the Prolong baseline (58.10) by a small margin.

The subtask analysis in Table 6 shows that Repo2Agent-Search excels in specific categories that mirror the "Recall-Plan-Act" structure of the synthetic trajectories. In the Recall category at 32k, Repo2Agent-Search achieves 99.81 versus 98.94 for Raw-Repos and 99.38 for Prolong. In In-Context Learning (ICL) at 32k, Repo2Agent-Search scores 76.32 versus 71.84 for Raw-Repos and 75.84 for Prolong. These are tasks that require the model to extract and apply patterns from long contexts — precisely the skill exercised by the Sub-Agents' Read-then-Write pattern in the trajectories. The Rerank category (which requires identifying relevant information within a large context) also shows consistent improvement: Repo2Agent-Search scores 33.69 versus 32.19 for Raw-Repos at 32k. However, in LongQA, the Prolong baseline maintains a slight edge at 64k (46.78 vs. 45.48 for Repo2Agent-Search), which drives Prolong's marginal overall advantage at that context length.

A key observation across both benchmarks is that Repo2Agent-Search consistently outperforms Repo2Agent (the unoptimised variant) on Helmet at 16k and 32k, and on Ruler at 64k — the search-based CoT refinement is adding value, not just the trajectory structure itself. However, the gap between Repo2Agent and Repo2Agent-Search is often smaller than the gap between either and Raw-Repos, suggesting the structural signal (the planning→reading→writing causal order) is the primary driver, with refinement providing incremental improvement.

Coding and Reasoning

The headline result is that agentic trajectory pretraining improves code generation across both short-form (HumanEval) and long-context (LongCodeBench) tasks, while general reasoning shows modest but consistent positive transfer despite the trajectories containing no explicit math or logic training data.

Coding benchmarks (Table 3). On HumanEval (the standard 164-problem Python function completion benchmark), Repo2Agent-Search achieves 37.20, outperforming Repo2Agent (36.59), Raw-Repos (34.76), and Prolong (16.46) by substantial margins. The gap between Raw-Repos and Repo2Agent-Search is 2.44 percentage points — a 7% relative improvement from replacing static code with search-optimised trajectories in just 12% of the training data. On LongCodeBench-32k (which evaluates code generation with 32k-token contexts), Repo2Agent-Search achieves 36.46 versus 34.51 for Repo2Agent, 34.16 for Raw-Repos, and 29.38 for Prolong — the trajectory variants all outperform the baselines, with search optimisation providing an additional ~2-point boost. However, on LongCodeBench-64k (64k-token contexts), the pattern is less consistent: Repo2Agent leads with 31.05, while Repo2Agent-Search (30.26) slightly underperforms and Raw-Repos (27.37) trails. The Prolong baseline (30.52) is competitive at this context length, though still behind Repo2Agent. The reversal at 64k (where unoptimised trajectories outperform optimised ones) may reflect noise — the LongCodeBench-64k numbers show relatively small absolute differences, and without confidence intervals it is unclear whether the ranking is statistically reliable.

Reasoning benchmarks (Table 3). Despite the trajectories containing no explicit math or formal logic content (they are purely software engineering narratives), the trajectory-trained models show positive transfer to general reasoning. On MATH (the challenging competition mathematics benchmark), Repo2Agent-Search achieves 3.76 versus 3.72 for Repo2Agent, 2.18 for Raw-Repos, and 1.64 for Prolong. The absolute scores are low across all variants — this is a known limitation of Llama-3-8B on complex mathematics — but the trajectory variants nearly double the Raw-Repos baseline and more than double the Prolong baseline. On GSM-8k (grade-school math word problems), Raw-Repos and Repo2Agent both achieve 61.94, while Repo2Agent-Search scores 60.96 and Prolong scores 59.67 — here the trajectories provide no advantage over raw code, and all three internal variants cluster tightly. On BBH (BigBench Hard, a suite of challenging reasoning tasks), Repo2Agent-Search achieves 67.03 versus 66.00 for Repo2Agent, 66.27 for Raw-Repos, and 66.69 for Prolong — all variants are within ~1 point, suggesting that neither trajectories nor raw code materially affect this capability. On AGI-Eval, the pattern is similarly tight: Repo2Agent-Search (36.85) and Prolong (36.91) are near-identical, with Repo2Agent (36.32) and Raw-Repos (35.78) trailing slightly.

The reasoning results reveal an asymmetric transfer pattern: the trajectory data provides meaningful benefits on MATH (the most challenging and multi-step reasoning task in the set) but essentially no benefit on GSM-8k, BBH, or AGI-Eval. This is consistent with the hypothesis that the trajectories teach structured multi-step reasoning (plan→read→write→next-file) which transfers to complex mathematical deduction but does not help with simpler or broader reasoning tasks that the base model already handles reasonably well.

Software-Engineering (Agentic) Capability

The headline result is that pretraining on synthetic trajectories improves fundamental agentic capabilities as measured by APTBench, with Repo2Agent (unoptimised trajectories) achieving the highest overall score, and the choice of optimisation (search vs. none) offering a tunable tradeoff between planning and debugging skills.

APTBench overall results (Table 4). APTBench evaluates models on atomic agentic skills derived from SWE-Bench and Deep-Research tasks without post-training, measuring inherent potential instilled during pretraining. Across all categories and subtasks, Repo2Agent achieves the highest overall average of 30.10, compared to 29.65 for Repo2Agent-Search and 29.02 for Raw-Repos. The Prolong baseline is not included in this table (the paper only compares internal variants), so the absolute ceiling is unknown.

The category-level breakdown reveals a divergence between Repo2Agent and Repo2Agent-Search that is the most nuanced finding in the paper. In the DeepResearch category (which tests open-ended research planning, citation, and quality assessment), Repo2Agent leads with 30.49 versus 30.02 for Repo2Agent-Search and 29.21 for Raw-Repos. The unoptimised trajectories provide better supervision for holistic planning tasks — the natural, unrefined CoT appears to teach a more generalisable approach to open-ended problem decomposition. Within DeepResearch, the Openend-Quality subtask shows the largest gap: Repo2Agent-Search scores 26.20 versus 24.74 for Repo2Agent and 21.99 for Raw-Repos — here search optimisation does help.

In the Issue-Fix category (which tests bug localization, patch generation, and test-patch verification), Repo2Agent again leads with 34.84 versus 33.80 for Repo2Agent-Search and 33.72 for Raw-Repos. The Plan subtask within Issue-Fix shows the strongest Repo2Agent advantage: 40.74 versus 38.68 for Repo2Agent-Search and 37.04 for Raw-Repos. However, the Fix-Patch subtask reverses: Repo2Agent (28.02) outperforms Repo2Agent-Search (25.43), while Locate is nearly identical across variants (~24.0). This suggests unoptimised CoT is better for high-level planning and strategy, while search-optimised CoT may be detrimental when the task requires reacting to concrete code (fixing a specific bug) rather than abstractly reasoning about structure.

In the Env-Setup category (which tests environment configuration, error diagnosis, and action planning), Repo2Agent-Search leads with 21.61 versus 21.01 for Repo2Agent and 20.61 for Raw-Repos. The Error subtask shows the clearest advantage for search optimisation: 24.49 for Repo2Agent-Search versus 23.13 for Repo2Agent and 22.45 for Raw-Repos. The Plan subtask shows Repo2Agent-Search at 19.22 versus 17.85 for Repo2Agent — search refinement helps here. This category represents meticulous, low-level debugging logic, and the search-optimised trajectories, with their more rigorous step-by-step reasoning, provide better supervision for this skill.

The key insight from APTBench is that trajectory optimisation is not uniformly beneficial — it creates a tradeoff. Unoptimised trajectories (Repo2Agent) provide better supervision for broad, holistic planning capabilities. Search-optimised trajectories (Repo2Agent-Search) provide better supervision for meticulous, detail-oriented debugging and error diagnosis. This aligns with the case study in Appendix C: search optimisation elaborates the reasoning with specific technical details (cache TTL values, exact variable names, edge cases), which helps with precision tasks but may narrow the reasoning's applicability to abstract planning scenarios. The overall score difference (30.10 vs. 29.65) is small, but the category-level divergence suggests that for a production system, one might want to include both trajectory types in the training mixture to capture complementary capabilities.

Ablation Studies and Robustness Checks

Trajectory composition analysis (Figure 2a, Section 5.1): The conversion of raw code to agentic trajectories fundamentally restructures the token distribution. In Repo2Agent trajectories, Sub-Agent-Call-Think tokens (the reasoning about how to implement a file) comprise approximately 900 tokens per trajectory on average. After search-based CoT optimisation, this more than doubles to 2,300 tokens in Repo2Agent-Search. This demonstrates that the search process does not merely rephrase existing reasoning — it substantially elaborates the logical steps, producing richer, more detailed thought processes. Tool-Call and Tool-Response tokens (the Read and Write actions and their observations) remain the dominant component, reflecting the action-heavy nature of software development. Main-Agent-Think tokens (architectural planning) are a much smaller fraction, consistent with the hierarchical design where most reasoning occurs at the Sub-Agent level.

Trajectory length expansion (Figure 2b, Section 5.1): Raw repositories average approximately 4,865.5 tokens when presented as flattened code files. Conversion to unoptimised trajectories (Repo2Agent) increases the average to approximately 9,500 tokens. Search optimisation (Repo2Agent-Search) further expands this to 12,083.4 tokens on average — roughly 2.5× the size of the raw code. Despite this per-repository expansion, all model variants are trained on the same total token budget (20B tokens, with 12% allocated to experimental data). This means the trajectory-trained models see fewer distinct repositories than the raw-code model (since each trajectory consumes more tokens per repository), yet they achieve better performance — direct evidence that the trajectories provide higher informational density per token than static code.

CoT optimisation iterations vs. reasoning quality (Figure 3, Section 5.2): On a sample of 100 trajectories tracked over 10 optimisation rounds, CoT length increases monotonically with iterations (Figure 3a), and code perplexity decreases steadily (Figure 3b). The inverse relationship between CoT length and code perplexity supports the central hypothesis: more detailed, elaborated reasoning provides a more predictive context for code generation. The decreasing perplexity without plateauing after 3 rounds (the configuration used for the main experiments) suggests that additional optimisation rounds might yield further improvements — the 3-round configuration may be conservative, and the reported Repo2Agent-Search results could be a lower bound on what longer optimisation would achieve. However, the paper does not report whether the trajectory models trained with >3 optimisation rounds would further improve downstream performance, leaving this as an open question.

Optimised vs. unoptimised trajectories on long-context tasks (Tables 2, 5, 6): The comparison between Repo2Agent and Repo2Agent-Search isolates the effect of CoT refinement independent of trajectory structure. This is not a standard ablation (both variants use the multi-agent simulation and grounding) but rather an evaluation of whether refining the reasoning matters. On Ruler at 64k, the difference is striking: Repo2Agent-Search (61.80) substantially outperforms Repo2Agent (58.10) and the Raw-Repos baseline (61.00). The unoptimised trajectories actually underperform raw code at this extreme context length, while search optimisation recovers and surpasses it. On Helmet at 32k, the gap is smaller but consistent: Repo2Agent-Search (62.65) vs. Repo2Agent (62.03). At 16k, the ranking reverses on Ruler: Repo2Agent (87.50) edges out Repo2Agent-Search (87.10). These results suggest that CoT refinement is particularly important at longer context lengths, where noisy or imprecise reasoning compounds over the trajectory and degrades the model's ability to track long-range dependencies.

Raw code vs. trajectory comparisons at fixed compute (Tables 2–4): The Raw-Repos baseline is the critical controlled ablation: same repositories, same 12% data slot, same total token budget, same model, same training recipe — differing only in whether the 300k repositories are presented as static code files or as reconstructed agentic trajectories. Across benchmarks:

• Ruler at 64k: Raw-Repos (61.00) vs. Repo2Agent-Search (61.80) — +0.80 points.
• Helmet at 32k: Raw-Repos (60.98) vs. Repo2Agent-Search (62.65) — +1.67 points.
• HumanEval: Raw-Repos (34.76) vs. Repo2Agent-Search (37.20) — +2.44 points.
• LongCodeBench-32k: Raw-Repos (34.16) vs. Repo2Agent-Search (36.46) — +2.30 points.
• APTBench overall: Raw-Repos (29.02) vs. Repo2Agent (30.10) — +1.08 points.

The improvements are consistent in direction but modest in magnitude (1–2.5 points across most benchmarks). The fact that replacing just 12% of training data with trajectories produces measurable gains across diverse tasks is evidence that the trajectory format provides genuinely complementary information, not just a different encoding of the same signal. However, the modest magnitude also suggests that trajectories are a supplement to, not a replacement for, raw code pretraining — the bulk of the model's capability still derives from the 88% of training data that is shared across variants.

Prolong baseline as external reference (Tables 2, 3, 5, 6): The Prolong baseline is not a controlled ablation (it uses a different training recipe with potentially different data sources), but it serves as a state-of-the-art long-context pretraining method. The fact that Repo2Agent-Search matches or exceeds Prolong on many benchmarks (Ruler at 64k: 61.80 vs. 57.10; Helmet at 32k: 62.65 vs. 61.57; HumanEval: 37.20 vs. 16.46) suggests that the trajectory approach is competitive with strong existing methods. The HumanEval gap (20+ points) is particularly notable, but this may partly reflect that Prolong was not specifically optimised for code generation, whereas the trajectory data is code-focused by design.

Negative result: Repo2Agent underperforms Raw-Repos at extreme context on Ruler (Table 2, Table 5): At 64k on Ruler, the unoptimised Repo2Agent variant achieves 58.10, which is below both Raw-Repos (61.00) and Repo2Agent-Search (61.80). This is a negative result that demonstrates unrefined trajectory reasoning can introduce noise that is more harmful than helpful at extreme context lengths. The search-based CoT optimisation is not merely a nice-to-have polish — it is necessary to prevent the trajectories from degrading long-context performance. This finding is not highlighted by the authors but is visible in the data.

Missing ablation: the effect of trajectory length vs. quality. A critical confound in the comparison between Raw-Repos and Repo2Agent-Search is that trajectories are approximately 2.5× longer per repository (Figure 2b). Since the experimental data slot is fixed at 12% of total tokens, the trajectory-trained models see fewer distinct repositories than the raw-code model. If the gains were purely due to trajectory length (more tokens = more training signal), this would weaken the paper's claim that trajectory structure is the causal factor. The paper addresses this by noting that the token budget is fixed, framing the gain as improved informational density, but a direct ablation — raw code padded to the same length as trajectories with some neutral filler (e.g., repeated or synthetic content), or trajectories truncated to match raw code length — would strengthen this claim. The paper does not run this control.

Missing ablation: grounding without trajectory structure. The grounding mechanism (replacing Read/Write outputs with real repository data) is evaluated only as part of the full trajectory pipeline. An informative ablation would be: raw code interleaved with grounded tool interactions but without the multi-agent hierarchical structure — essentially, a "flat" version of the trajectory that preserves the factual correctness of observations but removes the main-agent/sub-agent planning structure. This would isolate whether the planning hierarchy (Main Agent decomposing the project, Sub-Agents implementing files) contributes value beyond simply having Read/Write calls with verified responses. The paper does not include this ablation, making it unclear whether the multi-agent structure specifically, or simply having tool interactions with real code content, drives the observed gains.

Missing ablation: the 12% fraction sensitivity. All experiments use a fixed 12% experimental data slot. The paper does not ablate this fraction to determine whether a 6% slot would produce proportionally smaller gains, whether a 24% slot would saturate or continue improving, or whether the optimal fraction varies by benchmark. The 12% value appears to be determined by the amount of trajectory data generated (4B tokens) relative to the total budget (20B tokens × 12% = 2.4B consumed, well within the 4B generated). Without fraction sensitivity analysis, the practical guidance for how much trajectory data to include in a training mixture remains unclear.

Missing ablation: Qwen3-30B-A3B vs. other simulation engines. All trajectories are generated using a single model (Qwen3-30B-A3B-Instruct-2507). The paper does not test whether trajectories generated by a different LLM (e.g., Llama-3-70B, GPT-4) would produce different downstream results, or whether using Llama-3-8B itself (the model being trained) as the simulation engine would create a beneficial self-improvement loop. The choice of Qwen3-30B-A3B — a Mixture-of-Experts model — raises the possibility that its specific reasoning patterns (which may differ from Llama-family models) could affect the nature of the generated trajectories. This is not tested.

Critical Assessment

Does the paper demonstrate that "reverse-engineering latent agentic trajectories provides richer supervision than raw code alone"? The evidence is largely supportive but with important caveats about effect magnitude and benchmark specificity. The trajectory variants (Repo2Agent and Repo2Agent-Search) outperform the Raw-Repos baseline on most benchmarks (Tables 2–4), establishing that the trajectory format provides information beyond what is present in static code. However, the typical gain is 1–3 percentage points — meaningful but not transformative — and the improvement is not universal. On GSM-8k (Table 3), trajectories provide zero benefit over raw code. On BBH and AGI-Eval (Table 3), all variants are within ~1 point. On Ruler at 64k (Table 2), unoptimised trajectories actually underperform raw code. The claim holds on average but with domain-specific variability that the paper does not fully characterise. A fair summary is that trajectories provide modest but consistent improvements on tasks requiring structured, multi-step reasoning about code (coding benchmarks, long-context retrieval, debugging), but do not improve and may slightly degrade performance on tasks requiring broad general reasoning that is not specifically code-structured.

Does the paper demonstrate that "search-based CoT optimisation improves trajectory quality"? Yes, with both intrinsic and extrinsic evidence. Intrinsically, Figure 3b shows that code perplexity decreases steadily with optimisation rounds, and the case study in Appendix C shows qualitative improvements in reasoning specificity and causal grounding. Extrinsically, Repo2Agent-Search outperforms Repo2Agent on most benchmarks (Tables 2–4), with the most notable gains at extreme context lengths on Ruler (61.80 vs. 58.10 at 64k, Table 2) and on coding tasks (37.20 vs. 36.59 on HumanEval, Table 3). However, APTBench results (Table 4) complicate the picture: Repo2Agent achieves the highest overall average (30.10 vs. 29.65), and search optimisation creates a tradeoff between planning (where unoptimised CoT excels) and error diagnosis (where optimised CoT excels). The claim that optimisation improves quality is correct in aggregate but masks category-specific patterns where unoptimised trajectories are preferable — a nuance the paper could acknowledge more explicitly.

Does the paper demonstrate that "learning from reconstructed processes enhances coding, reasoning, and agentic capabilities"? Partially. The evidence for coding is strong: HumanEval (+2.44 over Raw-Repos, +20.74 over Prolong) and LongCodeBench-32k (+2.30 over Raw-Repos) show consistent, non-trivial improvements. The evidence for agentic capabilities is moderate: APTBench overall shows a +1.08 advantage (Repo2Agent over Raw-Repos, Table 4), but the Prolong baseline is not included for comparison, and the category-level analysis reveals tradeoffs rather than uniform improvement. The evidence for reasoning is weak and inconsistent: MATH shows a notable improvement (+1.58 over Raw-Repos for Repo2Agent-Search, Table 3), but absolute scores are very low (3.76%), GSM-8k shows no improvement, and BBH/AGI-Eval are flat. The paper claims "general reasoning" transfer, but the data suggests transfer is limited to complex multi-step reasoning (MATH) and does not generalise to broader reasoning categories. The paper's claim about "reasoning" would be more accurately scoped as "modest positive transfer to complex mathematical reasoning, with no benefit on simpler or broader reasoning tasks."

Is the 12% data slot design genuinely isolating the effect of trajectory structure? There is a confound that the paper acknowledges but does not fully resolve: trajectories are 2.5× longer per repository than raw code (Figure 2b). Since the experimental data slot is fixed at 12% of tokens, the trajectory models see fewer distinct repositories but more tokens per repository compared to the raw-code model. If the benefit derives partly from seeing each repository's content more intensively (multiple perspectives on the same code via planning, reading, writing) rather than from the specific trajectory format, the comparison is not purely about data format — it is partially about repetition versus diversity within the fixed token budget. The paper's framing (improved "informational density") is consistent with this interpretation, but a direct control — raw code repeated or reformatted to match trajectory length — would distinguish format effects from repetition effects. Without this control, the causal attribution to "trajectory structure" is plausible but not definitively established.

How robust are the findings to model scale and architecture? All experiments use a single model (Llama-3-8B-Instruct) at a single scale (8B parameters). The paper does not test whether trajectories would benefit larger models (e.g., Llama-3-70B) or models from different families (e.g., Qwen, Mistral, DeepSeek). It is possible that larger models, which already have stronger reasoning capabilities, would benefit less from synthetic trajectories because they already implicitly reconstruct some of the reasoning from static code — or conversely, that they would benefit more because they have greater capacity to absorb the richer signal. The generalisability of the findings across model scales and architectures is entirely untested.

How robust are the findings to the trajectory generation model? All trajectories are generated by Qwen3-30B-A3B-Instruct-2507. The quality and style of the generated reasoning — and therefore the effectiveness of the resulting pretraining data — may depend on the specific capabilities of this model. If Qwen3 produces reasoning that is unusually well-structured or well-aligned with Llama-3's processing style, the gains might not replicate with a different simulation engine. Conversely, if Qwen3's reasoning has systematic biases or blind spots, those biases may be transferred to the trained model. The paper does not discuss this dependency or test trajectories generated by alternative models.

Are the evaluation benchmarks measuring the capabilities that trajectories are hypothesised to improve? The benchmark selection is reasonable but has gaps. The long-context benchmarks (Ruler, Helmet) test retrieval and reasoning over long documents — relevant because trajectories are long sequential narratives. The coding benchmarks (HumanEval, LongCodeBench) test code generation — the primary domain. APTBench tests agentic skills — directly relevant to the trajectory format. However, the paper does not evaluate on benchmarks that specifically test the skills that trajectories are uniquely designed to teach: dependency-aware implementation ordering (given a requirement, can the model plan the correct file creation sequence?), Read-before-Write reasoning (does the model check interfaces before using them?), or repository-level code generation (can the model generate multi-file projects with correct cross-file imports?). Standard benchmarks like HumanEval test single-function completion, which captures only a fraction of what the trajectories teach. A custom benchmark — e.g., presenting a partial repository and asking the model to implement a new file that correctly depends on existing ones — would provide more direct evidence for the paper's causal claims about what trajectories teach.

What is the practical significance of the observed gains? The improvements are consistent but modest: 1–3 points on most benchmarks, with a few larger gains on specific subtasks (e.g., NIAH-Multi at 64k: +4.2 points over Raw-Repos). For a practitioner deciding whether to invest in trajectory generation for their own pretraining pipeline, the key question is whether these gains justify the substantial computational cost of the generation process: running a 30B MoE model to simulate 300k trajectories, plus search-based CoT optimisation (6 perplexity evaluations per reasoning step, plus candidate generation), plus the infrastructure for repository analysis and grounding. The paper does not report the total compute cost of trajectory generation relative to the pretraining compute, making cost-benefit analysis impossible from the reported data. The fact that only 12% of training data yields measurable gains is promising for scalability, but the absolute cost of generating that 12% may exceed the cost of simply training on more raw code to achieve equivalent improvements. Without cost reporting, the practical recommendation — "substitute 12% of your code pretraining data with reconstructed trajectories" — is supported for effectiveness but unquantified for efficiency.

What would strengthen the paper's claims? Several experiments would address the gaps identified above:

1. Scaling the trajectory data fraction (6%, 12%, 24%, 48%) to determine the dose-response curve and whether gains saturate or compound.
2. A trajectory length control — raw code padded or repeated to match trajectory length — to deconfound format from repetition.
3. A custom benchmark evaluating dependency-aware multi-file code generation, the skill the trajectories are uniquely designed to teach.
4. Cross-model-family validation — testing whether trajectories generated by Qwen3 benefit Llama-3 specifically (due to reasoning style similarity) or generalise to other model families.
5. Compute cost reporting — the total FLOPs for trajectory generation vs. pretraining, to enable cost-benefit comparisons.
6. Confidence intervals or variance estimates for benchmark scores, to assess whether reported differences (often 1–2 points) are statistically reliable given finite test sets.
7. A direct comparison of trajectory pretraining against simply scaling up raw-code pretraining by the same token budget — if adding 2.4B tokens of raw code to the Raw-Repos baseline matches the Repo2Agent-Search gains, the trajectory approach is not actually more data-efficient; it is just providing more effective tokens within the same slot size. The paper does not run this comparison.

The paper's contributions are real — the trajectory format demonstrably improves pretraining data quality for code-focused tasks, and the search-based CoT optimisation provides a principled mechanism for refining synthetic reasoning — but the evidence supports a more modest and conditional set of claims than the paper's framing suggests. The approach works, but primarily for code and code-adjacent reasoning, with modest magnitude, at unknown computational cost, and with tradeoffs that depend on the specific downstream task and whether reasoning is optimised or left unrefined.

6. Limitations and Trade-offs

6.1 The Trajectory Generation Cost Is Unquantified and Potentially Dominant

The constraint. The entire compute-optimal pretraining data paradigm rests on generating synthetic trajectories at scale — 300,000 repositories processed through a 30B-parameter Mixture-of-Experts model (Qwen3-30B-A3B-Instruct-2507), plus 3 rounds of search-based CoT optimisation with 2 candidates per reasoning step. The paper reports that this produces 4B tokens of trajectory data, but nowhere does it quantify the computational cost of this generation in FLOPs, GPU-hours, or dollar terms relative to the pretraining budget of 20B tokens. The only cost-related statement appears in Section 3.2:

"While this objective could be optimized using RL (with log p(x|z) as the reward), RL training is often complex, expensive, and unstable. We therefore opt for a simpler yet effective inference-time search strategy."

This comparison positions search-based optimisation as cheaper than RL, but provides no absolute cost. The actual generation pipeline involves: (1) running a 30B-parameter model to simulate multi-agent trajectories for 300k repositories, (2) executing repository analysis (AST parsing, dependency graph extraction, file tree traversal) for each repository, (3) grounding each trajectory by replacing tool outputs with ground-truth data, and (4) performing search-based CoT optimisation which requires, for each reasoning step in each trajectory, generating k=2 candidate refinements from the same LLM and performing k × T = 6 perplexity evaluations (each requiring a forward pass through a reference model) over T=3 rounds. The cost of step 4 alone — generating refinement candidates from a 30B model for potentially millions of reasoning steps across 300k trajectories, then evaluating perplexity — could exceed the cost of pretraining itself, especially since the 30B simulation model is larger (by active parameter count, ~3B) than the 8B model being trained.

The consequence. Without cost reporting, a practitioner cannot evaluate whether the 1–3 percentage-point gains on coding benchmarks (HumanEval: +2.44 over Raw-Repos, Table 3) and long-context tasks (Ruler 64k: +0.80, Table 2) justify the trajectory generation investment. The headline claim — that substituting 12% of training data with trajectories improves performance at fixed pretraining budget — is true as measured, but the total budget (generation + pretraining) may be substantially larger than simply pretraining on more raw code. If generating 4B trajectory tokens costs 10× more FLOPs than simply collecting 4B more raw code tokens and scaling pretraining by 20%, then the "efficiency" gain at fixed pretraining tokens is an accounting artifact — the real efficiency comparison would ask: given a fixed total FLOPs budget, is it better to spend it on trajectory generation and pretraining, or on pretraining alone with more data? This is directly analogous to the difficulty estimation cost problem flagged in Section 6.1 of the earlier paper: the $4\times$ efficiency gain over best-of-N did not include the cost of generating 2048 samples per question to estimate difficulty. Here, the trajectory generation cost is not amortised into the pretraining budget.

What evidence exists. The paper provides no FLOPs accounting, no wall-clock time reporting, and no cost comparison for any stage of the trajectory generation pipeline. The only cost-related figure is the trajectory length expansion (Figure 2b): raw code averages 4,865.5 tokens, Repo2Agent trajectories average ~9,500 tokens, Repo2Agent-Search averages 12,083.4 tokens. This tells us the output side (how many tokens are produced) but nothing about the input side (how much computation was required to produce them). We also know the CoT optimisation process was run on a sample of 100 trajectories tracked for 10 rounds (Figure 3), but the total computational cost of processing the full 300k-repository dataset through 3 rounds is not reported.

Mitigation status. The paper does not acknowledge this as a limitation. Section 5.1 notes that the trajectory generation produces longer per-repository sequences, and Section 6 (Conclusion) mentions that the method "transforms static repositories into dynamic, causally rich training data," but there is no discussion of the economic or computational tradeoff involved. A practitioner reading this paper cannot determine whether the approach is cost-effective or whether the trajectory generation step dominates the total budget.

6.2 The 12% Data Slot Design Does Not Isolate Format Effects from Repetition and Length Effects

The constraint. The experimental design fixes the experimental data slot at 12% of the total 20B-token pretraining budget and compares Raw-Repos (12% raw code) against Repo2Agent and Repo2Agent-Search (12% trajectories). However, as Figure 2b shows, trajectories are approximately 2.5× longer per repository than raw code (12,083.4 vs. 4,865.5 tokens on average for Repo2Agent-Search vs. raw code). Since the token budget is fixed, the trajectory-trained models see fewer distinct repositories but more tokens per repository compared to the Raw-Repos baseline. This creates a confound: the observed gains could arise from (a) the trajectory format providing richer supervision, (b) each repository being presented more intensively (the model sees the same codebase from multiple angles — plans, reads, writes — rather than just the static file content), or (c) the model being exposed to longer contiguous documents (12k tokens vs. 4.8k tokens per sample) which may improve long-context learning independent of content. The paper acknowledges the length difference in Section 5.1:

"Transforming raw code (avg. 4,865.5 tokens) into an agentic trajectory (Repo2Agent-Search, avg. 12,083.4 tokens) significantly increases the per-repository token count by making latent planning and execution steps explicit."

It then argues: "Importantly, despite the increased sample length, all model variants are trained on a fixed budget of 12% of 20B total tokens. This ensures a fair comparison: the performance gains are driven by the structural quality and informational density of the trajectories, rather than an increase in the total volume of training data."

This argument is incomplete. It correctly rules out the possibility that gains come from more total tokens (the budget is fixed), but it does not rule out the possibility that gains come from fewer, longer documents or repeated exposure to the same repositories from different perspectives, neither of which is "structural quality" in the sense the paper intends. The paper's preferred interpretation is that the trajectory format itself — the planning→reading→writing causal structure — provides superior supervision. An equally plausible interpretation is that any format which elaborates each repository into a longer, more repetitive document (e.g., raw code with synthetic comments, raw code with multiple restatements) would achieve similar gains, because the mechanism is simply more tokens of exposure per repository within the fixed budget.

The consequence. If the gains derive primarily from document length or repetition rather than from the specific trajectory structure, then the paper's central conceptual claim — that reverse-engineering the agentic development process provides uniquely valuable supervision — is not supported by the evidence. A practitioner could achieve similar gains by simply duplicating or reformatting existing code data to produce longer per-repository sequences, without the complex trajectory generation pipeline. The paper's policy recommendation — invest in trajectory generation infrastructure — would be incorrect if simpler, cheaper approaches to increasing per-repository token intensity produce equivalent benefits.

What evidence exists. The paper does not include any control that varies document length while holding format constant. Specifically, it does not test: (1) raw code padded with neutral filler to match trajectory length, (2) raw code repeated with different formatting or commentary, (3) trajectories truncated to match raw code length, or (4) a "flat" variant that preserves the Read/Write structure but removes the multi-agent hierarchy. Without at least one of these controls, the causal attribution to trajectory structure (as opposed to trajectory length) is speculative. The fact that Repo2Agent-Search trajectories are longer than Repo2Agent trajectories (~12k vs. ~9.5k tokens, Figure 2b) and also generally perform better (Tables 2–4) is consistent with a length-based explanation, although the Repo2Agent vs. Repo2Agent-Search comparison partially controls for format (both are trajectories) while varying only optimisation and length. The Repo2Agent vs. Raw-Repos comparison does not control for length at all.

Mitigation status. The paper does not acknowledge this confound or propose controls. The "informational density" framing in Section 5.1 implicitly assumes that the additional tokens in trajectories are causally valuable, but this is the claim under test, not a premise.

6.3 Performance Gains on Hard Problems and General Reasoning Are Near-Zero

The constraint. The paper demonstrates consistent gains on coding benchmarks (HumanEval: +2.44, LongCodeBench-32k: +2.30 over Raw-Repos, Table 3) and moderate gains on long-context understanding (Ruler 64k: +0.80, Helmet 32k: +1.67 over Raw-Repos, Table 2) and agentic capability (APTBench: +1.08 for Repo2Agent over Raw-Repos, Table 4). However, the gains are absent or negligible on several important categories:

• General reasoning: On BBH (BigBench Hard), all variants cluster within 1 point (66.00–67.03, Table 3). On AGI-Eval, the range is 35.78–36.91 — less than a 1.1-point spread across all variants including Prolong. On GSM-8k, Raw-Repos and Repo2Agent both score 61.94, and Repo2Agent-Search scores 60.96 — trajectories provide zero or negative benefit. The only reasoning benchmark showing gains is MATH (3.76 for Repo2Agent-Search vs. 2.18 for Raw-Repos, Table 3), but the absolute scores are very low (3.76%), and it is unclear whether a 1.58-point gain at this performance level represents meaningful capability or noise.

• Extreme long-context tasks: On Ruler at 64k, the unoptimised Repo2Agent variant (58.10) actually underperforms Raw-Repos (61.00, Table 2), showing that unrefined trajectories can hurt performance at the hardest context lengths. On Helmet at 64k, Repo2Agent-Search (57.84) trails Prolong (58.10, Table 2). The gains are concentrated at shorter context lengths (16k, 32k) and in specific task types (NIAH-Multi, ICL, Recall).

• Hardest coding tasks: On LongCodeBench-64k, Repo2Agent-Search (30.26) underperforms Repo2Agent (31.05) and barely edges out Prolong (30.52), while Raw-Repos (27.37) trails significantly. But the spread is small (3.68 points across all variants), and the ranking is inconsistent with the 32k results.

This mirrors a pattern observed in the prior paper's Section 7: test-time compute scaling provides gains on easy-to-medium difficulty problems but near-zero benefit on the hardest problems (difficulty bin 5). Here, the trajectory pretraining provides gains on standard benchmarks but negligible or negative benefit on the most challenging variants of those same benchmarks.

The consequence. The trajectory pretraining approach appears to improve performance on tasks that are relatively close to the base model's existing capabilities (standard code generation, moderate-length retrieval, structured planning) but provides no benefit — and sometimes degrades performance — on tasks that push the model to its limits (extreme context lengths, hard general reasoning, complex mathematics). This suggests that the trajectory data teaches specific patterns (plan-then-read-then-write) that generalise to tasks sharing that structure, but does not teach general reasoning ability that would transfer broadly. For a practitioner whose use case involves pushing model capabilities to the frontier rather than improving performance on standard tasks, trajectory pretraining offers limited value.

What evidence exists. The flat or negative results on BBH, AGI-Eval, GSM-8k (Table 3), and the degradation of Repo2Agent on Ruler at 64k (Table 2) directly demonstrate this limitation. The APTBench results (Table 4) provide a more nuanced picture — gains exist in specific categories (Issue-Fix Plan: 40.74 vs. 37.04; Env-Setup Error: 24.49 vs. 22.45) but the overall gain is modest (30.10 vs. 29.02), and there are categories where trajectories underperform (Openend-Plan: 13.42 for Repo2Agent vs. 16.11 for Raw-Repos). The limitation is also visible in the MATH results: a 1.58-point gain at 3.76% absolute accuracy is statistically unreliable without confidence intervals (which the paper does not report), and even if real, the model remains incapable of solving the vast majority of MATH problems — trajectory pretraining does not transform a model that cannot do mathematics into one that can.

Mitigation status. The paper does not explicitly discuss this domain-specificity. Section 4.2.2 states that "the structured logic within agentic trajectories provides a higher-quality supervision signal than raw code, enhancing specialized skills without compromising general intelligence," which is accurate only in the sense that general intelligence does not degrade — but it also does not improve. The paper's framing of "enhancing specialized skills" is a post-hoc narrowing of the original claim in the abstract that the method "significantly enhances Llama-3-8B's performance across diverse benchmarks, including long-context understanding, coding proficiency, and agentic capabilities." The "diverse benchmarks" claim would be more accurate if qualified with the observation that general reasoning benchmarks show no improvement.

6.4 The Approach Cannot Be Deployed Reciprocally — Trajectory Quality Depends on an External Model That Is Stronger Than the Model Being Trained

The constraint. The trajectory generation pipeline relies on Qwen3-30B-A3B-Instruct-2507, a 30B-parameter Mixture-of-Experts model (3B active parameters) that is substantially larger and more capable than the 8B Llama-3 model being trained. This creates an asymmetric dependency: to improve a smaller model via trajectory pretraining, you need access to a larger, more capable model to generate the trajectories. The paper does not test whether trajectories generated by Llama-3-8B itself (or a similarly-sized model) would produce comparable gains, nor does it test whether the trained Llama-3-8B could then generate improved trajectories to bootstrap further improvement (closing the self-improvement loop).

The consequence. This limits the practical applicability and scalability of the approach in two ways. First, the method is not self-bootstrapping: you cannot use the 8B model to generate its own training data and iteratively improve, because the trajectory quality depends on the simulator's reasoning capability exceeding the trainee's. If you attempted this, the 8B model would generate trajectories reflecting its own (limited) reasoning patterns, and training on those trajectories would at best reinforce existing capabilities rather than expand them. Second, the approach does not improve the state-of-the-art frontier: to train a model better than any existing model, you would need a trajectory generator that is already better than your target — but if you have such a generator, you could simply deploy it directly rather than using it to train a weaker model. The paper demonstrates that trajectory pretraining can close the gap between a smaller model and a larger one (Qwen3-30B's reasoning patterns improve Llama-3-8B's downstream performance), but it cannot help a model exceed the capabilities of the best available trajectory generator.

The paper's contribution is therefore best understood as a data augmentation technique for knowledge distillation: using a strong model to generate training data that a weaker model can learn from. This is valuable — distillation is widely used in practice — but it is more limited than the framing of "understanding via reconstruction" as a general paradigm for improving models. A genuinely general paradigm would not require an oracle model to generate the reconstruction.

What evidence exists. This is a conceptual limitation rather than an empirical one, but the paper's own experimental setup makes it visible. The simulation engine is explicitly Qwen3-30B-A3B-Instruct-2507 (Section 4.1), and the base model is Llama-3-8B-Instruct. The capability gap between these models (30B MoE vs. 8B dense) is substantial. The paper does not include any experiment where the simulation engine is a model of comparable or smaller scale to the trainee. Appendix K of the prior paper provides a cautionary parallel: the ReST^EM-trained revision model degraded substantially when trained on its own on-policy data, suggesting that self-generated training data can amplify errors rather than correct them. The same concern applies here: if the trajectory generator has systematic reasoning weaknesses, those weaknesses will be baked into the training data and transferred to the trainee. The paper's choice of a strong, external generator avoids this problem but at the cost of requiring access to a model that is already more capable.

Mitigation status. The paper does not discuss this limitation. Section 6 (Conclusion) presents the method as a general paradigm without noting the dependency on an external, stronger model. There is no discussion of whether the trained Llama-3-8B could serve as its own trajectory generator in a subsequent iteration, or whether the approach could work with a same-scale generator.

6.5 No Evaluation on Repository-Level, Multi-File Code Generation — the Core Skill the Trajectories Are Designed to Teach

The constraint. The paper's central hypothesis is that reconstructing the agentic development process teaches models the skills that static code misses: dependency-aware implementation ordering, cross-file reasoning, Read-before-Write information gathering, and multi-step planning across a repository. However, the evaluation benchmarks test only indirect proxies for these skills. HumanEval (Table 3) evaluates single-function completion from a docstring — no dependencies, no file interactions, no planning. LongCodeBench (Table 3) evaluates single-function generation with long contexts — the context contains other code, but the task is still isolated function completion, not repository-level construction. APTBench (Table 4) tests atomic agentic skills (planning, bug-fixing, error diagnosis) on SWE-Bench-style tasks, but these evaluate a model's ability to fix or extend existing code, not to construct a repository from scratch given requirements. Ruler and Helmet (Table 2) test long-context retrieval and reasoning, which are relevant to understanding trajectory structure but do not test code generation at all.

No benchmark in the paper evaluates the task that the trajectories are explicitly designed to teach: given a high-level requirement and a repository skeleton (or empty directory), generate a multi-file project with correct cross-file dependencies, implementing files in the right order, and reading dependencies before writing code that uses them. This is the core skill demonstrated in the trajectory example in Table 1 (creating a two-file calculator project where main.py is implemented after and depends on operations.py).

The consequence. The paper cannot distinguish between two competing explanations for the observed gains: (1) the trajectories teach the specific causal skills they are designed to teach (planning, dependency reasoning, tool use), and these skills transfer to the evaluated benchmarks through some common underlying capability; or (2) the trajectories provide a form of intensive, repetitive exposure to code (each repository seen from multiple perspectives) that improves general code understanding, and the gains on coding benchmarks reflect this general improvement rather than the acquisition of trajectory-specific skills. If explanation (2) is correct, then simpler approaches (e.g., adding code explanations, docstrings, or multiple representations of the same code) might achieve similar gains without the full trajectory generation pipeline. Without a direct test of trajectory-specific skills, the paper's causal claims about why the method works remain untested.

A practitioner who needs exactly the skills the trajectories teach — repository-level code generation from requirements — has no evidence from this paper that trajectory pretraining would help, because that skill is never evaluated. A practitioner who needs better single-function completion (HumanEval) or long-context retrieval (Ruler) might choose trajectory pretraining based on the reported gains, but would be relying on an unverified assumption about transfer.

What evidence exists. This limitation is evident from the benchmark descriptions in Section 4.1. HumanEval is described as testing code generation, LongCodeBench as testing "long-horizon tasks" in code, APTBench as assessing "foundational agentic capabilities." None of these descriptions mention multi-file repository construction, cross-file dependency management, or requirement-to-project generation. The closest benchmark is APTBench's Issue-Fix and Env-Setup categories (Table 4), which test bug-fixing and environment configuration within existing repositories — but these are modification tasks, not construction tasks. The trajectory example in Table 1 and the case study in Appendix C both show repository construction from scratch, but this skill is never tested downstream.

Mitigation status. The paper does not acknowledge this evaluation gap. There is no custom benchmark or targeted evaluation of the specific capabilities that the trajectory format is hypothesized to teach. This is arguably the most significant omission in the experimental design, because it leaves the paper's central causal claim — that reconstructing the development process teaches models how software is built, not just what it looks like — without direct empirical support.

6.6 The Trajectories Are Generated by a Single Model and Contain That Model's Systematic Biases

The constraint. All trajectories in the dataset are generated by a single model: Qwen3-30B-A3B-Instruct-2507. This model has specific reasoning patterns, coding style preferences, architectural biases, and failure modes that are characteristic of its training. The paper's grounding mechanism ensures that the terminal state of each trajectory is correct (Write outputs are replaced with ground-truth code), but the reasoning that leads to that state — the Think steps, the planning decisions, the order of Read calls, the justifications for implementation choices — reflects Qwen3's reasoning patterns, not necessarily those of a generic or expert developer. Even after CoT optimisation, the reasoning is optimised to maximise the predictability of the target code under Qwen3's own probability model (perplexity is measured using a reference LLM, presumably the same Qwen3 model). This means the "optimal" reasoning according to the search procedure is optimal for making Qwen3's predictions correct, which may not correspond to reasoning that is objectively valid, causally faithful, or transferable to other models.

The consequence. The trained Llama-3-8B model may internalise Qwen3's reasoning biases rather than learning general reasoning principles. If Qwen3 has a tendency to over-plan certain types of files, under-use certain dependency checks, or justify decisions with plausible-sounding but causally irrelevant rationales, these patterns will appear in the training data and be learned by Llama-3. The paper provides no characterisation of what systematic biases might exist in Qwen3's trajectory generation, nor any analysis of whether the trained Llama-3 model reproduces Qwen3-specific patterns.

A more subtle consequence concerns the generality of the findings. If another research group attempted to replicate this work using a different simulation engine (e.g., GPT-4, Claude, DeepSeek-Coder), they might generate trajectories with substantially different reasoning characteristics — more or less detailed, different planning styles, different dependency-checking patterns. The downstream gains might differ accordingly. The paper's results are conditional on Qwen3's specific trajectory generation style, and the extent to which they generalise to other simulation engines is unknown. This is a form of simulator bias — the same phenomenon that affects synthetic data generation in other domains (e.g., RL sim-to-real transfer, instruction tuning from specific teacher models), but the paper does not discuss it.

What evidence exists. The paper provides composition statistics for the generated trajectories (Figure 2a: token distribution across Main-Agent and Sub-Agent think/call/response components; Figure 2b: trajectory length). It also provides a case study showing the evolution of a single CoT step through optimisation rounds (Appendix C). But there is no analysis of reasoning quality diversity: do all trajectories follow a similar template? Do certain types of repositories systematically produce lower-quality reasoning? Does Qwen3 exhibit specific biases (e.g., overusing certain architectural patterns, under-specifying error handling, over-relying on external libraries)? The paper does not report any quality assessment of the generated trajectories beyond the intrinsic perplexity metric (Figure 3b) and the downstream performance of the trained models — both of which are circular if the goal is to detect simulator-specific biases, because the perplexity metric is itself evaluated using the same model that generated the data, and the downstream model might learn and reproduce the same biases.

Mitigation status. The paper does not acknowledge simulator bias as a limitation. The choice to use Qwen3-30B-A3B is mentioned once in Section 4.1 with no justification beyond naming the model, and there is no discussion of how the results might depend on this choice. Section 5.2 does note that "the search process doesn't merely refine thoughts but substantially elaborates on the logical steps required for implementation" (commenting on Figure 2a), but this treats the elaboration as inherently positive without considering whether the elaboration reflects Qwen3-specific reasoning patterns that may not generalise.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing rather than a paradigm shift. It does not propose a new architecture, training algorithm, or scaling law. Instead, it identifies a specific structural deficiency in how the field thinks about pretraining data for code LLMs — static repositories are causally incomplete artifacts whose generative history has been compressed away — and provides a concrete pipeline for decompressing them. The magnitude of the change is incremental in practical terms (1–3 point gains on most benchmarks) but potentially significant in conceptual terms: it shifts the conversation from "how much data" to "what kind of data contains the supervision signal for the capabilities we care about."

The paper resolves a latent tension between two observations that the field had previously held separately but not connected. First, models trained on massive code corpora (DeepSeek-Coder, Qwen2.5-Coder) achieve strong performance on isolated function-level tasks but struggle with repository-level reasoning that requires cross-file dependency management, architectural planning, and multi-step debugging. Second, synthetic agent trajectory data — even when LLM-generated and imperfect — can teach models to use tools, plan actions, and reason interactively (Chen et al., 2025; Li et al., 2025; Team, 2025). The paper connects these observations by arguing that the first problem (poor repository-level reasoning) is caused by the absence of the second signal (interactive, multi-step process data) in standard pretraining. The key insight is that the missing capability and the missing data format are the same thing: to reason about constructing software, a model needs to have observed software being constructed.

This reframing makes several research directions more attractive. Process-aware pretraining data design — asking not just "what data is available" but "what generative process produced this data, and can we reconstruct it" — becomes a first-class research question rather than an afterthought. Synthetic trajectory generation for pretraining (as opposed to fine-tuning) becomes credible, since the paper demonstrates that trajectory data works in the noisy, high-volume pretraining regime where it was previously unclear whether imperfect synthetic reasoning would help or harm. Repository-level code generation benchmarks become urgent, since the paper's core causal claims about what trajectories teach cannot be evaluated with existing single-function benchmarks. The paper also makes certain directions less attractive: scaling raw code volume as the sole path to better code models now faces a principled challenge — more static code may not help if the missing causal structure remains absent.

A less obvious but important shift concerns the role of search in data curation. The CoT optimisation procedure — iteratively refining reasoning to maximise the conditional probability of the ground-truth code — operationalises the idea that reasoning quality should be measured by predictive power, not coherence or plausibility. This is a departure from how synthetic reasoning data is typically generated (where the criterion is "does the reasoning sound reasonable to a human or an LLM judge") and provides a self-supervised quality metric that does not require human labels or downstream task evaluation. If this metric proves robust across domains, it could become a standard component of synthetic data pipelines beyond code.

Follow-Up Research This Work Enables

Repository-level code generation benchmarks that directly test the skills trajectories are designed to teach. The paper's central causal claim — that reconstructing the development process teaches models dependency-aware planning, cross-file reasoning, and Read-before-Write information gathering — is supported only by indirect proxy benchmarks (HumanEval, LongCodeBench, APTBench) that test isolated functions or bug-fixing within existing repos. What is needed is a benchmark that presents a model with a high-level requirement and an empty directory (or a partial repository skeleton), then evaluates whether the model can generate a multi-file project with correct cross-file imports, implement files in dependency order, and produce code that compiles and passes tests. A strong follow-up would construct such a benchmark from the 300k repositories already curated by the paper — for each repository, extract the requirement (already generated by the Main Agent during simulation), provide the file tree but not the file contents, and measure whether trajectory-trained models construct the repository more accurately than raw-code-trained models. This would provide the first direct test of whether trajectory pretraining teaches construction skills or merely improves general code understanding.

Closing the self-improvement loop by using the trained model as its own trajectory generator. The paper's trajectory generation depends on Qwen3-30B-A3B, a model larger and more capable than the Llama-3-8B being trained. This limits the approach to distillation — you need a stronger model to generate trajectories for a weaker one. A critical follow-up would test whether the trajectory-trained Llama-3-8B can generate trajectories of sufficient quality to train a further improved version of itself. The experiment would proceed in iterations: (1) train Llama-3-8B-v1 on Qwen3-generated trajectories, (2) use Llama-3-8B-v1 to generate trajectories for the same repositories, (3) train Llama-3-8B-v2 on these self-generated trajectories, (4) compare v2 against v1 on the repository-level benchmark described above and on standard coding benchmarks. If v2 outperforms v1, the approach becomes a bootstrapping mechanism rather than a one-time distillation, and the practical ceiling shifts from "as good as the trajectory generator" to "as good as the iterative self-improvement process can reach." If v2 does not outperform v1 (or degrades, analogous to the ReST^EM failure in Appendix K of the prior paper), this would reveal a fundamental limitation: the approach amplifies existing reasoning patterns rather than creating new ones, and the trajectory format is only valuable when generated by a model with qualitatively different (and superior) reasoning capabilities than the trainee.

Scaling the trajectory data fraction to determine the dose-response curve and whether gains saturate or compound. The paper tests exactly one data mixture: 12% trajectories, 18% Prolong repos, 70% general-domain. No sensitivity analysis is reported. A natural follow-up would sweep the trajectory fraction across values like 3%, 6%, 12%, 24%, and 48% (adjusting the general-domain fraction downward to maintain the total token budget), and measure performance on the coding, long-context, and agentic benchmarks from the paper. Three patterns are possible: (a) gains saturate at low fractions, indicating that trajectories provide a specific signal that is quickly learned and additional trajectory data is redundant; (b) gains increase monotonically, indicating that trajectory data is strictly higher-quality than the general-domain data it replaces and the optimal fraction is limited only by generation cost; (c) gains are U-shaped, with degradation at high fractions as model overfits to trajectory-specific patterns at the expense of general capability. The pattern would directly inform practical data mixture decisions for practitioners and would also test the implicit claim that trajectories provide "higher informational density" — if gains saturate at 12%, the density advantage is real but bounded; if gains continue growing at 48%, the density advantage is substantial and the 12% choice was conservative.

A trajectory length control to deconfound format effects from document length and repetition effects. The paper's trajectories are 2.5× longer per repository than raw code (Figure 2b: 12,083.4 vs. 4,865.5 tokens). Since the experimental data slot is fixed at 12% of total tokens, trajectory-trained models see fewer distinct repositories but more tokens per repository. A controlled follow-up would test: (1) raw code padded with neutral filler (e.g., syntactically valid but semantically empty code, or repeated comments) to match trajectory length, (2) raw code augmented with synthetic but non-agentic commentary (e.g., docstrings, line-by-line explanations) to match trajectory length without introducing planning/reading/writing structure, and (3) trajectories truncated to match raw code length. The key comparison is whether padded raw code (same length, no structure) achieves trajectory-level performance. If it does, the gains are attributable to document length or repetition rather than trajectory structure, and the paper's central claim is falsified. If it does not, the structural signal is genuinely causal. This experiment is cheap relative to trajectory generation (padding or truncation is post-hoc) and would substantially strengthen or qualify the paper's claims.

Verifier quality and over-optimisation analysis on synthetic reasoning data, parallel to the PRM over-optimisation documented in the prior paper. The prior paper on compute-optimal test-time scaling demonstrated that beam search against a process reward model degrades performance at high budgets due to verifier over-optimisation — search finds solutions that score highly under the verifier but are incorrect. A structurally analogous risk exists in this paper's CoT optimisation: the search procedure refines reasoning to minimise code perplexity (maximise log p(x|z)), but perplexity is measured under the same model (Qwen3-30B-A3B) that generated the candidate refinements. Optimising reasoning to satisfy Qwen3's own probability model may produce reasoning that is "overfit" to Qwen3's idiosyncratic predictive patterns — reasoning that makes the code predictable to Qwen3 specifically, but does not reflect genuine causal structure that would transfer to another model. A follow-up would test this by measuring code perplexity under a different reference model (e.g., Llama-3-8B before training, or a different model family entirely) after each round of CoT optimisation. If perplexity decreases for the optimisation model but plateaus or increases for the held-out model, this would be direct evidence of verifier over-optimisation in the reasoning refinement process — the search is exploiting Qwen3-specific patterns rather than discovering genuinely better explanations. This would parallel the finding in the prior paper's Figure 3 (right) where beam search over-optimised the PRM on easy problems, and would suggest that CoT optimisation should use a cross-model perplexity metric rather than a same-model metric.

Extending the reconstruction paradigm to non-code domains with clean terminal-state verification. The paper's approach — use a strong LLM to simulate the generative process behind a static artifact, ground the simulation in structural facts extracted from the artifact, and verify that the terminal state matches the artifact — generalises to any domain where: (a) the final output is a structured artifact with extractable ground-truth properties, and (b) the generative process involves multi-step reasoning with intermediate decisions that are not recoverable from the output alone. Candidate domains include: mathematical proofs (the proof is the terminal state; the reconstruction would simulate the exploration, false starts, and lemma discovery that led to it), legal documents (contracts or legislation as terminal states; reconstruction simulates negotiation, clause drafting, and revision), architectural designs or circuit schematics (terminal states with extractable dependency structures), and scientific papers (the published paper as terminal state; reconstruction simulates hypothesis formation, experimental design, and revision). A concrete first experiment would apply the pipeline to mathematical proofs from a dataset like NaturalProofs or the AFP: extract theorem dependencies as the "structural ground truth," simulate a mathematician agent that explores proof strategies before arriving at the final proof, apply CoT optimisation to refine the reasoning, and measure whether proof-trained models improve on theorem-proving benchmarks. The key test of generalisability is whether the reconstruction paradigm works when the "code" is replaced with a different structured artifact and the "development process" is replaced with a different generative process (proof construction), without changing the fundamental pipeline architecture.

Practical Applications and Downstream Use Cases

Data-efficient pretraining for code-specialised smaller models. The paper demonstrates that replacing just 12% of a 20B-token pretraining budget with reconstructed trajectories yields coding performance improvements of 2–4 points on HumanEval and LongCodeBench over an equivalent raw-code baseline (Table 3). For an organization training a code-specialised model in the 7–13B parameter range, this translates to a concrete recipe: rather than scaling raw code data volume by 20% (which would require collecting, filtering, and deduplicating millions of additional repositories), invest in trajectory generation for a curated subset of ~300k high-quality repositories. The 4B tokens of generated trajectory data can substitute for 2.4B tokens of the pretraining budget (12% of 20B) and yield measurable gains at fixed total pretraining cost. The primary practical barrier is the trajectory generation cost (unquantified in the paper), but if generation can be made efficient — e.g., by using a smaller or distilled simulation model, or by amortising generation cost across multiple training runs — the cost-benefit tradeoff is attractive for teams that have already exhausted easy gains from scaling raw data volume and are looking for data-quality-driven improvements.

Improving long-context code understanding in retrieval-augmented coding assistants. The paper's long-context results show that Repo2Agent-Search achieves 61.80 on Ruler at 64k context versus 57.10 for the Prolong baseline (Table 2), with particularly strong gains on multi-key retrieval tasks (NIAH-Multi at 64k: 80.40 vs. 66.20, Table 5). This directly translates to better performance in retrieval-augmented coding scenarios where an LLM must locate and integrate information from multiple files across a large repository. A coding assistant that retrieves relevant context from a repository before generating code — a standard architecture for tools like Sourcegraph Cody, GitHub Copilot's @workspace, or Cursor — benefits from a base model that more accurately tracks and integrates information across long, multi-file contexts. Integrating trajectory-pretrained models into such assistants would improve the quality of cross-file code generation, dependency-aware refactoring, and bug-fixing that requires understanding interactions between distant components. The 12% data slot design means this improvement can be achieved by swapping the model checkpoint without changing the retrieval architecture, context assembly logic, or prompt templates — it is purely a model quality upgrade.

Bootstrapping agentic capability in base models before post-training. The APTBench results (Table 4) show that pretraining on trajectories improves agentic skills (+1.08 overall for Repo2Agent over Raw-Repos), with category-specific tradeoffs: unoptimised trajectories improve planning (+3.7 points in Issue-Fix Plan), while search-optimised trajectories improve error diagnosis (+2.04 points in Env-Setup Error). For teams building agentic coding systems (SWE-agent-style, autonomous debugging, or automated PR generation), this suggests a data-centric strategy for the pretraining phase: include both optimised and unoptimised trajectories in the training mixture to cover both planning and debugging skills, and use APTBench-style evaluations to guide the mixture ratio. Since APTBench tests "foundational agentic capabilities of pre-trained models without post-training" (Section 4.1, citing Qin et al., 2025), improvements at this stage compound with downstream fine-tuning — a model that enters post-training with stronger planning and debugging instincts will converge faster and reach a higher ceiling on agentic tasks. The practical benefit is a stronger base model that requires less agent-specific fine-tuning data to reach a target performance level, reducing the cost and complexity of the post-training phase.