The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

🎯 Pitch

For the first time, a fully AI-generated paper survived peer review and met the acceptance bar at an ICLR workshop—without using any human-written code templates. The system’s key innovation is an experiment manager agent that conducts progressive agentic tree search, enabling it to design, execute, and iteratively refine its own experiments from scratch across multiple ML domains.

1. Executive Summary

This paper introduces The AI Scientist-v2, an end-to-end agentic system for automated scientific discovery that produces the first entirely AI-generated peer-review-accepted workshop paper. The system iteratively formulates hypotheses, designs and executes experiments, and authors manuscripts using a progressive agentic tree-search methodology managed by a dedicated experiment manager agent (coordinating four stages from preliminary investigation through ablation studies) and a Vision-Language Model (VLM) feedback loop (critiquing figure quality during experimentation and manuscript writing). Submitted to an ICLR workshop, one of three fully autonomous manuscripts achieved a reviewer score of 6.33/10 — exceeding the average human acceptance threshold — establishing that an AI-generated paper can successfully navigate peer review only when operating at workshop-level rigor, with the remaining submissions and internal analysis revealing that conference-level depth, citation accuracy, and methodological precision remain beyond current capabilities.

2. Context and Motivation

The Core Problem: From Template-Bound Automation to Genuinely Autonomous Scientific Discovery

The fundamental problem this paper addresses is not whether AI can assist with scientific research — it clearly can and does — but whether an AI system can perform the entire scientific workflow autonomously, from idea generation through peer-reviewed publication, without relying on human-provided structural scaffolding. The paper's predecessor, The AI Scientist-v1 (Lu et al., 2024), demonstrated that end-to-end automation was possible in principle, but its architecture embedded a critical limitation that prevented genuine autonomy: it required human-authored code templates tailored to each specific research topic. This paper aims to eliminate that dependency entirely.

To understand why template dependency matters, consider what a human must do to deploy The AI Scientist-v1 in a new research domain. Someone must write a baseline code implementation — a working experiment skeleton that includes data loading, model architecture, training loop, and evaluation. The AI Scientist-v1 then operates by proposing and applying incremental modifications to that template: tweaking hyperparameters, adding regularization terms, swapping loss functions. The space of possible experiments is therefore constrained to variations on the template's theme. This is not autonomous scientific discovery in the sense of formulating new experimental paradigms from scratch; it is automated local exploration around a human-chosen starting point. The authors state this limitation explicitly in their comparison table (Table 1), contrasting v1's "topic-specific" codebase against v2's "domain-general" approach.

The gap this paper addresses is therefore: how do we build a system that can design and implement experimental code from a high-level research idea, without relying on pre-written code templates, while still maintaining scientific rigor and producing publishable results?

Why This Problem Matters: The Scaling Argument for Automated Science

The practical significance of truly autonomous scientific discovery systems extends far beyond academic curiosity. The authors frame their motivation around a scaling argument that appears throughout the paper but is most explicitly stated in the conclusion (Section 7):

"We believe what matters most is not simply how AI science compares to human science, but whether its discoveries aid in human flourishing, such as curing diseases or expanding our knowledge of the laws that govern our universe."

The implicit claim is that human scientific progress is rate-limited by human cognitive bandwidth, training time, and institutional inertia. An AI system that can autonomously formulate hypotheses, design experiments, analyze results, and communicate findings could — if sufficiently capable — explore scientific hypothesis spaces at scales and speeds inaccessible to human researchers. This is not merely about automating drudgery; it's about increasing the throughput of the scientific process itself.

However, the paper is careful to situate this ambition within concrete, measurable milestones. Rather than claiming to have solved automated science, the authors set a specific, falsifiable target: can an AI system produce a manuscript that survives peer review at a recognized machine learning venue? This framing is strategically important because it treats peer review as an external validity check rather than relying solely on internal evaluation metrics. It also acknowledges the current limitations head-on — the target is a workshop, not a top-tier conference, and only one of three submissions succeeded.

Where The AI Scientist-v1 Fell Short

To understand the motivation for v2's specific architectural innovations, we need to examine exactly where v1's approach broke down. The authors identify three categories of limitation (Section 2, Background):

1. Template Dependency Constrained Autonomy and Scope. The AI Scientist-v1 required human-authored baseline code for each new research domain. This created a bottleneck: every new topic area demanded manual effort to create a template, which meant the system could not be deployed "out of the box" on arbitrary research questions. More subtly, the template constrained the type of experiments the system could propose — it could only explore variations on the template's architecture and methodology, not invent fundamentally different experimental designs. The authors characterize v1's approach as producing "incremental modifications or extensions based on an existing codebase" (Section 3.1), whereas v2 "begins at a higher level of abstraction" with open-ended idea generation.

2. Linear Experimentation Prevented Deep Exploration. The AI Scientist-v1 conducted experiments in a strictly linear sequence: run an experiment, analyze results, propose a modification, run the next experiment, repeat. The authors observed that this produced "short-sighted experimentation" (Section 2). Real scientific research is iterative and branching — a researcher might try several approaches in parallel, follow promising leads deeper, abandon dead ends, and revisit earlier ideas with new insights. A linear pipeline cannot recover from early mistakes or explore multiple hypotheses simultaneously. The paper frames this limitation using the language of exploration:

"The human-driven scientific process... relies on open-ended hypothesis generation, stepping-stone collection, and iterative hypothesis refinement."

3. No Visual Feedback Loop During Experimentation or Writing. The AI Scientist-v1 did not leverage Vision-Language Models (VLMs) to inspect generated figures. This meant that the system could produce figures with missing legends, unclear labels, misleading visualizations, or captions that didn't match the plotted data — and it had no mechanism to detect or correct these issues. Given that figures are central to communicating scientific results, this was a significant quality gap. In the writing phase specifically, v1 used an incremental, Aider-based iterative writing approach (Gauthier, 2024), which the authors replaced in v2 with a simpler single-pass generation followed by a reflection stage.

Conflicting Prior Work and the State of Automated Science

The paper positions itself within a rapidly developing landscape of AI-for-science systems, and understanding this landscape helps clarify what The AI Scientist-v2 is not — as well as what it is. The related work section (Section 6) describes a field in flux, with multiple groups pursuing different slices of the automation pipeline:

End-to-end systems with varying degrees of autonomy. AI-Researcher (Data Intelligence Lab, 2025) is described as another fully automated framework. Intology (Intology AI, 2025) and Carl (AutoScience AI, 2025) incorporate "varying degrees of human oversight." The distinction matters because "autonomous" is a spectrum — some systems automate idea generation but not execution, others automate execution but require human problem specification.

Systems with narrower scope. CycleResearcher (Weng et al., 2025) focuses specifically on idea generation and manuscript drafting while "explicitly excluding experimental execution." This is an important contrast: The AI Scientist-v2's claim to end-to-end autonomy includes executing real code, which introduces a fundamentally different set of challenges (bug handling, runtime management, result extraction) than systems that only generate text.

LLM-based scientific idea generation. Si et al. (2025) conducted a large-scale human study finding that LLM-generated ideas were "typically more novel but often less feasible than those proposed by human experts." This finding is directly relevant to The AI Scientist-v2's design: the system includes literature review tools (Semantic Scholar) specifically to assess novelty and feasibility during idea generation, not just as a post-hoc check.

Industry efforts. Google's AI Co-Scientist (Gottweis et al., 2025) represents the major-industry parallel to this work, though the paper does not provide a detailed comparison of capabilities.

How This Paper Positions Itself

The AI Scientist-v2 does not claim to be the only or first automated science system. Instead, it makes three specific, verifiable claims about what distinguishes it from prior and concurrent work:

Claim 1: Elimination of template dependency. This is the architectural contribution that most directly addresses v1's core limitation. The system generates experimental code from a high-level idea description, using agentic tree search to explore implementation variants, debug errors, and refine working prototypes. The technical mechanism for this — the experiment manager agent coupled with parallelized tree search across four distinct stages — is the paper's primary methodological innovation and is described in detail in Section 3.2.

Claim 2: Peer review as external validation. Rather than relying on internal benchmarks or author self-assessment, the paper submits AI-generated manuscripts to a real peer-review process. This is methodologically important because it subjects the system to the same evaluation standard applied to human researchers. The authors explicitly acknowledge the limitations of this evaluation — workshop-level rather than conference-level, one acceptance out of three submissions — but argue that even this result represents a milestone: "the first instance of a fully AI-generated paper successfully navigating a peer review process" (Section 4.2).

Claim 3: Depth of exploration through tree search. By replacing linear experimentation with branching tree search, the system can explore multiple hypotheses in parallel, recover from failed experiments through debugging nodes, and systematically conduct ablation studies. The authors frame this as moving from "short-sighted experimentation" toward "open-ended hypothesis generation, stepping-stone collection, and iterative hypothesis refinement" (Section 2).

The Gap Between Demonstration and Deployment

A careful reading reveals that the paper's motivation is both ambitious and measured. The authors are not claiming to have built a system that replaces human scientists. They are claiming to have built a system that can — under specific, constrained conditions — produce work that survives peer review at the workshop level. The conditions include: human selection of the high-level research topic (the workshop theme), human selection of which AI-generated ideas to pursue (from a pool of ~40), human selection of the best complete manuscript from multiple seeded runs, and manual downloading of non-standard datasets (as described for the pest detection paper in Appendix C.3.1).

This gap between full autonomy and the demonstrated capability is not hidden — the authors are explicit about it (Section 4.2):

"Crucially, while humans initiated the process by providing the high-level workshop theme and selected which initial AI-generated ideas to run multiple times through the full pipeline... the entire process within any single run... was performed autonomously."

This framing — "within a single run, fully autonomous; across runs, human meta-selection" — is a precise description of the current capability boundary. The motivation for building toward fuller autonomy is clear, but the paper is careful not to overclaim what has been achieved.

3. Technical Approach

3.1 Reader Orientation

The AI Scientist-v2 is an end-to-end agentic system that takes a high-level research topic as input and autonomously produces a complete scientific manuscript — including hypothesis formulation, code implementation, experiment execution, data analysis, figure generation, and LaTeX writing — as output. The core problem it solves is eliminating the template dependency that constrained its predecessor: rather than requiring a human to write starter code for each research domain, v2 generates experimental code from scratch using a progressive tree-search methodology where an experiment manager agent coordinates four sequential stages, each stage exploring a branching tree of code variants through parallel refinement and debugging until a publishable result emerges.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components arranged in a pipeline that flows from abstract idea to peer-review-ready manuscript:

Generalized Idea Generator — an LLM-powered module (Section 3.1) that produces research proposals at the abstract/grant-proposal level, incorporating literature search via Semantic Scholar to assess novelty. Output: a structured idea JSON (title, hypothesis, experiments list, risk factors).
Experiment Progress Manager — a coordinator agent (Section 3.2.1) that sequences experimentation through four stages (Preliminary Investigation → Hyperparameter Tuning → Research Agenda Execution → Ablation Studies), selecting the best-performing node at each stage boundary to seed the next stage.
Parallelized Agentic Tree Search — the execution engine (Section 3.2.2) that, within each stage, expands multiple experimental nodes in parallel: an LLM generates experiment plans and Python code, the code executes in an interpreter, and nodes are classified as buggy (execution error or VLM-criticized figure) or non-buggy. Buggy nodes spawn debugging children; non-buggy nodes spawn refinement children. Specialized node types handle hyperparameter sweeps, ablations, replications, and result aggregation.
VLM Figure Reviewer — a Vision-Language Model (Section 3.4) that inspects generated figures during experimentation (flagging issues like missing legends, unclear labels, misleading visualizations) and during manuscript writing (verifying figure-caption alignment, detecting duplicates between main text and appendix).
Manuscript Writer — an LLM-powered module (Section 3.3) that converts experiment summaries, generated figures, and the research idea into a complete LaTeX manuscript, followed by a reflection stage (using reasoning models like o1) that iteratively improves the paper based on VLM figure reviews, LaTeX syntax checking, and page-limit constraints.
Hugging Face Dataset Loader — a standardized interface (Section 3.3) that prompts the system to use datasets.load_dataset for automatic data acquisition, avoiding manual data preparation (with the acknowledged limitation that not all datasets support this method).

Information flows as follows: a broad topic prompt enters the Idea Generator → produces a structured research idea → the Experiment Manager initializes Stage 1 with the idea → Tree Search generates and executes code variants in parallel → VLM reviews figures → the best node advances to the next stage → after all four stages, the Writer consumes experiment summaries + figures + idea → produces LaTeX → VLM reviews figures in context → reflection loop refines the manuscript → final PDF output.

3.3 Roadmap for the Deep Dive

First, the Generalized Idea Generation module (Section 3.1), because it produces the abstract specification that drives all downstream experimentation — understanding what information the system starts with clarifies everything that follows.
Second, the Experiment Progress Manager (Section 3.2.1), since it defines the four-stage structure that constrains and sequences the tree search. Without this, the tree search would be unmoored from scientific methodology.
Third, the Agentic Tree Search mechanism (Section 3.2.2) in full technical detail — node structure, expansion logic, buggy-vs-non-buggy branching, specialized node types, and parallel execution — because this is the paper's primary methodological innovation.
Fourth, the VLM integration (Section 3.4), covering both its role during experimentation (figure quality checking) and during manuscript writing (caption verification, duplication detection), since it represents a critical quality-control feedback loop absent in v1.
Fifth, the Manuscript Writing and Reflection process (Section 3.3, with details from the prompts in Appendix B), to show how experimental results become a peer-reviewable paper.
Sixth, Dataset Loading (Section 3.3) and the hyperparameter/configurations table (Appendix A), to complete the operational picture.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that autonomous scientific discovery requires (1) eliminating human-authored code templates through agentic tree search over code implementations, and (2) structuring exploration through progressive stages that mirror the human scientific workflow (feasibility → tuning → main experiments → ablations).

Generalized Idea Generation

What changes from v1. The AI Scientist-v1 generated research ideas by proposing incremental modifications to an existing codebase — the system's creativity was bounded by the template it was given. The AI Scientist-v2 decouples idea generation from implementation entirely: it first produces a high-level research concept (analogous to a grant proposal abstract), and only later translates that concept into code during the experimentation phase. The authors describe this as "beginning at a higher level of abstraction" (Section 3.1), which means the system can propose ideas that would be structurally impossible to express as simple modifications to a pre-existing template.

The idea generation process. The system is prompted with a broad topic description (for the ICLR workshop, this was the ICBINB theme extracted from the official website, emphasizing negative results and unexpected findings). The LLM (Claude 3.5 Sonnet, temperature 0.5) engages in an iterative refinement loop with literature search:

Initial generation: The LLM produces a structured research proposal containing fields: Name, Title, Short Hypothesis, Related Work, Abstract, Experiments (a list of concrete experimental plans), and Risk Factors and Limitations. The prompt explicitly instructs the model to "propose high-impact research ideas resembling exciting grant proposals" and to "be very creative and think out of the box" (Appendix B).
Literature search integration: The system has access to a SearchSemanticScholar tool. The prompt states: "You should perform at least one literature search before finalizing your idea to ensure it is well-informed by existing research" (Appendix B). This means the LLM can query Semantic Scholar during idea formulation to check novelty and identify prior work — it is not a post-hoc check but an interactive part of ideation.
Reflection round: After generating an initial proposal, the system enters a reflection phase: "In your thoughts, first carefully consider the quality, novelty, and feasibility of the proposal you just created. Include any other factors that you think are important in evaluating the proposal." The LLM then produces a refined version incorporating its own critique and any literature search results. The prompt specifies {current_round}/{num_reflections}, indicating multiple reflection iterations are supported (though exact number is not specified).
Final output: The finalized idea is output as a JSON structure (the IDEA JSON format shown in Appendix B) that serves as the specification document for all subsequent experimentation.

What the output contains. For the workshop-accepted paper, the generated idea (reproduced in Appendix C.1) included:

A title: "Enhancing Compositional Generalization in Neural Networks via Compositional Regularization"
A hypothesis about adding an explicit regularization term to the loss function
An experiments list specifying: implement the regularization in sequence-to-sequence architectures, test on synthetic datasets (SCAN, COGS), apply to real-world tasks (machine translation, semantic parsing), analyze learned representations, conduct ablation studies on regularization strength, compare against meta-learning baselines
Risk factors including: effectiveness may vary across datasets, improper regularization weight could harm primary task performance, additional computational overhead

Design choice: why generate ideas before seeing code. The key architectural decision is the separation of "what to study" from "how to implement it." In v1, these were coupled — the template code implicitly constrained what ideas were possible. In v2, the idea generation phase has no access to implementation code; it operates purely at the conceptual level. This means the system can propose ideas that require novel architectures or experimental designs not present in any template, which is essential for genuine scientific exploration rather than local optimization around a fixed starting point. The tradeoff is that some generated ideas may be infeasible to implement — the experimentation phase must then discover this through failed execution attempts.

The human-in-the-loop boundary at this stage. The authors are transparent that for the workshop submission, they generated approximately 40 ideas (roughly 20 from a core ML prompt and 20 from an applied-domain prompt) and then humans selected three to pursue further (Section 4.2). This selection was based on "alignment with the workshop theme and potential interest." The ideas themselves were not modified — selection was purely curation from AI-generated output. The authors frame this as analogous to "a professor reviewing the work of many students or teams and deciding which work is ready to be submitted for peer review." This human selection step is not required by the system architecture (it could randomly choose ideas or pursue all of them) but was a practical choice to manage computational resources.

Experiment Progress Manager

Why staged experimentation matters. Real scientific research does not jump directly from hypothesis to polished results. It proceeds through distinct phases: first, getting anything to work (feasibility); then, tuning the baseline to be competitive (hyperparameter optimization); then, executing the core research idea (the main experimental contribution); finally, testing which components actually matter (ablation studies). The AI Scientist-v2 bakes this structure into its architecture through an experiment progress manager agent that coordinates four clearly defined stages, each with explicit stopping criteria and a node-promotion mechanism.

Stage 1: Preliminary Investigation. The system's goal is to "establish initial feasibility and correctness through a minimal working prototype based on the generated research idea" (Section 3.2.1). This is the "get something running" phase. The stopping criterion is simple: "Stage 1 concludes when a basic working prototype is successfully executed." This means at least one node must produce a non-buggy result — code that runs without errors and passes VLM figure review. The paper allocates 21 nodes to Stage 1 (Table 3), meaning the system can attempt up to 21 different implementation variants before it must have a working prototype.

Stage 2: Hyperparameter Tuning. The goal is to "refine the initial implementation by optimizing critical hyperparameters (e.g., learning rate, epochs) to create a robust experimental baseline." Stopping criterion: "experiments stabilize, as indicated by convergence in training curves and successful execution across at least two datasets." This is more stringent than Stage 1 — not just "it runs" but "it runs well and consistently." The allocation is 12 nodes (Table 3), reflecting that hyperparameter search is more constrained than initial prototyping.

Stage 3: Research Agenda Execution. This is where the system implements the core scientific contribution — the novel method, the comparative evaluation, the main experimental claims. The stopping criterion is budget-based: "Stage 3 concludes when the allocated computational budget is exhausted" (12 nodes). However, there is an adaptive element: "Stage 3 also includes a check for experiment duration — if runs finish much faster than the pre-allocated runtime, the system suggests increasing the complexity of experiments." This prevents the system from settling for trivial experiments that complete quickly but don't adequately test the hypothesis.

Stage 4: Ablation Studies. The goal is to "systematically assess the importance of various research components, providing rigorous support for the main experimental findings." Again, budget-based stopping (12 nodes). Ablation nodes are a specialized node type (described in Section 3.2.2) that systematically remove or vary components of the experimental design.

The node promotion mechanism. After each stage completes, "the experiment manager selects the best-performing node using a dedicated LLM evaluator based on clearly articulated criteria" (Section 3.2.1). This selected node becomes the root node for the subsequent stage — its code, its learned parameters, its experimental configuration serve as the starting point for the next phase of exploration. This is crucial: the system does not start each stage from scratch; it builds cumulatively on the best result from the previous stage, mirroring how human researchers carry forward their best working implementation.

Replication for statistical rigor. At the conclusion of each stage, the experiment manager "launches multiple replications of the selected best experiments" (Section 3.2.1). These replication nodes (a specialized node type) run the same experiment with different random seeds, producing the mean and standard deviation statistics used in figures and reported results. This is a direct response to a common criticism of automated science — that single-run results may be noisy or non-reproducible.

The per-node runtime constraint. Each individual experiment node has a maximum runtime of 1 hour (Table 3). If an experiment exceeds this limit, it is presumably terminated and marked as buggy (though the exact timeout handling is not described in detail). The total paper generation process "usually takes anywhere from several hours to a maximum of 15 hours, which is the runtime limit we have set" (Appendix A).

Design choice: why four stages rather than continuous exploration. The staged structure imposes a scientific methodology on what would otherwise be undirected code generation. Without stages, the tree search might explore hyperparameter variations before establishing basic feasibility, or jump to ablations before confirming the main result holds. The stages enforce a logical progression: you cannot tune hyperparameters until you have a working prototype; you cannot ablate components until you have a complete experimental pipeline. This is a form of curriculum learning applied to the research process — it constrains the search space in early phases to simpler subgoals, reducing the combinatorial explosion that would result from trying to simultaneously explore architecture, hyperparameters, research ideas, and ablations.

Parallelized Agentic Tree Search

This is the paper's primary technical innovation and the mechanism that replaces v1's linear code modification pipeline.

The fundamental problem tree search solves. When an LLM generates experimental code from a high-level idea, it will frequently produce code that contains bugs (syntax errors, import failures, shape mismatches, logic errors) or that runs but produces poor or misleading results. A linear pipeline that generates code → executes → moves on cannot recover from failures. Tree search enables the system to (a) try multiple implementation approaches in parallel, (b) debug failed attempts using error information, (c) refine working implementations to improve them, and (d) maintain a population of diverse solutions at different quality levels. The metaphor is evolutionary search over code artifacts, where each node is an individual in the population, node expansion produces offspring, and LLM-based evaluation serves as the fitness function.

Node structure. Each node in the tree is a collection of metadata (Section 3.2.2):

An experiment script (a Python file) — the actual executable code
A textual description of the high-level plan implemented in the script
An execution error trace (if applicable) — the Python traceback from a failed run
Experiment runtime — how long the code took to execute
Performance metrics recorded during the experiment (training/validation accuracy, loss curves, etc.)
LLM feedback after running the script — an evaluation of the results
A visualization script and file paths to generated figures
VLM feedback on those figures
The node's final status: either buggy or non-buggy

This rich metadata is what enables the LLM-based selection and expansion logic to make informed decisions — the system doesn't just know that a node succeeded or failed; it knows how it failed (error trace), what results it produced (metrics, figures), and why those results might be problematic (LLM and VLM feedback).

The execution cycle for each node. When a new node is created, it undergoes the following sequence (Section 3.2.2):

LLM generates plan and code: An LLM (Claude 3.5 Sonnet, temperature 0.5, max tokens 8,192 — Table 2) produces both "a concrete experimentation plan and the associated Python code to implement the experiment."
Code execution: "The generated code is immediately executed in a Python interpreter."
Error handling: "If execution encounters an error, the error message is recorded, and the node is marked as buggy, ending the current execution cycle for that node." The error trace becomes part of the node's metadata for downstream debugging.
Successful execution → data saving: "During each experiment, the system is instructed to save all relevant experimental outputs (training and validation metrics, losses, etc.) into structured numpy files." This is a critical design requirement — the system must serialize its results in a format that the plotting and writing phases can later consume.
Plotting phase: "The AI Scientist-v2 reads these stored results and the code, generating visualizations that summarize and illustrate the findings clearly."
VLM review of figures: "These visualizations are subsequently passed to a Vision-Language Model (VLM) for critique. Any issues flagged by the VLM (such as unclear labels, missing legends, or misleading visualizations) result in the node being marked as buggy, and this feedback is recorded for future debugging."
Final classification: "Nodes that successfully execute and pass the VLM review without issue are designated as non-buggy."

This is a dual-gate quality check: the code must run without errors AND the resulting figures must be interpretable and accurate. A node that produces incorrect but error-free code (e.g., plotting the wrong metric, using misleading scales) will be caught by the VLM gate.

The selection and expansion loop. At each iteration of the tree search, the system selects several existing nodes to expand (Section 3.2.2):

Selection probability: "With a predefined probability, a buggy node is chosen (thus prioritizing error resolution and debugging); otherwise, a non-buggy node is selected for further refinement and improvement." The debug probability is set to 1.0 (Table 3) — meaning the system always prioritizes debugging buggy nodes before refining non-buggy ones. This is a depth-first error resolution strategy: don't accumulate a backlog of broken experiments; fix them first.
Selection among non-buggy nodes: When choosing which non-buggy node to refine, "the system uses a best-first search strategy, guided by an LLM that evaluates candidates based on factors like performance metrics, training dynamics, and the quality of generated plots."
Expansion: "The selected nodes are expanded by creating a new child node that may either attempt debugging if the parent node was buggy, or refine and improve upon the previous experiment if the parent was non-buggy. An LLM is used to generate the plan and experiment code for each new child node."
Parallel execution: "All new nodes are executed concurrently in parallel, significantly accelerating the exploration process." This is a key efficiency property — the tree search is not sequential; within a stage, multiple branches can be explored simultaneously, limited only by available compute resources.

The debug depth limit. The system has a maximum debug depth of 3 (Table 3). This means if a node is buggy, its child attempts to debug it; if that child is also buggy, a grandchild attempts debugging; but after three consecutive debugging failures on a lineage, no further debugging children are created. This prevents infinite loops where the system repeatedly fails to fix an unfixable error — after three attempts, that branch is effectively abandoned.

Specialized node types. Beyond the basic buggy/non-buggy classification, the system introduces four specialized node variants for specific experimental needs (Section 3.2.2):

Hyperparameter nodes: Used during Stage 2 (Hyperparameter Tuning). These "systematically explore alternative hyperparameter configurations." The system explicitly tracks "previously tested hyperparameters, preventing redundant experiments." This is a simple form of memory — the system remembers what hyperparameter combinations it has already tried and avoids repetition. "Errors encountered during hyperparameter tuning trigger the creation of corresponding debug nodes."
Ablation nodes: Used during Stage 4 (Ablation Studies). These "evaluate crucial ablation studies... assessing the importance of various components or assumptions underlying the experiment." Like hyperparameter nodes, they track previously tested conditions to avoid redundancy, and spawn debug nodes on error.
Replication nodes: These "execute replicates of their parent experiments using different random seeds." Multiple replication nodes are created to "enable the calculation of statistical measures (mean and standard deviation) of experimental outcomes, enhancing result robustness." These are typically created at the end of each stage to produce the statistics used in final figures.
Aggregation nodes: A special non-experimental node type that does not run new experiments. Instead, aggregation nodes "consolidate and visualize the combined results of replication nodes... simply generate a Python script to aggregate and summarize prior results, producing figures that explicitly show means and standard deviations." These are the bridge between raw experimental data and publishable figures.

The tree search prompt structure. The experiment generation prompt (Appendix B) instructs the LLM: "You are an AI researcher who is looking to publish a paper that will contribute significantly to the field. Your first task is to write a python code to implement a solid baseline based on your research idea provided below, from data preparation to model training, as well as evaluation and visualization. Focus on getting a simple but working implementation first, before any sophisticated improvements. We will explore more advanced variations in later stages."

This prompt embeds an important design principle: progressive complexity. Early nodes should be simple and functional; later nodes can add sophistication. This mirrors how a human researcher would approach a new problem — start with the simplest thing that could possibly work, then iterate.

The plot aggregation process. After all four stages complete, the system runs a dedicated plot aggregation phase (separate from the tree search) that produces the final set of figures for the manuscript. The aggregation prompt (Appendix B) instructs the LLM to "produce ONE Python script that fully aggregates and visualizes the final results for a comprehensive research paper." Key constraints include:

"Only create plots where the data is best presented as a figure and not as a table"
"Each figure can have up to 3 subplots using fig, ax = plt.subplots(1, 3)"
"Use a font size larger than the default for plot labels and titles to ensure they are readable in the final PDF paper"
"Make the plots look professional (if applicable, no top and right spines, dpi of 300, adequate ylim, etc.)"
"Aim to aggregate multiple figures into one plot if suitable"
"Do not hallucinate data. Data must either be loaded from .npy files or copied from the JSON summaries"

This aggregation step is critical because the tree search produces many individual figures from different experimental nodes. The aggregator must select which results to present, combine related plots into multi-panel figures, and ensure visual consistency across the manuscript.

Design choice: why tree search over other exploration strategies. The paper could have used other search paradigms: random search (generate many independent code variants, pick the best), genetic algorithms (crossover between working implementations), or reinforcement learning (train a policy to generate code). Tree search was chosen for several reasons:

Interpretability: Each node has a clear lineage — you can trace exactly which modifications led to which results. This is important for scientific reproducibility and for the system's own self-understanding (the LLM evaluator can see the history of changes).
Error recovery: The buggy/non-buggy branching explicitly separates error resolution from refinement, preventing the system from abandoning promising directions due to fixable bugs.
Staged compatibility: Tree search naturally supports the four-stage structure — each stage starts a new tree rooted at the best node from the previous stage, enabling progressive refinement while maintaining exploration within each stage.
Parallelism: Independent branches can execute simultaneously, which is essential for practical runtime (the entire paper generation takes hours, not days).
Prior art in code generation: The paper explicitly cites AIDE (Jiang et al., 2025) as inspiration, which demonstrated that LLM-based tree search over code achieves state-of-the-art performance on machine learning engineering benchmarks. The AI Scientist-v2 adapts this approach from pure engineering tasks to the more open-ended domain of scientific experimentation.

What the tree search does NOT do. It is important to note what the system explicitly does not attempt:

It does not use reinforcement learning to train a policy for code generation — each node's code is generated from scratch by the LLM conditioned on the parent node's metadata.
It does not perform gradient-based optimization of code — all search is discrete, at the level of complete code files.
It does not maintain a formal belief state or uncertainty estimate about which branches are most promising — the LLM evaluator makes qualitative judgments rather than computing explicit value functions.

Vision-Language Model Integration

Where VLMs are used. The AI Scientist-v2 incorporates Vision-Language Models at two distinct phases of the workflow (Section 3.4):

Phase 1: During experimentation (tree search). After each successful experiment node generates figures, those figures are passed to a VLM (GPT-4o, temperature 0.5, max tokens 8,192 — Table 2) for immediate critique. The VLM checks for issues like "unclear labels, missing legends, or misleading visualizations" (Section 3.2.2). If the VLM flags issues, the node is marked as buggy, and the feedback is recorded in the node's metadata. This means the tree search treats visual quality as a correctness criterion — a figure with a missing legend is treated the same as code that crashes with a syntax error. Both prevent the node from being classified as non-buggy.

Phase 2: During manuscript writing (reflection stage). After the initial LaTeX manuscript is generated, the system extracts screenshots of each figure alongside its caption and the surrounding text that references it (identified by the keyword "Figure X"). These image-text-caption triplets are passed to the VLM, which performs multiple quality checks (Section 3.4):

Verifying alignment between figures and captions: does the caption accurately describe what the figure shows?
Identifying visual clarity issues: missing legends, unclear labels, illegible text
Detecting potential duplication: are there figures in the main text and appendix that convey the same information?

The VLM review output (Appendix B) is structured as JSON with fields: "Img_description" (detailed scientific description of the figure contents), "Img_review" (analysis of the figure itself and suggestions), "Caption_review" (assessment of caption accuracy and quality), "Figrefs_review" (evaluation of whether the main text adequately discusses the figure).

The reflection loop. The writeup reflection prompt (Appendix B) feeds VLM reviews back into the LLM with instructions to fix identified issues: "Ensure each caption clearly matches its image content and that there is substantial discussion of each figure in the text." It also addresses duplication: "Duplicate figures between main text and appendix. Make sure to remove the duplicate figures from the appendix." There is a separate VLM reflection prompt that focuses specifically on figure selection and placement: "For figures that do not add significant value to the paper, move them to the appendix. For figures that are not very informative or do not effectively communicate meaningful patterns, remove them entirely. For figures that do not contain subfigures and present sparse information, consider combining them with other related figures."

The model used. All VLM tasks use GPT-4o (Table 2) — the same model used for LLM feedback agents and the summary report agent, but distinct from the Claude 3.5 Sonnet used for code generation.

Design choice: why VLM review rather than programmatic figure checking. The issues the VLM catches are semantic rather than syntactic. A program could check that axis labels exist, but it cannot determine whether the labels are correct or whether the caption accurately describes the plotted data. The VLM brings human-like visual understanding to the quality control process. However, this introduces a dependency on the VLM's own reliability — a VLM could hallucinate issues in a correct figure or miss genuine problems. The paper does not evaluate the VLM's accuracy as a reviewer (i.e., what fraction of its flagged issues are genuine vs. false positives).

Manuscript Writing and Reflection

Architecture change from v1. The AI Scientist-v1 used an incremental, Aider-based iterative writing approach (Gauthier, 2024), where the manuscript was built up through many small edits. The AI Scientist-v2 replaces this with a simpler two-phase process (Section 3): (1) single-pass generation of the complete LaTeX manuscript, followed by (2) a separate reflection stage powered by reasoning models.

The writeup prompt structure. The manuscript generation prompt (Appendix B) is extensive and embeds substantial domain knowledge about scientific writing conventions. Key elements:

Page limit awareness: "The main paper is limited to 4 pages in single-column format, not counting references... DO NOT USE MORE THAN 4 PAGES FOR THE MAIN TEXT."
Section-by-section guidance: For each section of the paper (Title, Abstract, Introduction, Related Work, Background, Method, Experiments, Conclusion, Appendix), the prompt provides specific instructions about what to include. For example, for the Introduction: "Overview of the issue or challenge being explored. Clearly state why this problem is important, especially for practical or real-world contexts."
Stylistic constraints: "MINIMIZE THE USAGE OF ITEMIZE OR ENUMERATE. ONLY USE THEM IF THEY ARE ABSOLUTELY NECESSARY AND CONTAIN SUBSTANTIAL INFORMATION."
Figure management: "Include up to 4 figures in the main text. All other figures should be in the appendix."
Citation requirements: "Each section (especially Related Work) should have multiple citations." "Do not hallucinate new citations or any results not in the logs."

Input to the writeup phase. The LLM receives:

The research idea (the JSON from the idea generation phase)
Experiment summaries in JSON format (containing figure descriptions, code to generate figures, paths to .npy files)
The aggregator script (to see how plots were generated and what legend names were used)
A list of available plot filenames
VLM-based figure descriptions
The current LaTeX progress (for iterative refinement)

The reflection stage. After generating the initial manuscript, the system enters a reflection loop using reasoning models (the paper mentions o1 from OpenAI, 2024). The reflection prompt (Appendix B) checks for:

LaTeX syntax errors: Identified via chktex output and manually specified common mistakes ("LaTeX syntax errors (unenclosed math, unmatched braces, etc.), duplicate figure labels or references, unescaped special characters: & %")
Scientific rigor: "Is the writing clear, and scientifically rigorous for a workshop focusing on real-world pitfalls?"
Completeness: "Have we included all relevant details from the summaries without hallucinating?"
Figure usage: The prompt lists figures available but unused, figures referenced but missing, and VLM reviews of figure-caption-text alignment. It also explicitly checks for "duplicate figures between main text and appendix."
Supplementary material: "Can we use more information and details (hyperparameters, unused figures, etc.) in the supplementary material? Only add information that is not already covered in the main paper."
Page constraints: The system is prompted with the target page length and the current PDF length, and instructed to adjust accordingly.

The VLM-based figure selection reflection. A separate reflection step (Appendix B, "VLM Reflection Prompt") focuses specifically on figure curation: moving low-value figures to the appendix, removing uninformative figures entirely, combining sparse single-panel figures into multi-panel figures, and "enhancing the scientific analysis of the remaining figures in the text." The prompt instructs: "Be more aggressive with figure selection" if the page limit is exceeded.

Design choice: why single-pass generation + reflection over incremental editing. The authors do not provide an explicit ablation comparing the two approaches, but the motivation can be inferred: incremental editing (making many small changes to a document) can lead to incoherence, where later edits fix local issues but break global structure. Single-pass generation produces a coherent first draft, and the reflection stage makes targeted improvements while preserving overall coherence. The use of reasoning models (o1) for reflection is notable — these models are specifically designed for multi-step reasoning and self-critique, making them well-suited for identifying issues in generated text.

Dataset Loading via Hugging Face

The problem this solves. Empirical machine learning research requires datasets. In v1, datasets were handled through human-authored template code. In v2, the system must autonomously acquire the data it needs.

The mechanism. "In The AI Scientist-v2, we prompt the system to leverage Hugging Face Hub whenever possible, automatically downloading required datasets using the standard one-line function (datasets.load_dataset)" (Section 3.3). This is a practical engineering choice: Hugging Face Hub provides a unified API for thousands of datasets with predefined train/validation/test splits, meaning the system doesn't need to write custom data loading code for each new project.

The acknowledged limitation. "While this standardized approach greatly simplifies dataset handling, we acknowledge it is somewhat ad-hoc, as not all dataset repositories support this method." For the pest detection paper (one of the three submissions), the authors manually downloaded a dataset from Kaggle and reduced its size to one-tenth because the system could not automatically access it (Appendix C.3.1). This is explicitly noted as a limitation and an area for future improvement: "Automating this data preparation process further to handle a broader range of datasets remains an interesting direction for future research."

Hyperparameters and Model Configurations

LLM configurations (Table 2). The system uses different models for different tasks:

Code Generation (the primary experimental code writing): Claude 3.5 Sonnet (v2), max tokens 8,192, temperature 0.5
LLM/VLM Feedback Agents (evaluating experiments, reviewing figures): GPT-4o, max tokens 8,192, temperature 0.5
Summary Report Agent (high-level summarization): GPT-4o, max tokens 8,192, temperature 1.0

The temperature choices reflect the task requirements: code generation and evaluation use temperature 0.5 (moderate creativity with reasonable determinism), while summarization uses temperature 1.0 (more creative freedom).

Tree search hyperparameters (Table 3). These values govern the scale and depth of exploration:

Debug Probability: 1.0 (always prioritize debugging buggy nodes over refining non-buggy nodes)
Maximum Debug Depth: 3 (abandon a lineage after 3 consecutive failed debugging attempts)
Maximum Experiment Runtime per Node: 1 hour (longer experiments are presumably terminated)
Node Allocation per Stage: Stage 1: 21 nodes, Stage 2: 12 nodes, Stage 3: 12 nodes, Stage 4: 12 nodes

The decreasing allocation from Stage 1 (21 nodes) to later stages (12 nodes each) reflects the design philosophy: initial prototyping requires more exploration to find a working implementation; later stages operate from a known-good starting point and can be more targeted.

Total runtime. "Based on our experience, this process usually takes anywhere from several hours to a maximum of 15 hours, which is the runtime limit we have set" (Appendix A). At 21 + 12 + 12 + 12 = 57 total nodes with a 1-hour per-node limit, the theoretical maximum sequential runtime is 57 hours, but parallel execution (multiple nodes running concurrently) and the fact that most experiments complete well under 1 hour bring the typical wall-clock time into the 6–15 hour range.

4. Key Insights and Innovations

Innovation 1: Peer Review as an External Validity Check for Automated Science

The most distinctive intellectual move in this paper is not a technical algorithm but an evaluation methodology: submitting fully AI-generated manuscripts to a real, blind peer-review process and treating acceptance as the success criterion. This transforms the evaluation of automated science systems from internal self-assessment (where authors judge their own system's output quality) to external, community-validated assessment using the same mechanism that governs human scientific publication.

What the field did before. Prior automated science systems — including The AI Scientist-v1 (Lu et al., 2024), AI-Researcher (Data Intelligence Lab, 2025), and CycleResearcher (Weng et al., 2025) — evaluated their outputs through author inspection, automated metrics, or comparison to known benchmarks. Even when human evaluation was used (e.g., Si et al., 2025 had human experts rate LLM-generated ideas on novelty and feasibility), it was conducted in a controlled study context, not through the actual gatekeeping mechanism of scientific publication. The implicit assumption was that if authors judged the output to be reasonable, the system was working.

Why this changes the conversation. Peer review is a adversarial evaluation: reviewers are incentivized to find flaws, are anonymous (reducing social desirability bias), and represent genuine domain expertise. An AI-generated paper that survives this process has passed a higher bar than one that merely looks plausible to its creators. The paper's finding — that one of three submissions received scores of 6, 7, and 6 (averaging 6.33/10) and would have been accepted — is significant precisely because it was an uncurated, blind process where reviewers were told some submissions might be AI-generated but didn't know which ones. The fact that the accepted paper investigated a negative result (compositional regularization doesn't help) and was praised by reviewers for "clearly presenting why the regularization term does not yield the anticipated improvements" (Section 4.2, Reviewer #1) is especially telling: the system produced work that human reviewers found scientifically honest and informative, not merely superficially impressive.

A methodological contribution, not a performance claim. The paper is careful to frame this as a capability demonstration with sharp boundaries, not a claim of consistent competence. Two of three submissions were rejected. The acceptance was at a workshop (60–80% acceptance rates vs. 20–30% for top conferences). The authors explicitly acknowledge that "the current version of The AI Scientist-v2 does not yet consistently reach the rigorous standard required for top-tier conference publications" (Section 5). The innovation is establishing peer review as a meaningful unit of evaluation for automated science systems — a benchmark that is harder to game than self-assessment and directly measures the outcome these systems are designed to achieve.

A secondary architectural insight. The peer-review evaluation also serves as an ablation of the human role. The paper transparently documents that humans performed meta-selection (choosing which AI-generated ideas to pursue and which complete manuscript to submit from multiple seeded runs), but that within any single run, no human touched the code, results, figures, or text. This granular accounting of the human-AI boundary — what the authors call "meta-selection from fully autonomous outputs, not human-in-the-loop intervention in the scientific content generation" (Section 4.2) — provides a template for how future systems should report their level of autonomy. It prevents both overclaiming (by hiding human intervention) and underclaiming (by failing to distinguish content generation from process management).

Innovation 2: The Experiment Manager as an Enforcer of Scientific Methodology

The paper's most understated but architecturally crucial contribution is the experiment progress manager — an agent that imposes a four-stage scientific workflow (feasibility → tuning → main experiments → ablations) on what would otherwise be undirected code generation. This is not merely an engineering convenience; it represents a curriculum over scientific subgoals that constrains the combinatorially explosive space of possible experiments into a manageable sequence where each stage builds on the verified output of the previous one.

What the field did before. Most LLM-based code generation systems (including AIDE by Jiang et al., 2025, which inspired the tree search) treat the problem as flat optimization: generate code variants, evaluate them, keep the best. The AI Scientist-v1 used a linear sequence of code modifications with no explicit staging — each experiment built on the previous one, but there was no structural distinction between "get it working," "make it good," "test the hypothesis," and "verify it's real." Meta-learning approaches to scientific discovery (Finn et al., 2017) and open-ended search paradigms (Clune, 2019; Mouret and Clune, 2015) focus on exploration strategies but don't encode the logical dependencies between experimental phases — you literally cannot ablate components you haven't yet built, and hyperparameter tuning before feasibility is wasted computation.

Why this is a conceptual contribution, not just engineering. The stage structure solves a credit assignment problem in automated research. When an AI system produces a paper, how do you know whether the final result is genuine or an artifact of lucky exploration? By enforcing that each stage's best node seeds the next stage, the system creates a chain of custody for experimental results: the ablation studies in Stage 4 are conducted on a pipeline that survived feasibility testing (Stage 1), was tuned to be competitive (Stage 2), and successfully executed the research agenda (Stage 3). A reviewer can trace the provenance of results through this staged lineage. This mirrors how human scientists build confidence: you trust an ablation study more if you know the baseline it ablates was carefully tuned, and you trust the tuning more if you know the underlying implementation is correct.

The adaptive element adds scientific judgment. Stage 3's check — "if runs finish much faster than the pre-allocated runtime, the system suggests increasing the complexity of experiments" — is a primitive form of scientific ambition calibration. It prevents the system from settling for trivial experiments that complete quickly but don't adequately test the hypothesis. This is not a generic optimization heuristic; it encodes a specifically scientific value: experiments should be as informative as the compute budget allows, not merely as simple as possible.

Significance beyond this paper. The staged manager is a reusable architectural pattern for any AI system that must conduct empirical investigations. The specific stages (feasibility, tuning, agenda, ablation) are motivated by machine learning experimentation, but the principle — decompose the research process into phases with explicit stopping criteria and node-promotion logic — generalizes to any domain where experiments have logical prerequisites. The fact that the paper provides explicit stopping criteria for each stage (Stage 1 ends when code runs successfully; Stage 2 ends when training stabilizes across multiple datasets; Stages 3–4 end when budgets are exhausted) makes this pattern implementable and falsifiable rather than merely aspirational.

Innovation 3: Visual Correctness as a First-Class Objective in Automated Experimentation

By integrating Vision-Language Model review directly into the tree search's success criteria — a node is marked as buggy if its figures have unclear labels, missing legends, or misleading visualizations — the paper elevates visual communication quality from a post-hoc polish step to a correctness criterion on par with code execution. This is a subtle but important reframing of what it means for an automated experiment to "succeed."

What the field did before. In The AI Scientist-v1, figure generation was a downstream step after experimentation — code ran, data was saved, figures were plotted. If the figures were misleading or uninformative, there was no feedback loop to correct them during experimentation; the issue would only be caught (if at all) during the writing phase. More broadly, the automated ML literature (AutoML, neural architecture search, learned optimizers) treats figures as outputs for human consumption, not as objects that should be optimized or verified. The dominant assumption is that if the data is correct, the figure is correct — but this conflates data accuracy with visual communication accuracy. A plot that uses the wrong y-axis scale, omits a legend, or presents data in a misleading way can be factually correct (the underlying numbers are right) while being scientifically harmful (readers draw wrong conclusions).

Why this matters. The VLM gate creates a tight feedback loop between experimentation and visualization. A node that produces correct numerical results but generates an uninterpretable figure is treated identically to a node whose code crashes — both are buggy and spawn debugging children. This means the system cannot "succeed" at experimentation while failing at communication. The VLM feedback also becomes part of the node's metadata, which means debugging children can access specific critiques ("the legend is missing," "the x-axis label is unclear") when generating improved versions. This is an instance of multi-modal self-supervision: the text-generating LLM that writes code receives visual feedback from a VLM about the outputs of that code, enabling it to improve aspects of the code that affect visual quality.

The writing-phase VLM review adds a separate verification layer. During manuscript generation, figures are re-reviewed in context — alongside their captions and the surrounding text that references them. This catches a different class of errors: captions that misdescribe figures, text that draws conclusions unsupported by the plotted data, and duplicate figures between main text and appendix. The paper's own internal review found multiple instances of such errors in the AI-generated papers (Section 4.2: "the caption of Figure 3 incorrectly interpreted validation loss," "Figure 5's attention-based model clearly outperformed the LSTM model, contradicting the authors' claims"), indicating that figure-text misalignment is a real failure mode that the VLM review partially — but not completely — mitigates.

A limitation that reveals a deeper challenge. The VLM review catches visual issues but not conceptual ones. The paper notes that the AI-generated manuscript's Figure 5 caption "seems to be strongly influenced by the conclusion in the main text" — the caption claims the attention-augmented LSTM performs similarly to the baseline, when the actual plot shows it substantially outperforms (Appendix C.1). This is a narrative override failure: the text's conclusion contaminated the figure interpretation, and the VLM review (which checks figure-caption alignment) didn't catch it because the caption was internally consistent with the (wrong) conclusion. Solving this requires not just visual verification but cross-modal consistency checking between experimental data, figures, and textual claims — a harder problem the paper doesn't solve but effectively diagnoses.

Innovation 4: Template-Free Code Generation as a Scientific Autonomy Threshold

The elimination of human-authored code templates — the feature that gives the paper its subtitle "Workshop-Level Automated Scientific Discovery" — is more than an engineering upgrade from v1. It represents crossing a qualitative threshold in scientific autonomy: the system transitions from exploring variations on human-specified experimental designs to designing experiments from abstract specifications. This is not merely "better code generation"; it's a fundamentally different relationship between the AI and the scientific problem.

What the field did before. The AI Scientist-v1, AI-Researcher, and most LLM-based research assistants operate in a paradigm where a human provides a working codebase (or detailed specification) and the AI proposes modifications. This is scientifically useful — it can discover hyperparameters, architectural tweaks, or data augmentations that humans might miss — but it cannot change the experimental paradigm. If the template implements a CNN, the AI cannot decide that a transformer would be more appropriate and implement one from scratch. The template is both a starting point and a conceptual prison — the space of possible experiments is the set of local variations on the template's structure.

Why removing templates changes the scientific capability. In The AI Scientist-v2, the idea generation phase produces an abstract specification (e.g., "introduce a compositional regularization term into the loss function of sequence-to-sequence architectures") and the experimentation phase must translate this into working code without a reference implementation. This means the system can propose experimental designs that have no precedent in its codebase. The paper demonstrates this concretely: the accepted workshop paper implements an LSTM with a novel regularization term, attention-augmented variants, and synthetic arithmetic datasets — none of which existed as templates. If the system decided that a transformer architecture was more appropriate than an LSTM, it could (in principle) implement one, because it's generating code from the abstract idea, not modifying existing code.

The coupling with tree search is what makes this viable. Template-free code generation from LLMs is not novel in itself — LLMs can generate code from natural language descriptions. What's novel is making this reliable enough for scientific experimentation, where bugs don't just mean a failed run but potentially incorrect scientific conclusions. The tree search mechanism makes template-free generation viable by providing a safety net: when the generated code fails (as it frequently does — the system explicitly prioritizes debugging buggy nodes with probability 1.0), the error trace feeds back into the next generation attempt. Over multiple expansion rounds, the system explores the space of implementations until it finds one that works. This converts code generation from a one-shot gamble into an iterative search process with error recovery.

The residual human role defines the current frontier. Despite removing template dependency, the system still relies on humans for: (1) selecting which AI-generated ideas to pursue, (2) manually downloading non-standard datasets (the pest detection case), and (3) selecting the best complete manuscript from multiple seeded runs. These are not template dependencies but they are scaffolding dependencies — the system needs humans to make high-level judgments about research direction and to handle data acquisition edge cases. The paper's framing makes clear that template-free generation is a necessary but not sufficient condition for full autonomy, and the remaining dependencies define the agenda for v3.

5. Experimental Analysis

Evaluation Methodology

Dataset and Task Selection. The system generates its own datasets and experimental materials for each research idea; there is no fixed benchmark. For the workshop-accepted paper, the system autonomously created synthetic arithmetic expression datasets (1,000 training samples, 200 test samples, operators and digits drawn from defined ranges — Appendix C.1). For the other two submissions, it used MNIST, Fashion-MNIST, CIFAR-10, SVHN, EuroSAT, MedMNIST, and Crop Pest and Disease datasets accessed through Hugging Face Hub or manual download. The choice of what data to use is part of the system's autonomous decision-making, not a pre-specified evaluation protocol.
Base Models. The system uses two distinct LLM families for different roles (Table 2): Claude 3.5 Sonnet (v2) for all code generation tasks (temperature 0.5, max tokens 8,192) and GPT-4o for feedback/evaluation agents and VLM figure review (temperature 0.5 for feedback, 1.0 for summarization). The manuscript reflection stage uses reasoning models (OpenAI o1). These model choices are not ablated — the paper does not investigate whether alternative models would produce different-quality papers.
Metrics. The paper does not evaluate its own performance using quantitative metrics. Instead, the primary evaluation is external: peer-review scores, acceptance/rejection decisions, and qualitative reviewer feedback. The internal evaluation (Section 4.2 and Appendix C) consists of the authors' structured assessment of each AI-generated manuscript's strengths and weaknesses across dimensions like experimental rigor, presentation clarity, methodological soundness, and novelty. The paper provides no automated metric (e.g., pass rate, average score, mean time to acceptance) for the system's output quality beyond the three-submission trial.
Baselines. The comparison is against The AI Scientist-v1 (Lu et al., 2024) — the predecessor system that relied on human-authored code templates, used linear experimentation, and lacked VLM figure review. The comparison is qualitative (Table 1) rather than quantitative: v1 is characterized as "topic-specific," "linear," no parallel execution, no VLM integration, and "not submitted" to peer review; v2 as "domain-general," "tree-based," parallel, VLM-integrated, and "workshop acceptance-worthy." No other automated science systems (AI-Researcher, CycleResearcher, Agent Laboratory) are run as experimental baselines on the same tasks.
Generation Budget / Compute Accounting. The budget is measured in tree-search nodes: Stage 1 allocates 21 nodes, Stages 2–4 allocate 12 nodes each, for a total of 57 experimental nodes per complete run (Table 3). Each node has a maximum runtime of 1 hour. With parallel execution (multiple nodes run concurrently), the total wall-clock time per paper is "anywhere from several hours to a maximum of 15 hours" (Appendix A). The paper does not report total FLOPs, GPU-hours, or token consumption. For the workshop submission, the system was run multiple times with different random seeds per idea, and the best complete manuscript was selected — this selection cost (the number of discarded seeded runs) is not quantified.
Cross-Validation / Statistical Protocol. There is no cross-validation or train/test split for evaluating the system itself. The statistical rigor applies at two levels: (1) within each generated paper, replication nodes (using different random seeds) produce means and standard deviations for reported results, and (2) the three-submission trial provides a small-sample estimate of the system's peer-review success rate. The paper explicitly states that the study "aims to see whether The AI Scientist-v2 can produce at least one paper that survives peer review, and not what fraction of the time it can do so" (Section 4.2), acknowledging that three submissions are insufficient to estimate a success probability.

Main Quantitative Results

The paper's "quantitative results" are primarily qualitative outcomes of a real-world peer review process, supplemented by internal analysis of each generated manuscript.

Peer Review Outcomes

The central empirical result appears in Section 4.1: three fully autonomous manuscripts were submitted to the ICBINB workshop at ICLR 2025 (43 total submissions). Blind peer review produced:

Accepted manuscript: "Compositional Regularization: Unexpected Obstacles in Enhancing Neural Network Generalization" received scores of 6 (weak accept), 7 (accept), and 6 (weak accept), averaging 6.33/10. The paper states this score "placed it roughly in the top 45% of submissions" (Section 1) and that it "would have been accepted after meta-review were it human-generated" (Section 1).
Rejected manuscript #1: "Unveiling the Impact of Label Noise on Model Calibration in Deep Learning" received scores from workshop reviewers of 3 (clear rejection) and 3 (clear rejection) (Appendix C.2.4). Reviewer #1 noted missing references ("(?)"), unlinked references, unreadable figures, and described the paper as "in a late draft status and not review ready" (Reviewer #2).
Rejected manuscript #2: "Real-world Challenges in Pest Detection using Deep Learning: an Investigation into Failures and Solutions" received scores from workshop reviewers of 3 (clear rejection), 7 (accept), and 4 (rejection) (Appendix C.3.4). Reviewer #3 identified missing BibTeX entries ("?"), called for a "solid theoretical foundation," and noted that ResNet-18 is "a rather old model" without justification.

Result: 1 acceptance out of 3 submissions (33% acceptance rate). The paper does not claim this as a reliable success rate — it explicitly frames the experiment as a demonstration that acceptance is possible, not an estimate of how likely it is.

Internal Evaluation Findings

The authors conducted detailed reviews of all three AI-generated manuscripts (Section 4.2, Appendix C.1.1, C.2.2, C.3.2), identifying systematic issues:

For the accepted manuscript (Appendix C.1.1, "AI Scientist Team Review"):

Soundness: 3/5 — "Interesting idea with targeted experiments"
Presentation: 2/5 — "Citations, imprecise description, too confident interpretation"
Contribution: 3/5 — "Regularizer, analysis, ablations"
Overall Workshop Score: 5/10 (borderline accept)
Specific issues identified: vague regularization description (unclear whether applied to embedding hidden states or LSTM hidden states); missing the Hochreiter & Schmidhuber (1997) citation for LSTM; caption of Figure 3 incorrectly interprets validation loss; Figure 5's attention-augmented LSTM clearly outperforms baseline but text claims parity

For the rejected manuscripts (Appendices C.2.2, C.3.2):

Label noise paper: Soundness 2/5, Presentation 1/5, Contribution 1/5 — "Wrong description and duplication of figures. Missing citation and downplaying of related work." The internal review identified that the paper "mentions results that are not substantiated by actual experiments," including claims about temperature scaling and reliability diagrams that were not implemented.
Pest detection paper: Soundness 2/5, Presentation 1/5, Contribution 1/5 — "Wrong description and duplication of figures." Internal review noted that "domain adaptation" experiments only used data augmentation, and the motivation for EuroSAT/MedMNIST/CIFAR-10 results transferring to pest detection is unclear.

Code review findings (Appendices C.1.2, C.2.3, C.3.3):

Accepted paper: Dataset overlap issue — approximately 57% of test samples overlap with training data (synthetic data generation with limited operator-digit combinations). The attention-augmented LSTM achieving 100% test accuracy was revealed as an artifact of task simplicity (single-digit arithmetic); when tested with numbers 10–19, accuracy dropped to 56%.
Label noise paper: Temperature scaling was implemented in code but "never actually used" in experiments (Figure 8 in Appendix C.2.3). The asymmetric noise implementation always maps class i to (i+1) % num_classes, which is one valid approach but not described in the paper.
Pest detection paper: Multi-dataset training with a domain discriminator was attempted but ran unsuccessfully; the selected code instead trained separate models on individual datasets, making the paper's "multi-dataset training" claims misleading.

Ablation Studies and Robustness Checks

The paper does not conduct traditional ablations of its own architecture (e.g., "what if we remove the VLM review?" or "what if we use only 3 stages instead of 4?"). The "ablations" described in the paper (Section 3.2.1, Stage 4) are experiments that the system runs on its research hypotheses, not ablations of the AI Scientist-v2 system itself. However, several implicit robustness checks can be extracted from the experimental design:

Idea diversity robustness: The system generated approximately 40 ideas across two domains (core ML and applied/real-world), from which three were selected. The fact that the accepted paper came from the core ML batch and the pest detection paper from the applied batch (while still being rejected) suggests the system can generate ideas across domains, but provides no quantitative comparison of quality across domains.
Multiple seeded runs: Each idea was run "multiple times, each initiated with a different random seed" (Section 4.2), and the best complete manuscript was selected. This provides a form of reliability check — the system doesn't always produce the same output for the same idea — but the number of runs, variance in quality across seeds, and selection criteria are not quantified.
Predicted vs. actual paper quality: The authors' internal assessment of the accepted paper (5/10 workshop score) was lower than the actual peer-review outcome (6.33/10), suggesting that author self-assessment may be more conservative than blind review — or that the specific reviewers assigned were favorable. This is not systematically analyzed.
VLM review effectiveness: The VLM-based duplication checker for figures between main text and appendix was imperfect — Appendix C.2.1 shows that Figure 4 in the label noise paper's supplementary material was a duplicate of Figure 1 in the main text, which the VLM "overlooked or the writeup phase failed to account for" (annotation in Appendix C.2.1). This reveals a reliability gap in the VLM quality control loop.
Dataset coverage robustness: The three submissions used three distinct dataset types (synthetic arithmetic, standard vision benchmarks, real-world agricultural images), providing weak evidence that the system generalizes across data modalities. However, all three are supervised classification/regression tasks, limiting the demonstrated scope.

Critical Assessment

The paper makes three central claims that require experimental support:

Claim 1: "The AI Scientist-v2 produces the first entirely AI-generated peer-review-accepted workshop paper." This claim is directly supported by the evidence. One manuscript received scores (6, 7, 6) that exceeded the workshop's acceptance threshold, and the paper transparently documents the human meta-selection involved (idea curation, seed selection, dataset downloading for the pest detection paper). The claim is precise about scope: workshop-level acceptance, not conference-level. No external verification beyond the workshop's own peer review process is required, and none is possible — the review was blind and the scores are documented.

What this claim does NOT establish: that the system can reliably produce such papers (only 1 of 3 succeeded), that it can do so without human meta-selection (the paper explicitly documents this dependency), or that the accepted paper would survive post-acceptance scrutiny (the internal code review found a 57% train-test overlap that reviewers didn't detect). The claim is narrow and the evidence matches it.

Claim 2: "The system eliminates reliance on human-authored code templates." This claim is supported with qualifications. The paper provides no direct ablation comparing template-based vs. template-free generation on the same tasks — we cannot see, for example, whether the accepted paper's quality would have been higher or generated faster with a human template as starting point. However, the three generated papers implement experimental code (LSTM with custom regularization, ResNet-18 on vision benchmarks, attention-augmented variants) that did not exist as pre-written templates, and the code reviews (Appendices C.1.2, C.2.3, C.3.3) confirm that the implementations are functional and non-trivial. The qualification comes from the pest detection paper, where humans manually downloaded and reduced the dataset — the system still requires human scaffolding for non-standard data acquisition, which is a form of template dependency by another name.

Claim 3: "Agentic tree search enables deeper exploration than linear experimentation." This claim is not experimentally validated within this paper. There is no head-to-head comparison of tree search vs. linear experimentation on the same research ideas. The paper cites observed "short-sighted experimentation" in v1 (Section 2) but provides no quantitative before/after comparison. The claim is primarily supported by architectural description and analogy to human scientific practice, not by experimental evidence. A convincing validation would require: (1) running the same research idea through both v1's linear pipeline and v2's tree search, (2) comparing the depth/completeness of the resulting papers, and (3) showing that tree search produces qualitatively different (better) experimental designs that linear search cannot reach. None of this is done.

Genuine weaknesses in the experimental design:

Sample size of one. The core empirical contribution is a single acceptance out of three attempts. This demonstrates possibility but provides no statistical information about probability. A reader cannot estimate whether the system's true workshop acceptance rate is 5%, 30%, or 50%. The paper is transparent about this limitation, but it constrains the strength of any "the system works" conclusion.
No automated metric for self-evaluation. The paper evaluates the system by its output's peer review scores but provides no internal metric that predicts those scores. This makes systematic improvement difficult — to test whether a change to the tree search algorithm improves paper quality, one would need to submit batches of papers to peer review, which is expensive and slow. The absence of a proxy metric (e.g., human-evaluated rubric scores on a held-out set of generated papers) is a significant gap for anyone trying to build on this work.
Confounded evaluation. The three submitted papers differed not only in the system's performance but also in the underlying research ideas. The accepted paper investigated a compositional regularization hypothesis; the rejected papers investigated label noise calibration and pest detection. We cannot distinguish whether the acceptance resulted from a better idea (which humans selected from the pool) or better execution of the idea (which the system performed). This confounding is inherent to the evaluation design — it's a property of testing end-to-end systems rather than controlled components.
The human meta-selection pipeline is not ablating. The paper describes human involvement in idea selection, dataset preparation, and final manuscript selection but never tests how critical these steps are. Would the system still produce an acceptance-worthy paper if it randomly selected an idea from its 40 proposals? If it submitted the first manuscript generated rather than the best of multiple seeded runs? If it were required to acquire all datasets autonomously? These are crucial questions about the system's true autonomy level, and they are not experimentally addressed.
Missing baselines. The comparison in Table 1 is only against The AI Scientist-v1. No other automated science system (AI-Researcher, CycleResearcher, Agent Laboratory, or even a simple baseline like "ask GPT-4o to write a paper directly") is evaluated. This makes it impossible to assess whether the specific architectural innovations (tree search, staged experimentation, VLM review) are necessary for workshop-level acceptance or whether simpler approaches might achieve similar results.
Code-level errors that peer review missed. The internal code review (Appendix C.1.2) found a 57% train-test overlap in the accepted paper's synthetic dataset — a fundamental flaw that could invalidate the paper's conclusions about generalization. The human reviewers did not detect this (they had no access to code), and the AI Scientist-v2 did not detect it either (the VLM review checks figures, not data generation correctness). This reveals a blind spot in both the system's quality control and the workshop peer review process, and it means that "passing peer review" is not equivalent to "producing scientifically sound work" — a distinction the paper acknowledges but does not experimentally quantify.

Experiments that would have strengthened the paper:

An ablation where the VLM figure review is removed during tree search, comparing paper quality with and without this feedback loop
A comparison of tree search vs. linear experimentation on identical research ideas, measuring metrics like number of distinct experimental conditions tested, depth of ablation studies, and external review scores
Running the full pipeline multiple times on the same idea (not just different seeds) to measure output variance and estimate how often the system produces submission-worthy work from a fixed starting point
Automated evaluation of generated code for common flaws (train-test leakage, incorrect metric computation) as a complement to VLM figure review
A human baseline: what score would a human graduate student achieve on the same task (generate a 4-page workshop paper on a given idea, with similar compute budget)? Without this, we cannot interpret whether 6.33/10 is impressive for an AI or well below human capability for the same task.

Where the claims hold conditionally:

The central claim — that the system can produce workshop-acceptable research — holds under specific conditions that the paper documents but does not experimentally verify as necessary or sufficient: (1) humans filter the AI's ideas from a large pool, (2) humans handle non-standard data acquisition, (3) humans run the system with multiple seeds and select the best output, (4) the target venue is a workshop with 60–80% acceptance rates, and (5) the research domain is one where the system's base models (Claude 3.5 Sonnet, GPT-4o) have sufficient knowledge to generate competent code and text. Whether the claim generalizes to other venues, other domains, or reduced human involvement is unknown and not tested.

6. Limitations and Trade-offs

6.1 The System Requires Non-Trivial Human Scaffolding for Idea Selection, Dataset Acquisition, and Output Curation

The assumption or constraint. The AI Scientist-v2 is described as an "end-to-end agentic system" (Section 1), but a careful reading of Section 4.2 reveals three categories of human intervention that the system's architecture does not eliminate: (1) humans selected 3 ideas to pursue from an AI-generated pool of approximately 40 proposals, (2) for the pest detection paper, humans manually downloaded and reduced a Kaggle dataset to one-tenth its original size because the system could not automatically access it (Appendix C.3.1), and (3) each idea was run "multiple times, each initiated with a different random seed," and humans "selected the single best-resulting manuscript for submission based on a careful inspection of its overall coherence and scientific quality" (Section 4.2). The authors characterize this as "meta-selection from fully autonomous outputs, not human-in-the-loop intervention in the scientific content generation" — a distinction that is precise but pragmatically significant: the system does not autonomously decide which ideas to pursue, how to acquire non-standard data, or which output is ready for submission.

The consequence. The reported result — 1 acceptance out of 3 submissions — reflects the performance of a human-AI pipeline, not a fully autonomous system. A fully autonomous deployment where the system randomly selects an idea from its pool, runs a single seed, and submits the result without human curation would almost certainly have a lower acceptance rate (possibly zero, given that 2 of 3 human-curated submissions were rejected). The paper provides no estimate of performance degradation as human involvement is removed. For a practitioner seeking to deploy this system at scale — where human review of every run's output would defeat the purpose of automation — the headline capability is substantially inflated relative to what would be achievable without human filtering.

What evidence exists in the paper. The paper acknowledges this limitation but does not measure it. Section 4.2 states: "Our current study aims to see whether The AI Scientist-v2 can produce at least one paper that survives peer review, and not what fraction of the time it can do so." Section 3.3 acknowledges that the Hugging Face dataset loading approach is "somewhat ad-hoc, as not all dataset repositories support this method," and the pest detection case in Appendix C.3.1 provides a concrete failure instance. However, no experiment tests whether removing idea selection, dataset assistance, or seed-based curation would materially change outcomes.

Mitigation status. The paper frames these as practical choices to "manage computational resources" rather than fundamental architectural limitations, and suggests that future work could address autonomous data acquisition (Section 3.3) and that "the system, if run for sufficiently many seeds, would have generated similar outputs, requiring only the final selection step to be performed by humans" (Section 4.2). These are hypotheses, not demonstrated capabilities. The idea selection and manuscript curation steps remain unsolved problems — the paper provides no mechanism for autonomously deciding which generated ideas are worth pursuing or which manuscript from multiple seeds is submission-ready.

6.2 Difficulty Estimation Cost Is Excluded from the Operational Budget, and No Internal Quality Metric Exists to Replace Peer Review

The assumption or constraint. The paper evaluates the system exclusively through external peer review — three submissions to an ICLR workshop (Section 4.1). This evaluation methodology treats peer review as a black-box oracle: the system's output is good if reviewers accept it. However, the paper provides no internal quality metric that predicts peer review outcomes. During the system's operation, the experiment manager selects the best-performing node at each stage boundary using an LLM evaluator that considers "performance metrics, training dynamics, and the quality of generated plots" (Section 3.2.2), but these criteria are never calibrated against external review scores. The VLM figure reviewer (Section 3.4) provides feedback on visual quality but does not assess overall scientific merit.

The consequence. This creates a fundamental measurement bottleneck: the only way to determine whether a change to the system (e.g., a different tree search strategy, a different LLM, a modified VLM review prompt) improves or degrades output quality is to submit batches of generated papers to peer review — a process that takes weeks to months, costs real reviewer labor, and is ethically constrained (the authors had to arrange withdrawal of accepted papers with ICLR leadership, Section 4.1). Without an internal proxy metric, iterative improvement of the system is prohibitively slow. A practitioner cannot run 100 experiments on the system's hyperparameters and rank the results — they would need to guess which configuration produced the best paper, submit it, and wait for reviews. This makes The AI Scientist-v2 essentially impossible to systematically optimize using current methods.

Additionally, the "experiment progress manager selects the best-performing node" (Section 3.2.1) at each stage boundary, but this selection is itself an unvalidated LLM judgment. If the LLM evaluator systematically favors superficially clean results over scientifically meaningful ones — and the paper's internal review found that the accepted manuscript's attention-augmented LSTM achieved 100% accuracy only because the task was trivially simple (Appendix C.1.2) — then the system may promote nodes that look good but are scientifically hollow. Without an external validity check (peer review) applied internally, the system has no mechanism to detect this failure mode.

What evidence exists in the paper. The paper provides abundant indirect evidence of this limitation. The internal reviews (Appendix C.1.1, C.2.2, C.3.2) reveal systematic flaws that the system did not self-detect: the accepted paper had a 57% train-test overlap (Appendix C.1.2), the label noise paper claimed results for temperature scaling experiments that were never run (Appendix C.2.3), and the pest detection paper described "multi-dataset training" when it actually trained separate models on single datasets (Appendix C.3.3). The VLM duplicate figure detection failed to catch a duplicate between Figure 1 (main text) and Figure 4 (appendix) in the label noise paper (Appendix C.2.1). All of these represent failures of the system's internal quality control — failures that peer review partially caught (the label noise paper was rejected for "mentioning results that are not substantiated") but partially missed (the 57% overlap in the accepted paper went undetected by reviewers). The paper does not report any quantitative correlation between internal metrics (LLM evaluation scores, VLM criticism counts) and external review outcomes.

Mitigation status. The paper does not address this limitation. It acknowledges that "peer review" serves as the evaluation mechanism but does not discuss the problem of developing internal quality proxies. Section 5 notes that "the current version of The AI Scientist-v2 does not yet consistently reach the rigorous standard required for top-tier conference publications," but frames this as a capability gap rather than a measurement gap. The open-sourcing of the codebase (Section 1) could enable the community to develop internal evaluation metrics, but the paper itself provides no foundation for such work.

6.3 Tree Search vs. Linear Experimentation Is Not Experimentally Validated, Making the Central Architectural Claim Untested

The assumption or constraint. The paper's primary technical contribution — replacing v1's linear experimentation with agentic tree search across four stages (Section 3.2) — is justified through analogy to human scientific practice and prior work on LLM-based code search (AIDE, Jiang et al., 2025), but never experimentally compared to a simpler alternative. There is no ablation where the same research ideas are run through linear experimentation and tree search, with the resulting papers compared on any metric (peer review scores, internal quality ratings, number of experimental conditions tested, depth of ablation coverage). The paper's Table 1 lists "Linear" vs. "Tree-Based" as a distinguishing feature between v1 and v2, but this is a cross-system comparison confounded by multiple simultaneous changes (template removal, VLM integration, parallel execution, different LLM models). The incremental contribution of tree search alone is unknown.

The consequence. A practitioner implementing an automated science system faces a direct design choice: is the substantial engineering complexity of tree search (node metadata management, buggy/non-buggy branching, debug depth limits, four-stage progression, specialized node types, parallel execution coordination) worth the implementation cost compared to a simpler approach — for example, generating N independent code variants and selecting the best, or running a single linear chain with error recovery? The paper provides no evidence to guide this decision. It is entirely possible that a system using linear experimentation but with the same base LLMs (Claude 3.5 Sonnet, GPT-4o) and VLM review would produce papers of comparable quality. The cited motivation — that v1's linear approach led to "short-sighted experimentation" (Section 2) — is an anecdotal observation, not a measured performance gap.

Furthermore, several components of the tree search architecture have design choices that are not justified empirically: why a debug probability of 1.0? Why a maximum debug depth of 3? Why 21 nodes in Stage 1 but only 12 in later stages? Why four stages rather than three or five? These hyperparameters (Table 3) are presented as fixed choices without sensitivity analysis. A practitioner cannot know whether increasing Stage 3's allocation from 12 to 24 nodes would meaningfully improve paper quality, or whether the debug depth limit of 3 prematurely abandons fixable branches.

What evidence exists in the paper. The paper contains zero ablations of its own architecture. Section 4.2 describes human evaluation of the AI-generated papers' scientific content, not evaluation of the AI Scientist-v2's design decisions. The experiments in Appendix C are code reviews of the generated papers, not system ablations. The total evidence for the tree search approach is: (1) the system produced one workshop-accepted paper using tree search, (2) v1 (which used linear search) was not submitted to peer review. This is insufficient to attribute the outcome to tree search rather than to improvements in base LLM capability (Claude 3.5 Sonnet vs. whatever v1 used), template removal, VLM integration, or the human curation pipeline.

Mitigation status. The paper makes no attempt to address this through experimental design. It does not acknowledge this as a limitation, does not suggest a future ablation study, and does not provide the controlled comparison that would allow a reader to assess the value of the tree search component independently. This is the most significant methodological gap in the paper, because the claimed technical contribution is architectural but the evaluation only measures end-to-end output quality.

6.4 The System's Scientific Soundness Is Brittle: Code-Level Errors That Survive Both Internal and External Review

The assumption or constraint. The AI Scientist-v2 relies on two quality-control mechanisms: VLM figure review (Section 3.4) to catch visual errors and the LLM-based experiment manager (Section 3.2.1) to select promising experimental nodes. The implicit assumption is that these mechanisms are sufficient to ensure that the final manuscript's scientific claims are supported by its experimental evidence. However, the paper's own internal code reviews (Appendices C.1.2, C.2.3, C.3.3) reveal multiple instances where the system produced, and failed to detect, scientifically consequential code-level errors — errors that undermine the validity of the generated paper's conclusions but that neither the VLM reviewer (which checks figures) nor the LLM evaluator (which checks performance metrics) was designed to catch.

The consequence. The accepted workshop paper contains a 57% train-test overlap in its synthetic dataset (Appendix C.1.2). This means that the paper's central finding — that compositional regularization does not improve generalization — is based on an evaluation where more than half of the "unseen" test examples were actually present in training. The human workshop reviewers did not detect this (they had no code access), and the AI Scientist-v2 did not detect it (data generation correctness is not checked by any component). The paper's conclusion might be correct (the regularization might genuinely not help), but the evidence supporting it is compromised. A reader of the accepted paper who trusts the peer review process would be misled about the reliability of the experimental results.

The label noise paper (Appendix C.2.3) claims to have evaluated temperature scaling for calibration improvement, and the paper's text discusses these results — but the code review reveals that temperature scaling was implemented but "never actually used" in any experiment. The manuscript writer hallucinated experimental results from the implementation plan rather than from actual execution output. This is a plan-to-paper hallucination: the system knew it intended to run temperature scaling experiments, the code existed, but the experiments were never executed, and the writing phase did not verify that claimed results corresponded to actual experimental outputs.

The pest detection paper describes "multi-dataset training" but the code (Appendix C.3.3) actually trained separate models on individual datasets — a fundamentally different experimental design than what the paper reports. The domain adaptation experiment that the paper references was attempted but ran unsuccessfully; the system paper proceeded to describe experiments that were planned but never completed.

These are not cosmetic issues. They are fabricated or compromised evidence in peer-reviewed scientific manuscripts. The current quality-control architecture has no mechanism to detect any of these failure modes because they occur at a level of abstraction — data generation correctness, experiment execution completeness, code-to-text consistency — that neither VLM figure review nor LLM performance-metric evaluation addresses.

What evidence exists in the paper. The internal code reviews (Appendices C.1.2, C.2.3, C.3.3) extensively document these errors. The authors "manually created test cases" to verify the ECE calculation, discovered the temperature scaling discrepancy, and re-ran the attention-augmented LSTM experiment with modified numbers to reveal that 100% accuracy was a simplicity artifact. The paper is transparent about these findings, which strengthens its credibility as an evaluation but reveals a fundamental weakness in the system being evaluated.

Mitigation status. The paper acknowledges some of these issues in its internal reviews but does not propose systemic solutions. Section 5 notes that "the current version... does not yet consistently reach the rigorous standard required for top-tier conference publications," but frames this as a capability limitation rather than a specific architectural gap. The VLM reviewer checks figures, and the LLM evaluator checks metrics, but no component checks data generation correctness, experiment execution completeness, or code-to-text consistency. The paper does not propose adding such components, and does not discuss the architectural challenge of verifying these properties autonomously. Until such mechanisms exist, papers generated by The AI Scientist-v2 require human code review to verify their scientific validity — which undermines the system's claim to end-to-end autonomy.

6.5 Single Benchmark (ICBINB Workshop) and Single Model Family Provide No Evidence of Domain or Venue Generalization

The assumption or constraint. All experimental validation of The AI Scientist-v2 comes from three submissions to a single workshop (ICBINB at ICLR 2025) that specifically solicits negative results and unexpected findings — a scope that is deliberately narrower and more forgiving than mainstream machine learning venues (Section 4.1). The system's code generation relies on Claude 3.5 Sonnet, its evaluation agents on GPT-4o, and its reasoning reflection on OpenAI o1 (Table 2). The three generated papers all fall within supervised learning for classification or regression tasks (compositional generalization, calibration under label noise, pest detection). The paper provides no evidence that the system can generate research in other ML subfields (reinforcement learning, unsupervised learning, generative modeling), other scientific disciplines (biology, physics, chemistry), other types of research contributions (theoretical, systems, benchmark creation), or other publication venues (conferences with lower acceptance rates, journals with different formatting requirements).

The consequence. The paper's central empirical claim — "The AI Scientist-v2 is capable of autonomously generating manuscripts that successfully pass peer review at a workshop of a major machine learning conference" (Section 7) — is supported only for the specific intersection of (ICBINB workshop × supervised learning × negative results × Claude 3.5 Sonnet/GPT-4o). A practitioner seeking to deploy this system for a different workshop theme, a different ML subfield, or with different base models has no data on expected performance. The ICBINB workshop's focus on negative results is particularly significant: generating a paper that convincingly reports a method doesn't work is fundamentally different from generating one that convincingly demonstrates a method does work and outperforms baselines on established benchmarks. The latter requires the system to achieve state-of-the-art results and make credible comparative claims — a substantially harder task that the paper does not evaluate.

The model dependency is also unexamined. Claude 3.5 Sonnet is a specific capability level; if a practitioner uses a less capable code-generation model (or a different provider's model with different strengths), the tree search's ability to produce working experimental code may degrade substantially. Conversely, a more capable model might make the tree search unnecessary — perhaps a single generation from a sufficiently powerful LLM would produce correct, well-structured code. The paper provides no guidance on these tradeoffs.

What evidence exists in the paper. The paper's evidence is confined to what it explicitly tests. Section 4.1 specifies the ICBINB workshop. Table 2 specifies the models. Appendix C provides the three full papers, all in supervised learning. The authors are transparent about scope: "the acceptance occurred at a workshop level rather than at the main conference track, and only one of the three AI-generated submissions was accepted" (Section 5). However, the paper does not discuss model dependency or domain generalization as limitations. The idea generation prompt (Appendix B) explicitly instructs the LLM to propose ideas that "can be done starting from the provided codebase, and does not require resources beyond what an academic lab could afford" — a constraint that implicitly limits the system to computationally modest ML experiments.

Mitigation status. The paper does not attempt to demonstrate generalization across venues, domains, or model families. It acknowledges scope limitations regarding workshop vs. conference standards (Section 5) but does not frame venue/domain/model dependency as a limitation requiring investigation. The open-sourcing of the codebase (Section 1) could enable community evaluation across these dimensions, but the paper provides no foundation for predicting how the system would perform under different conditions. Future work on "more complex datasets" and "larger models" is suggested in the context of improving the generated papers' depth (Section 5), not in the context of testing the system's robustness to different deployment conditions.

6.6 The Hardest Scientific Tasks — Truly Novel Hypotheses, Deep Mechanistic Analysis, and Conference-Level Rigor — Remain Beyond the System's Demonstrated Capability

The assumption or constraint. The paper frames The AI Scientist-v2 as making progress toward autonomous scientific discovery, and its conclusion envisions AI that will "generate papers that match or exceed human quality, even at the highest levels of scientific publishing" (Section 7). However, the demonstrated capability — one accepted workshop paper investigating whether a simple regularization term helps with a synthetic arithmetic task — represents a narrow slice of scientific work. The system did not formulate a genuinely novel hypothesis (compositional regularization had been explored in prior work), did not provide deep mechanistic analysis of why the regularization failed (the paper offers speculative explanations), did not compare against a comprehensive set of baselines (the initial idea listed SCAN, COGS, machine translation, and semantic parsing — none of which were implemented), and produced work that the authors rated as 5/10 for workshop quality (Appendix C.1.1). The internal review identifies that "the experimental evaluation could benefit from more depth," "the considered sequence lengths are very short and the considered task is only synthetic," and the paper "lacks sufficient depth and rigor for acceptance into a full conference" (Section 4.2).

The consequence. There is a substantial gap between the paper's empirical demonstration and its rhetorical framing about scientific discovery. The system successfully executed a routine empirical investigation — design a regularization term, test it on synthetic data, observe it doesn't help, write up the negative result — that a competent graduate student could complete in days. The hardest aspects of scientific research — formulating questions that advance the field's understanding, designing experiments that definitively test causal mechanisms, interpreting results in the context of theoretical frameworks, identifying which null results are informative vs. which reflect implementation errors — are either absent from the demonstrated capability or performed at a level the authors themselves rate as below conference standard. The paper acknowledges that "formulating genuinely novel, high-impact hypotheses, designing truly innovative experimental methodologies, or rigorously justifying design choices with deep domain expertise — remain challenging for purely automated systems" (Section 5), but does not characterize how far the current system is from these capabilities.

A practitioner should understand that The AI Scientist-v2 is not a "scientist" in the sense of generating new knowledge that advances human understanding. It is an automated empirical investigation pipeline that can take a rough research direction, implement a reasonable experimental test, and write up the results in passable academic prose. The distinction matters because deploying this system with the expectation of generating novel insights — rather than efficiently exploring variations on known themes — would lead to disappointment. The paper's own internal evaluation: none of the three generated papers "met the quality standards typical of top-tier main-track conferences" (Section 4.1).

What evidence exists in the paper. The authors' structured internal reviews (Appendices C.1.1, C.2.2, C.3.2) provide detailed evidence. The accepted paper's weaknesses include: vague method description, missing fundamental citations (Hochreiter & Schmidhuber, 1997), incorrect figure captions, misleading claims about attention model performance, and dataset overlap compromising the reliability of conclusions. The rejected papers had more severe issues: missing experiments, duplicate figures, misleading terminology ("multi-dataset training" for single-dataset models), and conclusions unsupported by the presented evidence. The human workshop reviews (Sections 4.2, Appendices C.2.4, C.3.4) corroborate these assessments: reviewers noted insufficient justification, limited experimental breadth, missing citations, and lack of theoretical grounding. These are not failures of implementation that more compute could fix — they reflect fundamental limitations in the system's ability to reason about scientific methodology, contextualize results within literature, and maintain consistency between experimental execution and textual description.

Mitigation status. The paper is transparent about the workshop-vs-conference gap (Section 5), acknowledges that "significant challenges remain in consistently achieving top-tier quality and generating truly groundbreaking hypotheses" (Section 7), and frames the current work as "promising early signs of progress" rather than a solved problem. However, the paper does not analyze why the system fails to achieve conference-level depth, does not identify specific architectural limitations that cause the failure modes observed (e.g., hallucinating experimental results, producing internally inconsistent papers), and does not propose concrete mechanisms for addressing these limitations in future versions beyond the generic observation that "LLMs rapidly advance" (Section 5). The gap between demonstrated capability and the envisioned future of "AI science contributing to the betterment of humanity, fostering collaboration and accelerating the pace of discovery" (Section 7) is acknowledged in spirit but not characterized in measurable terms.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm, a new benchmark, or a new theoretical result. What it introduces is a new unit of evaluation for automated science systems: the blind peer-reviewed workshop paper. By submitting AI-generated manuscripts to a real venue and reporting the outcomes — including the reviews, the scores, and the internal code audits — the authors shift the conversation from "does this system produce plausible-looking output?" to "does this system produce output that survives the same adversarial scrutiny applied to human scientists?" This is a methodological contribution disguised as a systems paper, and it matters because the field has lacked standardized, community-validated success criteria for automated research.

This shift has several concrete consequences for the research landscape:

Peer review becomes a (flawed but operational) benchmark. Prior automated science systems evaluated themselves through author inspection or automated metrics. The AI Scientist-v2 demonstrates that external peer review — for all its well-documented problems — provides a harder-to-game signal than self-assessment. The fact that two of three submissions were rejected, and that the accepted paper contained a 57% train-test overlap that reviewers missed (Appendix C.1.2), reveals that peer review is neither sufficient nor necessary for scientific soundness, but it is more informative than no external check at all. Future systems can adopt this evaluation methodology with the same transparency — submit to real venues, publish the reviews, audit the generated code — creating a cumulative record of capability improvements over time. However, the practical barriers (IRB approval, venue cooperation, withdrawal agreements, reviewer consent) are substantial, and the paper's success in navigating them represents a non-trivial infrastructure contribution.

The role of AI in science becomes experimentally testable rather than purely philosophical. Debates about whether AI can "do science" have historically been fought at the level of definitions and thought experiments. The AI Scientist-v2 makes this question empirical: generate papers, submit them to peer review, report the scores. The answer — at least for workshop-level ML research using Claude 3.5 Sonnet and GPT-4o in April 2025 — is "yes, occasionally, with significant human curation, and with brittleness that internal audit reveals but peer review does not." This is a more useful answer than any philosophical argument because it establishes a baseline against which future systems can be measured. When someone claims that a v3 or v4 system "can do science," the natural question becomes: what peer-reviewed venues has it passed, at what rate, and what did the code audit reveal? The paper provides a template for answering that question with evidence rather than rhetoric.

The human-AI boundary becomes precisely auditable. The paper's granular accounting of human involvement — idea selection from an AI-generated pool, dataset downloading for the pest detection case, final manuscript curation from multiple seeded runs (Section 4.2) — establishes a reporting standard that the field can adopt. A claim of "fully autonomous" research becomes falsifiable: if a system requires human intervention at any point between topic specification and final submission, those interventions should be enumerated. This prevents both overclaiming (systems that hide human scaffolding) and underclaiming (systems that are more autonomous than the authors describe). The distinction between "meta-selection from fully autonomous outputs" and "human-in-the-loop intervention in content generation" provides a vocabulary for discussing partial autonomy that is more precise than binary "autonomous vs. not" claims.

Negative results and self-audit become features rather than bugs. The paper's most distinctive rhetorical move is its willingness to publish detailed internal reviews that are more critical than the external peer reviews. The authors rate their own accepted paper at 5/10 (Appendix C.1.1), below the 6.33/10 it received from reviewers, and their code audits reveal flaws (train-test overlap, hallucinated experiments, misleading terminology) that workshop reviewers missed. This establishes a norm where the developers of automated science systems are expected to be their own harshest critics — not to showcase capability, but to honestly characterize limitations. For a field where overclaiming is common and reproducibility is poor, this norm of aggressive self-audit, if adopted, would be a significant cultural contribution.

The template-removal threshold is established as a meaningful autonomy milestone. Table 1's comparison between v1 ("topic-specific" codebase) and v2 ("domain-general") provides a clear, binary criterion for assessing progress toward autonomy: can the system generate experimental code for a novel research idea without human-provided starting code? The paper demonstrates this for three distinct supervised learning tasks (compositional generalization, calibration under label noise, pest detection), establishing that template-free code generation is feasible for routine ML experimentation. Future systems can be evaluated on whether they cross this threshold for other domains (reinforcement learning, theoretical analysis, benchmark creation) and other languages (R, Julia, matlab). The criterion is crisp, verifiable, and directly measures the system's ability to translate abstract research ideas into executable experiments — a capability that is necessary (though not sufficient) for genuine scientific autonomy.

What does NOT change. The paper does not demonstrate that AI can generate novel, high-impact research directions. The accepted paper's idea — "add a regularization term and see if it helps with compositional generalization" — is a routine empirical investigation that the authors themselves rate as insufficient for conference publication. The paper does not demonstrate that AI can conduct deep mechanistic analysis or theoretical work. It does not demonstrate that AI can design experiments whose results would surprise domain experts. And it does not demonstrate that the system can operate without human curation of its outputs. These remain open problems, and the paper's primary contribution is establishing a methodology for measuring progress on them rather than solving them.

Follow-Up Research This Work Enables

Closed-loop quality control: detecting train-test leakage, hallucinated results, and code-to-text inconsistency during generation. The paper's internal code audits (Appendix C) reveal three failure modes that the current quality-control architecture does not detect: data generation bugs (57% train-test overlap in the accepted paper), plan-to-paper hallucination (temperature scaling described but never executed in the label noise paper), and code-to-text mismatches (single-dataset training described as "multi-dataset" in the pest detection paper). A natural follow-up would add dedicated verification agents that operate alongside the VLM reviewer during tree search: a data integrity agent that computes train-test overlap statistics and flags excessive leakage, an execution completeness agent that cross-references the experiment plan JSON against actual execution logs to ensure every claimed experiment was run, and a code-to-text consistency agent that compares the manuscript's method and results claims against the actual code and logged metrics. The evaluation would measure whether these agents reduce the rate of the specific errors documented in Appendix C, using the same internal audit methodology the paper demonstrates.

Ablating the tree search: head-to-head comparison against linear experimentation on identical research ideas. The paper's central architectural claim — that agentic tree search enables deeper exploration than linear experimentation — is never tested. A direct follow-up would run the same set of research ideas (perhaps the three from the paper, plus additional ideas from the AI-generated pool of ~40) through both v2's tree search and a stripped-down linear baseline that generates code sequentially with error recovery but no branching, parallel exploration, or stage-based progression. The evaluation would use the same external peer-review methodology (submit to a workshop, report scores) plus internal metrics: number of distinct experimental conditions tested, number of ablation studies conducted, presence of specific error types documented in Appendix C, and author-assigned quality scores using the structured review format from Appendix C.1.1. A null result — tree search produces papers indistinguishable from linear experimentation — would be highly informative, suggesting that template removal and VLM review, not search architecture, drive v2's improvements over v1.

Idea selection automation: can the system decide which of its own ideas to pursue? The paper's human pipeline involved selecting 3 ideas from approximately 40 AI-generated proposals based on "alignment with the workshop theme and potential interest" (Section 4.2). A natural extension would train or prompt the system to perform this curation itself: after generating N ideas, have the LLM (or a separate evaluator LLM) rank them on dimensions like novelty (checking against Semantic Scholar), feasibility (can the idea be implemented given the system's tool constraints?), and alignment with the target venue's theme. The evaluation would compare the quality of papers generated from top-ranked ideas vs. randomly selected ideas vs. human-selected ideas, submitted under the same peer-review protocol. A positive result — AI-selected ideas produce papers of comparable quality to human-selected ones — would close one of the three major human-scaffolding gaps the paper identifies.

Stress-testing the system on deliberately adversarial research tasks. The paper's three submissions all fall within supervised learning for classification/regression — tasks where the base LLMs have abundant training data and the system's tools (PyTorch, Hugging Face datasets) are well-suited. A stress-test would design a set of research tasks that deliberately probe the system's known weaknesses: tasks requiring non-Python implementation (R, Julia), tasks requiring access to non-Hugging-Face data (web scraping, API calls, custom file formats), tasks requiring comparison against methods the LLM may not know (very recent papers, niche subfields), and tasks where the "obvious" approach is flawed (to test whether the system can recognize and correct its own questionable assumptions). The evaluation would measure the system's failure modes — not just whether papers are accepted, but what specific errors occur and whether the documented limitations (Section 5) generalize or are specific to the ICBINB/ML domain.

Internal quality proxy: training a reviewer model on the paper's peer-review data. The paper identifies but does not address the measurement bottleneck: the only way to evaluate a generated paper's quality is external peer review, which takes weeks and cannot be used for rapid iteration. With 3 papers, 3 sets of reviews (scores + detailed feedback), and the authors' own structured internal reviews, a small but valuable dataset exists for training an automated reviewer. A follow-up would use this data (potentially augmented with human reviews of additional AI-generated papers) to fine-tune a model that predicts peer-review scores and identifies specific flaw categories (missing experiments, hallucinated results, misleading terminology, figure-caption mismatch) from the paper PDF alone. The evaluation would measure correlation with actual peer-review scores on held-out submissions and test whether the automated reviewer catches the errors documented in Appendix C that human reviewers missed (the 57% train-test overlap, the temperature scaling hallucination). A working proxy — even one that only identifies the most egregious errors — would dramatically accelerate the system's development cycle.

Domain generalization: testing the system outside supervised ML. The paper's scope is entirely within supervised learning experiments. A domain-expansion study would test the system on research tasks from adjacent ML subfields: reinforcement learning (implement a new exploration bonus in a standard environment), generative modeling (test a new regularization on a VAE or diffusion model), and meta-learning (implement and evaluate a new meta-training objective). Each domain requires different standard libraries (gymnasium, diffusers, learn2learn), different evaluation protocols, and different figure conventions — testing whether the system's template-free code generation and VLM review generalize. The evaluation would follow the same peer-review submission protocol but target venues in those subfields (e.g., RLDM for reinforcement learning, a generative modeling workshop). The key metric is not just acceptance rate but whether the system can produce runnable code for a research idea in each domain, since the paper's own pest detection example (Appendix C.3.1) shows that domain-specific data requirements can defeat the current toolset.

Practical Applications and Downstream Use Cases

Automated empirical investigation of routine research hypotheses. An ML research lab with a backlog of "we should test whether X helps with Y" ideas — simple empirical questions that a graduate student could investigate in a few days but that never get prioritized — could deploy The AI Scientist-v2 to systematically explore this space. A human provides the high-level idea (e.g., "does label smoothing help with calibration under asymmetric noise?"), the system generates the code, runs the experiments, and produces a manuscript with the answer. The human's role reduces to idea specification and final verification. Based on the paper's runtime of "several hours to a maximum of 15 hours" per paper (Appendix A), a single GPU machine could process multiple ideas per day in parallel. The practical value is not in producing publishable papers (though some might be workshop-worthy) but in accelerating the empirical exploration that precedes deeper investigation — efficiently answering the "is this worth pursuing?" question that currently consumes human researcher time.

Rapid prototyping of negative-result papers for venues that value them. The ICBINB workshop specifically solicits negative results and unexpected findings — a niche that is well-matched to the system's demonstrated capabilities (the accepted paper was exactly this: a regularization method that didn't work). A researcher with a collection of failed experiments that never got written up could use The AI Scientist-v2 to generate initial drafts that convert experimental logs into structured manuscripts. The system's tree search would handle the ablation studies and hyperparameter tuning that make negative results credible (showing that the failure isn't due to poor tuning), and the VLM review would ensure figures are publication-ready. The human provides the experimental logs from their own failed attempts; the system generates the manuscript and identifies missing experiments (through the four-stage process) that would strengthen the negative-result argument. Given that negative-result venues are underserved relative to the volume of unpublished null findings, this could increase the rate at which the community learns from failures.

Educational tool for teaching experimental design and scientific writing. The paper's internal reviews (Appendices C.1.1, C.2.2, C.3.2) reveal systematic flaws in the AI-generated papers — vague method descriptions, missing citations, misleading figure captions, conclusions unsupported by evidence — that are also common in novice human papers. An educational deployment could use The AI Scientist-v2 to generate first drafts of papers on simple research questions, then have students perform the same structured review that the authors demonstrate (Section 4.2, Appendix C) to identify and correct these flaws. The pedagogical value is that students learn scientific critique on papers where the errors are systematic and well-characterized, and the authors' detailed annotations provide a model for what to look for. The system's code audit capability (showing exactly where the generated code diverges from the paper's claims) would be particularly valuable for teaching the distinction between "the paper says X" and "the code actually does Y" — a skill that usually requires painful firsthand experience.