Language Modeling by Language Models

ArXiv: 2506.20249

🎯 Pitch

A multi-agent LLM system autonomously discovers 1,062 novel language model architectures that beat GPT-2 and Mamba2 on 6 out of 9 benchmarks. It achieves this using a genetic programming backbone that improves successful code generation by ~86 percentage points over standard prompting, and a 'ladder of scales' strategy that smartly allocates training compute across model sizes.

1. Executive Summary

This paper introduces Genesys, an autonomous multi-agent LLM system that simulates the full research pipeline for discovering novel language model architectures, from ideation and literature review to code generation, pretraining, and downstream evaluation. Using a genetic programming backbone with a factorized Generalized Autoregressive Unit (GAU) tree representation—which decomposes architecture designs into composable, mutatable sub-components—and a Ladder of Scales verification strategy that allocates exponentially fewer training trials at larger model sizes (14M to 350M parameters), Genesys discovers 1,062 fully verified novel designs when evaluated against the Pythia/GPT-2 training and LM-Eval benchmarking suite. The system's best discovered architectures outperform standard transformer and Mamba2 baselines on 6 of 9 common zero-shot benchmarks, and the unit-based code generation approach yields an ~86 percentage point improvement in successful design generation over direct prompting, establishing that structured, factorized search spaces dramatically improve the viability of LLM-driven architecture discovery by avoiding the exponential failure modes of single-shot code generation.

2. Context and Motivation

The Core Problem: Can We Automate Novel Architecture Discovery?

The paper addresses a deceptively ambitious question: can LLMs be used to model the entire process of discovering novel language model architectures that meaningfully improve on the standard transformer? This is not just about generating code snippets — it's about simulating the full research pipeline: ideation grounded in literature, adversarial peer review of ideas, incremental implementation of designs with correctness checks, budget-aware experimental verification through pretraining, and downstream evaluation against established benchmarks. The paper frames this as a specialized instance of Automated Scientific Discovery (ASD), but one with a clear, objectively measurable goal — producing executable architecture designs that outperform human-designed baselines on standard metrics.

This problem sits at the intersection of three fields — neural architecture search (NAS), LLM-driven code generation, and automated scientific discovery — but the paper argues that none of them, individually or in combination, have previously tackled this specific challenge in a way that is both broad in scope and verifiable in outcome.

Why This Problem Matters: Beyond Incremental Improvements

The significance of this problem operates on multiple levels:

Scientific significance. Transformer architectures (Vaswani et al., 2017) remain the de facto standard for language modeling, but research into alternatives — state-space models (Gu et al., 2022; Dao & Gu, 2024), modern RNNs (Peng et al., 2024; Beck et al., 2024), test-time training approaches (Sun et al., 2024) — is an active, rapidly evolving field. If autonomous systems can systematically explore this design space, they could accelerate the discovery of architectures that are more efficient, more scalable, or better suited to specific downstream tasks than what human researchers find through intuition and manual experimentation. The paper explicitly connects this to the "mature field of neural architecture search" (Section 2), but positions itself as pursuing a broader goal: not just finding good architectures, but modeling the process of discovery itself.

Practical significance for research infrastructure. The paper's central claim — that fully automated discovery pipelines can produce competitive architectures — has direct consequences for how research organizations allocate resources. If autonomous systems can reliably generate and verify novel architectures, this could dramatically reduce the human effort required for architecture research, freeing researchers to focus on higher-level scientific questions. The paper's scale is noteworthy here: 1,162 newly discovered designs, 1,062 fully verified through pretraining, involving more than 1 billion tokens, 2.76 million lines of code, and 86,000 agent interactions. This is, to the authors' knowledge, "the largest ASD experiment of its kind" (Section 1), and the sheer scale demonstrates that autonomous discovery at this level is not just theoretically possible but practically achievable.

Theoretical significance for LLM capabilities. The problem serves as a stress test for what LLMs can do in complex, multi-step reasoning tasks. Architecture discovery requires deep understanding of the research literature (to propose genuinely novel ideas), the ability to write correct, efficient code in an unbounded design space (not just fill in templates), and the judgment to allocate limited computational resources wisely (deciding which designs to verify at which scales). Success in this domain would demonstrate that LLMs can go beyond the "generate and evaluate" paradigm that dominates most LLM-based discovery systems and can engage in something closer to the deliberative, iterative process that characterizes human research.

Where Prior Approaches Fall Short

The paper identifies specific limitations across three relevant fields:

1. ASD systems have focused on open-ended, hard-to-verify tasks. Recent LLM-driven ASD systems — such as AI Scientist (Lu et al., 2024a; Yamada et al., 2025), AgentLab (Schmidgall et al., 2025a), CodeScientist (Jansen et al., 2025a), and AIGS (Liu et al., 2024a) — have targeted open-ended research with unclear goals and evaluation protocols. The paper argues (Section 2) that this creates a fundamental problem: "discoveries are hard to verify." Without a clear, objective success criterion, it's difficult to assess whether a system is genuinely making progress or simply generating plausible-sounding artifacts. This motivates the paper's focus on architecture discovery specifically because it offers "a clear objective yet involves many new challenges for ASD" — the fitness function is well-defined (downstream task performance), and designs can be objectively compared.

2. NAS operates in constrained, fixed operation spaces. Traditional NAS approaches (Esken et al., 2019; White et al., 2023; Chitty-Venkata et al., 2022) search over a fixed inventory of operations — attention heads, convolution kernels, pooling layers — to find the best combination for a given task. The paper argues (Section 2) that this is too restrictive: the goal is not just to recombine known operations but to discover genuinely novel mechanisms and architectural principles. The paper explicitly states that it aims for "a broader space of operations and architectures" and, critically, "attempt[s] to model the broader scientific discovery process" — not just the search itself, but the literature review, ideation, proposal writing, and adversarial review that precede implementation. This is a fundamentally different framing from standard NAS.

3. Direct LLM code generation fails catastrophically on complex, multi-constraint tasks. The paper identifies a critical bottleneck that any LLM-driven architecture discovery system must solve: generating code that simultaneously satisfies multiple constraints — syntactic validity (correct Python), semantic validity (differentiable, causal, numerically stable, as checked by the symbolic checker in Table 1), and novelty (genuinely different from prior designs). The paper shows (Section 4.2, Table 4) that a direct prompting approach — present the model with the task and regenerate from scratch on failure — achieves only 6% validity. This is not surprising from a theoretical standpoint: if a design has N sub-components that must each be correct, and each has independent success probability p, the probability of a single-shot success is p^N, which decays exponentially. The paper formalizes this as a geometric distribution problem (Appendix A.1), showing that the expected number of calls for direct prompting is 1/p^N, which becomes prohibitive for complex designs. This failure mode is "familiar to many code generation systems" (Section 4.2), but the paper argues that existing work has not systematically addressed it in the context of multi-constraint scientific code generation.

4. Genetic programming with LLMs is nascent but underexplored for architecture discovery. Some recent work has combined genetic programming (GP) with LLMs — notably Hemberg et al. (2024) on evolving code and Romera-Paredes et al. (2024) on mathematical discovery — but these systems typically operate in domains with simpler correctness criteria (e.g., program output matching a target). Architecture discovery adds the challenge that correctness cannot be determined by a single execution: a design must be pretrained and evaluated, which is expensive and stochastic. The paper builds on the GP + LLM paradigm but introduces substantial innovations in how the search space is structured (the GAU tree factorization) and how resources are allocated (the Ladder of Scales) that are specific to the architecture discovery domain.

How This Paper Positions Itself

The paper positions Genesys as addressing a gap in the literature that falls between several existing paradigms:

Relative to ASD systems: It argues for moving from open-ended discovery to tasks with clear, verifiable objectives, while preserving the ambition of modeling the full research process (not just the final artifact).
Relative to NAS: It argues for expanding the search space from fixed operation inventories to an unbounded code-based space, and for incorporating the "pre-implementation" stages of research (literature review, ideation, proposal review) that NAS typically ignores.
Relative to LLM code generation: It argues that naive prompting strategies are fundamentally inadequate for the multi-constraint, multi-component code that architecture discovery requires, and proposes a structured, factorized approach (the GAU tree + Viterbi-style unit-by-unit generation) that is theoretically justified (Appendix A.1) and empirically validated (Table 4).
Relative to prior GP + LLM systems: It argues that architecture discovery introduces unique challenges — most notably the cost of verification — that require innovations in budget allocation (the Ladder of Scales) and design selection (the quadrant-based exploitation-exploration strategy) not present in prior work.

The paper's motivating tension is thus: the ambition of modeling the full scientific discovery process for a complex, impactful domain is appealing, but the component sub-problems — generating valid, novel code; evaluating designs at scale; allocating limited compute — are individually hard in ways that prior work has not solved. Genesys is positioned as a system that addresses all of these simultaneously through a combination of structural innovations (the GAU factorization, the Viterbi-style implementation strategy) and resource management strategies (the Ladder of Scales, the quadrant-based selection), with the goal of demonstrating that end-to-end autonomous architecture discovery is not just aspirational but feasible at meaningful scale.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a systems and empirical analysis paper that builds an autonomous multi-agent pipeline called Genesys for discovering novel language model architectures. The core idea is that by factorizing architecture code into a tree of composable, mutatable sub-components (called Generalized Autoregressive Units, or GAUs) and using a genetic programming (GP) backbone—where LLM-powered agents propose, implement, and verify modifications to these sub-components—the system can efficiently explore a vast design space while avoiding the catastrophic failure modes that cripple direct, single-shot code generation approaches. The system solves the problem of navigating an unbounded, multi-constraint code generation space (syntactic validity, semantic correctness, novelty, and empirical performance) by decomposing it into a sequence of locally-checkable sub-problems, each of which can be retried independently, and by allocating expensive verification compute (pretraining) according to a budget-aware "Ladder of Scales" strategy that tests many designs cheaply at small scales before committing to fewer, more expensive evaluations at larger scales.

3.2 Big-Picture Architecture (Diagram in Words)

The Genesys system has five major components that interact asynchronously through a shared data structure:

LMADE (Language Model Architecture Discovery Environment): The external environment providing foundational tools. It contains two sub-engines:
- Knowledge Engine (KE): Provides access to the academic literature via a manually curated reference library, arXiv, Semantic Scholar, and web search. It services agent queries for background research.
- Verification Engine (VE): Provides tools for checking code correctness and executing expensive experiments. This includes a Symbolic Checker (for static and runtime code analysis) and an automated pipeline for pretraining models and evaluating them on downstream benchmarks.
Evolution Tree: A central, shared database that stores all discovered architecture designs. Each node in the tree contains a design's executable code, its factorized GAU tree representation, design traces, and empirical performance metrics (fitness and confidence). It serves as the communication medium between designer and verifier agents.
Designer Agents: A network of five specialized LLM agents (Proposer, Reviewer, Planner, Coder, Observer) that form a pipeline to generate new designs. They select parent designs from the Evolution Tree, research the literature via the Knowledge Engine, propose and review novel architectural modifications, and then implement them piece-by-piece, adding the new designs back to the Evolution Tree.
Verifier Agents: Agents that select unevaluated designs from the Evolution Tree and run the full pretraining and evaluation pipeline using the Verification Engine. They manage the budget via the Ladder of Scales strategy, performing many cheap evaluations at small scales and few expensive ones at large scales, and update the Evolution Tree with performance results.
Ladder of Scales Budget Controller: A resource management algorithm that enforces a pyramidal verification budget, allowing hundreds of trials at the 14M parameter scale but only a handful at 350M. It dynamically releases higher-scale budgets as lower-scale verifications are completed.

The workflow runs asynchronously and in parallel: Designer agents continuously consume promising designs from the tree, consult the literature, and produce new candidates; Verifier agents continuously consume unevaluated designs from the tree, train and evaluate them, and write back performance scores. This distributed architecture ensures that the slow bottleneck of pretraining does not stall the faster ideation and code generation processes.

3.3 Roadmap for the Deep Dive

First, the GAU factorization and Evolution Tree (§3.4.1): This is the fundamental structural innovation that enables all downstream operations. I'll explain how arbitrary architecture code is decomposed into a tree of typed, composable units, why this guarantees a valid search space, and how the tree serves as the evolutionary substrate.
Second, the Proposer-Reviewer loop for ideation (§3.4.2): This covers how the system generates novel research ideas grounded in the literature, how it selects parent designs for modification, and how an adversarial reviewer ensures novelty before implementation even begins.
Third, the Viterbi-style, unit-based code generation pipeline (§3.4.3): This is the core implementation machinery involving the Planner, Coder, and Observer agents. I'll detail the recursive, step-by-step code generation process, the role of the Symbolic Checker, and provide the formal justification from Appendix A.1 for why this is exponentially more efficient than direct prompting.
Fourth, the Verifier, Ladder of Scales, and design selection (§3.4.4): This covers how expensive verification is managed, how the budget is dynamically allocated across scales, and how the quadrant-based selection strategy balances exploration and exploitation when choosing designs for both further evolution and verification.
Fifth, the formal justification for the factorized approach (§3.4.5): This synthesizes the theoretical guarantees from Appendices A.1, A.2, and A.3, explaining why a GAU tree representation is universally sufficient, why unit-based generation yields an exponential reduction in expected cost, and why this enables a "few high-quality samples" strategy to outperform a "many low-quality trials" one.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that the challenge of LLM-driven architecture discovery can be made tractable by (1) factorizing the search space into a GP-friendly tree representation of composable code units, (2) using this factorization to implement a unit-by-unit, Viterbi-style code generation pipeline that exponentially reduces failure probability, and (3) managing the cost of experimental verification through a budget-adaptive Ladder of Scales strategy that exploits the correlation of performance across model sizes.

3.4.1 The GAU Factorization and Evolution Tree

The problem of representing architecture designs. The first major challenge for any system that aims to discover or evolve neural architectures is how to represent a design in a way that (a) is expressive enough to capture a wide range of possible mechanisms (attention, state-space models, RNNs, etc.), (b) is structured enough to allow for meaningful, local modifications (mutations and crossovers), and (c) guarantees that any combination of valid sub-components produces a valid overall design. A flat, monolithic code representation fails on all three counts: it is difficult to make local changes without breaking the whole, and there is no guarantee that splicing code from two different designs yields a coherent result.

The Generalized Autoregressive Block (GAB). The paper establishes a common interface that all architecture designs must adhere to. At the top level, a language model is a stack of repeating blocks. Each block is a PyTorch module that implements a function of type (X, Z) -> (X, Z), where X is an input tensor of shape (B, L, D) (batch size, sequence length, embedding dimension) and Z is a dictionary of intermediate variables (e.g., memory states, caches). The block outputs a new tensor Y of the identical shape and an updated dictionary Z'. This interface is encoded in the GABBase class (Figure 3, and detailed in Appendix B.3, Figure 17). A full model, encoded in the GAM class (Figure 18), simply embeds the input tokens and then applies a sequence of these GAB blocks.

The Generalized Autoregressive Unit (GAU) tree. The key structural insight is that a complex GAB block can be recursively decomposed into a tree of smaller, functionally identical sub-modules called Generalized Autoregressive Units (GAUs). Each GAU, like a GAB, is a PyTorch module (GAUBase) that also implements a function of type (X, Z) -> (X, Z). A GAU can be atomic (implementing a specific operation like multi-head attention or a gated MLP) or composite (implementing a sequence of calls to its child GAUs). The overall block, then, is represented as a tree where the root is the full block, internal nodes are composite GAUs that orchestrate their children, and leaves are atomic GAUs with concrete implementations. Figure 3 illustrates this with the GPT block: the root GPTBlock decomposes into GPTMultiheadAttention and GatedMLP child units, and the attention unit itself might decompose further into sub-units.

Why this representation is universally sufficient. The paper provides a formal justification in Appendix A.2. Since all blocks follow the (X, Z) -> (X, Z) (or more generally Σ -> Σ) type signature, and the language of GAU programs is closed under composition (the sequential execution of two Σ -> Σ functions is itself a Σ -> Σ function), any valid block program can be factorized into a finite GAU tree. The base case is an atomic operation; the recursive case composes the trees of sub-functions. A crucial extension (Appendix A.2.2) shows that even operations that do not naturally have this signature—such as a residual connection which has type X -> X—can be "lifted" into the Σ -> Σ type via encoder/decoder pairs, meaning the representation is fully general for any architecture that can be typed in this way.

The Evolution Tree as the search population. The Evolution Tree (Figure 4) is a graph where each node represents a single discovered architecture design. A node stores the design's executable GAB code, its complete GAU tree representation (a hierarchical structure showing which units compose the block), a "design trace" (the lineage of modifications that led to it), and empirical performance metrics (fitness and confidence) populated by verifier agents. Edges in the tree represent parent-child relationships. A new design can be created by applying a genetic programming (GP) operation—mutation or crossover—to one or two parent designs. Mutation (Figure 5A) selects a single parent and modifies one specific GAU within it, replacing a sub-unit with a new design. Crossover (Figure 5B) selects two parents and merges their units into a novel combination. A third operation, "design from scratch," is a special case of mutation on an empty root. The tree is initialized with five state-of-the-art human-designed architectures: GPT, Mamba2, RetNet, RWKV6, and TTT.

3.4.2 The Proposal Stage: Ideation and Adversarial Review

Parent selection with quadrants. Before generating a new idea, the system must select one or two parent designs from the Evolution Tree to modify. This selection is handled by the QuadrantSelect algorithm (Section 4.3, Algorithm 4, and Figure 10). Every design in the tree is ranked along two dimensions: its fitness F, defined as the average downstream task performance aggregated across all scales at which it has been verified, and its confidence, defined as the number of scales at which verification has been performed. The medians of these two dimensions are used to partition the population into four quadrants:

Good & Confident (high fitness, high confidence): Promising designs that have been well-tested. Designers primarily exploit these for further improvement.
Good & Unconfident (high fitness, low confidence): Designs that look promising in small-scale tests but haven't been verified at larger scales. Verifiers primarily exploit these by promoting them to larger-scale evaluation.
Poor & Confident (low fitness, high confidence): Well-tested designs that underperform. Designers explore these to see if a mutation can salvage a promising structural idea.
Poor & Unconfident (low fitness, low confidence): Untested, low-performing designs. Verifiers explore these opportunistically.

A probabilistic scheduler selects from exploitation quadrants with probability p_explore = 0.15 and exploration quadrants otherwise. A restart mechanism (with an annealing probability starting at 1 and decreasing to 0.05) periodically re-selects from the initial five seed designs to ensure diversity. The GP operation type is chosen probabilistically: mutation with probability 0.75, crossover with 0.2, and design from scratch with 0.05.

Literature-grounded ideation. Once parents are selected, the Proposer agent (an LLM) initiates the proposal stage (Figure 7). It first conducts multiple rounds of literature search using the Knowledge Engine. It formulates precise keyword queries for external academic sources (arXiv, Papers with Code, Semantic Scholar) and semantic descriptions for the internal vector database of the curated reference library (297 papers with associated code snippets). A general web search via Perplexity.ai provides broader context. The proposer records all findings and its analysis in a running note, as search results are cleared between rounds to prevent context overflow. The goal is to produce a research proposal that specifies:

Title and abstract: A high-level description of the novel design.
Motivation and related work: Why the proposed modification addresses a limitation in existing architectures, grounded in the literature.
Design plan: A specific selection of a single GAU in the parent tree to modify, and a detailed, mathematically-formulated description of the new sub-unit(s) to replace it with, including pseudo-code.
Novelty: The modification must be "local" (influence as few existing units as possible) to be traceable and factorizable.

Adversarial review cycle. The proposal is then passed to a separate Reviewer agent (Figure 7). The reviewer conducts its own literature search to verify the novelty and soundness of the idea. It assigns a float rating from 0 to 5, evaluating clarity, innovation, theoretical soundness, scalability, and efficiency. The reviewer's system prompt explicitly instructs it to be "objective, strict, and fair," to check for replication of "sibling" proposals that share the same parents, and to compare against similar prior unit codes retrieved via embedding-based search from a library of past proposals. The loop continues: if the proposal is rejected, the feedback is passed back to the proposer for refinement, which triggers another round of literature search and proposal revision. This continues for up to MAX_ROUNDS = 5 attempts. If the final rating exceeds a threshold (THRESHOLD = 4.0), the proposal is accepted and proceeds to implementation. The paper reports that reviewer agents exhibit varying stringency: GPT-4o and Claude 3.5 Sonnet concentrated their ratings in a narrow band around 4.0-4.2, while the O1 models produced more diverse ratings (Figures 28-32, Appendix E.2.1).

3.4.3 The Implementation Stage: Viterbi-Style Unit-by-Unit Code Generation

Why not direct prompting? The paper argues, both formally and empirically, that a "direct prompting" approach to code generation is fundamentally inadequate for this task. In this approach, an LLM would be given the entire proposal and asked to generate the complete, working code for the modified block in a single shot. If the code fails any check (syntactic, semantic, or novelty-based), the entire output is discarded, and the process is retried. As formalized in Appendix A.1, if a design has N sub-components that must each be correct, and each has an independent probability p of being correctly generated, the probability of a single-shot success is p^N. The expected number of calls to the model is 1/p^N, which grows exponentially with N. The empirical consequence is stark: Table 4 shows the "Direct" strategy achieves a validity rate of only 6%, with an average of just 1.1 attempts generating trivial, short code (avg. 49 lines of function-body code, or LFC).

The Viterbi-style, recursive implementation process (Algorithm 3). The system instead implements the proposed modification by recursively building up the new GAU tree piece by piece. The process (visualized in Figure 8) maintains a list of Unimplemented GAUs, initialized with the root of the subtree to be modified. In each step, three specialized agents work in sequence:

Planner Agent: The Planner receives the current state of the GAU tree (which units are implemented, which are placeholders) and the overall proposal. It selects the next unimplemented unit to work on, prioritizing based on dependencies, and provides a high-level implementation plan and key points to the Coder agent. The Planner is explicitly instructed not to provide exact implementation details, only guidance. In practice (Table 4), ablating the Planner has minimal quantitative impact but is considered qualitatively important for guiding the implementation order.
Coder Agent: The Coder receives the plan, the parent design code, a pool of existing units available for reuse (retrieved via embedding-based search from the Evolution Tree), and the proposal. Its task is to generate a complete, compilable Python file for the selected GAU. This file must conform to the GAUBase template (Figure 16): it must contain the class implementation (with __init__ and _forward methods), at least one unit test decorated with @gau_test, and a CHILDREN_DECLARATIONS list that explicitly declares any new child GAUs the unit depends on as placeholders. The Coder can choose to either implement a unit fully or defer the implementation of its declared children to later steps. Crucially, once a unit passes checks, its code is "frozen" and never modified again—only its children's placeholders are filled in later. This is the core of the Viterbi-style search: partial progress is checkpointed.
Observer Agent: The Observer reviews the Coder's output for alignment with the proposal, code quality, novelty, and potential integration issues. It compares the generated code against a database of prior unit codes (using cosine similarity of embeddings) to flag potential self-replication. It provides a rating (threshold: 3/5) and detailed suggestions for improvement.

The Symbolic Checker (Table 1). After the Coder generates code and the Observer approves it, the code is passed to the Symbolic Checker, a non-LLM, rule-based system in the Verification Engine. This checker performs a sequence of static and runtime analyses to guarantee the unit's validity before it is frozen:

Static checks: An AST-based parser verifies syntactic correctness. A formatter verifies that the code adheres to the GAB/GAU protocol.
Functional checks: A PyTorch runtime check verifies that the module can be initialized, that a forward pass can be completed, and that a backward pass can be completed. A differentiability check ensures all parameters are used in the computation graph. A causality check verifies that the t-th output does not depend on inputs beyond the t-th position, by systematically perturbing future inputs and checking for invariance on a test tensor.
Efficiency/test training check: A quick 10-step training run on the Wikitext-2 dataset is launched. The checker monitors for exploding gradients, non-decreasing loss, and whether the FLOPs or step time exceed 5x that of a baseline GPT model. This catches numerically unstable but syntactically correct designs early.

The retry loop. If either the Observer rejects the code (rating < 3) or the Symbolic Checker fails, the state of the tree is rolled back to before the Coder's attempt, and the Planner, Coder, Observer cycle is retried for the same unit. This continues for up to K_fails = 5 attempts. If the unit cannot be implemented after 5 attempts, the entire proposal is abandoned and may be retried later up to K_attempts = 4 times before being deemed un-implementable. The step then repeats: the Planner selects the next unimplemented GAU, and the cycle continues until the Unimplemented list is empty, at which point the design is fully implemented and added to the Evolution Tree.

Formal advantage of the Viterbi-style approach (Appendix A.1). The expected number of LLM calls in the unit-based approach is the sum of the expected attempts for each of the N units: E[calls_VS] = Σ (1/p_k). Even if all steps have equal success probability p, this is N/p. In contrast, the direct approach has an expected cost of 1/p^N. This constitutes an exponential reduction in the expected number of model calls, which holds even when accounting for the growing token costs from the accumulating history context (Appendix A.1.2). The empirical confirmation is in Table 4: the full system achieves a 92% validity rate, versus 6% for direct prompting, and 73% when the unit-based generation (UG) strategy is ablated.

Evidence for the Viterbi-quality advantage (Appendix A.1.3). Beyond mere validity, the paper argues that the stepwise process improves the quality of the generated code. Because each step can involve multiple retries with detailed debug information and corrective instructions, the total number of "design tokens" (effective reasoning tokens that contribute to the final artifact) is strictly greater than in a single-shot approach. Under a monotonicity assumption (more design tokens lead to better quality), this implies higher-quality final artifacts. The evidence is the complexity of the generated code: the full system produces an average of 181 lines of function-body code (LFC) per unit, comparable to the human-written reference library's average of 220 LFC.

3.4.4 The Verification Stage: Ladder of Scales and Design Selection

The cost problem. Verifying a single architecture design requires pretraining it on a language corpus and evaluating it on downstream benchmarks, a process that takes hundreds of GPU-hours at even the 125M parameter scale (Table 19). With a total compute budget, the system cannot afford to fully verify every design at every scale; it must allocate its verification budget non-uniformly.

The Ladder of Scales (LoS) strategy (Figure 11, Algorithm 5). The LoS strategy is a budget allocation scheme that structures verification trials pyramidally. The total verification budget B_m is partitioned across N_S + 1 scales (e.g., 125M and 350M parameters), with the constraint that the number of trials β_{i+1} at the next larger scale is a fraction of the trials at the current scale, controlled by a selection ratio sr_i < 1 (e.g., β_{i+1} ≈ sr_i * β_i). In the paper's main experiment, this translates to:

~1,000 trials at 14M parameters (0.7B training tokens).
~5 trials at 350M parameters (50B training tokens).

The budget is released dynamically. A higher-scale budget β_{i+1} is initially set to zero and is only released when the used lower-scale budget exceeds 1/sr_i. This prevents all high-scale slots from being consumed early by the first available designs, ensuring that later, potentially better designs still have a chance to be verified at large scale. This dynamic allocation is formally defined in Algorithm 5.

Why this works. The LoS strategy is grounded in scaling laws (Kaplan et al., 2020; Tay et al., 2022a), which show that the relative performance of models is highly correlated across scales. A design that performs well at 14M parameters is likely, though not guaranteed, to perform well at 350M. The pyramid tests many hypotheses cheaply at the base, uses the results to select the most promising candidates, and then verifies this smaller subset at progressively larger scales. This mimics the methodology commonly used in research on small LMs (Hu et al., 2024; Lu et al., 2024b).

Verifier agent workflow. A Verifier agent operates in a loop. It first selects a design from the Evolution Tree using the same quadrant-based strategy described in §3.4.2, but with a key difference: it primarily exploits from the "Good & Unconfident" quadrant (designs with promising fitness from small scales that haven't been tested at larger scales) and explores from the "Poor & Unconfident" quadrant. Once a design is selected, the Verifier identifies the lowest unverified scale i for that design for which a LoS budget slot α_i > 0 is available. If none is, it either skips or selects another design. It then passes the design to the Verification Engine's Auto-Tuner (Table 7), which adjusts n_blocks and embed_dim to fit the target parameter count (within 80-120% of a reference GPT's parameters, prioritizing depth over width per Tang et al. (2024)), and determines gradient accumulation steps to avoid out-of-memory errors. Finally, the model is pretrained using the Hugging Face Trainer on the custom SmolLM-1/8-Corpus (a filtered high-quality subset of SmolLM, ~34B tokens), and evaluated on 29 selected LM-Eval benchmarks. A Runtime Checker (Table 8) monitors for divergence, exploding gradients, or errors during the run and terminates early if detected.

Updating fitness and confidence. Upon completion, the verifier writes the results back to the Evolution Tree, updating the design node's fitness (the aggregate performance) and incrementing its confidence (the number of completed scales). A design is marked as "erroneous" only if it fails verification on more than three separate V-Node machines, preventing transient hardware issues from disqualifying a potentially good design.

3.4.5 Formal Justification for the System's Design Decisions

The paper synthesizes three formal arguments (Appendix A) that together justify the core architectural choices of Genesys: the GAU factorization, the Viterbi-style implementation strategy, and the focus on generating a small number of high-quality designs.

1. Expressive completeness of the GAU tree (Appendix A.2). The Σ -> Σ type structure is not just a convenient interface; it's a guarantee. Theorem 4 proves that any program P: Σ -> Σ in a language closed under composition admits a finite unit tree decomposition. The proof proceeds by structural induction: atomic programs are trivial trees, and composite programs P_1 ∘ ... ∘ P_k form a root with the trees of each sub-program as children. Proposition 5 extends this to programs with more general types Q: X -> Y via a lifting construction with encoder/decoder pairs, showing that any architecture block can be represented in this framework. This guarantees the search space is complete.

2. Exponential advantage of Viterbi-style search (Appendix A.1). Lemma 1 and Lemma 2 formalize the expected number of calls for direct versus unit-based generation. Proposition 1 establishes the exponential advantage of the unit-based approach when the probability of single-shot success p_valid is the product of per-step probabilities ∏ p_k. Crucially, this advantage grows dramatically with the number of sub-components N. This explains why direct prompting fails (6% validity) and the unit-based approach succeeds (92% validity) for complex designs with N in the range of 3-5 (the median number of units per design is 5; Appendix E.3.2).

3. Synergy with "few high-quality samples" (Appendix A.3). In evolutionary search, there is a fundamental trade-off: one can either generate many cheap, low-quality trials (e.g., simple architectures with a low probability Q of being beneficial and a high probability E of being valid) or a few expensive, high-quality trials (complex architectures with higher Q and near-certain validity E ≈ 1 due to the Viterbi-style process). Proposition 6 shows that the yield of beneficial discoveries is proportional to (Q * E) / c, where c is the average cost per sample. Proposition 7 combines this with the exponential advantage of Viterbi-style search: by raising E from the direct approach's ∏ p_k to nearly 1, the effective Q*E term becomes exponentially larger, more than compensating for the increased per-sample cost c_VS. This provides a theoretical basis for the paper's design philosophy of "design more complex, and ultimately more expensive, agent systems that are more deliberate."

4. Key Insights and Innovations

Innovation 1: Discovery as a Factorizable Search Problem — The GAU Tree as a Unified Representation

The paper's most fundamental conceptual move is reframing the challenge of neural architecture discovery from the task of generating monolithic, opaque code strings to the task of searching over a structured, factorizable space of composable functional units. This is not merely a convenient engineering choice; it is a theoretical insight that changes what kinds of search algorithms are possible and guarantees completeness.

What the field did before. Prior work in neural architecture search (NAS) operates in fixed, hand-specified operation spaces — attention heads, convolution kernels, pooling layers — where the search algorithm selects from a pre-defined inventory and the space of possible architectures is bounded by the designer's imagination when constructing the search space (Elsken et al., 2019; White et al., 2023). Prior work in LLM-driven code generation (including for NAS, such as Chen et al., 2023) typically treats the output as a single, unstructured string to be generated in one shot and validated post-hoc. Both paradigms share a limitation: the structure of the search space is either imposed externally by human designers (in NAS) or not leveraged at all by the generation algorithm (in direct prompting). This makes search inefficient, modification non-local, and novel cross-paradigm combinations (e.g., merging attention from a transformer with state-space memory from Mamba) difficult to express without breaking the whole program.

What's distinctive about this work. The paper demonstrates — both formally and empirically — that by imposing a single type-theoretic constraint (all components are typed as Σ → Σ functions), the design space becomes automatically factorizable. This is not an ad-hoc decomposition; it is a provable property. Theorem 4 (Appendix A.2) shows that any program in a language closed under composition admits a finite unit tree, and Proposition 5 extends this to more complex type signatures via a lifting construction. The insight is that the type constraint is simultaneously restrictive enough to enable efficient GP operations (mutation, crossover) and decomposition (the Viterbi-style implementation) yet expressive enough to capture a wide range of architectures (transformers, SSMs, RNNs, hybrids). The empirical consequence is that the system discovers architectures with a median of 5 composable units (Appendix E.3.2), comparable in complexity to human designs, and that these units can be freely recombined — the crossover operation (Figure 5B) successfully merges units from Mamba2 and GPT into a novel hybrid that would be extremely fragile to construct via monolithic code splicing.

Significance beyond performance. This is a fundamental reframing of the architecture discovery problem, not an incremental improvement. It shifts the bottleneck from "can we generate valid code?" (which direct prompting shows is catastrophic — 6% validity) to "can we search effectively over valid configurations?" The factorization makes the search problem tractable by decomposing it along two axes: (1) a structural axis (the GAU tree), which enables local modifications with guaranteed validity, and (2) a procedural axis (the Viterbi-style implementation), which guarantees exponential efficiency in generating valid code. The universality proof means this framework is not specific to language models; it generalizes to any domain where the artifacts of interest can be typed as compositions of Σ → Σ functions, which covers a broad class of modular deep learning systems. This turns an empirical observation ("factorizing code helps") into a principled design rule with formal guarantees.

Evidence grounding. The theoretical claim is anchored in the formal proofs of Appendix A.2 (completeness of the representation) and Appendix A.1 (exponential efficiency of Viterbi-style generation). The empirical claim is anchored in Table 4, where the unit-based generation strategy achieves 92% validity versus 6% for direct prompting, and in Table 5, where discovered hybrid architectures (e.g., HMamba, which combines hierarchical state-space modeling with a double-layer Mamba design) achieve competitive performance against pure-breed human designs.

Innovation 2: The Viterbi-Style Code Generation as a Principled Solution to the Multi-Constraint Problem

The paper identifies and formally characterizes a failure mode that is pervasive in LLM-driven code generation but rarely diagnosed with precision: the exponential penalty for simultaneous constraints. When a generated artifact must satisfy N independent sub-constraints (syntactic validity, differentiability, causality, numerical stability, novelty), and generation is treated as a single-shot Bernoulli trial, the expected number of attempts grows as 1/p^N, which becomes prohibitive for even modest N. This is not a limitation of a particular model or prompt — it is a structural property of the direct generation approach.

What the field did before. Both the LLM code generation literature and the prompt-engineering community have implicitly treated generation failures as a model capability problem to be solved by better models, better prompts, or better post-hoc filtering. The dominant paradigm is "generate, test, retry," where failures are handled by regenerating the entire artifact from scratch with modified instructions (Figure 9A). Prior GP + LLM systems (Hemberg et al., 2024; Romera-Paredes et al., 2024) also generate complete programs, albeit with iterative improvement across generations. The concept of decomposing the generation procedure itself to match the artifact's compositional structure — checkpointing partial successes and only retrying failed sub-components — has not been articulated as a formal algorithmic principle with provable efficiency guarantees in this domain.

What's distinctive about this work. The paper doesn't just implement a multi-step generation pipeline; it formalizes why this pipeline is exponentially better than the default approach. Lemmas 1 and 2 (Appendix A.1.1) establish the expected calls for direct versus Viterbi-style generation, and Proposition 1 proves the exponential advantage when the single-shot success probability factorizes into per-step probabilities. The Viterbi analogy — freezing the most probable partial solution at each step before proceeding — makes explicit the connection to the classic decoding algorithm and grounds the design in a well-understood computational principle rather than ad-hoc engineering. The refinement in Appendix A.1.2 extends this to token cost models, showing the advantage persists when accounting for the growing history context.

This framing has a second, subtler consequence: it implies that the structure of the generation process should mirror the structure of the artifact being generated. The GAU tree isn't just a convenient representation for search; it's a roadmap for generation. Each node in the tree becomes a checkpoint, and the recursive decomposition of the block into sub-units maps directly to the recursive decomposition of the generation task. This is a design principle — match procedural decomposition to structural decomposition — that is absent from prior work and generalizable beyond architecture discovery.

Significance beyond performance. This is a diagnostic reframing of a widespread problem. It explains why direct prompting fails catastrophically (6% validity) — not because LLMs are bad at code generation, but because the probability of simultaneous success decays exponentially with complexity. It explains why ablation of the Symbolic Checker causes the most drastic validity drop (from 92% to 30%, Table 4) — because the checker acts as a per-unit gate that prevents incorrect partial work from being propagated, which is essential for the Viterbi property. And it explains why the unit-based strategy without UG still achieves 73% validity — because even a partial decomposition of the generation task provides some of the checkpointing benefit. This diagnostic moves the conversation from "how can we write better prompts?" to "how can we decompose the generation task to match the artifact's structure?" — a more principled and scalable question.

Evidence grounding. The formal argument is in Appendices A.1.1-A.1.3. The empirical evidence is Table 4, where the direct approach achieves 6% validity with an average of 49 LFC (small programs), while the full Viterbi-style system achieves 92% validity with an average of 181 LFC (complex, human-competitive programs). The ablation of the unit-based strategy (No UG) dropping to 73% validity and 75 LFC directly supports the claim that structural decomposition, not just iterative refinement, drives the gain.

Innovation 3: Evolutionary Fitness as a Two-Dimensional Signal — Decoupling Performance from Confidence

In standard evolutionary algorithms and most NAS approaches, the fitness of an individual is a scalar value — the performance on the target task — and selection operates on this scalar alone. The paper introduces a second, independent dimension to fitness: confidence, defined as the number of scales at which the design has been verified. This transforms selection from a one-dimensional problem (pick the best performer) to a two-dimensional one (pick the best performer given what we know), enabling a principled separation of exploitation and exploration.

What the field did before. Evolutionary NAS and GP approaches (Real et al., 2020; Chen et al., 2024) typically evaluate every individual at a fixed scale and use the resulting scalar fitness for parent selection, tournament selection, or truncation selection. The exploration-exploitation trade-off is managed implicitly — through population diversity, mutation rates, or tournament size — rather than being explicitly parameterized by an individual's evaluation history. In Bayesian optimization and multi-fidelity optimization, the concept of uncertainty in fitness estimates is formal (e.g., via Gaussian process posteriors), but these methods typically select the next evaluation (acquisition function) rather than incorporating uncertainty into the parent selection for evolution.

What's distinctive about this work. The quadrant-based selection strategy (Figure 10, Algorithm 4) makes the two-dimensional fitness signal first-class. Designs are not just "good" or "bad"; they are "good and we know it" (exploitation targets for further improvement), "good but we're not sure" (exploitation targets for verification at larger scales), "bad and we know it" (exploration targets — can a mutation salvage something?), and "bad but we're not sure" (opportunistic exploration for verification). This framing is novel in the evolutionary NAS context and has practical consequences: designers and verifiers use different quadrants as their primary targets, enabling the distributed architecture where designer and verifier agents operate asynchronously without conflict. The designer primarily exploits Good & Confident designs (refining proven winners) while the verifier primarily exploits Good & Unconfident designs (confirming promising candidates at scale). This decoupling means verification doesn't stall ideation and vice versa.

Significance beyond performance. This is a conceptual innovation in evolutionary search that is particularly well-suited to domains where fitness evaluation is expensive, stochastic, and multi-fidelity (as in architecture discovery with pretraining). The insight generalizes: any problem where evaluation cost varies and intermediate evaluations provide noisy-but-correlated signal can benefit from making uncertainty about fitness an explicit selection axis. The fact that the "w/o Exp." ablation (Table 2, removing experimental feedback and fitness-based selection entirely) reduces fitness improvement by a factor of ~2.4× and Sharpe ratio by a factor of ~5.7× demonstrates that the selection strategy is not just a theoretical convenience — it is critical for stable evolutionary progress. The "hubness" pattern in the evolutionary tree (Figure 13, Appendix E.1.2) — where a few highly-fit designs become central hubs that most subsequent designs branch from — is an emergent consequence of this selection strategy that mirrors the structure of scientific citation networks (Wu et al., 2009), suggesting the system is recapitulating a pattern of cumulative scientific progress.

Evidence grounding. Table 2 shows the impact of removing experimental verification (w/o Exp.): the end fitness improvement drops from 4.10% to 2.20%, and the Sharpe ratio drops from 0.69 to 0.263. Table 3 shows that w/o Exp. designs have a ~19 percentage point higher error rate during verification (27.31% vs. 8.61%), indicating that poor designs are being selected for expensive verification — a direct consequence of losing the confidence signal. Figure 13 visualizes the structural difference: the w/o Exp. tree is randomly connected, while the full tree shows the "hubness" pattern characteristic of selection-driven evolution.

Innovation 4: The Ladder of Scales — Budget-Aware Verification as a Dynamic Allocation Problem

The paper introduces a budget allocation strategy for multi-fidelity evaluation that is structurally distinct from both the fixed-fidelity evaluation of standard NAS and the acquisition-function-based sequential selection of Bayesian optimization. The Ladder of Scales (LoS) strategy (Figure 11, Algorithm 5) formalizes the intuition — familiar to human researchers — that one should test many ideas cheaply and only a few ideas thoroughly, but it does so with a dynamic release mechanism that prevents early depletion of high-fidelity evaluation slots.

What the field did before. Small-scale LM research (Hu et al., 2024; Biderman et al., 2023) uses multi-scale evaluation, but typically as a fixed protocol: train at a few predetermined scales and compare. NAS with weight-sharing (Elsken et al., 2019) avoids the multi-scale problem entirely by evaluating architectures within a single supernetwork. Bayesian optimization (and multi-fidelity BO) uses acquisition functions to sequentially select both the next candidate and the fidelity level at which to evaluate it, but this requires maintaining a surrogate model and typically operates in a much smaller search space. None of these approaches addresses the specific challenge faced by an asynchronous, parallel evolutionary system where multiple verifier agents concurrently select designs for evaluation and the high-fidelity budget must be protected from early exhaustion by the first-arriving candidates.

What's distinctive about this work. The LoS strategy's key innovation is the dynamic release mechanism: higher-scale budgets are initially zero and are only released when the corresponding lower-scale budget is used. This decouples the rate of design generation from the rate of high-fidelity evaluation — the budget at scale i+1 is a function of cumulative lower-scale usage, not wall-clock time or the number of designs in the queue. This ensures that a late-arriving, potentially superior design still has a chance at large-scale verification, even if the system has been running for a long time and many designs have already been evaluated. This is a significant practical contribution for any long-running autonomous discovery system where the quality of designs is expected to improve over time.

The paper also provides an analysis of the optimal ratio of designer to verifier nodes (Appendix E.4.2). By modeling the pipeline as a two-stage queue with rates ND/TD (design throughput) and NV/TV (verification throughput), the optimal ratio r* = TV/TD balances the pipeline and prevents either stage from bottlenecking the other. For the paper's hardware configuration, this works out to approximately 2 design threads per verifier node, a concrete operational insight that is grounded in the system's empirical timing characteristics.

Significance beyond performance. This is an operational innovation that addresses a real bottleneck in scaling autonomous discovery: the tension between the desire for comprehensive evaluation (test every design at every scale) and the reality of limited compute. It provides a principled, parameterizable strategy (controlled by the selection ratio sr_i) for navigating this trade-off, and the dynamic release mechanism solves a subtle but important race condition that would otherwise undermine the fairness of the evolutionary process. The strategy is not specific to architecture discovery; it applies to any multi-fidelity search where performance correlates across fidelities and the budget at higher fidelities is constrained.

Evidence grounding. The LoS strategy is not independently ablated (doing so would require an alternative budget allocation scheme at equal total compute, which the paper does not compare against), but its downstream consequences are visible in the system's ability to scale: 1,062 designs are fully verified, with ~1,000 trials at 14M parameters and ~5 at 350M (Section 1, Section 4.3). The evolutionary progress over time (Figure 12, Table 2) — fitness continues to improve up to 1,000 designs — would be impossible without a budget allocation strategy that reserves large-scale slots for later, higher-fitness designs. The V-D ratio analysis (Appendix E.4.2) provides practical evidence for the throughput model.

Innovation 5: Discovery as a Deliberate, Literature-Grounded Process — A Negative Result for Pure Evolutionary Search

The paper's commitment to modeling the full research process — not just the artifact generation — produces an important negative result: removing the literature search and proposal review stages measurably degrades the stability and trajectory of evolutionary progress, even when the downstream code generation and verification pipelines remain unchanged. This establishes that the "research overhead" (literature review, adversarial review, proposal writing) is not cosmetic scaffolding but a functional component of the discovery pipeline.

What the field did before. LLM-driven ASD systems (AI Scientist, AgentLab) and LLM + GP systems (Romera-Paredes et al., 2024) typically include literature access and some form of ideation, but the contribution of these stages to downstream discovery outcomes has not been systematically isolated. The implicit assumption is that these stages improve the semantic coherence of the artifacts (they read like real research) but that the empirical quality of the artifacts is determined primarily by the search algorithm and verifier. The literature and proposal stages are valued for interpretability and human-in-the-loop interaction, not necessarily for their causal impact on fitness.

What's distinctive about this work. The ablation results in Table 2 directly contradict this assumption. Removing literature access (w/o Lit.) reduces the Sharpe ratio from 0.69 to 0.567 and increases the Maximum Drawdown (worst fitness decline) from -0.38% to -0.62% — meaning the evolutionary process is less stable and more prone to regressions. Removing experimental verification (w/o Exp.) is catastrophic (Sharpe ratio drops to 0.263), confirming that fitness-based selection is essential. But the key negative result is the Base variant: when access to the accumulated design tree is removed and search is limited to only the five seed designs (with the w/ Mem variant allowing new designs to be used only as background references, mimicking the Romera-Paredes et al. (2024) setting), the end fitness improvement is negligible (0.01%) and the Sharpe ratio is near zero (0.002). This means that genealogical accumulation — building on previous generations of discovered designs, not just the original seeds — is the primary driver of progress, and simple re-sampling from a fixed pool of background knowledge does not replicate it.

Significance beyond performance. This is a diagnostic negative result that clarifies where the value in the system resides. It's not in the LLMs' raw ability to generate code (which is unreliable without the Viterbi-style process), and it's not in the elegance of the GP operations (which are standard). The value is in the coevolution of the population and the knowledge it generates: each generation of designs produces both better architectures and richer context for the next generation's proposals. The literature review anchors novel proposals in existing knowledge, preventing random drift; the adversarial review filters out ideas that are self-plagiarizing or poorly grounded; and the accumulated tree provides a diverse but validated pool of building blocks for crossover. The w/ Mem result (inclusion of past designs as references only, without genealogical selection) shows that merely having access to past artifacts is insufficient — the selective pressure from the evolutionary tree structure (parent-child relationships, fitness-based quadrant selection) is what converts knowledge accumulation into fitness improvement.

Evidence grounding. Table 2 provides the full ablation: Full (∆ = 4.10%, SR = 0.69), w/o Lit. (∆ = 3.37%, SR = 0.567), w/o Exp. (∆ = 2.20%, SR = 0.263), Base (∆ = 0.01%, SR = 0.002), Base w/ Mem (∆ = 2.81%, SR = 0.196). The difference between Base w/ Mem and Full (∆ improvement of 1.29 percentage points, SR improvement of 0.494) isolates the contribution of the evolutionary tree's active selection pressure over passive knowledge accumulation.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021), a dataset of high-school competition-level mathematics problems. The paper adopts the specific split from Lightman et al. (2022): 12,000 training questions and 500 test questions. MATH was chosen because test-time compute is expected to help most "when the model already possesses the necessary knowledge and the challenge is drawing complex inferences," and mathematical reasoning fits this profile (Section 4).
Base model(s). All experiments use PaLM 2-S (Codey)* (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful performance regime: non-trivial performance on MATH (roughly 10–19% pass@1 depending on prompt and sampling configuration) but far from saturation, leaving room for test-time compute to improve outcomes. For the FLOPs-matched comparison, a second model with approximately ~14× more parameters is used as the pretraining-scaled baseline. The 14× larger model uses greedy decoding with no additional test-time compute.
Metrics. The primary metric is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). For difficulty-dependent analysis, accuracy is reported within each of the five difficulty quintiles separately.
Baselines. The paper compares against:
1. Majority voting: selecting the most common final answer among N sampled solutions, with no learned verifier.
2. ORM best-of-N weighted: scoring N solutions with an outcome reward model and applying best-of-N weighted selection.
3. PRM best-of-N weighted: scoring N solutions with the process reward model and applying best-of-N weighted selection (this is the primary verifier-based baseline for search experiments).
4. Parallel sampling (for revision experiments): generating N independent solutions from the revision model and selecting the best via verifier or majority voting.
5. Greedy decoding from the ~14× larger model: the baseline for the FLOPs-matched pretraining comparison.
Generation budget / compute accounting. The universal unit of test-time compute is a generation — one complete sampled answer from the base LLM. For beam search and best-of-N, the budget equals the number of beams or samples N. For lookahead search with k lookahead steps, the cost is N × (k + 1) to account for the additional rollout computation. For revisions, the budget is the total number of generated solutions across all chains and parallel samples. Budgets are swept across powers of 2, typically from 1 to 512 generations. For the FLOPs-matched comparison, pretraining FLOPs are estimated as X = 6 N D_pretrain and inference FLOPs as Y = 2 N D_inference (standard approximations from the scaling laws literature).
Cross-validation / statistical protocol. To avoid the circularity of selecting the best strategy and evaluating it on the same data, the paper uses two-fold cross-validation within each difficulty bin on the 500-question test set. The best-performing strategy is selected on one fold and evaluated on the other, with results averaged. Difficulty bins are estimated using 2,048 samples per question, with the ground-truth correctness rate (oracle) or the PRM's predicted correctness rate (predicted) used to partition questions into quintiles.

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

The headline finding for verifier-guided search is that beam search significantly outperforms best-of-N at low generation budgets but its advantage diminishes or reverses at high budgets, and this behavior is strongly difficulty-dependent. With compute-optimal difficulty-conditioned allocation, the efficiency gain over best-of-N reaches ~4× (e.g., matching best-of-N at 64 generations with only 16 generations; Figure 4).

Aggregate search performance (Figure 3, left). Across all 500 MATH test questions:

At low budgets (2–8 generations), beam search with M = 4 significantly outperforms best-of-N weighted. At 4 generations, beam search achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted.
At high budgets (64–256 generations), beam search performance flattens and falls slightly below best-of-N weighted. Best-of-N weighted reaches approximately 38% at 512 generations; beam search (M = 4) plateaus around 34%.
Lookahead search (both k = 1 and k = 3) generally underperforms all methods at the same budget due to its higher per-step cost consuming the generation budget. The 3-step lookahead variants never surpass simpler methods.
Majority voting trails substantially, reaching only about 29% at 512 generations.

Difficulty-dependent behavior of search (Figure 3, right). Breaking results down by difficulty quintile (beam search M = 4 vs. best-of-N weighted) reveals the core pattern:

Bin 1 (easiest): Beam search accuracy decreases from roughly 78% to 77% as budget increases from 4 to 256, while best-of-N weighted increases from 68% to 88%. This is clear evidence of PRM over-optimization: aggressive search finds solutions that exploit the verifier signal.
Bin 2: Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets.
Bin 3 (medium): Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations.
Bin 4 (medium-hard): Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations.
Bin 5 (hardest): Both methods hover near 1–3% regardless of budget. No method makes meaningful progress.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level:

At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a ~4× compute reduction.
At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).
Compute-optimal with predicted difficulty bins tracks the oracle version closely, with the curves "largely overlapping" per the authors (Figure 4). The predicted version reaches approximately 37% at 256 generations.
Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%).

PRM vs. ORM scaling (Figure 14, Appendix F). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples, confirming the PRM's superior scaling properties.

Revision Model Results (Section 6)

The headline finding for revisions is that sequential revisions marginally outperform parallel sampling in aggregate, but the optimal ratio of sequential-to-parallel sampling is strongly difficulty-dependent, with easy problems benefiting from purely sequential refinement and hard problems requiring a balanced mix. Compute-optimal ratio selection achieves ~4× efficiency improvement over the parallel best-of-N baseline (Figure 8).

Revision model pass@1 trajectory (Figure 6, left). Starting from approximately 18.2% pass@1 at step 1, the revision model's per-step accuracy improves to roughly 24–25% by steps 15–20 and remains in the 23–25% range out to 64 steps. The model generalizes beyond its 4-step training horizon, demonstrating that the revision skill transfers to longer chains.

Sequential vs. parallel aggregate comparison (Figure 6, right). At 64 generations:

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35% Sequential sampling outperforms parallel under both selection mechanisms.

Difficulty-dependent optimal ratio (Figure 7, right). At a fixed budget of 128 generations:

Bin 1: Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to allocation strategy.
Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel.
Bin 3: A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2:1 to 8:1), reaching approximately 42% vs. 35% at the extremes.
Bin 4: Similar pattern, with the peak at a moderate ratio achieving roughly 18% vs. 14% at fully parallel.
Bin 5: All ratios produce roughly 2–3% accuracy. No allocation strategy helps.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin:

At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a ~4× improvement in compute efficiency.
At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
Compute-optimal with predicted bins performs slightly below oracle bins at high budgets (approximately 41% at 256 generations) but still substantially outperforms the parallel baseline.
The parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

The headline finding for the FLOPs-matched comparison is that test-time compute with a smaller model can outperform a ~14× larger model, but only on easy-to-medium problems and primarily when the ratio of inference to pretraining tokens is low (R ≪ 1). On hard problems, pretraining is almost always more effective, and test-time compute provides essentially zero benefit.

Revisions vs. pretraining (Figure 9, left; Figure 1). Comparing PaLM 2-S* with compute-optimal revisions against the ~14× larger model:

At R ≪ 1 (0.16): Test-time compute outperforms across all difficulty levels, with a +27.8% relative improvement on medium questions (bins 2–3).
At R ≈ 1 (0.79): Test-time compute remains preferable on easy and medium problems (+3.5% and +16.7% relative, respectively) but loses on hard problems (bins 4–5).
At R ≫ 1 (22): Test-time compute only remains preferable on easy questions, with hard questions showing a -37.2% relative disadvantage.

PRM search vs. pretraining (Figure 9, right; Figure 1). The comparison is starker:

At R ≪ 1: Test-time compute shows a +19.1% advantage on easy questions but only breaks even on medium (0.0%) and loses slightly on hard (-3.6%).
At R ≈ 1: Test-time compute is at a substantial disadvantage on medium (-35.3%) and hard (-35.3%) problems.
At R ≫ 1: Disadvantages grow to -30.8% on medium and -52.9% on hard problems. Only easy questions remain neutral (+2.0%).

Figure 9 detail. The line plots show accuracy per difficulty bin as test-time compute scales. The ~14× larger model's greedy performance is plotted as stars at three x-axis positions corresponding to the three R values. Where the compute-optimal scaling line lies above a star, test-time compute wins. On bin 1 (easiest, purple line), the scaling line is above all three stars for revisions. On bin 5 (hardest, blue line), the line is below all three stars and essentially flat near 0–5%, confirming that no amount of test-time compute helps on the hardest problems.

Ablation Studies and Robustness Checks

PRM step-wise aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise aggregation methods: "Last" achieves roughly 37% at 256 samples, "min" achieves roughly 35%, and "prod" achieves roughly 27%. An ORM baseline scores roughly 34%. The "last" aggregation's superiority is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM. The authors interpret this as evidence that step-level PRM training provides beneficial representation learning even when intermediate predictions are not directly used.
PRM vs. ORM scaling comparison (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%, and the gap continues to grow. This confirms that per-step supervision, even with last-step aggregation, produces a better verifier than outcome-only supervision.
Revision model verifier distribution shift (Appendix J, Figure 15a). The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs: sequential + base-LM PRM achieves roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern: a verifier trained on base model outputs does not transfer perfectly to revision model outputs.
Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (approximately 1–2 percentage points at 64 generations), but both variants outperform the parallel baseline, confirming that the sequential sampling benefit is not solely attributable to the verifier seeing more context.
Oracle vs. predicted difficulty bins (Figures 4, 8, Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (roughly 41% vs. 44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4, curves "largely overlap"). This is the critical robustness check confirming the compute-optimal strategy works without ground-truth labels.
Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This confirms the ratio effect is not an artifact of the verifier.
ReST^EM revision model optimization (Appendix K, Figure 16). An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) backfires substantially: additional sequential revisions degrade performance rather than improving it. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly. This is a notable negative result highlighting the sensitivity of revision training to the data generation procedure.
Beam search and lookahead search budget degradation (Figure 3, left vs. right). At high budgets, beam search plateaus and sometimes declines, while lookahead search — the strongest optimizer — paradoxically performs worst overall. Appendix M provides qualitative examples (Figure 29) showing search producing degenerate outputs: repetitive low-information steps at the end of solutions and overly short 1–2 step solutions that score highly under the PRM but are incorrect. This confirms that verifier over-optimization is a genuine failure mode, not a statistical artifact.

Critical Assessment

Does the evidence actually demonstrate a 4× efficiency improvement over best-of-N?

The claim that compute-optimal scaling achieves a 4× efficiency gain over best-of-N is supported by specific data points: Figure 4 shows compute-optimal search at 16 generations matching PRM best-of-N at 64 generations, and Figure 8 shows compute-optimal revisions at 64 generations matching parallel best-of-N at 256 generations. These are genuine comparisons at equivalent accuracy thresholds.

However, there are two important caveats:

First, the 4× figure does not amortize the cost of difficulty estimation. The difficulty bins used to select the optimal strategy require generating 2,048 samples per question (for predicted bins) or accessing ground-truth labels (for oracle bins). Generating 2,048 samples per question is enormously expensive — comparable to or exceeding the largest test-time budgets studied. The paper acknowledges this cost explicitly (Section 3.2) but does not include it in any budget calculation. The reported 4× efficiency gain is therefore computed after difficulty is known, not including the cost of learning it. In a realistic deployment, difficulty estimation would need to be amortized across many queries or done via a cheaper method (e.g., a lightweight classifier trained on question text, which the paper suggests as future work but does not develop). Until the estimation cost is accounted for, the 4× figure represents an upper bound on achievable efficiency, not a realized deployment gain.

Second, the 4× figure is most reliable in the low-to-moderate compute regime. At the highest budgets (256–512 generations), the gains narrow somewhat — particularly with predicted difficulty bins, where the revision curve in Figure 8 shows the oracle version at ~44% and the predicted version at ~41%, a non-trivial gap. This suggests the difficulty estimation method introduces some variance that matters more at higher budgets, possibly because the bin boundaries become less reliable when the difference between adjacent strategies shrinks.

Does the evidence demonstrate that test-time compute can substitute for a 14× larger model?

The FLOPs-matched comparison in Figure 9 and the bar charts in Figure 1 provide direct evidence, but with sharp, explicit boundary conditions that the paper acknowledges. The claim holds for easy-to-medium problems when R ≪ 1 (the inference-to-pretraining token ratio is small), with relative improvements ranging from +11.8% to +27.8% for revisions. As R increases or problem difficulty increases, the advantage shrinks and eventually reverses. On the hardest problems (bin 5), test-time compute provides essentially zero benefit across all R values — accuracy remains near 0–5% regardless of budget.

Several aspects of this comparison warrant scrutiny:

The 14× larger model baseline is not compute-optimal. The paper scales parameters only while holding training data fixed, following the LLaMA paradigm rather than Chinchilla-optimal scaling (where both parameters and data are scaled). A compute-optimally trained larger model would likely perform better, making the pretraining baseline weaker than it optimally could be. The paper acknowledges this explicitly (Section 7) and leaves the Chinchilla-optimal comparison to future work.

The larger model uses only greedy decoding. No majority voting, no best-of-N, no search augmentation is applied to the larger model. This is a deliberate choice to isolate the test-time compute versus pretraining compute trade-off, but it makes the baseline substantially weaker than a fair deployment scenario where some test-time compute would typically be allocated to the larger model as well. A comparison where both models receive compute-optimal test-time strategies would be more informative but more complex to design and interpret.

The R values are specific to the MATH benchmark and PaLM 2 model sizes. The ratio R = D_inference / D_pretrain depends on the specific pretraining token count of the base model and the inference tokens generated per query. These numbers are particular to PaLM 2-S* and the MATH problem lengths. For models with different pretraining budgets or tasks with different typical output lengths, the R values would shift, and the condition under which test-time compute is preferable would change.

Single model family, single benchmark. All FLOPs-matched comparisons use PaLM 2 models on MATH. The difficulty-dependence pattern — test-time compute helps on easy-to-medium problems, fails on hard ones — may be general, but the specific thresholds and the magnitude of the advantage at different R values could be model-specific. A model with different calibration properties, different in-context learning capabilities, or different base performance on MATH might exhibit different substitution characteristics.

Does the evidence demonstrate that all components (literature, verification feedback, accumulated designs) are necessary?

Table 2 provides ablation results for the evolutionary process, showing that removing literature access (w/o Lit.), experimental verification (w/o Exp.), or access to accumulated designs (Base) systematically degrades fitness improvement and stability metrics. These results support the claim that all components contribute positively.

However, the ablation has important scope limitations:

The ablation is on the first 300 designs. For extended runs (500 and 1000 designs), only the Full and w/o Exp. configurations are compared. The w/o Lit. and Base variants are not evaluated at larger scales due to "computational constraints" (Section 5.1). It is possible that the gap between Full and w/o Lit. would narrow as the population grows, since the accumulated design tree itself becomes a source of knowledge that partially substitutes for external literature. Similarly, the Base variant's negligible improvement might partially recover with more generations if the re-sampling from seed designs eventually discovers good architectures through brute force. The evidence for the necessity of literature access and genealogical accumulation is therefore strongest for the early stages of discovery; the long-horizon contribution of these components is less certain.

The "w/o Exp." ablation removes not just experimental feedback but fitness-based selection. The ablated system selects designs randomly (or uniformly) rather than using the quadrant-based strategy. This conflates two distinct mechanisms: the information provided by experimental verification (knowing which designs perform well) and the selective pressure applied by fitness-based parent selection. A more fine-grained ablation — e.g., one that keeps fitness-based selection but reduces the number of verification trials — would help disentangle whether the stability gains come primarily from the information or from the selection strategy that uses it.

The metrics (Sharpe ratio, Maximum Drawdown) are borrowed from financial economics. While these metrics are well-defined and the paper justifies their use (Section 5.1), they are sensitive to the choice of population size and step size. Figure 19 (Appendix D.2) shows sensitivity analysis for these parameters, and the Full system's advantage persists across a range of values (population size >20, step size >10), but the absolute magnitude of the metrics shifts. The qualitative conclusion that Full outperforms the ablations is robust; the precise quantitative differences (e.g., SR = 0.69 vs. 0.567) are less so.

What would strengthen the paper's claims?

A cheap, amortized difficulty estimator. The current difficulty estimation costs 2,048 generations per question — prohibitive for deployment. Demonstrating that a lightweight classifier trained on question text alone can recover most of the compute-optimal gains would transform the 4× figure from an upper bound to a realistic deployment estimate. The paper flags this as future work but does not develop it.

Replication on a second model family and dataset. All experiments use PaLM 2-S* on MATH. Whether the difficulty-dependent strategy patterns (beam search for medium difficulty, best-of-N for easy, sequential revisions for easy) generalize to other models (e.g., Llama, Gemma) and other reasoning domains (code generation, logical reasoning, scientific QA) is untested. A secondary experiment on, for example, HumanEval with a different base model would substantially strengthen the generality claim.

Joint optimization of search and revisions. The paper studies PRM tree-search and iterative revisions independently but never combines them — the revision model is never used as the proposal distribution within beam search, and the PRM is never used to guide which revision chains to pursue. This is acknowledged in Section 8 as a key limitation. A combined experiment would test whether the complementary strengths of the two mechanisms (revisions improve proposal quality, search improves selection) yield gains beyond either alone.

A fairer pretraining baseline. The 14× larger model with Chinchilla-optimal pretraining (scaling both parameters and data) and some modest test-time compute budget (e.g., best-of-8) would be a stronger baseline against which to measure the test-time compute advantage. The current comparison against a parameter-only-scaled model with greedy decoding likely overstates the practical advantage of investing in test-time compute over pretraining.

Statistical reliability for the compute-optimal strategy selection. The 500-question test set split into five difficulty quintiles of ~100 each, further split by two-fold cross-validation, means the optimal strategy is selected based on ~50 questions per fold per bin. Confidence intervals on the compute-optimal scaling curves are not reported, making it difficult to assess whether the observed differences between the oracle and predicted curves, or between compute-optimal and best-of-N at specific budgets, are statistically significant or within the noise of a small evaluation set.

Assessment of the central claims vis-à-vis the evidence

The paper's most important claim — that the effectiveness of test-time compute strategies depends critically on problem difficulty, and that ignoring this dependence leaves substantial efficiency on the table — is strongly supported across search methods, revision strategies, and selection mechanisms. The difficulty-bin analyses in Figures 3 (right) and 7 (right) show qualitatively different, and sometimes opposite, effects of the same strategy at different difficulty levels, replicated across PRM search and iterative revisions. This is the most robust finding in the paper.

The claim that compute-optimal, difficulty-conditioned allocation recovers ~4× efficiency gains over uniform best-of-N is empirically supported but qualified by the unaccounted cost of difficulty estimation. The 4× figure is best understood as a proof-of-concept upper bound that would require amortization or cheaper estimation to realize in practice.

The claim that test-time compute with a smaller model can outperform a ~14× larger model is supported with well-defined boundary conditions (easy-to-medium problems, low R), and the paper is transparent about where the substitution fails. The strength of this finding is in its specificity, not its universality.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted For, Making the 4× Efficiency Gain an Upper Bound

The assumption or constraint. The entire compute-optimal scaling framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2,048 samples per question and averaging either ground-truth correctness (oracle bins) or the PRM's final-answer score (predicted bins) — is extraordinarily expensive. The paper acknowledges this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The 2,048 samples required for difficulty estimation consume more compute than the largest test-time budgets studied in the paper (256–512 generations). This cost is not included in any of the budget calculations or efficiency comparisons.

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty_estimation + strategy_execution, and the former could dominate the latter. For single-use queries, where difficulty must be estimated per-prompt, the net cost would be worse than simply applying uniform best-of-N — the 4× gain would invert. The gains would only be realizable in scenarios where difficulty estimation can be amortized across many similar queries (e.g., a fixed benchmark, a stationary problem distribution), but the paper does not characterize when such amortization is feasible or what the break-even point is.

What evidence exists in the paper. The paper does not measure or model the amortized cost of difficulty estimation. It does not evaluate alternatives, such as using a smaller number of samples (e.g., 4–8) for a coarse difficulty estimate, training a lightweight classifier to predict difficulty from question text, or dynamically estimating difficulty during the solution process itself. Section 3.2 acknowledges the gap but provides no empirical characterization of how the difficulty estimation cost scales with its accuracy.

Mitigation status. Not addressed. The paper explicitly flags this as future work (Section 8): "estimating difficulty... exploring the tradeoff between accuracy and cost of difficulty estimation, and training models to predict difficulty directly from the prompt." Until a cheap, reliable difficulty estimator is demonstrated, the 4× efficiency gain should be understood as a proof-of-concept upper bound, not a realized deployment gain.

6.2 Hard Problems (Bin 5) Receive Essentially Zero Benefit From Any Test-Time Compute Strategy

The assumption or constraint. The paper's framework partitions problems by difficulty based on the base model's pass@1 rate. The hardest quintile (bin 5) represents problems where the base model's pass@1 is near zero — it almost never produces the correct answer, even with 2,048 independent samples. The paper shows that for these problems, no test-time compute strategy — beam search, best-of-N, lookahead search, sequential revisions, or any combination thereof — produces meaningful improvement. Accuracy remains at roughly 1–3% regardless of compute budget, across all methods studied.

The consequence. This establishes a hard capability boundary: test-time compute can only amplify existing capability, not create it. If the base model's proposal distribution does not contain correct solutions at any non-trivial rate, search and revisions cannot recover them — there are simply no correct answers to find or refine. For problem classes outside the base model's training distribution or reasoning capabilities, increasing inference compute is futile, and pretraining a larger model (which can acquire new capabilities) is the only viable path. The paper is candid about this (Section 7 takeaway box), but it means the approach offers no path forward for genuinely novel or out-of-distribution reasoning.

What evidence exists in the paper. Figure 3 (right, bin 5) shows search methods achieving 1–3% accuracy at all budgets. Figure 7 (right, bin 5) shows the same for revisions. Figure 9 (bin 5 scaling lines) shows the compute-optimal strategy is essentially flat and near zero, lying well below the ~14× larger model's greedy performance at all values of R. The FLOPs-matched bar charts in Figure 1 show relative disadvantages of up to -52.9% for PRM search on hard problems at R ≫ 1.

Mitigation status. Not addressed, and fundamentally not addressable within this framework. The paper does not claim otherwise. This limitation is inherent to the proposal distribution + verifier framework: the verifier can only select among what the proposal distribution generates, and if the proposal distribution generates nothing correct, the verifier is powerless. The paper's contribution is precisely in characterizing this boundary, not solving it — it tells practitioners when to stop spending inference compute and invest in pretraining instead.

6.3 Search and Revisions Are Studied Independently; The Two Axes Are Never Combined

The assumption or constraint. The paper decomposes test-time compute into two independent mechanisms: modifications to the verifier (search against the PRM, Section 5) and modifications to the proposal distribution (iterative revisions, Section 6). However, the two mechanisms are studied in isolation. The revision model is never used as the proposal distribution within PRM-guided beam search, and the PRM is never used to guide which revision branches to pursue. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The paper's compute-optimal policy selects between search strategies and revision ratios independently per difficulty bin, but never considers a joint allocation — e.g., using the revision model's outputs as candidates for beam search, or using the PRM's per-step scores to decide when a revision chain should be abandoned and restarted.

The consequence. The paper's results represent a lower bound on what combined approaches could achieve. The two mechanisms have complementary strengths: revisions improve the quality of generated candidates (making the proposal distribution more likely to contain correct answers), while PRM search improves the selection among candidates (finding the correct answer among those generated). If, for example, the revision model is particularly effective at improving pass@1 on medium-difficulty problems (Figure 7, right), and beam search is particularly effective at finding correct answers among candidates on those same problems (Figure 3, right), the combination could yield gains beyond either alone. The paper provides no evidence about whether these gains would be additive, superadditive, or subadditive. A practitioner building on this work cannot know whether to prioritize one mechanism, deploy both independently, or invest in a tightly integrated combination.

What evidence exists in the paper. None. The paper provides separate analyses for search (Section 5) and revisions (Section 6) with separate compute-optimal policies. There is no experiment, ablation, or analysis of a combined system. Section 8 lists this as future work.

Mitigation status. Not addressed. The paper explicitly acknowledges this as a limitation and a direction for future work but provides no empirical or theoretical guidance on how the mechanisms would interact.

6.4 The Revision Model Correct-to-Incorrect Reversion Rate Is 38%, and Training Methodology Is Fragile

The assumption or constraint. The revision model is trained via supervised fine-tuning on sequences of incorrect answers followed by a correct answer, constructed from independently sampled solutions — not on-policy rollouts. A critical consequence of this training procedure is that the model is never shown examples where the current answer is already correct and should be preserved. As a result, at inference time, when the revision chain happens to produce a correct intermediate answer, the model will often incorrectly "revise" it into a wrong answer in the next step. Section 6.1 reports:

"approximately 38% of correct answers get converted back to incorrect ones"

The paper mitigates this with a selection mechanism — using majority voting or verifier-based selection across the entire chain to pick the best answer from any revision step, rather than always taking the final revision. However, this is a post-hoc patch, not a solution to the underlying model behavior.

Furthermore, the ReST^EM experiment (Appendix K, Figure 16) reveals that the revision training approach is fragile: attempting to further optimize the revision model using on-policy RL-style training (ReST^EM; Singh et al., 2024) substantially degrades performance rather than improving it. At 256 generations, fully sequential performance with the ReST^EM revision model drops to approximately 33.5%, compared to roughly 38.5% at the optimal ratio with the original revision model. The authors hypothesize:

"on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly"

The consequence. The practical implication is that deploying the revision model requires a selection mechanism that evaluates all steps in the chain — which incurs additional verifier calls and latency — rather than simply taking the final output. This adds complexity and compute overhead to what is otherwise a conceptually simple procedure (generate a chain, take the last answer). More fundamentally, the fragility of the training methodology means that the revision approach cannot be straightforwardly improved through standard RL fine-tuning — the positive results depend on specific, non-obvious design choices (offline data construction, edit-distance-based pairing of incorrect and correct answers) that may not transfer to other settings, models, or tasks.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The ReST^EM degradation is shown in Appendix K, Figure 16. The paper does not ablate the training data construction choices (edit-distance pairing vs. random incorrect answer selection, uniform vs. non-uniform sampling of the number of incorrect answers) to determine which are critical for performance.

Mitigation status. Partially addressed through the within-chain selection mechanism, which recovers the best answer across the chain. However, this is a compensatory strategy, not a fix for the underlying model behavior. The paper does not experiment with training the model to recognize and preserve correct answers (e.g., by including "no revision needed" examples in the training data), nor does it diagnose why ReST^EM degrades performance. Both are left as implicit future work.

6.5 All Results Are on a Single Benchmark (MATH) With a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper — PRM training, search algorithm comparison, revision model training, compute-optimal strategy selection, and the FLOPs-matched comparison — uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* (Anil et al., 2023) as the base model. The 500-question test set is split into five difficulty quintiles of ~100 questions each, further split by two-fold cross-validation for strategy selection, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper states (Section 4) that it believes PaLM 2-S* is "representative of the capabilities of many contemporary LLMs," but provides no evidence for this claim.

The consequence. Several aspects of the findings could be model- or domain-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution and error patterns. A model with different calibration (e.g., better calibrated confidence) or different typical errors (e.g., more systematic vs. more random) might exhibit different difficulty-dependent scaling curves and different optimal strategy allocations.
The revision model's effectiveness depends on the base model's in-context learning capabilities, which vary substantially across model families (e.g., Llama vs. GPT vs. Gemini). A model that is better or worse at learning from in-context incorrect examples might show a different optimal sequential-to-parallel ratio or different absolute gains from revisions.
The difficulty bins are defined relative to the base model's pass@1 rate on MATH. These bins are model-specific: a problem that is "hard" for PaLM 2-S* (bin 5) might be "medium" for a stronger model. The observation that beam search helps on medium problems and hurts on easy ones might hold relative to each model's own capability threshold, but the absolute difficulty boundaries would shift.
MATH is exclusively competition-level math requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to other reasoning domains — code generation (which has different error modes), logical reasoning, scientific QA — or to tasks requiring factual knowledge rather than pure inference, where the model either knows the answer or doesn't, and iterative refinement is less meaningful.

What evidence exists in the paper. None outside the MATH + PaLM 2-S* setting. The paper does not compare against a second model family or a second benchmark. The 500-question test set, while standard for MATH, provides limited statistical power when further partitioned by difficulty: the per-bin sample sizes of ~100 (split in half by cross-validation) mean that the selected strategies are based on small samples, and no confidence intervals are reported on the compute-optimal scaling curves.

Mitigation status. Acknowledged implicitly (the paper does not claim universality) but not addressed. Replication on code generation benchmarks (HumanEval, MBPP) or with an alternative base model family would substantially strengthen the generality claims but is absent. The authors state the model is "representative" (Section 4) but provide no supporting evidence.

6.6 The FLOPs-Matched Comparison Uses a Weak Pretraining Baseline: Parameter-Only Scaling With Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a ~14× larger model. The larger model is constructed by scaling parameters only while holding training data fixed (following the LLaMA paradigm), and it is evaluated using greedy decoding with no test-time compute augmentation. The paper acknowledges this deviation from optimal practice in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

A Chinchilla-optimal model (Hoffmann et al., 2022), where both parameters and data are scaled, would likely achieve better performance at the same total pretraining FLOPs. Additionally, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or compute-optimal allocation) would create a substantially stronger baseline.

The consequence. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy-to-medium questions for revisions at R ≪ 1 (Figure 1) — are likely overstated relative to what a fairer comparison would show. The paper is comparing an optimized inference strategy (compute-optimal allocation) against an unoptimized pretraining strategy (parameter-only scaling) with an unoptimized inference strategy (greedy decoding). This is a reasonable first comparison but conflates the choice of where to spend compute (pretraining vs. inference) with the choice of how to spend it (optimal vs. naive allocation). The correct counterfactual for a practitioner is: "given a total compute budget, should I train a larger model with some inference strategy, or a smaller model with a more aggressive inference strategy?" The paper answers a narrower question: "given a total compute budget, should I train a larger model with greedy decoding, or a smaller model with compute-optimal test-time strategies?"

What evidence exists in the paper. The paper explicitly documents both aspects of the weak baseline: the parameter-only scaling (Section 7) and the greedy decoding. No comparison against a Chinchilla-optimal larger model or against a larger model with any test-time compute augmentation is provided.

Mitigation status. The paper acknowledges the parameter-only scaling issue explicitly and defers the Chinchilla-optimal comparison to future work. The greedy decoding choice is not addressed as a limitation — the paper treats the larger model's inference strategy as an immutable part of the "pretraining" option, which is a framing choice rather than a technical necessity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM-driven scientific discovery from a focus on open-ended generation—where the quality and novelty of output is hard to verify—toward a paradigm where discovery is framed as a structured search over a formally-guaranteed compositional space. It is not an incremental improvement to existing automated scientific discovery (ASD) or neural architecture search (NAS) pipelines; it is a methodological reframing grounded in a specific formal insight: imposing a single type-theoretic constraint (Σ → Σ on all components) simultaneously makes the search space factorizable (enabling local GP operations with guaranteed validity) and enables a Viterbi-style code generation procedure that is exponentially more efficient than the single-shot "generate, test, retry" approach that dominates LLM-based code generation.

The magnitude of this shift is substantial within the specific domain of automated architecture discovery. Prior to this work, the field lacked any systematic demonstration that LLMs could autonomously produce genuinely novel, competitive architectures at scale—not just recombine known operations in a fixed NAS space. The paper's empirical results establish a baseline that did not previously exist: 1,062 fully verified novel designs, with the best outperforming GPT-2, Mamba2, and other strong human baselines on 6 of 9 benchmarks (Table 5). The 86 percentage point gap between direct prompting (6% validity) and the unit-based generation approach (92% validity) in Table 4 is not just a large effect size; it is evidence that the naive approach most systems would default to is fundamentally non-viable for problems of this complexity, and that the structured decomposition is what makes the problem tractable. This is a diagnostic contribution: it identifies why prior attempts at LLM-driven complex code generation failed, and it provides a principled, formally-justified solution rather than ad-hoc prompt engineering.

This work also reconciles a latent tension between two research communities. On one side, the NAS community (Elsken et al., 2019; White et al., 2023; Real et al., 2020) has developed sophisticated search algorithms but operates in constrained, human-specified operation spaces. On the other side, the LLM-driven ASD community (Lu et al., 2024a; Yamada et al., 2025; Schmidgall et al., 2025a) has demonstrated ambition for end-to-end discovery but has largely focused on open-ended tasks where verification is ambiguous. Genesys demonstrates that the combination—using LLMs as proposers within a formally-structured, NAS-inspired search framework—is not just viable but necessary: the Viterbi-style generation depends on the GAU tree factorization, which depends on the Σ → Σ type constraint, which is a structural choice inspired by the modular design philosophy of NAS. The paper shows that LLM-driven discovery needs structured search spaces to be effective, and conversely, that structured search benefits from LLMs' ability to operate in unbounded, semantically-rich operation spaces that no fixed inventory can capture. This resolves the apparent contradiction between the ambition of ASD and the rigor of NAS.

The paper also provides a negative result that reshapes research priorities: the ablation of the Base and w/ Mem configurations (Table 2) shows that merely having access to past designs as background knowledge (mimicking the Romera-Paredes et al., 2024 setting) does not produce meaningful evolutionary progress without the selective pressure of the Evolution Tree's genealogical structure. The end fitness improvement for Base is negligible (0.01%, Sharpe ratio 0.002), while Base w/ Mem recovers some improvement (2.81%, Sharpe ratio 0.196) but still falls substantially short of Full (4.10%, Sharpe ratio 0.69). This means that passive knowledge accumulation is insufficient; the specific mechanism of genealogical selection with fitness and confidence as two-dimensional signals is what converts knowledge into progress. This steers future work away from simply improving LLMs' ability to read and summarize the literature, and toward designing better evolutionary structures and selection strategies that actively shape the search trajectory.

Finally, the identification of hubness in the evolutionary tree (Figure 13, Appendix E.1.2)—where a few highly-fit designs become central hubs that most subsequent designs derive from—is a structural finding that connects autonomous discovery to the sociology of science (the long-tail distribution of paper citations; Wu et al., 2009). This is not just an interesting parallel; it suggests that the system is recapitulating a pattern of cumulative, citation-like knowledge development, which has implications for how we evaluate and interpret the output of autonomous discovery systems. If the evolutionary tree structure mirrors citation networks, then metrics from network science (degree distribution, community structure, centrality) become diagnostic tools for assessing whether a discovery system is genuinely building on prior knowledge versus randomly sampling.

Follow-Up Research This Work Enables

Cheap, amortized difficulty estimation to make the 4× efficiency gain realizable in deployment. The paper's most glaring practical gap is the unaccounted cost of difficulty estimation (2,048 samples per question; Section 3.2, Section 6.1). The 4× efficiency gains over best-of-N are computed after difficulty is known, and the estimation cost would dominate total compute in any single-query setting. A concrete experiment would train a lightweight classifier—a small transformer or even a linear probe on top of a frozen LLM embedding—to predict the difficulty bin (the five-quintile PRM score average) directly from the question text, trained on a dataset of questions with oracle difficulty labels computed offline. The evaluation would measure (a) the accuracy of bin prediction, (b) the end-to-end accuracy of compute-optimal allocation using this cheap estimator versus the full 2,048-sample method, and (c) the break-even point in terms of number of queries where the amortized cost of the classifier makes the approach net-beneficial over uniform best-of-N. This experiment is directly enabled by the paper's finding that predicted (non-oracle) difficulty bins track oracle bins closely (Figures 4 and 8), suggesting that difficulty is a learnable function of the question text, not an intrinsically expensive signal.

Joint optimization of proposal-distribution modification and verifier-guided search to test for complementarity. The paper studies revisions (modifying the proposal distribution) and PRM search (modifying the verifier) as independent mechanisms, never combining them (Section 8). The natural next experiment is to use the revision model's outputs as the candidate pool for PRM-guided beam search: at each step of the beam search tree, the proposal distribution is not the base LLM but the revision model conditioned on its own previous (incorrect) attempts. The key question is whether the mechanisms are superadditive: the revision model improves pass@1 (making the proposal distribution more likely to contain correct solutions), while beam search improves the probability of finding those correct solutions among candidates. If the gains are superadditive—i.e., the combined method outperforms the sum of their individual improvements over a baseline—this would suggest that the two axes target different failure modes and should always be deployed together. If they are subadditive, it would suggest that the revision model's outputs are already "easy" for best-of-N selection to handle, limiting the marginal benefit of sophisticated search. The paper's existing difficulty-dependent analysis (Figures 3 right, 7 right) provides the hypothesis: combinations should help most on medium-difficulty problems (bins 3-4), where both mechanisms individually show their strongest advantages.

On-policy or iterative training for the revision model to eliminate the correct-to-incorrect reversion problem. The 38% correct-to-incorrect reversion rate (Section 6.1) is a direct consequence of the training data construction: the model never sees examples where the current answer is already correct and should be preserved. A concrete experiment would augment the training trajectories with examples where a correct answer appears in-context and the target is to preserve it (identity mapping) or to refine it with a better but still-correct answer. This requires generating on-policy data: run the current revision model, collect chains that contain a correct intermediate answer, and add training examples where the model is trained to output the same correct answer (or a verified-better one) when the context contains that correct answer. The evaluation metric is the reversion rate (fraction of correct intermediate answers changed to incorrect on the next step) and the end-to-end pass@1 after a chain of length K. This experiment is enabled by the paper's negative result with ReST^EM (Appendix K, Figure 16), which showed that naive on-policy training degrades performance, but which did not diagnose whether the degradation was due to the reversion problem specifically or to other distributional issues in the on-policy data. A targeted intervention that only adds "preserve correct" examples would isolate the mechanism.

Cross-model and cross-domain replication to map the boundary conditions of difficulty-dependent strategy allocation. All results are on PaLM 2-S* with the MATH benchmark. A replication study using, for example, Llama-3 on HumanEval (code generation) or ARC (logical reasoning) would answer: do the difficulty-dependent strategy patterns (beam search for medium difficulty, best-of-N for easy, sequential revisions for easy) generalize to structurally different reasoning domains and model families with different calibration properties? The key measurement is whether the qualitative pattern—beam search over-optimizes on easy problems, helps on medium problems, and fails on hard problems—holds across domains, even if the absolute difficulty thresholds shift. If the pattern is robust, it suggests a general principle: the optimal test-time strategy depends on the model's base pass@1, not on the specific benchmark. If it fails to replicate, it suggests the findings are specific to mathematical reasoning (where errors are systematic and verifiable) and may not transfer to domains where errors are more random or where correctness is less cleanly defined. The paper's explicit partitioning of difficulty by base model pass@1 (not dataset difficulty labels) provides the methodological template for this replication.

Verifier robustness training to raise the over-optimization ceiling. The paper demonstrates that verifier over-optimization is the primary bottleneck limiting further scaling of test-time compute (Figure 3 right: beam search degrades on easy problems at high budgets; Appendix M: degenerate outputs that score highly). A direct experiment would be to train the PRM not only on i.i.d. samples from the base model but also on adversarially generated samples from beam search trajectories—specifically, the solutions that beam search finds and that the PRM scores highly but that are actually incorrect. This is the standard recipe for adversarial robustness: include the failure modes of the optimizer in the training data of the verifier. The evaluation would measure whether the over-optimization threshold (the budget at which beam search performance plateaus or declines) shifts to higher budgets for easy and medium problems, and whether the qualitative failure modes (repetitive steps, overly short solutions) are reduced. This experiment is directly motivated by the paper's finding that over-optimization is the hard ceiling, and by its demonstration that the PRM's training methodology (Monte Carlo soft labels) matters for its properties.

Scaling the discovery experiment to billion-parameter scales with an improved LoS strategy. The paper's discovery experiments are capped at 350M parameters (with ~5 trials at that scale). The LoS strategy and V-D ratio analysis (Appendix E.4.2) provide the resource allocation framework, but the paper does not empirically validate whether the evolutionary gains observed at 14M-350M continue at larger scales. A scaled experiment with, say, a 1.4B parameter top scale would test whether the fitness improvements observed in Figure 12 (continued improvement up to 1,000 designs) translate to architectures that are competitive with human-designed models at deployment-relevant sizes. The key open question is whether the scaling laws correlation (performance at small scales predicts performance at large scales) holds strongly enough for the architectures discovered by the system itself—which may have different scaling properties than the human-designed architectures for which scaling laws have been validated. A null result (evolutionary progress at small scales that fails to transfer to large scales) would be an important diagnostic, revealing whether the system is optimizing for small-scale proxies that don't generalize, and would motivate research into scale-invariant fitness metrics or multi-scale training of the verifier itself.

Dynamic, adaptive strategy allocation within a single query, replacing the fixed difficulty bin pre-classification. The current compute-optimal policy is static: difficulty is estimated once (via 2,048 samples or a classifier), a strategy is selected, and the full budget is spent under that strategy. A natural extension, enabled by the observation that difficulty can be estimated from the PRM's score distribution, is dynamic allocation: begin with a small number of parallel samples (e.g., 4), compute the PRM's average final-answer score on those samples as an on-the-fly difficulty estimate, and then allocate the remaining budget accordingly—switching to sequential revisions if the problem appears easy, or to beam search if it appears medium-difficulty. This converts difficulty estimation from a pure cost to a shared cost that also contributes to the solution process (the initial parallel samples can be used in the final answer selection). The evaluation would measure total compute (including the initial estimation samples) and end-to-end accuracy, comparing against the static compute-optimal policy. This experiment connects to the multi-armed bandit and Bayesian optimization literatures but with the specific structure that the arm (strategy) is chosen once based on an initial signal, not sequentially updated.

Practical Applications and Downstream Use Cases

Cost-efficient, automated architecture discovery for specialized deployment scenarios. For organizations building custom language models for specific domains (e.g., medical coding, legal document processing, scientific literature mining), the standard approach is to take an off-the-shelf architecture (typically a transformer variant) and fine-tune it on domain data. Genesys offers an alternative: run an autonomous discovery experiment targeting the specific computational constraints and data characteristics of the deployment scenario, potentially discovering architectures that are more compute-efficient or better-performing for that specific distribution than the general-purpose default. The concrete benefit is grounded in Table 5: Genesys-discovered architectures (e.g., HMamba, Geogate) match or outperform strong baselines like Mamba2 and GPT on benchmarks like RTE, CoLA, and MRPC, suggesting that domain-specific discovery could yield further gains. The practical barrier is the computation required for the discovery process itself; an organization would need to weigh the one-time cost of running Genesys against the recurring cost of using a suboptimal architecture in production.

Data generation for self-improvement pipelines, using discovered architectures as specialized proposal models. The paper envisions "distilling the outputs of applying additional test-time compute back into the base LLM, enabling an iterative self-improvement loop" (Section 8). A practical instantiation: use Genesys to discover an architecture that is particularly good at generating high-quality training data for a target task (e.g., mathematical reasoning), then use that architecture—rather than a generic large model—as the proposal distribution in a bootstrapping pipeline like STaR. The discovered architecture would be smaller and more inference-efficient than a large general-purpose model, reducing the cost of the generation step in the self-improvement loop. The unit-performance correlation analysis (Appendix E.3.3, Table 18) shows that specific unit combinations are predictive of performance on specific tasks (e.g., units predictive of CoLA performance differ from those predictive of MRPC), suggesting that task-specialized architectures are discoverable. The key practical question is whether the discovery cost amortizes over multiple iterations of the self-improvement loop.

A principled template for constructing domain-specific ASD systems. The paper's decomposition of the architecture discovery problem into a discovery environment (LMADE) and a discovery system (Genesys) is explicitly designed to be generalizable beyond language model architectures. The Σ → Σ type constraint that enables the GAU factorization and Viterbi-style generation (Appendix A.2) is not specific to autoregressive language models—it applies to any domain where the artifacts of interest can be expressed as compositions of typed modules. A practitioner building an ASD system for, say, molecular dynamics simulators or robotics control policies could adopt Genesys's architecture wholesale, replacing the Knowledge Engine's reference library with domain-specific literature, the Verification Engine's pretraining pipeline with domain-specific simulation, and the GAB/GAU base classes with domain-specific module interfaces that preserve the Σ → Σ composition structure. The paper's formal guarantees (Theorems 1 and 4 from the reference example, Theorems 4 and 5 in the paper's appendix) provide assurance that the factorization and efficiency properties would transfer. This is a transferable system design pattern, not a point solution for language models.

When to Prefer This Method

The paper explicitly positions Genesys as a system for automated architecture discovery, but it does not articulate a clear tradeoff matrix against named alternative methods for architecture design (e.g., manual design by human experts, standard NAS with fixed operation spaces, or simply using an off-the-shelf transformer). The implicit tradeoff is between the cost of discovery (running Genesys is computationally expensive—1,062 designs required >1 billion tokens, 2.76M lines of code, and 86K agent interactions) and the benefit of customization (the discovered architectures are competitive with or outperform standard baselines on specific benchmarks, but no single discovered design dominates all tasks). The decision rule for a practitioner is therefore: run Genesys when the recurring benefit of a potentially superior, domain-specialized architecture outweighs the one-time cost of discovery. The paper does not provide a quantitative cost-benefit analysis (e.g., FLOPs spent on discovery vs. FLOPs saved by a more efficient architecture in production), so a precise decision rule is not extractable from the current results. In the absence of explicit positioning, a forced decision matrix would be speculative rather than grounded in the paper's evidence.