Understanding the Challenges in Iterative Generative Optimization with LLMs

🎯 Pitch

The success of an LLM-driven optimizer can hinge entirely on invisible design choices like how you modularize the initial code—one monolithic function versus many modules can shift benchmark performance by over 10 percentile points. Short, truncated execution traces often work just as well as full-episode data for reinforcement learning in Atari, and larger batches of trial-and-error examples do not monotonically improve generalization. These findings reveal that the brittleness limiting LLM-based optimization in production stems not from inadequate infrastructure, but from a lack of universal defaults for three critical, hidden hyperparameters.

1. Executive Summary

This paper analyzes why LLM-based iterative generative optimization—where a model repeatedly revises artifacts like code or prompts using execution feedback—remains brittle in practice, with only 9% of surveyed agents employing any automated optimization. Through case studies on MLAgentBench, Atari, and BigBench Extra Hard using Claude Sonnet-3.5, the authors isolate three hidden design decisions that critically determine whether a learning loop succeeds: the starting artifact (how the initial system is modularized—one monolithic function versus many modular functions), the credit horizon (how many execution steps to include per update—single-step immediate rewards versus full multi-step rollout traces), and experience batching (how many independent trials to aggregate into one learning context—batch sizes of 1, 3, or 5 examples). The paper finds that one-function versus many-function initialization of an ML pipeline can shift Kaggle leaderboard performance by over 10 percentile points, that short credit horizons suffice for four of eight Atari games without sacrificing full-episode performance, and that larger batches do not monotonically improve generalization on BBEH—with optimal batch size being strictly task-dependent (e.g., batch size 1 achieves 0.537 accuracy on Disambiguation QA while batch size 5 achieves 0.531). Establishing that no single universal recipe works across domains—different tasks demand different configurations of all three design choices—the paper frames these challenges as parallels to well-studied ML concepts (architecture initialization, truncated backpropagation through time, and stochastic gradient descent batch size), arguing that the lack of simple defaults is a major hurdle for productionization rather than a consequence of inadequate infrastructure.

2. Context and Motivation

The Core Gap: Generative Optimization Works in Labs, Not in Production

The paper addresses a striking disconnect between research enthusiasm and real-world adoption. LLM-based generative optimization—where an LLM repeatedly edits code, prompts, or workflows using execution feedback to improve a target metric—has demonstrated impressive results across specialized domains: discovering faster matrix multiplication algorithms (Novikov et al., 2025), optimizing GPU kernel latency (Ouyang et al., 2025a; Lange et al., 2025b), designing novel parallel programs (Wei et al., 2025a), and proposing drug candidates (Ghareeb et al., 2025). Yet Pan et al. (2025b) report that only 9% of surveyed production agentic systems use any form of automated design, including simple LLM-assisted prompt tuning. This is not for lack of software infrastructure—the past two years have produced a rich ecosystem of agent-building libraries (Khattab et al., 2024; Wu et al., 2024; LangChain, 2024; Cheng et al., 2024) with built-in optimization mechanisms. The paper's central question is: if the tools exist and the research results are compelling, why isn't generative optimization being widely deployed?

The authors argue the answer lies not in engineering gaps but in hidden design complexity. Setting up a learning loop requires an engineer to make choices that are rarely discussed explicitly in prior work: what to provide as the initial artifact, how much of an execution trace to include as evidence, and how many trials to batch together. These choices matter enormously—they can determine whether optimization succeeds or fails—but they lack principled defaults. This creates what the paper calls a "setup burden": engineers must invest substantial effort and guesswork to get a learning loop working for each new task, which is antithetical to the production requirement of simple, reusable solutions.

Why This Problem Matters, Practically and Conceptually

Practical significance. The low adoption rate despite mature tooling represents a concrete failure mode for the vision of self-improving AI systems. If generative optimization cannot be made reliable across diverse applications without per-task custom engineering, then the dream of "end-to-end automation that scales with compute" (Sutton, 2019)—which the paper explicitly references—remains out of reach. For organizations building agentic systems, the setup burden translates directly to labor costs: every new optimization task requires an engineer to experimentally determine the right modularization, credit horizon, and batch size, rather than applying known defaults. This limits generative optimization to specialized, high-value domains (alphaEvolve for algorithmic discovery, kernel optimization for hardware companies) and prevents its widespread adoption in more routine agent design.

Conceptual significance. The paper identifies a deeper structural issue: the design decisions in a learning loop closely parallel well-understood concepts in traditional machine learning, but remain unexplored in the LLM optimization context. The starting artifact problem resembles neural network architecture design (Zoph & Le, 2017) and weight initialization (Glorot & Bengio, 2010), where different starting points determine which solutions are reachable. The credit horizon problem mirrors debates in episodic reinforcement learning about how many timesteps to include before computing returns (Arjona-Medina et al., 2019) and truncated back-propagation through time (Tallec & Ollivier, 2017; Shaban et al., 2019). The experience batching problem parallels batch size selection in stochastic gradient descent, where the number of examples aggregated per update affects both learning dynamics and generalization (Smith et al., 2018). By drawing these parallels explicitly, the paper argues that generative optimization's challenges are not ad hoc engineering issues but systematic research problems that can be studied with the same rigor applied to traditional ML. This reframing is the paper's core intellectual contribution: it converts "this is hard to set up" into a research agenda with identifiable sub-problems.

Where Prior Work Falls Short

The paper identifies specific limitations across three categories of prior work:

1. Optimization libraries focus on search algorithms, not loop design. Frameworks like DSPy (Khattab et al., 2024), TextGrad (Yuksekgonul et al., 2025), Trace (Cheng et al., 2024), and LangChain (LangChain, 2024) provide mechanisms for iterative modification—cross-validation-based prompt selection, Pareto optimization over multiple objectives, fine-grained gradient-like feedback propagation through computational graphs. However, these works "primarily showcase successful applications rather than investigating the design choices and instabilities that make learning loops difficult to implement" (Section 3). They provide tools for optimization but offer little guidance on how to configure the loop itself—what to initialize, what traces to feed back, how many examples to aggregate. An engineer using DSPy still faces the same three hidden decisions, just within a more convenient API.

2. Work on self-improving agents conflates within-task refinement with cross-task learning. The agent loop literature (Zhao et al., 2025; Anthropic, 2025; Bolin, 2026; Huntley, 2026) focuses on making an agent succeed on a single task through self-debugging (Chen et al., 2024), self-correction (Xiong et al., 2025), and self-refinement (Madaan et al., 2023). The paper draws a critical distinction: these are within-task agent loops that aim for highest success rate on one execution, whereas the paper's learning loop accumulates experience across tasks where "the success or failure of any single attempt is secondary to the agent's eventual mastery" (Section 3). Prior work has not systematically studied how to construct the learning context for cross-task optimization.

3. Memory and context engineering research addresses retrieval, not fundamental loop configuration. Recent work on agent memory (Wang et al., 2025b; Zhou et al., 2025a; Ouyang et al., 2025b; Zhang et al., 2025d) focuses on how to retrieve relevant past experiences—analogous to developing better database indexes. The paper's contribution is orthogonal and more fundamental: even assuming perfect retrieval, the engineer must still decide what unit of experience to store (credit horizon) and how many to show at once (experience batching). These decisions precede retrieval and shape what the optimizer can learn, yet they have received "little systematic investigation" (Section 3). The paper is not arguing against memory systems; it is arguing that memory alone does not solve the loop configuration problem.

How This Paper Positions Itself

The paper explicitly positions itself as identifying and systematically characterizing the design decisions in a learning loop, not proposing a new optimization algorithm or achieving state-of-the-art results on any benchmark. This is stated upfront in Section 1:

"We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence... We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption."

The paper's contribution is therefore diagnostic, not prescriptive. It does not claim to have solved these problems; it claims that they are the right problems to solve.

Methodologically, the paper uses controlled case studies where each of the three factors is varied in isolation while holding other aspects of the learning loop constant. This contrasts with prior work that varies multiple factors simultaneously (e.g., comparing different search algorithms while also implicitly changing the artifact structure or feedback design) or reports single successful configurations without ablating alternatives. In Section 4 (Starting Artifact), the paper compares one-function versus many-function initialization on the same ML pipeline task with the same optimizer, same feedback design, and same LLM backend—the only variable is how the code is modularized. In Section 5 (Credit Horizon), the paper compares one-step versus multi-step traces on the same Atari games with identical starting artifacts and feedback templates. In Section 6 (Experience Batching), the paper varies only batch size on the same BBEH tasks with fixed training set, optimizer, and feedback. This isolation enables attribution: if performance changes, it is because of the design choice being studied.

Conceptually, the paper introduces a graph-theoretic formalism in Appendix D that makes these design choices explicit and rigorous. The distinction between a workflow graph (a single execution trace of the parameterized system) and a learning graph (the combined graph constructed by a learning template and shown to the optimizer) provides precise language for what was previously implicit engineering. The learning template is the key abstraction: it specifies how individual experiences are combined—through concatenation (batch learning), sequential linking (episodic learning), or single-example presentation (interactive learning)—before being sent to the optimizer. The three factors studied in the paper map directly onto components of this template: starting artifact determines the structure of workflow graphs, credit horizon determines the length of sequential linking in episodic templates, and experience batching determines the aggregation width in batch templates. This formalism does not appear in prior work on generative optimization and represents the paper's main theoretical contribution.

Scoping. The paper is careful to bound its claims. It does not study all possible design choices—feedback design (staged vs. flat rewards, directional vs. scalar feedback) is acknowledged as important but left to prior work (Nie et al., 2024; Xu et al., 2025). The specific optimization algorithm (OptoPrime from Cheng et al., 2024) is held fixed, as is the LLM backend (Claude Sonnet-3.5-v2). The paper is not claiming these don't matter; it is claiming that the three factors it studies matter enough to be worth their own investigation, and that their task-dependence is sufficient to explain why adoption lags despite algorithmically capable optimizers.

The implicit argument. The paper's structure—three cleanly separated case studies, each isolating one factor, each showing task-dependent optimal configurations—makes an implicit argument through its form: these decisions are independent enough to study separately but consequential enough that ignoring any of them can cause optimization to fail. The fact that no configuration is universal (a different batch size is optimal for each BBEH task; short credit horizons work for Freeway but not Space Invaders; one-function initialization beats many-function on Housing Price but not Spaceship Titanic) is the paper's main empirical finding. This task-dependence is what prevents simple design guidelines from emerging naturally—and it is precisely what distinguishes generative optimization from traditional ML, where practitioners have developed robust defaults (Adam optimizer, ReLU activations, batch size 32 or 128) that work across many applications.

3. Technical Approach

3.1 Reader Orientation

This paper builds a framework for understanding and diagnosing why LLM-based iterative generative optimization—where a language model repeatedly revises code, prompts, or workflows using execution feedback—succeeds in some settings but fails in others. The core insight is that three hidden engineering decisions—what to provide as the starting system, how much execution trace to include as evidence, and how many independent trials to aggregate into one update—are the primary determinants of whether a learning loop works, and no single configuration of these decisions transfers across domains.

3.2 Big-Picture Architecture (Diagram in Words)

The system being analyzed is a learning loop—a feedback-driven optimization process with five conceptual components:

A parameterized system ( $\theta$ ): the artifact being optimized. This can be code (an ML pipeline, an Atari game-playing program), a prompt template with postprocessing logic (for BBEH tasks), or any combination of text, code, and configuration files that an LLM can edit.
An environment or evaluation harness: executes the parameterized system on inputs, collects outputs, and produces feedback. This can be a Kaggle leaderboard with a hidden test set, an Atari emulator returning step-level rewards, or a correctness checker for language understanding tasks.
A trace oracle: records the execution as a computational graph—a workflow graph showing which parameters influenced which intermediate computations and final outputs, with feedback attached to the output node.
A learning template: defines how individual execution traces are combined into a learning graph before being shown to the optimizer. This is where the three hidden decisions live: the template determines (1) which parts of the system are optimizable (starting artifact), (2) how many sequential steps to link together before updating (credit horizon), and (3) how many independent execution traces to concatenate into one context (experience batching).
An LLM optimizer: receives the learning graph as input, proposes revisions to the parameterized system, and the cycle repeats. The paper uses OptoPrime (Cheng et al., 2024) with Claude Sonnet-3.5-v2 as the backend, but the architecture is optimizer-agnostic.

Information flows as follows: the current system $\theta$ is executed on inputs → the trace oracle records one or more workflow graphs → the learning template stitches them into a learning graph using either batch aggregation (concatenating independent traces), episodic linking (sequentially chaining environment-dependent traces), or interactive presentation (single trace per update) → the LLM optimizer receives this learning graph along with feedback → the optimizer proposes a new $\theta'$ → the cycle repeats for a fixed number of iterations.

3.3 Roadmap for the Deep Dive

First, the OPTO formalism (Appendix D) that provides precise language for describing learning loops, because it defines what exactly the optimizer sees and how the three hidden choices manifest mathematically.
Second, the three learning templates—interactive, batch, and episodic—since they are the mechanism by which credit horizon and experience batching are operationalized.
Third, the starting artifact problem in MLAgentBench, including the one-function vs. many-function initialization, the staged feedback design, and the meta-overfitting phenomenon.
Fourth, the credit horizon problem in Atari, including the per-game configurations, the one-step vs. multi-step trace design, and the staged feedback templates.
Fifth, the experience batching problem in BigBench Extra Hard, including the batchify operator, the per-task batch size sweep, and the train/validation/test split protocol.
Sixth, the cross-cutting experimental methodology shared across all three case studies: the optimizer choice (OptoPrime), the LLM backend (Claude Sonnet-3.5-v2), the iteration budgets, and the evaluation protocols.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a diagnostic empirical analysis paper whose core idea is that generative optimization's brittleness arises from three under-studied design decisions—starting artifact, credit horizon, and experience batching—and that these decisions map directly onto well-understood concepts in traditional machine learning when viewed through the lens of the OPTO (Optimization with Trace Oracle) formalism.

The OPTO Formalism: A Unified Language for Learning Loops

The paper introduces a graph-theoretic framework (detailed in Appendix D) to make precise what is normally left implicit in learning loop implementations. This is not the main technical contribution—it is scaffolding for the empirical investigation—but understanding it is essential because the three hidden choices (starting artifact, credit horizon, experience batching) are each different points of intervention in this framework.

The OPTO problem definition. An OPTO problem is a triple $(\Theta, \omega, \mathcal{T})$ where $\Theta$ is the parameter space (all possible configurations of the system being optimized), $\omega$ is the problem context (task description, constraints, evaluation criteria), and $\mathcal{T}$ is a Trace Oracle. When invoked with a parameter $\theta \in \Theta$ , the Trace Oracle returns a pair $(f, g)$ where $g$ is a computational graph involving $\theta$ and $f$ is a feedback signal attached to exactly one node in $g$ —the output node.

In plain language: every time you run your system with a particular configuration, the Trace Oracle records exactly how information flowed through the system (which parameters influenced which computations) and what feedback was received. This recorded flow—the computational graph $g$ —is what the optimizer will use to decide how to revise $\theta$ .

Workflow graph vs. learning graph. The key distinction is between the workflow graph $g_i$ (what a single execution produces) and the learning graph $G_{\text{learn}}$ (what the optimizer actually sees). A workflow graph is produced by executing the parameterized system $W_\theta$ on one input $x_i$ :

For a system with two functions $h_{\theta_1}$ and $h_{\theta_2}$ , input $x_i$ flows through $h_{\theta_1}$ producing intermediate output $o_i = h_{\theta_1}(x_i)$ , then through $h_{\theta_2}$ producing final output $y_i = h_{\theta_2}(o_i)$ . The workflow graph $g_i$ has edges $(x_i, \theta_1) \to o_i$ and $(o_i, \theta_2) \to y_i$ , with feedback $f_i$ attached at $y_i$ .

One workflow graph represents one experience: one input, one execution, one output, one feedback signal. But the optimizer shouldn't always update from a single experience. The learning graph $G_{\text{learn}}$ is constructed from one or more workflow graphs using a learning template $T$ :

$G_{\text{learn}} = T(g_1, g_2, \ldots, g_k)$

where $T$ is the learning template, $g_1, \ldots, g_k$ are individual workflow graphs, and $k$ is the number of experiences aggregated.

What it computes: the learning template determines how individual execution traces are combined into the single optimizer-facing context. If $k=1$ , the optimizer sees one trace per update. If $k>1$ and the experiences are independent, they are concatenated via a batchify operator $\oplus$ . If the experiences are causally linked (output of one determines input of next), they are chained via an environment transition operator $\Rightarrow$ .

Why this form: the template abstraction separates the structure of the optimization problem (how experiences relate to each other) from the content of individual experiences (what happened in each execution). This lets us vary credit horizon and experience batching independently of the system being optimized or the feedback design.

The three template types map directly onto the paper's three hidden choices:

Interactive learning template $T_{\text{interactive}}(g_1) = g_1$ : the optimizer receives exactly one workflow graph per update. This corresponds to online learning where the system is revised after every single experience. No cross-example composition occurs.
Batch learning template $T_{\text{batch}}(g_1, \ldots, g_B)$ : the optimizer receives $B$ independent workflow graphs concatenated via $\oplus$ . The batchify operator $\oplus$ is the formal name for what the paper calls experience batching—it takes $B$ separate (input, output, feedback) triples and stitches them into one contiguous learning context. This template is used in the BBEH case study (Section 6).
Episodic learning template $T_{\text{episodic}}(g_1, \ldots, g_T)$ : the optimizer receives $T$ workflow graphs that are causally linked via environment transitions $\Rightarrow$ . The key difference from batch learning is that $g_{t+1}$ depends on the output of $g_t$ through the environment—the experiences are not independent. This template is used in the Atari case study (Section 5), where each $g_t$ represents one game step and $T$ is the credit horizon (how many steps to unfold before updating).

Why this matters for the paper's argument. The formalism makes explicit what the three hidden choices correspond to: starting artifact determines the internal structure of each workflow graph $g_i$ (which functions are editable, how they connect); credit horizon determines $T$ in episodic templates (how many $g_i$ to chain before presenting to optimizer); experience batching determines $B$ in batch templates (how many independent $g_i$ to concatenate). The paper's empirical claim is that these choices are not implementation details—they fundamentally change the optimization problem the LLM is solving.

Design of the Three Case Studies (Shared Methodology)

Before diving into each factor individually, the paper establishes a consistent experimental protocol across all three case studies:

Optimizer and LLM backend. All experiments use OptoPrime (Cheng et al., 2024) as the generative optimizer and Claude Sonnet-3.5-v2 as the LLM backend. The experiments were conducted between February and April 2025. OptoPrime is chosen because it is a general-purpose generative optimizer that works with arbitrary parameterized systems (code, prompts, workflows) rather than being specialized to a particular domain. This choice is deliberate: the paper is studying factors that affect any LLM-based optimization loop, not factors specific to one optimizer's algorithm. The authors note that "the main focus of our paper is to study the factors that impact the learning loop, not the small differences between individual libraries" (Section 2).

Memory size. Across all experiments, the optimizer maintains a memory of size 5—it keeps the 5 most recent learning graphs in its context. This provides the optimizer with a history of past attempts and their outcomes, enabling it to avoid repeating failures.

Iteration budgets. The number of optimizer update steps varies by task: 20 steps for MLAgentBench (Section 4), 20-30 steps for Atari depending on the game (Table B2 in Appendix B, ranging from 20 for Pong and Space Invaders to 30 for Breakout, Freeway, Asterix, Enduro, Q*bert, and Seaquest), and 15 steps for BigBench Extra Hard (Appendix C.1, Table C1).

Evaluation protocol. All case studies follow a train-evaluate-on-held-out protocol:

In MLAgentBench, an 80/20 train/validation split is used to provide internal feedback, and the best checkpoint (according to validation metric) is submitted to Kaggle's hidden test set for final evaluation (Appendix A.4).
In Atari, the learned agent is evaluated on full episodes (typically 4000 steps, with game-specific variations in Table B2) after the optimization budget is exhausted.
In BigBench Extra Hard, 15 training examples are used for optimization, 10 validation examples for monitoring, and 175+ held-out test examples for final evaluation (Appendix C.1, Table C1).

Trials and statistical reporting. Each configuration runs for multiple trials with different random seeds (5 trials for MLAgentBench and Atari, 3 trials for BBEH). Results report means, and where error bars are shown (BBEH validation curves in Figure 7), they represent standard error across trials. The 3 trials in BBEH correspond to different random batch shufflings of the same training set.

The Starting Artifact Problem: MLAgentBench Case Study

The starting artifact is not a single file—it is a bundle of design decisions about what the LLM optimizer is allowed to edit and what prior structure it starts from. The OPTO formalism maps these to three concrete aspects (Appendix D.4): workflow structure (the number of functions, files, or components and how they connect—determines the shape of $g_i$ ), program documentation (docstrings, comments, instructions that explain each component's intended purpose—shapes the optimizer's interpretation), and initial implementation (how much working code is provided before optimization begins—determines whether early executions succeed or fail). The MLAgentBench case study varies the first of these—workflow structure—while holding the others constant.

Task description. The task is to create an ML training pipeline for Kaggle competition datasets: given a dataset (Housing Price or Spaceship Titanic), write a codebase that handles data ingestion, preprocessing, model building, training, and hyperparameter search, then produce predictions that score well on Kaggle's hidden test set. This is a popular task in the AI research agent literature (Huang et al., 2024; Chan et al., 2025; Toledo et al., 2025). The input to the LLM optimizer includes the task description, the dataset (pre-downloaded, outside the agent's scope), and a train/validation split. The output is a codebase that produces a trained model.

The two initialization conditions. The paper compares what it calls "one-function" and "many-function" starting artifacts (Figure 3 in the main paper, detailed in Figure A1 in Appendix A.2). The key methodological claim is that both initializations contain equivalent information—the one-function docstring is a concatenation of all the docstrings in the many-function version—and the only difference is the level of modularization:

One-function initialization (Figure 3 left, Figure A1a): The system is a single Pipeline class with one editable function train_model that handles the entire pipeline from data ingestion to prediction. The class's __call__ method simply calls self.train_model(x, y, test_data). The docstring for this function is long—it contains all the implementation hints concatenated together: suggestions about preprocessing, feature selection, ensemble construction, training, and prediction, all in one block of text. The optimizer's @trace.bundle(trainable=True) annotation means only this one function can be modified; the rest of the class structure is fixed.
Many-function initialization (Figure 3 right, Figure A1b): The same Pipeline class is decomposed into four separate editable functions—preprocess, select_features, train_model, and predict—each with its own @trace.bundle(trainable=True) annotation and its own docstring. The __call__ method explicitly chains them: x = self.preprocess(x) → z = self.select_features(x) → m = self.train_model(z, y) → return self.predict(m, z). Each function's docstring covers only its specific responsibility (preprocessing hints in the preprocessing docstring, feature selection hints in the feature selection docstring, etc.), but the union of all docstrings equals the one-function version's docstring.

Why this design matters for the experiments. This is a clean ablation: the information content is held constant (both conditions see the same total guidance), and the only variable is whether that guidance is presented as one monolithic block or decomposed across modular interfaces. The paper explicitly draws the parallel to prior work on task decomposition—least-to-most prompting (Zhou et al., 2023) and Parsel (Zelikman et al., 2023)—which suggests that decomposing hard tasks into multiple easier subtasks can help LLMs reason more effectively. The experiment asks: does the same principle apply when the LLM is acting as an optimizer rather than a solver?

Staged and suggestive feedback design (Appendix A.3). The optimizer does not receive bare metrics. Instead, the paper uses staged feedback templates that vary the natural-language guidance based on the current validation performance level. This is a deliberate design choice: the paper wants the optimizer to have useful directional feedback while still having to discover the specific code changes. The templates are task-specific:

Spaceship Titanic (Table A1a, Figure A2a): Feedback is based on validation F1 score. If F1 < 0.5: "Model performance is poor. Try better feature engineering and preprocessing." If 0.5 ≤ F1 < 0.7: "Model is showing promise but needs improvement. Consider class balancing techniques." If 0.7 ≤ F1 < 0.8: "Model is performing well. Fine-tune hyperparameters for further improvements." If F1 ≥ 0.8: "Excellent performance! Focus on preventing overfitting."
Housing Price (Table A1b, Figure A2b): Feedback is based on validation $r^2$ . If $r^2 \leq 0$ : "Model is performing worse than baseline. Focus on better feature engineering and selection." If $0 < r^2 < 0.5$ : "Model has poor predictive power. Try more advanced preprocessing or different algorithms." If $0.5 \leq r^2 < 0.7$ : "Model is improving but still has room for growth. Consider feature interactions." If $r^2 \geq 0.7$ : "Model is performing well. Fine-tune hyperparameters for further improvements."

Additionally, the paper applies improvement-style feedback: when the model fails to improve the validation metric relative to the previous optimization step, an improvement suggestion is appended to the feedback string. This gives the optimizer a signal about whether its last revision helped or hurt, beyond the absolute metric value.

Meta-overfitting: a distinct phenomenon from traditional overfitting (Appendix A.4). The paper identifies a subtle but important phenomenon it calls meta-overfitting, which it distinguishes from traditional model overfitting. In traditional overfitting, a trained model performs well on training data but poorly on held-out data. In meta-overfitting, the optimizer itself learns to propose revisions that improve the immediate validation metric at the expense of generalization—it discovers pipeline code revisions that fit the training split rather than the true data-generating distribution.

Figure A3 illustrates this: at each optimization step (x-axis), the agent produces an entirely new model trained from scratch. The y-axis shows validation F1. What increases over optimization steps is the optimizer's tendency to find pipeline architectures that work on the validation split but fail on the held-out test set. The paper states this explicitly: "the generative optimizer learns to make workflow revisions that improve the immediate validation-driven objective while drifting toward brittle pipelines."

This is a critical finding because it means having a separate held-out validation set is not sufficient to prevent meta-overfitting—the optimizer can overfit to the validation set through repeated querying, analogous to how hyperparameter search can overfit validation performance if performed repeatedly on the same split. The paper reports Kaggle test results (Table 1) to validate that the best checkpoints generalize, but the optimization curves in Figure A3 show that aggressive optimization leads to validation improvements that don't transfer.

Baseline comparison. The paper compares against ResearchAgent (Huang et al., 2024), which is an earlier ML agent that can also produce Kaggle submissions. To ensure fair comparison, the authors pre-downloaded datasets for ResearchAgent and confirmed it could produce valid test submissions. Both systems use the same underlying LLM (Claude Sonnet-3.5-v2).

Key configurations summarized:

Optimizer: OptoPrime, memory size 5, 20 optimization steps
Evaluation: 80/20 train/validation split; best checkpoint selected by validation metric (F1 for Spaceship Titanic, $r^2$ for Housing Price); final evaluation on Kaggle hidden test set
Trials: 5 runs per configuration
Feedback: staged templates + improvement suggestions when metric doesn't improve

The Credit Horizon Problem: Atari Case Study

The credit horizon determines how many steps of a multi-step execution trace to include in the learning graph before presenting it to the optimizer. In the OPTO formalism (Appendix D.5), this corresponds to the parameter $T$ in an episodic learning template—how many workflow graphs $g_1, \ldots, g_T$ are chained via environment transitions $\Rightarrow$ before the resulting learning graph is shown to the optimizer. The Atari case study isolates this factor by comparing two regimes:

One-step credit horizon: The optimizer receives a trace containing a single observation, a single action, and its immediate reward, and updates the agent's code after every step. This corresponds to $T=1$ in the episodic template.
Multi-step credit horizon: The optimizer receives a full rollout trace before each update—the agent plays for multiple steps, all observations/actions/rewards are recorded, and the full sequence is presented as one learning context. The rollout length $T$ varies by game (see Table B2 in Appendix B.1).

Task description. The task is to write a Python program that plays Atari games: given structured object-centric state representations (object positions, velocities, lives, rewards) from the OCAtari environment (Delfosse et al., 2024), the program outputs an action for the game controller at each timestep. The program itself is stateless—it does not maintain memory across steps, making it a Markov policy that maps current observation to action, similar to what traditional RL algorithms learn. The paper uses eight games spanning different strategic demands: Pong and Breakout (ball trajectory prediction and paddle positioning), Space Invaders (coordinated shooting and movement under firing constraints), Freeway (dodging traffic for immediate rewards), and Asterix, Enduro, Q*bert, and Seaquest.

Why Atari? Game playing is a natural multi-step task with a clear episodic structure: the system takes game screen observations at each timestep and outputs actions, accumulating reward over potentially hundreds or thousands of steps. Atari games also provide dense per-step rewards, creating a controlled testbed for the credit horizon question: should the optimizer consider immediate rewards only, or wait for episode completion? The paper notes that this parallels classic debates in reinforcement learning about discount factors and effective horizons (Laidlaw et al., 2023; Cheng et al., 2021), and draws a direct analogy to truncated back-propagation through time in recurrent neural networks (Pascanu et al., 2013; Tallec & Ollivier, 2017).

State representation: OCAtari. A critical implementation detail is that the agent does NOT receive raw pixels. Instead, it receives structured object-centric dictionaries from OCAtari (Delfosse et al., 2024). Figure B2 in Appendix B.1 shows an example from Breakout: the observation is a Python dictionary containing objects like Player (with fields x, y, w, h, dx, dy), Ball (with position and velocity), RB/OB/YB/GB/AB/BB (brick rows with bounding boxes), lives (integer), and reward (float). Importantly, these objects are not annotated—the LLM sees raw acronyms like "RB" and "OB" and must infer their meaning from context, the same way the trajectory prediction functions they implement must work with raw coordinate data.

Why object-centric representations? The paper makes this choice for two reasons. First, it makes the learned code more interpretable—the optimizer writes code that processes named objects rather than convolutional neural network weights, allowing qualitative inspection of learned strategies (as shown in Figure B4). Second, it provides a clean comparison with deep RL baselines: the same object information can be converted to numeric vectors for neural networks (DQN, PPO) while being passed as raw dictionaries to the LLM agent, keeping the information content constant across approaches.

Agent design (Appendix B.2). The paper designs modular initial agents for each game, with the same many-function philosophy as the MLAgentBench case study. Figure B3 shows representative workflows:

Pong and Breakout: Both center on trajectory prediction and paddle control. The agent has functions like predict_ball_trajectory(obs) and select_action(predicted_ball_y, obs) for Pong, or generate_paddle_target(pre_ball_x, obs) for Breakout. The initial implementations are intentionally simple (return current ball position, choose random movement), giving the optimizer room to discover more sophisticated strategies.
Space Invaders: The workflow decomposes into decide_shoot(obs), decide_movement(obs), and combine_actions(shoot, movement). This decomposition makes explicit that the game requires concurrent control over shooting (attack) and movement (defense), unlike pure interception games.

Staged feedback design (Appendix B.3). Similar to MLAgentBench, the Atari experiments use staged feedback templates keyed to performance levels. Tables B3 and B4 show representative examples:

Pong (Table B3a): If reward ≤ 0: "Your score is -5 points. Try to improve paddle positioning to prevent opponent scoring." If 0 < reward < 19: "Keep it up! You're scoring 12 points against the opponent but you are still 9 points from winning the game. Try improving paddle positioning to prevent opponent scoring." If reward ≥ 19: "Good job! You're close to winning the game! You're scoring 20 points against the opponent, only 1 point short of winning."
Space Invaders (Table B4): If reward ≥ 300: "Great job! You're performing well with an average score of 320. Try to improve your shooting accuracy and dodging." If 100 ≤ reward < 300: "Good progress! Your average score is 180. Focus on better timing for shooting and avoiding enemy projectiles." If reward < 100: "Your average score is 70. Try to improve your strategy for shooting aliens and dodging projectiles."

The paper notes that this staged feedback is analogous to reward shaping in RL—it gives the optimizer more granular guidance than a single scalar reward—but it is template-based and pre-determined by the engineer rather than learned or dynamically adapted.

Rollout length design (Appendix B.4). The per-game multi-step rollout lengths are reported in Table B2: 400 steps for Pong, 300 for Breakout, 25 for Space Invaders, 100 for Freeway/Asterix/Enduro/Q*bert/Seaquest. Why are they different? The paper states that these values "were chosen to balance two opposing forces. Longer traces reveal delayed consequences and are more faithful to the eventual control objective, but they also consume more of the optimizer's context budget and reduce update frequency." For example, Space Invaders uses a short multi-step horizon of 25 steps because its dense reward structure already provides informative per-step feedback (shooting aliens yields immediate reward), while Pong uses 400 steps because a single step reveals very little about whether the paddle positioning strategy is correct (you need to see whether the ball is returned successfully).

Key configurations summarized:

Environment: ALE {game}-NoFrameskip-v4 via Gymnasium, action repeat 4, sticky action probability 0.0
State representation: OCAtari object dictionaries (no raw pixels, no manual annotation)
Optimizer: OptoPrime, memory size 5, 20-30 optimization steps (game-dependent)
Credit horizons: one-step (update after every action) vs. multi-step (update after full rollout, per-game lengths in Table B2)
Feedback: staged natural-language templates based on reward thresholds
Evaluation: 4000-step episodes (or game-dependent lengths), 5 trials per configuration
Deep RL baselines: DQN and PPO with object-centric inputs, 10M environment steps, 5 seeds per game (Appendix B.5, Table B5a)

The Experience Batching Problem: BigBench Extra Hard Case Study

Experience batching determines how many independent execution traces to concatenate into one learning context before presenting it to the optimizer. In the OPTO formalism (Appendix D.6), this corresponds to the number of workflow graphs $g_1, \ldots, g_B$ that are aggregated via the batchify operator $\oplus$ in a batch learning template. The BBEH case study isolates this factor by comparing three batch sizes while holding everything else constant.

Task description. The task is to optimize a prompted LLM system for challenging language understanding tasks from the BigBench Extra Hard (BBEH) benchmark (Kazemi et al., 2025). The eight tasks span logical reasoning (Dyck Languages, Boolean Expressions), spatial reasoning (Geometric Shapes), language understanding (Linguini, Disambiguation QA), recommendation and rule-based reasoning (Movie Recommendation, Boardgame QA), and causal reasoning (Causal Understanding). Each task requires the agent to produce a prompt template and postprocessing code that generalizes across diverse question types within that task—a common real-world LLM engineering scenario.

Why BBEH? The tasks provide clean correctness signals (binary right/wrong with ground truth solutions revealed for incorrect answers), making feedback unambiguous. The tasks are diverse enough to test whether optimal batch size is domain-dependent. And the agent being optimized is simple—just a prompt and an answer extraction function—which means the experiment isolates batching without confounding factors like code modularization or multi-step credit assignment.

Agent design (Appendix C.2, Figure C1). The agent has exactly two optimizable components:

call_llm: Combines the trainable prompt template with the task query and sends the resulting string to the backend LLM (Claude Sonnet-3.5-v2). The prompt is the main optimization target—it is revised across updates to improve accuracy.
answer_extraction: Parses the raw LLM response into the final answer format expected by the evaluator. The initial implementation simply splits on the string "Answer:", making formatting part of the optimization problem rather than assuming it is solved in advance.

Both components are marked @trace.bundle(trainable=True), meaning the optimizer can modify both the prompt text and the extraction logic. This is deliberately minimal: the agent is a pure text-processing system with no external tools, no retrieval, no multi-step reasoning—just a prompt and a parser. This minimalism ensures that the only factor affecting learning is how many examples are batched together.

The batchify operator (Appendix C.4). The paper defines a batchify operator $\oplus$ that concatenates multiple independent execution traces into a single learning context. For batch size $k$ , the learning graph is:

$\text{BatchTrace} = \text{Trace}_1 \oplus \text{Trace}_2 \oplus \cdots \oplus \text{Trace}_k$

where each $\text{Trace}_i$ contains the question, the agent's predicted answer, and the feedback string, and $\oplus$ denotes text concatenation.

What it computes: $k$ complete (question → prompt → LLM response → extracted answer → feedback) traces are concatenated into one text block. The optimizer sees all $k$ examples simultaneously, with their individual correctness outcomes and (for incorrect answers) the ground truth.

Why this form: concatenation is the simplest possible aggregation—it presents all information without summarization or filtering. This means the experiment's results can be attributed to the number of examples rather than any clever aggregation scheme. If a more sophisticated aggregation were used (e.g., summarizing errors across examples), we wouldn't know whether batch size effects were due to the quantity of information or the quality of the aggregation.

Training data protocol (Appendix C.1, Table C1). The paper uses a fixed data split that enables clean comparison across batch sizes:

15 training examples (first 15 from dataset order)
10 validation examples (next 10)
175+ held-out test examples (remaining)
All configurations run for 15 optimizer update steps
3 trials per configuration, where trials correspond to different random shuffles of the 15 training examples

Critical detail: epochs vary with batch size. Because the training set is fixed at 15 examples and the update budget is fixed at 15 steps, the number of passes over the training set (epochs) varies: batch size 1 uses 1 epoch (15 examples ÷ 1 example per update = 15 updates), batch size 3 uses 3 epochs (3 examples per update × 15 updates = 45 examples seen = 3 passes over 15 examples), and batch size 5 uses 5 epochs (5 examples per update × 15 updates = 75 examples seen = 5 passes over 15 examples). The paper keeps the total update budget fixed rather than the total data seen, which is a deliberate choice: it studies whether seeing more examples per update (larger batch) is better than seeing fewer examples per update but more frequently (smaller batch), matching the classical batch size vs. learning rate tradeoff in SGD.

Feedback design (Appendix C.3). Feedback is deliberately simple and task-agnostic: a binary correctness signal with ground truth revealed for failures. When prediction is correct, the guide returns a success message. When incorrect, the guide reveals the expected answer and asks for a revision. This minimal feedback design ensures that any differences in batch size effects are due to the information aggregation structure, not to interaction with task-specific reward shaping.

Meta-overfitting in Boardgame QA (Appendix C.5). The paper explicitly flags Boardgame QA as "the clearest failure case" where the unoptimized baseline (accuracy 0.371) outperforms all optimized variants (0.341, 0.278, 0.276 for batch sizes 1, 3, 5 respectively). The interpretation is meta-overfitting: "the optimizer can over-specialize the prompt and extraction logic to the small training set rather than learn changes that generalize to held-out questions." This is the same phenomenon identified in MLAgentBench (Appendix A.4), but occurring in a pure text optimization setting where the artifact being optimized is a prompt rather than code.

Key configurations summarized:

Optimizer: OptoPrime, 15 update steps, Claude Sonnet-3.5-v2 backend
Agent: two-component system (trainable prompt + trainable answer extraction)
Data split: 15 train / 10 validation / 175+ test (per task)
Batch sizes: 1, 3, 5 examples per optimizer update
Epochs: 1 (batch=1), 3 (batch=3), 5 (batch=5) — total updates held constant at 15
Evaluation: held-out test accuracy, 3 trials per configuration
Feedback: task-agnostic binary correctness with ground truth for failures

Cross-Cutting Observations: Task-Dependence as the Central Finding

The paper's experimental design—three cleanly separated case studies, each isolating one factor—enables a powerful cross-cutting observation: in every case study, the optimal configuration is task-dependent. This is not a secondary finding; it is the paper's primary empirical contribution.

For starting artifact: One-function initialization produces the best ML pipeline on Housing Price (75.6% vs. 54.6% Kaggle percentile for best runs), but many-function initialization wins on Spaceship Titanic (86.6% vs. 72.7% for best runs, Table 1). The ordering flips between tasks—no single modularization strategy is universal.

For credit horizon: Multi-step optimization outperforms one-step in 4 of 8 Atari games (Pong, Breakout, Space Invaders, Asterix), while one-step outperforms multi-step in the other 4 (Freeway, Enduro, Q*bert, Seaquest) (Figure 5). The split is not random—games requiring strategic coordination (Space Invaders) benefit from longer traces, while games with aligned short-term and long-term rewards (Freeway) benefit from more frequent updates.

For experience batching: The optimal batch size varies across all 8 BBEH tasks (Table 2). Batch size 1 is best for Disambiguation QA (0.537) and Movie Recommendation (0.889). Batch size 3 is best for Geometric Shapes (0.389), Linguini (0.234), and Boolean Expressions (0.238). Batch size 5 is best for Dyck Languages (0.190) and Causal Understanding (0.531). Boardgame QA shows degradation under all batch sizes.

The implication for practice. If the optimal configuration of starting artifact, credit horizon, and experience batching is task-dependent—and the paper's evidence strongly suggests it is—then an engineer cannot set these configurations once and reuse them across domains. Each new optimization problem requires experimental determination of the right modularization, trace length, and batch size. This is the "setup burden" the paper identifies as the fundamental barrier to productionization. The paper does not claim to have solved this burden; it claims to have named and characterized it in a way that enables future systematic research.

4. Key Insights and Innovations

Innovation 1: The Learning Loop as a Design Space — Not an Algorithm

The paper's most fundamental contribution is reframing generative optimization from an algorithmic question ("what is the best optimizer or search strategy?") to a design space characterization ("what hidden engineering decisions determine whether any optimizer can succeed?"). Prior work on generative optimization has overwhelmingly focused on the algorithm itself—developing better search procedures (Lange et al., 2025a; Agrawal et al., 2025; Ren et al., 2026), more sophisticated feedback propagation mechanisms (Yuksekgonul et al., 2025; Cheng et al., 2024), or more effective candidate selection strategies like cross-validation and Pareto optimization (Khattab et al., 2024; Conway et al., 2025). The implicit assumption in these works is that if you build a powerful enough optimizer, it will work across tasks. The paper systematically dismantles this assumption: the same optimizer, with the same LLM backend, on similar tasks, can succeed or fail entirely depending on configuration decisions orthogonal to the optimizer itself.

What makes this reframing intellectually distinctive is not the empirical demonstration (though that is solid) but the conceptual move it enables. By positioning the three hidden decisions—starting artifact, credit horizon, experience batching—as the primary determinants of optimization success rather than implementation details, the paper transforms "this is hard to set up" from a complaint into a research agenda with identifiable sub-problems. The OPTO formalism (Appendix D) provides the theoretical language for this reframing: the learning template abstraction makes explicit that before the optimizer ever runs, the engineer has already committed to a particular structure for what evidence the optimizer will see and how it will be aggregated. This is a genuinely novel concept in the LLM optimization literature—prior work does not have a formal distinction between the workflow graph (what the system does) and the learning graph (what the optimizer sees), and consequently cannot analyze how changes to the template affect optimization dynamics.

This reframing is fundamental rather than incremental because it changes what researchers should study: not the next 5% improvement from a better search algorithm, but the systematic characterization of how template choices affect optimization outcomes across domains. The paper draws explicit parallels to traditional ML to underscore this point: just as the choice of neural network architecture, truncated backpropagation horizon, and batch size are not afterthoughts but core design parameters with well-studied tradeoffs, the learning loop's analogous decisions deserve the same rigorous treatment. The evidence that this reframing is necessary—and not merely conceptual—comes from the systematic task-dependence demonstrated across all three case studies: one-function initialization wins on Housing Price but many-function wins on Spaceship Titanic (Table 1), multi-step credit horizon helps in Space Invaders but hurts in Freeway (Figure 5), and no single batch size is optimal across all eight BBEH tasks (Table 2). If the algorithm were the dominant factor, these task-dependence patterns would not exist—or at least, they would be second-order effects. The paper shows they are first-order.

Innovation 2: Task-Dependence as the Central Empirical Finding — And Its Implication That No Simple Defaults Exist

While individual case studies demonstrating that different configurations work for different tasks might seem obvious in retrospect, the paper's systematic characterization of task-dependence across three orthogonal design dimensions, in three distinct domains, with controlled ablation is a novel empirical contribution that the field had not assembled before. Prior work either demonstrated a single successful configuration (e.g., DSPy's few-shot prompt optimization for a specific task; Khattab et al., 2024) or reported aggregate performance without breaking down which design choices mattered (e.g., Trace's demonstrations across diverse problems; Cheng et al., 2024). The paper's design—three cleanly separated case studies, each isolating one factor while holding the optimizer and LLM backend constant—enables a claim that no prior work could make: the brittleness of generative optimization is not due to any one of these factors being poorly configured in isolation, but rather to their interaction and the fact that optimal configurations do not transfer across tasks.

The intellectual significance of this finding goes beyond the empirical data. The paper is making a diagnostic argument about why production adoption lags despite research success: if every new optimization problem requires experimental determination of the right starting artifact, credit horizon, and batch size, then the engineering cost of setup exceeds the automation benefit for all but the highest-value problems. This explains the observed pattern where generative optimization succeeds in specialized domains (algorithm discovery at DeepMind, kernel optimization for hardware companies) where the setup investment can be amortized across many problems, but fails to achieve broad adoption in general agent engineering. The 9% adoption rate from Pan et al. (2025b) is thus not a failure of algorithmic capability but a rational economic response to configuration complexity.

This finding has a negative-results character that is unusual and valuable: the paper demonstrates that no universal configuration exists, which means the field cannot simply converge on "defaults" as traditional ML eventually did (Adam with learning rate 3e-4, ReLU activations, batch size 32). The paper's explicit parallel to traditional ML's development trajectory is telling here: just as Transformers (Vaswani et al., 2017) provided a broadly useful inductive bias and Adam (Kingma & Ba, 2014) worked well across architectures, the paper conjectures that generative optimization may eventually admit robust defaults—but its empirical results demonstrate that we are far from that point. The task-dependence across all three dimensions, replicated across three domains, makes this negative conclusion unusually strong: it is not the absence of evidence for defaults, but positive evidence that optimal configurations are genuinely task-specific.

Innovation 3: Meta-Overfitting as a Distinct Failure Mode in Generative Optimization

The paper identifies and names a phenomenon—meta-overfitting—that has no direct analog in the prior LLM optimization literature. In traditional machine learning, overfitting refers to a trained model performing well on training data but poorly on held-out data. In the context of generative optimization, the paper identifies a higher-level phenomenon: the optimizer itself learns to propose revisions that improve the immediate validation metric while degrading generalization, even though each proposed revision produces an entirely new model trained from scratch (Figure A3 in Appendix A.4). This is not the same as hyperparameter overfitting (where repeated evaluation on a validation set eventually overfits), though it is related—it is the optimizer learning to exploit the specificities of the validation split through its choice of pipeline architecture and training procedure, rather than improving the underlying approach.

What makes this conceptually distinctive is that it identifies a failure mode that is invisible in the standard evaluation protocol used by most prior work. If you only report the best validation performance achieved during optimization—as many papers do—meta-overfitting would appear as successful optimization. The gap between validation improvement and held-out test performance is the signature (Figure A3 vs. Table 1), and the paper explicitly identifies it as the same phenomenon driving the Boardgame QA failure in the BBEH case study, where the unoptimized baseline (0.371 accuracy) outperforms all optimized variants (0.341, 0.278, 0.276 for batch sizes 1, 3, 5; Table 2). This is a unification across domains: the same mechanism—the optimizer over-specializing to a small set of examples—manifests in code optimization (MLAgentBench) and prompt optimization (BBEH).

The significance of this observation is not just as a warning to practitioners but as a structural insight about the nature of LLM-based optimization. Unlike gradient-based optimization, where overfitting is controlled through regularization, early stopping, and held-out validation, the paper demonstrates that these standard defenses are insufficient for generative optimization. The optimizer has access to the validation feedback on every step and can exploit it in ways that are hard to detect from validation curves alone. The paper does not propose a solution—it identifies a problem that the field has not adequately addressed—and this diagnostic move is more valuable at this stage of the field's development than another incremental algorithmic improvement would be.

Innovation 4: The AI-as-Optimizer vs. Human-as-Optimizer Correspondence as a Lens for Systematic Research

The paper's most subtle contribution is its argument that generative optimization's challenges are structurally analogous to well-understood problems in traditional machine learning, and that this correspondence provides a lens for systematic research. The starting artifact problem parallels neural architecture search (Zoph & Le, 2017) and weight initialization (Glorot & Bengio, 2010); the credit horizon problem parallels debates in episodic reinforcement learning (Arjona-Medina et al., 2019) and truncated back-propagation through time (Tallec & Ollivier, 2017; Shaban et al., 2019); the experience batching problem parallels batch size selection in stochastic gradient descent (Smith et al., 2018).

Prior work on LLM-based optimization has occasionally noted superficial similarities to traditional ML (e.g., DSPy's "compiling" analogy; Khattab et al., 2024), but these have been metaphors rather than research frameworks. The paper's innovation is to treat the correspondence literally and systematically: the OPTO formalism (Appendix D) maps the learning template onto the structure of optimization problems studied in classical machine learning, making it possible to ask whether insights from decades of ML research transfer. For example, the finding that larger batches enable faster initial learning but can plateau earlier on BBEH tasks (Figure 7, e.g., Geometric Shapes where batch size 5 initially rises fastest but batch size 3 catches up) echoes the well-documented phenomenon in neural network training where larger batches converge to sharper minima with poorer generalization (Keskar et al., 2017). The staged feedback design in the Atari experiments (Appendix B.3) explicitly mirrors reward shaping in RL—a theoretical framework exists for analyzing when shaped rewards preserve optimal policies relative to sparse rewards.

This is a conceptual contribution rather than an empirical one: it does not solve any of the three problems, but it provides the intellectual scaffolding for solving them systematically rather than through ad hoc engineering. The paper's conjecture in Section 7—that sustained research may eventually yield robust defaults analogous to the Transformer architecture or the Adam optimizer—is not wishful thinking but a testable prediction derived from the correspondence: if the structural analogies hold, then the same research strategies that produced robust defaults in traditional ML (large-scale empirical sweeps over design space, theoretical analysis of failure modes, tasks-specific heuristics giving way to generalizable principles) should work for generative optimization's hidden design decisions.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three distinct task domains, each with its own dataset: (1) MLAgentBench — two Kaggle competition datasets, Housing Price and Spaceship Titanic, with pre-downloaded training data split into 80% train / 20% internal validation for optimization feedback, and a hidden Kaggle test set for final evaluation (Appendix A.1, A.4). (2) Atari — eight games from the Arcade Learning Environment (Pong, Breakout, Space Invaders, Freeway, Asterix, Enduro, Q*bert, Seaquest) using {game}-NoFrameskip-v4 environments via Gymnasium with action repeat 4 and sticky action probability 0.0, with object-centric state representations from OCAtari (Delfosse et al., 2024) rather than raw pixels (Appendix B.1, Table B2). (3) BigBench Extra Hard (BBEH) — eight tasks (Dyck Languages, Boolean Expressions, Geometric Shapes, Linguini, Disambiguation QA, Movie Recommendation, Boardgame QA, Causal Understanding) from Kazemi et al. (2025), with a fixed split of 15 training examples, 10 validation examples, and 175+ held-out test examples per task (Appendix C.1, Table C1).
Base model(s). All experiments use Claude Sonnet-3.5-v2 as the LLM backend, accessed during February–April 2025 (Appendix E.3.1). The optimizer is OptoPrime (Cheng et al., 2024) with memory size 5 across all experiments — it maintains the 5 most recent learning graphs in context to provide the optimizer with a history of past attempts. The paper explicitly states that the choice of optimizer and LLM is held fixed to isolate the three design factors being studied, noting that "the main focus of our paper is to study the factors that impact the learning loop, not the small differences between individual libraries" (Section 2). For baseline comparisons in MLAgentBench, the previously published ResearchAgent (Huang et al., 2024) is re-run with the same LLM backend where possible (Appendix E.3.1).
Metrics. Three task-specific metrics are used: (1) MLAgentBench — Kaggle leaderboard percentile and raw task metrics (RMSE for Housing Price, accuracy for Spaceship Titanic), with internal validation using F1 (Spaceship Titanic) or $r^2$ (Housing Price) for staged feedback and checkpoint selection (Section 4, Table 1, Appendix A.3). (2) Atari — raw game score normalized to map 0% to random performance and 100% to human performance (Figure 6), with per-game evaluation protocols (typically 4000-step episodes, with game-specific variations in Table B2) and 5 trials per configuration. (3) BBEH — held-out test accuracy, computed as exact match or (for multiple-choice tasks) the last parenthesized answer character (Appendix C.3), with 3 trials per batch size configuration where trials correspond to different random shufflings of the 15 training examples (Appendix C.1).
Baselines. The paper compares against several baselines across the three case studies: (1) ResearchAgent (Huang et al., 2024) for the MLAgentBench tasks — an earlier LLM-based ML research agent that can produce Kaggle submissions (Section 4, Table 1). (2) Object-centric deep RL baselines (DQN and PPO) for the Atari tasks, implemented using CleanRL (Huang et al., 2022) with the same OCAtari object representations converted to numeric vectors, trained for 10M environment steps with 5 seeds per game (Appendix B.5, Table B5a). (3) Unoptimized baseline for BBEH — the initial two-component agent (prompt template + answer extraction) evaluated on the held-out test set without any optimizer updates (Table 2, "Un-Optimized" rows).
Generation budget / compute accounting. The paper measures optimization progress in optimizer update steps, not model parameters or FLOPs: 20 steps for MLAgentBench, 20–30 steps for Atari (game-dependent, Table B2), and 15 steps for BBEH (Section 4, 5, 6). For BBEH, the fixed 15-step budget means different batch sizes see different numbers of total training examples: batch size 1 sees 15 examples (1 epoch), batch size 3 sees 45 examples (3 epochs), batch size 5 sees 75 examples (5 epochs) — the paper deliberately holds the update budget constant rather than the total data seen, to isolate whether seeing more examples per update (larger batch) is better than seeing fewer examples per update but more frequently (Appendix C.4). For Atari, the deep RL baselines use 10M environment steps for fair comparison (Appendix B.5), while the LLM-based optimizer uses substantially fewer environment interactions but is not directly FLOPs-matched to the neural baselines — the comparison in Figure 6 is intended to show competitive performance with less wall-clock time, not a controlled compute-efficiency study.
Cross-validation / statistical protocol. The paper uses distinct training, validation, and held-out test splits across all three domains, with no information leakage from test to optimization: (1) MLAgentBench — 80/20 train/validation split generated outside the agent; the best checkpoint on the validation metric is selected and submitted to Kaggle's hidden test set for final evaluation, with 5 independent trials per configuration (Appendix A.4). Kaggle does not permit sufficient test submissions for the test set to be used as an optimization signal, so the held-out validation split becomes the reward source. (2) Atari — 5 trials per game per credit horizon condition (one-step vs. multi-step), where each trial runs optimization for 20–30 steps and then evaluates the final learned agent on full episodes; results in Figure 5 report mean scores across trials. (3) BBEH — 3 trials per batch size configuration per task, where trials use different random shuffles of the same 15 training examples; validation is evaluated throughout training (Figure 7) and held-out test accuracy is computed after optimization completes (Table 2, Appendix C.1). Error bars in Figure 7 show standard error across the 3 trials.

Main Quantitative Results

Starting Artifact (MLAgentBench Case Study)

The headline finding in Section 4 is that the choice of starting artifact (one-function vs. many-function initialization) produces measurably different final ML pipelines, and the direction of the advantage flips between tasks. Table 1 reports the following:

Spaceship Titanic (evaluated by accuracy): The many-function initialization outperforms the one-function initialization on both average and best-case performance. The best pipeline from many-function initialization achieves 80.43% accuracy (surpassing 86.6% of Kaggle leaderboard submissions — Figure 4a, rightmost purple bar), while the best one-function pipeline achieves 80.00% (surpassing 72.7% of submissions). On average across 5 runs, many-function achieves 79.69% vs. one-function's 79.65% — a small average difference but with substantially higher best-case performance.
Housing Price (evaluated by RMSE, lower is better): The ordering flips. The best one-function pipeline achieves RMSE 0.129 (surpassing 75.6% of Kaggle leaderboard submissions — Figure 4b, rightmost green bar), while the best many-function pipeline achieves RMSE 0.141 (surpassing only 54.6% of submissions). The average RMSE across 5 runs is 0.135 for one-function vs. 0.147 for many-function.
Comparison to ResearchAgent baseline (Table 1): Both learned ML pipeline configurations substantially outperform ResearchAgent (Huang et al., 2024). On Spaceship Titanic, ResearchAgent achieves 78.17% average accuracy and 79.84% best accuracy — the learned pipelines improve by roughly 1.5–2.4 percentage points on average. On Housing Price, ResearchAgent achieves 0.149 average RMSE and 0.145 best RMSE — the one-function learned pipeline improves by roughly 10.4% on average (0.135 vs. 0.149). The paper states this as "around 11.5%-22.4% on average" improvement over ResearchAgent (Section 4, Results paragraph), though the specific percentage calculation method is not detailed.

The meta-overfitting trajectory (Figure A3). Figure A3 in Appendix A.4 shows a single optimization trajectory from Spaceship Titanic where validation F1 improves over 20 steps (from roughly 0.65 to 0.75) while the gap between train and validation performance widens. The paper interprets this as the optimizer discovering pipeline architectures that fit the validation split rather than improving the underlying ML approach. Unlike traditional overfitting (where a single model overfits), each point in Figure A3 represents an entirely new model trained from scratch with a different pipeline architecture — the optimizer is meta-overfitting by proposing architectures that work well on the specific validation split.

Credit Horizon (Atari Case Study)

The headline finding in Section 5 is that the optimal credit horizon is game-dependent: multi-step optimization outperforms one-step in 4 of 8 games, while one-step outperforms multi-step in the other 4 (Figure 5). The specific games and outcomes are:

Multi-step wins (4 games): Pong, Breakout, Space Invaders, and Asterix. For Space Invaders, multi-step approximately doubles the normalized score relative to one-step (roughly 45% vs. 25% by visual inspection of Figure 5, exact numbers not reported in text).
One-step wins (4 games): Freeway, Enduro, Q*bert, and Seaquest. For Freeway, the advantage is particularly clear — one-step achieves a substantially higher normalized score than multi-step (roughly 70% vs. 35% by visual inspection).
Interpretation from the paper (Section 5, Results): Games where multi-step helps (Pong, Breakout, Space Invaders, Asterix) involve "coordinating shooting and movement under delayed consequences" or "paddle-and-ball games where action quality depends on how returns shape future trajectories." Games where one-step helps (Freeway, Enduro, Q*bert, Seaquest) have "immediate rewards that accurately reflect progress toward the final goal," making frequent short-horizon updates more beneficial than waiting for episode completion.

Comparison to deep RL baselines (Figure 6): The LLM-based generative optimization achieves competitive scores with substantially less wall-clock time. The paper reports median runtime in minutes (Table B5b): LLM-based optimization takes 8.3 minutes (IQR: 4.3–21.0) vs. DQN at 291.6 minutes (IQR: 184.3–469.1) and PPO at 219.3 minutes (IQR: 163.3–255.2). However, the LLM approach achieves a median normalized score of 44.3% (IQR: 3.0–100.2) compared to DQN at 71.5% (IQR: 47.5–114.2) and PPO at 108.1% (IQR: 71.6–143.7). The paper notes this comparison is not strictly FLOPs-matched — the deep RL baselines use 10 parallel environments while LLM uses 1, and the scores are normalized differently (random = 0%, human = 100%, with PPO sometimes exceeding human performance). The comparison is intended to demonstrate that generative optimization achieves non-trivial performance with dramatically less engineering effort and wall-clock time, not to claim superiority over deep RL.

Key configuration detail (Table B2): The multi-step rollout lengths are game-specific: 400 steps for Pong, 300 for Breakout, 25 for Space Invaders, 100 for Freeway/Asterix/Enduro/Q*bert/Seaquest. The Space Invaders value (25 steps) is notably short — the paper states this was chosen because "its dense reward structure already provides informative per-step feedback," so even a short multi-step trace captures enough context to improve over single-step updates.

Experience Batching (BigBench Extra Hard Case Study)

The headline finding in Section 6 is that larger batch sizes do not monotonically improve generalization, and the optimal batch size is strictly task-dependent (Table 2). The paper reports held-out test accuracy across 8 BBEH tasks for 4 conditions (unoptimized baseline, batch sizes 1, 3, 5), each with 3 trials:

Batch size 1 is best for: Disambiguation QA (0.537 ± 0.036) and Movie Recommendation (0.889 ± 0.038). For Movie Recommendation, this represents a massive improvement over the unoptimized baseline (0.238) — the largest absolute gain for any task-batch combination in the table.
Batch size 3 is best for: Geometric Shapes (0.389 ± 0.040), Linguini (0.234 ± 0.012), and Boolean Expressions (0.238 ± 0.006). For Geometric Shapes, batch size 3 outperforms batch size 5 (0.200) by a factor of nearly 2×.
Batch size 5 is best for: Dyck Languages (0.190 ± 0.031) and Causal Understanding (0.531 ± 0.018). For Causal Understanding, batch size 5 substantially outperforms batch size 1 (0.375) and batch size 3 (0.408).
Optimization degrades performance for: Boardgame QA, where the unoptimized baseline achieves 0.371 ± 0.003 and all optimized variants perform worse (batch 1: 0.341, batch 3: 0.278, batch 5: 0.276). The paper identifies this as meta-overfitting — the optimizer finds prompts and code that perform well on the 15 training examples but fail to generalize to the test distribution (Appendix C.5).

Validation learning curves (Figure 7): The paper reports validation accuracy across optimization iterations for selected tasks, showing different convergence patterns. Larger batch sizes often enable faster initial learning — in Geometric Shapes, batch size 5 rises fastest in the first 5 iterations before plateauing, while batch size 3 continues to improve and eventually catches up. Smaller batches show noisier learning curves (batch size 1 for Dyck Languages oscillates substantially) but sometimes sustain improvement longer. The paper draws an explicit parallel to "classical batch size trade-offs in neural network training, where larger batches provide more stable gradients but may converge to different local optima than smaller batches" (Section 6, Results).

Critical experimental detail (Appendix C.4): Because the 15-step update budget is fixed, batch size determines the number of epochs: batch size 1 uses 1 epoch, batch size 3 uses 3 epochs, batch size 5 uses 5 epochs. This means batch size 5 configurations see each training example 5 times during optimization, while batch size 1 sees each example only once. The paper deliberately holds updates constant rather than data seen to match the classical SGD batch size analysis where the question is: given a fixed number of gradient steps, is it better to use larger or smaller batches?

Ablation Studies and Robustness Checks

Staged feedback design (Appendix A.3, B.3, C.3): The paper uses task-specific staged feedback templates in MLAgentBench and Atari, but task-agnostic binary feedback in BBEH. This is not presented as a formal ablation (the feedback design is not varied within any single case study), but the paper includes enough detail to assess sensitivity: the MLAgentBench templates provide different natural-language guidance at different performance thresholds (Table A1), while BBEH uses a minimal binary correctness signal with ground truth revealed for failures (Appendix C.3). The fact that all three case studies show task-dependent optimal configurations despite different feedback designs suggests the three factors under study are robust to feedback variation, though this is not experimentally verified.

Deep RL baseline implementation details (Appendix B.5): The paper transparently reports that the deep RL baselines represent "what a single developer can achieve in a reasonable amount of time" rather than state-of-the-art tuned systems. A master's student implemented the baselines in approximately three days, with two weeks for training and debugging. The hyperparameters are inherited from CleanRL defaults with only "a few hand-chosen MLP size adjustments that varied by game" — no broad hyperparameter sweep or systematic architecture search was performed. The paper acknowledges this explicitly: the goal is "to obtain reasonable neural baselines with modest engineering effort, not to carry out an exhaustive baseline-optimization campaign."

Meta-overfitting across domains: The paper identifies meta-overfitting in two of the three case studies: MLAgentBench (Figure A3, Appendix A.4) and BBEH Boardgame QA (Table 2, Appendix C.5). For Boardgame QA, this is a particularly clean demonstration because the artifact being optimized is a prompt and extraction function rather than code — it shows that meta-overfitting is not specific to program synthesis but is a general phenomenon in LLM-based optimization when the training set is small relative to the optimizer's capacity to over-specialize. The paper notes that "this problem can be addressed by reshuffling the training and validation set as a whole, though such a design choice is beyond the scope of this paper" (Appendix C.5), suggesting a direction for future work without pursuing it experimentally.

Improvement-style feedback (Appendix A.3): In MLAgentBench, the paper adds improvement suggestions to the feedback when the model fails to improve relative to the previous step. This is not ablated — we cannot tell from the reported experiments whether this additional feedback signal meaningfully affects optimization outcomes compared to staged feedback alone. The paper presents it as part of the feedback design but does not analyze its contribution.

One-function vs. many-function ablation control: The paper states that both MLAgentBench initializations "contain equivalent information in their docstrings; the only difference is the level of modularization" (Figure 3 caption). The one-function docstring is "a concatenation of all docstrings for the many-function initialization" (Figure A1a caption). However, the actual code skeletons differ beyond modularization: the many-function version explicitly chains function calls in __call__ (preprocess → select_features → train_model → predict), while the one-function version hands all logic to train_model. This means the many-function version imposes a specific pipeline structure that the optimizer cannot change (the sequence of operations is fixed), while the one-function version leaves the optimizer free to implement any internal structure. The experimental comparison is therefore between imposed modular structure vs. optimizer-chosen structure, not just between two equivalent representations of the same constraint.

Critical Assessment

The paper's central claim is that the three hidden design decisions — starting artifact, credit horizon, and experience batching — "can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work" and that "the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization." The experiments provide solid evidence for the task-dependence of each individual factor, but several important gaps exist between what was tested and what the claim asserts.

Do the experiments demonstrate that starting artifact determines optimization success? Partially. The MLAgentBench experiments show that one-function vs. many-function initialization produces different Kaggle leaderboard performance (Table 1), and the advantage flips between tasks (Spaceship Titanic favors many-function, Housing Price favors one-function). However, this demonstrates that starting artifact matters — it does not demonstrate that it determines success, because the paper does not report cases where one initialization completely fails while the other succeeds. Both initializations produce valid Kaggle submissions with non-trivial performance on both tasks. The worst reported result (many-function on Housing Price, 54.6% Kaggle percentile) is still substantially above random. To demonstrate that starting artifact determines success, the paper would need to show at least one configuration where optimization completely fails — produces invalid code, fails to converge, or performs below baseline. This is not shown.

Additionally, the two initialization conditions differ in more than modularization. The many-function version forces a specific pipeline structure (preprocess → select_features → train_model → predict) that the optimizer must work within. The one-function version gives the optimizer freedom to implement any internal structure. The experiment therefore confounds modularization with constraint on the search space: the many-function condition imposes an architectural prior, while the one-function condition allows the optimizer to discover architecture. It is possible that the Housing Price advantage for one-function is not about modularization per se, but about the optimizer discovering a pipeline structure that happens to work better for that dataset — a structure it could not express in the constrained many-function setup. The paper does not discuss this confound.

Do the experiments demonstrate that credit horizon determines optimization success? Solidly for the comparative claim, but weakly for the absolutist claim. Figure 5 clearly shows that one-step and multi-step credit horizons produce different normalized scores, and the ordering reverses across games. However, neither condition "fails" — both produce non-trivial scores on all 8 games. The worst-performing condition in Figure 5 appears to be one-step on Pong (roughly 5–10% normalized score by visual inspection), but this is not zero. The paper frames the credit horizon as a "genuine design choice" (Section 5, Results), and the experiments support this characterization — they show that the choice matters and that the optimal depends on the task — but they do not show that an incorrect credit horizon causes catastrophic failure.

The deep RL comparison (Figure 6, Table B5b) is under-analyzed. The LLM approach achieves only 44.3% median normalized score vs. PPO's 108.1% — less than half the performance with dramatically less compute. The paper presents this as evidence that "generative optimization achieves competitive scores with substantially less wall-clock time," which is true but incomplete: it also achieves substantially lower absolute performance. A reader interested in whether to use generative optimization for game-playing should know both numbers — the speed advantage and the performance ceiling. The paper's framing emphasizes the former.

Do the experiments demonstrate that experience batching determines optimization success? The strongest evidence among the three case studies. Table 2 shows that batch size matters systematically, with optimal batch size varying across 7 of 8 tasks (Boardgame QA being the exception where all batch sizes degrade performance). The gaps are substantial: on Geometric Shapes, batch size 3 (0.389) nearly doubles batch size 5 (0.200). On Causal Understanding, batch size 5 (0.531) substantially outperforms batch size 1 (0.375). These are not noise-level differences — they are large enough to determine whether optimization is worthwhile.

However, the experiment conflates batch size with number of epochs. Batch size 5 sees each training example 5 times, while batch size 1 sees each example once. It is possible that the apparent benefit of larger batches on some tasks (e.g., Causal Understanding) is actually a benefit of seeing the training data more times — i.e., more epochs rather than larger batches. To disentangle these, the paper would need to either hold epochs constant (varying batch size and total updates) or hold total examples seen constant (varying batch size and epochs inversely). The current design cannot distinguish between "batch size matters" and "epochs matter," because they are perfectly confounded. The paper does not discuss this confound.

Missing experiment: interaction effects. Each case study isolates one factor, but in practice an engineer must set all three simultaneously. The paper provides no evidence about whether the optimal starting artifact depends on the credit horizon, or whether the optimal batch size depends on the starting artifact. These interactions are almost certainly non-trivial — for example, a modular starting artifact might benefit from longer credit horizons because the optimizer can attribute credit to specific components — but they are completely unexplored. This means the paper's practical guidance ("choose the right starting artifact, credit horizon, and batch size") cannot be operationalized: if the optimal depends on interactions that were never measured, the engineer still faces a combined search space rather than three independent decisions.

Missing experiment: difficulty estimation cost. The paper identifies the setup burden as the core problem but does not measure the cost of determining the optimal configuration for a new task. To claim that "no simple defaults exist" is a practical barrier, the paper should estimate how many trials are needed to find good configurations — or at minimum, report the variance in outcomes within a single configuration to give a sense of how expensive hyperparameter search would be. The 5-trial averages in MLAgentBench and Atari, and the 3-trial averages in BBEH, provide some evidence of variance, but the paper does not analyze whether a practitioner could identify good configurations with reasonable effort (e.g., by running a few trials per candidate and comparing).

Missing baseline: random search over configurations. The paper compares specific configurations (batch sizes 1, 3, 5; one-step vs. multi-step; one-function vs. many-function) but does not report the expected performance if an engineer randomly chose among these options. Given the task-dependence, a random choice would sometimes pick the optimal and sometimes the worst — the paper could strengthen its argument that these choices matter by showing that random configuration selection produces substantially worse performance than the best configuration per task.

Single LLM backend limitation. All experiments use Claude Sonnet-3.5-v2. The paper's claims about task-dependence of optimal configurations may be specific to this model — a stronger model might be less sensitive to starting artifact (because it can recover from poor initializations more reliably), or a weaker model might benefit more from shorter credit horizons (because longer traces exceed its effective context utilization). The paper acknowledges that "framework-specific factors could exist" (Section 2) but extends this caveat to the optimizer, not the underlying LLM. The choice of Claude Sonnet-3.5-v2 is reasonable for a study conducted in early 2025, but the generality of the findings to other model families (GPT-4, Gemini, open-source models) is an open question that the experiments do not address.

The Boardgame QA negative result is under-explored. The paper flags Boardgame QA as a case where optimization degrades performance (unoptimized baseline 0.371 vs. best optimized 0.341), attributing this to meta-overfitting. This is the strongest evidence in the paper that generative optimization can actively make things worse, which is arguably more important for practitioners than the finding that optimal configurations vary by task. Yet the paper provides no analysis of why Boardgame QA is susceptible to meta-overfitting while other tasks with the same training set size (15 examples) are not. Is it a property of the task distribution? The prompt initialization? The feedback signal? Understanding this failure mode would be more practically valuable than knowing the optimal batch size for each task.

Strengths in experimental design. Despite these gaps, the paper's experimental methodology has several genuine strengths: (1) The isolation of individual factors in separate case studies enables relatively clean attribution, even if the confounds discussed above exist. Most prior work on generative optimization varies multiple factors simultaneously and cannot attribute performance differences to specific design choices. (2) The use of held-out test sets across all three domains (Kaggle hidden test set, full Atari episodes, BBEH test split) prevents the common failure mode of reporting optimization progress on the same data used for feedback. (3) Multiple trials with standard errors provide a sense of variance, even if formal statistical tests are not reported. (4) The transparent reporting of implementation details (the OCAtari wrapper, the staged feedback templates, the exact batch size-to-epoch mapping) enables replication and critical analysis that would be impossible with less detailed appendices. (5) The negative result on Boardgame QA is reported prominently rather than buried, which is unusual and valuable in a field that often emphasizes success cases.

6. Limitations and Trade-offs

The Difficulty Estimation Cost for Learning Loop Configuration

The assumption. The paper's central claim—that the three hidden design decisions (starting artifact, credit horizon, experience batching) must be configured per-task because no universal defaults exist—implicitly assumes that determining the optimal configuration for a new task is practical. The paper does not measure or account for the cost of this configuration search.

The consequence. If the optimal starting artifact, credit horizon, or batch size is genuinely task-dependent (as the paper demonstrates), then an engineer deploying generative optimization to a new problem must experimentally determine these settings before the learning loop can operate effectively. For MLAgentBench, this means running multiple 20-step optimization trials with different initializations to determine which modularization works better. For Atari, it means comparing one-step and multi-step horizons across games to identify which produces higher scores. For BBEH, it means sweeping batch sizes {1, 3, 5} to find the one that yields the best held-out test accuracy. The paper's own protocol for each case study—5 trials per configuration in MLAgentBench, 5 trials per game per credit horizon in Atari, 3 trials per batch size per task in BBEH—represents the minimum cost to detect the differences reported. In production, this cost would be incurred for every new task before any optimization benefit is realized, potentially exceeding the benefit of automation itself for tasks with modest performance requirements. The paper flags this meta-problem indirectly—"the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization" (Section 1)—but does not analyze the meta-cost of configuration search as a barrier.

What evidence exists in the paper. None directly. The paper reports the number of trials per configuration (5 for MLAgentBench and Atari, 3 for BBEH) and the iteration budgets (20 steps for MLAgentBench, 20–30 for Atari, 15 for BBEH), from which the total optimization cost can be estimated. However, the paper never reports the cost to arrive at the conclusion that a particular configuration is optimal—e.g., how many total LLM API calls or wall-clock hours were needed to determine that one-function initialization is better than many-function for Housing Price. The Boardgame QA negative result (Table 2) is particularly telling: running 3 trials each at batch sizes 1, 3, and 5 (9 total trials, each with 15 optimizer steps) would be required to discover that none of these batch sizes improve over the unoptimized baseline—a substantial cost for a negative finding.

Mitigation status. The paper does not address this at all. The Discussion (Section 7) envisions future "robust defaults" analogous to Adam or Transformers, but does not propose methods for reducing the configuration search burden in the interim. The task-dependence finding itself makes the configuration search problem harder rather than easier: it establishes that the search cannot be avoided by adopting a known default. No method is proposed for predicting optimal configuration from task characteristics, no adaptive strategy for interleaving configuration search with optimization is suggested, and no sensitivity analysis indicates whether near-optimal configurations exist that reduce the precision needed for selection.

No Evidence of Interaction Effects Between the Three Design Factors

The assumption. Each case study isolates a single factor (starting artifact in Section 4, credit horizon in Section 5, experience batching in Section 6) and draws conclusions about that factor's task-dependence independently. The implicit assumption is that these factors are separable—that the optimal credit horizon does not depend on the starting artifact, and the optimal batch size does not depend on the credit horizon.

The consequence. In practice, an engineer must set all three simultaneously. If interactions exist—for example, if modular starting artifacts benefit from longer credit horizons because they allow the optimizer to attribute feedback to specific components, or if smaller batch sizes enable more frequent updates that compensate for shorter credit horizons—then the per-factor task-dependence reported in the paper does not provide sufficient guidance. The search space becomes the Cartesian product of configurations: for MLAgentBench alone, choosing between 2 initializations × 2 credit horizons × 3 batch sizes would require 12 configurations to explore, each with 5 trials and 20 optimization steps. The paper's claim that "no single universal recipe works across all tasks" (Section 7) understates the problem: even if per-task recipes existed, the interaction space makes finding them combinatorially expensive. The paper's framing of the problem as "three independent decisions" rather than "one decision with a combined configuration space" may therefore overstate how solvable the setup burden is.

What evidence exists in the paper. None. No experiment combines two or more of the three factors. The MLAgentBench case study uses a fixed feedback design (staged templates) and fixed optimizer configuration; the Atari case study uses a fixed starting artifact (modular agent design) and fixed feedback; the BBEH case study uses a fixed agent design and fixed optimizer. The paper is explicit about its isolation strategy—"We sought to isolate learning-loop design choices that are often treated as implementation details" (Section 7)—but does not acknowledge that isolating factors for analysis does not imply they are independent in practice. The OPTO formalism (Appendix D) provides a unified language that could in principle represent interactions (the learning template encodes credit horizon and experience batching simultaneously), but the experiments never test joint configurations.

Mitigation status. Not addressed. The paper does not flag the absence of interaction studies as a limitation, nor does it suggest future work to characterize interactions between the three factors. The Discussion envisions eventual "robust defaults" that "transfer across agent designs and domains," but does not consider whether interactions might make such defaults inherently impossible—i.e., whether the optimal starting artifact inherently depends on the credit horizon in a way that prevents either from having a task-independent default.

Single LLM Backend Limits Generality of Findings

The assumption. All experiments in the paper use Claude Sonnet-3.5-v2 as the LLM backend, accessed between February and April 2025 (Appendix E.3.1). The paper's claims about task-dependence of starting artifact, credit horizon, and experience batching are drawn exclusively from this model's behavior.

The consequence. The task-dependence patterns may be specific to Claude Sonnet-3.5-v2 rather than universal properties of LLM-based optimization. Several plausible model-dependent effects exist: (1) A stronger model might be less sensitive to starting artifact because it can recover from poor initial modularization more reliably—if the one-function initialization on Spaceship Titanic underperforms, a more capable optimizer might deduce the need for modular decomposition internally. (2) A model with longer effective context utilization might benefit differently from credit horizon choices—a model that loses coherence over long traces would show stronger preference for short horizons than one that handles 400-step Atari rollouts robustly. (3) The optimal batch size may depend on the model's ability to reason globally across multiple examples (Schnabel et al., 2025, cited in the paper)—models with different global reasoning capacity could show different batch-size optimization curves. If these model-dependent effects exist, the paper's specific findings (one-function wins on Housing Price, multi-step helps in Space Invaders, batch size 3 is best for Geometric Shapes) may not transfer to GPT-4, Gemini, or open-weight models—and more importantly, the qualitative conclusion that optimal configurations are task-dependent might hold for Claude but not for a sufficiently capable model that is robust to configuration variation.

What evidence exists in the paper. None. The paper uses only one LLM backend throughout all three case studies. The authors note that "the experiments in this paper were conducted during the period of February 2025 to April 2025" (Appendix E.3.1) and that they re-ran ResearchAgent (Huang et al., 2024) with the same model endpoint "whenever possible so that differences in access conditions do not dominate the comparison." This is good practice for controlled comparison within the paper but provides no evidence about model generality. The paper acknowledges that "framework-specific factors could exist" (Section 2) but extends this caveat to the optimization framework (Trace/OptoPrime), not the underlying LLM.

Mitigation status. The paper does not discuss this limitation or propose multi-model experiments for future work. The Discussion focuses on discovering "starting artifacts for agents that are broadly optimizable across tasks" and "robust ways to structure the learning context" (Section 7), implicitly assuming that such defaults would transfer across models. Given the pace of LLM development (Claude Sonnet-3.5-v2 is already superseded by Claude 4 as of 2025), the shelf life of per-model configuration findings is an unexamined concern.

Interaction Between Batch Size, Epoch Count, and Total Updates in BBEH Experiments

The assumption. The BBEH experiments compare batch sizes 1, 3, and 5 while holding the number of optimizer update steps fixed at 15. This means the number of epochs (passes over the 15-example training set) varies: batch size 1 uses 1 epoch, batch size 3 uses 3 epochs, batch size 5 uses 5 epochs (Appendix C.4). The paper interprets differences between these conditions as effects of batch size.

The consequence. This design conflates batch size with two other variables: number of epochs (how many times each training example is seen) and total training examples processed (15, 45, and 75 examples for batch sizes 1, 3, and 5 respectively). Any observed difference between batch size conditions could be due to: (1) the batch size itself (as the paper claims), (2) the number of epochs (batch size 5 sees each example 5 times, potentially enabling more thorough learning from the small training set), or (3) the total examples seen (batch size 5 processes 5× more examples, providing more total information to the optimizer). The task-dependent optimal batch sizes reported in Table 2 (e.g., batch size 5 best for Causal Understanding at 0.531 vs. 0.375 for batch size 1) cannot be attributed to batch size alone without disentangling these confounds. If Causal Understanding benefits from batch size 5 primarily because of the additional epochs (each example seen 5 times rather than once), then a batch size 1 configuration with 5× more update steps (75 steps) might achieve the same or better performance—but this configuration was never tested.

What evidence exists in the paper. Appendix C.4 explicitly describes the design: batch size 1 → 1 epoch, batch size 3 → 3 epochs, batch size 5 → 5 epochs. The paper notes this in passing but does not analyze it as a confound. Figure 7 shows validation learning curves across optimization iterations, where the x-axis is "optimizer update steps" not total examples seen—batch size 5 completes 75 examples by step 15, while batch size 1 completes only 15 examples. The curves therefore show different amounts of total information at each x-axis position, making direct comparison of learning speed difficult. The paper cites Smith et al. (2018) on batch size in SGD as motivation (Section 6), but classically, SGD batch size experiments hold total data seen or total epochs constant while varying batch size and steps inversely—the opposite of the paper's design.

Mitigation status. The paper does not acknowledge this confound or propose alternative experimental designs. No experiment holds total epochs constant (varying batch size and steps inversely) or holds total examples seen constant. This is the most significant methodological weakness in the otherwise careful experimental design, because it undermines the interpretation of one of the paper's three main empirical findings.

The Boardgame QA Meta-Overfitting Failure Mode Is Unexplored

The assumption. The paper identifies Boardgame QA in the BBEH case study as a task where all three batch sizes degrade performance relative to the unoptimized baseline (unoptimized: 0.371; batch 1: 0.341; batch 3: 0.278; batch 5: 0.276; Table 2). The paper attributes this to "meta-overfitting" and notes it as "the clearest failure case" (Appendix C.5).

The consequence. This negative result is arguably the most practically important finding in the paper—it demonstrates that generative optimization can actively make a system worse—yet the paper provides almost no analysis of why it occurs. If meta-overfitting on small training sets is a general failure mode (as the MLAgentBench evidence in Figure A3 also suggests), then practitioners need to know: (1) What task properties predict susceptibility to meta-overfitting? Is it training set size, task complexity, prompt initialization quality, or some interaction? (2) What batch sizes or credit horizons mitigate meta-overfitting? The BBEH results show that larger batches (batch 5: 0.276) perform even worse than smaller batches (batch 1: 0.341), but this pattern is observed for only one task and may not generalize. (3) At what training set size does meta-overfitting disappear? The paper uses 15 training examples for all BBEH tasks—would 30 or 50 examples prevent the Boardgame QA degradation? Without answers to these questions, a practitioner cannot assess whether their target task is at risk of optimization-induced degradation, making generative optimization an unpredictable intervention.

What evidence exists in the paper. Table 2 reports the raw numbers for Boardgame QA. Appendix C.5 devotes one short paragraph to it, stating that "this problem can be addressed by reshuffling the training and validation set as a whole, though such a design choice is beyond the scope of this paper." No experiment tests this reshuffling hypothesis. MLAgentBench Figure A3 shows a related phenomenon (optimizer overfitting to validation split), but the Boardgame QA case is arguably worse because it affects a simpler artifact (prompt + extraction function rather than full ML pipeline) and occurs on a task where even modest improvement should be achievable given the baseline. The paper does not compare Boardgame QA to the tasks where optimization succeeded (e.g., Movie Recommendation, where batch size 1 achieved 0.889 from a baseline of 0.238) to identify distinguishing characteristics.

Mitigation status. Minimally. The paper acknowledges the existence of the failure but provides no diagnostic framework, no ablation to identify causes, and no guidance for practitioners to detect or avoid meta-overfitting on their own tasks. The suggestion to reshuffle the training/validation split is untested and amounts to speculation. Given that the paper's central argument is about the difficulty of setting up learning loops, this specific failure mode—where the learning loop not only fails to help but actively harms performance—should receive more attention than a brief appendix paragraph.

Atari Deep RL Baseline Comparison Is Not Controlled for Compute or Performance

The assumption. Figure 6 presents wall-clock time on the x-axis and normalized score on the y-axis, comparing the LLM-based generative optimization approach against object-centric DQN and PPO baselines. The paper interprets this as showing that "generative optimization achieves competitive scores with substantially less wall-clock time" (Section 5, Results; Appendix B.5).

The consequence. The comparison conflates multiple uncontrolled variables and provides limited information for a practitioner deciding between approaches. Specifically: (1) Compute is not controlled. The LLM optimizer runs on 1 environment instance while DQN uses 1 and PPO uses 10 parallel environments (Appendix B.5, Table B5a). The LLM optimizer benefits from a pretrained model (Claude Sonnet-3.5-v2) whose training cost is not amortized into the comparison, while the deep RL baselines learn from scratch. This is not a like-for-like comparison—it compares fine-tuning a large pretrained model against training a small neural network from scratch, where the former has orders of magnitude more pretraining FLOPs embedded in it. (2) Performance is substantially lower for the LLM approach. The median normalized score for LLM is 44.3% (IQR: 3.0–100.2) compared to DQN's 71.5% (IQR: 47.5–114.2) and PPO's 108.1% (IQR: 71.6–143.7) (Table B5b). The LLM approach achieves less than half the performance of PPO with roughly 30× less wall-clock time (8.3 minutes vs. 219.3 minutes). A practitioner interested in maximizing final score would choose PPO; one interested in rapid prototyping might choose LLM—but the paper's framing ("competitive scores") obscures the tradeoff. (3) The deep RL baselines are not tuned. Appendix B.5 states that the baselines used "only a few hand-chosen MLP size adjustments," no hyperparameter sweep, and a master's student's implementation in three days. The reported PPO and DQN scores are therefore lower bounds on what an RL practitioner could achieve, making the LLM approach look more competitive than it would against properly tuned baselines.

What evidence exists in the paper. Table B5b reports the exact numbers; Appendix B.5 transparently describes the baseline implementation limitations. The paper acknowledges that the baselines represent "what a single developer can achieve in a reasonable amount of time" and that the goal is "not to carry out an exhaustive baseline-optimization campaign." To the paper's credit, this transparency prevents the comparison from being misleading to careful readers. However, the main text (Section 5, Figure 6) presents the comparison without these caveats, and the takeaway "competitive scores with substantially less wall-clock time" is the narrative a casual reader would retain.

Mitigation status. Partial. The appendix is transparent about limitations, but the main text presentation is not. The paper does not propose a more controlled comparison (e.g., FLOPs-matched, or an ablation of the LLM approach with scaled-down pretraining) and does not acknowledge the embedded-pretraining asymmetry as a limitation of the comparison. The suggestion that generative optimization "may eventually admit robust defaults" (Section 7) implicitly assumes that the performance gap is closable through better configuration, but the experimental evidence shows a large absolute gap between LLM (44.3% median) and PPO (108.1% median) that better batching or credit horizons are unlikely to close entirely.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new algorithm, optimizer, or benchmark. It makes a diagnostic intervention in a field that has been overwhelmingly focused on building better optimization machinery. The shift is from asking "what's the best way to optimize?" to asking "what hidden decisions, made before optimization even starts, determine whether any optimizer can succeed?" This is a reframing of the problem rather than a solution to it, and its impact will be measured not by citation count but by whether the field redirects research attention from algorithmic improvements to systematic characterization of the learning loop design space.

The specific conceptual move that matters. The OPTO formalism's distinction between the workflow graph (what a single execution produces) and the learning graph (what the optimizer actually sees, constructed by a learning template) makes explicit something that was previously only implicit engineering intuition. Prior work on generative optimization treated the structure of the optimization loop—what traces are shown to the optimizer, how many are aggregated, how much of a multi-step trajectory is included—as implementation detail. This paper demonstrates that these choices are not detail; they are the primary determinants of whether optimization succeeds. The learning template abstraction provides a formal language for discussing these choices, enabling researchers to ask: given a particular template structure, what optimization dynamics should we expect? This is analogous to how the distinction between model architecture and training algorithm in traditional ML enabled systematic study of each independently.

Resolving the tension between research success and production failure. The paper's central empirical observation—that the same optimizer, with the same LLM backend, on superficially similar tasks, can produce dramatically different outcomes depending on how the learning loop is configured—provides a unified explanation for the disconnect that motivated the work. Research papers demonstrate successful generative optimization by carefully (but implicitly) choosing loop configurations that work for their target domains. Production engineers, facing arbitrary new tasks, lack the configuration knowledge and discover through trial and error that optimization is unreliable. The paper's contribution is not to close this gap—it doesn't propose a solution—but to name and characterize it in terms that make systematic investigation possible. The 9% adoption rate from Pan et al. (2025b) is thus reinterpreted: it is not evidence that generative optimization is ineffective, but evidence that the configuration knowledge required to make it effective has not been codified or made transferable.

Which research directions become more attractive, and which become less so. The paper's findings argue strongly against continued investment in incremental optimizer improvements as the primary research strategy. If the limiting factor is not optimizer quality but loop configuration—as the paper's systematic evidence suggests, since the same OptoPrime optimizer with the same Claude backend can succeed or fail depending on artifact structure, credit horizon, or batch size—then developing a 5% better search algorithm will not meaningfully improve production reliability. The paper instead redirects attention toward: (1) understanding the structure of the learning loop design space (what are the dimensions, how do they interact, can we predict optimal configurations from task characteristics?), (2) developing methods that are robust to configuration rather than requiring per-task tuning, and (3) building diagnostic tools that help engineers identify when their loop is misconfigured. These are fundamentally different research questions from "can we build a better optimizer?", and the paper's evidence suggests they are the right questions for closing the research-to-production gap.

The meta-overfitting finding adds an underappreciated failure mode to the field's vocabulary. Prior work on generative optimization has largely focused on whether optimization improves performance, with the implicit assumption that more optimization steps are monotonically beneficial. The paper demonstrates—in two distinct domains (MLAgentBench in Appendix A.4, Boardgame QA in Section 6 and Appendix C.5)—that optimization can actively degrade performance relative to the unoptimized baseline, and that this degradation is invisible if you only monitor validation metrics during optimization. This finding changes how practitioners should evaluate generative optimization: the relevant comparison is not "did validation improve during optimization?" but "does the final artifact outperform the initial artifact on held-out data?" The paper gives this failure mode a name (meta-overfitting) and identifies its structural cause (the optimizer learns to exploit the specificities of the finite training/validation split rather than learning generalizable improvements), which makes it a target for future mitigation research rather than an inexplicable pathology.

The correspondence to traditional ML as a research accelerator. The paper's argument that the three hidden design choices map onto well-understood concepts in traditional ML (architecture initialization, truncated backpropagation through time, SGD batch size) is not merely a metaphor—it is a research strategy. If the correspondence is substantive rather than superficial, then decades of theoretical and empirical work in traditional ML can be adapted to generative optimization. For example: the finding that larger batch sizes enable faster initial learning but can plateau earlier or converge to worse generalization (Figure 7, e.g., Geometric Shapes where batch size 5 initially rises fastest but batch size 3 catches up) mirrors the well-documented "sharp minima" phenomenon in neural network training (Keskar et al., 2017). If this parallel holds, then techniques developed for SGD batch size tuning—learning rate scaling rules, adaptive batch size schedules, gradient noise scale analysis—may transfer to generative optimization with appropriate modification. The paper does not pursue these transfers, but by establishing the correspondence clearly, it enables other researchers to do so.

Follow-Up Research This Work Enables

Interaction effects between starting artifact, credit horizon, and experience batching. The paper isolates each factor in separate case studies, but in practice an engineer must set all three simultaneously. A natural follow-up would test whether the optimal credit horizon depends on the starting artifact, or whether the optimal batch size depends on the credit horizon. For example: modular starting artifacts might benefit from longer credit horizons because the optimizer can attribute feedback to specific components (the predict_ball_trajectory function vs. the select_action function in Atari), while monolithic artifacts might be insensitive to horizon because all changes affect a single function. A concrete experiment would use the Atari setup from Section 5, cross the one-step/multi-step credit horizon conditions with two starting artifact designs (the current modular design vs. a single monolithic policy function containing all logic), and measure whether the credit horizon effect size changes. If interactions are large, the search space for configuration becomes combinatorial; if interactions are small, the three factors can be tuned independently, substantially reducing the setup burden. The paper's silence on this question is its largest open empirical gap.

Predicting optimal configuration from task characteristics. The paper demonstrates that optimal configurations are task-dependent but provides no method for predicting which configuration will work for a new task without running experiments. A strong follow-up would develop a task featurization that predicts optimal starting artifact, credit horizon, or batch size from properties of the task description, dataset, or initial execution traces. For credit horizon: can we predict whether a game benefits from multi-step traces using features like reward density (average reward per step), action-repeat structure, or whether the initial agent's score correlates with episode length? For experience batching: can we predict optimal batch size from training set size, inter-example diversity (measured by embedding distance between questions in BBEH tasks), or the optimizer's initial success rate? A concrete experiment would use the 8 BBEH tasks as a training set for a meta-learner that predicts optimal batch size from task features, then test on held-out BBEH tasks (the benchmark contains more than 8) or on entirely different prompt optimization benchmarks. Even a modest correlation between predicted and actual optimal batch size would demonstrate that configuration search can be partially automated.

Scaling training set size to study meta-overfitting thresholds. The Boardgame QA failure—where all optimized agents underperform the unoptimized baseline (Table 2)—raises a question the paper does not answer: at what training set size does meta-overfitting disappear? A direct follow-up would replicate the BBEH experiment on Boardgame QA with training set sizes of 15, 30, 50, 100, and 200 examples (the benchmark provides sufficient examples), measuring whether there exists a threshold beyond which optimization reliably improves over baseline. If a threshold exists and is consistent across tasks, it provides an actionable guideline for practitioners: "do not attempt generative optimization on tasks with fewer than N training examples." If no threshold exists—if some tasks exhibit meta-overfitting regardless of training set size—then the phenomenon is more fundamental and requires architectural solutions (regularization of the optimizer, early stopping based on validation trends, ensembling across optimization trajectories). This experiment is straightforward to run using the paper's released code and could be completed in weeks.

Cross-model robustness of configuration sensitivity. The paper's experiments all use Claude Sonnet-3.5-v2. A critical robustness check is whether the task-dependence of optimal configurations is model-specific or general. A follow-up would replicate the BBEH batch size sweep (Section 6) using two or three additional LLM backends: a stronger model (GPT-4 or Claude 4), a weaker model (Claude Haiku or GPT-3.5), and an open-weight model (Llama-3). The question is twofold: (1) Do the optimal batch sizes change? If every model prefers batch size 3 for Geometric Shapes, the finding is robust. If Claude prefers batch size 3 but GPT-4 prefers batch size 5, then optimal configuration depends on both task and model—a substantially harder design problem. (2) Does the magnitude of configuration sensitivity change? If stronger models show smaller gaps between best and worst batch sizes (i.e., they are more robust to misconfiguration), then the setup burden is a transient problem that will diminish as models improve. If sensitivity is constant or increases with model capability, it is a fundamental property of LLM-based optimization that must be addressed architecturally.

Combining generative optimization with traditional ML hyperparameter optimization for learning loop configuration. The paper identifies a meta-problem—configuring the learning loop itself requires optimization—but does not address how to solve it. A natural follow-up would treat the learning loop's design decisions as hyperparameters of the optimization process and apply traditional hyperparameter optimization (Bayesian optimization, multi-armed bandits, population-based training) to select them. For a new task: run a small number of optimization steps (5-10) under multiple configurations (2 starting artifacts × 2 credit horizons × 3 batch sizes = 12 configurations), evaluate each on a validation set, and use the results to select the configuration for the full optimization budget. The paper's own data provides a benchmark for this approach: can a configuration selector trained on 7 BBEH tasks and tested on the 8th predict the optimal batch size, using only early-validation performance as features? If even simple heuristics (e.g., "choose the batch size that yields the highest validation accuracy after 5 steps") work well, the practical setup burden is substantially reduced—the configuration search is amortized into the optimization budget rather than being a separate pre-processing step. The paper's reported validation curves (Figure 7) make this experiment immediately feasible: the curves show that batch size ordering is often visible within the first 5-7 optimization steps (batch size 5 leads early in Geometric Shapes, batch size 1 is noisy but eventually catches up in Dyck Languages).

Stress-testing the OPTO formalism on diverse learning templates. The paper introduces the workflow graph / learning graph distinction and three template types (interactive, batch, episodic) but only experiments with two (batch in BBEH, episodic in Atari). A follow-up could develop more complex templates that combine temporal and cross-example structure: for example, an episodic-batch template where the optimizer sees multiple complete episodes concatenated (e.g., 3 full Breakout episodes of 300 steps each, concatenated into one 900-step learning graph). This would test whether the paper's credit horizon and experience batching factors interact in templates that blend both dimensions—a scenario directly relevant to multi-task RL or meta-learning settings. A negative result (the combined template performs no better than the best single-factor template) would suggest the factors are largely independent; a positive result (the combined template enables learning that neither factor alone supports) would demonstrate that the real design space is richer than the paper's tripartite decomposition captures.

Practical Applications and Downstream Use Cases

Agent pipeline design teams adopting configuration sweeps as standard practice. The paper's most actionable finding for practitioners is that starting artifact, credit horizon, and experience batching must be treated as tunable hyperparameters rather than fixed defaults. For a team building an automated ML pipeline generator (analogous to the MLAgentBench case study), the paper provides a concrete protocol: test both one-function and many-function initializations (the difference was 11.1 Kaggle percentile points on Spaceship Titanic best-case, Table 1), vary the amount of execution trace included in optimizer context (short vs. long horizons produced opposite orderings in 4 of 8 Atari games, Figure 5), and experiment with batch sizes of 1, 3, and 5 (optimal batch size varied across all 8 BBEH tasks, with gaps as large as 0.189 accuracy points between best and worst, Table 2). The cost of this sweep—roughly 2-3× the optimization budget of a single configuration—is modest relative to the performance differences observed, and the paper's experimental appendices provide templates for feedback design, evaluation splits, and iteration budgets that can be adapted directly. This is not a glamorous application, but it is the most immediately impactful: teams that treat configuration as a first-class design step rather than an afterthought will achieve substantially better optimization outcomes.

Prompt optimization services with task-adaptive batching. The BBEH results (Table 2) have direct implications for any service or library that automatically optimizes prompts—a growing category that includes DSPy, TextGrad, and various internal tools at AI companies. Currently, these tools typically use fixed batch sizes (often 1 example per update, or a fixed minibatch of 4-8) regardless of task characteristics. The paper's finding that the optimal batch size varies from 1 (Movie Recommendation: 0.889, Disambiguation QA: 0.537) to 3 (Geometric Shapes: 0.389, Boolean Expressions: 0.238) to 5 (Causal Understanding: 0.531) across tasks—and that using the wrong batch size can degrade performance by 0.05-0.20 accuracy points—implies that task-adaptive batching is a low-hanging fruit for improving these services. A practical implementation would run a short pilot phase (5-7 optimization steps) at batch sizes 1, 3, and 5, select the best-performing based on validation, and proceed with the full optimization budget. The pilot cost (15-21 steps) is modest relative to the 15-step full budget reported in the paper, and the potential gains—correctly identifying batch size 1 for Movie Recommendation vs. batch size 3—are large.

Game-playing agent development with credit horizon as a first-class design parameter. The Atari case study (Section 5) provides practical guidance for engineers building game-playing agents or any multi-step interactive system: the credit horizon should be chosen based on the alignment between short-term and long-term rewards, not based on convention or convenience. When immediate rewards accurately reflect progress toward the final goal (Freeway, Enduro, Q*bert, Seaquest), short credit horizons suffice—they provide more frequent updates and faster iteration. When success requires coordinating actions over time with delayed consequences (Space Invaders, Pong, Breakout, Asterix), longer credit horizons are necessary despite the computational cost. The paper's per-game rollout lengths (Table B2: 400 steps for Pong, 300 for Breakout, 25 for Space Invaders) provide concrete starting points, and the staged feedback templates (Appendix B.3) offer a template-based approach to injecting domain knowledge without per-step human intervention. The deep RL comparison (Figure 6, Table B5b) also provides a pragmatic benchmark: the LLM-based approach achieved competitive scores with 8.3 minutes median wall-clock time vs. 219-292 minutes for DQN/PPO, suggesting that generative optimization is particularly attractive for rapid prototyping and iteration, even if absolute performance ceilings remain below well-tuned RL.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Gap: Generative Optimization Works in Labs, Not in Production

Why This Problem Matters, Practically and Conceptually

Where Prior Work Falls Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The OPTO Formalism: A Unified Language for Learning Loops

Design of the Three Case Studies (Shared Methodology)

The Starting Artifact Problem: MLAgentBench Case Study

The Credit Horizon Problem: Atari Case Study

The Experience Batching Problem: BigBench Extra Hard Case Study

Cross-Cutting Observations: Task-Dependence as the Central Finding

4. Key Insights and Innovations

Innovation 1: The Learning Loop as a Design Space — Not an Algorithm

Innovation 2: Task-Dependence as the Central Empirical Finding — And Its Implication That No Simple Defaults Exist

Innovation 3: Meta-Overfitting as a Distinct Failure Mode in Generative Optimization

Innovation 4: The AI-as-Optimizer vs. Human-as-Optimizer Correspondence as a Lens for Systematic Research

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Starting Artifact (MLAgentBench Case Study)

Credit Horizon (Atari Case Study)

Experience Batching (BigBench Extra Hard Case Study)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

The Difficulty Estimation Cost for Learning Loop Configuration

No Evidence of Interaction Effects Between the Three Design Factors

Single LLM Backend Limits Generality of Findings

Interaction Between Batch Size, Epoch Count, and Total Updates in BBEH Experiments

The Boardgame QA Meta-Overfitting Failure Mode Is Unexplored

Atari Deep RL Baseline Comparison Is Not Controlled for Compute or Performance

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation