optimize_anything: A Universal API for Optimizing any Text Parameter

ArXiv: 2605.19633

🎯 Pitch

A single LLM-based optimizer discovers agent architectures that triple Gemini's ARC-AGI score to 89.5% and finds scheduling policies that slash cloud costs by 40%, matching or beating specialized tools across code, prompts, and images—by simply treating everything as a text artifact refined through diagnostic feedback.

1. Executive Summary

This paper introduces optimize_anything, a declarative API that unifies LLM-based optimization of arbitrary text artifacts — code, prompts, agent architectures, images, and scheduling policies — under a single interface matching or surpassing domain-specific tools across six fundamentally different domains. The system supports three optimization modes (single-task search, multi-task search with cross-problem transfer, and generalization to unseen inputs) and elevates Side Information — diagnostic feedback such as stack traces, profiler data, and rendered visualizations — to a first-class API contract that accelerates convergence by 4–6× compared to score-only feedback. Across its evaluation suite, optimize_anything discovers agent architectures that nearly triple Gemini Flash’s ARC-AGI accuracy from 32.5% to 89.5%, finds scheduling algorithms cutting cloud costs by 40%, generates CUDA kernels where 87% match or beat PyTorch baselines, and outperforms AlphaEvolve’s reported circle packing solution (n=26), establishing that a single LLM-based text optimization system can serve as a general-purpose problem-solving paradigm — subsuming tasks traditionally requiring domain-specific algorithms — only when the artifact is representable as text and the evaluator can surface actionable diagnostics.

2. Context and Motivation

The Core Problem: Every Domain Has Its Own Optimizer

The fundamental problem this paper addresses is the fragmentation of AI-driven optimization. In 2025–2026, we have powerful systems that can optimize code functions (FunSearch, AlphaEvolve), another set that can optimize LLM prompts (GEPA, MIPROv2, TextGrad), and yet others that can search over agent architectures (ADAS, AFlow). Each of these tools was built for exactly one artifact type, with bespoke interfaces, and no system has demonstrated effectiveness across fundamentally different domains simultaneously.

This matters because the underlying pattern is always the same: you have a candidate represented as text, you evaluate it against a scoring function, and you use the evaluation results to propose an improvement. Whether the text is a CUDA kernel, a cloud scheduling policy, an SVG image, a 3D model script, or a system prompt, the optimization loop is structurally identical — the artifact is serialized, evaluated, and refined based on feedback. Yet prior to this work, a practitioner who wanted to optimize a prompt would need one library, and a practitioner who wanted to optimize a scheduling algorithm would need an entirely different library, with different APIs, different configuration paradigms, and different assumptions about what types of feedback are available.

The paper captures this insight directly:

"We observe that a wide range of problems can be formulated as optimizing a text artifact. Whether the artifact is a CUDA kernel, a cloud scheduling policy, an agent architecture, Scalable Vector Graphics (SVGs), or a system prompt, the structure is the same: serialize the artifact as a string, evaluate it, and let an LLM propose improvements based on diagnostic feedback." (Section 1)

This fragmentation is not merely an inconvenience — it obscures a deeper unity and prevents cross-domain insights. If prompt optimization benefits from per-aspect sub-scores (as GEPA showed), does that insight transfer to code optimization? If multi-task search accelerates CUDA kernel generation, could it help with scheduling policies? Prior frameworks made these questions impossible to ask because each operated in its own silo.

Why This Matters: Beyond Convenience to Capability

The paper makes a subtle but consequential argument: unification is not just about developer ergonomics — it enables capabilities that no single-domain system can achieve. Specifically:

Multi-task search across problems is nonexistent in prior frameworks. AlphaEvolve, OpenEvolve, and ShinkaEvolve all operate in single-task mode: optimize one artifact for one problem at a time. If you have 31 CUDA kernels to optimize, you run 31 independent optimization jobs. This means that insights discovered while optimizing Kernel A — say, that float4 vectorization dramatically improves memory throughput — are thrown away when you start optimizing Kernel B. The optimizer must rediscover the same pattern independently for each problem. Section 2 explicitly states: "No prior system supports multi-task search, where solving a batch of related problems together enables cross-transfer of discovered optimization patterns." This is a genuine capability gap, not just a convenience gap.

Generalization beyond prompts is unexplored. GEPA and MIPROv2 can optimize a prompt to generalize across unseen inputs, but they cannot optimize an agent architecture to generalize across unseen tasks, or a scheduling policy to generalize across unseen infrastructure scenarios. The generalization pattern — train on a set of examples, validate on held-out examples, produce a single artifact that performs well on new inputs — is a core machine learning concept that should apply to any text artifact, not just prompts. But no prior system generalized it.

Diagnostic feedback is ad-hoc and framework-specific. AlphaEvolve feeds execution results back to the LLM. GEPA uses per-example scores. TextGrad generates natural-language "gradients." Each framework implements feedback differently, and none treats it as a uniform API concept. This means that surfacing a stack trace in one framework requires different code than surfacing profiler data in another, even though both are just diagnostic text that an LLM can reason about. The paper argues that this feedback — which it calls Side Information (SI) — should be a first-class evaluator contract that works identically regardless of domain.

Where Prior Approaches Fall Short

The paper identifies concrete limitations across the existing landscape. Rather than simply listing prior systems, it characterizes what each cannot do and why those limitations matter.

Code Evolution Systems: Powerful but Narrow

AlphaEvolve [18] pioneered the LLM-evolution paradigm, using Gemini models with island-based MAP-Elites to discover algorithms for Google's infrastructure, including a 56-year-old matrix multiplication bound and data center scheduling heuristics. OpenEvolve [24] provides an open-source reimplementation. ShinkaEvolve [13] adds novelty-based rejection sampling and adaptive LLM ensemble selection. FunSearch [22] applies evolutionary LLM search to mathematical discovery, discovering new constructions for the cap set problem.

The paper acknowledges these as important predecessors but identifies three specific gaps (Section 2, Table 2):

1. Artifact type is restricted to code. AlphaEvolve, OpenEvolve, and ShinkaEvolve operate on code artifacts exclusively. They cannot optimize prompts, agent architectures, or images. FunSearch specifically evolves Python functions. This is not an implementation detail — the frameworks are architecturally coupled to code: they insert EVOLVE-BLOCK markers, configure island topologies, and assume a code execution evaluator. Extending them to non-code artifacts would require substantial re-engineering.

2. Only single-task mode is supported. Each optimization job targets exactly one problem. There is no mechanism for sharing insights across related problems. The paper quantifies the cost of this limitation in Section 5.8: on CUDA kernel generation, single-task mode plateaus early while multi-task mode continues improving, because patterns like warp shuffle reductions must be independently rediscovered for each kernel.

3. Interfaces are framework-specific rather than declarative. Using AlphaEvolve requires understanding island topologies, configuring mutation prompts, and marking code blocks with evolve directives. ShinkaEvolve adds prompt samplers and novelty rejection schedules. This creates a steep learning curve: a domain expert who understands their problem (e.g., circle packing geometry) must also learn the optimization framework's internal abstractions before they can use it.

Prompt Optimization Systems: Generalization but Limited Artifact Scope

GEPA [3] achieves state-of-the-art prompt optimization using reflective mutation with Pareto-based search, outperforming both MIPROv2 [19] and GRPO [23] on prompt optimization benchmarks. MIPROv2 optimizes instructions and few-shot demonstrations for multi-stage LM programs. TextGrad [28] uses LLM-generated "gradients" for iterative text improvement. Other systems include OPRO [27], APE [30], ProTeGi [21], and PromptBreeder [8].

These systems are effective within their domain but share a critical limitation: the artifact type is hardcoded to prompts. GEPA's search algorithm knows it is optimizing a prompt — it structures reflection around prompt-specific concepts, assumes the evaluator is an LM call with a system prompt, and cannot represent a CUDA kernel or a scheduling algorithm as the optimization target. The paper's contribution is to extract GEPA's Pareto-based search from its prompt-specific scaffolding and generalize it to arbitrary text artifacts (Section 4, "several concrete algorithmic modifications were necessary to generalize from prompts to arbitrary text artifacts").

A subtler limitation: existing prompt optimizers assume the evaluation feedback is an LM's output on a dataset of inputs. They cannot consume the diagnostic feedback types that matter in other domains — compiler errors, profiler traces, rendered visualizations, or execution time measurements. This is because their feedback channels are baked into framework internals rather than exposed as a user-definable contract.

Agent Architecture Search Systems: Architecture-Specific Interfaces

ADAS [11] and AFlow [29] search over agent architectures, but they operate with architecture-specific interfaces. ADAS searches over a predefined space of agent components; AFlow automates workflow generation using a graph-based representation. Neither can be repurposed to optimize prompts or scheduling algorithms, because their search spaces and evaluator assumptions are tightly coupled to agent architectures. The paper notes that optimize_anything's generalization mode "subsumes these as special cases: the artifact is the agent code, the evaluator runs it on tasks, and the system evolves both architecture and prompts jointly" (Section 2).

Common Limitation: No Unified Diagnostic Feedback Contract

Across all prior systems, diagnostic feedback is handled inconsistently. AlphaEvolve feeds execution traces into the proposer LLM through framework-specific code paths. GEPA uses per-example scores and sub-scores but bakes the feedback structure into its prompt optimization logic. TextGrad generates "gradient" text through LLM calls, which works for natural language tasks but cannot incorporate compiler errors or profiler data unless the user manually massages them into the gradient format.

The paper's diagnosis is that these ad-hoc mechanisms prevent a key capability: domain experts should be able to provide the diagnostics they already understand (compiler errors, profiler summaries, rendered images) and have the optimizer use them directly, without learning the framework's internal feedback format. Section 8 makes this explicit:

"optimize_anything trades optimization expertise for domain expertise. The user, most often a domain expert, need not configure backends, tune algorithmic hyperparameters, or engineer prompting strategies, only surface the diagnostics they already understand."

How This Paper Positions Itself

optimize_anything does not claim to invent new optimization algorithms. The default backend extends GEPA's Pareto-based search, and the system is explicitly backend-agnostic: "as new optimization strategies emerge, they plug in without changing user code" (Section 9). The paper's contribution is the unifying abstraction that makes a single algorithm work across domains.

This is not a trivial refactoring. Section 4 details concrete algorithmic modifications required to generalize from prompt-only to arbitrary text artifacts: new frontier types for single-task and multi-task search (since GEPA's Pareto-frontier selection assumed multiple data points, but single-task admits only one), a refiner step that catches malformed code blocks and import errors before evaluation (unnecessary for prompts, essential for code), content-addressed evaluation caching for expensive rollouts, SI as a typed primitive, and an adapter layer between backends and the unified interface.

The paper draws a direct parallel to DSPy's [12] philosophy of "programming, not prompting" — the concept that declarative abstractions, rather than hand-crafted templates, are the right interface for LLM-based systems. Just as DSPy abstracts away the details of prompt construction so users can focus on specifying what they want, optimize_anything abstracts away optimization framework configuration so users can focus on specifying their artifact, evaluator, and domain knowledge.

The paper positions its three optimization modes as filling a gap in the landscape (Table 2 makes this explicit with a feature matrix). No prior system supports all three modes. No prior system treats SI as a first-class API contract. No prior system supports image feedback (enabling VLM proposers to see rendered visualizations). No prior system works across code, prompts, and agent artifacts without changing interfaces.

This positioning is important because it frames the contribution not as "here is a better optimization algorithm" but as "here is a generalization that reveals the underlying unity of text-based optimization and enables capabilities (multi-task transfer, cross-domain diagnostic feedback, generalizable agent architectures) that were impossible in single-domain frameworks." The paper's experiments are designed to demonstrate this: they don't just show that optimize_anything works on each domain, but that it works with the same API call, requiring only the seed artifact, evaluator, and optional dataset — no domain-specific configuration, no mutation templates, no island topologies.

3. Technical Approach

3.1 Reader Orientation

The paper introduces optimize_anything, a unified declarative API that treats LLM-based optimization of any text artifact as a single, general-purpose problem-solving system. The system solves the problem of fragmented optimization — where different domains (code, prompts, agents, images) each require separate frameworks with bespoke interfaces — by abstracting all text optimization into one loop: a candidate text artifact is evaluated by a user-provided scoring function, diagnostic feedback (Side Information) flows back to an LLM proposer, and the LLM generates an improved artifact. The "shape" of the solution is a backend-agnostic wrapper around evolutionary search that works identically whether you're optimizing a CUDA kernel, a system prompt, a cloud scheduling policy, an ARC-AGI agent architecture, or an SVG image, with the same API call and no domain-specific configuration.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a closed optimization loop (Figure 1):

1. Text Artifact ($x$) — the thing being optimized, serialized as a string. It could be Python code, a natural-language prompt, an agent system definition, SVG markup, or a CAD model script. Everything flows from and to this artifact.

2. Evaluator ($f(x)$) — a user-provided function that takes the artifact string and returns a score (higher is better) plus an optional Side Information (SI) dictionary containing diagnostic feedback. The evaluator encapsulates all domain-specific execution: running code, calling an LLM, rendering an image, or simulating a scheduling policy.

3. Side Information (SI) — the diagnostic output from the evaluator, flowing back to the proposer. This can include text (compiler errors, execution traces, natural-language critiques), structured data (per-test-case results, multi-objective sub-scores), or images (rendered SVGs, 3D model screenshots) for Vision-Language Model (VLM) proposers.

4. LLM Proposer — a large language model (GPT-5, Gemini 3 Flash, Claude Opus 4.6, depending on the domain) that receives the current artifact, its score, and its SI, then reasons about failures and generates an improved artifact. The proposer operates through a structured reflection step that diagnoses what went wrong and proposes targeted fixes.

5. Pareto-Based Search Engine (GEPA backend) — the optimization algorithm that manages a population of candidate artifacts, tracks their performance across multiple objectives (per-example scores, per-metric sub-scores), maintains a Pareto frontier of non-dominated candidates, and selects which candidates to mutate. This component includes content-addressed evaluation caching and a refiner that fixes malformed outputs before evaluation.

Information flows cyclically: the current artifact enters the evaluator → evaluator returns (score, SI) → Pareto engine records per-objective scores and selects candidates for mutation → selected candidate plus a minibatch of examples plus SI enter the proposer → proposer generates an improved artifact → artifact is optionally refined (syntax/import fixes) → artifact enters evaluator → cycle repeats. The loop terminates when an evaluation budget is exhausted, returning the best artifact from the Pareto frontier.

3.3 Roadmap for the Deep Dive

• First, the formal problem formulation (Section 4.1) that mathematically defines the three optimization modes — single-task, multi-task, and generalization — under one equation, because the mode determines how the Pareto frontier is constructed and what "optimal" means.
• Second, the Side Information (SI) mechanism (Section 4.2) as the core innovation that makes the system general, because SI is the text-optimization analogue of a gradient and understanding its contract is prerequisite to understanding why the system works across domains.
• Third, the Pareto-based search algorithm (Section 4.3) including candidate selection, minibatch reflection, and mutation, because this is the engine that drives optimization and its design choices — frontier-based selection rather than aggregate scoring, minibatch reflection rather than full-dataset feedback — directly enable multi-task transfer and prevent premature convergence.
• Fourth, the three optimization modes as instantiations of the formalism, because the same algorithm operates differently depending on whether the dataset is empty (single-task), contains related problems (multi-task), or is split into train/validation (generalization).
• Fifth, the refiner and caching mechanisms that make the system practical for code and agent artifacts, because without these, malformed LLM outputs would cause evaluation failures that waste budget.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and interface paper whose core idea is that a single declarative API with a backend-agnostic evolutionary search engine can match or outperform domain-specific optimization tools across fundamentally different text artifacts, provided the evaluator surfaces actionable diagnostic feedback as a first-class contract.

The Core API Contract

The optimize_anything function signature establishes the minimal information required to optimize any text artifact:

result = optimize_anything(
    seed_candidate="<your artifact>",
    evaluator=evaluate,
)

The key design principle is declarative specification: the user declares what they want to optimize (the artifact and how to score it), not how to optimize it (no mutation prompts, no search hyperparameters, no island topologies). The paper explicitly draws this parallel to DSPy's "programming, not prompting" philosophy, where the abstraction boundary hides optimization mechanics behind a declarative interface.

Required inputs:

• seed_candidate: a starting text artifact (string). This is the initial guess that the optimizer will improve. It can be as simple as a naive implementation — the 10-line ARC-AGI agent that makes a single LLM call, the Dijkstra routing algorithm for CloudCast, or a basic greedy circle packing heuristic.
• evaluator: a function f(candidate: str) -> tuple[float, dict] that runs the artifact and returns a numerical score (higher is better) and an optional dictionary of Side Information.

Optional inputs that activate different modes:

• dataset: a list of examples. If provided, the optimizer tracks per-example scores and optimizes for average performance across them. If dataset contains different problems (e.g., different CUDA kernels to optimize), this activates multi-task mode; if it contains training examples for a single task, this activates generalization mode. If absent, the system runs in single-task mode (Section 3.2).
• valset: a held-out set of examples used only for measuring generalization performance, never for guiding search. Required for generalization mode; absent for single-task and multi-task modes.
• objective: a natural-language description of the optimization goal, used when no seed_candidate is provided (seedless mode).
• background: domain knowledge text that the proposer receives to understand the problem context.
• config: engine settings that override defaults, though the system works without any configuration in most cases.

What the system does NOT require (and why this matters): Unlike prior frameworks, the user does NOT provide mutation prompts (AlphaEvolve requires these), task-specific templates (GEPA bakes prompt structure into its search), island configurations (AlphaEvolve's MAP-Elites requires island topologies), or EVOLVE-BLOCK markers (used by AlphaEvolve and OpenEvolve to designate mutable code regions). The paper states: "optimize_anything doesn't require mutation prompts, task-specific templates, island configurations, or EVOLVE-BLOCK markers (all common in prior frameworks). The user declares the what (artifact, evaluator, domain knowledge), and optimize_anything, through its optimization backends, handles the execution" (Section 3.1).

This is not merely cosmetic. The elimination of framework-specific configuration is what enables the same API call to work across domains — a domain expert optimizing circle packing does not need to learn what an island topology is, and a prompt engineer does not need to insert evolve-block markers into their prompt text. The paper argues this trades "optimization expertise for domain expertise" (Section 8): the user surfaces the diagnostics they already understand (compiler errors, profiler traces) rather than learning optimization framework internals.

Seedless mode: When even writing a bad seed requires domain expertise (e.g., 3D modeling with build123d), the user provides only an objective string and the LLM bootstraps the first candidate from scratch. This is demonstrated in Appendix C on a 3D unicorn generation task, where the system starts from no code and iteratively refines geometry based on VLM feedback on rendered images. The mechanism is simple: on the first iteration, the proposer receives the objective, background knowledge, and no current artifact, generating the initial candidate entirely from the LLM's knowledge.

Formal Problem Formulation

The paper formalizes text optimization mathematically to unify the three modes under a single framework (Section 4.1).

Let the space of text artifacts be denoted $\mathcal{X}$ (all possible strings). An evaluator is a function:

$f: \mathcal{X} \times \mathcal{E} \cup \{\bot\} \rightarrow \mathbb{R} \times \mathcal{I}$

where:

• $\mathcal{X}$ is the space of text artifacts (strings),
• $\mathcal{E}$ is the space of examples (individual problem instances),
• $\bot$ is a null value representing "no example" (used in single-task mode),
• $\mathbb{R}$ is the space of real-valued scores (higher is better),
• $\mathcal{I}$ is the space of Side Information dictionaries (arbitrary key-value diagnostic data).

The evaluator maps an artifact $x \in \mathcal{X}$ and an optional example $e \in \mathcal{E} \cup \{\bot\}$ to a pair $(s(x, e), \iota(x, e))$, where $s(x,e)$ is the scalar score and $\iota(x,e)$ is the SI dictionary.

What this computes: the evaluator is the domain-specific bridge between the text artifact and the optimizer. When called with $e = \bot$ (single-task mode), it scores the artifact directly — run the packing algorithm and measure the sum of radii, compile the CUDA kernel and benchmark it. When called with a specific example $e$ (multi-task or generalization mode), it scores the artifact on that particular problem instance — run the agent on ARC-AGI puzzle #37, evaluate the prompt on AIME problem #12.

Why this form: the evaluator signature is the key design decision that enables the API to span domains. By requiring the evaluator to accept an optional example parameter, the same function signature works whether the optimization target is a single problem (example is always $\bot$), multiple related problems (different examples passed on each call), or a training set for generalization (examples from dataset during search, examples from valset during evaluation). The SI dictionary uses Python's native dict type with no schema constraints, meaning evaluators can return any diagnostic data — compiler errors as strings, profiler summaries as nested dicts, rendered images as base64-encoded PNGs — without the optimization framework needing to understand their structure.

The three modes correspond to:

Single-task search: $\mathcal{E} = \emptyset$. The objective is to maximize $s(x)$ directly. The "dataset" is conceptually a singleton — there is exactly one problem to solve, so every evaluation call invokes $f(x, \bot)$. Example: in circle packing, the artifact is the packing algorithm code, and each evaluation runs that code and returns the packing score plus geometric diagnostics. There is no notion of "different examples" because there is only one problem.

Multi-task search: Given a dataset $\mathcal{D} = \{e_1, \ldots, e_n\}$ of related problems, the objective is to find an artifact $x \in \mathcal{X}$ that maximizes:

$\frac{1}{n} \sum_{i=1}^{n} s(x, e_i)$

where $n$ is the number of tasks, $e_i$ is the i-th task (e.g., a specific CUDA kernel to optimize), and $s(x, e_i)$ is the score of artifact $x$ on task $e_i$.

What this computes: the average performance of artifact $x$ across all $n$ related problems. This is a scalar that the optimizer tries to maximize, but critically, the optimizer tracks per-task scores individually for Pareto frontier construction — a candidate that excels at one kernel but is mediocre on others may be preserved because it carries transferable insights.

Why this form: the simple average is the right aggregate for multi-task optimization because all tasks are equally important (there is no notion of task priority in the KernelBench setup). But the real power comes from the Pareto frontier tracking per-task scores separately, which the average alone would obscure. A candidate with scores [0.9, 0.9, 0.1, 0.1] has the same average as one with [0.5, 0.5, 0.5, 0.5], but the first candidate is preserved on the frontier because it dominates on tasks 1 and 2, and its strategies for those tasks can transfer to improving tasks 3 and 4 in subsequent mutations.

Generalization: Given a training set $\mathcal{D}_{\text{train}}$ and a validation set $\mathcal{D}_{\text{val}} = \{e^{\text{val}}_1, \ldots, e^{\text{val}}_k\}$, the objective is to find an artifact $x \in \mathcal{X}$ that maximizes:

$\frac{1}{k} \sum_{j=1}^{k} s(x, e^{\text{val}}_j)$

What this computes: the generalization performance — accuracy on unseen examples. Search uses feedback from $\mathcal{D}_{\text{train}}$ (the proposer sees scores and SI from training examples), but the final artifact selection and any reported metrics use $\mathcal{D}_{\text{val}}$, which the proposer never sees. This is the same train/validation split used in classical machine learning, applied to text artifacts.

Why this form: the separation of train and validation prevents over-optimization to the specific examples the proposer sees. Without this separation, the optimizer could discover artifacts that memorize training-example-specific patterns rather than learning generalizable strategies. The paper demonstrates this in prompt optimization (AIME), where the train set is AIME 2022-2024 and the validation/test set is AIME 2025 — problems the system has never seen during optimization.

Critical distinction between multi-task and generalization: In multi-task search, the output is $n$ specialized artifacts — one per task — that have benefited from shared optimization context. Patterns discovered while optimizing task $e_i$ are available as parents when proposing for task $e_j$, but each artifact independently selects its best candidate from the Pareto frontier at output time. Section 3.2 states: "each task independently selects its own best candidate from the frontier. This means multi-task search produces $N$ specialized artifacts (one per task) that have benefited from shared optimization context." In generalization mode, the output is one artifact that must perform well across all unseen examples — there is one global optimum, not $n$ per-task optima. Section 3.2: "multi-task search yields $N$ specialized artifacts while generalization yields one globally generalized artifact."

Side Information (SI): The Text-Optimization Analogue of a Gradient

The paper's central technical insight is that diagnostic feedback — what it calls Side Information (SI) — should be a first-class evaluator contract rather than an ad-hoc, framework-specific mechanism. Section 4.2 makes the analogy explicit:

"SI is the text-optimization analogue of the gradient. Where gradients tell a numerical optimizer which direction to move, SI can tell the LLM proposer why a candidate failed and how to fix it."

The SI contract. The evaluator returns a side_info dictionary containing any diagnostic it can produce. The system imposes no schema; the dictionary can contain any Python values that can be serialized into the proposer's prompt:

def evaluate(candidate: str) -> tuple[float, dict]:
    result = execute_code(candidate)
    return result.score, {
        "Error": result.stderr,          # Text: compiler errors
        "Output": result.stdout,         # Text: program output
        "Runtime": f"{result.time_ms:.1f}ms",  # Text: timing
        "Subscores": {                   # Structured: per-metric
            "correctness": 0.95,
            "speedup": 2.3,
        },
        "Visualization": oa.Image(...),  # Image: rendered output
    }

Supported SI types (Section 4.2):

• Text: compiler errors (NVCC for CUDA, go test output for Bleve), runtime exceptions with stack traces, profiler summaries showing memory bandwidth utilization, natural-language critiques from VLM evaluation.
• Structured data: per-test-case pass/fail results, sub-scores for multiple objectives (correctness, speed, memory usage), execution traces showing which agent component failed on which puzzle.
• Images: rendered SVGs showing the current packing arrangement, 3D model screenshots from multiple camera angles, chart visualizations of performance trends. These are passed to VLM proposers (Gemini Flash, Claude Opus) that can "see" what they are improving.

The reflection step: how SI is consumed. During optimization, when a candidate is selected for mutation, the system executes a reflection step rather than a blind mutation (Section 4.3). The proposer LLM receives:

1. The current artifact text.
2. The artifact's score(s) on a minibatch of examples.
3. The SI dictionary for each example in the minibatch.
4. A structured prompt asking it to diagnose failures and propose an improved artifact.

The proposer reasons over the SI to identify which failure mode occurred and how to fix it. In circle packing, SI reveals that radii collapsed to near-zero (diagnosis: the greedy algorithm over-constrained itself; fix: switch to LP-based radius optimization). In CUDA kernel generation, SI reveals an NVCC error at line 47 (diagnosis: wrong data type for vectorized load; fix: use float4* instead of float*). In ARC-AGI, SI reveals that the agent's code generation stage produced correct Python but the output grid had wrong dimensions (diagnosis: the code didn't handle the specific puzzle's grid size; fix: add dimension inference logic).

Why SI is opt-in but zero-friction. The paper emphasizes that evaluators returning only a score (with an empty SI dict) still work — the system degrades gracefully to score-only feedback. But the default capture_stdio=True option automatically captures any print() output as SI text, meaning domain experts who already print diagnostics during development get SI for free. The paper states: "SI is opt-in but zero-friction: evaluators that return only a score work fine, and existing print() statements can be captured automatically via capture_stdio=True" (Section 4.2).

The design choice: why not hardcode SI types? Prior frameworks embed feedback into framework internals — AlphaEvolve has specific code paths for execution traces, GEPA has per-example score tracking baked into prompt optimization logic. optimize_anything's dictionary-based contract means the framework never interprets SI; it simply serializes it into the proposer's prompt. This is why the system can surface a compiler error, a VLM's aesthetic critique, and a profiler trace through the same mechanism — the proposer LLM, not the framework, interprets the diagnostic content. The paper argues this is the right abstraction boundary because LLMs are already capable of understanding diverse diagnostic formats (compiler errors, profiler output, natural language critiques), and enforcing a schema would restrict what diagnostics can be surfaced without adding value.

Pareto-Based Search Algorithm

The default optimization backend extends GEPA's [3] Pareto-based search, which was originally studied only for prompt optimization. Several concrete modifications were necessary to generalize to arbitrary text artifacts (Section 4):

1. New frontier types for single-task and multi-task search with distinct selection semantics, since GEPA's original Pareto-frontier selection assumed evaluation across multiple data points (prompts evaluated on a dataset of inputs), but single-task search admits only one evaluation per candidate.

2. A refiner step that catches common LLM generation artifacts — malformed code blocks, import errors, syntax issues — before evaluation. This is essential for code and agent artifacts where minor formatting errors cause complete evaluation failure, wasting budget on candidates that were conceptually correct but syntactically broken.

3. Content-addressed evaluation caching to avoid redundant expensive rollouts. If the same artifact is generated twice (e.g., through different mutation paths), the cached score and SI are reused rather than re-running the evaluator.

4. SI as a first-class typed primitive enabling domain-portable proposer logic and multimodal feedback, as described in the previous subsection.

5. An adapter layer between various optimization backends and the unified interface, so that future optimization algorithms can plug in without changing user code.

The core algorithm (Algorithm 1 in Appendix D) proceeds as follows:

Initialization: The candidate pool $\mathcal{P}$ is initialized with the seed artifact $\Phi_0$. The seed is evaluated on all examples in the dataset (or on the single problem for single-task mode), recording per-example scores.

Main loop (repeats until evaluation budget $B$ is exhausted):

1. ParetoSelect: Select a candidate $\Phi_k$ from the pool for mutation. The selection mechanism is described below.

2. Minibatch sampling: Draw a minibatch $\mathcal{M}$ of size $b$ (typically 2–3) examples from the dataset.

3. Execute and reflect: Run the selected candidate on the minibatch, collect scores and SI, and present them to the proposer LLM in a structured reflection prompt. The proposer diagnoses failures and produces an updated artifact $\Phi'$.

4. Minibatch check: If $\Phi'$ improves performance on the minibatch (higher average score than $\Phi_k$), proceed to full evaluation. If not, discard $\Phi'$ and sample a new candidate.

5. Full evaluation: Evaluate $\Phi'$ on the full dataset, recording per-example scores.

6. Frontier update: Add $\Phi'$ to the candidate pool. Prune any candidates that are strictly dominated — i.e., candidates for which there exists another candidate in the pool that is equal or better on every objective and strictly better on at least one.

7. Output: When the budget is exhausted, return the artifact $\Phi^* \in \mathcal{P}$ that maximizes the average score across all examples.

Pareto-based candidate selection. The candidate selection mechanism is the key algorithmic choice that differentiates this approach from simple "pick the top scorer." Let $\mathcal{J}$ index the objectives used to form the Pareto scores — these could be per-example tasks (each ARC-AGI puzzle trackes separately), per-metric scores (correctness vs. speed for CUDA kernels), or both. Each candidate $\Phi$ in the pool induces a score $s_j(\Phi)$ for every objective $j \in \mathcal{J}$.

Let $\mathcal{P}$ denote the set of Pareto-nondominated candidates — candidates for which no other candidate in the pool achieves equal-or-better scores on all objectives and strictly better on at least one. For each objective $j \in \mathcal{J}$, let $\mathcal{B}[j]$ be the subset of $\mathcal{P}$ that achieve the best score on objective $j$. A candidate's frontier frequency is the number of objectives for which it is the best:

$\text{freq}(\Phi) = |\{j \in \mathcal{J} : \Phi \in \mathcal{B}[j]\}|$

Candidates are sampled for mutation with probability proportional to their frontier frequency:

$\mathbb{P}(\text{select } \Phi) \propto \text{freq}(\Phi)$

What this computes: a candidate's probability of being selected for the next mutation round is proportional to how many objectives it is the best at, not how high its average score is. A candidate that is the absolute best on 3 out of 10 tasks but terrible on the other 7 will be selected more often than a candidate that is above-average on all 10 tasks but never the best at any single one.

Why this form: this selection mechanism explicitly favors complementary diversity. The proposer sees candidates that excel at different subsets of tasks, and can combine their strategies — the memory coalescing pattern from the kernel that aces matrix multiplication with the warp reduction pattern from the kernel that aces LayerNorm. If selection were based on average score, the frontier would collapse to a single strategy family that does well on average but never explores specialized patterns that could transfer across problems. The paper states this mechanism "focuses exploration on broadly effective solutions" — candidates that are best on many objectives are likely to contain generally useful patterns.

Minibatch reflection (not full-dataset reflection). A subtle but important design choice: the reflection step shows the proposer only $b = 2$–$3$ examples at a time, not the full dataset. This enables focused, targeted improvements: the proposer sees "this artifact failed on examples #12 and #37 because of pattern X" rather than "this artifact has an average score of 0.67 across 200 examples."

The minibatch check — only proceeding to full evaluation if the new artifact improves on the minibatch — acts as a quick filter that saves evaluation budget. The paper's ablation on prompt optimization (Figure 9) shows that SI on a minibatch yields 6× faster convergence than score-only feedback on the full dataset, partly because the proposer can focus its reasoning on specific failure modes rather than trying to improve everything at once.

The refiner: a safety net for code artifacts. Before any generated artifact enters the evaluator, it passes through a refiner that catches common LLM generation artifacts (Section 4, item 2). This is unnecessary for prompt optimization (where malformed output just means a bad natural-language prompt, which the evaluator will score poorly), but essential for code artifacts where a single syntax error causes the entire evaluation to fail with score = 0.

The refiner checks for:

• Malformed code blocks (missing triple backticks, wrong language tag).
• Import errors (missing standard library imports that the LLM assumed were present).
• Syntax errors that Python's ast.parse can catch without execution.
• Common CUDA-specific issues (missing kernel launch configuration, wrong pointer types).

If the refiner can fix the artifact automatically, the fixed version enters evaluation. If not, the original (broken) version enters evaluation, receives score = 0, and the SI (the syntax error) flows back to the proposer for the next reflection step. The paper notes in Appendix F that "even broken code mutations (score=0.0) are recovered by the refiner and retained on the front, acting as a safety net that preserves exploration" — a candidate that failed due to a trivial syntax error might contain a valuable algorithmic insight that can be recovered when the refiner or the next proposer call fixes the syntax.

Content-addressed evaluation caching. When the evaluator is expensive (e.g., ARC-AGI with 100-puzzle evaluations costing $144), redundant evaluations waste budget. The system hashes each artifact string (after refiner normalization) and caches (score, SI) pairs. If the same artifact is generated through different mutation paths, the cached result is reused without re-running the evaluator. This is a simple but essential engineering optimization that makes the system practical for expensive evaluators.

Three Optimization Modes: How the Same Algorithm Instantiates Differently

The three modes — single-task, multi-task, and generalization — are not separate algorithms but different configurations of the same Pareto-based search loop. The mode is determined entirely by whether dataset and valset are provided (Section 3.2):

Single-task search (dataset=None, valset=None): The "dataset" is a singleton — there is exactly one problem. The Pareto frontier is constructed over per-metric scores extracted from the SI dictionary. If the evaluator returns sub-scores for multiple objectives (e.g., correctness + speed for CUDA kernels, or sum_radii + min_separation for circle packing), each sub-score becomes a separate objective on the frontier. If the evaluator returns only a single score, the frontier degenerates to a single-objective best-so-far tracker — still functional but losing the diversity benefits of Pareto selection.

In circle packing (Section 5.6), the frontier tracks multiple metrics simultaneously: the raw sum_radii score, an exponential moving average (EMA) of recent scores to measure stability, and an improvement rate (how fast the score is increasing). A candidate with slightly lower sum_radii but higher improvement rate survives on the frontier because it might be on a trajectory to surpass the current best. This is why the system discovers the bilevel L-BFGS approach: the LP-based candidates dominate on sum_radii, but the CMA-ES candidates survive on improvement rate, and the proposer eventually combines them.

Multi-task search (dataset provided as a list of related problems, valset=None): Each element of dataset is an independent problem — a different CUDA kernel to optimize, a different SVG image goal, or a different mathematical optimization problem. The Pareto frontier is constructed over per-task scores: each problem is a separate objective, and a candidate that excels at Kernel A but is terrible at Kernel B is preserved because it dominates on the Kernel A dimension.

The cross-transfer mechanism (Section 5.8) works as follows: when the ParetoSelect step samples a candidate for mutation, it might select a candidate that is specialized for Kernel A (best on that dimension) and present it to the proposer alongside a minibatch from Kernel B. The proposer sees the specialized candidate, sees the minibatch failures on Kernel B, and generates a new candidate that combines Kernel A's optimization pattern with adaptations for Kernel B's specific requirements. This is qualitatively different from single-task mode, where the proposer only ever sees candidates from the same kernel's optimization history.

At output time, multi-task search produces $n$ specialized artifacts — one per task — each selected independently from the Pareto frontier. Section 3.2 states: "each task independently selects its own best candidate from the frontier. This means multi-task search produces $N$ specialized artifacts (one per task) that have benefited from shared optimization context, patterns discovered while optimizing task $e_i$ are available as parents when proposing for task $e_j$, but each artifact can specialize to its task."

The paper quantifies the benefit in Section 5.8: on 10 KernelBench problems, multi-task mode achieves 90% of kernels matching baseline vs. 60% for single-task at equivalent per-problem budget, with the gap widening at higher speedup thresholds. MT20 (20 problems) outperforms MT10 (10 problems), which outperforms single-task — the cross-transfer benefit scales with the number of related problems because more problems means more diverse patterns discovered and available for transfer.

Generalization (dataset and valset both provided): dataset serves as the training set; valset is held out. The Pareto frontier is constructed over per-example training scores (each training example is a separate objective). The proposer sees only training examples during reflection; valset is used only for final artifact selection and reporting.

The key distinction from multi-task: the output is one globally generalized artifact, not $n$ per-problem artifacts. The artifact must perform well on unseen examples. This generalizes classical supervised learning: the artifact is the "model" (but it's a text string — a prompt, an agent architecture, or a policy), the training set provides supervised signal, and validation measures generalization.

In ARC-AGI (Section 5.3), the dataset is a set of training puzzles with known solutions; valset is the standard ARC-AGI test set. The agent architecture evolved on training puzzles achieves 89.5% on unseen test puzzles, demonstrating that the system learned generalizable architectural patterns (code generation with verification, iterative debugging with fallback) rather than overfitting to training puzzle specifics.

The Proposer LLM: Reflection and Mutation

The proposer is the LLM that generates improved candidate artifacts from (current artifact + minibatch scores + SI). The paper uses different proposer models depending on the domain (Table 1): GPT-5/5.1 for CUDA kernels, circle packing, and AIME prompts; Gemini 3 Flash for ARC-AGI (where it serves as both proposer and the underlying agent model); Gemini 3 Pro for cloud scheduling; Claude Opus 4.6 for coding agent skills and 3D modeling.

The reflection prompt structure. The system constructs a structured prompt containing:

1. The original artifact text (the "current candidate").
2. A header explaining the optimization context: "You are improving a [circle packing algorithm / CUDA kernel / agent architecture]."
3. For each example in the minibatch: the example description, the artifact's score on that example, and the full SI dictionary (compiler errors, execution traces, rendered images, per-aspect sub-scores).
4. An instruction to diagnose what went wrong and propose a specific, targeted fix.
5. Background knowledge if provided (domain context about what the artifact should do).

The proposer's output is the new artifact string — the complete, updated version, not a diff or patch. The system expects the proposer to produce a self-contained artifact that can be directly evaluated.

Why reflection rather than blind mutation. The paper contrasts this with prior evolutionary approaches that use blind genetic operators (crossover, random mutation) on code strings. SI-driven reflection is closer to how a human engineer iterates: run the code, read the error, understand why it failed, and make a targeted fix. The paper's ablation (Section 5.9, Figure 9) quantifies the benefit: on prompt optimization for the Facility Support Analysis dataset, SI-driven reflection reaches a validation score of 0.80 in approximately 100 rollouts, while score-only feedback (where the proposer sees only aggregate scores without per-aspect breakdowns) requires approximately 600 rollouts — a 6× acceleration in convergence speed. The final test score is also higher with SI: 86.32 vs. 82.5 without.

The mechanism is that SI converts "score decreased from 0.7 to 0.6" (which tells the proposer something went wrong but not what) into "score decreased because the correctness sub-score dropped from 0.9 to 0.4 while efficiency remained at 0.8 — the recent change to the normalization step introduced a bug in edge-case handling" (which tells the proposer exactly where to look and what to fix).

Proposer sensitivity (Section 5.10, Table 8). The quality of proposed artifacts depends on the proposer LLM's capabilities. The paper compares GPT-5.1 against the cheaper GPT-5-nano:

• On AIME prompt optimization: GPT-5.1 improves from 46.67% to 60.0% (cost $6.44); GPT-5-nano improves to 50.0$3.71).
• On circle packing: GPT-5.1 achieves 2.636 (cost $6.00); GPT-5-nano achieves 2.512 (cost$0.50).

The nano model consistently underperforms the larger model on final artifact quality but still improves substantially over the seed. The cost reduction is dramatic (over 90% on circle packing), suggesting a practical tradeoff: use nano for rapid prototyping and exploration, then switch to the larger model for final convergence.

Cost breakdown (Section 5.10, Table 9). Total optimization costs range from $1 (numerical blackbox optimization) to$144.70 (ARC-AGI agent architecture search). The reflection cost — the LLM calls for the proposer — is consistently minimal compared to the evaluator cost. In ARC-AGI, reflection costs $0.70 while the evaluator (running the agent on 200 puzzles) costs$144. In AIME prompt optimization, reflection costs $2.17 while LLM evaluation calls cost$4.27. This reflects a key property of LLM-based optimization: it is highly sample-efficient, calling the expensive evaluator fewer times than traditional blackbox optimization methods, and spending most of its budget on evaluation rather than optimization overhead.

Why the Three Mechanisms Work Together: Trajectory Analysis

Section 6 and Appendix F provide an optimization trajectory analysis that reveals three mechanisms underlying the system's effectiveness. These are not separate components but emergent properties of the SI + Pareto frontier + reflection combination.

Mechanism 1: SI enables targeted algorithmic shifts. In circle packing, SI-driven reflection produces a characteristic pattern of directed algorithmic improvements:

• Current algorithm is greedy: radii collapse because greedy over-constrains itself. SI reveals "radii near zero, constraint violations high at step 15." Proposer switches to linear programming (LP) for radius optimization.
• LP optimizes radii well but centers are fixed: score saturates. SI reveals "dual variables indicate high sensitivity of score to center positions #3 and #7." Proposer switches to sequential linear programming (SLP) for joint center-radius optimization.
• SLP gets stuck in local optima. SI reveals "improvement rate has flatlined for 50 iterations." Proposer switches to bilevel L-BFGS-B with exact LP-derived gradients.
• Bilevel L-BFGS-B is deterministic and gets trapped. SI reveals "multiple restarts converge to similar scores." Proposer adds CMA-ES global exploration with automatic restarts.

The critical property: each algorithmic shift is targeted — it addresses the specific failure mode that SI revealed, rather than being a random mutation. Score-only feedback can only tell the proposer "the score stopped improving," not why it stopped or what kind of algorithmic change might help.

Mechanism 2: Multi-module Pareto leapfrogging. optimize_anything optimizes both the code artifact and a refiner prompt — the prompt that tells the proposer how to reflect on and fix broken artifacts — simultaneously, tracking both on the shared Pareto frontier. In circle packing, this creates a productive coordination dynamic:

• The code module is still a weak heuristic (score ≈ 0.98). The refiner prompt discovers that LP-based optimization would solve the radius collapse problem (refiner prompt score ≈ 1.93 — measured by how well it diagnoses failures).
• The code module absorbs the LP approach: next proposer call, building on the refiner's diagnosis, generates an LP-based packing algorithm (code score jumps to ~2.61).
• The refiner prompt, seeing that LP is now implemented, pushes further by diagnosing the center-placement bottleneck and suggesting SLP (refiner prompt score ~2.63).
• The code module absorbs SLP and reaches the world record score.

Each module's advances become the foundation for the other's next improvement. The paper notes that even broken code mutations (score = 0.0) are recovered by the refiner and retained on the frontier, acting as a "safety net that preserves exploration" — a candidate that failed due to a trivial bug might contain a valuable algorithmic insight that the refiner can salvage.

Mechanism 3: Pareto diversity prevents premature convergence. At convergence, the Pareto frontier retains candidates from multiple algorithmic families simultaneously: greedy, LP, SLP, bilevel L-BFGS, CMA-ES. They survive on different quality dimensions — max score, mean score, EMA stability, improvement rate. When the proposer samples a parent for mutation, it might select a CMA-ES candidate (high improvement rate but lower raw score) rather than always selecting the L-BFGS champion. This structural diversity means the proposer can recombine strategies across algorithmic families rather than being locked into refining a single approach.

Even when LP dominates on raw score, greedy and CMA-ES candidates survive on stability and improvement-rate metrics. If the L-BFGS approach later stagnates, the CMA-ES candidate can be selected and hybridized with LP-derived gradient information — a combination that would never emerge from a single-family optimizer.

4. Key Insights and Innovations

Innovation 1: The Artifact-Agnostic Optimization Interface is the Contribution, Not a Better Algorithm

The paper's most fundamental move is a reframing of the problem itself: LLM-based optimization is not a collection of domain-specific tools that happen to use similar techniques, but a single unified paradigm that can be expressed through one declarative API. This is a conceptual contribution about what the problem is, not an algorithmic contribution about how to solve it better.

Before this paper, the field treated code optimization (AlphaEvolve, FunSearch, OpenEvolve) and prompt optimization (GEPA, MIPROv2, TextGrad) and agent architecture search (ADAS, AFlow) as separate subproblems requiring separate frameworks. Each system had its own interface, its own configuration abstractions (island topologies, EVOLVE-BLOCK markers, mutation prompt templates), and its own assumptions about what types of feedback were available. A practitioner optimizing a CUDA kernel and a practitioner optimizing a system prompt used entirely different tools, despite the fact that both were running the same fundamental loop: generate text, evaluate it, feed results back to an LLM, repeat.

The reframing is: all text optimization is the same problem. Whether the artifact is Python code, a natural-language prompt, SVG markup, or a 3D modeling script, the structure is identical — serialize as string, evaluate, return score + diagnostics, let LLM propose improvement. The paper captures this with an observation that is obvious in retrospect but was not the default assumption before this work:

"We observe that a wide range of problems can be formulated as optimizing a text artifact. Whether the artifact is a CUDA kernel, a cloud scheduling policy, an agent architecture, Scalable Vector Graphics (SVGs), or a system prompt, the structure is the same: serialize the artifact as a string, evaluate it, and let an LLM propose improvements based on diagnostic feedback."

What makes this a genuine innovation rather than a trivial observation is that acting on this insight required solving real generalization problems. Section 4 details five concrete algorithmic modifications needed to lift GEPA's prompt-specific search to arbitrary artifacts: new frontier types for single-task and multi-task search (since prompt optimization always assumes multiple data points), a refiner step for malformed code (irrelevant for prompts, essential for CUDA), content-addressed caching for expensive evaluators, SI as a typed primitive, and a backend adapter layer. These are not cosmetic changes — they represent the gap between "these problems look similar" and "a single system actually works across all of them without domain-specific hacks."

The significance extends beyond developer convenience. The unified interface enables cross-domain methodological insights that single-domain frameworks make impossible to ask. Does multi-task transfer work for scheduling policies the way it works for CUDA kernels? Does SI-driven reflection accelerate convergence on images the way it does on prompts? Prior frameworks couldn't answer these questions because they couldn't run the experiments — each domain required a separate framework, making controlled cross-domain comparisons infeasible. The paper doesn't fully explore all cross-domain questions, but it establishes the intellectual scaffolding for doing so.

The evidence that this reframing is real, not just aspirational, is Table 1: the same API call, with no domain-specific configuration, achieves state-of-the-art or competitive results across six fundamentally different domains — code optimization (circle packing beats AlphaEvolve), prompt optimization (AIME matches GEPA's reported gains), agent architecture search (ARC-AGI nearly triples baseline accuracy), cloud scheduling (tops ADRS leaderboard), CUDA kernel generation (87% match PyTorch), and image generation (unanimously preferred by human evaluators). A single system that works across all of these has never been demonstrated before.

This is a fundamental shift in how to think about LLM-based optimization: the right abstraction level is the text artifact and its evaluator, not the specific problem domain. Future optimization algorithms can target this interface and automatically work for all text artifacts, rather than requiring per-domain reimplementation.

Innovation 2: Side Information as a First-Class Gradient for Text Optimization

The paper introduces Side Information (SI) as a new abstraction layer between the evaluator and the proposer, and argues — with cross-domain evidence — that this abstraction is the key to making LLM-based optimization general. The conceptual move is to treat diagnostic feedback not as an ad-hoc implementation detail but as a first-class evaluator contract with its own design principles and measurable impact.

Prior frameworks handled diagnostic feedback inconsistently and opaquely. AlphaEvolve feeds execution traces back to the LLM, but through framework-specific code paths that assume code execution — you cannot surface a VLM's aesthetic critique of a rendered image through the same mechanism. GEPA uses per-example scores and sub-scores, but the feedback structure is baked into its prompt optimization logic — you cannot surface CUDA compiler errors without modifying the framework. TextGrad generates "gradient" text via LLM calls, which handles natural-language tasks well but cannot incorporate profiler traces or rendered visualizations without manual preprocessing by the user.

In each case, the feedback mechanism is framework-internal — the user must understand how the framework expects to receive diagnostics and format their evaluator output accordingly. This means that switching domains requires learning a new feedback mechanism, and that domain experts cannot simply surface the diagnostics they already produce (compiler errors, profiler summaries, rendered outputs) without translating them into the framework's expected format.

optimize_anything's move is to invert this relationship: the framework makes no assumptions about what SI contains, and simply serializes the evaluator's side_info dictionary into the proposer's prompt. The proposer LLM — not the framework — interprets the diagnostic content. This is the right abstraction boundary because LLMs are already capable of understanding diverse diagnostic formats (NVCC errors, Python stack traces, natural-language critiques, rendered images), and any framework-side interpretation would restrict what diagnostics can be surfaced without adding value.

The paper's analogy makes the conceptual contribution explicit:

"SI is the text-optimization analogue of the gradient. Where gradients tell a numerical optimizer which direction to move, SI can tell the LLM proposer why a candidate failed and how to fix it."

This analogy is precise: in gradient-based optimization, the gradient is a first-class object that the optimizer computes from the loss function, and the optimization algorithm operates on gradients regardless of what the loss function represents. Similarly, SI is a first-class object computed by the evaluator, and the Pareto-based search algorithm operates on SI regardless of what domain the evaluator represents. The optimization algorithm is strictly more general than any specific SI format, just as gradient descent is strictly more general than any specific loss function.

What elevates SI from a feature (every framework has some way to pass feedback) to an innovation is the cross-domain evidence that the format and quality of SI directly controls optimization effectiveness. The SI ablation (Section 5.9, Table 4) shows that across three different domains — prompt optimization, circle packing, and CUDA kernel generation — actionable SI yields 4–6× faster convergence and substantially higher final scores than score-only feedback. On CUDA kernels with multi-task search, SI enables 40% of kernels to exceed 1.1× speedup vs. 0% with score-only feedback. On circle packing, SI achieves the optimal solution while score-only reaches only 93.96% of the best score.

The mechanism differs by domain, but the principle is consistent: SI converts "score decreased from A to B" into a diagnostic that identifies which failure mode occurred and where to look. For code, SI surfaces the specific compiler error and line number. For agents, per-puzzle traces reveal which component failed. For cloud scheduling, SI exposes the temporal decision structure that led to deadline violations. In each case, the proposer can make targeted fixes rather than blind mutations, and the convergence acceleration is measured in multiples, not percentages.

This is a fundamental innovation in how to design LLM-based optimization interfaces: SI should be an explicit user-facing contract, not an internal implementation detail. The paper argues this turns domain expertise — knowing what diagnostics to surface — into an optimization superpower, while hiding optimization expertise (mutation strategies, frontier management) behind the interface. Section 8 states it clearly: "optimize_anything trades optimization expertise for domain expertise. The user, most often a domain expert, need not configure backends, tune algorithmic hyperparameters, or engineer prompting strategies, only surface the diagnostics they already understand."

Innovation 3: Multi-Task Search as Cross-Problem Transfer via Pareto Frontier Sharing

The paper introduces multi-task search as a new optimization mode that enables cross-problem transfer of discovered optimization patterns through a shared Pareto frontier, and demonstrates both its benefits (on CUDA kernels) and its failure conditions (on circle packing). This is a genuinely new capability that no prior LLM-evolution framework supports, and the paper provides empirical evidence for when it helps and when it hurts.

Prior frameworks (AlphaEvolve, OpenEvolve, ShinkaEvolve, FunSearch) all operate in single-task mode: optimize one artifact for one problem at a time. If you have N related problems, you run N independent optimization jobs. Patterns discovered for problem A — float4 vectorization for matrix multiplication, warp shuffle reductions for LayerNorm — are discarded when you start optimizing problem B. The optimizer must rediscover these patterns independently for each problem, incurring the same exploration cost N times.

Multi-task search changes this by sharing the Pareto frontier across tasks. When a candidate is selected for mutation, it might be a specialist for problem A (best on that objective) presented alongside a minibatch from problem B. The proposer sees what worked for A, sees where it fails on B, and generates a candidate that transfers A's optimization pattern with adaptations for B's specific requirements. This is structurally impossible in single-task mode, where the proposer only sees candidates from the same problem's history.

The evidence in Section 5.8 and Figure 8 demonstrates that this mechanism works in practice: on 10 CUDA kernel problems, multi-task mode achieves 90% of kernels matching baseline vs. 60% for single-task at equivalent per-problem budget, with the gap widening at higher speedup thresholds (20%+ speedup: multi-task continues improving while single-task plateaus early). The benefit scales with the number of related tasks — MT20 (20 problems) outperforms MT10 (10 problems) at moderate speedup thresholds (Table 6-7) — because more problems mean more patterns discovered and available for transfer.

But the paper is equally instructive about when multi-task search fails. Section 7 and Table 5 show that on circle packing, where optimizing different values of N jointly introduces noise rather than useful cross-transfer, multi-task mode degrades performance. The paper's diagnosis is precise:

"Circle packing problems for different N are fundamentally independent, optimal configurations change unpredictably with N, with no transferable structure"

This negative result is a contribution in itself: it establishes that the benefit of multi-task search depends on tasks sharing underlying optimization patterns. CUDA kernels on the same hardware share memory access patterns, vectorization strategies, and reduction techniques — the optimization landscape has transferable structure. Circle packing for different N values involves independent geometric constraints with no shared optimization principles — the landscape has no transferable structure, and multi-task search introduces noise by presenting the proposer with irrelevant candidates.

This is a fundamental innovation because it defines a new axis of optimization capability (cross-problem transfer) and provides partial boundary conditions for when it works. It converts the implicit question "should I optimize these problems together or separately?" from intuition to empirical evidence. The mechanism — Pareto frontier sharing — is simple but the insight that it enables cross-transfer is non-obvious: prior work assumed that evolutionary search requires problem-specific populations, and the idea that a single population across problems accelerates convergence through pattern transfer is a genuine conceptual advance.

Innovation 4: Verifier-Free Optimization via Diagnostic-Driven Reflection

The paper establishes — partly by negative result — that LLM-based text optimization does not require a learned verifier or reward model. Instead, the combination of the evaluator's raw score plus diagnostic Side Information, processed through a reflection step, provides sufficient signal for effective optimization. This distinguishes the approach from the RLHF/PRM paradigm that dominates much of the LLM optimization literature, and has implications for how to think about optimization signal quality.

In the dominant paradigm exemplified by RLHF and PRM-based approaches (as seen in the inference-time compute literature), optimization quality depends critically on the quality of a learned verifier. The verifier is trained to predict human preferences or correctness, and optimization algorithms (PPO, best-of-N, beam search) optimize against the verifier's scores. The bottleneck is verifier calibration: over-optimization occurs when search finds solutions that score highly under the verifier but are actually incorrect, and substantial research effort goes into building more robust verifiers.

optimize_anything takes a fundamentally different approach: the evaluator is the ground truth. There is no learned proxy — the artifact is executed (code is run, the agent is tested on puzzles, the image is rendered and scored by a VLM) and the resulting score is the optimization signal. SI augments this signal with diagnostics, but the diagnostics are factual (compiler errors, execution traces, rendered visualizations), not learned estimates of quality.

This is not a minor implementation detail — it's a fundamental architectural choice with significant implications. The paper's approach cannot over-optimize a proxy because there is no proxy. The evaluator's score is the actual performance metric, so every improvement in the optimization loop corresponds to a genuine improvement in artifact quality. The failure mode shifts from "the optimizer is gaming the verifier" (the RLHF problem) to "the proposer is failing to generate improvements given the diagnostics" (an LLM capability problem), which is a qualitatively different — and arguably more tractable — challenge.

The evidence that this approach works at scale comes from the diversity of evaluators that drive successful optimization: a V100 GPU benchmarking CUDA kernels, an ARC-AGI puzzle runner scoring agent outputs, a cloud infrastructure simulator computing data egress costs, a circle packing renderer computing geometric overlap, and a VLM scoring rendered SVGs on aesthetic properties. None of these evaluators are learned verifiers; all of them compute the actual performance metric they claim to compute. The system achieves state-of-the-art results without training a single reward model.

This is a fundamental reframing of what makes LLM-based optimization work. The question is not "how do we train a better verifier" but "how do we surface the right diagnostic information so the LLM can reason about improvements." The paper's trajectory analysis (Section 6, Appendix F) provides mechanistic evidence: SI-driven reflection enables targeted algorithmic shifts (greedy → LP → SLP → bilevel L-BFGS in circle packing) that arise from the proposer diagnosing specific failure modes from factual diagnostics, not from optimizing against a learned score function. The gap between score-only and score+SI feedback (4–6× convergence acceleration, Table 4) quantifies how much of the optimization power comes from diagnostic reasoning rather than from numerical optimization.

This insight generalizes beyond this paper's implementation. It suggests that future LLM-based optimization systems should invest in evaluator quality and diagnostic richness, not in learned verifier training, and that the key skill for domain experts using these systems is knowing what diagnostics to surface rather than knowing how to tune optimization hyperparameters.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates across six primary domains, each with its own dataset: (1) ARC-AGI [7] — a visual reasoning benchmark using the standard train/test split; (2) ADRS cloud scheduling benchmark [6] with training and validation splits over infrastructure scenarios; (3) AIME mathematics competition problems, using 2022–2024 for training and 2025 for testing; (4) KernelBench [20] — 31 reference PyTorch operations evaluated on a V100 32GB GPU; (5) Circle packing with n=26 in a unit square (single problem, no split); (6) SVG/CAD image generation with 4 goals evaluated by a VLM. Additional domains in appendices: EvalSet blackbox optimization benchmark (56 problems) [16] and a 3D modeling task.

• Base model(s). The proposer LLM — the model that generates improved candidate artifacts during reflection — varies by domain: GPT-5/5.1 for CUDA kernels, circle packing, and AIME prompts; Gemini 3 Flash for ARC-AGI (serving as both proposer and the underlying agent model); Gemini 3 Pro for cloud scheduling; Claude Opus 4.6 for coding agent skills and 3D modeling. The paper also compares proposer quality using GPT-5-nano as a cheaper alternative (Table 8). The choice is pragmatic — different models perform better on different tasks — and the system is designed to work with any sufficiently capable LLM.

• Metrics. Primary metrics are domain-specific: ARC-AGI accuracy (% of puzzles solved correctly), cloud scheduling cost savings (%), AIME accuracy (% problems correct), CUDA kernel speedup ratio versus PyTorch baseline, circle packing sum of radii (continuous score), VLM-based visual quality scores (0–100 scale per aspect), and code agent pass rate (% of repository tasks resolved). For multi-task CUDA, the paper reports Fast_p(s): the fraction of kernels achieving speedup ≥ s. For cloud scheduling, the ADRS leaderboard aggregate score combines metrics across scenarios.

• Baselines. The paper compares against a wide range of domain-specific baselines: AlphaEvolve [18] and OpenEvolve [24] for circle packing (controlled rerun with matched proposer LLM); MIPROv2 [19] for AIME prompt optimization; PyTorch reference implementations for CUDA kernels (KernelBench baseline); Dijkstra routing for CloudCast and a greedy deadline-check heuristic for Can't Be Late; zero-shot LLM generation for image tasks (same model without optimization); Optuna [4] for blackbox mathematical optimization; and naive single-call agents for ARC-AGI. For multi-task ablation: single-task mode of optimize_anything itself at equivalent per-problem budget.

• Generation budget / compute accounting. The unit of optimization budget is evaluator calls — each call runs the candidate artifact and returns a (score, SI) pair. Total evaluations vary by domain: circle packing uses 63–100 evaluations, KernelBench uses a per-kernel budget, ARC-AGI consumes significant budget due to expensive puzzle evaluations ($144.70 total). The system tracks evaluation count rather than proposer LLM calls, since evaluator cost typically dominates (Table 9: reflection is$0.70 vs. evaluator $144 for ARC-AGI). For fair comparison against OpenEvolve on circle packing, the paper matches proposer LLM (GPT-5.1) and reports evaluation counts for both systems. For multi-task vs. single-task ablation, per-problem budget is held constant.

• Cross-validation / statistical protocol. For generalization mode, the paper uses standard train/validation splits — AIME 2022–2024 vs. 2025, ARC-AGI public training vs. test sets, cloud scheduling with predefined scenario splits. For circle packing, results are confirmed by controlled rerun of OpenEvolve under matched conditions. For image generation, five human evaluators compared optimized vs. zero-shot images. The paper does not report confidence intervals or statistical significance tests; results are presented as point estimates from single optimization runs.

Main Quantitative Results

Agent Architecture Search: ARC-AGI (Generalization Mode)

The headline result is that optimize_anything evolves a 10-line naive agent seed into a 300+ line 4-stage pipeline that nearly triples Gemini 3 Flash's ARC-AGI test accuracy from 32.5% to 89.5% — a 57 percentage point gain (Section 5.3, Figure 4). The validation accuracy reaches 93.5%, suggesting minimal overfitting to training puzzles.

The optimized architecture implements qualitatively novel components not present in the seed: (1) rule induction via pattern analysis, (2) code generation with exec()-based verification, (3) iterative debugging with up to 2 fix attempts, and (4) structured fallback from code-first to direct LLM prediction (Appendix J.2, Figure 10). These architectural patterns — verify-then-fallback, iterative refinement with retry limits — are the kind of design decisions that typically require manual engineering iterations. The system discovers them automatically from per-puzzle SI (execution traces, error tracebacks, model outputs) driving targeted reflection.

The evaluation cost is high ($144.70 total, with$144 spent on running the agent against puzzles and only $0.70 on proposer LLM reflection calls), but this reflects evaluation expense rather than optimization overhead — the system is sample-efficient in proposer calls, making most of its budget work in the evaluator.

Cloud Scheduling Algorithms (Generalization Mode)

CloudCast achieves 40.2% cost savings over Dijkstra routing for multi-cloud data transfer (Section 5.2, Figure 3a). The evolved algorithm (178 lines, Appendix J.3) qualitatively departs from the shortest-path seed by discovering provider-aware Steiner tree routing — it biases path selection toward intra-provider links to minimize egress costs while maintaining bandwidth constraints. This capability is absent from the initial Dijkstra seed, which routes purely by shortest path without provider awareness.

Can't Be Late achieves 7.8% cost savings (Section 5.2, Figure 3b) by evolving a simple deadline-check heuristic into an adaptive strategy (110 lines, Appendix J.4) with three learned behaviors: break-even switching cost analysis that avoids costly SPOT→ON_DEMAND transitions when remaining work is small, persistent spot-unavailability tracking via a counter that detects when SPOT instances are unlikely to return, and graduated decision thresholds based on slack ratio that become increasingly aggressive as deadlines approach.

Both results top the ADRS leaderboard: optimize_anything achieves 96.6 aggregate score vs. 92.9 for OpenEvolve and 72.0 for ShinkaEvolve. The optimization trajectories (Figure 3) show steady improvement over iterations rather than sudden jumps, suggesting cumulative refinement rather than a single lucky mutation.

ARC-AGI Agent Architecture (Already covered above — this is the same as §5.3)

AIME Prompt Optimization (Generalization Mode)

Prompt optimization for GPT-4.1-mini improves AIME 2025 accuracy from 46.67% to 60.00% — a 13.3 percentage point gain from changing only the system prompt (Section 5.4, Figure 5). This outperforms MIPROv2 (51.33% on the same benchmark), demonstrating that the general interface does not sacrifice performance on prompt-specific tasks.

The optimized prompt (Appendix I) evolves from a single generic sentence into a structured 6-rule reasoning framework: restate the problem, set up notation cleanly, show logically ordered reasoning with theorem justification, handle dead ends explicitly, keep reasoning focused and minimal while rigorous, and isolate the final answer on its own line. The system discovers these rules from per-problem SI (reasoning chain, extracted answer, ground truth, correct/incorrect flag) without being told what makes a good math prompt.

Validation score improves from 46.67% to 57.78%, with test performance (60.00%) slightly exceeding validation — an unusual pattern that likely reflects AIME 2025 being marginally easier than AIME 2022–2024 rather than genuine positive generalization gap.

CUDA Kernel Generation (Multi-Task Search)

The headline: 87% of generated kernels match or beat the PyTorch baseline; 48% achieve 10%+ speedups; 25% achieve 20%+ speedups (Section 5.5, Figure 6). The best individual kernel — LayerNorm — achieves a 3.32× speedup over PyTorch (Appendix J.5).

The evolved kernels employ techniques such as float4 vectorization (loading four values per memory transaction), two-pass algorithms (compute statistics first, then normalize), warp shuffle reductions (direct register-to-register partial sums bypassing shared memory), and shared memory tiling. These are non-trivial GPU optimization strategies that the system discovers from NVCC compiler errors, correctness test failures, profiler data, and CUDA documentation snippets included in the SI.

Multi-task mode is critical: on 10 selected problems, multi-task achieves 90% kernels matching baseline vs. 60% for single-task at equivalent per-problem budget (Figure 8). The gap widens at higher speedup thresholds — single-task plateaus early while multi-task continues improving. The mechanism is cross-transfer via the shared Pareto frontier: optimization patterns discovered for one kernel (vectorized memory access for matrix multiplication) transfer to others (the same pattern adapted for LayerNorm).

Circle Packing (Single-Task Search)

For n=26 circles in a unit square, optimize_anything reaches a score of 2.63598+, outperforming AlphaEvolve's, OpenEvolve's, and ShinkaEvolve's reported solutions (Section 5.6, Figure 7). The evolved algorithm (480+ lines, Appendix J.6) is a bilevel optimizer with multiple components not present in the seed: linear programming (LP) over radii with dual-variable sensitivity analysis providing exact gradients for center optimization, L-BFGS-B over centers using LP-derived gradients, block SLP trust-region boosts for worst-performing circles, CMA-ES global exploration with automatic restarts, and six diverse seeding strategies (hexagonal, uniform, edge-ring, farthest-point, corner-spokes, edge-biased hex).

Controlled comparison with OpenEvolve (Table 3): Under matched conditions (GPT-5.1 as proposer), optimize_anything achieves 2.63598 in 63 evaluations (~$3.18), while OpenEvolve reaches only 2.4583 at 100 evaluations and 2.6307 at 200 evaluations (~$6.85). optimize_anything achieves a better score with less than one-third the evaluation budget, confirming sample efficiency.

Image Generation (Multi-Task Search)

Five human evaluators unanimously preferred optimize_anything-optimized images over zero-shot baselines across all four goals (Section 5.7). Quantitatively, the "pelican riding a bicycle" task achieves a VLM score of 0.726 vs. 0.330 for the zero-shot baseline — a 2.2× improvement (Appendix Figure 11).

The multi-task setup here differs from CUDA: each evaluator call scores one visual aspect (one of 12–13 natural-language properties rated 0–100 by a VLM), making this a natural multi-task search over the Pareto frontier of visual properties. The optimized images show qualitative improvements in composition, structure, detail, and overall visual quality across all four tasks (octopus on a pipe organ, sloth steering an excavator, pelican riding a bicycle, 3D unicorn).

Blackbox Mathematical Optimization (Single-Task Search, Appendix B)

On the 56-problem EvalSet benchmark against Optuna, with a budget of 8,000 evaluations per problem, optimize_anything ties Optuna on 40 problems, wins 7, and loses 9. On 10 selected problems where Optuna struggles with lower budgets (2,000 evaluations), optimize_anything finds better solutions on 7 out of 10.

The mechanism: Optuna's fixed TPE-CMA-ES pipeline fails in predictable ways (TPE's per-dimension sampling converges to trap basins; CMA-ES assumes smooth unimodal landscapes). optimize_anything tailors the solver algorithm to each problem — discovering L-BFGS-B for boundary optima and multi-start search for deceptive traps — by optimizing the solver code itself rather than tuning parameters within a fixed algorithm.

Ablation Studies and Robustness Checks

Multi-task vs. single-task search (Section 5.8, Figure 8): On 10 CUDA kernel problems re-optimized from scratch in single-task mode with equivalent per-problem budget, multi-task consistently outperforms across all speedup thresholds. MT10 achieves 90% kernels matching baseline vs. 60% for single-task; the gap widens at higher thresholds (Fast_p(1.2): single-task plateaus early while multi-task continues improving). The benefit scales with the number of related tasks — MT20 (20 problems) outperforms MT10 (Table 6–7), establishing that cross-transfer benefits compound with more related problems.

When multi-task search hurts (Section 7, Table 5): On circle packing with different N values, multi-task search degrades performance. Single-task achieves 2.6360; MT7 (7 different N values jointly) achieves 2.6313; MT11 achieves 2.5973. The paper attributes this to the fundamental independence of circle packing problems for different N — optimal configurations change unpredictably with N, with no transferable structure. This negative result is important: it establishes that multi-task search helps when tasks share underlying optimization patterns (CUDA kernels on same hardware) and hurts when they are fundamentally independent.

Side Information vs. score-only feedback (Section 5.9, Figure 9, Table 4): On prompt optimization for the Facility Support Analysis dataset, SI with per-aspect sub-scores reaches a validation score of 0.80 within ~100 rollouts, while score-only requires ~600 rollouts — a 6× acceleration. The test score with SI is 86.32 vs. 82.5 without, confirming that convergence speed and final quality both benefit.

Cross-domain SI ablation (Table 4): On circle packing, SI achieves the optimal solution (100% of best score); score-only reaches 93.96%. On CUDA kernels in single-task mode, SI enables 32.3% of kernels to exceed 1.1× speedup (mean speedup 4.11×) vs. 12.9% (mean 1.15×) with score-only. In multi-task mode, SI enables 40% exceeding 1.1× vs. 0% with score-only (mean 1.15× vs. 1.03×). The mechanism is consistent: SI reveals which failure mode to address next (compilation error, incorrect output, memory bandwidth bottleneck), while score-only feedback only indicates that performance changed.

Proposer LLM sensitivity (Section 5.10, Table 8): GPT-5.1 vs. GPT-5-nano comparison reveals a clear cost-performance tradeoff. On AIME prompts: GPT-5.1 reaches 60.0% (cost $6.44); GPT-5-nano reaches 50.0$3.71) — still a 3.3pp improvement over the 46.67% seed. On circle packing: GPT-5.1 achieves 2.636 (cost $6.00); GPT-5-nano achieves 2.512 (cost$0.50) — 92% cost reduction with substantial but sub-optimal improvement. The paper argues this enables a practical workflow: use nano for rapid exploration, then the larger model for final convergence.

Cost analysis (Section 5.10, Table 9): Total optimization costs range from $1 (Numerical Blackbox) to$144.70 (ARC-AGI). Across all domains, reflection cost (proposer LLM calls) is consistently minimal compared to evaluator cost: ARC-AGI reflects $0.70 vs. evaluator$144, AIME reflects $2.17 vs. evaluator$4.27. This confirms a key property of LLM-based optimization: it is highly sample-efficient in evaluator calls, spending the majority of budget on evaluation rather than optimization overhead.

Refiner mechanism (Appendix F): The refiner catches malformed code blocks, import errors, and syntax issues before evaluation. Even broken code mutations (score=0.0 from syntax errors) are recovered by the refiner and retained on the Pareto frontier, acting as a safety net that preserves algorithmic insights despite trivial formatting failures.

Multi-module Pareto leapfrogging (Section 6, Appendix F): The paper's trajectory analysis on circle packing reveals that optimizing the code artifact and refiner prompt simultaneously on a shared Pareto frontier creates a productive coordination dynamic: the refiner discovers LP-based optimization while code is still weak (code=0.98, refiner=1.93); code absorbs LP approach (→2.61); refiner pushes further with SLP (→2.63); code absorbs SLP and reaches the world record. This mechanism is absent from single-artifact systems.

Pareto diversity preservation (Section 6, Appendix F): At convergence, the Pareto frontier retains candidates from multiple algorithmic families simultaneously (greedy, LP, SLP, bilevel L-BFGS, CMA-ES) across quality dimensions (max score, mean score, EMA stability, improvement rate). This prevents premature convergence to a single strategy family and enables the proposer to recombine structurally diverse approaches.

Seedless mode (Appendix C): Demonstrated on a 3D unicorn generation task using build123d + pyrender. Starting from no code — only a natural-language objective — the system bootstraps the first candidate and iteratively refines geometry, proportions, and anatomical detail based on multi-view rendered PNGs scored by a VLM, confirming that the system can operate without a seed when domain expertise to write even a bad initial artifact is scarce.

Critical Assessment

Claim 1: "A single LLM-based Text Optimization system matches or surpasses domain-specific tools across six fundamentally different domains."

What was demonstrated: The paper shows that optimize_anything achieves strong results on each of six domains: ARC-AGI (89.5% vs. 32.5% seed), cloud scheduling (40.2% cost savings), AIME prompts (60.0% vs. 46.67% seed), CUDA kernels (87% match baseline), circle packing (2.636 vs. AlphaEvolve's reported solution), and image generation (unanimous human preference). The same API call, with no domain-specific configuration, was used across all domains.

What was NOT demonstrated: The paper does not systematically compare against the best domain-specific tool in every domain with matched proposer models and budgets. The ARC-AGI result's primary comparison is against its own seed — no comparison against ADAS [11] or AFlow [29] under matched conditions is reported. The cloud scheduling results compare against the ADRS leaderboard (which includes OpenEvolve and ShinkaEvolve entries), but those entries may have used different LLMs and different budgets; the comparison is leaderboard-based, not a controlled rerun. The AIME prompt comparison against MIPROv2 is a single number (51.33%) with no detail on matching conditions. The circle packing comparison is the strongest — a controlled rerun of OpenEvolve with the same proposer LLM — and it convincingly shows optimize_anything's advantage (2.636 in 63 evals vs. 2.6307 in 200 evals). For CUDA kernels, there is no comparison against a domain-specific CUDA optimization tool (the baseline is PyTorch, not another optimization system).

Conditional strength: The claim that a single system can work across domains is well-supported — the diversity of artifacts (code, prompts, agents, SVGs, scheduling policies, 3D models) is genuinely impressive and unprecedented. The claim that it "matches or surpasses" domain-specific tools is supported most strongly for circle packing (controlled comparison) and cloud scheduling (leaderboard), moderately for prompts (MIPROv2 comparison exists but is underexplained), and weakly for agent architectures and CUDA kernels (no head-to-head against domain-specific search tools).

A genuine strength: the paper does not cherry-pick easy domains. The six tasks require fundamentally different optimization dynamics — single-task geometric optimization (circle packing), multi-task code generation with cross-transfer (CUDA), generalization of agent architecture to unseen puzzles (ARC-AGI), policy optimization under infrastructure uncertainty (cloud scheduling), prompt refinement for mathematical reasoning (AIME), and generative visual quality improvement (SVG/CAD). That the same system handles all of these without per-domain hacks is the paper's strongest evidence.

Claim 2: "Three optimization modes — single-task, multi-task, and generalization — unified under one interface, including the first multi-task mode."

What was demonstrated: The three modes are clearly distinguished, well-motivated, and each is demonstrated on at least one domain. Multi-task mode's benefits are quantified against single-task (Figure 8, Tables 6–7) with a clear mechanism (Pareto frontier sharing) and scaling behavior (MT20 > MT10 > ST). Generalization mode is demonstrated on three domains (ARC-AGI, cloud scheduling, AIME prompts) with standard train/validation splits. The paper also demonstrates when multi-task fails (circle packing, Table 5), which strengthens credibility.

What was NOT demonstrated: The paper does not systematically explore the boundary between multi-task and generalization modes. If you have related problems and a held-out set, should you use multi-task (produce N specialized artifacts) or generalization (produce one global artifact)? This is not tested. The ARC-AGI domain uses generalization mode (one agent architecture for all puzzles), but it could also be cast as multi-task search over puzzle types — the choice seems to be based on what the domain requires rather than a principled comparison of modes.

The multi-task scaling results (MT10 vs. MT20 vs. ST) are based on subsets of the 31 KernelBench problems with unclear selection criteria ("10 best multi-task problems," "20 randomly sampled"). A more systematic scaling analysis would use all 31 problems and show the marginal benefit of each additional problem.

A subtle distinction: the paper frames multi-task mode as producing N specialized artifacts while generalization produces one global artifact. But the CUDA kernels in multi-task mode are all generated from a single evolving prompt (Appendix J.5 shows the prompt drives kernel generation), so the "specialized artifact per task" framing is somewhat at odds with the actual implementation, where a shared prompt generates specialized kernels. This doesn't invalidate the results but suggests the distinction between modes is more nuanced than the paper presents.

Claim 3: "Side information yields 4–6× faster convergence and substantially higher final performance versus score-only feedback."

What was demonstrated: The SI ablation is cleanly designed, tested across three domains (prompt optimization Figure 9, circle packing and CUDA kernels Table 4), and the mechanism is well-articulated through trajectory analysis (Section 6, Appendix F). The 4–6× claim is supported by the prompt optimization convergence curves (6× faster to reach 0.80 validation score) and the CUDA multi-task result (40% vs. 0% kernels exceeding 1.1× speedup).

What was NOT demonstrated: The SI ablation is only measured on a subset of domains. There is no SI ablation for agent architecture search (ARC-AGI), cloud scheduling, agent skills, or image generation. Given that the paper's central claim is that SI is the key enabler of cross-domain generality, the ablation should ideally span all major domains rather than three. It's plausible that SI matters even more in domains like ARC-AGI (where per-puzzle traces are rich diagnostics) or matters less in domains like cloud scheduling (where the score is a composite of many scenarios and individual SI may be noisy), but this is unknown.

The ablation also conflates multiple aspects of SI: in the prompt optimization ablation (Figure 9), "with SI" provides per-aspect sub-scores, while "without SI" provides only aggregate score. But "without SI" still provides which examples performed well or poorly (since the Pareto frontier tracks per-example scores) — it's not truly score-only in the sense of a single scalar with no diagnostic whatsoever. The CUDA kernel ablation provides richer SI (compiler errors, test failures, documentation snippets) vs. only a pass/fail score. What specific component of SI matters most — sub-scores vs. error messages vs. documentation context — is not ablated.

The ablation does not test whether SI quality matters. Would noisy or misleading SI (e.g., wrong error attribution) hurt performance below score-only? The paper frames SI as always beneficial, but in domains where the evaluator cannot produce reliable diagnostics, score-only might be preferable. This boundary condition is unexplored.

Claim 4: "Multi-task search outperforms independent single-task optimization given equivalent per-problem budget through cross-task transfer, with benefits scaling with the number of related tasks."

What was demonstrated: This claim is well-supported by Figure 8 and Tables 6–7, with a clear mechanism explanation (Pareto frontier sharing enabling pattern transfer). The negative result on circle packing (Table 5) provides a boundary condition: cross-transfer helps when tasks share underlying patterns and hurts when they are fundamentally independent. The scaling result (MT20 > MT10 > ST) suggests the benefit is monotonic in the number of related tasks.

What was NOT demonstrated: The paper does not systematically characterize what makes tasks "related enough" for multi-task to help. The contrast between CUDA kernels (helps) and circle packing N values (hurts) is clear, but these are extreme cases. What about a mix of partially related problems? At what degree of relatedness does the crossover occur? The paper doesn't probe this boundary.

The ablation uses subsets of KernelBench problems (10 or 20 out of 31) with unclear selection criteria. The "10 best multi-task problems" phrasing suggests the problems were selected because multi-task worked well on them, which would bias the comparison. The MT20 result uses "20 randomly sampled" problems and still shows benefit, which provides some robustness, but a systematic analysis across all 31 problems would be stronger.

The mechanism of cross-transfer is described qualitatively ("patterns discovered for one kernel transfer to others automatically through the shared Pareto frontier") but not verified quantitatively. Do specific patterns (float4 vectorization, warp shuffle reductions, shared memory tiling) appear earlier or more frequently in multi-task mode? Are the same patterns independently rediscovered in single-task mode? The qualitative trajectory analysis (Appendix F) provides some evidence for code and refiner prompt co-evolution, but the specific cross-task transfer mechanism is not traced.

Claim 5: "Our system discovers agent architectures that nearly triple Gemini Flash's ARC-AGI accuracy (32.5% → 89.5%)."

What was demonstrated: The accuracy improvement is dramatic and the trajectory (Figure 4) shows steady progress rather than a lucky jump. The final architecture (Appendix J.2, Figure 10) is qualitatively sophisticated — a 4-stage pipeline with verify-then-fallback logic that would require significant manual engineering to design.

What was NOT demonstrated: The paper does not characterize what the agent is actually doing on the test puzzles. Does it succeed on the same puzzle types that the seed succeeded on, just more reliably? Or does it succeed on qualitatively harder puzzles that the seed couldn't touch? The per-puzzle analysis is absent, making it hard to assess whether the improvement represents better execution of the same strategies or the discovery of entirely new reasoning capabilities.

The ARC-AGI result uses Gemini 3 Flash as both the proposer (optimizing the agent architecture) and the underlying agent model (executing the optimized architecture). This is a self-optimization loop — the same model is improving itself — which raises questions about whether the discovered architecture would transfer to other models. The agent skills experiment (Section 5.1) demonstrates cross-model transfer for prompts, but no analogous experiment is reported for the ARC-AGI architecture.

The comparison is against the seed (a naive single-call agent), not against a domain-specific agent architecture search system (ADAS [11], AFlow [29]). A head-to-head comparison under matched conditions would test whether the claimed unification actually outperforms specialized tools in this domain.

Claim 6: "Outperforms AlphaEvolve's reported circle packing solution (n=26)."

What was demonstrated: This is the strongest comparison in the paper — a controlled rerun of OpenEvolve (the open-source AlphaEvolve reimplementation) with the same proposer LLM (GPT-5.1). optimize_anything achieves 2.63598 in 63 evaluations vs. OpenEvolve's 2.4583 at 100 and 2.6307 at 200 (Table 3). The cost accounting is favorable: $3.18 vs.$6.85.

Caveats: The paper reports one run. Circle packing with evolutionary methods has stochastic outcomes — the comparison would be stronger with multiple runs and confidence intervals. The OpenEvolve rerun uses 100 and 200 evaluation budgets, while optimize_anything uses 63; the comparison is fair (optimize_anything achieves a better score with fewer evaluations) but the budget asymmetry makes it harder to interpret the per-evaluation efficiency difference.

The claim "outperforms AlphaEvolve's reported solution" compares against a published number that may have been generated with a different LLM, different hardware, and different hyperparameters. The controlled OpenEvolve rerun partially addresses this, but it's OpenEvolve (a reimplementation), not AlphaEvolve itself. The original AlphaEvolve paper used Gemini models, not GPT-5.1, so the "outperforms AlphaEvolve" claim conflates algorithm quality with proposer LLM quality.

Cross-cutting weaknesses

No statistical rigor. All results are point estimates from single optimization runs. There are no confidence intervals, no multiple random seeds, no significance tests. For domains with inherent stochasticity (circle packing, CUDA kernel generation, agent architecture search), this makes it impossible to assess whether the reported improvements are reliable or within noise.

Baseline strength varies. The strongest baselines are in circle packing (controlled OpenEvolve rerun) and cloud scheduling (ADRS leaderboard). The weakest baselines are in ARC-AGI (only compared to seed, not to ADAS/AFlow), agent skills (no comparison to other skill optimization methods), and image generation (human evaluation but no comparison to other optimization tools). The CUDA kernel baseline is PyTorch, which is a performance target rather than an optimization method comparison — useful for establishing absolute quality but silent on whether a specialized CUDA optimization tool would do better.

Single optimization backend. The paper is "backend-agnostic" but all experiments use the GEPA-based Pareto search backend. There is no comparison of different backends on the same domain, so the contribution of the Pareto search vs. other search strategies (MAP-Elites as in AlphaEvolve, novelty search as in ShinkaEvolve, simple best-of-N) is unknown. The paper argues that the interface is the contribution, not the algorithm, but this claim would be stronger with evidence that multiple backends work through the same interface.

Missing experiments. Several experiments would strengthen the paper: (1) a systematic cross-domain SI quality ablation to establish when SI helps vs. when it's neutral or harmful; (2) comparison of multi-task vs. generalization mode on the same dataset to establish mode selection criteria; (3) ARC-AGI comparison against ADAS or AFlow under matched proposer LLM; (4) a full 31-problem KernelBench multi-task vs. single-task comparison; (5) multiple random seeds for stochastic domains; (6) an ablation testing whether the refiner mechanism matters (code domains without refiner vs. with refiner) — the paper mentions it as a necessary modification but doesn't ablate its contribution.

Overall Assessment

The experiments convincingly demonstrate that a single optimization interface can work across fundamentally different text artifacts — this is the paper's core claim and the evidence is strong. The multi-task mode is a genuine capability advance with clear evidence of cross-transfer benefits on related problems and clear failure conditions on unrelated problems. The SI ablation provides credible evidence that diagnostic feedback matters, though the claim of "4–6×" is established on a subset of domains and conflates multiple SI aspects.

The paper's weaker claims are those requiring head-to-head superiority over domain-specific tools. Only circle packing has a controlled head-to-head comparison; other domains rely on leaderboard comparisons, seed-based baselines, or performance targets rather than optimization-method comparisons. The paper's true contribution is unification and generality, not necessarily algorithmic superiority — and the experiments are structured to demonstrate unification (same API across six domains) more than superiority (few controlled comparisons against the best domain-specific tool). Readers should interpret "matches or surpasses domain-specific tools" as "achieves results competitive with the best reported results in each domain using a single unified system," not "systematically outperforms the best tool in each domain under controlled conditions."

6. Limitations and Trade-offs

6.1 The Evaluator Must Be Crafted by a Domain Expert, and SI Quality Dictates Optimization Success

The assumption or constraint. The paper's central abstraction — that the user provides an evaluator returning (score, side_info) and the system handles everything else — relies on the evaluator being well-designed by a domain expert who knows what diagnostics to surface. Section 8 acknowledges this explicitly:

"designing effective SI still requires domain expertise; while evaluators returning only a score work, the demonstrated gains come from expert-designed SI (compiler errors, profiler traces, VLM scoring rubrics). That said, optimize_anything trades optimization expertise for domain expertise. The user, most often a domain expert, need not configure backends, tune algorithmic hyperparameters, or engineer prompting strategies, only surface the diagnostics they already understand."

The paper frames this as a favorable trade — domain experts already understand their diagnostics, so surfacing them is easy. But this assumes the domain expert knows which diagnostics are most informative for an LLM trying to improve the artifact, and that those diagnostics are actually available from the evaluation environment.

The consequence. SI quality is not just a nice-to-have — the ablation in Section 5.9 and Table 4 shows it is the difference between success and failure. On CUDA kernels with multi-task search, score-only feedback achieves 0% of kernels exceeding 1.1× speedup, while SI enables 40%. On circle packing, score-only reaches only 93.96% of the best score. The gap is not marginal — it is the difference between the system working and not working. This means that a domain expert who cannot articulate rich diagnostics cannot use the system effectively, even if they understand their domain deeply. The system does not discover what diagnostics to surface; it relies entirely on the user to provide them.

Furthermore, the paper does not characterize what happens when SI is misleading rather than merely absent. If the evaluator surfaces a compiler error that points to the wrong line, or a VLM critique that misunderstands the image, does the proposer get derailed? The ablation only tests SI vs. no SI — it does not test the quality-sensitivity of SI, so we do not know whether noisy or incorrect SI can degrade performance below score-only baseline. In domains where ground-truth diagnostics are unavailable (e.g., aesthetic quality of images, where the evaluator is itself a VLM with uncalibrated judgments), the SI may encode the evaluator's biases rather than actionable improvement directions.

What evidence exists in the paper. The SI ablation (Section 5.9, Figure 9, Table 4) demonstrates that SI is critical for performance, but only across three domains (prompt optimization, circle packing, CUDA kernels). There is no SI ablation for ARC-AGI, cloud scheduling, agent skills, or image generation — all domains where the SI surface is different (per-puzzle traces, scenario-level cost breakdowns, task descriptions and agent traces, VLM aesthetic scores). The cross-domain SI ablation (Table 4) shows the largest SI benefit in multi-task CUDA (40% vs. 0% at 1.1×), a moderate benefit in single-task CUDA (32.3% vs. 12.9%), and a smaller benefit in circle packing (100% vs. 93.96%). This variation suggests SI impact is domain-dependent, but the paper does not characterize which evaluator properties drive this variation.

Mitigation status. The paper does not attempt to address this limitation. It argues that the trade (optimization expertise for domain expertise) is favorable and that domain experts "need only surface the diagnostics they already understand" (Section 8). But it provides no guidance on how to design effective SI, no characterization of what makes SI "actionable" vs. "noisy," and no sensitivity analysis to SI quality. The capture_stdio=True option lowers the friction of surfacing diagnostics, but it does not help a user decide what to print. The paper suggests no future work on automatic SI discovery or SI quality assessment.

6.2 The System Cannot Optimize Non-Text Artifacts, and the Text-to-Artifact Translation Is a Bottleneck

The assumption or constraint. The entire framework rests on the artifact being representable as text. Section 8 states:

"The system assumes the artifact is representable as text; optimization of continuous parameters or binary artifacts requires a text-based proxy."

This assumption is baked into the API contract — the seed_candidate is a string, the evaluator receives a string, and the proposer generates a string. Anything that is not natively representable as text (continuous weight vectors, binary executables, hardware configurations) must be serialized into a text format that an LLM can both understand and generate. The paper's domains were chosen specifically because they admit natural text representations — Python code, SVG markup, natural-language prompts, CAD model scripts. But many optimization problems do not: hyperparameter tuning over continuous spaces, circuit design, molecular geometry, or any domain where the natural representation is a tensor or graph rather than a string.

The consequence. When the artifact requires a text proxy, the optimization becomes doubly indirect. First, the LLM must generate valid text in the proxy format (e.g., a Python script that sets hyperparameters and runs training). Second, the evaluator must parse and execute this text representation. This introduces two failure modes: syntactic failures where the LLM generates malformed proxy text (partially addressed by the refiner, but the refiner only catches simple errors), and representational failures where the text proxy is a poor encoding of the actual artifact space (e.g., an LLM generating code that describes a neural architecture is less efficient than directly searching over architecture parameters).

The paper does not explore this limitation because all six main domains have natural text representations. The appendix includes blackbox mathematical optimization (Appendix B), where the artifact is solver code rather than the continuous parameters being optimized — the system optimizes the algorithm that searches for parameters, not the parameters themselves. This is a clever workaround, but it only works when the parameter optimization algorithm itself can be expressed as text code. For problems where even the search algorithm is not naturally textual, the system provides no path forward.

What evidence exists in the paper. The limitation is acknowledged explicitly in Section 8, but no experiment probes the boundary. The six main domains all use text-native artifacts (code, prompts, agent definitions, SVG markup, scheduling policies). The 3D CAD domain (Appendix C) demonstrates that some non-text domains can be addressed by text-to-render pipelines (build123d Python scripts), but this is still code. There is no experiment with an artifact that is fundamentally non-textual and must be approximated through a text proxy, so we do not know how much optimization efficiency is lost in the translation.

Mitigation status. The paper acknowledges the limitation explicitly (Section 8) and does not claim to address it. It suggests no mitigation beyond the implicit assumption that "text-based proxy" is sufficient for many problems. Future work on direct multimodal artifact representations (e.g., the proposer outputting weight vectors or structured configurations) is not discussed. The refiner mechanism partially addresses syntactic failures in code artifacts, but representational failures (the proxy format is clumsy) are unaddressed and likely fundamental for certain problem classes.

6.3 Optimization Cost Is Dominated by Evaluator Expense, and the Paper Provides No Guidance on When the System Is Cost-Effective

The assumption or constraint. The paper reports total optimization costs without contextualizing them against the value of the improvements. Table 9 shows costs ranging from $1 (Numerical Blackbox) to$144.70 (ARC-AGI), with evaluator cost consistently dominating proposer cost. For ARC-AGI, the $144 evaluator cost (running the agent on 200 puzzles) dwarfs the$0.70 proposer cost. For CUDA kernels, $4.51 per kernel across 31 kernels totals$140. For cloud scheduling, the cost is $52.42. These are non-trivial absolute costs for a single optimization run, and the paper provides no framework for deciding whether the resulting improvement justifies the expenditure.

The consequence. A practitioner reading this paper cannot answer: "For my domain, should I use optimize_anything or should I spend the same $50–150 on a human expert improving the artifact manually?" The paper quantifies improvements in domain-specific metrics (accuracy percentage points, speedup ratios, cost savings percentages) but does not translate these into a common currency or provide cost-effectiveness comparisons against alternatives. The cost-accuracy tradeoff curves that would enable this decision are absent.

For domains where the evaluator is cheap (prompt optimization: the evaluator is an LLM call costing fractions of a cent), the total optimization cost is modest ($6.44 for AIME) and the system is clearly cost-effective. For domains where the evaluator is expensive (ARC-AGI, cloud scheduling, CUDA kernels on real hardware), the optimization cost is significant, and the paper does not characterize how the benefit scales with budget — would spending$300 instead of $144 on ARC-AGI yield another 5 percentage points? The optimization trajectories (Figures 3, 4, 5) show improvement as a function of iterations, not as a function of dollars, making cost-effectiveness analysis impossible from the reported data.

Furthermore, the paper does not compare against the cost of a human expert. A human engineer might spend 2–4 hours designing an ARC-AGI agent architecture at a cost comparable to or exceeding $144. But they might produce a better architecture, or one that transfers more robustly across models. The paper's silence on this comparison leaves practitioners without a decision framework.

What evidence exists in the paper. Table 9 provides total costs per domain. The proposer sensitivity analysis (Table 8) shows a cost-performance tradeoff between GPT-5.1 and GPT-5-nano. But neither table connects cost to improvement magnitude in a way that enables cost-effectiveness decisions. There is no cost-benefit analysis, no cost-per-percentage-point metric, and no comparison against human expert cost. The optimization trajectories (Figures 3a, 3b, 4, 5) use iterations/rollouts as the x-axis, not dollars, obscuring the financial scaling behavior.

Mitigation status. The paper does not address cost-effectiveness analysis. It reports costs as factual data points (Table 9) and notes that "reflection cost is minimal; total spend is dominated by the evaluator" (Section 5.10), which is an observation, not a mitigation. The paper provides sample-efficiency evidence (e.g., 63 evaluations for circle packing vs. OpenEvolve's 200) but does not frame this as a cost argument. A future practitioner would need to estimate their own evaluator cost per call and multiply by the expected number of evaluations to budget an optimization run, but the paper provides no guidance on how many evaluations to expect for a new domain.

6.4 Multi-Task Search Hurts on Unrelated Problems, and the Paper Provides No Criterion for Predicting Relatedness

The assumption or constraint. Multi-task search — the paper's flagship innovation — assumes that the problems in the dataset share transferable optimization patterns. Section 7 identifies the boundary condition:

"Multi-task search helps when tasks share underlying patterns (e.g., CUDA kernels on the same hardware) and hurts when they are fundamentally independent."

The paper demonstrates this boundary with a clear negative result: on circle packing with different N values, multi-task search degrades performance (Table 5: single-task 2.6360, MT7 2.6313, MT11 2.5973). But it provides no method for determining, before running optimization, whether a given set of problems is "related enough" for multi-task to help.

The consequence. A practitioner with a collection of problems faces an unresolvable decision: do I use multi-task mode (and risk degradation if the problems are insufficiently related) or single-task mode (and forgo potential cross-transfer benefits)? The paper's only guidance is the qualitative principle "tasks sharing underlying patterns," but underlying pattern similarity is not observable before optimization — it is precisely what the optimization process reveals. This creates a circular dependency: to know whether multi-task search will help, you need to understand what optimization patterns the problems share, but to discover those patterns, you need to run optimization.

The negative result on circle packing is stark: MT11 achieves a score of 2.5973, which is worse than the seed's eventual performance in single-task mode (2.6360) — meaning multi-task search can actually produce artifacts worse than what single-task would discover, not just fail to improve. A practitioner who incorrectly chooses multi-task mode incurs not just wasted compute but a genuinely worse final artifact. The paper provides no diagnostic that would have predicted this failure in advance.

The scaling results (MT20 > MT10 > single-task for CUDA kernels, Tables 6–7) suggest that adding more related tasks monotonically improves multi-task performance. But this doesn't help with the relatedness decision — if the tasks are unrelated, adding more of them would likely worsen performance further (as MT11 is worse than MT7 for circle packing).

What evidence exists in the paper. Table 5 provides the negative result on circle packing. Section 7 provides the qualitative diagnosis: "Circle packing problems for different N are fundamentally independent, optimal configurations change unpredictably with N, with no transferable structure." The CUDA kernel experiments (Figure 8, Tables 6–7) demonstrate the positive case. But there is no intermediate case — no experiment where problems are partially related, and multi-task search helps marginally or is neutral. The paper only explores the two extremes (clearly related CUDA kernels, clearly unrelated circle packing N values), providing no evidence about where the crossover occurs.

Mitigation status. The paper does not attempt to provide a relatedness criterion, a pre-optimization diagnostic, or a method for dynamically detecting whether multi-task transfer is occurring during optimization. Section 7 describes the boundary qualitatively but does not operationalize it. The suggestion in the paper is implicit: if you believe your problems share structure, try multi-task; if you're unsure, compare against single-task. But a comparison requires running both modes, doubling the optimization budget — precisely the cost that multi-task search was supposed to reduce. The paper suggests no future work on automatic relatedness detection or adaptive mode switching.

6.5 the System Inherits All Limitations of the Underlying Proposer LLM, Including Capability Ceilings and Model-Specific Biases

The assumption or constraint. The quality of every artifact improvement depends on the proposer LLM's ability to (1) understand the current artifact, (2) interpret the SI diagnostics, and (3) generate a genuinely improved artifact. If the proposer LLM cannot reason about the domain, the optimization stalls regardless of SI quality or search strategy. Section 8 states:

"The quality of proposals depends on the proposer LLM's capabilities; weaker models produce weaker candidates, as confirmed by our proposer sensitivity analysis (Table 8)."

The proposer sensitivity analysis (Table 8) shows that GPT-5-nano consistently underperforms GPT-5.1: on circle packing, 2.512 vs. 2.636; on AIME, 50.0% vs. 60.0%. But this only captures the difference between two GPT-5 variants. The deeper limitation is that all current LLMs have capability ceilings — domains requiring reasoning beyond the LLM's capacity are fundamentally unoptimizable by this approach, regardless of SI quality or compute budget.

The consequence. The system is bounded by the proposer LLM's domain knowledge and reasoning ability, and this bound is invisible to the user during optimization. If the proposer LLM doesn't understand GPU memory hierarchies well enough to suggest tiling strategies, or doesn't grasp the geometric constraints of circle packing well enough to propose LP formulations, the optimization will converge to whatever the LLM can produce — which may be far below the domain's achievable optimum. Worse, the LLM may confidently propose incorrect "improvements" that look plausible but are wrong, and the evaluator's SI may not be rich enough to catch the error.

This creates a diagnosability problem: when optimization plateaus, the user cannot distinguish between "the artifact has reached a genuine local optimum in the space of text artifacts" and "the proposer LLM has exhausted its domain knowledge and cannot generate further improvements." The trajectory analysis in Appendix F attributes convergence to genuine optimization dynamics (Pareto diversity, algorithmic shifts), but it cannot rule out that the proposer simply ran out of ideas. The paper provides no tool for diagnosing proposer capability saturation.

The model-specific nature of improvements is partially addressed by the agent skills experiment (Section 5.1), where skills optimized for one model transfer to another. But this is a single domain (coding agent prompts), and the transfer is explicitly between models in the same family (Claude Haiku 4.5 → Sonnet 4.5). It does not demonstrate that an ARC-AGI agent architecture optimized with Gemini 3 Flash as proposer would transfer to a GPT-5-based agent, or that a CUDA kernel optimization pattern discovered by GPT-5 would be reproducible by Claude. The proposer LLM's biases and knowledge gaps are baked into the optimized artifact in ways that may not be portable.

What evidence exists in the paper. Table 8 quantifies the performance gap between GPT-5.1 and GPT-5-nano, providing direct evidence of proposer sensitivity. But this comparison only spans two models from the same family. There is no comparison across model families (GPT vs. Gemini vs. Claude) on the same domain, so the paper provides no evidence about whether some models are systematically better at certain types of optimization or whether results are robust to proposer choice. The agent skills transfer experiment (Figure 2) suggests some robustness for prompts, but no analogous experiment exists for code or agent architectures.

Additionally, the paper uses different proposer LLMs for different domains (Table 1) — GPT-5 for CUDA/circle packing/AIME, Gemini 3 Flash for ARC-AGI, Gemini 3 Pro for cloud scheduling, Claude Opus 4.6 for agent skills/3D modeling. This domain-specific model selection is pragmatic but makes it impossible to attribute performance differences to the domain vs. the model. Does Gemini 3 Flash achieve 89.5% on ARC-AGI because the domain is well-suited to the approach, or because Gemini 3 Flash is a better proposer for agent architectures than GPT-5 would be? The paper cannot answer this because it never tests multiple proposers on the same domain (except the GPT-5.1 vs. GPT-5-nano comparison on AIME and circle packing).

Mitigation status. The paper acknowledges proposer sensitivity in Section 8 and quantifies it in Table 8, but does not attempt to mitigate it. It does not propose techniques for detecting proposer saturation, switching proposers mid-optimization, or combining multiple proposer models for robustness. The backend-agnostic design (Section 9) implies that as better LLMs become available, they can be swapped in, which is a future-facing mitigation. But this doesn't address the fundamental limitation: any given LLM has a capability ceiling, and the system cannot exceed it. The paper's language of "discovers agent architectures" and "finds scheduling algorithms" should be understood as "the proposer LLM, guided by SI, generates improved artifacts within the scope of its capabilities" — the system discovers what the LLM is capable of discovering, which may be less than what is discoverable in principle.

6.6 Only One Optimization Backend Is Evaluated, and the Contribution of Pareto Search vs. the Overall Framework Is Unexplored

The assumption or constraint. The paper presents optimize_anything as "backend-agnostic" (Section 4, Section 9) — the idea being that the API is a unified interface and various optimization algorithms can plug in. However, all experiments in the paper use a single backend: the GEPA-based Pareto search extended from Agrawal et al. [3]. The paper does not evaluate any alternative backend through the optimize_anything interface, and it does not ablate the specific algorithmic choices that distinguish its Pareto search from simpler alternatives.

The consequence. The paper's claimed contribution is the unifying interface, not a specific algorithm. But because only one algorithm is tested, the reader cannot separate three confounded factors: (a) the benefit of the unified API abstraction, (b) the benefit of Pareto-based search over simpler strategies (best-of-N, random mutation with selection, MAP-Elites as in AlphaEvolve), and (c) the benefit of SI-driven reflection over alternative proposer prompting strategies. The paper's results might depend heavily on the specific GEPA algorithm, in which case the "backend-agnostic" claim is aspirational, or the results might be robust to backend choice, in which case the interface is the true contribution. We cannot know which is true because no comparison between backends exists.

Specific algorithmic choices that are unexplored in ablation:

• Pareto-based selection vs. selecting the top-k by average score. The paper argues that Pareto diversity prevents premature convergence (Section 6, Appendix F, Mechanism 3) and enables multi-task transfer (Section 5.8). But there is no ablation comparing Pareto selection against a simple top-k baseline. Would top-5 by average score perform similarly on single-task domains? Does Pareto selection matter more for multi-task than for single-task? Unknown.

• Minibatch reflection size (b = 2–3). The paper argues that small minibatches enable focused, targeted improvements (Section 4.3), but does not ablate the minibatch size. Would b = 1 work better (ultra-focused) or worse (too narrow, overfits to single example)? Would b = 10 work better (more context) or worse (diffuse feedback)? Unknown.

• Frontier frequency proportional sampling vs. uniform sampling from the frontier. The paper samples candidates with probability proportional to how many objectives they are best on (Section 4.3). Would uniform sampling from the frontier work equally well? If frontier frequency sampling is critical, the paper should demonstrate this; if it's incidental, the algorithm is overcomplicated.

• The refiner mechanism. The refiner catches syntax errors and malformed code blocks before evaluation. The paper describes it as a "necessary" modification for code artifacts (Section 4, item 2), but never ablates it. What fraction of candidates would fail without the refiner? Does the refiner ever "fix" a candidate incorrectly, producing a valid but semantically wrong artifact that wastes evaluation budget? Unknown.

What evidence exists in the paper. None. There is no backend comparison experiment and no algorithmic ablation beyond the SI and multi-task analyses. The controlled comparison against OpenEvolve on circle packing (Table 3) is the closest the paper comes to comparing algorithms, but OpenEvolve is a different framework with a different interface, using MAP-Elites rather than Pareto search. This comparison tests the overall system against a competitor, not the specific algorithmic choices within the Pareto backend.

The paper's own framing — "the API is backend-agnostic" — implies that multiple backends exist, but only one is used. This is fine for a systems paper introducing a new interface, but it means the performance claims are inextricably tied to the GEPA algorithm. A practitioner who uses the optimize_anything interface but plugs in a different backend (e.g., simple random mutation with reflection) has no evidence about whether their results will be comparable.

Mitigation status. The paper does not address this limitation. It describes the backend adapter layer in Section 4 (item 5) as enabling future backend plugins, but provides no experimental evidence that the interface works with multiple backends. The paper suggests in Section 9 that "as new optimization strategies emerge, they plug in without changing user code," which is a forward-looking claim untested in the current work. A stronger paper would have implemented at least one alternative backend (e.g., MAP-Elites or simple greedy search) through the same interface and compared performance on a subset of domains, establishing that the interface genuinely abstracts over algorithmic choices. Without this, the "backend-agnostic" claim is a design principle rather than a demonstrated property.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new optimization algorithm. It makes a more fundamental move: it changes what we think LLM-based optimization is a problem of. Before this work, the field implicitly treated code optimization, prompt optimization, and agent architecture search as distinct subproblems requiring distinct frameworks — each with its own API, its own configuration abstractions, and its own assumptions about what types of feedback are available. The evidence for this fragmentation is in Table 2: no prior system supports all artifact types, all optimization modes, or diagnostic feedback as a first-class contract. Each system was designed for exactly one domain.

optimize_anything's reframing — that all of these are instances of a single problem: generate a text artifact, evaluate it, feed the score and diagnostics back to an LLM, repeat — is obvious in retrospect but was not the operating assumption of the field. The paper's contribution is to demonstrate that this reframing is actionable: a single system, with a single API, achieves competitive or state-of-the-art results across six fundamentally different domains without per-domain algorithmic modifications. The evidence is not that optimize_anything invents better algorithms than AlphaEvolve or GEPA, but that the same Pareto-based search, the same reflection mechanism, and the same SI contract work across code, prompts, agents, images, and scheduling policies — domains that previously required entirely separate tools.

This is a reframing, not a paradigm shift. The underlying algorithmic ideas (evolutionary search with LLM-based mutation, diagnostic-driven reflection) existed in prior work. What is new is the abstraction boundary that makes these ideas portable. The paper effectively argues that the field has been drawing the wrong boundary — between code and prompts and agents — when the right boundary is between the artifact text and its evaluator. The practical consequence is that future optimization algorithms can target the (artifact, evaluator) -> improved artifact interface and automatically work across domains, rather than requiring per-domain reimplementation. This is the same dynamic that DSPy introduced for prompt engineering: define a declarative abstraction, prove it works across tasks, and let the ecosystem build on it.

The paper also reconciles a latent tension in how LLM-based optimization systems handle feedback. Prior work split into two camps: evolutionary systems (AlphaEvolve, FunSearch) that used execution results as raw feedback but in framework-specific, code-only ways, and prompt optimization systems (GEPA, TextGrad) that used structured scores or LLM-generated "gradients" but couldn't consume compiler errors or profiler traces. The SI contract resolves this by treating all diagnostic feedback uniformly — dictionaries of arbitrary types — and pushing the interpretation burden onto the proposer LLM rather than the framework. This is a conceptual contribution, not an algorithmic one, but it changes the design philosophy for future systems: invest in evaluator quality and diagnostic richness, not in framework-internal feedback formats.

The paper also redirects research attention from verifier training to diagnostic engineering. In the RLHF/PRM paradigm, optimization quality depends on learned verifier quality, and the primary research challenge is building verifiers that remain calibrated under optimization pressure. optimize_anything's approach — use the evaluator as ground truth, not a learned proxy, and augment it with diagnostic SI — suggests a different investment: make the evaluator more informative (richer diagnostics, multimodal outputs) rather than making a learned verifier more robust. This is a shift in where research effort should go, and the SI ablation (Table 4: 4–6× convergence acceleration from SI vs. score-only) provides concrete evidence that diagnostic quality is a high-leverage investment.

The multi-task search mode introduces something genuinely new to the landscape: cross-problem transfer through shared evolutionary populations. Prior evolutionary systems (AlphaEvolve, FunSearch) operated on one problem at a time. The demonstration that sharing a Pareto frontier across related CUDA kernels accelerates convergence and improves final quality (Figure 8, Tables 6–7) — and that this benefit scales with the number of related tasks — establishes multi-task evolutionary search as a viable approach for problems that come in batches with shared structure. This is not a minor feature; it's a new capability that no prior framework supported.

However, the paper's negative result on circle packing (Table 5) establishes that this capability has a sharp boundary: multi-task search degrades performance when tasks lack transferable structure. This is equally important because it prevents over-claiming — multi-task search is not universally beneficial, and its failure condition is identifiable (task independence). The paper thus provides both a new capability and a partial characterization of its applicability condition, which is the right level of maturity for a first demonstration.

The practical impact on the field depends on adoption of the interface. If optimize_anything becomes a standard API that optimization algorithms target — analogous to how the transformers library standardized model interfaces — then the paper's reframing will have been genuinely consequential. If the system remains one of several frameworks, each with its own interface, the conceptual contribution will have been prescient but not transformative. The paper's open-source release and the explicit "backend-agnostic" design (Section 9) suggest the authors intend the former.

Follow-Up Research This Work Enables

Systematic characterization of SI quality and its impact on optimization efficiency. The SI ablation (Section 5.9) establishes that diagnostic feedback matters, but it compares only two conditions: rich SI vs. scalar-only. A deeper investigation would systematically degrade SI quality — removing error messages but keeping sub-scores, adding noise to compiler error line numbers, replacing VLM critiques with random text — and measure the impact on convergence speed and final artifact quality. This would establish what properties of SI (accuracy, specificity, actionability) drive the benefit, and whether there is a point where noisy SI becomes worse than no SI. The experiment would use 2–3 domains from the paper (circle packing, CUDA kernels, prompt optimization) with controlled SI degradation across 5–10 noise levels and multiple random seeds, producing curves that map SI quality to optimization efficiency. The paper's current "SI vs. no SI" design leaves the middle of this curve completely unexplored.

Cross-model proposer comparison on identical domains to separate algorithm from proposer capability. The paper uses different proposer LLMs for different domains (Table 1: GPT-5 for CUDA/circle packing, Gemini 3 Flash for ARC-AGI, Claude Opus 4.6 for agent skills). This makes it impossible to attribute performance differences to domain difficulty vs. proposer capability. A systematic study would run optimize_anything on 3–4 domains (ARC-AGI, circle packing, AIME, one CUDA kernel) with 3–4 different proposer LLMs (GPT-5, Gemini 3 Flash, Claude Opus 4.6, and a weaker model like Claude Haiku) under matched budgets, measuring both final artifact quality and convergence speed. This would reveal whether certain proposers are systematically better at certain types of optimization (e.g., Gemini better for agent architectures, GPT better for mathematical code), and whether the optimize_anything interface genuinely abstracts over proposer choice or if results are highly proposer-dependent. The proposer sensitivity analysis (Table 8) only compares two GPT-5 variants, which is too narrow to answer this question.

Exploring the multi-task/single-task decision boundary with a continuum of task relatedness. The paper demonstrates that multi-task search helps on clearly related tasks (CUDA kernels on the same hardware) and hurts on clearly unrelated tasks (circle packing with different N). What happens in between? A controlled experiment would construct a family of tasks with tunable relatedness — for example, optimizing prompt formats for N different NLP tasks where task similarity is measured by embedding distance or performance correlation — and measure multi-task benefit as a function of similarity. The experiment would sweep across similarity levels (from nearly identical tasks to completely unrelated tasks) and measure whether the multi-task benefit degrades smoothly or drops off at a threshold. This would operationalize the paper's qualitative principle ("helps when tasks share underlying patterns") into a quantitative decision rule that a practitioner could apply before choosing optimization mode. The negative result on circle packing (Table 5) currently serves as the only counterexample; a continuum experiment would map the full curve.

Ablation of Pareto frontier selection against simpler baselines to isolate the contribution of diversity preservation. The paper attributes several benefits to Pareto-based selection: cross-task transfer in multi-task mode (Section 5.8), prevention of premature convergence (Section 6, Mechanism 3), and multi-module leapfrogging (Section 6, Mechanism 2). But there is no ablation comparing Pareto selection against alternatives. A controlled experiment would run optimize_anything on 3–4 domains (circle packing, AIME prompts, CUDA kernels) with the Pareto frontier mechanism replaced by: (1) top-K by average score (K matched to frontier size), (2) uniform random selection from the candidate pool, (3) epsilon-greedy (select best 90% of the time, random 10%), and (4) MAP-Elites with grid-based diversity (as in AlphaEvolve). Each variant would use the same proposer LLM and SI quality, isolating the selection mechanism's contribution. This would test the paper's core claim that Pareto diversity matters — if top-K by average score performs identically, then the algorithmic contribution of Pareto search is minimal, and the interface is indeed the main contribution.

Generalization of the multi-task transfer mechanism to domains beyond CUDA kernels. The paper demonstrates cross-task transfer on CUDA kernel generation (Figure 8) but does not test it on other multi-task-capable domains. Does multi-task search accelerate convergence on solving multiple AIME problems (different problems, shared mathematical reasoning patterns)? On optimizing prompts for multiple NLP tasks simultaneously (different task formats, shared prompt engineering patterns)? On image generation for multiple visual goals (different compositions, shared rendering techniques)? Each of these would test whether the transfer mechanism generalizes beyond the specific domain where it was demonstrated. The AIME multi-task experiment would be particularly informative: mathematical reasoning problems share patterns (case analysis, algebraic manipulation, constraint satisfaction) but differ in specifics, analogous to CUDA kernels sharing optimization patterns but differing in operations. If multi-task search accelerates AIME prompt optimization, it suggests the mechanism is broadly applicable; if it doesn't, it suggests CUDA's regular structure (same hardware, similar operations) is critical.

Dynamic mode switching: can the system detect during optimization whether multi-task transfer is occurring and fall back to single-task if not? The paper's negative result on circle packing (Table 5) shows that choosing the wrong mode produces worse artifacts than single-task optimization. A practical system would monitor the overlap in Pareto frontier membership across tasks during optimization — if candidates that excel on task A consistently underperform on task B (no shared Pareto champions), the tasks are likely unrelated and multi-task mode is degrading performance. This would enable adaptive mode switching: start in multi-task mode, periodically assess whether cross-transfer is occurring (using a statistical test on frontier overlap), and if not, fork into independent single-task runs. The experiment would test this adaptive strategy against pure multi-task and pure single-task on the circle packing domain (where multi-task is known to hurt) and on CUDA kernels (where it helps), demonstrating that the adaptive strategy automatically selects the right mode without prior knowledge of task relatedness. This addresses the current limitation that practitioners must guess whether their tasks are "related enough."

Applying the SI contract to domains with learned verifiers rather than ground-truth evaluators. The paper's approach relies on the evaluator being the ground truth — the optimization signal is the actual performance metric. This is feasible for code (execution), prompts (LLM accuracy on known answers), and images (VLM scoring), but many important domains lack clean ground-truth evaluators: dialogue quality, creative writing, complex planning, or any task where "correctness" is ambiguous or multi-dimensional. A natural extension would train a learned verifier (similar to the PRM in the inference-time compute literature) that returns both a score and diagnostic SI (explaining why a response is suboptimal), and test whether SI from a learned verifier provides similar convergence acceleration as SI from a ground-truth evaluator. The experiment would compare optimize_anything with: (1) ground-truth evaluator + SI, (2) learned verifier (trained on human preference data) + no SI, and (3) learned verifier + SI (where the verifier generates natural-language diagnostics). This would test whether the SI benefit survives when the evaluator itself is fallible — a critical boundary condition that the paper does not explore.

Practical Applications and Downstream Use Cases

Batch optimization of infrastructure code in cloud services. The cloud scheduling results (Section 5.2) translate directly to production infrastructure: CloudCast achieves 40.2% cost savings on multi-cloud data transfer, and Can't Be Late achieves 7.8% cost savings on spot instance scheduling, both topping the ADRS leaderboard. For a cloud provider or large cloud consumer, deploying optimize_anything to periodically re-optimize scheduling policies as infrastructure evolves would directly reduce operational costs. The cost of optimization ($52.42, Table 9) is negligible compared to the cost savings over months of operation. The system's sample efficiency (63–100 evaluations per policy) means re-optimization can run as a daily or weekly batch job without significant compute expenditure.

Automated prompt engineering for LLM-powered products. The AIME prompt optimization result (Section 5.4: 46.67% → 60.00% with GPT-4.1-mini, costing $6.44) demonstrates that optimize_anything can replace manual prompt engineering for mathematical reasoning. For any LLM-powered product where prompt quality directly impacts user experience — tutoring systems, code generation assistants, customer support bots — the same pipeline (provide a dataset of representative tasks, an evaluator that scores LLM outputs against expected answers, and optional SI like error categorization) can automatically discover prompts that outperform hand-crafted ones. The cost is dominated by evaluator calls ($4.27 of the $6.44 total), which are LLM API calls themselves — meaning prompt optimization costs roughly the same as a small batch evaluation run. For a product already spending thousands of dollars per month on LLM inference, a one-time$6–50 prompt optimization cost is a negligible investment for measurable accuracy improvements.

On-device agent architecture discovery for edge deployment. The ARC-AGI agent architecture result (Section 5.3: 32.5% → 89.5%) demonstrates that the system can discover multi-stage pipelines with verify-then-fallback logic that are qualitatively more sophisticated than what a human engineer would design in a reasonable timeframe. For edge deployment scenarios where model size is constrained (no datacenter-scale LLMs available), discovering specialized agent architectures that make a small model competitive on specific tasks is a high-value application. The $144.70 optimization cost (Table 9) is dominated by agent evaluation ($144), not proposer calls ($0.70), meaning the cost scales with the complexity of the evaluation benchmark rather than the optimization algorithm. For a team deploying an on-device agent for a specific task (medical coding, field service diagnostics, language tutoring), spending$100–200 on automated architecture search is vastly cheaper than the engineering time required to manually design and iterate on agent architectures.

Training data generation pipelines for self-improving systems. The multi-task CUDA kernel results (Section 5.5: 87% of kernels match or beat PyTorch) suggest a concrete pipeline for generating specialized high-performance implementations: maintain a library of reference operations, run optimize_anything in multi-task mode to generate optimized kernels for new hardware targets or new operations, and feed the best discovered kernels back as training data for future optimization runs. The cross-transfer mechanism means each new kernel added to the library benefits from patterns discovered for previous kernels, creating a positive-feedback loop where the system gets more efficient as the library grows. The MT20 > MT10 > single-task scaling result (Tables 6–7) directly supports this: larger batches of related tasks produce better per-task results, so a growing library of kernels would continuously improve the optimization quality for new additions.

When to Prefer This Method

The paper's positioning against alternatives is explicit, and a decision rule emerges from the experiments and limitations. Prefer optimize_anything (or its underlying paradigm) when:

• The artifact is representable as text (code, prompts, configurations, policies, markup), and you have a reliable evaluator that returns both a score and rich diagnostic feedback. The SI ablation (Table 4) shows that score-only feedback produces dramatically worse results, so domains where you cannot surface actionable diagnostics (compiler errors, test failures, profiler traces, VLM critiques) should not use this approach.

• You have a batch of related optimization problems that share underlying structure. The multi-task mode (Figure 8, Tables 6–7) provides cross-transfer benefits that scale with the number of related tasks, making this approach strictly better than single-task optimization for batched problems like kernel libraries, prompt collections, or scheduling policy families. If the problems are clearly unrelated (Table 5), use single-task mode instead.

• You want to trade optimization expertise for domain expertise. The system eliminates the need to configure mutation prompts, island topologies, or search hyperparameters — tasks that require understanding the optimization framework's internals. Instead, the user invests effort in designing a high-quality evaluator with rich SI. For a domain expert who understands their problem's diagnostics but not evolutionary algorithms, this is a favorable trade.

• The proposer LLM is capable in the domain. The proposer sensitivity analysis (Table 8) shows that weaker LLMs produce weaker artifacts, though they still improve over the seed. If the domain requires reasoning beyond the proposer's capabilities, the system cannot compensate — it can only discover what the LLM is capable of discovering. The system works best when the proposer LLM has strong domain knowledge and can reason about the diagnostics the evaluator produces.

Prefer domain-specific tools (AlphaEvolve, GEPA, MIPROv2) when:

• You are optimizing in a single domain with a mature, well-tuned existing pipeline and the marginal benefit of switching frameworks does not justify the migration cost. The paper does not claim optimize_anything outperforms every domain-specific tool on every metric — it claims competitive results with a unified interface. In prompt optimization, for example, GEPA (which optimize_anything's backend extends) is already state-of-the-art, so migrating gains interface simplicity but not necessarily performance.

• You need fine-grained control over the optimization algorithm. The system's declarative interface abstracts away mutation strategies, population management, and selection mechanisms. If your domain requires custom algorithmic modifications (e.g., novelty-based rejection sampling as in ShinkaEvolve, or domain-specific genetic operators), the current backend-agnostic interface may be too opaque — though the paper intends future backends to plug in, this capability is not yet demonstrated.

Prefer scaling pretraining or using larger models when:

• The optimization problem cannot be represented as a text artifact (continuous parameter vectors, binary executables, non-textual representations) and the text proxy would be too lossy. The paper explicitly acknowledges this limitation (Section 8) and does not claim to address it.

• The evaluator is prohibitively expensive per call and you cannot amortize the cost. The paper's most expensive experiments ($144 for ARC-AGI, Table 9) are still modest by research standards, but for production domains where each evaluation costs thousands of dollars (large-scale simulation, physical experiments), the hundreds of evaluations required by evolutionary search may be infeasible. The paper's sample efficiency is good relative to other evolutionary methods (63 evaluations for circle packing vs. OpenEvolve's 200), but it still requires non-trivial evaluation budgets.