CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery

🎯 Pitch

Giving LLM agents control over their own search decisions, rather than hard-coding evolutionary rules, yields 3–10× higher improvement rates and 20% better kernel optimization results. Four autonomous agents collaborating through shared memory discover a faster GPU kernel (1103 cycles) than any prior system, demonstrating that multi-agent co-evolution measurably extends the frontier beyond what independent agents can achieve.

1. Executive Summary

This paper introduces CORAL, the first framework for autonomous multi-agent evolution on open-ended discovery problems, evaluated on 11 mathematical, algorithmic, and systems optimization tasks — including Anthropic's kernel engineering and Polyominoes packing — using Claude Opus 4.6 and an open-source MiniMax M2.5 + OpenCode stack. CORAL replaces the fixed heuristics and hard-coded exploration rules of prior evolutionary search with long-running autonomous agents that control retrieval, proposal, evaluation scheduling, and knowledge accumulation through shared persistent memory (a structured file system of attempts, notes, and skills), asynchronous multi-agent execution, and heartbeat-based interventions (periodic reflection, consolidation, and stagnation-triggered redirection). A single autonomous agent achieves 3–10× higher improvement rates with up to 10× fewer evaluations than fixed evolutionary search baselines across all 11 tasks, while four co-evolving agents push the best known score on the kernel engineering task from 1,363 to 1,103 cycles — a 20% gain — establishing that both agent autonomy over search decisions and multi-agent co-evolution through shared memory causally extend the search frontier beyond what independent agents can achieve.

2. Context and Motivation

The Core Problem: Discovery Without Ground Truth

The paper addresses a fundamental class of problems that do not admit one-shot solutions: open-ended discovery tasks where the objective is clear but the optimal solution is unknown. The authors crystallize this in their opening sentence:

"Many important scientific problems do not come with ground-truth answers. What is the best heuristic for a logistics problem? How should one write the most efficient kernel?"

These are not problems where an LLM can be prompted once and expected to produce the correct answer — they require iterative proposal, testing, revision, and sustained progress over time. The objective function exists (e.g., minimize GPU kernel cycle count, maximize packing density), but there is no labeled dataset of correct answers to train on, and the solution space is vast and poorly structured.

This problem class spans practical domains that matter enormously for scientific and engineering progress: discovering novel mathematical algorithms (Romera-Paredes et al., 2024), optimizing systems-level code such as GPU kernels and database heuristics (Ouyang et al., 2025a; Chen et al., 2025), and solving combinatorial optimization problems like packing and scheduling (Mang et al., 2025). Each of these domains shares the same structure — an evaluator that can score candidate solutions, but no gradient, no ground truth, and no closed-form path to the optimum. Progress in automating discovery on these problems translates directly to faster scientific iteration, more efficient computing infrastructure, and reduced human engineering effort.

The Dominant Paradigm: Fixed Evolutionary Search

The current state-of-the-art approach to these problems, established by systems like FunSearch (Romera-Paredes et al., 2024) and AlphaEvolve (Novikov et al., 2025), embeds LLMs inside an outer-loop evolutionary search procedure. The LLM acts as a mutation operator: it proposes candidate programs conditioned on previously high-scoring solutions, an external evaluator executes and scores these candidates, and a predetermined evolutionary algorithm governs parent selection and population updates. This is what the paper calls fixed evolutionary search.

The paper decomposes each improvement step in such systems into four stages (Section 3.1):

RETRIEVE: construct a working context from the population of prior solutions
PROPOSE: generate a candidate solution conditioned on that context
EVALUATE: obtain a score and feedback from the evaluator
UPDATE: incorporate the new candidate into the population

In fixed evolutionary search, the RETRIEVE and UPDATE stages are governed by externally specified, hard-coded rules that are independent of the LLM. The LLM's role is confined almost entirely to PROPOSE — it receives a constructed prompt with selected parent solutions and generates a mutation in a single forward pass. It does not decide which parents to inspect, when to run intermediate tests, what knowledge to preserve for future reuse, or how to react to failure trajectories.

The paper argues that this division of labor leaves significant performance on the table:

"For challenging open-ended problems, these choices are integral to the evolutionary algorithm and can substantially affect performance."

The key insight here is that search meta-decisions — what to explore next, when to pivot, what to remember — are themselves part of the optimization problem. A fixed rule that works well on one problem class (e.g., selecting parents via MAP-Elites fitness-novelty tradeoffs) may be suboptimal on another. The LLM, which has access to the full history of attempts and their outcomes, possesses the reasoning capability to make context-sensitive decisions about these meta-choices, but in fixed evolutionary search it is never given the opportunity.

Why Fixed Search Falls Short: Four Specific Gaps

The paper identifies four concrete limitations of the fixed evolutionary search paradigm:

1. Retrieval is determined by predefined heuristics, not by the LLM's analysis of what information is relevant. In AlphaEvolve, working context M̂_t is constructed from the population M_t using predetermined selection rules inspired by MAP-Elites and island models (Novikov et al., 2025). The LLM never chooses which prior solutions to inspect — it receives whatever the algorithm selects. For open-ended problems where relevant information might be scattered across qualitatively different solutions (a kernel optimization insight from one attempt, a scheduling trick from another), this rigid retrieval can miss non-obvious connections that an agent actively browsing the solution history might discover.

2. The LLM has no agency over evaluation timing or intermediate testing. In fixed search, EVALUATE is triggered by the outer loop — the LLM proposes, the evaluator scores, and the cycle repeats. The LLM cannot decide to run local tests before submitting for formal evaluation, cannot validate intermediate results to catch bugs early, and cannot decide that a candidate is unpromising and should be abandoned before consuming an evaluation budget. This matters because evaluation calls are the scarce resource in open-ended discovery — each evaluation consumes wall-clock time and, in many real-world settings, actual computational or monetary cost (e.g., running a GPU kernel benchmark, simulating a physical system). As the results in Table 1 demonstrate, fixed evolutionary search methods waste the majority of their evaluation budget on unproductive candidates, achieving improvement rates of only 6–33% compared to CORAL's 30–100%.

3. Knowledge accumulation is limited to the population of solutions themselves, with no mechanism for abstracting reusable insights. Fixed evolutionary search maintains a population of candidate programs, and the LLM conditions on high-scoring individuals. But there is no persistent repository for observations about why something worked, notes documenting failed approaches, or reusable skills that can be transferred across attempts. The population encodes what solutions performed well, but not why they performed well or what general principles can be extracted. This means that each new proposal starts from scratch in interpreting the significance of prior solutions — the system does not accumulate interpretable knowledge over time.

4. The search strategy itself is static. Even in more adaptive systems like AdaEvolve (Cemri et al., 2026) and EvoX (Liu et al., 2026a), which make the search strategy itself subject to meta-evolution, the adaptation happens at the level of hyperparameters (mutation rates, selection pressure) rather than at the level of individual decisions. The LLM still follows a fixed pipeline: receive a prompt, generate a candidate, receive a score. It does not decide to pause and reflect, to switch approaches when progress stalls, or to consolidate learnings across multiple attempts. The paper's heartbeat mechanism (Section 3.3) directly addresses this gap by injecting structured reflection prompts when stagnation is detected — an intervention that has no analog in fixed evolutionary search.

The Multi-Agent Dimension: Vertical Scaling vs. Horizontal Exploration

The paper identifies a parallel limitation in how existing systems scale to multiple agents. Most multi-agent LLM systems for discovery rely on what the authors call vertical scaling: humans decompose the task, assign specialized roles to different agents, and define a fixed communication structure. Systems like Sakana AI's AI Scientist (Lu et al., 2024) and Google's AI Co-Scientist (Gottweis et al., 2025) exemplify this paradigm — they are powerful, but they assume that the optimal decomposition and interaction topology are known in advance.

For open-ended problems, the authors argue this assumption is restrictive:

"For open-ended problems, that assumption is restrictive. This raises a further question: Can multiple autonomous agents scale more effectively through horizontal parallelism, by exploring in parallel, exchanging discoveries, and building on each other's progress over time?"

In fixed evolutionary search systems like FunSearch and AlphaEvolve, parallelism is limited to running multiple stateless evaluation workers concurrently — there is no memory shared across workers, no cross-pollination of ideas, and no emergent coordination. Each worker is an independent instance of the same fixed pipeline. The paper's vision of horizontal scaling is fundamentally different: agents explore different regions of the solution space autonomously, exchange discoveries through shared persistent memory, and build on each other's progress without requiring a predefined communication protocol or role assignment.

The Autonomous Agent Literature: A Missing Bridge

A separate line of work has demonstrated that LLM agents can operate with substantial autonomy in open-ended environments. Autonomous coding agents like SWE-Agent (Yang et al., 2024) and OpenHands (Wang et al., 2024) navigate codebases, execute code, and iteratively debug within sandboxed environments. The AI Scientist (Lu et al., 2024) automates the full research cycle — generating ideas, running experiments, and writing papers. Self-improvement techniques such as Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023) enable agents to learn from their own mistakes through verbal self-feedback. Systems like ReAct (Yao et al., 2023) interleave reasoning and tool use, while MemAgent (Yu et al., 2026) and MEM1 (Zhou et al., 2026) develop learned memory consolidation for long-horizon tasks.

However, the authors note a critical gap:

"These systems demonstrate the power of agent autonomy, but they target one-off task completion rather than sustained, goal-driven optimisation."

Autonomous agents in prior work are designed to complete a specific task (fix a bug, answer a question, write a paper) and then stop. They do not engage in continuous, open-ended improvement where the goal is not to satisfy a completion criterion but to keep finding progressively better solutions over an extended horizon. The paper's contribution is to bridge these two traditions: bringing the autonomy of LLM agents into the evolutionary search loop, so that the same agent intelligence that can navigate a codebase and debug errors can also decide what to explore next, when to consolidate knowledge, and when to pivot strategies.

Recent position papers have argued for elevating deployment-time adaptation to an autonomous evolver agent (Gao et al., 2025), and concurrent open-source projects like AutoResearch (Karpathy, 2026) and Hive (rllm-org, 2026) are exploring similar directions. The paper positions CORAL as a systematic exploration and strong baseline for this emerging paradigm.

How CORAL Positions Itself

CORAL does not propose a new evolutionary algorithm, a new mutation operator, or a new verifier architecture. It proposes a paradigm shift in how the evolutionary search loop is organized. The paper's contribution is architectural and organizational: replacing fixed control flow with agent autonomy, replacing static population management with shared persistent memory, and replacing predefined multi-agent roles with emergent coordination through shared knowledge.

The paper frames this progression explicitly in Figure 1, which shows three paradigms:

Fixed Evolutionary Search: The LLM is confined to PROPOSE; all other decisions are externally specified.
Autonomous Single-Agent Evolution: A single agent controls RETRIEVE, PROPOSE, EVALUATE scheduling, and UPDATE, deciding for itself what to inspect, when to test, and what to remember.
Autonomous Multi-Agent Evolution: Multiple agents operate asynchronously, coordinating only through shared persistent memory — exploring in parallel, exchanging discoveries, and building on each other's work without predefined roles or communication protocols.

CORAL implements the third paradigm and provides the infrastructure to make it practical: shared persistent memory structured as a file system with attempts, notes, and skills; asynchronous multi-agent execution with isolated workspaces; heartbeat-based interventions for long-horizon robustness; and execution safeguards including evaluator isolation, workspace guards, and process management.

The paper's empirical claim is that this reorganization is not merely aesthetic — it produces substantially stronger performance across a diverse set of open-ended discovery tasks. A single autonomous agent already outperforms fixed evolutionary search baselines (Table 1), and multi-agent co-evolution pushes the frontier further than independent agents with equivalent total compute (Table 2, Table 3). The mechanistic analyses in Section 4.4 trace these gains to specific behaviors that autonomy enables: local verification before formal evaluation, knowledge accumulation through notes and skills, cross-agent information transfer, and exploration diversity — none of which are possible in fixed evolutionary search but all of which emerge naturally when agents control their own search process.

In the broader landscape of LLM-based discovery, CORAL represents a bet that the intelligence should be in the loop, not around it — that as LLMs become more capable as autonomous agents, delegating search meta-decisions to those agents will outperform even carefully hand-designed evolutionary algorithms. The paper's results provide the first systematic evidence for this hypothesis across a range of task types and difficulty levels.

3. Technical Approach

3.1 Reader Orientation

CORAL is a lightweight infrastructure framework — not a new learning algorithm or model architecture — that lets large language model (LLM) agents run continuously as autonomous researchers, controlling their own exploration, testing, and knowledge accumulation rather than being told what to do at each step by a fixed outer loop. It solves the open-ended discovery problem by replacing the four externally-scripted stages of traditional evolutionary search (RETRIEVE, PROPOSE, EVALUATE, UPDATE) with a persistent multi-agent operating system where each agent decides for itself what to examine, when to test, what to remember, and when to change direction, coordinated only through a shared library of solutions, observations, and reusable tools.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major modules, shown in Figure 4 of the paper:

Configuration (dataclass CoralConfig): Parses a YAML task specification that defines the problem description, evaluator parameters, agent count and model settings, workspace layout, and sharing flags. This is the single source of truth consumed by every other component.
Workspace Setup (create_project, create_agent_worktree): On startup, clones the task's seed repository, creates a shared .coral/ directory for persistent memory, and spins up one isolated git worktree per agent — each on its own branch so concurrent code changes cannot interfere. Symbolic links wire each worktree's knowledge folders (notes, skills, attempts) back to the centralized shared persistent memory.
Agent System (AgentManager, AgentRuntime, HeartbeatRunner): Spawns and monitors the long-running LLM agent processes. Each agent receives a generated instruction file (CORAL.md) and a CLI tool (coral eval, coral log, coral show, etc.) for interacting with the evaluation pipeline and shared memory. The HeartbeatRunner periodically checks trigger conditions and injects structured reflection prompts into agent sessions.
Grader Hierarchy (TaskGrader, FunctionGrader): A pluggable evaluation interface. The task-specific grader class (subclass of TaskGrader) is sealed in .coral/private/eval/grader.py — invisible to agents — and exposes only grade(codebase_path, tasks) -> ScoreBundle. When an agent calls coral eval, the system stages a git commit, spawns the grader in a subprocess with a hard timeout, and records the score.
Hub — Shared Persistent Memory (.coral/public/): A file-system-based collective knowledge base with three root folders: attempts/ (JSON records of every evaluated solution with scores), notes/ (markdown files of observations, analyses, and reflections), and skills/ (reusable procedures with natural-language descriptions and executable artifacts). All agents read from and write to this space through symlinks from their isolated worktrees.
Core Types (Task, Score, ScoreBundle, Attempt): Data classes that define the contract between components — what a task looks like, what evaluation results contain, and what gets recorded in shared memory.

Information flows as follows: at startup, a YAML config is parsed → workspace setup clones the seed repo and creates agent worktrees with symlinks to the shared hub → AgentManager writes CORAL.md into each worktree and launches agent processes → each agent, in a loop, plans changes, edits code, runs coral eval, reads the score and feedback, inspects prior attempts via coral log/show, reads and writes notes and skills, and continues → the HeartbeatRunner monitors the evaluation counter and per-agent scores, and when triggers fire (interval, plateau), interrupts the agent with a structured prompt → all attempts, notes, and skills are written to the shared hub, where other agents can discover them.

3.3 Roadmap for the Deep Dive

First, the problem formulation for open-ended discovery (Section 3.1 of the paper), which defines the four-stage abstraction (RETRIEVE, PROPOSE, EVALUATE, UPDATE) that CORAL reinterprets — this vocabulary is needed to understand what CORAL changes.
Second, the three paradigms in Figure 1 (fixed evolutionary search, autonomous single-agent evolution, autonomous multi-agent evolution), explaining what decisions shift from external rules to the agent at each stage.
Third, the shared persistent memory — the structured file system of attempts, notes, and skills — since it is the central coordination mechanism that all other components depend on.
Fourth, the multi-agent organization — how independent agents execute asynchronously in isolated workspaces and coordinate through shared memory rather than direct messaging.
Fifth, the heartbeat mechanism — periodic reflection, consolidation, and stagnation-triggered redirection — since this is what replaces the outer loop's fixed control with agent-internal steering.
Sixth, the evaluation pipeline and CLI tools (coral eval, coral log, coral show, etc.) — the interface through which agents interact with the grader and shared memory.
Seventh, the execution safeguards — evaluator isolation, workspace guards, process management, and evaluation timeouts — which make long-running autonomous operation practical and safe.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an infrastructure and systems paper whose core idea is that delegating search meta-decisions — what to explore next, when to test, what to remember, when to change direction — from fixed algorithms to autonomous LLM agents, and enabling multiple such agents to coordinate through shared persistent memory, yields substantially more efficient open-ended discovery than traditional evolutionary search with externally-scripted control flow.

Problem Formulation for Open-Ended Discovery

The paper defines open-ended discovery tasks through a precise formalism that cleanly separates what is fixed (the problem and evaluator) from what must be discovered (the solution) — a separation that makes it possible to substitute autonomous agents for fixed algorithms at every stage.

An open-ended discovery task instance is specified by a task description x (natural language, provided to the agent) and an evaluator E. The evaluator is a function that takes the task description and a candidate solution y, and returns:

$E(x, y) := (s, f)$

where $s$ is a numeric score measuring the quality of $y$ , and $f$ is auxiliary feedback — this could be a sub-score breakdown, a textual critique from an LLM-powered evaluator, execution traces, or any other structured or unstructured diagnostic information.

What it computes: given a candidate solution, the evaluator produces a scalar quality signal and optional diagnostic context. The score $s$ is the optimization target; the feedback $f$ provides the agent with information about why the score is what it is.

Why this form: this abstraction is intentionally minimal — it requires only an evaluative signal, not a differentiable loss, not ground-truth labels, and not a model of the solution manifold. The separation of $x$ and $E$ from $y$ means that the same evaluator can be applied to any candidate solution, making iterative proposal-and-test possible. The inclusion of $f$ is important because real evaluators for open-ended problems (e.g., a GPU kernel benchmark, a packing simulator) produce rich diagnostics — execution cycles per sub-operation, utilization percentages, constraint violation messages — that a reasoning agent can use to diagnose failures and plan improvements, rather than treating the score as a black box.

Let $M_t$ denote the shared persistent memory available at search step $t$ — this includes prior candidate solutions, their evaluation outcomes, and any notes or skills that agents have recorded. At an abstract level, each improvement step consists of four stages:

RETRIEVE: construct a working context $\hat{M}_t$ from $M_t$
PROPOSE: generate a candidate solution $y_{t+1}$ conditioned on $x$ and $\hat{M}_t$
EVALUATE: obtain score and feedback $(s_{t+1}, f_{t+1}) = E(x, y_{t+1})$
UPDATE: incorporate new information into shared persistent memory to form $M_{t+1}$

The paper's central insight is that who or what controls each of these four stages is the key architectural design choice that distinguishes different paradigms for open-ended discovery — and that moving control from fixed external rules to autonomous agents yields substantial gains.

From Fixed Search to Autonomous Evolution: What Changes at Each Stage

The paper's Figure 1 maps the progression across three paradigms by showing, for each of the four stages, the mechanism that controls it. Understanding this table is essential to understanding what CORAL actually does differently.

Fixed Evolutionary Search (prior work: FunSearch, AlphaEvolve, EvoX). In this paradigm:

RETRIEVE: Context selected by fixed rules. The working context is constructed from the population using predetermined selection heuristics — for example, MAP-Elites selects parents based on fitness and novelty scores, island models select from sub-populations with migration rules. The LLM has no agency over what it sees.
PROPOSE: Model proposes from given context. The LLM receives the constructed prompt and generates a candidate in a single forward pass. This is the only stage where the LLM participates at all.
EVALUATE: Triggered by a fixed loop. The outer evolutionary algorithm decides when evaluation happens — typically after every proposal. The LLM cannot decide to test locally, skip evaluation for unpromising candidates, or delay evaluation until it has refined a solution further.
UPDATE: Updated by fixed rules. The population is updated according to predetermined procedures — elitism, tournament selection, diversity maintenance. The LLM has no role in deciding what to preserve or discard.

Autonomous Single-Agent Evolution (CORAL with 1 agent). In this paradigm:

RETRIEVE: Agent chooses what to inspect. The agent can browse the leaderboard (coral log), inspect specific prior attempts (coral show <hash>), search for related attempts by keyword (coral log --search), read notes from previous runs, and check available skills. It decides what information is relevant to its current approach.
PROPOSE: Agent plans, implements, and tests. Rather than producing a candidate in a single forward pass, the agent engages in a multi-step process: it analyzes evaluation feedback and prior attempts, formulates a hypothesis about what would improve performance, implements code changes, and can optionally run local tests before submitting for formal evaluation. The proposal is the result of an extended reasoning-and-coding interaction, not a single token-generation step.
EVALUATE: Agent decides when to evaluate. The agent calls coral eval when it believes a candidate is ready. It can run local tests, fix bugs, and iterate within a single proposal before consuming an evaluation slot. It can abandon unpromising directions without ever calling the formal evaluator.
UPDATE: Agent decides what to store. The agent writes notes recording its observations and analyses, creates skills for reusable procedures, and decides what knowledge to externalize into shared persistent memory. The memory is not a fixed-size population of solutions — it is an open-ended repository that grows as the agent accumulates insights.

Autonomous Multi-Agent Evolution (CORAL with N agents). This paradigm extends single-agent autonomy with coordination through shared memory:

RETRIEVE: Agent reads from shared memory. Each agent can inspect not only its own prior attempts and notes, but also those produced by other agents. One agent's discoveries become another agent's starting point without any explicit communication protocol.
PROPOSE: Multiple agents propose in parallel. Agents operate asynchronously, each pursuing its own line of exploration. They can build on each other's commits, combine ideas from different agents, and pursue complementary strategies that collectively cover more of the solution space.
EVALUATE: Each agent schedules eval independently. There is no centralized scheduler — agents submit evaluations when they choose, and the evaluation pipeline handles concurrent submissions through file-system isolation (each attempt is written to a unique file keyed by commit hash).
UPDATE: Agents write to shared memory. Notes and skills are written to a directory structure visible to all agents. The paper reports that on advanced tasks, agents create and consume knowledge at much higher rates than on standard tasks, and that cross-agent information transfer — building on another agent's commit or referencing another agent's notes — has substantially higher improvement rates than the average attempt.

The key architectural shift is that in fixed evolutionary search, the RETRIEVE→PROPOSE→EVALUATE→UPDATE pipeline is procedural — executed by the framework code, with the LLM called as a subroutine during PROPOSE only. In CORAL, the pipeline is agent-driven — the agent is the top-level controller that decides when and how to invoke evaluation, retrieval, and memory update as tools in service of its own exploration strategy.

Shared Persistent Memory as File System

The shared persistent memory is the central coordination mechanism that makes autonomous multi-agent evolution possible. It is implemented as a structured file system within the .coral/public/ directory, with symbolic links making it accessible from each agent's isolated worktree. The design principle is progressive disclosure: the file system structure provides agents with a browsable, searchable knowledge repository that grows over time, and the use of standard file-system operations (read, write, list directory) means that agents can interact with it using the same Bash and file-reading tools they use for code.

The memory is organized into three root folders, each serving a distinct purpose:

attempts/ — historical evaluations and solutions. Each evaluation produces a JSON file named by the git commit hash of the candidate solution. An example record (from a 4-agent run on the Kernel Engineering task, shown in Box C.4) contains:

{
  "commit hash": "00d466e3...",
  "agent id": "agent-2",
  "title": "Pre-compute idx=2*idx+1 before hash...",
  "score": 1274.0,
  "status": "improved",
  "parent hash": "08e3b759...",
  "timestamp": "2026-03-14T14:55:16+00:00",
  "feedback": "eval: Cycles: 1,274 | Speedup: 115.96x | ..."
}

The status field is one of improved, baseline, regressed, crashed, or timeout — determined by comparing the new score against the agent's previous best. The feedback field preserves the evaluator's output verbatim. The parent hash records which prior attempt this one was built on, enabling the reconstruction of exploration trajectories.

What this enables: agents can browse the leaderboard (coral log), inspect specific attempts in detail (coral show <hash> with optional --diff flag), search by keyword (coral log --search), and reset their worktree to any prior attempt (coral checkout <hash>). This turns the history of all evaluations into a searchable library of approaches — an agent can study what worked for other agents, compare implementations, and identify patterns across attempts.

notes/ — observations, learnings, and reflections. Notes are markdown files with YAML frontmatter recording metadata (creator agent, creation timestamp). Agents are instructed to write notes after each evaluation and during heartbeat-triggered reflections. Notes can be organized hierarchically into subdirectories — the paper provides an example from Kernel Engineering where notes reside in insights/depth0-alu-xor-breakthrough-1181.md, documenting a specific architectural optimization that reduced cycles from 1187 to 1181 by converting depth-0 XOR operations from VALU to ALU. The note records the mechanism (saves 64 VALU at cost of 512 ALU, ALU has headroom), why related approaches failed (depth-1 XOR conversion created scheduling dependencies), and what to try next (convert depth-3 LT ops to ALU).

The heartbeat consolidation prompt (Box C.1.2) instructs agents to create three specific synthesis artifacts in the notes directory:

A synthesis note in notes/synthesis/ for any topic with 3+ individual notes, stating conclusions upfront, citing specific attempts as evidence, and explaining causal mechanisms
A connections map at notes/connections.md documenting cross-category patterns
An open questions list at notes/open-questions.md documenting contradictions and gaps

What this enables: notes encode why something worked, not just what the score was. This is the crucial difference from the solution population in fixed evolutionary search — the population says "solution X scored Y," but notes say "reducing VALU usage helps because VALU is the bottleneck at 6944/6 ≈ 1158 cycles vs. ALU at 12544/12 ≈ 1046 cycles." An agent can read notes to understand the principles that underlie high-scoring solutions, even if it plans to pursue a completely different implementation strategy.

skills/ — reusable procedures, tools, and implementation patterns. A skill consists of a SKILL.md file (natural-language description with YAML metadata) together with executable artifacts such as functions, example scripts, and usage instructions. The paper provides an example skill from the LLM-SQL task (skills/column-stats-sort/SKILL.md) that documents a column-reordering heuristic: it computes a scoring formula avg_length_sq * (avg_rows_per_group - 1) that prioritizes columns with high unique value count and long average string length for prefix caching. The skill records the formula, the results it achieved (score 0.7018, average hit rate 0.6864, runtime 0.27s), and when to use it ("as a baseline before trying more advanced approaches").

Agents are bootstrapped with a skill creator skill — a meta-skill that guides them through the process of abstracting reusable procedures from their experience and documenting them in the skills directory.

What this enables: skills capture transferable implementation patterns. When an agent develops an effective technique — a data preprocessing pipeline, an optimization heuristic, a testing script — it can package it as a skill that other agents can immediately use. This is fundamentally different from knowledge transfer through solution populations: in a population, an agent can only inherit the specific code of a parent solution; with skills, an agent can inherit a general technique and apply it in a different context.

Symlink architecture. The shared persistent memory is exposed to agents through symbolic links. For the Claude Code runtime, the agent's .claude/notes symlinks to .coral/public/notes/, and similarly for skills and attempts. This means agents use their runtime's native file access tools — Read, Write, Bash — to interact with shared memory, while the actual storage remains centralized in .coral/public/. A .gitignore rule in each worktree excludes .coral/ and runtime directories from git operations, preventing agents from accidentally committing shared memory files.

Concurrency model. Because agents operate asynchronously and each attempt is written to a unique file keyed by commit hash, no explicit locking is required for attempt recording. Notes and skills use unique filenames — the paper reports that "in practice, we observe no file-level conflicts across agents." This is a deliberate design choice: file-system-based coordination with unique naming is simpler and more robust than a database with transactional guarantees, at the cost of requiring agents to handle rare conflict cases (which the prompt instructions address by recommending unique note filenames based on topic).

Artifact quality differences across task types. The paper's trajectory analysis (Section 4.4.1, Table 4) reveals an important qualitative difference in how knowledge artifacts function on standard versus advanced tasks:

On standard tasks, agents create only 0.05 knowledge artifacts per attempt, and knowledge access yields only a small gain (+2 percentage points improvement rate over attempts without knowledge access). Notes on standard tasks tend to be "lightweight progress logs, such as records of parameter changes."
On advanced tasks, agents create over 10× more knowledge per attempt (0.55 for Polyominoes, 0.68 for Kernel Engineering), and knowledge access is much more strongly associated with improvement (55% on Kernel Engineering vs. 26% on standard tasks). The knowledge itself is qualitatively different — Kernel Engineering notes "identify architectural bottlenecks such as VALU or record cases where relaxing WAR dependencies hurts performance," while Polyominoes includes a "what NEVER worked" folder documenting failed approaches.

This asymmetry — that the value of shared persistent memory increases with task complexity — is central to understanding when and why CORAL's architecture matters. On simple tasks where improvement comes from incremental parameter tuning, the overhead of knowledge externalization may not pay for itself. On complex tasks where improvement requires understanding system-level architectural tradeoffs, the ability to accumulate and reuse interpretable insights is critical.

Multi-Agent Organization

CORAL's multi-agent architecture extends from a single autonomous agent to a population of $N$ agents with a specific organizational principle: coordination through shared memory, not direct messaging.

Agent isolation. Each agent $i$ maintains its own local context $C^{(i)}_t$ (its conversation history, current code state, and working memory) and executes in an isolated git worktree — its own branch of the seed repository, created at startup. This isolation ensures that concurrent agents cannot interfere with each other's code state. If agent-1 modifies kernel_builder.py and agent-2 simultaneously modifies the same file, they do so on different branches and each sees only their own version until one's changes are committed and the other explicitly checks out that commit.

Shared access. All agents share access to the same evaluator (via the coral eval CLI, which routes to the same grader implementation) and the same shared persistent memory $M$ via symbolic links from their individual worktrees. The evaluator is a stateless service — each evaluation is independent of all others. The shared memory is the only state that persists across agents and across time.

Coordination mechanism. Unlike many peer-to-peer multi-agent systems where agents directly talk to each other through structured messages (LangChain, 2024; Wu et al., 2023), coordination between agents in CORAL occurs exclusively through shared persistent memory. Each agent writes artifacts $W^{(i)}_t$ — attempts, notes, skills — to $M$ as a byproduct of its autonomous workflow. Later, another agent $j$ may retrieve those artifacts as part of constructing its own working context $\hat{M}^{(j)}_t$ . The paper describes this as agents "interacting indirectly through shared persistent memory":

"This way, one agent's discoveries can influence another agent's future search through what it writes to the shared workspace, without requiring a messaging protocol."

What this enables. The indirect coordination model has several properties that the paper argues are beneficial for open-ended discovery:

Asynchronicity without protocol overhead. Agents do not need to be synchronized, do not need to wait for responses from other agents, and do not need to maintain a model of other agents' current state. An agent can read a note written by another agent hours ago and benefit from it immediately — there is no "stale message" problem because knowledge is versioned by the git commits that produced it.
Emergent information transfer without predefined roles. The paper reports that on Kernel Engineering, "36% of attempts use another agent's commit as their parent, and these improve at 17% versus 9% for all attempts. The majority (66%) of new records originate from a cross-agent parent." This cross-pollination is not orchestrated by any central scheduler — it emerges because agents autonomously inspect the leaderboard, find promising work by other agents, and build on it.
Complementary transfer modes. The two stress-test tasks exhibit different dominant transfer modes that the system supports without modification. On Kernel Engineering, transfer occurs more through referencing others' code (36% of attempts have cross-agent parents). On Polyominoes, direct code transfer is rarer (12% of attempts) but still powerful when it happens (50% improvement rate vs. 19% average), and transfer instead occurs more through shared notes and skills — 87% of rounds reference knowledge committed by other agents. The architecture supports both modes because the shared memory contains both code (attempts) and interpretable knowledge (notes, skills).
Exploration diversity without explicit diversity mechanisms. Because each agent autonomously decides what to explore, the population naturally diversifies. The paper quantifies this by extracting strategy keywords from attempt titles and computing pairwise Jaccard similarity. On Kernel Engineering, agents average 0.43 pairwise overlap; on Polyominoes, 0.31. More than half of each agent's strategy vocabulary is unique. This diversity is an emergent property of autonomous decision-making — no diversity-maintenance heuristic (e.g., MAP-Elites' novelty score) is required.

Initialization and heterogeneity. An important architectural choice: all agents are initialized identically with the same CORAL.md instruction file, the same access to shared memory, and the same model configuration. The paper explicitly notes this as a limitation in Appendix A:

"Multi-agent evolution currently lacks bootstrapped heterogeneity: all agents are initialized identically and given access to the same information."

The diversity that emerges is therefore purely a consequence of the stochastic, autonomous decision-making of each agent — different initial random choices by the LLM lead to different exploration trajectories, which diverge further as each agent encounters different evaluation results and reads different subsets of the shared memory at different times.

Contribution balance. The paper reports an important empirical pattern: on Kernel Engineering, all four agents produce similar numbers of attempts (130–165), similar numbers of improvements (10–16 each), and all four independently reach the best score of 1103 cycles. Records are evenly split (14/15/10/15). This suggests that the shared persistent memory creates a "rising tide lifts all boats" effect — each agent benefits from others' discoveries sufficiently that no single agent dominates. Leader tenure is more skewed (agent-1 holds the best score for 45% of the run), but this reflects the timing of submissions rather than capability differences. On Polyominoes, contributions are less balanced (agent-3 sets 6 of 13 records), suggesting that task characteristics influence how evenly exploration effort translates to improvements.

Heartbeat: Reflection, Consolidation, and Redirection

The heartbeat mechanism addresses a specific failure mode of autonomous evolution: agents may drift into local optima, forget to externalize knowledge, or pursue micro-optimizations instead of exploring innovative directions. Because CORAL does not enforce a fixed workflow — agents are free to decide what to do next — there is no external signal that tells an agent when it should stop tweaking a diminishing-returns approach and try something fundamentally different. The heartbeat mechanism provides this signal by periodically injecting structured prompts into the agent's ongoing conversation.

Conceptual model. The paper describes heartbeats as functioning "like a Reminder App, periodically prompting the agents to exercise self-reflection and pivoting for new ideas when existing approaches plateau." A heartbeat event consists of a trigger condition and a prompt template. When the trigger fires at step $t$ , the heartbeat applies a modification to the agent's local context:

$C^{(i)}_t \rightarrow C'^{(i)}_t$

where $C^{(i)}_t$ is the agent's current context (conversation history, code state, and working memory) and $C'^{(i)}_t$ is the context after the heartbeat prompt is injected.

What it computes: at each monitoring cycle (every 5 seconds), the agent manager checks each configured heartbeat action's trigger condition against the current state. If a trigger fires, the agent is interrupted, and the rendered heartbeat prompt is appended to its conversation context. The agent then continues autonomously from the augmented context — the heartbeat does not force a specific action, but steers subsequent behavior by drawing the agent's attention to specific meta-cognitive tasks.

Why this form: the heartbeat design reflects a deliberate choice about the level of intervention. CORAL does not force an agent to change direction at a plateau — it prompts the agent to consider whether a change is warranted. This preserves agent autonomy (the agent can examine the evidence and decide that continued optimization of the current approach is still promising) while combating the known tendency of LLM agents in long-horizon tasks to drift into repetitive patterns or lose track of the bigger picture. The alternative — having the framework automatically reset the agent's approach when a plateau is detected — would override the agent's judgment and could interrupt productive optimization. The alternative — having no intervention at all — is what the paper shows leads to premature plateauing in single-agent runs.

Three heartbeat types. CORAL implements three built-in heartbeat prompts with different triggers and purposes:

Per-iteration reflection heartbeat (every = 1, trigger = interval, scope = local). Fires after every single evaluation by each agent. The prompt (Box C.1.2) instructs the agent to pause and write a note capturing: (a) concrete results from recent attempts with specific score changes and what caused them, (b) surprises and unexpected outcomes, (c) causal analysis of the most significant result, (d) confidence assessment of the current approach, and (e) a plan for the next experiment with an explicit prediction of what will happen. The prompt emphasizes anchoring in concrete evidence ("Attempt abc123 improved score from 0.72 to 0.78 by adding batch normalization after each conv layer") rather than vague impressions.

Purpose: this heartbeat ensures that knowledge externalization happens contemporaneously with experiments, rather than retrospectively when details have been forgotten. It addresses the problem that agents, when focused on coding, may neglect to document their reasoning.
Periodic consolidation heartbeat (every = 10, trigger = interval, scope = global). Fires after every 10 evaluations across all agents (global counter). The prompt (Box C.1.2) instructs the agent to synthesize the shared knowledge base by: (a) creating synthesis notes for any topic with 3+ individual notes, (b) updating a connections map documenting cross-category patterns, (c) documenting open questions and contradictions, (d) reorganizing the notes hierarchy, and (e) promoting well-validated techniques to skills. The required outputs are specific files: a synthesis note in notes/synthesis/, the connections map at notes/connections.md, and the open questions list at notes/open-questions.md.

Purpose: this heartbeat transforms the accumulating stream of per-eval notes into structured, navigable knowledge. Individual reflection notes capture raw observations; consolidation synthesizes those observations into conclusions. The distinction between the reflection heartbeat (during-work note-taking) and consolidation heartbeat (after-work organization) is an explicit design choice: separating capture from synthesis reduces the cognitive burden on the agent during active exploration.
Stagnation-triggered redirection heartbeat (every = 5, trigger = plateau, scope = local). Fires when an agent has produced no score improvement for 5 consecutive evaluations. The prompt (Box C.1.2) instructs the agent to: (a) diagnose why the current approach has plateaued (are scores flat? oscillating? what is the theoretical limit?), (b) study what's different in the top-scoring attempts — especially from other agents, (c) choose a fundamentally different approach (different algorithm family, different problem formulation, different representation, or techniques from other domains), (d) reset to the best-scoring attempt as a foundation, and (e) commit a minimal implementation of the new approach quickly.

Purpose: this heartbeat directly addresses the local optima problem. The prompt explicitly frames the situation: "You are likely stuck in a local optimum. It's time to try something fundamentally different." The instruction to study other agents' top attempts leverages multi-agent diversity — what is a novel approach for one agent may already be demonstrated in another agent's work. The instruction to "commit quickly" with a minimal implementation counteracts the tendency to over-invest in an unproven direction.

Trigger mechanism detail. The agent manager's monitoring loop polls .coral/public/attempts/ every 5 seconds. For each new attempt detected, it updates per-agent tracking state: the local evaluation count, the best score achieved, and the number of consecutive evaluations without improvement (for plateau detection). The HeartbeatRunner then evaluates trigger conditions:

Interval triggers: fire when count mod every == 0, using either the local eval counter (for scope = local actions) or the global eval counter (for scope = global actions). The mod operation ensures periodic firing at regular intervals.
Plateau triggers: fire when evals_since_improvement >= every, with a cooldown mechanism that prevents re-firing until another every evaluations of continued stalling have accumulated. The cooldown prevents the agent from being interrupted by the same plateau prompt repeatedly in quick succession — once the agent has been told it is stuck and has attempted a new direction, it gets every more evaluations to try that direction before being prompted again.

Delivery mechanism. When heartbeat actions are triggered, the manager interrupts the agent process via SIGINT, which triggers graceful session saving in the Claude Code runtime (preserving conversation history). The agent is then resumed with a combined prompt containing: (1) the evaluation results that triggered the heartbeat (score, commit hash, status, feedback), and (2) the rendered heartbeat prompt(s) with {shared dir} and {agent id} substituted. This injection places the heartbeat prompt directly in the agent's conversation context — the agent processes it as if it were the next instruction in its workflow.

Agent-modifiable heartbeats. Agents can customize their heartbeat configuration at runtime via CLI commands:

coral heartbeat set adds or updates a heartbeat action with a custom prompt, interval, trigger type, and scope.
coral heartbeat remove removes a heartbeat action.
coral heartbeat reset restores default configuration.

Protected actions (reflect, consolidate) cannot be deleted to ensure minimum knowledge externalization. This is a safety mechanism: even if an agent decides it does not need to reflect (perhaps because it is "on a roll" and does not want interruption), the system enforces periodic reflection as a non-negotiable part of the autonomous workflow.

Default configuration. Unless otherwise noted, all CORAL experiments use the default heartbeat configuration shown in Table 7: reflect every 1 eval (local), consolidate every 10 evals (global), pivot after 5 non-improving evals (local). The paper does not report ablations varying the heartbeat configuration — the default settings are used uniformly across all tasks. Understanding the sensitivity of results to heartbeat frequency and trigger thresholds is left as future work.

Evaluation Pipeline and CLI Interface

The evaluation pipeline is the mechanism by which agents submit candidate solutions for formal scoring. It is exposed to agents through a command-line interface (CLI) with 17 commands organized into four categories — workflow, query, orchestration, and heartbeat management — but the central command for agents is coral eval.

The coral eval pipeline. When an agent runs coral eval -m "description", the system executes an eight-step sequence:

Stage and commit. Run git add -A followed by git commit -m "description" in the agent's worktree. The agent's description becomes the commit message, preserving a human-readable (and agent-readable) record of what was changed and why. Because each agent operates in its own git worktree on its own branch, commits from different agents do not conflict.
Load grader. Dynamically import the Grader class from .coral/private/eval/grader.py — a file that is copied to the private directory at run initialization and is inaccessible to agents. The agent cannot read this file, modify it, or discover implementation details that would enable reward hacking.
Grade. Spawn the grader in a child process with a configurable hard timeout (default 300 seconds). The grader receives the path to the agent's worktree and any task-specific arguments, executes the candidate solution (e.g., runs a kernel simulator, invokes a packing algorithm, evaluates a mathematical function), and returns a ScoreBundle containing a numeric score and textual feedback.
Determine status. Compare the returned score against the agent's previous best score: improved if strictly better (for maximize tasks) or strictly lower (for minimize tasks), baseline if equal, regressed if worse, crashed if the grader returned None (indicating an execution failure), or timeout if the grader exceeded the time limit.
Record attempt. Write an Attempt JSON file to .coral/public/attempts/<hash>.json containing the commit hash, agent ID, user-provided description (stored as title), score, status, parent commit hash, timestamp, and evaluator feedback.
Checkpoint. Snapshot the current shared persistent memory (notes, skills) with a hash for versioning, enabling rollback if needed.
Increment counter. Update the global evaluation counter at .coral/public/eval_count, which drives interval-based heartbeat triggers.
Return results. The CLI returns the score and feedback to the agent, which can then read the results and decide what to do next.

Why this pipeline design. Several design choices are motivated by the autonomy of agents:

Git as the submission mechanism means that every evaluation corresponds to an immutable, inspectable snapshot of the codebase. Agents can later inspect any prior attempt with coral show <hash> to see the exact code changes and evaluation results. The commit message serves as a lab notebook entry — the paper explicitly tells agents: "Eval messages are your paper trail — write like lab notebook entries."
Grader isolation (private directory, subprocess execution, hard timeout) prevents agents from inspecting or modifying the evaluation logic. This is a critical safeguard in autonomous systems where agents might otherwise learn to exploit grader vulnerabilities. The paper notes this explicitly: "This reduces opportunities for reward hacking."
Status determination gives the agent immediate feedback on whether its change improved, maintained, or degraded performance — information that is essential for the agent's own reasoning about what direction to pursue.

Query commands. Agents use query commands to inspect the shared memory and plan their next steps:

coral log displays the leaderboard — the top 20 attempts by score, with commit hashes, scores, and agent IDs. coral log --recent shows the most recent attempts. coral log --search "keywords" searches attempt titles and feedback for specific terms.
coral show <hash> displays the full details of a specific attempt, including the commit message, score, feedback, and metadata. With the --diff flag, it shows the code diff between that commit and its parent.
coral notes lists, searches, and reads shared notes. coral skills lists and reads shared skills.
coral checkout <hash> resets the agent's worktree to the state of a specific prior attempt — the core mechanism for building on others' work.
coral diff shows uncommitted changes; coral revert undoes the last commit.

Why these commands. The set of query commands enables the agent behaviors that the paper's trajectory analysis shows are associated with successful improvement: inspecting prior attempts (25% attempt inspection rate on standard tasks, 47% on advanced tasks), accessing knowledge (7% on standard tasks, 17–30% on advanced tasks), and resetting to a strong base before trying a new approach. The commands provide the agent with what information exists without prescribing how to use it — the agent decides which attempts to inspect, which notes to read, and when to reset.

Orchestration commands (used by the human operator, not agents) manage the overall experiment:

coral start -c task.yaml launches the full multi-agent run with optional dotlist overrides for any configuration field.
coral resume resumes from a previous run, restoring agent sessions from saved session IDs.
coral stop performs graceful shutdown (SIGINT → SIGTERM → SIGKILL with escalating timeouts).
coral status displays agent health (alive/dead), evaluation counts, and current leaderboard.

Execution Safeguards

Autonomous agents running for hours with the ability to execute arbitrary code, modify files, and make API calls require safeguards to prevent them from corrupting the evaluation, exhausting resources, or interfering with each other. CORAL implements four categories of safeguards:

1. Evaluator isolation. At run initialization (coral start time), the grader implementation (eval/grader.py) is copied to .coral/private/eval/grader.py. The private directory is explicitly excluded from the symlinks that expose shared memory to agent worktrees — agents can see .coral/public/ but not .coral/private/. This separation means:

Agents cannot read the grader source code to discover edge cases or scoring implementation details that they could exploit.
Agents cannot modify the grader to inflate their scores.
The grader remains immutable across the entire run (unless the operator manually updates it between runs).

The grader interface itself is abstracted behind a protocol (GraderInterface) with a single method grade(codebase_path, tasks) -> ScoreBundle, making it impossible for an agent to influence the grader through any channel other than the code in its worktree.

2. Workspace guard. Each agent's worktree has a .gitignore that excludes .coral/ and runtime directories from git operations. This prevents agents from accidentally (or deliberately) committing shared memory artifacts or configuration files as part of their code changes. A .coral_dir breadcrumb file in each worktree records the path to the shared persistent memory — this is used by the evaluation pipeline to locate the shared hub regardless of the agent's current working directory. The breadcrumb is writable only by the framework, not by the agent.

3. Process management. The agent manager records its own PID at .coral/public/manager.pid and all agent PIDs at agent.pids. This enables coral stop to locate and terminate all processes without scanning the process table. Graceful shutdown follows a graduated sequence:

First, SIGINT is sent to all agents, triggering graceful session saving (the Claude Code runtime saves conversation state on SIGINT).
After a configurable timeout, SIGTERM is sent to any remaining processes.
After a further timeout, SIGKILL is sent as a last resort.

Dead agent restart. If an agent process terminates unexpectedly — due to max-turns exhaustion (the Claude Code runtime has a configurable turn limit) or a crash — the monitoring loop detects the exit within 5 seconds (its polling interval) and automatically restarts the agent. The restarted agent receives a prompt containing the latest evaluation results (so it knows what happened while it was down) and instructions to review the leaderboard and recent activity (so it can re-orient itself). This ensures that a single agent crash does not silently reduce the effective agent count for the remainder of the run.

4. Evaluation timeout. Each grader invocation runs in a child process with a configurable hard timeout (default 300 seconds, configurable per task in task.yaml under grader.timeout). If the grader exceeds this limit — for example, because the agent's solution enters an infinite loop, or because the evaluation involves solving an intractable subproblem — the child process is killed, and the evaluation is recorded with status timeout and a null score. The timeout is set per task because different tasks have different natural evaluation latencies: compiling and running a GPU kernel simulation takes longer than evaluating a mathematical function.

Why these safeguards matter for the paper's claims. The safeguards are not incidental engineering details — they are what make autonomous multi-agent evolution practical as an experimental paradigm. Without evaluator isolation, there is no way to trust that score improvements represent genuine algorithmic progress rather than grader exploitation. Without workspace guards, agent errors (or adversarial behavior) could corrupt the shared memory that other agents depend on. Without process management, runs could hang indefinitely due to a single crashed agent. Without evaluation timeouts, an agent could consume the entire wall-clock budget on a single pathological evaluation. The fact that CORAL achieves its performance gains with these safeguards in place — rather than in a simplified environment where agents have unrestricted access — strengthens the paper's claim that autonomous multi-agent evolution is a viable paradigm for real-world open-ended discovery.

Agent Instruction Document (CORAL.md)

Each agent receives a structured instruction document at startup that is automatically generated and placed in the agent's worktree. This document is the agent's sole source of task-level instructions and system interface documentation — it defines the task, explains how to use the CORAL CLI, establishes workflow conventions, and sets ground rules.

Template instantiation. The instruction file is instantiated from one of two templates — multi-agent or single-agent — by substituting task-specific fields: {task name}, {task description}, {score direction} (maximize or minimize, so the agent knows which direction is better), {shared dir} (path to the shared persistent memory), and {agent id}.

Multi-agent template (Box C.1.1). The multi-agent version emphasizes collaboration:

"You are one of several agents working on this task in parallel. These agents are your colleagues. Each agent has its own git worktree... but you all share a .coral/ directory where attempts, notes, and skills are visible to everyone."

It frames the multi-agent dynamic explicitly: "Have a collaborative mindset: frequently check in with your agent mates, learn from what they have done well, and actively contribute your findings."

The workflow is specified as a loop: plan → edit → eval → repeat. The plan step instructs agents to review the leaderboard, inspect top attempts, check notes and skills from other agents, and "think creatively" — but keep plans lightweight. The edit step tells agents to "make focused changes. One idea per eval. Bias toward speed." The eval step specifies coral eval -m "what you changed and why", followed by mandatory note and skill creation. The read results step tells agents to use coral checkout to navigate to prior attempts.

Single-agent template. The single-agent version omits collaborative language — there are no "agent mates" and no shared notes from concurrent agents. Instead, it emphasizes persistence: "You should never stop until you reach / beat the best score." It makes skill creation mandatory after every evaluation (in the multi-agent template, it is a "strong recommendation"), and references notes as "from previous runs" rather than "from other agents." These differences reflect the different failure modes: in single-agent, the risk is giving up too early; in multi-agent, the risk is failing to leverage others' work.

Ground rules. Both templates include explicit prohibitions:

"You are fully autonomous. Do not ask for permission." — This is important because LLM agents sometimes default to asking for user confirmation before taking actions; CORAL needs agents to proceed without human-in-the-loop.
"Never run git commands directly. Use coral eval/checkout/revert/diff." — This ensures the framework maintains control over version management and can track all code state changes.
"Never touch .coral/ with git." — This prevents agents from committing shared memory files or configuration.
"Eval messages are your paper trail — write like lab notebook entries." — This ensures that attempt descriptions are useful for future retrieval.
"Eval early and often." — This pushes agents toward the rapid experimentation cycle that the paper's results show is effective.

Orientation sequence. Before writing any code, agents are instructed to orient themselves by: reading the task description, reading key files to understand the current code state, checking the leaderboard via coral log, checking recent activity via coral log --recent, inspecting top attempts via coral show <hash>, searching for prior art via coral log --search, reading notes in the shared directory, and checking available skills. This orientation sequence is deliberately detailed — it ensures that agents begin with a thorough understanding of what has already been tried, reducing redundant exploration.

Task Configuration and Grader Interface

CORAL provides a unified task interface that decouples the evolution loop (which is CORAL's responsibility) from task-specific evaluation logic (which is provided by the user). This decoupling is what makes CORAL "task-agnostic" — the same framework runs on mathematical optimization, systems optimization, and kernel engineering without modification.

Configuration schema. A task is fully specified by a YAML file (task.yaml) with six sections:

task: Metadata including name, description (the full problem statement — this is what gets substituted into CORAL.md), files (key files for the agent to examine), seed (initial files to copy into the workspace), and tips (evaluation-specific hints such as timeout and scoring details).
grader: Evaluation configuration including timeout (in seconds), direction (maximize or minimize — determines what counts as "improved"), args (task-specific arguments passed to the grader constructor), and private (additional files copied to .coral/private/ and hidden from agents).
agents: Agent configuration including count, runtime (e.g., claude code), model (e.g., claude-opus-4-6), max turns, heartbeat (action list), and research (boolean — enable web search access).
workspace: Paths for results dir and repo path (seed code), and setup (shell commands to run before agent startup, e.g., uv sync to install Python dependencies).
run: Operational flags: verbose, ui (enable web dashboard), tmux.
sharing: Boolean flags for sharing attempts, notes, skills across agents.

Grader implementation. Task-specific evaluation logic is implemented as a Python class that inherits from TaskGrader (which itself inherits from BaseGrader). The class must implement a single method:

def evaluate(self) -> float | ScoreBundle:

The method receives the agent's worktree path via self.codebase_path, the task-specific arguments via self.args, and access to private files via self.private_dir. It has access to helper methods: self.run_program(filename) executes a file in a subprocess with timeout; self.run_script(code) runs inline Python; self.run_script_json(code) runs a script and parses its stdout as JSON.

Score representation. Evaluation results are represented as a ScoreBundle containing:

Multiple named Score objects, each with a value (float) and name (string). For tasks with multiple sub-objectives, the bundle can contain per-component scores.
An aggregated numeric score computed as a weighted average of component scores.
Optional feedback text (string) that appears in the attempt record.
An is_public flag controlling whether the score is visible to agents.

The ScoreBundle abstraction allows tasks to provide rich, multi-dimensional feedback while still defining a scalar optimization target. For example, a kernel engineering task might report both cycle count and correctness, where correctness is binary (pass/fail) and cycle count is the optimization target — the bundle can contain both, with the aggregated score reflecting whether the solution was correct before considering cycle count.

Score direction. The direction field in the grader configuration determines how CORAL compares scores. For maximize tasks, "improved" means new_score > best_score. For minimize tasks (like Kernel Engineering, where lower cycles are better), "improved" means new_score < best_score. The scoring function for Kernel Engineering uses linear interpolation:

The paper provides the exact formula in the task tips: score is linearly interpolated with 0.0 at the baseline (147,734 cycles) and 1.0 at the best known (1,363 cycles). This bounds the optimization landscape: any solution worse than the naive baseline receives a negative score, and the target is to exceed 1.0 by improving beyond the previously best known human result.

Task diversity. The paper evaluates on tasks spanning three categories (mathematical optimization, systems optimization, and stress-test problems), and the grader implementations vary accordingly:

Mathematical optimization tasks (circle packing, signal processing, Erdős overlap, MMD, autocorrelation inequalities): graders typically evaluate mathematical functions, verify constraints, and return the objective value. Evaluation is purely computational — running the candidate program as a subprocess and parsing its output.
Systems optimization tasks (EPLB, PRISM, LLM-SQL, transaction scheduling, Cloudcast): graders run simulations or benchmarks against test datasets, often with stochastic elements. The paper documents several grader bug fixes (Appendix D.3, Table 8) that were necessary because the original evaluators had edge cases that could produce incorrect scores — for example, the PRISM evaluator silently skipped failed GPU placements rather than penalizing them, allowing solutions that crashed on difficult inputs to appear well-balanced.
Stress-test problems: Kernel Engineering uses a VLIW SIMD kernel simulator that counts execution cycles for a tree-traversal program; correctness is mandatory (incorrect kernels are rejected regardless of cycle count). Polyominoes uses a packing simulator that computes the fraction of grid cells covered.

The diversity of grader implementations — subprocess execution, JSON result parsing, constraint validation, benchmark-relative scoring, delegation to external frameworks — demonstrates that CORAL's task interface is genuinely general, not tailored to a specific evaluation pattern.

Evaluator Corrections

During integration of the systems optimization tasks from the SkyDiscover repository (Liu et al., 2026b), the authors discovered and corrected four bugs in the evaluator implementations that could lead to incorrect scoring. These corrections are important for reproducibility because they affect the absolute scores reported in the paper's experiments — without the fixes, solutions with specific failure modes could achieve artificially inflated scores.

PRISM: failed placements silently skipped. The original evaluator caught TimeoutError and general exceptions during GPU placement evaluation but called continue to skip the failed test case. This meant a solution that crashed on difficult inputs would only be graded on the easy cases it successfully completed, receiving an artificially high average score. The fix appends a worst-case penalty value (10⁶, corresponding to maximum load imbalance) for each failed placement, so failures are explicitly reflected in the final score.

Transaction Scheduling: invalid schedules scored above zero. The original evaluator computed score = 10⁶ / (1 + makespan) regardless of whether the schedule was valid (respecting read-write and write-write conflict ordering). Invalid schedules could receive positive scores proportional to their makespan, rewarding incorrect solutions — a shorter incorrect schedule would score higher than a longer incorrect schedule, even though neither is valid. The fix gates the scoring formula on a validity check: invalid schedules receive a score of 0.

EPLB: dropped experts and redundant averaging. Two issues existed. First, when an expert had zero replicas assigned, the evaluator skipped it entirely rather than penalizing the imbalance — allowing solutions that simply dropped difficult-to-balance experts to appear well-balanced. The fix concentrates all of the dropped expert's load onto a single physical slot (the worst-case imbalance scenario). Second, the grader averaged results over 3 redundant runs of the same deterministic evaluator, adding noise without informational value. The fix removes this averaging and uses a single evaluation.

LLM-SQL: type handling. The seed program's column analysis assumed homogeneous column types in the input DataFrame, but real datasets contain mixed types (integers, strings, nulls) that caused prefix-matching operations to crash. The fix converts all DataFrame values to string dtype before analysis, ensuring robust handling of heterogeneous types without changing the optimization objective.

Why these corrections matter beyond reproducibility. These bugs illustrate a subtle challenge in open-ended discovery: the evaluator itself may be imperfect, and optimization can exploit evaluator flaws. In fixed evolutionary search, these flaws might go undetected because the search is less efficient — it might not explore the edge cases that trigger them. CORAL's agents, by autonomously trying diverse strategies, are more likely to stumble upon evaluator vulnerabilities. The paper's documentation of these corrections is a form of transparency about the engineering work required to make open-ended discovery benchmarks fair, and it suggests that evaluator robustness is an important consideration for future work in this area.

4. Key Insights and Innovations

Innovation 1: A New Diagnostic Architecture for Open-Ended Discovery — Control Over Search Meta-Decisions, Not Search Algorithms

What is distinctive at the idea level. CORAL's contribution is not a better evolutionary algorithm, mutation operator, or verifier. It is a reorganization of the evolutionary search loop's control architecture — a shift in who decides what rather than what the decision rules are. In fixed evolutionary search (FunSearch, AlphaEvolve, EvoX), the framework code controls RETRIEVE, EVALUATE triggering, and UPDATE; the LLM is confined to PROPOSE. CORAL inverts this: the LLM agent controls all four stages, and the framework provides infrastructure (shared memory, heartbeat, evaluator isolation) rather than control flow. This is an architectural and conceptual contribution, not an algorithmic one.

The paper captures this shift explicitly in Figure 1 and the table below it. In fixed search, RETRIEVE is "context selected by fixed rules," EVALUATE is "triggered by a fixed loop," and UPDATE is "updated by fixed rules." In autonomous evolution, each of these becomes agent-driven: "agent chooses what to inspect," "agent decides when to evaluate," "agent decides what to store." The verbs shift from passive (selected, triggered, updated) to active (chooses, decides). The conceptual contribution is recognizing that search meta-decisions — what to explore, when to test, what to remember — are themselves part of the optimization problem and that LLMs, which can read evaluation histories, analyze failure patterns, and reason about what to try next, are better positioned to make these decisions than any fixed heuristic.

Prior work and the dominant assumption it challenges. Prior LLM-based evolutionary search systems — FunSearch (Romera-Paredes et al., 2024), AlphaEvolve (Novikov et al., 2025), EvoX (Liu et al., 2026a), AdaEvolve (Cemri et al., 2026) — all treat the LLM as a subroutine within an externally-scripted loop. Adaptive variants like AdaEvolve and EvoX evolve the hyperparameters of the search strategy (mutation rates, selection pressure), but the LLM still follows a fixed pipeline: receive prompt, generate candidate, receive score. The dominant assumption was that search orchestration belongs in the framework, not in the model. Even the autonomous agent literature (SWE-Agent, OpenHands, AI Scientist) targets one-off task completion — fixing a bug, running experiments for a paper — rather than sustained, open-ended optimization. CORAL challenges the assumption by showing that the agent's reasoning capabilities, applied to the meta-decisions of search, produce more efficient exploration than hand-designed population management.

Significance beyond raw performance. The performance gains (3–10× improvement rates, 10× fewer evaluations, 4× efficiency) are downstream of a conceptual reframing: the intelligence should be in the loop, not around it. This is a bet about scaling: as LLMs become more capable as coding agents, delegating increasingly high-level search decisions to the agent will outperform even carefully hand-designed evolutionary algorithms. The paper's evidence supports this bet for the current generation of models (Claude Opus 4.6, MiniMax M2.5), but the conceptual framing matters beyond any specific performance number. It opens the door to a class of systems where the framework provides scaffolding (memory, tool interfaces, safety) and the agent provides intelligence (planning, diagnosis, redirection), rather than the framework providing intelligence (search strategy) and the agent providing stateless generation.

Incremental or fundamental? This is a fundamental architectural shift, not an incremental refinement. It changes the locus of control from framework code to model reasoning. The evidence for the shift being more than cosmetic comes from the ablation in Table 3: knowledge accumulation (which is only meaningful when agents control what to remember) and co-evolution (which is only meaningful when agents can build on each other's work) both causally improve performance independent of additional compute. The gap between co-evolution and independent best-of-4 single-agent runs shows that the organizational structure matters beyond simply running more agents — the architecture enables behaviors that independent agents cannot replicate.

Innovation 2: Shared Persistent Memory as Coordination Without Communication Protocols

What is distinctive at the idea level. CORAL introduces a specific coordination model for multi-agent evolution: agents coordinate solely through a shared, persistent, file-system-based knowledge repository — not through direct messaging, predefined roles, or structured communication channels. This is fundamentally different from the dominant multi-agent paradigm in LLM systems, which relies on vertical scaling: humans decompose the task, assign specialized roles, and design a fixed communication topology (Lu et al., 2024; Gottweis et al., 2025; Hong et al., 2024; Wu et al., 2023). CORAL instead enables horizontal scaling: identical agents explore in parallel, write their discoveries to shared memory, and later read each other's findings — coordination emerges from asynchronous read-write patterns on a shared artifact, not from agent-to-agent messages.

The conceptual innovation is in what gets shared and how. In fixed evolutionary search, what gets shared is the population of solutions — raw code, implicitly encoding design decisions through its structure. In CORAL, what gets shared is code plus interpretable knowledge: attempt records (what was tried and what score it got), notes (why something worked, what bottlenecks were identified, what never works), and skills (reusable procedures with documentation). This is a progression from implicit knowledge (code that must be reverse-engineered to understand its principles) to explicit knowledge (natural language explanations of principles). The paper's trajectory analysis confirms this distinction matters: on advanced tasks, agents create 10× more knowledge artifacts per attempt than on standard tasks, and knowledge access on advanced tasks is associated with much higher improvement rates (55% on Kernel Engineering vs. 26% on standard tasks). The knowledge itself is qualitatively different — notes on advanced tasks identify architectural bottlenecks, while notes on standard tasks are lightweight parameter-change logs.

Why file-system-based, not database-based. The choice of a file system over a database or structured message bus is architecturally significant. A file system provides progressive disclosure (agents can browse directory trees, search filenames, read files incrementally), uses standard tool interfaces agents already possess (Bash, file read/write), and handles concurrency through unique naming rather than transactional coordination. This is not an implementation detail — it is a design choice that makes shared memory accessible to off-the-shelf coding agents without requiring them to learn a new query language or API. The symlink architecture (each agent's worktree links to the shared hub) makes shared memory appear as local files, so agents interact with it using the same tools they use for code.

Evidence that coordination emerges, not imposed. The paper provides quantitative evidence that cross-agent information transfer is real and effective without being hardcoded. On Kernel Engineering, 36% of attempts use another agent's commit as their parent, and these cross-agent attempts improve at nearly twice the average rate (17% vs. 9%). The majority (66%) of new records originate from a cross-agent parent. On Polyominoes, direct code transfer is rarer (12%) but even more powerful when it happens (50% improvement rate), while transfer through notes and skills is ubiquitous (87% of rounds reference knowledge from other agents). None of this cross-pollination is orchestrated — agents independently inspect the leaderboard, find promising work, and check it out. The two tasks exhibit complementary transfer modes (code-heavy on Kernel Engineering, knowledge-heavy on Polyominoes) without any task-specific configuration, demonstrating that the architecture supports emergent coordination rather than imposing a single transfer mode.

Incremental or fundamental? The shared persistent memory as the sole coordination mechanism is a fundamental architectural innovation for multi-agent systems. It replaces the need for message-passing protocols, role assignment, and task decomposition with a simpler, more general mechanism that scales to heterogeneous transfer patterns without reconfiguration. The evidence that co-evolution outperforms independent best-of-4 single-agent runs (Table 3) confirms that the coordination mechanism is causal, not just a framing device.

Innovation 3: Stagnation-Triggered Redirection as an Embedding of Meta-Cognition in the Search Loop

What is distinctive at the idea level. CORAL's heartbeat mechanism — particularly the stagnation-triggered redirection prompt — introduces explicit meta-cognitive intervention into the evolutionary search loop as a first-class design element. In fixed evolutionary search, the outer loop has no concept of "stuck" — it continues applying the same mutation-selection rules regardless of whether the population is improving. CORAL detects plateaus and prompts agents to diagnose why they are stuck, study alternative approaches (especially from other agents), and pivot to fundamentally different strategies.

The conceptual move is recognizing that detecting and escaping local optima is itself a reasoning task that can be delegated to the LLM agent, not a mechanical procedure that the framework should automate. Rather than automatically switching to a different mutation operator when progress stalls (which would be the fixed-evolutionary-search equivalent), CORAL asks the agent to reflect on why the current approach has plateaued and to propose a new direction. This preserves agent autonomy — the agent can conclude that continued optimization is still warranted — while combating the known tendency of LLM agents in long-horizon tasks to drift into repetitive patterns.

The diagnostic framing. The redirection prompt (Box C.1.2) is structured as a diagnostic workflow, not a command: "Diagnose the ceiling," "Study what's different at the top," "Choose a new direction." It frames the plateau as a signal about the quality of the current search region, not a failure of the agent. The prompt explicitly instructs the agent to study other agents' top attempts — a design choice that leverages multi-agent diversity as a source of alternative approaches. This turns the plateau from a dead end into an opportunity for knowledge transfer: what is a novel direction for the stuck agent may already be demonstrated (at least partially) in another agent's work.

Why this is more than a simple restart mechanism. The default heartbeat configuration (reflect every eval, consolidate every 10 evals, pivot after 5 non-improving evals) creates a three-tier system: continuous note-taking captures raw observations, periodic consolidation synthesizes those observations into structured knowledge, and stagnation-triggered redirection uses that accumulated knowledge (plus cross-agent inspection) to inform pivots. This layered architecture is conceptually distinct from both the fixed-loop approach (no intervention) and a naive restart-on-plateau approach (which would discard accumulated context). The consolidation heartbeat ensures that when an agent does pivot, the knowledge from its previous approach is well-documented for other agents to learn from.

Incremental or fundamental? This is an incremental but important innovation. The idea of detecting stagnation and restarting search from a different point is well-established in evolutionary computation (island models with migration, random restarts). What is novel is embedding the diagnosis and redirection in the agent's own reasoning process rather than treating it as a mechanical procedure. The significance is practical: without this mechanism, the paper's results show that single-agent runs plateau early on stress-test problems, and multi-agent co-evolution is needed to push further (Table 2). The heartbeat mechanism makes single-agent runs more robust, but its greater importance may be in sustaining multi-agent diversity — by prompting stuck agents to study others' work, it actively drives cross-pollination rather than passively hoping for it.

Innovation 4: Empirical Evidence That Multi-Agent Evolution Extends the Search Frontier Beyond 4× Compute

What is distinctive at the idea level. The paper provides causal evidence that multi-agent co-evolution produces solutions that no single agent finds, even when the single agent is given equivalent total compute. This is not a claim about parallel speedup (getting to the same result faster) or ensemble diversity (getting a better result by picking the best of independent runs). It is a claim about emergent collective capability: the interaction of agents through shared memory enables discoveries that are inaccessible to any individual agent regardless of how much time it spends exploring alone.

The key evidence is in Table 3: on Kernel Engineering, 4-agent co-evolution achieves 1103 cycles, while the best of 4 independent single-agent runs achieves 1180 cycles — a meaningful gap (6.5% better). On Polyominoes, co-evolution achieves 84.2 vs. 80.8 for independent best. On Transaction Scheduling, 4694 vs. 4629. These gaps cannot be attributed to additional compute because the independent-best comparison controls for total compute (4× single-agent budget). They cannot be attributed to ensemble selection because co-evolution's final score is from a single run, not from selecting the best across trials. The gap must come from the interaction itself — agents building on each other's intermediate discoveries in ways that no single trajectory captures.

Why this challenges a common intuition. A natural intuition is that multi-agent systems are primarily about parallelism: run N agents to explore N times faster, and the best one will find the same thing a single agent would find given N times the budget. Under this view, multi-agent systems are an engineering convenience, not a fundamental capability improvement. The paper's evidence contradicts this. Single-agent runs plateau on stress-test problems — they get stuck in local optima that multi-agent co-evolution escapes because different agents explore different regions and one agent's escape becomes another agent's starting point. The plateau is not a function of insufficient compute; it is a function of exploration diversity.

The paper quantifies this diversity directly: on Kernel Engineering, pairwise Jaccard similarity between agents' strategy vocabularies averages 0.43 — less than half of each agent's strategy vocabulary is shared with any other agent. On Polyominoes, the similarity is 0.31. These are not minor differences in implementation; they represent qualitatively different approaches to the problem. A single agent, even with 4× the compute, does not spontaneously generate this diversity because its exploration is path-dependent — early choices constrain later directions in ways that a fresh agent does not inherit.

The mechanistic story. The paper's trajectory analysis (Section 4.4.2) traces how this diversity translates to improved final scores. Cross-agent attempts — those that build on another agent's commit — improve at nearly twice the average rate on Kernel Engineering (17% vs. 9%) and produce 66% of new records. On Polyominoes, cross-agent code transfer is rarer but even more effective (50% improvement rate). This suggests a specific mechanism: agents independently discover partial solutions or useful sub-components, and cross-agent building combines these partial discoveries into solutions that no single exploration trajectory would produce. A single agent would need to discover the partial solution, recognize its value, and then discover the complementary component — all within one path-dependent trajectory. Co-evolution decouples these discoveries: agent-1 finds component A, agent-2 finds component B, and agent-3 combines them by checking out agent-1's commit and incorporating agent-2's insight from a note.

Incremental or fundamental? This is a fundamental empirical finding with implications for how open-ended discovery should be organized. If co-evolution merely provided parallel speedup, the case for multi-agent systems would be economic (faster wall-clock time) rather than capabilities-based. The evidence that co-evolution finds solutions that no single agent finds — even with equivalent total compute — changes the argument: multi-agent organization is a capability multiplier, not just a speed multiplier. The finding aligns with broader intuitions from collective intelligence research but provides specific, quantitative evidence in the context of LLM-based discovery. The limitation is that this finding is demonstrated on three stress-test tasks (Table 3) — its generality to other problem domains and agent configurations is an open question.

5. Experimental Analysis

Evaluation Methodology

Dataset. Two benchmark suites and two stress-test problems. The benchmark suites follow the setup in EvoX (Liu et al., 2026a) and TTT-Discover (Yuksekgonul et al., 2026), consisting of 6 mathematical optimization tasks (circle packing, signal processing, Erdős minimum overlap, MMD-16-2, MMD-14-3, 3rd-autocorrelation inequality) and 5 systems optimization tasks (EPLB expert placement load balancing, PRISM GPU placement, LLM-SQL column caching, transaction scheduling, Cloudcast cross-cloud transfer). The stress-test problems are Anthropic's kernel engineering — a VLIW SIMD tree-traversal task with a known best score of 1,363 cycles — and Polyominoes packing from the Frontier-CS benchmark (Mang et al., 2025), described as the hardest among all 172 problems in that benchmark. All tasks share the open-ended discovery structure: an evaluator that scores candidate solutions, but no ground-truth answers or gradients.
Base model(s). The primary backbone model is Claude Opus 4.6, used with the Claude Code agent runtime for single-agent experiments (Table 1), the multi-agent stress-test experiments (Table 2, top section), and all baselines (OpenEvolve, ShinkaEvolve, EvoX). To verify that gains generalize beyond proprietary models, the multi-agent experiments on the math and systems suites (Table 2, bottom section) use a fully open-source stack: MiniMax M2.5 (MiniMax, 2026) as the LLM with OpenCode (OpenCode, 2025) as the agent runtime. The authors argue Claude Opus 4.6 is representative of frontier coding agents capable of handling "relatively complex coding-agent workflows" (Appendix A). No internet access is provided to agents unless the research flag is explicitly enabled in the task configuration.
Metrics. Three metrics are reported per task:
- Final score: the best score achieved within the evaluation budget. This is the primary metric — the optimization target. For tasks marked ↑ in tables, higher is better; for tasks marked ↓, lower is better.
- Improvement rate (%): the fraction of evaluations that yield an improvement over the current best score for that task. Formally, (number of improved-status attempts) / (total attempts). This measures the efficiency of the search — a higher improvement rate means fewer evaluations are wasted on non-productive candidates.
- # Evals: the total number of evaluator calls required to reach the final score. This measures evaluation budget efficiency, since each evaluator call consumes wall-clock time.
For the stress-test problems, performance is also reported relative to the best known human/AI result. On Kernel Engineering, the score is linearly interpolated: 0.0 at baseline (147,734 cycles) and 1.0 at best known (1,363 cycles), so the target is to exceed 1.0. On Polyominoes, coverage percentage is the raw score, with the previous SOTA at 87%.
Baselines. Four methods are compared:
- OpenEvolve (Sharma, 2025): open-source implementation of AlphaEvolve with static elite populations and diversity maintenance.
- ShinkaEvolve (Lange et al., 2025): adaptive sampling with bandit-based selection.
- EvoX (Liu et al., 2026a): meta-evolved search strategy — the strongest fixed-evolutionary-search baseline, since its outer loop itself adapts through co-evolutionary meta-learning.
- SOTA: the best previously known results (human or AI) for each task, used as a reference ceiling rather than a directly comparable method.
All baselines receive identical seed programs, evaluators, and wall-clock budgets. For the multi-agent experiments, an additional comparison is best-of-4 independent single-agent runs (abbreviated as "Indep. Best" in Table 3), which approximates 4× compute without multi-agent coordination — this controls for the possibility that multi-agent gains are simply from running more agents in parallel.
Generation budget / compute accounting. The primary resource constraint is wall-clock time, not a fixed number of evaluations or proposals. For the math and systems suites, all methods receive a 3-hour wall-clock budget. For baselines with a fixed iteration structure (OpenEvolve, ShinkaEvolve, EvoX), runs proceed for 3 hours or 100 iterations, whichever is longer. For CORAL, to ensure fairness, runs are given "the minimum duration among all baseline runs" — this is a conservative choice that prevents CORAL from benefiting from longer runtimes. For stress-test problems, experiments terminate when there is no improvement over 100 evaluations or 2 hours, whichever comes first — these problems are hard enough that fixed iteration budgets would be insufficient. Multi-agent experiments use 4 agents with matched wall-clock time to the single-agent runs, making the comparison about organizational structure rather than total compute. The paper reports that "for a typical 3-hour single-agent run on a mathematical optimization task using Claude Opus 4.6, total API cost ranges from approximately $30–60 USD" (Appendix E.2).
Cross-validation / statistical protocol. All results on the math and systems suites are averaged over 4 independent trials. The paper does not report confidence intervals or statistical significance tests on these means. For the stress-test problems (Kernel Engineering, Polyominoes), where each run is substantially more expensive, the number of trials is not explicitly stated for Table 2 — the ablation results in Table 3 appear to be from single runs (given the integer cycle counts for Kernel Engineering). The lack of reported variance on stress-test results is a limitation for assessing the reliability of the claimed 20% cycle reduction.

Main Quantitative Results

Single-Agent Autonomous Evolution vs. Fixed Evolutionary Search (Table 1)

The headline result: CORAL with a single autonomous agent achieves the best final score on all 11 tasks, establishing new SOTA on 8 of them, while operating at much higher efficiency than any fixed evolutionary search baseline.

Final scores across tasks. Table 1 reports per-task final scores for SOTA, OpenEvolve, ShinkaEvolve, EvoX, and CORAL. On the mathematical optimization tasks:

Circle Packing: CORAL achieves 2.6360, slightly above the previous SOTA of 2.6359. EvoX, the strongest baseline, reaches 2.6320. The absolute margins are small because this task is close to saturation — all methods cluster within ~1.4% of each other.
Signal Processing: CORAL achieves 0.8229, substantially above SOTA (0.7429) and the best baseline (ShinkaEvolve at 0.8171). This is a task where the prior SOTA was relatively low, leaving room for large absolute gains.
Erdős Minimum Overlap: CORAL achieves 0.38089, essentially matching SOTA (0.38088) — a difference of 0.00001, which is likely within noise. EvoX achieves 0.38125.
MMD-16-2: CORAL matches SOTA at 12.89, as does ShinkaEvolve. EvoX achieves 12.96.
MMD-14-3: CORAL matches SOTA at 4.16. OpenEvolve achieves 4.21, while ShinkaEvolve and EvoX trail at 4.46.
3rd-Autocorrelation Inequality: CORAL matches SOTA at 1.4557. The baselines range from 1.4731 (OpenEvolve) to 1.5552 (EvoX), with EvoX performing notably worse than other baselines on this specific task.

On the systems optimization tasks:

EPLB: CORAL achieves 0.149, establishing a new SOTA above the previous 0.145. EvoX reaches 0.146; OpenEvolve and ShinkaEvolve trail at 0.127 and 0.129.
PRISM: All methods match the SOTA of 26.26 — this task appears to be at a performance ceiling that current methods cannot exceed, making it a poor discriminator.
LLM-SQL: CORAL achieves 0.731, slightly above SOTA (0.730). EvoX reaches 0.726; OpenEvolve and ShinkaEvolve trail.
Transaction Scheduling: CORAL achieves 4566, substantially above SOTA (4348). The best baseline is EvoX at 3984 — a gap of 582 points, or ~14.6% relative improvement. This is one of the tasks where CORAL's advantage is largest in absolute terms.
Cloudcast: CORAL achieves 618.4 (lower is better), establishing a new SOTA below the previous 632.7. The best baseline is EvoX at 623.5.

Cyan cells in Table 1 mark tasks where CORAL surpasses the previous SOTA. Eight of 11 tasks are marked cyan — the exceptions are MMD-16-2, MMD-14-3, and PRISM, where SOTA was already matched by one or more baselines.

Improvement rate and evaluation efficiency. The most striking pattern in Table 1 is not the final scores but the efficiency metrics. CORAL's improvement rates are 3–10× higher than any baseline across nearly all tasks:

Circle Packing: CORAL 100.0% vs. EvoX 27.1% (best baseline)
Signal Processing: CORAL 30.3% vs. EvoX 21.1%
Erdős Overlap: CORAL 36.8% vs. EvoX 27.8%
MMD-16-2: CORAL 83.3% vs. EvoX 33.3%
MMD-14-3: CORAL 75.0% vs. OpenEvolve 25.0%
3rd-Autocorrelation: CORAL 60.0% vs. ShinkaEvolve 32.4%
EPLB: CORAL 78.9% vs. ShinkaEvolve 12.6%
PRISM: CORAL 100.0% vs. OpenEvolve 31.6%
LLM-SQL: CORAL 53.3% vs. ShinkaEvolve 23.8%
Transaction Scheduling: CORAL 27.3% vs. ShinkaEvolve and EvoX at 16.0%
Cloudcast: CORAL 33.3% vs. ShinkaEvolve 20.8%

A 100% improvement rate (Circle Packing, PRISM) means that every single evaluation produced a new best score — the agent never submitted a candidate that failed to improve over its own prior best. This is an extraordinary claim and deserves scrutiny: on tasks where CORAL achieves 100% improvement rate with very few evaluations (11 on Circle Packing, 3 on PRISM), the absolute number of improvements is small, and a single lucky trajectory could produce this pattern. The paper does not report how many of the 4 independent trials achieved 100% vs. lower rates.

The number of evaluations tells a complementary story. CORAL typically converges within 5–20 evaluations, while baselines require 60–100:

Circle Packing: CORAL 11 evals vs. 48–100 for baselines
MMD-16-2: CORAL 6 evals vs. 18–97
MMD-14-3: CORAL 8 evals vs. 12–96
3rd-Autocorrelation: CORAL 5 evals vs. 37–97
PRISM: CORAL 3 evals vs. 16–32
EPLB: CORAL 19 evals vs. 60–100

The combination of high improvement rate and low evaluation count means CORAL is not merely finding better solutions — it is finding them with dramatically less trial-and-error. Since evaluation calls are the bottleneck resource (they consume wall-clock time, and in many real settings, actual computational cost), this efficiency translates directly to practical speedup.

Why EvoX is the strongest baseline, and why CORAL still wins. EvoX (Liu et al., 2026a) uses a meta-evolved search strategy — the outer loop itself adapts through co-evolution. It is conceptually the closest baseline to CORAL's philosophy of adaptive search, yet CORAL outperforms it on every task. The gap is particularly instructive on tasks where EvoX does well (e.g., Circle Packing: EvoX 2.6320 vs. CORAL 2.6360) vs. tasks where EvoX does poorly (e.g., 3rd-Autocorrelation: EvoX 1.5552 vs. CORAL 1.4557). This variability suggests that EvoX's meta-learned strategy — while adaptive — still operates at the level of hyperparameter selection rather than per-decision reasoning, and its adaptation may not transfer well across qualitatively different optimization landscapes. CORAL's agent-level reasoning can adjust to task-specific characteristics in ways that a fixed meta-evolved policy cannot.

Multi-Agent Co-Evolution vs. Single-Agent CORAL (Table 2)

The headline result: 4-agent co-evolution pushes performance beyond single-agent CORAL on most tasks, with the largest gains on stress-test problems where single-agent runs plateau early. Table 2 splits results into three sections: stress-test problems using Claude Code + Opus 4.6 (top), and math/systems tasks using OpenCode + MiniMax M2.5 (bottom two).

Stress-test problems (Claude Opus 4.6).

Kernel Engineering: 4-agent co-evolution achieves 1,103 cycles, an 18.3% reduction from the single-agent result of 1,350 cycles and a 19.1% reduction from the previous best known of 1,363 cycles. This is the paper's flagship result. However, the improvement rate drops from 43.0% (single-agent) to 9.0% (4-agent), and the number of evaluations increases from 56 to 596 — more than 10×. This tradeoff is worth noting: multi-agent evolution explored much more broadly (596 evaluations across 4 agents vs. 56 for a single agent), at the cost of a lower per-evaluation success rate. The large evaluation count suggests that many agents' attempts were unproductive individually, but the collective exploration eventually found a breakthrough.
Polyominoes: 4-agent achieves 84.2, a 5.0% improvement over single-agent (80.2), but still below the previous SOTA of 87.0. The improvement rate drops from 42.4% to 19.4%, and evaluations increase from 33 to 67. Notably, with web search enabled, CORAL achieves 89.4 (Appendix B.1, Figure 3), surpassing SOTA — but the main Table 2 reports the no-web-search result for fairness, since web search provides access to external human knowledge that goes beyond the evolutionary search paradigm.

Math and systems tasks (MiniMax M2.5 + OpenCode). The open-source results serve primarily as a generalization check — do multi-agent gains persist with weaker models and a different agent runtime? The answer is mixed:

Multi-agent improves over single-agent on 8 of 11 tasks, with gains (%) ranging from 0.15% (MMD-16-2, essentially tied) to 20.80% (Cloudcast).
On 3 tasks, multi-agent performs worse: MMD-16-2 (12.89 vs. 12.91 — single-agent is actually slightly better, though the difference is 0.02, likely within noise), Cloudcast measured in a different direction (672.8 for 4-agent vs. 849.4 for 1-agent — this is a 20.8% improvement in the score metric, but Cloudcast is a minimize task where lower is better, and 4-agent's 672.8 is worse than SOTA 632.7; the single-agent 849.4 is anomalously poor, suggesting a failed run pulling the average down), and transaction scheduling (3774 vs. 3704 — single-agent is slightly better, 1.89% difference).
Improvement rates and evaluation counts vary widely across tasks. On EPLB, 4-agent achieves 65.4% improvement rate vs. 50.0% for single-agent, using more evaluations (26 vs. 6). On MMD-14-3, 4-agent achieves only 15.0% vs. 63.6% for single-agent, with many more evaluations (80 vs. 11).

The key takeaway from the open-source table is not the specific per-task numbers (which are noisy with only 4 trials) but the direction of the effect: multi-agent co-evolution generally helps, and the gains are not artifacts of Claude Opus 4.6's specific capabilities. The MiniMax M2.5 results are substantially weaker than the Claude Opus 4.6 results on the same tasks (compare single-agent MiniMax M2.5 on Circle Packing, 2.3531, to Claude Opus 4.6, 2.6360, in Table 1 — the model gap dominates). But within the MiniMax M2.5 regime, multi-agent still provides a boost.

Comparison to SOTA. The open-source 4-agent results match or approach SOTA on several tasks: Circle Packing (2.5391 vs. SOTA 2.6359 — close), Signal Processing (0.7383 vs. SOTA 0.7429 — very close), MMD-16-2 (12.89 — matches SOTA), and LLM-SQL (0.730 — matches SOTA). This is notable because SOTA values were set by the best of all prior methods (including those using stronger models). That a MiniMax M2.5 4-agent run can approach or match these numbers suggests the multi-agent architecture partially compensates for a weaker base model.

Trajectory Analysis of Autonomous Self-Evolution (Section 4.4.1, Tables 4 and 5)

The headline result: local verification and knowledge accumulation are strongly associated with successful improvement, but their importance varies dramatically across task types.

Local testing. Across standard tasks (Table 4, "Average" row), 24% of attempts include local testing, and among those, the improvement rate is 37% — substantially above the 24% average improvement rate across all attempts. This means local testing is not just correlated with improvement (which could be because agents only test on promising candidates); it increases the likelihood that a submission succeeds, presumably by catching bugs before the formal evaluation. On specific tasks (Table 5):

Circle Packing: 100% local test rate, 100% improvement rate. Every attempt was tested locally, and every evaluation succeeded — a strong pattern, though with only 11 total attempts (Table 1), this may not generalize.
Transaction Scheduling: 61% local test rate, 20% improvement rate among those tested. This is lower than the task's average 27.3% improvement rate (Table 1), suggesting local testing is used broadly (including on attempts that ultimately fail), which is the expected behavior.
Kernel Engineering (Table 4, Advanced): 57% local test rate with 47% improvement rate among tested attempts, vs. 43% average. Local testing catches compilation failures before evaluation consumption — a critical efficiency gain on a task where each evaluation simulates a VLIW kernel.
PRISM: 0% local test rate. The evaluator generates randomized tests that cannot be replicated locally, making pre-submission testing impossible. This is a natural limitation — not all tasks support local verification.

Knowledge accumulation. On standard tasks (Table 4, Standard Average), agents create only 0.05 knowledge artifacts per attempt, and knowledge access yields 26% improvement rate vs. 24% average — only a 2 percentage point gain. On advanced tasks, the pattern shifts dramatically: agents create 0.55–0.68 knowledge artifacts per attempt, and knowledge access is associated with 38–55% improvement rates vs. 30–43% averages. On Kernel Engineering, attempts that access knowledge improve at 55%, substantially above the 43% baseline.

The paper's interpretation is that knowledge artifacts on standard tasks are "lightweight progress logs" (parameter change records), while on advanced tasks they capture "reusable insights" (architectural bottlenecks, documented failure modes). This is a post-hoc characterization — the paper does not provide a systematic taxonomy of note quality across tasks — but it aligns with the quantitative pattern: if notes are shallow, reading them provides little marginal benefit over reading the raw evaluation scores; if notes capture genuine architectural insights, they enable an agent to understand why a solution worked without reverse-engineering the code.

Attempt inspection. On standard tasks, agents inspect prior attempts in 25% of rounds; on advanced tasks, 17–47%. The inspection rate on Kernel Engineering (47%) is particularly high — nearly half of all attempts begin with the agent examining prior work. This makes sense for a task where the solution space involves complex architectural tradeoffs (VALU vs. ALU balancing, dependency graph optimization) where understanding previous design decisions is essential.

Why Multi-Agent Organization Helps (Section 4.4.2)

The headline result: multi-agent co-evolution improves performance through three mechanisms — cross-agent information transfer, exploration diversity, and balanced contribution — with the transfer mode adapting to task characteristics.

Cross-agent information transfer. On Kernel Engineering (596 total attempts):

36% of attempts use another agent's commit as their parent.
These cross-agent attempts improve at 17% vs. 9% for all attempts — nearly double the baseline improvement rate.
66% of new records (score improvements that set a new best) originate from a cross-agent parent.

On Polyominoes (67 total attempts):

Direct code transfer (building on another agent's commit) is rarer: 12% of attempts.
When it does occur, the improvement rate is 50% vs. 19% average.
Transfer through knowledge (notes, skills) is the dominant mode: 87% of rounds reference knowledge committed by other agents.

The complementary transfer modes — code-heavy on Kernel Engineering, knowledge-heavy on Polyominoes — emerge without task-specific configuration. Kernel Engineering involves optimizing a single kernel builder program where direct code changes are the natural unit of improvement; Polyominoes involves packing strategies where design principles (what shapes fit together) are more transferable than specific packing configurations.

Exploration diversity. The paper quantifies diversity by extracting strategy keywords from attempt titles and computing pairwise Jaccard similarity between agents' strategy vocabularies:

Kernel Engineering: average pairwise similarity 0.43
Polyominoes: average pairwise similarity 0.31

In both cases, more than half of each agent's strategy vocabulary is unique to that agent. This is an emergent property — agents are initialized identically with no role specialization. The diversity arises from stochastic exploration decisions compounding over time, amplified by each agent reading different subsets of the shared memory at different times.

Contribution balance. On Kernel Engineering, all four agents contribute relatively evenly: 130–165 attempts each, 10–16 improvements each, all four independently reach the best score of 1103 cycles. Records are split 14/15/10/15 across agents. Leader tenure is more skewed: agent-1 holds the best score for 45% of the run. On Polyominoes, contributions are less balanced: agent-3 sets 6 of 13 records, and agent-4 leads for 34% of the total time. The paper does not investigate why contribution balance differs across tasks — possible explanations include differences in task structure (Kernel Engineering's optimization landscape may have multiple accessible paths to the same optimum; Polyominoes may have a narrower path that one agent happened to find earlier) or stochastic variation in the small sample of trials.

Ablation Studies and Robustness Checks

The paper reports two ablation studies in Table 3, both conducted on three stress-test tasks (Kernel Engineering, Polyominoes, Transaction Scheduling) using Claude Code + Opus 4.6. All results appear to be from single runs rather than averaged over trials, which limits statistical reliability.

Knowledge accumulation (1-agent): Disabling note and skill creation tests whether the shared persistent memory causally improves performance or is merely correlated with it. Results:

Kernel Engineering: without knowledge, cycles degrade from 1,350 to 1,601 — an 18.6% regression. This is the largest effect, and it aligns with the trajectory analysis showing that knowledge access on Kernel Engineering is associated with 55% improvement rate (Table 4).
Polyominoes: score drops from 80.2 to 77.3 — a 3.6% regression.
Transaction Scheduling: score drops from 4,566 to 4,444 — a 2.7% regression.

All three tasks show degradation, confirming that knowledge artifacts causally contribute to search quality. The magnitude of the effect varies substantially (2.7% to 18.6%), suggesting that the importance of explicit knowledge externalization depends on task characteristics — on tasks where improvement requires understanding complex architectural tradeoffs (Kernel Engineering), knowledge is critical; on tasks where improvement comes from incremental parameter tuning (Transaction Scheduling), the effect is smaller.

A limitation of this ablation: disabling note/skill creation removes the agent's ability to write knowledge, but the agent can still read the raw evaluation scores and feedback in attempts. The ablation therefore isolates the value of interpretable, agent-authored knowledge over and above raw evaluation data. The fact that this matters (even modestly on Polyominoes and Transaction Scheduling) supports a key design claim: storing what agents think about results is more useful than storing only the results themselves.

Co-evolution vs. independent runs (4-agent): This ablation tests whether multi-agent gains come from coordination through shared memory or simply from running more agents — the equivalent of an ensemble. It compares 4-agent co-evolution (agents share memory) against the best score from 4 independent single-agent runs (each agent operates in isolation, with no shared memory). Results:

Kernel Engineering: co-evolution achieves 1,103 cycles vs. 1,180 for the independent best — a 6.5% gap. This is the strongest evidence that coordination matters beyond compute scaling.
Polyominoes: co-evolution achieves 84.2 vs. 80.8 — a 4.2% gap.
Transaction Scheduling: co-evolution achieves 4,694 vs. 4,629 — a 1.4% gap.

The gap is on all three tasks, though the magnitude varies. The independent-best comparison is a conservative baseline: it assumes you can run 4 agents in isolation and pick the best result, which gives you 4× the exploration budget of a single agent. Co-evolution outperforms this baseline, meaning the interaction between agents — building on each other's intermediate discoveries — enables solutions that no single trajectory finds even with equivalent total exploration. This is the paper's strongest causal evidence for the value of the multi-agent architecture specifically, as opposed to simply scaling compute.

Missing ablations. Several ablations that would strengthen the paper's claims are not reported:

Heartbeat configuration sensitivity: all experiments use the default heartbeat settings (reflect every eval, consolidate every 10 evals, pivot after 5 non-improving evals). How sensitive are results to these thresholds? Would a less frequent consolidation heartbeat degrade knowledge quality? Would a more aggressive pivot trigger (after 3 non-improving evals instead of 5) prematurely interrupt productive optimization?
Number of agents: all multi-agent experiments use 4 agents. How do 2-agent, 8-agent, and 16-agent configurations perform? Does performance plateau or degrade with more agents (due to shared memory noise or exploration redundancy)?
Shared memory components: the knowledge accumulation ablation disables all notes and skills. What is the relative contribution of notes vs. skills specifically? Does the "never worked" folder matter, or is it primarily synthesis notes that drive improvement?
Agent autonomy over specific stages: CORAL gives agents autonomy over all four stages (RETRIEVE, PROPOSE, EVALUATE scheduling, UPDATE). Which stages' autonomy matters most? For example, would fixed RETRIEVE (parent selection by the framework) with autonomous PROPOSE/EVALUATE/UPDATE still outperform pure fixed search?
Single-agent with 4× time vs. 4-agent: the co-evolution ablation compares 4-agent against best-of-4 independent single-agent runs. A stronger comparison would be a single agent given 4× the wall-clock budget — this tests whether the benefit is from parallelism specifically or from the total exploration budget.

Generality to other model families. All ablation results are on Claude Opus 4.6. The multi-agent co-evolution results on MiniMax M2.5 (Table 2, bottom) provide some evidence of generality, but no MiniMax M2.5 ablations are reported. It is unknown whether the knowledge accumulation and co-evolution effects replicate with weaker models, which may lack the reasoning capability to write useful notes or to effectively build on other agents' work.

Evaluator bug fixes. The paper documents four evaluator bugs discovered and corrected during integration (Appendix D.3, Table 8). These corrections are important for reproducibility — without them, solutions with specific failure modes could achieve artificially inflated scores on PRISM (crashes silently skipped), Transaction Scheduling (invalid schedules scored >0), EPLB (dropped experts ignored), and LLM-SQL (type mismatches causing crashes). The bug fixes were applied uniformly to all methods (CORAL and baselines), so they do not bias the comparison, but they highlight a general challenge: open-ended discovery with imperfect evaluators can reward solutions that exploit evaluator flaws rather than solving the underlying problem. CORAL's grader isolation prevents agents from inspecting evaluator code, but it cannot prevent agents from discovering evaluator bugs through trial and error — a particularly effective agent might learn that certain types of invalid outputs happen to produce high scores.

Critical Assessment

Claim 1: A single autonomous agent outperforms fixed evolutionary search baselines, achieving 3–10× higher improvement rates and up to 10× fewer evaluations.

This claim is strongly supported by Table 1, with qualifications about the magnitude of the improvement rate metric.

The evidence: on all 11 tasks, CORAL achieves the best final score, with improvement rates 3–10× higher than any baseline on most tasks. The "up to 10× fewer evaluations" claim holds for several tasks (MMD-16-2: 6 vs. 97; MMD-14-3: 8 vs. 96; 3rd-Autocorrelation: 5 vs. 97). The consistency across 11 diverse tasks and 4 independent trials per task makes this the paper's most robust finding.

However, the improvement rate metric deserves scrutiny. A 100% improvement rate (Circle Packing, PRISM) means every evaluation improved over the agent's own prior best. But on PRISM, CORAL made only 3 evaluations total — a "100% improvement rate" with 3 evaluations is less informative than the same rate with 50 evaluations, because the confidence interval is wide. The metric conflates efficiency (fewer total evaluations) with effectiveness (higher fraction of evaluations improving), which are different things. CORAL's agents make far fewer evaluations than baselines (3–22 typically vs. 60–100), which inflates the improvement rate: you need fewer improvements to maintain a high rate when the denominator is small. The fairer interpretation is that CORAL achieves similar final scores with substantially fewer evaluations, which is a genuine efficiency gain regardless of how the improvement rate is calculated.

A substantial weakness: all baselines and CORAL use the same Claude Opus 4.6 model. The paper's claim is about the organizational paradigm (autonomous vs. fixed search), but the baselines use a fundamentally different agent architecture — the baselines call the LLM as a stateless mutation operator, while CORAL uses Claude Code, a full coding agent with tool use, file editing, and multi-turn reasoning. The performance gap could be partly attributable to Claude Code's superior code-generation and debugging capabilities (independent of the evolutionary organization) rather than to autonomy over search decisions. An ideal baseline would use Claude Code as the proposal mechanism within a fixed evolutionary search loop — the LLM gets the same coding agent capabilities, but the outer loop still controls retrieval, evaluation scheduling, and population management. The paper does not run this baseline, making it impossible to isolate the effect of autonomy from the effect of better coding infrastructure.

Claim 2: Multi-agent co-evolution extends the search frontier beyond what independent agents can achieve with equivalent total compute.

This claim is supported with qualifications based on Table 2 and Table 3.

The evidence: 4-agent co-evolution outperforms single-agent on most tasks (Table 2), and critically, co-evolution outperforms the best of 4 independent single-agent runs on three stress-test tasks (Table 3). The latter comparison controls for total compute, isolating the effect of coordination through shared memory. The gap is meaningful on Kernel Engineering (6.5%) and Polyominoes (4.2%), and small but present on Transaction Scheduling (1.4%).

The qualifications: the ablation results in Table 3 appear to be from single runs (the integer cycle counts for Kernel Engineering — 1,103, 1,180, 1,350, 1,601 — suggest single-trial outcomes rather than averaged means). With one trial per condition, the observed gap between co-evolution and independent best could be due to stochastic variation rather than a causal effect of coordination. The paper would need multiple trials per ablation condition with reported variance to support strong causal claims. This is a significant methodological gap given that the co-evolution vs. independent comparison is the paper's central evidence for the multi-agent architecture specifically.

Additionally, the stress-test tasks (Kernel Engineering, Polyominoes) are evaluated with Claude Opus 4.6, a very strong model. The paper does not report the co-evolution vs. independent ablation for weaker models (MiniMax M2.5). It is plausible that with weaker models, the benefits of coordination diminish — if agents struggle to write useful notes or to effectively build on others' code, the shared memory adds noise without benefit. This is an untested boundary condition.

Claim 3: Both agent autonomy and multi-agent co-evolution causally contribute to performance gains, not just additional compute.

This claim is supported in two parts, with asymmetric evidence quality.

For knowledge accumulation: the ablation in Table 3 (with vs. without knowledge) shows that removing notes and skills degrades performance on all three tested tasks. This is causal evidence that the knowledge accumulation mechanism matters. However, three tasks is a small sample (out of 13 total tasks evaluated), and the magnitude varies substantially (2.7% to 18.6%). The paper does not investigate why the effect varies — is it task complexity, task type, or the number of evaluations?

For multi-agent co-evolution: the best-of-4 comparison in Table 3 provides causal evidence that coordination matters beyond compute scaling. The same caveats about single-trial reliability apply here.

A missing piece: the paper does not ablate the specific components of autonomy — for example, comparing CORAL with autonomous RETRIEVE against a variant where RETRIEVE is fixed (the agent always sees the top-k candidates) but other stages are autonomous. This makes it difficult to attribute the gains to specific design decisions (heartbeat, shared memory structure, CLI tools) rather than to the general paradigm of "let the agent decide more things."

Claim 4: CORAL establishes new state-of-the-art on 8 tasks and improves the best known kernel engineering score by 20%.

This claim is supported by Table 1 (8 tasks marked cyan, surpassing prior SOTA) and Table 2 (Kernel Engineering at 1,103 cycles vs. 1,363 prior best known). However, "state-of-the-art" in this context means "best among the methods compared in this paper" — it does not mean CORAL universally dominates all possible approaches on these tasks. The SOTA values in Table 1 are drawn from the SkyDiscover repository and the specific baselines run, but there may be unpublished or concurrent results that exceed these numbers (the paper itself notes concurrent open-source projects like AutoResearch and Hive exploring similar directions).

The kernel engineering result (1,103 cycles, a 19.1% improvement over the previous best known of 1,363) is genuinely impressive — it represents a substantial advance on a well-studied task with a known human best. However, the paper does not report the variance on this result. With a single multi-agent run producing this number, it is unclear whether CORAL reliably achieves ~1,100 cycles or whether this was a particularly fortunate run. For a result that the paper's abstract and introduction prominently feature, this is a significant omission.

Genuine weaknesses that the experiments do not address:

Single model family for ablations: all causal ablation evidence (Table 3) uses Claude Opus 4.6. The paper does not demonstrate that the conclusions about knowledge accumulation and co-evolution generalize to weaker models or different model families.
Task diversity is concentrated in one benchmark family: the math and systems suites both come from the SkyDiscover/ADRS repository (Liu et al., 2026b), which may have systematic properties (evaluator design, task structure, seed program quality) that favor or disfavor certain approaches. The stress-test tasks (Kernel Engineering, Polyominoes) are more distinct but only two in number.
No ablation of the heartbeat mechanism: the heartbeat is a core architectural component but is never varied. We cannot assess whether the specific default configuration (reflect every eval, consolidate every 10 evals, pivot after 5 non-improving evals) is close to optimal, or whether any heartbeat at all is necessary for the reported performance.
The single-agent CORAL vs. best-of-4 independent comparison would strengthen the autonomy claim: if a single agent with 4× wall-clock time (matching the total compute of 4 agents) matches or exceeds the 4-agent performance, then the multi-agent benefit is purely about parallelism and wall-clock time, not about coordination. This experiment is not run.
Evaluator cost and latency are unaccounted for in the "efficiency" narrative: the paper reports improvement rate (fraction of evaluations that improve) and number of evaluations, but does not measure total wall-clock efficiency including the agent's reasoning time between evaluations. A fixed evolutionary search baseline calls the LLM once per evaluation; CORAL agents engage in extended multi-turn reasoning, local testing, and knowledge management before calling coral eval. The 10× reduction in evaluation count may come with increased per-evaluation overhead — the paper's reported API costs ($30–60 per 3-hour run) suggest the overhead is not prohibitive, but a direct wall-clock-to-performance comparison is not provided.
The four evaluator bug fixes (Appendix D.3) raise questions about evaluator quality: if the original evaluators had bugs that produced incorrect scores for certain edge cases, the absolute performance numbers on those tasks (PRISM, Transaction Scheduling, EPLB, LLM-SQL) should be interpreted cautiously. The fixes were applied uniformly, so relative comparisons remain valid, but a method that is more effective at finding evaluator bugs (as opposed to solving the underlying problem) could appear artificially strong. CORAL's agent-driven exploration, which tries diverse strategies, might be more likely to encounter evaluator edge cases than a fixed search with narrower exploration.

What experiments would have strengthened the paper:

Baseline with Claude Code in a fixed evolutionary loop: isolate the effect of the coding agent infrastructure from the effect of autonomous search control.
Ablation of each heartbeat type separately: how much does reflection contribute vs. consolidation vs. stagnation redirection?
Multiple trials for the Table 3 ablation: enable statistical comparison of co-evolution vs. independent best rather than point estimates.
Scaling the number of agents: 2, 4, 8, 16 — does performance plateau or degrade?
Single-agent with equivalent total compute: would a single agent given 12 hours match the 4-agent × 3-hour performance?
Cross-model replication of the knowledge accumulation and co-evolution ablations: do the effects hold for MiniMax M2.5 or other open-source models?

6. Limitations and Trade-offs

The Difficulty Estimation Cost Problem: Knowledge Accumulation Requires Overhead That Is Not Amortized in the Efficiency Metrics

The assumption or constraint. CORAL's headline efficiency metrics — 3–10× higher improvement rates, up to 10× fewer evaluations — measure only formal evaluator calls (coral eval). They exclude the substantial computational overhead of the agent's own reasoning, local testing, knowledge management, and heartbeat processing between evaluations. Unlike fixed evolutionary search, where each evaluation cycle is a single LLM call (generate a candidate), CORAL agents engage in extended multi-turn interactions: browsing the leaderboard, inspecting prior attempts, reading notes, writing reflections, running local tests, and iteratively debugging before calling coral eval. The paper acknowledges this indirectly in Appendix E.2:

"CORAL agents typically perform fewer evaluation calls than structured baselines within the same wall-clock budget, because each agent step involves reasoning and implementation before submission."

The paper reports API costs of $30–60 USD per 3-hour single-agent run (Appendix E.2), which indicates the overhead is real but does not normalize performance by total cost or total tokens consumed. The improvement rate metric further compounds this: because improvement rate divides improvements by number of evaluations, and CORAL makes far fewer evaluations (3–22 typically vs. 60–100 for baselines), a small number of successful evaluations can produce very high rates that are sensitive to small denominators.

The consequence. A practitioner comparing CORAL to a fixed evolutionary search baseline cannot determine from the paper's metrics alone whether CORAL achieves better per-dollar or per-wall-clock-second performance, only better per-evaluator-call performance. If evaluator calls are cheap (e.g., a mathematical function evaluation taking milliseconds) but the agent's reasoning between calls is expensive (minutes of LLM inference), the per-evaluation efficiency could be misleading — CORAL might spend 10× more total compute to achieve 10× fewer evaluations, yielding no net efficiency gain. Conversely, if evaluator calls dominate cost (e.g., simulating a GPU kernel architecture taking tens of seconds), reducing evaluations by 10× is genuinely valuable regardless of agent overhead. But the paper provides no breakdown of where time and tokens are spent across tasks, leaving this tradeoff opaque.

What evidence exists in the paper. Table 1 reports improvement rate and number of evaluations but not wall-clock time per evaluation or total tokens per run. The 3-hour wall-clock budget is used as fairness constraint across methods, but within that budget CORAL makes far fewer evaluations (e.g., 11 for Circle Packing vs. 48–100 for baselines), suggesting that agent reasoning consumes most of the 3 hours while baselines spend their time on rapid evaluation cycles. The trajectory analysis (Table 4) shows 24% local test rate on standard tasks and 57% on Kernel Engineering — every locally tested attempt incurs additional LLM inference that is not captured in the evaluation count. The paper does not measure or report total inference cost (tokens, API dollars) as a function of final score for any task.

Mitigation status. The paper does not address this directly. Appendix E.2 provides approximate per-run costs but does not normalize improvement by cost. The authors frame the improvement rate as a measure of "evaluation efficiency" rather than total compute efficiency, which is a reasonable metric when evaluation calls are the scarce resource, but the paper does not argue that this is the case — it does not characterize which of the evaluated tasks have expensive evaluators vs. cheap evaluators. This is a notable gap for a paper whose central claim is about efficiency.

The Hardest Problems Remain Unsolved: The Method Amplifies Existing Capability but Cannot Create It

The assumption or constraint. CORAL, like all test-time search methods, requires the base model to possess some non-trivial capability on the target problem — the agent must be able to generate solutions that are sometimes on the right track, even if suboptimal. If the base model's initial attempts are essentially random with respect to the evaluation metric, no amount of autonomous iteration will converge to a strong solution, because the agent receives no informative feedback signal. The paper acknowledges this implicitly in its choice of models (Claude Opus 4.6, a frontier coding model) and in the observation that on some tasks, even CORAL matches but does not exceed prior SOTA.

The paper does not explicitly characterize what level of base-model competence is required, nor does it evaluate CORAL on problems where the base model is known to fail completely. All tasks in the evaluation suite are ones where prior methods have achieved non-trivial scores — the seed programs and base models provide a starting point above zero. This is a selection bias: the benchmark tasks are filtered to those where LLM-based approaches are already somewhat viable.

The consequence. CORAL cannot be expected to work on genuinely novel problems where the base model has no relevant knowledge or heuristic intuition. This limits the scope of "open-ended discovery" to tasks within the base model's approximate capability envelope — the method can refine, optimize, and discover novel combinations within that envelope, but it cannot cross fundamental capability boundaries that the model has not acquired during pretraining. This is directly analogous to the finding in test-time compute scaling papers (referenced in the prior sections of this analysis) that test-time strategies fail on the hardest difficulty bin where the base model's pass@1 is near zero.

A deployed system using CORAL on a stream of problems would need a way to detect when a problem is fundamentally outside its capability range and avoid wasting compute on fruitless exploration — but CORAL provides no such capability boundary detection mechanism. The stagnation-triggered pivot heartbeat might eventually exhaust approaches and stop improving, but there is no explicit "give up and flag for human" mechanism, which could lead to unbounded resource consumption on impossible problems.

What evidence exists in the paper. The paper does not systematically evaluate CORAL on problems where the base model is known to fail. The closest evidence is the variation in improvement rates across tasks: PRISM achieves 100% improvement rate (3/3 evaluations improved), while Transaction Scheduling achieves only 27.3% — suggesting that some tasks are intrinsically harder for the agent, but not revealing whether there are tasks where 0% improvement rate would occur. The MiniMax M2.5 results (Table 2, bottom) show substantially lower absolute performance than Claude Opus 4.6, and on Cloudcast the single-agent MiniMax M2.5 result (849.4) is anomalously poor compared to SOTA (632.7, lower is better), hinting that with a weaker model, CORAL may struggle to make progress. But this is not systematically explored — there is no experiment showing CORAL's performance as a function of base-model capability on a controlled difficulty ladder.

Mitigation status. Not addressed in the paper. Appendix A notes that CORAL "relies on frontier foundation models that can handle relatively complex coding-agent workflows" but frames this as a deployment limitation (difficulty running locally) rather than a capability boundary. The paper does not discuss how to determine whether a given problem is within scope for a given model, nor does it propose fallback mechanisms for problems that exceed the model's capability. This is a significant gap for practitioners who need to decide whether CORAL is appropriate for their specific problem domain.

Task and Model Diversity Is Narrow: All Causal Evidence Comes from Three Tasks on One Proprietary Model

The assumption or constraint. The paper's strongest causal claims — that knowledge accumulation improves performance, and that multi-agent co-evolution outperforms independent agents — are supported by ablation experiments (Table 3) conducted on only three stress-test tasks (Kernel Engineering, Polyominoes, Transaction Scheduling) using a single model (Claude Opus 4.6) and a single agent runtime (Claude Code). The single-agent vs. fixed-search comparison (Table 1) covers 11 tasks, also all on Claude Opus 4.6. The generalization experiment (Table 2, bottom) tests multi-agent co-evolution on 11 tasks with MiniMax M2.5 + OpenCode, but does not include the ablations — we cannot determine whether the knowledge accumulation and co-evolution mechanisms are causal with the open-source stack, or whether the MiniMax M2.5 multi-agent gains are attributable to the same mechanisms.

The paper acknowledges the model limitation in Appendix A:

"CORAL relies on frontier foundation models that can handle relatively complex coding-agent workflows, which makes full deployment on local devices difficult."

But this acknowledges the model strength requirement, not the narrowness of the ablation evidence base. The paper does not discuss whether the observed mechanisms (knowledge accumulation, cross-agent transfer) depend on Claude Opus 4.6's specific capabilities — for example, its ability to write high-quality analytical notes, to identify architectural bottlenecks in GPU kernels, or to build on another agent's code without introducing bugs.

The consequence. A practitioner cannot confidently generalize the paper's causal findings to other model families (GPT-4, Gemini, open-source models), to other agent runtimes (SWE-Agent, OpenHands), or to other task domains (drug discovery, materials design, theorem proving). The mechanisms that drive CORAL's performance — writing useful notes, extracting skills from experience, building on others' code — depend on the base model's reasoning, code generation, and self-reflection capabilities. A weaker model might write superficial notes that add noise rather than signal to the shared memory, or might introduce bugs when modifying another agent's code, making cross-agent building counterproductive. The paper provides no evidence about where the "capability threshold" lies for CORAL's mechanisms to function.

The task diversity is similarly narrow. The mathematical and systems optimization suites share a common origin (SkyDiscover/ADRS repository, Liu et al., 2026b) and a common structure (optimize a Python program against a provided evaluator). The stress-test tasks (Kernel Engineering, Polyominoes) are more distinct but only two in number. It is unknown whether CORAL's architecture generalizes to open-ended discovery tasks with fundamentally different structure: tasks with stochastic evaluators (where the same solution produces different scores on different runs), tasks with multi-objective or partially specified evaluation (where the "best" solution is ambiguous), or tasks requiring physical experimentation (where evaluations are slow, expensive, and non-deterministic).

What evidence exists in the paper. The Table 2 generalization experiment shows that multi-agent co-evolution with MiniMax M2.5 improves over single-agent on 8 of 11 tasks, but the improvement margins are smaller and the single-agent baselines are weaker by roughly 5–15% compared to the Claude Opus 4.6 results in Table 1 on the same tasks. This suggests that the multi-agent architecture provides some benefit independent of model strength, but the lack of ablations on MiniMax M2.5 leaves open whether the mechanism of benefit is the same (knowledge accumulation, cross-agent transfer) or whether the weaker model benefits primarily from parallel exploration without effective knowledge reuse. The MiniMax M2.5 co-evolution results are also noisier: on some tasks (MMD-16-2, Transaction Scheduling), 4-agent performs slightly worse than 1-agent, though the differences are within what might be expected from 4-trial variance.

Mitigation status. Partial. The paper includes the MiniMax M2.5 experiment specifically to test generalization beyond proprietary models, and the multi-agent advantage generally persists (8/11 tasks). However, the authors do not replicate the causal ablations (knowledge accumulation removal, co-evolution vs. independent) with the open-source stack, which would be necessary to confirm that the benefits arise from the same mechanisms. The paper does not discuss task domain generalizability beyond noting that the stress-test problems are "challenging" and "hardest among all 172 problems in the benchmark" (Section 4.1).

The Multi-Agent Contribution Balance and Emergent Coordination Are Underexplored: We Cannot Predict When Co-Evolution Will Help

The assumption or constraint. The paper's multi-agent experiments use exactly 4 agents in all cases, with identical initialization (same CORAL.md, same model, same seed program access). The paper reports that contribution balance varies across tasks — on Kernel Engineering, all four agents contribute roughly equally; on Polyominoes, agent-3 sets 6 of 13 records and agent-4 leads 34% of the time — but does not investigate why or characterize what task properties produce balanced vs. skewed contributions. More fundamentally, the paper does not explore how sensitive the multi-agent benefit is to the number of agents, to their heterogeneity, or to the shared memory configuration.

The paper acknowledges this in Appendix A:

"Multi-agent evolution currently lacks bootstrapped heterogeneity: all agents are initialized identically and given access to the same information. Future work could inject distinct personalities, roles, or private information into different agents to encourage greater behavioral diversity."

But this frames heterogeneity as a potential future improvement, not as a gap in understanding the current system. The paper does not discuss whether there are tasks where adding more agents would degrade performance (due to shared memory noise, redundant exploration, or conflicting advice in notes), nor whether the 4-agent configuration was chosen based on optimization or convenience.

The consequence. A practitioner deploying CORAL on a new task cannot determine how many agents to allocate, whether to introduce heterogeneity, or how to configure shared memory (e.g., should agents share all notes or maintain private working notes with periodic public summaries?). The paper provides no guidance on whether the multi-agent benefit saturates at some agent count, grows monotonically, or exhibits diminishing returns. If the benefit saturates quickly (e.g., 2 agents capture most of the gain), deploying 8 or 16 agents would waste compute. If the benefit requires a minimum number of agents to achieve meaningful exploration diversity, deploying 2 agents might be insufficient.

Worse, the paper's current evidence suggests that multi-agent benefit varies substantially across tasks — large on Kernel Engineering (6.5% gap between co-evolution and independent best), small on Transaction Scheduling (1.4% gap) — but provides no diagnostic to predict this variance. Running 4-agent co-evolution on a new task is expensive (3–4× the single-agent cost, per Appendix E.2), and the practitioner has no way to estimate the expected return on that investment.

What evidence exists in the paper. The paper reports multi-agent results for 4 agents only. The co-evolution vs. independent-best ablation (Table 3) establishes that coordination matters on three tasks, but with single-trial results (integer cycle counts on Kernel Engineering) that do not support statistical characterization of the effect size or its variance. The contribution balance analysis (Section 4.4.2) describes what happened in the specific runs but does not explain why or assess whether different runs of the same task would show different balance patterns. The paper does not ablate agent count (2, 4, 8) or heterogeneity (different prompts, different models, different access to shared memory).

Mitigation status. Not addressed. Appendix A notes the lack of bootstrapped heterogeneity as a limitation and suggests distinct personalities/roles/private information as future work, but does not discuss agent count scaling or contribution balance diagnostics. The paper treats the 4-agent configuration as a fixed architectural parameter rather than a variable to be studied.

The Heartbeat Mechanism Is Never Ablated or Tuned: A Core Architectural Component Has Unknown Sensitivity

The assumption or constraint. CORAL's heartbeat mechanism — periodic reflection, consolidation, and stagnation-triggered redirection — is described as a core architectural component that "promotes explicit memory formation and reduces myopic local search" (Section 3.3). The specific default configuration (Table 7: reflect every 1 eval locally, consolidate every 10 evals globally, pivot after 5 non-improving evals locally) is used in all experiments. The paper never ablate the heartbeat mechanism — no experiment compares CORAL-with-heartbeat to CORAL-without-heartbeat — and never varies the heartbeat configuration to assess sensitivity to trigger frequency, threshold, or prompt content.

The heartbeat prompts themselves are detailed (Boxes C.1.2–C.1.2) and prescriptive — for example, the consolidation prompt mandates six specific steps with required output files (notes/synthesis/, notes/connections.md, notes/open-questions.md). The pivot prompt instructs the agent to try "a fundamentally different approach: different algorithm family, different problem formulation, different representation, or techniques from other domains." These are not neutral interventions — they strongly shape agent behavior — yet the paper provides no evidence that they improve performance relative to a counterfactual without them, or that the specific wording matters.

The consequence. A practitioner cannot determine whether the heartbeat mechanism is load-bearing. If heartbeats are critical to performance, implementing them correctly (with appropriate intervals, trigger conditions, and prompt design) is essential to replicating CORAL's results. If heartbeats are incidental — if autonomous agents with shared persistent memory would eventually reflect, consolidate, and pivot on their own — then the mechanism adds implementation complexity without benefit. The paper's claims about "greater agent autonomy" are in tension with a heartbeat mechanism that forces specific meta-cognitive behaviors at fixed intervals — how much autonomy remains when the agent is interrupted after every evaluation to write a reflection note?

More specifically, the pivot heartbeat (triggered after 5 non-improving evals) could be harmful in some settings: it might interrupt productive local optimization that would have found an improvement on evaluation 6, or it might cause the agent to prematurely abandon an approach that is on the verge of a breakthrough. The paper provides no evidence about whether 5 is the right threshold, whether the pivot prompt's specific instructions ("choose a fundamentally different approach," "start fresh from a strong base") are well-calibrated, or whether a softer nudge ("consider whether you're stuck") would be equally effective with less disruption.

What evidence exists in the paper. None. There is no ablation of the heartbeat mechanism, no sensitivity analysis of heartbeat parameters, and no comparison of different heartbeat prompt designs. The only evidence that heartbeats affect behavior is the trajectory analysis showing that agents do write notes and create skills (Table 4), but this does not establish that these behaviors are caused by the heartbeat rather than by the base instruction ("After every eval, update or create a note" in CORAL.md). The consolidation heartbeat's required outputs (synthesis/, connections.md, open-questions.md) are mentioned in the system description but never analyzed — we do not know whether agents actually produce these files, whether they are useful to other agents, or whether they contain accurate information.

Mitigation status. Not addressed. The heartbeat mechanism is presented as a design feature with a default configuration, and all experiments use that configuration without variation. The paper does not identify heartbeat sensitivity as a limitation or propose ablation studies for future work. This is a significant methodological gap: a core component of the architecture that could plausibly account for substantial performance variation is never tested.

Evaluator Quality Is Assumed but Not Guaranteed: The Method Can Exploit Evaluator Flaws Without Detecting Them

The assumption or constraint. CORAL assumes the availability of a "reasonably well-specified evaluator" (Appendix A) that returns scores aligned with the true optimization objective. The paper acknowledges that this assumption can fail:

"For many important open-ended problems, evaluators are themselves difficult to obtain, incomplete, or even fundamentally ambiguous. In such settings, evaluation may also need to co-evolve with the solutions."

However, even when an evaluator appears well-specified, it may contain bugs or unintended shortcuts — behavior that scores highly under the evaluator but does not represent genuine solution quality. The paper provides concrete evidence of this risk in Appendix D.3, documenting four evaluator bugs they had to fix during integration: PRISM silently skipped failed GPU placements, Transaction Scheduling scored invalid schedules above zero, EPLB ignored experts with zero replicas, and LLM-SQL crashed on mixed-type data.

CORAL's architecture includes evaluator isolation (agents cannot read grader code) as a safeguard against deliberate reward hacking, but this does not prevent agents from discovering evaluator bugs through trial and error. An effective agent that tries diverse strategies is more likely to encounter evaluator edge cases than a narrow fixed-evolutionary search — the very exploration diversity that makes CORAL effective also increases the risk of finding and exploiting evaluator flaws. The paper does not discuss how to detect when a score improvement is due to genuine solution quality vs. evaluator exploitation.

The consequence. In a deployment setting where the evaluator has undetected bugs (which the paper's own experience suggests is common), CORAL's agents may converge to solutions that score well on the buggy evaluator but fail on the true objective. Worse, because agents document their reasoning in shared notes, an agent that discovers an evaluator shortcut may write a note describing it as a "breakthrough," causing other agents to pursue the same flawed strategy — knowledge accumulation amplifies rather than corrects the error.

This is not a theoretical concern. The paper's bug fixes corrected flaws that could produce arbitrarily large score inflation: on PRISM, a solution that crashed on difficult cases could achieve a higher average score than one that handled all cases correctly but performed slightly worse on easy ones. An agent optimizing against the buggy evaluator would be rewarded for producing solutions that crash selectively. In fixed evolutionary search, this might go undetected because the search is less efficient and less likely to discover the edge case. CORAL's effectiveness paradoxically makes it more vulnerable to evaluator flaws.

What evidence exists in the paper. Appendix D.3 documents four evaluator bugs that were present in the original benchmark code. The paper applies fixes uniformly to all methods, so the comparative results are valid, but the existence of these bugs demonstrates that evaluator quality is a real and practical concern. The trajectory analysis does not include any measure of "evaluator exploitation" or "genuine improvement vs. cheating" — there is no analysis of whether any of CORAL's improvements exploited edge cases before the fixes were applied, or whether the fixed evaluators still contain undiscovered bugs. The paper reports that CORAL matches or sets new SOTA on 8 tasks (Table 1), but these are post-fix results; we cannot assess whether pre-fix runs would have achieved higher (but illegitimate) scores.

Mitigation status. Partial and forward-looking. The paper acknowledges evaluator quality as a limitation in Appendix A and suggests "iterative refinement of the evaluator, learned critics, or human-agent negotiation over what constitutes progress" as future directions. The grader isolation mechanism prevents agents from reading evaluator source code but cannot prevent them from observing score patterns that reveal evaluator behavior. The paper does not propose any mechanism for detecting or flagging when an agent's improvement might be due to evaluator exploitation rather than genuine progress — no adversarial validation, no held-out test cases, no automated detection of suspicious score patterns. This is a significant gap for a system that is designed to run autonomously for hours without human oversight.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new architectural paradigm for LLM-based open-ended discovery — shifting control of search meta-decisions from fixed external algorithms to autonomous agents — and provides the first systematic evidence that this shift yields substantial efficiency and capability gains across diverse optimization tasks. The contribution is not a better mutation operator, verifier, or evolutionary algorithm; it is a reorganization of who decides what during iterative search, supported by lightweight infrastructure (shared persistent memory, heartbeat interventions, evaluator isolation) that makes long-running autonomous evolution practical.

What kind of shift is this? It is best understood as an architectural paradigm shift, not an algorithmic refinement. Prior work in LLM-based evolutionary search (FunSearch, AlphaEvolve, EvoX, AdaEvolve) treated the LLM as a subroutine — a stateless proposal generator called by an external loop that managed retrieval, evaluation scheduling, and population updates. The dominant assumption was that search orchestration belongs in the framework, not in the model, because hand-designed evolutionary algorithms encode domain expertise about how to balance exploration and exploitation. CORAL challenges this assumption directly: the paper's central empirical claim is that on the tasks studied, an LLM agent's context-sensitive reasoning about what to explore next, when to test, and what to remember outperforms even carefully hand-designed population management heuristics. The evidence for this is the 3–10× higher improvement rates and up to 10× fewer evaluations across 11 tasks (Table 1), with the strongest fixed-evolutionary baseline (EvoX, which meta-evolves its own search strategy) still trailing CORAL on every task.

This shift is not merely about the degree of adaptation — it is about the locus of intelligence. In fixed evolutionary search, intelligence resides in the algorithm designer's choice of selection mechanism, mutation strategy, and diversity maintenance. In CORAL, intelligence resides in the agent's analysis of evaluation history, causal reasoning about why approaches worked or failed, and decisions about knowledge externalization. The heartbeat mechanism formalizes this: rather than having the framework automatically switch strategies on a plateau, CORAL prompts the agent to diagnose why it is stuck and propose a new direction. The framework provides scaffolding (memory, tools, safety); the agent provides reasoning. This division of labor — framework as infrastructure, agent as intelligence — is a conceptual reframing that opens a different research trajectory from the dominant one of designing better evolutionary operators.

What contradictions does this work resolve? The paper reconciles two previously disconnected literatures. On one side, LLM-based evolutionary search (FunSearch, AlphaEvolve) showed that LLMs can be effective mutation operators but treated them as stateless components within fixed pipelines. On the other side, autonomous agent research (SWE-Agent, OpenHands, AI Scientist) showed that LLMs can navigate codebases, debug errors, and complete complex tasks autonomously, but targeted one-off task completion rather than sustained optimization. CORAL bridges these traditions by showing that the same agent capabilities that enable autonomous debugging — reading evaluation feedback, formulating hypotheses, running local tests, and iterating — can be applied to the meta-decisions of evolutionary search, and that doing so produces more efficient discovery than reserving those meta-decisions for the framework. This synthesis is likely to accelerate convergence between the evolutionary computation and autonomous agent communities.

Which research directions become more attractive? CORAL's results suggest that investment in agent infrastructure for discovery — shared memory architectures, meta-cognitive intervention mechanisms, safe evaluation sandboxes — may yield higher returns than investment in more sophisticated evolutionary algorithms. This is a bet that as LLMs become more capable as coding agents, the marginal value of delegating search decisions to the agent grows faster than the marginal value of optimizing hand-designed search heuristics. The paper's concurrent work context (AutoResearch, Hive, AutoEvolver) suggests this bet is already being placed by multiple groups.

Which directions become less attractive? The strong performance of CORAL's simple shared-file-system coordination over task-specific role decomposition (as used in AI Scientist, AI Co-Scientist, MetaGPT) suggests that for open-ended optimization, horizontal scaling through emergent coordination may be more general and more scalable than vertical scaling through pre-designed role specialization. If agents can autonomously discover which other agents' work to build on and which knowledge to transfer, the human effort of designing communication topologies and role assignments may be better spent on improving the shared memory infrastructure that enables emergent coordination. This does not render role-based systems obsolete — they may still be superior for tasks with well-understood decomposition — but it shifts the default assumption: for problems where the optimal decomposition is unknown, emergent coordination through shared memory is a strong baseline that requires no per-task engineering.

A cautionary note on the scope of the shift. The paper's evidence is concentrated on a specific problem class (open-ended optimization with well-specified evaluators), a specific model tier (Claude Opus 4.6, a frontier coding model), and a specific task structure (Python programs scored by automated evaluators). The paradigm shift claim is broader than the evidence base currently supports. Whether autonomous multi-agent evolution outperforms fixed search on tasks requiring physical experimentation, multi-objective optimization, creative design without clear metrics, or collaborative human-AI discovery is entirely untested. The paper's contribution is to establish that the paradigm is viable and promising on a meaningful benchmark suite, not to prove it is universally superior. The next generation of work should stress-test the paradigm across task domains and model capability levels to map its boundary conditions.

Follow-Up Research This Work Enables

1. Capability threshold characterization: How capable must a model be for CORAL's mechanisms to function? The paper's causal evidence (Table 3 ablations) uses Claude Opus 4.6, a frontier model with strong code generation, self-reflection, and causal reasoning capabilities. The generalization experiment (Table 2, MiniMax M2.5) shows that multi-agent co-evolution still provides benefits with a weaker model, but does not include the knowledge accumulation or co-evolution-vs-independent ablations. A critical open question is whether CORAL's key mechanisms — writing useful analytical notes, extracting transferable skills, building on another agent's code without introducing bugs — require a minimum capability threshold, and if so, where that threshold lies on current model leaderboards. A strong follow-up would systematically evaluate CORAL's single-agent and multi-agent performance across a capability ladder (e.g., Claude Opus 4.6 → Claude Sonnet 4 → GPT-4o → DeepSeek-V3 → Qwen 2.5 Coder 32B → CodeLlama 7B), with the full set of ablations (knowledge accumulation on/off, co-evolution vs. independent best) at each capability level. The prediction: knowledge accumulation benefits should diminish sharply below some capability threshold where models cannot reliably distinguish architectural insights from superficial correlations in their notes, and cross-agent code transfer should degrade when models introduce bugs more frequently than they contribute improvements. Mapping this threshold would provide practical guidance for deployment decisions and theoretical insight into the relationship between base model capability and test-time organizational structure.

2. Heartbeat mechanism necessity and sensitivity. The heartbeat prompts (Boxes C.1.2–C.1.2) are detailed, prescriptive, and used in all experiments without ablation. This is the single largest unexamined component of the architecture. A controlled ablation study would compare CORAL performance with: (a) no heartbeats at all (agents follow only the base CORAL.md instructions), (b) reflection-only heartbeat (remove consolidation and pivot), (c) pivot-only heartbeat (remove reflection and consolidation), (d) consolidation-only heartbeat, (e) default configuration, and (f) aggressive configuration (reflect every eval, consolidate every 3 evals, pivot after 2 non-improving evals). The experiment should measure final score, improvement rate, number of evaluations, and qualitative note quality (via LLM-as-judge or human evaluation of whether notes contain causal insights vs. superficial observations) across at least the three stress-test tasks with multiple trials. A negative result — no significant difference across configurations — would suggest that the base instructions sufficiently encourage reflection and that the heartbeat adds implementation complexity without benefit, simplifying CORAL's design. A positive result with specific configurations dominating would provide guidance for per-task heartbeat tuning. The current paper provides no evidence either way, which is a significant gap for a mechanism described as core to the architecture.

3. Agent count scaling and shared memory saturation. All multi-agent experiments use exactly 4 agents. The paper provides no evidence about whether 2 agents capture most of the benefit, whether 8 agents provide further gains, or whether performance degrades beyond some threshold due to shared memory noise (conflicting notes, redundant exploration). A scaling study on Kernel Engineering and Polyominoes — tasks where the paper shows clear multi-agent benefit — would evaluate 1, 2, 4, 8, and 16 agents, each with matched total wall-clock time (e.g., 1 agent × 12 hours vs. 4 agents × 3 hours vs. 16 agents × 45 minutes). Key metrics: final score, cross-agent attempt rate (what fraction of attempts build on another agent's commit), note quality diversity, and contribution balance (Gini coefficient of improvements across agents). The prediction: cross-agent transfer should increase with agent count up to some saturation point, after which agents cannot effectively process the volume of shared information, and redundant exploration increases. Identifying the saturation point would provide practical guidance for resource allocation. A parallel question is whether initializing agents with different exploration biases (different temperature settings, different first-attempt strategies, or different subsets of seed programs) improves diversity without requiring more agents — the paper's appendix A notes this as future work but provides no preliminary exploration.

4. Evaluator robustness and automatic exploitation detection. Appendix D.3 documents four evaluator bugs that produced incorrect scores — bugs that were present in the original benchmark code and could have been exploited by effective search. CORAL's exploration diversity makes it more likely to discover such bugs than fixed search, creating a perverse dynamic where more effective optimization could produce less genuine progress. A valuable follow-up would develop and evaluate methods for detecting when a score improvement might be due to evaluator exploitation rather than genuine solution quality. Concrete approaches include: (a) held-out validation cases that test for known exploit patterns (e.g., test the PRISM solution on edge-case inputs that should trigger failures), (b) anomaly detection on score trajectories (a sudden large improvement without corresponding code complexity increase is suspicious), (c) requiring agents to explain why a solution improved and flagging explanations that reference evaluator-specific details rather than domain principles, and (d) adversarial evaluation where a second, independently implemented evaluator (or a human judge for a subset of solutions) provides a cross-check. Running CORAL on the original (buggy) evaluators and measuring how often it discovers and exploits bugs would quantify the severity of this risk. The paper's evaluator isolation is necessary but insufficient — it prevents agents from reading grader code but not from learning grader behavior through interaction. Developing defense-in-depth against reward hacking in autonomous discovery systems is an important and currently underexplored problem.

5. Domain generalization beyond Python programs with automated evaluators. All tasks in the paper share a common structure: optimize a Python program against a provided scoring function. Many important open-ended discovery problems differ along one or more dimensions: (a) stochastic evaluators where the same solution produces different scores on different runs (e.g., training a neural network with random initialization, simulating a noisy physical system), (b) expensive evaluators where each call costs dollars or hours rather than seconds (e.g., wet-lab experiments, large-scale simulations), (c) multi-objective or partially specified evaluation where no single scalar captures quality (e.g., drug candidates must balance efficacy, toxicity, and synthesizability), and (d) evaluators that are themselves learned or human-provided (e.g., preference judgments, expert reviews). A systematic extension would adapt CORAL to one or more of these settings and measure where the autonomous multi-agent paradigm continues to provide benefits and where it breaks down. For stochastic evaluators, agents would need to learn to average multiple evaluations and distinguish signal from noise — does the heartbeat's reflection mechanism naturally support this, or does it require explicit uncertainty quantification tools? For expensive evaluators, the 3–10× reduction in evaluation count that CORAL demonstrates (Table 1) becomes proportionally more valuable — but the agent's own reasoning overhead (local testing, knowledge management) must remain small relative to evaluation cost. For multi-objective settings, notes and skills would need to capture tradeoffs rather than single-dimensional insights — does the file-system-based shared memory support this naturally, or would it require structured representations?

6. Training specialized small models for the CORAL agent role. The paper notes in Appendix A that CORAL "relies on frontier foundation models that can handle relatively complex coding-agent workflows, which makes full deployment on local devices difficult." All experiments use Claude Opus 4.6 or MiniMax M2.5, both large models accessed via API. CORAL's trajectory data — thousands of agent attempts with associated code changes, evaluation outcomes, notes, skills, and heartbeat reflections — constitutes a rich dataset for fine-tuning. A natural follow-up would train a smaller open-source model (e.g., Qwen 2.5 Coder 7B or 32B) on CORAL trajectories to serve as the agent backbone, then evaluate whether the fine-tuned small model can approach the frontier model's performance when embedded in the CORAL infrastructure. The training objective would be multi-task: generate code changes given task context and evaluation history, write analytical notes, extract skills, and respond to heartbeat prompts. The key metric is whether the organizational benefits of CORAL (shared memory, multi-agent co-evolution, heartbeat) can partially compensate for the model capability gap, similar to how the MiniMax M2.5 results (Table 2) show that multi-agent evolution approaches SOTA on some tasks despite a weaker base model. This direction directly addresses the deployment limitation and would test whether CORAL's architecture is valuable primarily because it leverages frontier model reasoning, or because its organizational structure provides benefits that are partially independent of model scale.

Practical Applications and Downstream Use Cases

1. Automated systems optimization for cloud infrastructure and hardware. The paper's results on EPLB (expert placement load balancing), PRISM (GPU placement), LLM-SQL (column caching), Transaction Scheduling, and Cloudcast (cross-cloud transfer) demonstrate that CORAL directly applies to practical systems optimization problems where the evaluator is an existing benchmark or simulator. The Kernel Engineering result — pushing a VLIW SIMD kernel from 1,363 to 1,103 cycles, a 19% improvement over the best known human result — is the most compelling example. For organizations that maintain performance-critical kernels, database heuristics, or resource allocation algorithms, deploying CORAL with an internal evaluator could automate a significant fraction of optimization work that currently requires senior engineering time. The 3-hour runtime and $30–60 cost per task (Appendix E.2) make this economical for tasks where a 5–20% performance improvement translates to meaningful infrastructure savings. The practical deployment path is straightforward: wrap existing benchmarks or simulators as CORAL graders, provide a seed implementation, and run a 4-agent co-evolution overnight.

2. Scientific algorithm discovery with moderate human oversight. The mathematical optimization tasks (circle packing, Erdős overlap, signal processing, MMD, autocorrelation inequalities) represent a class of problems where the objective is well-defined, ground truth is unknown, and improvements require algorithmic insight rather than scale. CORAL's 100% improvement rate on Circle Packing (11/11 evaluations improved) and 83.3% on MMD-16-2 (6 evaluations, 5 improvements, Table 1) suggest that on tasks where the solution space admits incremental improvement, the framework can make steady progress with minimal wasted computation. For researchers in applied mathematics, operations research, or theoretical computer science who have well-specified optimization problems but lack the time to explore the full design space manually, CORAL provides a tool for automated exploration that produces interpretable artifacts (notes documenting why approaches work, skills encoding reusable techniques) alongside improved solutions. The interpretability is practically important — it means researchers can review the agent's notes to understand the principles behind improvements, rather than receiving only a black-box optimized solution.

3. Bootstrapping self-improvement pipelines with verifiable rewards. CORAL's architecture is well-suited for generating high-quality training data in domains with verifiable rewards. Running CORAL on a training set of problems produces trajectories of (attempt, score, feedback, note) that capture the reasoning behind successful optimizations. These trajectories could be used to fine-tune a base model on the skill of iterative optimization itself — teaching a model not just to solve problems, but to analyze failures, extract principles, and systematically improve. The paper's trajectory data (Table 4, Tables 4–5) demonstrates that CORAL agents on advanced tasks produce rich knowledge artifacts (0.55–0.68 knowledge items per attempt on Polyominoes and Kernel Engineering), with notes that "identify architectural bottlenecks" and document "what NEVER worked." This data has value beyond the specific solutions it produces — it captures a form of meta-cognitive reasoning about optimization that could be distilled into a model's capabilities. For organizations building AI systems that need to improve over time from feedback (e.g., code generation with execution feedback, mathematical reasoning with proof verification), CORAL-generated trajectories provide a source of training data for the improvement skill itself.

4. Competitive baseline for open-ended discovery benchmarks. As the field of LLM-based discovery matures, standardized benchmarks will emerge (the paper itself draws tasks from EvoX, TTT-Discover, SkyDiscover, Frontier-CS, and Anthropic's kernel engineering task). CORAL provides a strong, well-documented baseline that future methods should compare against — not just on final score, but on improvement rate and evaluation efficiency. The paper's comprehensive reporting of these metrics (Table 1, Table 2) and its ablation methodology (Table 3) establish a standard for what a thorough evaluation of an open-ended discovery system should include. Researchers proposing new methods should demonstrate not just that their approach achieves higher final scores, but that it does so with comparable or better evaluation efficiency — a method that achieves 2% higher final score but requires 10× more evaluations may not represent genuine progress if evaluator calls dominate cost. CORAL's open-source release and detailed documentation (Appendix C) lower the barrier for others to use it as a baseline.

When to Prefer This Method

The paper does not explicitly position CORAL against named alternatives with a decision rule, nor does it provide evidence about which task characteristics predict CORAL's relative advantage over fixed evolutionary search. The conditions below are inferred from patterns in the experimental results rather than stated by the authors, and should be treated as hypotheses requiring validation rather than established guidance.

The experimental evidence suggests that CORAL's advantages are largest when:

The base model has non-trivial initial capability on the task. On tasks where Claude Opus 4.6 achieves high improvement rates (Circle Packing 100%, PRISM 100%, EPLB 78.9%), CORAL converges in very few evaluations. On tasks where the base model struggles (Transaction Scheduling 27.3% improvement rate for single-agent Claude), the efficiency gap narrows. The MiniMax M2.5 results (Table 2, bottom) show weaker absolute performance and noisier multi-agent benefits, consistent with a capability threshold effect. This suggests CORAL is most appropriate when the model can already produce solutions that are sometimes directionally correct, even if suboptimal.
The evaluator supports rapid, deterministic feedback. Tasks with local test capability (Circle Packing at 100% local test rate, Kernel Engineering at 57%) show strong CORAL performance because agents can validate before submitting. Tasks without local testability (PRISM at 0%) still work but lose the benefit of pre-submission debugging. For tasks where each evaluation takes hours or costs significant money, CORAL's 10× reduction in evaluation count is proportionally more valuable.
The improvement landscape rewards understanding over brute force. On Kernel Engineering, where improvement requires understanding architectural tradeoffs (VALU vs. ALU balancing, dependency graph optimization), CORAL's knowledge accumulation is strongly beneficial (18.6% degradation when disabled). On Transaction Scheduling, where improvement may involve more incremental parameter tuning, the knowledge effect is smaller (2.7% degradation). This suggests CORAL is most valuable when the solution space admits interpretable principles that can be documented and reused.
The total compute budget allows for 3+ hours of agent runtime. CORAL's efficiency advantage is in evaluations per improvement, not necessarily in wall-clock time per improvement — agents spend significant time reasoning between evaluations. For tasks where wall-clock time is the binding constraint and evaluator calls are cheap, fixed evolutionary search (which calls the LLM once per evaluation) may achieve more evaluations per hour, potentially compensating for a lower improvement rate. The paper does not measure this tradeoff directly.
Multi-agent parallel compute is available. The multi-agent benefit (Table 2, Table 3) is real but requires running 4 agents simultaneously, increasing API costs proportionally. For resource-constrained settings, single-agent CORAL already outperforms fixed search baselines (Table 1) and may be the practical sweet spot — the additional gain from multi-agent coordination is most pronounced on stress-test problems (Kernel Engineering 18.3%, Polyominoes 5.0%) and smaller on the math/systems suites (Table 2, bottom).