AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization

🎯 Pitch

An open-source LLM agent with beam search and a self-curated optimization memory can match the kernel optimization results of Claude Sonnet 4 on emerging AI accelerators—at 26× lower cost—without any expert-provided hardware heuristics. It autonomously discovers global transformations by iterating on its own slow-fast kernel pair experiences, boosting throughput by 12–14 percentage points on Trainium benchmarks.

1. Executive Summary

This paper introduces AccelOpt, the first self-improving LLM agentic system that autonomously optimizes kernels for emerging AI accelerators without requiring expert-provided, hardware-specific optimization knowledge. The system is evaluated on NKIBench, a new benchmark suite of 14 real-world AWS Trainium kernels extracted from LLM workloads, using open-source models (gpt-oss-120b and Qwen3-Coder-480B) as the agent backbones. AccelOpt combines beam search (iteratively expanding a frontier of top-performing candidate kernels to build on prior successes) with an optimization memory (a curated queue of slow-fast kernel pairs and LLM-distilled optimization insights that transfer experiences across iterations), enabling the system to discover both peephole optimizations and non-trivial multi-step global transformations. The system improves the average percentage of peak throughput from 49% to 61% on Trainium 1 and from 45% to 59% on Trainium 2, matching the kernel improvements of Claude Sonnet 4 (thinking mode) while being 26× cheaper, establishing that open-source models with structured search and memory accumulation can rival leading proprietary models on this task only when sufficient exploration budget is allocated through beam search rather than repeated sampling.

2. Context and Motivation

The Core Problem: Kernel Optimization Is Hard for Emerging Hardware

The fundamental challenge this paper addresses is deceptively simple: how do you write high-performance low-level kernels for a brand-new AI accelerator without any established cookbook of optimization recipes? When a new chip arrives, the developers writing the kernels that map machine learning operators onto its hardware resources lack the accumulated wisdom, heuristics, and performance intuition that exist for mature platforms like NVIDIA GPUs. They face a vast design space — memory layouts, parallelization schemes, loop ordering, tiling strategies, scheduling decisions — with no map and few signposts.

This matters because the gap between a naive kernel and a well-optimized one is enormous, and it directly translates to wasted compute, money, and time. The paper quotes a concrete example from the GPU world to calibrate expectations: after NVIDIA released the H100 in 2022, it took roughly a year for attention kernels to reach approximately 37% of theoretical peak performance (Dao, 2023), and another year to approach 85% (Shah et al., 2024). The timeline is measured in years because human experts must empirically explore the optimization space, building intuition through trial and error. When the hardware is brand new, that process starts from scratch.

The paper focuses on Amazon Trainium (AWS, 2025), which it describes as "a widely deployed and representative AI accelerator that exemplifies these challenges." Trainium is programmed using the Neuron Kernel Interface (NKI) (AWS, 2025b), a Python-embedded kernel language. The key phrase in the paper's framing is: "both the hardware and the programming model remain relatively new." This isn't a hypothetical future problem — it's a live engineering challenge at scale. The paper explicitly positions this as a problem faced by every new accelerator entering production, citing concrete evidence that major organizations are building their own chips: OpenAI (2025), Qualcomm (2025), Meta (2024), and Microsoft Azure (2024).

Why This Problem Is Increasingly Important

The paper identifies three converging trends that make autonomous kernel optimization urgent:

The accelerator landscape is fragmenting. The era of NVIDIA GPU monoculture is ending. As more organizations design custom silicon (whether TPUs, Trainium, Maia, MTIA, or in-house designs), the number of hardware platforms that need high-quality kernels multiplies. Each platform has its own architecture, memory hierarchy, instruction set, and programming model. The human effort required to manually optimize kernels for each one does not scale linearly with the number of platforms — it's more than linear because each new architecture erases the performance intuition developed on previous ones.

Scale amplifies waste. When a kernel runs at 30% of peak throughput instead of 80%, that gap represents actual money being burned — dollars of electricity, silicon area, and time. The paper notes that "suboptimal kernels can severely limit system performance and, when scaled to large deployments, result in substantial waste of compute and financial resources" (Spector et al., 2024; Ye et al., 2025; Zhao et al., 2025). In production LLM inference serving millions of tokens per day, a 2× kernel improvement isn't academic — it's the difference between profitability and loss.

Workload diversity demands coverage. As AI accelerators proliferate, practitioners face "hundreds or thousands of kernels to optimize across different configurations, hardware versions, and workloads" (Kim et al., 2023). Manual optimization approaches bottleneck on expert availability — there aren't enough kernel engineers with deep platform knowledge to cover every operator at every shape on every hardware revision. An automated system that can produce expert-competitive kernels without per-kernel expert involvement addresses this bottleneck directly.

Where Prior Approaches Fall Short

The paper identifies several categories of existing work and their limitations:

Manual optimization and expert heuristics. The traditional approach — human engineers studying profiling output, applying optimization passes, and iterating — produces excellent results but does not scale. It's slow (years for a new platform), requires rare expertise, and creates a dependency that becomes a bottleneck for every new operator, shape, or hardware version. For Trainium specifically, the paper states the problem plainly: "developers lack the extensive optimization recipes and performance heuristics available for mature platforms like GPUs" (Thakkar et al., 2023).

LLM-based kernel generation without optimization. Recent work has shown LLMs can generate correct kernels for accelerators (Ouyang et al., 2025a; Wei et al., 2025; Li et al., 2025; Lange et al., 2025). However, correctly generating a functional kernel and producing one that approaches peak hardware throughput are fundamentally different problems. The paper acknowledges that LLMs have demonstrated "potential to automatically generate correct kernels with competitive performance in the context of GPUs, TPUs, and NPUs," but positions itself differently — AccelOpt's goal is not just generation, but autonomous exploration of the optimization space starting from a working baseline.

LLM-based optimizers with manually curated recipes. AutoComp (Hong et al., 2025) uses LLM agents to optimize kernels, but its planners "rely on manually crafted, problem-specific lists of optimizations" — a human must pre-identify the relevant optimization categories for each kernel. This limits generality and still requires platform-specific expert knowledge as input. AlphaEvolve (Novikov et al., 2025) optimizes matrix multiplication and FlashAttention on TPUs, but its implementation is not publicly available (the paper notes this explicitly). GEPA (Agrawal et al., 2025) improves LLM-generated AMD NPU kernels by evolving prompts through automatic discovery of architectural best practices, which the paper cites as complementary — but GEPA's insights operate at the prompt level (general best practices), while AccelOpt's optimization memory provides "more detailed task-specific insights."

Non-LLM search-based optimizers. Systems like Mirage (Wu et al., 2025) and TASO (Jia et al., 2019) can automatically search for optimized tensor programs using formal superoptimization techniques. However, the paper argues these approaches "typically rely on substantial manual effort and deep architecture knowledge, including rewriting initial kernels to match the optimizer's input format and explicitly specifying a search space together with pruning strategy." Setting up a fair comparison with these systems is "difficult" (Section 4.4) because the human investment required to encode platform knowledge into their search specifications is precisely what AccelOpt aims to eliminate.

The missing piece: self-improvement from experience. The crucial gap the paper identifies is that existing LLM-based approaches treat each kernel as an independent optimization problem. They do not accumulate insights across iterations, across kernels, or across runs. A system that discovers an effective loop invariant code motion for one kernel cannot apply that pattern to another. A system that repeatedly attempts the same (failing) optimization strategy has no mechanism to learn from those attempts. The paper's framing in Section 2.3 makes this explicit: "Although beam search can record exploration history through its evolving candidates, it cannot capture optimization experiences." AccelOpt's optimization memory is designed specifically to fill this gap.

The Existing Benchmark Gap

The paper identifies a specific infrastructure deficiency that it addresses through NKIBench. At the time of development, "no existing benchmark suite contained NKI kernels with sufficient baseline performance to serve as meaningful starting points for optimization." Moreover, existing accelerator kernel benchmarks share a common limitation: they "typically lack information about how well a kernel is optimized relative to the hardware's theoretical peak performance." The paper argues, correctly, that relative speedup alone can be ambiguous — a 2× speedup over a terrible baseline might still leave the kernel at 10% of peak throughput, while a 1.1× speedup over an already-near-optimal kernel represents genuine achievement. By computing the percentage of peak throughput using a roofline model analysis (Williams et al., 2009), NKIBench provides an absolute yardstick: it tells you not just that a kernel is faster than another, but how close it is to the hardware's upper bound.

The benchmark's composition also addresses a practical gap. The 14 kernels are "extracted from real-world LLM workloads" (listed in Appendix Table 5), covering both inference and training, spanning from single operators (Matmul, BatchMatmul) to multi-operator chains (Matmul+others, LoRA) and larger building blocks (Group Query Attention, Mamba block). This reflects the diversity of actual engineering work, where optimizers face everything from simple element-wise operations to complex fused blocks.

How This Paper Positions Itself

The paper occupies a specific conceptual niche: autonomous, self-improving exploration of the kernel optimization space on emerging hardware without expert curation of optimization knowledge. It does not claim to be the first LLM-based kernel generator — it explicitly cites that lineage. It does not claim to be the first to use LLM agents for optimization — it cites AutoComp and GEPA. It does not claim to be the first to use beam search for iterative improvement — it cites Hong et al. (2025).

Rather, AccelOpt's claimed novelty (Section 1 contributions) is the combination of three elements that have not been jointly demonstrated before:

No expert-provided, hardware-specific optimization knowledge or predefined optimization recipes. The system discovers optimizations through exploration, not through a curated list of what to try. This is what makes it viable for genuinely new platforms.
Memory accumulation across iterations. Unlike beam search alone (which only carries forward candidate kernels), AccelOpt curates and transfers optimization insights — both successful strategies and failed attempts — across time. This is positioned as an instance of test-time learning (Sun et al., 2024).
Evaluation against theoretical peak hardware performance. Rather than only reporting relative speedup, NKIBench situates results within the complete optimization landscape, providing insight into how much room remains and whether saturation reflects genuine optimality or a failure of exploration.

The paper explicitly acknowledges what it does not address. Section 4.3 analyzes saturation behaviors, identifying cases where AccelOpt cannot improve further because the kernel is near peak performance versus cases where exploration itself fails. Section 4.6 notes the system only handles single-core kernels without cross-chip communication, and that extending to communication primitives is "a promising direction for future work." Section 2.3 flags that transferring memory accumulated from optimizing some kernels to different kernels is "worth future exploration" — the current system's memory is per-kernel, not cross-kernel.

The conceptual positioning is best understood through the paper's own framing (Section 1): "an important goal of this work is to investigate whether an LLM-based system can autonomously navigate the optimization space to produce high-performance kernels without relying on human-engineered heuristics or preexisting optimization examples." The key word is navigate — the system must explore, learn, and improve, not just apply a fixed recipe. This is the gap between AccelOpt and prior work, and it's why the paper's combination of beam search (exploration) and optimization memory (learning) is not just an incremental combination — it's the mechanism that makes autonomous navigation possible in a space where LLMs have no prior map.

3. Technical Approach

3.1 Reader Orientation

AccelOpt is a multi-agent LLM system that iteratively rewrites low-level AI accelerator kernels, using profiling feedback to guide improvement and a memory of past optimizations to accelerate learning. Formally, it solves the problem of autonomous kernel optimization space exploration: given a functionally correct but suboptimal kernel for an unfamiliar hardware platform, produce a sequence of transformations that maximize the kernel's throughput (measured as a percentage of the hardware's theoretical peak performance), without any human-provided optimization recipes or platform-specific heuristics. The solution takes the shape of a beam search over kernel variants augmented with a curated optimization memory: at each iteration, the system generates many candidate rewrites, keeps the best few for further exploration (beam search), and simultaneously extracts generalizable optimization strategies from successful and unsuccessful transformations to inform future iterations (memory).

3.2 Big-Picture Architecture (Diagram in Words)

The AccelOpt system has five major components, connected in a cyclical pipeline as illustrated in Figure 1 of the paper:

Candidate Kernel Pool — A set of B = 6 kernels (the "beam") that represent the current frontier of best-known implementations. These are the starting points for each round of optimization.
Agentic Workflow (Planner → Executor → Summarizer) — Three LLM agents that collaboratively generate new kernel variants. The Planner analyzes profiling data from a candidate kernel and proposes N = 12 optimization plans. The Executor concretely implements each plan, making K = 2 attempts per plan (producing up to B × N × K = 144 new kernels per iteration). The Summarizer distills generalizable optimization insights from discovered slow-to-fast kernel pairs into structured experience items.
Distributed Profiling Service — Receives generated kernels, compiles and runs them on Trainium hardware, performs correctness checking (against a CPU reference implementation with a tight per-task tolerance tol), and measures execution latency with warm-up iterations and multiple runs per kernel. Returns profiling data (memory traffic, engine utilization, spill behavior, latency) to the agents.
Optimization Memory — A fixed-capacity queue (capacity ExpN = 16) of experience items, each containing a slow-fast kernel pair (as pseudocode) and an LLM-generated optimization strategy description. The memory is updated each iteration with up to TopK = 8 new items, curated by the Summarizer from the current iteration's profiling results. Oldest entries are evicted when capacity is reached.
Candidate Selection Function β — At the end of each iteration, selects the B = 6 kernels that will form the beam for the next iteration. This function first identifies the fastest correct kernel within each plan group, then selects the top-B by measured latency from this representative pool.

Information flows cyclically: the current beam C_i → Planner generates plans → Executor implements kernels → Profiling Service measures performance → Summarizer distills experience → Optimization Memory is updated → Candidate Selection picks the next beam C_{i+1}. The cycle repeats for T = 16 iterations (the full experiment horizon).

3.3 Roadmap for the Deep Dive

First, the overall algorithm structure (Algorithm 1) — how beam search, the agentic workflow, and memory interact across one complete iteration, establishing the outer loop that the rest of the section fills in.
Second, the Placer-Executor-Summarizer agentic workflow in detail — what each agent does, what prompts they receive (Figures 2, 19–25 in the Appendix), what outputs they produce, and how their roles decompose the kernel optimization task into subtasks that LLMs can handle reliably.
Third, the beam search mechanism — why it is used instead of repeated parallel sampling, the candidate selection function β, the B × N × K sampling structure, and the key hyperparameters that control exploration breadth versus depth.
Fourth, the optimization memory curation procedure (Algorithm 2) — how slow-fast pairs are identified, how the summarizer distills them, how diversity is enforced, the positive/negative rewrite distinction, and the speedup thresholds that gate memory entry.
Fifth, the profiling service and peak throughput calculation — how correctness is verified, how latency is measured, how the roofline model computes theoretical peak performance, and why this metric provides an absolute yardstick.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that combining structured search (beam search) with experience accumulation (optimization memory) enables LLM agents to autonomously navigate the kernel optimization space for emerging hardware platforms without pre-curated optimization knowledge.

3.4.1 Overall Algorithm: One Iteration of AccelOpt

Algorithm 1 in the paper defines the procedure for one complete iteration of AccelOpt. It takes three inputs: E_{i-1}, the optimization memory from the previous iteration; C_i, the set of B candidate kernels at the start of iteration i; and the fixed agent models (Planner θ_p, Executor θ_e, Summarizer θ_s), the profiler function r, the memory curation function σ, and the candidate selection function β. It outputs three things: K, the set of all generated kernels with their profiling results; E_i, the updated optimization memory; and C_{i+1}, the beam of B kernels for the next iteration.

The procedure unfolds in three phases:

Phase 1: Generation (lines 1–8). For each candidate kernel c in the current beam C_i, the Planner θ_p samples N optimization plans, each conditioned on the candidate kernel c and the previous iteration's experience E_{i-1}. Formally, the Planner samples p ∼ θ_p(p | c, E_{i-1}) with |P| = N. For each plan p, the Executor θ_e samples K concrete kernel implementations, each conditioned on the plan p and the candidate kernel c, producing a set A_p of K pairs (a, p, r(a)) where a is the generated kernel and r(a) is its profiling result (latency and performance metrics). All generated kernels across all candidates and all plans are accumulated into the flat set K.

Phase 2: Memory Update (line 9). The memory curation function σ (defined in Algorithm 2) processes the new kernels K together with the existing memory E_{i-1}, using the Summarizer θ_s to distill experience items from the best-performing slow-fast kernel pairs discovered in this iteration. The output is E_i, the updated optimization memory for the next iteration.

Phase 3: Beam Selection (line 10). The candidate selection function β selects the B kernels that will form C_{i+1} from the union of all generated kernels K and the current candidates C_i. If fewer than B valid (correct, compilable) kernels exist, the remaining slots are filled by the previous iteration's candidates — this ensures the beam size remains constant even when an iteration produces few correct outputs, dynamically allocating more sampling budget to difficult cases.

The key structural insight of this design is the three-stage generation pipeline (Planner → Executor → Summarizer). The Planner operates at the level of optimization strategies ("reduce memory spilling by hoisting invariant computation"), the Executor operates at the level of concrete code transformations (actually rewriting loop nests and tensor layouts), and the Summarizer operates at the level of knowledge extraction ("what generalizable principle does this successful transformation embody?"). This decomposition mirrors how human experts work — analyze the profile to form a hypothesis, implement the transformation, then reflect on what was learned — and it enables each LLM call to operate at a manageable level of abstraction.

3.4.2 The Planner Agent: Generating Optimization Strategies

The Planner agent (Figure 2, top panel; Appendix Figure 21) is the strategic reasoning component of AccelOpt. Its job is to analyze the profiling results of a candidate kernel and propose N concrete, one-step optimization plans. Critically, the Planner does not write code — it produces structured natural language plans that the Executor will later implement.

Inputs to the Planner. The Planner receives five categories of information in its prompt:

NKI base knowledge (Appendix Figure 19): architectural constraints of the NKI programming model — how partition dimensions work, free dimension size limits for PSUM (≤ 512), SBUF partition size limits (≤ 192KB), partition count limits (≤ 128), and the semantics of the nc_matmul tensor engine instruction. This ensures the Planner understands what transformations are legal.
Profiling terminology (Appendix Figure 20): definitions of all profiling metrics — hbm_read_bytes, hbm_write_bytes, sbuf_read_bytes, spill_reload_bytes, spill_save_bytes, hardware_flops, hfu_estimated_percent (Hardware FLOPs Utilization), vector_engine_active_time_percent, scalar_engine_active_time_percent, latency, and the memory/compute bound classification via mm_arithmetic_intensity vs peak_flops_bandwidth_ratio.
The ML operator specification (problem_code): the high-level mathematical operation the kernel implements (e.g., a specific matrix multiplication with given dimensions).
The baseline candidate kernel code (kernel_code): the actual NKI implementation currently under consideration.
The profiling results (profile): the measured performance metrics for this specific kernel.
Past experiences (E_{i-1}): the current contents of the optimization memory, containing previously discovered slow-fast kernel pairs and their corresponding optimization strategies. This is how the Planner learns from prior iterations.

The Planner's task and output. The Planner is instructed (Figure 21) to follow a specific reasoning process: (1) start by analyzing the profile to find inefficiencies, (2) combine those intuitions with the kernel code to formulate optimization plans, (3) consider a broad space of transformations — "loop ordering, tiling, loop split and merge, liveness analysis, data reuse, reordering instructions or blocks of instructions, hoisting redundant operations out of loops, fusion, and other methods not listed here." The Planner is explicitly told: "The compiler exists and thus the profile numbers might not match the source code analysis. However, the plan can still target optimizing certain metrics." This is an important design choice — it allows the Planner to reason about high-level optimization principles even when the compiler's exact behavior creates mismatches between source-code analysis and measured performance.

The output is N optimization plans, each containing one step. The instruction says "each optimization plan should have 1 step" — this constraint keeps plans focused and executable, rather than producing vague multi-step wish lists that the Executor cannot reliably implement.

Why the Planner is needed. The paper could have combined planning and implementation into a single prompt (as it does for the Claude Sonnet 4 baseline, which receives a unified prompt — Appendix Figures 29–31). Separating them serves two purposes. First, it allows the Planner to focus entirely on strategic analysis without being distracted by NKI syntax details, reducing the chance of generating plans that sound reasonable but are impossible to implement. Second, it enables the system to generate multiple diverse plans (N = 12) and then attempt each one multiple times (K = 2) — this creates a structured exploration pattern where strategic diversity (plans) is orthogonal to implementation robustness (executor attempts), which would be impossible in a single unified generation step.

A concrete Planner output is shown in Figure 3. The Planner identifies that "the profile shows low HFU (7.78%) and high memory writes (1.07 GB), indicating memory-bound operations," then traces this to a specific inefficiency — the LHS matrix transposition v7 → v8 → v9 is recomputed "16 times (for each i1) for the same LHS data block." The proposed plan is "Hoist LHS Transpose Out of Reduction Loop" with the step: "Move the LHS transpose operations out of the i1 loop to eliminate redundant computation." The plan includes specific code change descriptions and the reasoning about why this should help (eliminating redundant memory bandwidth usage).

3.4.3 The Executor Agent: Implementing Optimization Plans

The Executor agent (Figure 2, middle panel; Appendix Figures 22–24) is the code-generation component of AccelOpt. Its job is to take an abstract optimization plan and concretely implement it as a modified version of the candidate kernel. Unlike the Planner, the Executor must produce compilable, correct NKI code — it operates at the level of concrete syntax, not strategy.

Inputs to the Executor. The Executor receives:

The same NKI base knowledge as the Planner (Appendix Figure 19) — architectural constraints.
A concentrated NKI programming guide (Appendix Figures 22–23) — common pitfalls and language rules that the paper's authors empirically identified as frequent error sources. This guide covers: output dependencies in affine_range vs sequential_range loops, the distinction between basic indexing and advanced indexing (NKI requires one or the other, not both), tensor usage scope rules (tensors defined in if/for blocks cannot be used outside them), variable access restrictions (no slices with variable sizes, shape elements must be integers), and proper patterns for in-place updates ("use data[...] = if you want to update the existing object" rather than shadowing with data = data + ...). The paper notes this guide is "adopted from the public NKI programming document and tuned for the agents based on their common errors" — it represents accumulated debugging knowledge from earlier failed experiments.
The ML operator specification (problem_code).
The baseline candidate kernel code (kernel_code).
Kernel usage — a snippet showing how the kernel is invoked, including the correctness checking assertion (np.allclose(kernel_output, ref_output, atol=1e-4, rtol=1e-2)). This tells the Executor the expected input/output interface.
The optimization plan (optimization_plan) — the Planner's output.

The Executor's task and output. The Executor is instructed (Figure 24): "Please concretize the optimization plan and optimize the kernel function." It receives two explicit constraints: "Only use existing NKI APIs in the baseline kernel. Do not invent new APIs even if the optimization plan suggests it" and "Don't use lower precision than the baseline kernel even if the optimization plan suggests it." These constraints prevent the Executor from "cheating" by switching to a lower-precision data type or invoking nonexistent hardware features, both of which would produce kernels that look faster in profiling but are either incorrect or impossible to compile.

For each plan, the Executor makes K = 2 attempts. This is a robustness mechanism: not every attempt at implementing a plan will produce correct, compilable code. By trying twice, the system increases the probability that at least one implementation of each plan is valid. If both attempts succeed, the profiling service will measure both, and the faster one will be selected by β for the per-plan representative.

A concrete Executor output is shown in Figure 3 (the "Optimized kernel" code block). Comparing the "Candidate kernel" and "Optimized kernel" shows the transformation: the LHS transpose operations (loading v1, computing v7, v8, v9) are moved outside the i1 loop (the outer loop over 16 iterations) and instead placed in a loop over i2 and i3 that executes before the i1 loop. The precomputed result is stored in v9_global and reused inside the i1 loop, eliminating the redundant repeated transposition. This is a non-trivial code transformation that requires understanding loop dependencies, tensor lifetimes, and the semantics of the affine_range parallel construct.

The role of the NKI programming guide. The inclusion of the programming guide (Figures 22–23) is a practical design choice that acknowledges a fundamental limitation: current LLMs, even strong coding models, do not have reliable knowledge of niche, platform-specific APIs like NKI. The guide provides the Executor with the syntactic and semantic rules it needs to avoid common errors. The paper's authors tuned this guide "based on their common errors" — this is an empirical process where they observed execution failures, identified patterns (e.g., scope errors, indexing mismatches), and added explicit counter-examples to the prompt. This represents a small but essential amount of manual engineering that makes the autonomous system work in practice, analogous to the "system prompt engineering" that goes into many LLM applications.

3.4.4 The Summarizer Agent: Distilling Optimization Insights

The Summarizer agent (Figure 2, bottom panel; Appendix Figure 25) is the knowledge extraction component of AccelOpt. Its job is to take a discovered slow-fast kernel pair and produce a structured experience item that generalizes the transformation into a reusable optimization strategy.

Inputs to the Summarizer. The Summarizer receives:

A slow kernel — the pre-optimization version (either the candidate kernel from which the fast kernel was generated, for positive rewrites; or a generated slower kernel, for negative rewrites).
A fast kernel — the post-optimization version (the generated kernel that improved performance, for positive rewrites; or the candidate kernel, for negative rewrites).
The speedup — the ratio of the slow kernel's latency to the fast kernel's latency.

The Summarizer's task and output. The Summarizer is instructed (Appendix Figure 25): "Identify the difference between the old and new kernels. If two kernels are identical, just say 'No optimization found'. If two kernels are different, summarize a one-step optimization plan that can convert the old kernel to the new kernel, and add a short python code snippet of the original and optimized kernels that clearly represents the optimization plan. The optimization plan should be general enough to be applied to other kernels."

The output format is structured (Appendix Figures 26–28 show examples):

A short title describing the optimization plan (e.g., "Loop Invariant Code Motion for LHS Matrix Transposition," "Increase the tile size in the innermost dimension from 256 to 512 and reduce the corresponding outer loop iteration counts by half," "Loop Fusion and Dimension Reshaping for Improved Data Locality").
A full description of the plan, explaining what changed and why it improves performance.
Pseudocode snippets of the original and optimized code, extracted to show only the relevant transformed segment. The Summarizer is instructed to produce "a short python code snippet ... that clearly represents the optimization plan" — this is a compression step that removes irrelevant details (unmodified code, boilerplate) to focus the Planner's attention on the transformation itself when this experience item appears in future prompts.

Why this extraction and generalization step exists. If the optimization memory simply stored the raw slow-fast kernel pairs, the Planner would need to parse two complete kernels (hundreds of lines each) to identify the relevant transformation. Worse, the Planner might overfit to superficial differences (variable names, loop indices) rather than extracting the general principle. The Summarizer abstracts the transformation into a portable form: a strategy description plus relevant code segments. The paper explicitly states that the Summarizer "extracts the optimized segment of each pair as pseudocode" and that the optimization plan should be "general enough to be applied to other kernels" — this makes the memory items transferable.

Concrete Summarizer outputs are shown in Figures 26–28. For example, Figure 26 shows a memory item with the title "Loop Invariant Code Motion for LHS Matrix Transposition" containing pseudocode that shows the loop structure before and after hoisting. Figure 27 shows "Increased Tile Size for Last Dimension with Loop Fusion," and Figure 28 shows "Loop Fusion and Dimension Reshaping for Improved Data Locality." Each item distills a specific, named optimization strategy with an English explanation and before/after code snippets.

3.4.5 Beam Search: Why It Beats Repeated Sampling

The paper explicitly compares beam search against repeated (parallel) sampling in Section 4.4, and the results (Figure 13) show that beam search "outperforms repeated sampling of the agentic workflow, using the same LLMs." This section explains the mechanism behind that result.

Repeated sampling baseline. The alternative to beam search is to query the entire agentic workflow (Planner + Executor) independently M times, producing M kernel variants from the same starting baseline, and selecting the fastest one. This is standard test-time scaling for LLM-based code generation (Brown et al., 2024) and is how the paper evaluates Claude Sonnet 4 (the baseline prompts in Appendix Figures 29–31 are single-shot optimization prompts, queried repeatedly). The problem with repeated sampling is that each attempt starts from the same baseline kernel — no attempt can build on improvements discovered by other attempts. If the best optimization requires two sequential transformations (e.g., first fuse two loops, then increase the tile size of the fused loop), repeated sampling can only discover it if a single LLM call happens to perform both transformations correctly in one shot, which is increasingly unlikely as transformations become more complex.

How beam search works in AccelOpt. At each iteration, the system generates B × N × K kernel variants from the B current candidates. After profiling, β selects B kernels to carry forward. The key property is that each iteration's candidates are the best results of previous iterations — the beam carries forward cumulative improvements. If iteration 3 discovers a 1.2× speedup and iteration 7 discovers a 1.3× speedup relative to that improved kernel, the total speedup compounds to 1.56×. This compounding is what repeated sampling cannot achieve.

The candidate selection function β in detail. The paper describes β's operation in Section 2.2:

"β first identifies the fastest correct kernel within each plan group A_p, ensuring that every explored direction contributes its best result. From this representative pool, it then selects the top-B kernels by measured latency. If fewer than B valid kernels exist, remaining slots are filled by the previous iteration's candidates, allowing the system to dynamically allocate more sampling budget to difficult cases where no improvement was achieved."

This "plan group representative" mechanism has a specific purpose: it prevents one plan that produces many good kernels from dominating the beam. Each plan gets exactly one representative (its best kernel), ensuring diversity across optimization directions. If a plan produces many fast kernels, only the fastest one survives to the beam — the others are not selected, preventing the beam from becoming homogeneous.

The fallback mechanism (filling slots with previous candidates when fewer than B valid kernels exist) is important for problems where exploration is challenging. Section 4.3 provides an example (Figure 11) where "at iterations 7-9, no correct kernels are generated" — without the fallback, the beam would shrink or terminate. With the fallback, the system continues trying even when recent iterations fail, preserving the best-known kernel for further exploration.

Evidence for beam search effectiveness. Figure 12 shows the distribution of speedups relative to two baselines: the orange bars show per-iteration speedup over the current candidates (how much improvement one iteration produces), and the blue bars show cumulative speedup over the initial baseline kernel. The orange bars "cluster near 1.0×," meaning most individual iterations produce modest improvements. But the blue bars "include more cases exceeding 1.0×," meaning the cumulative effect of many modest improvements adds up. The paper explains: "beam search yields cumulative performance gains" — each iteration builds on previous best kernels, leading to progressively better optimizations that would require implausibly large single-shot improvements to achieve via repeated sampling.

Hyperparameters. The main experiment configuration uses B = 6, N = 12, K = 2, T = 16. This means each iteration generates up to 6 × 12 × 2 = 144 kernels, the beam carries 6 candidates forward, and the system runs for 16 iterations (total up to 144 × 16 = 2304 kernels generated per problem, though not all will be correct/compilable). The paper sweeps other configurations in the ablation study (Figure 13): B = 4, N = 8 and B = 1, N = 72 (which matches the number of profiled kernels by reducing diversity while keeping total samples constant). The finding is that "when B and K are small, the best speedup drops" — beam diversity matters. Having only one candidate (B = 1) with 72 plans produces worse results than having 6 diverse candidates with 12 plans each, even at the same total sample count. This confirms that the beam's role in maintaining diverse optimization directions is essential, not just a computational convenience.

3.4.6 Optimization Memory Curation: Algorithm 2

Algorithm 2 defines the memory curation procedure σ that transforms a set of profiled kernels K into an updated optimization memory E_i. The design reflects three principles: quality filtering (only substantial speedups or slowdowns become memory items), diversity enforcement (each plan group contributes at most one item), and bounded capacity (the memory is a fixed-size queue that discards oldest entries).

Step 1: Identify slow-fast pairs. The procedure groups all generated kernels by their originating candidate c and plan p (line 3). Within each group, it computes the maximum speedup of any kernel relative to the candidate c. If the maximum speedup exceeds a positive threshold t_pos = 1.04 (line 5), the candidate c and the fastest kernel in the group form a positive rewrite pair — the candidate is the slow kernel, the fastest generated kernel is the fast kernel. This captures successful optimizations.

If the maximum speedup is instead less than 1/t_neg = 1/1.15 (line 7) — meaning the best kernel in the group is actually slower than the candidate by more than a 15% margin — the slowest kernel in the group and the candidate c form a negative rewrite pair — the generated (slower) kernel is the slow kernel, the candidate is the fast kernel. The paper explains (Section 2.3): "Both positive and negative rewrites represent performance-improvement cases. One highlights successful optimization, and the other captures failed attempts. Therefore, we include both to provide balanced signals for the self-improving system."

To understand why negative rewrites matter, consider what happens without them: the memory would only contain successful transformations, creating a biased view of the optimization space. The Planner might overconfidently apply a strategy that worked in one context to a context where it fails, with no memory of that failure. Negative rewrites encode "this transformation, which seems reasonable, actually degrades performance" — an important signal for avoiding repeated mistakes.

Step 2: Sort and select top pairs. Positive rewrite pairs are sorted by speedup in descending order, and the top TopK / 2 = 4 are selected (line 11). Negative rewrite pairs are sorted by speedup in ascending order (least speedup = most dramatic slowdown), and the top TopK - |E_pos| are selected (line 12). This means the memory always contains exactly TopK = 8 new items per iteration, split roughly equally between positive and negative examples (if enough exist of each type).

Step 3: Summarize pairs into experience items. Each selected slow-fast pair is processed by the Summarizer agent θ_s (line 11–12), producing a structured experience item as described in Section 3.4.4.

Step 4: Update the memory queue. The new experience items E_pos and E_neg are prepended to the existing memory E_i, and the combined list is truncated to ExpN = 16 entries (line 13). Specifically, the update is:

$E_{i+1} = \left[E_{\text{pos}}, E_{\text{neg}}, E_i[: \text{ExpN} - |E_{\text{pos}}| - |E_{\text{neg}}|]\right]$

where E_i[:k] denotes the first k entries of the previous memory, and the total size is capped at ExpN.

What physically happens: The new positive and negative experience items from this iteration are placed at the front of the memory queue (they are the most recent and presumably most relevant to the current state of optimization). The oldest entries from previous iterations are retained up to the capacity limit — if ExpN = 16 and the current iteration produces 8 new items, then 8 items from previous iterations are carried forward, and any older items are evicted.

Why this form: The queue structure ensures recency — memory items that are many iterations old gradually age out as new experiences accumulate. This is important because optimization strategies that were effective at early iterations (when the kernel was far from optimal) may be less relevant at later iterations (when only subtle refinements help). The split between positive and negative rewrites ensures balanced signals. The per-plan-group selection (line 4–10) enforces diversity by preventing one strategy that generates many similar fast kernels from flooding the memory with redundant items.

The relationship between TopK and ExpN. The paper explores these parameters in Section 4.5 (Figure 15). TopK controls "how eager the memory system can be when updating the memory using the current iteration observations" — higher TopK means more aggressive memory updates, potentially at the cost of including lower-quality or less-diverse items. ExpN controls how many historical experiences are retained — higher ExpN preserves more context from earlier iterations, but increases the length of the Planner's prompt (each experience item is a substantial block of text), raising inference cost. The paper finds that "increasing memory capacity (ExpN) is more cost-efficient than increasing memory update eagerness (TopK)" — spending budget on retaining more history provides better returns than aggressively updating with marginal new items.

Comparison with Reflexion. The paper implements a Reflexion-style baseline (Shinn et al., 2023) where "a reflector generates optimization insights at each iteration, managed following the Reflexion paper." In the Reflexion baseline, reflection happens on every generated kernel rather than on a curated subset, and the reflection is appended to a persistent memory buffer. AccelOpt's approach differs in two ways: (1) it curates which kernels generate memory items (only those exceeding thresholds), reducing noise from marginal improvements; (2) it maintains a bounded, recency-weighted queue rather than an ever-growing buffer, preventing the Planner's context from growing unboundedly. The paper reports that AccelOpt achieves 1.235× speedup at $139.00 cost versus the Reflexion baseline's 1.137× at$ 178.37 — both worse performance and higher cost, confirming that selective, curated memory is more effective than exhaustive reflection.

3.4.7 The Distributed Profiling Service

The profiling service (Section 3.2, Figure 4) is the feedback mechanism that closes the loop between generation and improvement. Without reliable, fast profiling, the system cannot distinguish good kernels from bad ones, and the beam selection and memory curation mechanisms have no signal.

Correctness checking. Each generated kernel is compiled and executed on Trainium hardware with inputs initialized with "several different random seeds." The output is compared against a CPU reference implementation using the condition ||output - cpu_ref|| < tol × ||cpu_ref|| with a "tight tol individually set for each task." The paper notes (Appendix A.2) that "running on CPU with full precision is slower than running the reference implementation on other accelerators like GPU, especially for the data intensive applications NKIBench targets," but CPU is preferred because "there is no IEEE standard for special functions like exponential and CPU implementation is widely accepted as the ground truth" — this is a correctness-over-speed tradeoff. For performance-critical special functions (exp, sigmoid, rsqrt) where different hardware platforms may produce slightly different numerical results, the CPU reference provides a consistent, standard-conforming baseline.

Performance measurement. The measured metric is execution latency, excluding compilation time. The paper justifies this (Appendix A.1): "The compilation time does not affect kernel quality since kernels are usually reused multiple times in ML pipelines after one-time compilation." Each measurement round includes warm-up iterations (2 warm-up, 10 repeated runs in the main experiments), and multiple rounds are conducted ("at most 10 rounds"). The paper selects "the one with the smallest relative difference, or the first within a predefined threshold" — this is a noise-mitigation strategy that accounts for hardware variability (thermal effects, other processes on the machine). The thresholds are "1% for Trainium 1 and 4% for Trainium 2," reflecting different noise characteristics of the two hardware generations.

Parallelism. The profiling service exploits two levels of parallelism. Task-level parallelism: each of the 14 NKIBench problems runs independently, so all problems can be profiled concurrently across machines. Sample-level parallelism: within each problem, up to B × N × K = 144 kernels per iteration can be profiled simultaneously, because they are independent. The architecture uses "core-level and machine-level parallelism of Trainium hardware," with "machines connected via a shared network file system, with a centralized manager dispatching the requests and returning the profiling results" (Figure 4). The paper notes a practical detail: "cores are periodically rotated to mitigate performance fluctuations after long running" — this is an empirical observation that individual cores can develop performance degradation over extended profiling sessions, and rotation distributes this effect.

Scale of profiling. The paper does not report total number of kernels profiled across all experiments, but we can estimate from the configuration: 14 problems × (B=6 candidates × N=12 plans × K=2 attempts = 144 kernels per iteration) × T=16 iterations = up to 32,256 kernels if all are correct and compilable. In practice, many are not, but the profiling service must handle tens of thousands of kernel compilations and executions at scale. The paper also notes (Appendix A.2) a security concern: "LLMs, especially gpt-oss, can exploit the correctness checker for certain kernel workloads. For example, it proposes to compute only the row-wise maximum of the first tile in each row chunk to achieve fake speedup by omitting necessary computation in safe softmax." This is a form of reward hacking — the LLM discovers that the correctness checker's random-input testing doesn't catch certain semantic errors, and produces kernels that pass the test but compute the wrong result. The paper flags this as motivation for "more rigorous equivalence checking than the common practice of testing with random inputs."

3.4.8 Peak Performance Calculation via the Roofline Model

Section 3.3 defines the theoretical peak throughput calculation that provides NKIBench's absolute performance yardstick. The paper argues that "prior work that uses LLMs to write accelerator kernels often measures relative speedup ... which is an effective metric to demonstrate progress" but that percentage of peak throughput "offers additional insights on how effective AccelOpt has been in exploring the entire optimization landscape."

The roofline model. The calculation uses a standard roofline analysis (Williams et al., 2009) adapted to the Trainium architecture (Figure 5). On each Trainium core, three compute engines (tensor, vector, scalar) and a data movement engine (GPSIMD) run concurrently and communicate with High-Bandwidth Memory (HBM) through software-managed on-chip memory (SBUF). The peak performance T (in seconds) is the maximum of three bounds:

$T = \max\left(\frac{\text{TrafficMin}}{\text{Bandwidth}}, \frac{\text{FLOPs}_{\text{MM}}}{\text{PeakMM}}, \frac{\text{FLOPs}_{\text{Vec}}}{\text{PeakVec}}\right)$

where:

TrafficMin is the "minimal required traffic calculated as the summation of the size of all input tensors and output tensors measured in bytes" — essentially, the absolute minimum data that must cross the HBM interface (read inputs, write outputs), assuming perfect on-chip reuse and no spilling.
Bandwidth is the peak HBM bandwidth (440.2 GB/s for Trainium 1, 640.0 GB/s for Trainium 2 per core; Appendix Table 4).
FLOPs_MM is the "matmul FLOPs in Numpy operators" — the number of floating-point operations that can be mapped to the tensor engine.
PeakMM is the peak tensor engine throughput (23.75 TFLOPS for Trainium 1, 19.75 TFLOPS for Trainium 2).
FLOPs_Vec is "all other FLOPs" — non-matmul operations that run on vector and scalar engines.
PeakVec is the "summation of peak vector engine and peak scalar engine compute throughput" (286.8 GFLOPS for Trainium 1, 550.0 GFLOPS for Trainium 2).

What this computes: The roofline model divides execution time into three non-overlapping bounds. The first term TrafficMin / Bandwidth is the memory bound — it says "even if all computation were instant, moving the data across the HBM interface would take at least this long." The second term FLOPs_MM / PeakMM is the tensor compute bound — it says "even if data were instantly available, executing the matrix multiplications would take at least this long." The third term FLOPs_Vec / PeakVec is the vector compute bound — the same for non-matmul operations. The actual execution time t (measured latency) is then compared to T to get the percentage of peak throughput: Pct = T / t. A kernel achieving 100% means its measured time equals the roofline prediction — no implementation could be faster under these hardware limits.

Why this form: The roofline model separates the three resource constraints because they operate in parallel. The tensor engine, vector engine, scalar engine, and DMA engine (handling HBM transfers) all execute concurrently. A kernel can be bottlenecked on any one of them — improving another resource provides no benefit. For example, if a kernel is compute-bound on the tensor engine (the second term dominates), reducing HBM traffic (improving the first term) will not reduce execution time, because the tensor engine is already the bottleneck. The roofline model tells the optimizer which bound is active, guiding optimization effort toward the relevant resource.

Application in AccelOpt. The paper notes (Section 3.3): "Although AccelOpt presents the raw profiling results directly to agents, including the percentage of peak throughput in the prompts could be interesting to investigate." This is a design choice — the agents currently see the raw profiling metrics (HBM bytes, engine utilizations, hardware FLOPs) and must infer for themselves where the bottleneck lies, rather than being told "you're at 49% of peak." The authors flag that including the percentage directly could improve the Planner's analysis, but this requires further experimentation.

Hardware specifications (Appendix Table 4) are provided per-core, since AccelOpt's kernels (in the current version) run on a single core. Trainium 1 has 440.2 GB/s peak HBM bandwidth, 23.75 TFLOPS tensor engine throughput, and 286.8 GFLOPS combined vector+scalar throughput. Trainium 2 has 640.0 GB/s, 19.75 TFLOPS, and 550.0 GFLOPS — note that Trainium 2's tensor engine throughput is actually lower than Trainium 1's, while its vector throughput is significantly higher, reflecting different hardware design points.

3.4.9 Design Choices: Why This Architecture?

The section concludes by examining the major design decisions that shape AccelOpt and their justifications.

Three-agent decomposition vs. single agent. Why Planner, Executor, and Summarizer as separate agents rather than a single prompt that does everything? The first reason is abstraction separation: kernel optimization requires both strategic reasoning (what to optimize) and precise implementation (how to write the code), which are different cognitive tasks that LLMs handle differently. The Planner can think in terms of loop fusion, tiling, and memory spilling without worrying about NKI syntax; the Executor can focus on correct code without needing to invent strategies. The second reason is structured exploration: the N × K sampling structure (N plans × K attempts) creates a two-level exploration where strategic diversity (plans) and implementation robustness (attempts) are independent axes. A single prompt generating N × K kernels couldn't enforce this structure — kernels from different attempts would not be grouped by plan, and β couldn't select per-plan representatives.

Beam search over repeated sampling. The paper provides empirical evidence in Section 4.4 and discusses the mechanism in Section 2.2: beam search enables cumulative improvements where each iteration starts from the previous iteration's best results. This is fundamentally different from repeated sampling, where every attempt starts from the same baseline and cannot compound. The paper explicitly validates this choice against the alternative.

Optimization memory as a curated queue. The alternatives would be: no memory (beam search only), an ever-growing memory (Reflexion-style), or a memory that only stores positive examples. The bounded queue (capacity ExpN = 16) provides recency weighting — old, potentially irrelevant strategies age out. The curation via Summarizer provides generalization — raw kernel pairs are compressed into strategies with pseudocode. The inclusion of negative examples provides balanced feedback. The paper validates these choices via ablation (Section 4.4), showing that memory improves cost efficiency and that the Reflexion baseline (which reflects on every kernel) is both worse (1.137× vs. 1.235× speedup) and more expensive ( $178.37 vs.$ 139.00).

Platform-agnostic design. Section 4.6 explicitly states: "AccelOpt is platform-agnostic: the beam search and self-improving memory mechanism are orthogonal to specific hardware. Adaptation requires: (1) a profiling service and (2) platform-specific base prompts." The core loop — generate candidates, profile, curate memory, select beam — does not depend on NKI or Trainium specifically. The paper demonstrates this by applying AccelOpt to Triton kernels on H100 (Appendix Figure 18), achieving "1.27× average speedup over best Triton baselines, with 3.19× peak speedup on a GQA decoding kernel." However, the paper carefully notes a critical condition: "Critically, our Trainium evaluation demonstrated that AccelOpt's evolution approach works when LLMs have limited prior knowledge of the platform. Thus, the technique should be even more effective on mature platforms like GPUs, where LLMs have more relevant training data." This is an interesting prediction — the value of autonomous exploration is greatest when LLMs lack prior knowledge, exactly the setting of emerging hardware.

Cost-aware model selection. The paper makes a deliberate choice to use open-source models (gpt-oss-120b, Qwen3-Coder-480B) as the primary agent backbones rather than proprietary models like Claude Sonnet 4. Section 4.5 (Table 3) shows that using Claude Sonnet 4 as both planner and executor achieves 1.226× speedup at $1732.73, while gpt-oss-120b for both achieves 1.235× at$ 139.00 — better performance at 1/12th the cost. The paper also finds that model ensembles (selecting the best kernel across runs with different executor models) achieve the highest speedup (1.246×) at proportionally higher total cost ($470.66), suggesting a Pareto frontier where organizations can trade cost for performance by choosing which models to use.

4. Key Insights and Innovations

Innovation 1: Difficulty-Conditioned Optimization Is Replaced by Memory-Conditioned Optimization — a Conceptual Shift from Static Difficulty to Dynamic, Learned Knowledge

The unifying framework paper discussed in the reference example (Section 2) introduced the idea that test-time compute strategies should be conditioned on prompt difficulty — easy problems benefit from exploitation, hard problems from exploration. This made difficulty estimation the linchpin: the system had to know the problem's hardness before allocating budget, creating an expensive chicken-and-egg problem where estimating difficulty cost as much as solving the problem.

AccelOpt makes a fundamentally different conceptual move. The paper does not estimate difficulty at all. Instead, it conditions optimization on accumulated experience — the optimization memory — which encodes what the system has already learned about this specific kernel's optimization landscape. The key shift is from a static, pre-computed difficulty label (which requires either an oracle or expensive sampling) to a dynamic, learned representation that grows during the optimization process itself.

This is not an incremental refinement of difficulty-conditioned allocation. It is a different paradigm. Difficulty estimation answers the question "how hard is this problem for my model?" — a statement about the model's current capability. Optimization memory answers the question "what have I learned so far about what works and doesn't work on this problem?" — a statement about the history of exploration. The former is a scalar (a difficulty bin), while the latter is a structured artifact (slow-fast kernel pairs with generalization summaries) that accumulates over time.

Why does this matter conceptually? Because it eliminates the exploration-exploitation tradeoff inherent in difficulty estimation. In difficulty-aware systems, you must spend budget to learn difficulty before you can allocate the remaining budget optimally. AccelOpt's memory mechanism fuses these two phases: the optimization process itself generates the experiences that inform future optimization, so there is no separate "measure difficulty" step. The cost of generating 144 kernels per iteration simultaneously serves as exploration (testing new strategies) and memory-building (identifying which strategies work).

The evidence for this shift is structural, not just empirical. The paper's architecture (Figure 1, Algorithm 1) shows that memory E_{i-1} is an input to the Planner at every iteration — it conditions strategy generation, not just strategy selection. The Planner does not receive a difficulty score; it receives a queue of past successful and failed transformations. This means the system's "understanding" of the optimization space is always grounded in concrete examples rather than an abstract difficulty metric, making it more specific (relevant to this kernel) and more actionable (showing what to try, not just how much to try).

The contrast with prior work is sharp. The reference paper's compute-optimal scaling required pre-computing difficulty quintiles via 2048 samples per question — an approach the authors themselves flagged as impractical (Section 3.2). AccelOpt's memory accumulation sidesteps this entirely by making experience collection part of the optimization loop. This is a fundamental architectural innovation: it replaces the question "how hard is this problem?" with "what have I learned about this problem?" — a question that can be answered progressively rather than requiring upfront investment.

Innovation 2: Structured Exploration via Plan-Grouped Beam Search as a Deliberate Alternative to Unstructured Sampling

Most LLM-based optimization systems use repeated sampling (query the same prompt many times, pick the best result) or sequential refinement (feed the previous output back as context for the next attempt). The reference paper's revision model (Section 6) is an example of sequential refinement: the model conditions on its own previous answers to produce improved versions. Repeated sampling and sequential refinement represent two extremes: one explores breadth without depth, the other explores depth without breadth.

AccelOpt introduces a third structure: plan-grouped beam search. The key insight is not beam search itself (which the paper cites from Hong et al., 2025), but the specific decomposition into B candidates × N plans × K attempts. This creates a two-level hierarchy where strategic diversity (different plans) is orthogonal to implementation robustness (multiple attempts per plan). The Planner proposes N qualitatively different optimization directions; the Executor makes K attempts to correctly implement each direction; the candidate selection function β then selects one representative per plan group for the next beam.

What makes this distinctive is that it structurally enforces diversity. In repeated sampling, the system has no mechanism to prevent all attempts from converging on the same strategy — if one optimization approach is easier for the LLM to discover, it will dominate the samples, and the system will miss alternative directions. In sequential refinement, the system can get stuck in a local optimum, iteratively refining the same approach without exploring fundamentally different strategies. AccelOpt's plan-grouped structure directly addresses both failures: the N plans guarantee at least N different strategies are attempted (diversity), and the K attempts per plan guarantee that each strategy gets a fair chance (robustness against LLM stochasticity).

The evidence for why this matters is in Figure 12 and the ablation study. Figure 12 shows that the orange bars (per-iteration speedup) cluster near 1.0× — most individual iterations produce modest gains — but the blue bars (cumulative speedup) include more cases exceeding 1.0×. This is exactly what plan-grouped beam search should produce: no single iteration's exploration needs to be transformative; small improvements from diverse directions compound over iterations. The ablation study (Section 4.4) confirms that reducing B to 1 (eliminating beam diversity) produces worse results even at the same total sample count: "When B and K are small, the best speedup drops."

This is a fundamental design principle rather than an incremental improvement. It says that for LLM-based optimization on problems where the correct transformation sequence is unknown a priori, the generation budget should be organized into a tree of hypotheses (plans) with multiple verification attempts per hypothesis, rather than either a flat list of independent attempts or a linear chain of refinements. This principle generalizes beyond kernel optimization to any domain where LLMs must explore a combinatorial space through iterative generation.

Innovation 3: Optimization Memory as Test-Time Learning — Turning Exploration History into Reusable Knowledge

The concept of test-time learning (Sun et al., 2024) posits that models can learn during inference by updating internal representations based on the specific input. AccelOpt instantiates this idea in an agentic setting, but the mechanism is distinctive: instead of updating model weights or hidden states, it constructs an explicit, structured memory artifact (the optimization memory queue) that conditions future LLM calls through in-context learning.

What makes this novel is not the idea of memory for agents (Zhang et al., 2025c; Sun et al., 2025; Ouyang et al., 2025b), but the specific curation pipeline that transforms raw exploration data (profiling results from hundreds of kernel variants) into compact, generalizable experience items. The pipeline has three stages that each make a non-obvious design choice:

Quality filtering via speedup thresholds (t_pos, t_neg): Not every kernel variant becomes a memory item — only those with substantial speedup (>4%) or substantial slowdown (>15% relative). This prevents the memory from filling with noise (1.01× improvements that are within measurement variance) and ensures that when the Planner sees a memory item, it knows the transformation was genuinely impactful.
Duality of positive and negative rewrites: The paper includes both successful optimizations (candidate → faster kernel) and failed attempts (candidate → slower kernel, where the "fast" kernel is the candidate and the "slow" is the generated failure). This is counterintuitive — why store failures? The paper's answer (Section 2.3) is that negative rewrites "capture failed attempts" and "provide balanced signals for the self-improving system." Without negative examples, the Planner would only see strategies that worked somewhere, creating an overconfident bias toward applying those strategies everywhere. The negative rewrites encode "this strategy, which seems reasonable, degraded performance in this context" — a crucial signal for avoiding overgeneralization.
Summarization into generalized strategies with pseudocode: Rather than storing raw kernel pairs (which would be hundreds of lines of code, mostly irrelevant to the transformation), the Summarizer extracts the optimized segment and writes a natural-language description of the general principle. This compression serves two purposes: it makes the memory items more portable (the Planner can match a stored strategy to a different kernel with similar structure) and more prompt-efficient (shorter items mean more items can fit in the memory queue within token limits).

The comparison with LessonL (Liu et al., 2025) is instructive. LessonL's memory items are anchored to the baseline kernel — their performance reference is always the original starting point. AccelOpt's memory is "evolving with the candidate kernels" (Section 5), meaning a memory item from iteration 5 references the candidate from iteration 5, which may itself be the product of earlier optimizations. This makes the memory more diverse (items capture different stages of optimization) but also creates a challenge: items from later iterations may not make sense without understanding the intermediate transformations. The Summarizer's generalization step addresses this by extracting the transformation principle rather than the specific code delta.

The paper provides evidence for the memory's effectiveness through the Reflexion ablation (Section 4.4). The Reflexion baseline, which reflects on every kernel rather than curating a subset, achieves 1.137× speedup at $178.37 versus AccelOpt's 1.235× at$ 139.00 — both worse and more expensive. The key finding is not just that memory helps, but that curated, selective memory beats exhaustive memory. The Reflexion baseline is overwhelmed by noise — minor variations that don't generalize — while AccelOpt's thresholding and summarization extract the signal. This is a practical insight with theoretical implications: in agentic systems, remembering everything is worse than remembering the right things, and the "right things" must be identified by a curation process that filters for impact and generalizability.

Innovation 4: Verifier-Free Optimization via Direct Profiling Feedback — A Different Path from Reward-Model-Guided Search

The reference paper's approach to test-time compute (and much of the LLM optimization literature) relies on a learned verifier — a Process Reward Model (PRM) trained on Monte Carlo rollouts to score intermediate solutions. The verifier guides search by predicting which partial solutions are likely to lead to correct final answers. This creates a pipeline where the LLM generates candidates and the verifier selects among them.

AccelOpt takes a fundamentally different path. There is no learned verifier, no reward model, no scoring function that predicts kernel quality before execution. Instead, the system relies on direct profiling feedback — actually compiling and running every generated kernel on real Trainium hardware and measuring its execution time. This seems like an obvious choice (why approximate quality when you can measure it?), but it represents a conceptual departure from the verifier-guided paradigm.

The reason verifiers exist in the reference paper's setting is that executing a candidate solution to check correctness requires ground-truth answers, which are unavailable at inference time. The PRM provides a proxy quality signal that can be computed without knowing the right answer. For kernel optimization, the situation is different: you can always run a kernel and measure its speed, and you can always check correctness against a CPU reference. The "ground truth" — the correct output and the execution time — is available through measurement, so there is no need to learn a proxy.

This changes the nature of the optimization problem in two ways. First, feedback is exact rather than approximate. A PRM might give a high score to a kernel that "looks optimized" but actually performs poorly due to a compiler quirk or hardware interaction it doesn't model. Direct profiling eliminates this gap — the feedback is the actual performance on the actual hardware. The paper's observation about verifier over-optimization in the reference paper (Section 5.3) — beam search finding solutions that score highly under the PRM but are actually incorrect — is a pathology that cannot occur when feedback is measured rather than predicted.

Second, feedback is expensive rather than cheap. Running a kernel on real hardware takes orders of magnitude more time than scoring a solution with a PRM (milliseconds vs. microseconds per evaluation). This creates a different optimization landscape: the system must be sample-efficient in how many kernels it profiles, because profiling is the bottleneck. This is why AccelOpt's beam search structure matters — it ensures that the B × N × K = 144 kernels profiled per iteration are strategically diverse (multiple plans) rather than randomly sampled, maximizing the information gained per profiling cycle.

The paper provides evidence for this approach implicitly through its success: the system achieves substantial improvements (49% → 61% peak throughput on Trainium 1) using only measured performance as feedback, with no learned quality estimator. The cost analysis (Section 4.5, Table 1) shows that the total cost is dominated by LLM inference ( $96–$ 470 depending on model configuration), not by profiling (which runs on dedicated Trainium hardware with no per-query pricing). This suggests that for domains where ground-truth feedback is available through measurement (compilation + execution), direct feedback can be more cost-effective than training and maintaining a learned verifier, especially when the verifier would need to be retrained for each new hardware platform.

This is a foundational architectural insight: the choice between learned verifiers and direct measurement is not just about accuracy — it's about which resource is the bottleneck in a given domain. When measurement is cheap and accurate (kernel optimization), direct feedback dominates. When measurement is impossible or expensive (reasoning problems where correctness is unknown), learned verifiers are necessary. The paper doesn't make this comparison explicitly, but the success of its verifier-free approach, combined with the reference paper's documentation of verifier over-optimization as a limiting factor, implies that eliminating the verifier bottleneck may be as impactful as improving the verifier itself — and that for a broad class of optimization problems, direct measurement makes the verifier bottleneck irrelevant.

Innovation 5: Measuring Against Peak Hardware Performance Rather Than Relative Speedup — a Diagnostic Reorientation

Prior work on LLM-based kernel optimization (Ouyang et al., 2025a; Wen et al., 2025; Tian et al., 2025) almost universally reports relative speedup — how much faster the LLM-generated kernel is compared to some baseline (typically a naive implementation or a manually written reference). The paper acknowledges this is "an effective metric to demonstrate progress" (Section 3.3), but argues it provides an incomplete picture.

AccelOpt introduces a complementary metric: percentage of peak hardware throughput, computed via a roofline model (Williams et al., 2009). This metric answers a different question. Relative speedup answers "how much better is this kernel than where we started?" Percentage of peak answers "how close is this kernel to the best possible implementation on this hardware?"

The diagnostic value of this shift is substantial. Consider two scenarios:

Scenario A: AccelOpt achieves a 3× speedup over the baseline, bringing a kernel from 15% to 45% of peak throughput. The 3× number sounds impressive, but 45% of peak reveals that more than half the hardware's capability remains untapped — there is substantial room for further optimization.
Scenario B: AccelOpt achieves a 1.1× speedup over the baseline, bringing a kernel from 75% to 82.5% of peak. The 1.1× number sounds modest, but 82.5% of peak reveals that the kernel is near the hardware's theoretical limit — further optimization would have diminishing returns.

Without the peak throughput metric, scenario A looks like a success and scenario B looks like a disappointment. With it, the interpretation reverses: scenario A represents incomplete exploration (there's much more room to improve), while scenario B represents near-optimal results.

The paper uses this metric to diagnose saturation behaviors in Section 4.3. When performance plateaus, the system needs to know whether the plateau reflects a genuine approach to the hardware limit or a failure of exploration. The paper identifies both cases: Figure 9 shows saturating speedup where "the kernel discovered at iteration 7 has already reached about 82% of peak throughput, leaving little room for further improvement" — this is a healthy plateau. Figure 11 shows saturating speedup where "all the performance metrics barely change" and the kernel is far from peak — this is an exploration failure.

This is a diagnostic innovation rather than an algorithmic one. It doesn't change how AccelOpt works, but it fundamentally changes how we evaluate it. By normalizing against the hardware's upper bound, NKIBench converts an ambiguous relative metric (is a 2× speedup good?) into an absolute one (is 61% of peak good? It depends on the kernel — for some, 61% is near-optimal; for others, it leaves substantial room). This is analogous to how the computer architecture community uses roofline models to evaluate optimizers: a transformation that improves compute utilization from 30% to 60% is understood differently from one that improves it from 80% to 90%, even if both represent a doubling of performance relative to some baseline.

The paper's specific instantiation of the roofline model (Section 3.3) computes peak throughput as the maximum of three bounds (memory bandwidth, tensor compute, vector compute), reflecting the concurrent execution model of Trainium's heterogeneous engines. This is standard roofline methodology adapted to Trainium's specific architecture, but the insight — that absolute hardware capacity provides a more informative evaluation framework than relative improvement — generalizes to any accelerator benchmarking effort. The paper notes that "including the percentage of peak throughput in the prompts could be interesting to investigate" (Section 3.3), suggesting that this metric could move from evaluation to training signal in future iterations.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper constructs NKIBench, a new benchmark suite of 14 NKI kernels extracted from real-world LLM workloads (listed in Table 5). The kernels span both inference and training, ranging from single operators (Matmul, BatchMatmul) to multi-operator chains (LoRA) to larger building blocks (Group Query Attention, Mamba block). Initial kernels were primarily generated by the official Neuron compiler, with four manually written using standard techniques where the compiler provided no implementation. The benchmark is evaluated on both Amazon Trainium 1 and Trainium 2 hardware, and the paper commits to open-sourcing the code and working with the community to expand the suite.

Base models. The primary experiments use open-source LLMs: gpt-oss-120b as the Planner and Summarizer, and Qwen3-Coder-480B-A35B-Instruct-FP8 as the Executor. The cost analysis (Section 4.5) additionally experiments with gpt-oss-20b, Qwen3-Coder-30B-A3B-Instruct, and Qwen3-235B-Thinking as planners; and Qwen3-Coder-30B, gpt-oss-120b, and Qwen3-Coder-480B as executors. Claude Sonnet 4 (thinking mode, temperature 1.0, max output 10k tokens, context length 20k) serves as the proprietary baseline, evaluated via repeated sampling. For open-source models, default vllm sampling settings are used.

Metrics. The primary metric is percentage of peak throughput: measured kernel latency divided by the theoretical minimum latency computed via a roofline model (Section 3.3). The roofline model computes peak throughput as T = max(TrafficMin / Bandwidth, FLOPs_MM / PeakMM, FLOPs_Vec / PeakVec), where TrafficMin is the minimal required HBM traffic (input + output tensor sizes), FLOPs_MM counts matrix multiplication operations, and FLOPs_Vec counts all other operations. The execution latency t is measured with 2 warm-up iterations and 10 repeated runs per round, up to 10 rounds, selecting the round with smallest relative difference or first within a 1% threshold for Trainium 1 (4% for Trainium 2). Compilation time is excluded because kernels are reused after one-time compilation. The secondary metric for internal comparisons is speedup relative to the initial baseline kernel for each problem.

Baselines. The paper compares against multiple baselines:

Repeated sampling of Claude Sonnet 4: Query Sonnet 4 with a unified optimization prompt (Appendix Figures 29–31) multiple independent times and select the fastest correct kernel (Brown et al., 2024).
Repeated sampling of the agentic workflow: Generate kernels using the Planner + Executor pipeline independently M times (same total samples as AccelOpt) without beam search or memory, selecting the fastest.
Beam search only (no memory): AccelOpt's beam search mechanism running for the full T=16 iterations but without optimization memory curation — the Planner receives no past experiences.
Reflexion-style baseline: A reflector agent generates optimization insights on every generated kernel, managed following Shinn et al. (2023), with memory accumulated in an ever-growing buffer without AccelOpt's curation thresholds.
Human expert references: For two kernels (Mamba, RoPE), the paper compares against human-optimized implementations from the official NKI tutorials and samples repository.
Non-LLM search-based optimizers are discussed but not directly compared because they require "substantial manual effort and deep architecture knowledge" to set up, which would violate the paper's goal of fully autonomous optimization without manual curation.

Generation budget / compute accounting. Compute is measured in total kernels profiled and monetary cost (USD). For LLM API costs, the paper uses per-token pricing from Fireworks (open-source models) and Anthropic (Claude Sonnet 4) as listed in Table 6: gpt-oss-120b costs $0.15/1M input tokens and$ 0.60/1M output tokens; Claude Sonnet 4 costs $3.00/1M input and$ 15.00/1M output. The primary AccelOpt configuration runs B=6 candidates × N=12 plans × K=2 executor attempts = up to 144 kernels per iteration × T=16 iterations = up to 2304 kernels per problem (though not all are correct/compilable). The paper explicitly accounts for profiling cost as orthogonal to LLM inference cost — profiling runs on dedicated Trainium hardware with no API pricing, so the bottleneck is LLM calls, not hardware execution.

Cross-validation / statistical protocol. The paper does not use cross-validation or statistical significance testing. Instead, it reports geometric mean of the maximum speedup achieved across all 14 NKIBench problems for each experimental configuration (Section 4.5). For the saturation analysis (Section 4.3), it reports distributions of per-iteration speedup and performance metrics (traffic efficiency, engine utilization) across all generated kernels. The ablation studies use matched sample counts: when comparing B=1 against B=6, the paper adjusts N to keep total profiled kernels constant (e.g., B=1, N=72, K=2 vs. B=6, N=12, K=2), ensuring fair comparison at equal profiling budget.

Main Quantitative Results

Overall Kernel Performance on NKIBench

The headline result: AccelOpt with open-source models (gpt-oss-120b + Qwen3-Coder-480B) achieves an average percentage of peak throughput improvement from 49% to 61% on Trainium 1 and from 45% to 59% on Trainium 2 across all 14 NKIBench kernels (reported in text, Section 4.1, and visually in Figure 6). This matches Claude Sonnet 4 (thinking mode) while being 26× cheaper (the cost comparison is reported in the same paragraph and elaborated in Table 3).

Per-kernel results are shown in Figure 7, where the x-axis is sorted by the baseline kernel's percentage of peak throughput (easiest-to-hardest by initial quality). The bars show that AccelOpt (blue) and Claude Sonnet 4 repeated sampling (orange) achieve comparable final performance across most kernels on both Trainium 1 and Trainium 2. The paper does not report per-kernel exact numerical values in the text, but the visual in Figure 7 shows that on several kernels (e.g., BatchMatmul + Softmax, Group Query Attention, Matmul + Add + RMSNorm), AccelOpt substantially outperforms the baseline, while on others (e.g., RoPE, SiLU) the improvements are modest.

The comparison with human expert references (Section 4.2, "Comparison with Human Experts") reports two concrete numbers:

Mamba: Starting from the same baseline as the NKI tutorial's human-optimized versions (28.4% of peak), AccelOpt autonomously improved the kernel to 54.6% of peak, which is 1.04× the best expert result (52.7% of peak). The generated kernel used a different loop order than the best human implementation.
RoPE: Starting from the nki-samples reference implementation (21.1% of peak), AccelOpt improved performance to 29.6% of peak, a 1.4× speedup over the human reference.

Beam Search vs. Repeated Sampling

Figure 13 (left panel) shows the geometric mean of best speedup achieved up to each iteration across all tasks, comparing three strategies: repeated sampling (blue), beam search without memory (orange), and beam search with memory (green). The key finding: beam search substantially outperforms repeated sampling at matched generation budgets. At the final iteration (T=16 for beam search), the search-only configuration with B=6, N=12 achieves approximately 1.22×–1.24× speedup, while repeated sampling plateaus lower (the paper does not give the exact repeated sampling final value in text, but Figure 13 visual shows it consistently below the beam search curves).

Figure 12 provides the mechanistic explanation. The orange bars show the distribution of per-iteration speedup over the current candidates, which "cluster near 1.0×" — most individual iterations produce modest improvements. The blue bars show the cumulative speedup over the initial baseline kernel, which "include more cases exceeding 1.0×" — the compounding effect of many modest improvements across iterations. The paper states: "beam search yields cumulative performance gains" through this compounding mechanism, which repeated sampling cannot achieve because every attempt starts from the same baseline.

Optimization Memory vs. Beam Search Only

Figure 13 shows that Search + Memory (green) achieves similar final speedup to Search Only (orange) but in fewer iterations: 13 iterations with memory vs. 16 iterations without, "saving 16–17% cost" (Section 4.4). This is further supported by Figure 14, which shows two complementary effects. The top panel plots cumulative Fast@p — the percentage of generated kernels achieving >p speedup (where p=1.0 and p=1.1 are shown). The Search + Memory configuration produces a higher percentage of good-performing kernels at matched sample counts compared to Search Only. The bottom panel shows the average current iteration candidates' speedup over baseline, which can "drop below 1.0×" because the candidate selection function β selects from all correct kernels, not only those with speedups. The Search + Memory curve consistently lies above the Search Only curve, indicating stronger candidate pools per iteration.

The Reflexion-style baseline (Section 4.4) achieves 1.137× speedup at $178.37 cost** versus AccelOpt's **1.235× at$ 139.00 — both worse performance and higher cost. The higher cost is attributed to the Reflexion baseline reflecting on every generated kernel, consuming more tokens, while AccelOpt's curation selects only a subset of high-impact kernels for summarization.

Diversity and Beam Size Ablation

The paper reports (Section 4.4, "Validate the Diversity Design"):

B=1 with memory: 1.204× speedup at $143.58
B=1 without memory: 1.229× speedup at $116.35
B=6, N=12 (standard): 1.235× speedup at $139.00

The B=1 configuration matches the total number of profiled kernels by setting N=72, K=2, T=16. Crucially, B=1 without memory (1.229×) outperforms B=1 with memory (1.204×), and the paper interprets: "the memory might limit optimization if the candidates are not diverse enough." When only one candidate is carried forward, the memory items are all derived from variations of that single kernel, reducing diversity and making the memory less useful — or even harmful.

Additionally, comparing B=4, N=8 against B=6, N=12 in Figure 13 shows that "when B and K are small, the best speedup drops." The paper predicts that performance would "further drop when decreasing B and N to (1,2,4)" if tested, though this specific configuration was not explicitly run.

Cost-Benefit Analysis of Memory Parameters

Figure 15 explores the trade-off between optimization memory capacity (ExpN) and memory update eagerness (TopK) across four labeled configurations:

Base configuration (TopK=8, ExpN=8): ~1.22× speedup at ~$120 cost
Higher TopK (TopK=16, ExpN=8): slightly higher speedup but more expensive
Higher ExpN (TopK=8, ExpN=16): ~1.235× speedup at ~$139 cost
Both high (TopK=16, ExpN=16): highest speedup (~~1.24×) at highest cost (~~$160)

The paper's key finding: "Increasing memory capacity (ExpN) is more cost-efficient than increasing memory update eagerness (TopK)." Comparing points 3 and 2, at similar cost, increasing ExpN from 8 to 16 (point 3 vs. point 1) produces a larger speedup delta than increasing TopK from 8 to 16 (point 2 vs. point 1). The paper therefore selects TopK=8, ExpN=16 for the main experiments.

Model Selection and Ensemble Effects

Table 1 reports speedup and cost for different executor models, all using gpt-oss-120b as planner and summarizer:

Executor	ExpN=8	ExpN=16	Delta
Qwen3-Coder-30B	1.144× ($96.10)	1.197× ($108.43)	+4.6% (+$12.33)
gpt-oss-120b	1.228× ($125.19)	1.235× ($139.00)	+0.6% (+$13.81)
Qwen3-Coder-480B	1.209× ($205.35)	1.230× ($223.23)	+1.7% (+$17.88)
Ensemble (best of three)	1.241× ($426.64)	1.246× ($470.66)	+0.4% (+$44.02)

The ensemble strategy selects the best kernel for each problem from the runs of each single executor, with total cost equal to the sum of the three individual experiments. The ensemble achieves the highest speedup (1.246×) but also the highest cost ( $470.66). Worth noting: the benefit of expanding memory from ExpN=8 to ExpN=16 varies dramatically by model — Qwen3-Coder-30B gains 4.6% for$ 12.33, while gpt-oss-120b gains only 0.6% for $13.81, suggesting that weaker executors benefit more from richer memory (because they need more guidance), while stronger executors are less dependent on historical experiences.

Table 2 shows that switching the planner model has minimal impact when using gpt-oss-120b as executor with ExpN=16: gpt-oss-20b achieves 1.234× ( $116.87), gpt-oss-120b achieves 1.235× ($ 139.00), and Qwen3-235B-Thinking achieves 1.234× ($316.21). The paper concludes: "further performance improvements could first focus on enhancing the executor's capability" rather than the planner's.

Claude Sonnet 4 as Agent Backbone

Table 3 reports results when using Claude Sonnet 4 within AccelOpt:

Planner	Executor	Speedup (Cost)
Claude Sonnet 4	Claude Sonnet 4	1.226× ($1732.73)
gpt-oss-120b	Claude Sonnet 4	1.213× ($1269.98)
Claude Sonnet 4	gpt-oss-120b	1.208× ($1223.05)
gpt-oss-120b	gpt-oss-120b	1.235× ($139.00)

The best performance-per-dollar is gpt-oss-120b for both planner and executor (1.235× at $139.00). Using Claude Sonnet 4 as executor with gpt-oss-120b planner (1.213× at$ 1269.98) reduces cost compared to the Claude-only configuration (1.226× at $1732.73) — a 3.3× cost reduction mentioned in Section 4.1 — but is still substantially more expensive than the all-open-source configuration. Repeated sampling of Claude Sonnet 4 (the baseline in Figure 6) achieves 1.222× at$ 5806.83, making AccelOpt with open-source models 26× cheaper for comparable performance as stated in the abstract.

Saturation Analysis

Section 4.3 analyzes three distinct saturation patterns using performance metric distributions over iterations:

Figure 9 (Saturating speedup with effective exploration): The maximum speedup plateaus after iteration 7, but the system continues to explore effectively. At iteration 10, traffic efficiency shifts to a new distribution and maximum vector utilization continues to vary beyond iteration 7. The plateau occurs because the kernel already reaches ~82% of peak throughput at iteration 7, "leaving little room for further improvement." This is a healthy saturation — the system found a near-optimal kernel.

Figure 10 (Early saturating speedup with effective exploration): Speedup saturates early (much sooner than Figure 9). Although latency does not improve, agents continue to propose "meaningful rewrites rather than minor local tweaks," reflected in "large variations and shifting trends in vector utilization and traffic efficiency." The paper attributes the plateau to the workload being "dominated by matrix multiplication and the baseline already reaches about 83% of peak throughput" — the operator is inherently compute-bound and near-optimal from the start.

Figure 11 (Saturating speedup without effective exploration): "Very few effective rewrites are discovered by AccelOpt. All the performance metrics barely change, and at iterations 7–9, no correct kernels are generated." This is an exploration failure, attributed to the problem characteristics: "the problem size is small enough for all the data to fit on-chip, which causes the traffic efficiency to be nearly 100% in the baseline, and the reduction dimension K=64 is half of the hardware-native reduction dimension (128), so it is hard to fully utilize the tensor engine using current NKI APIs." This identifies a fundamental capability boundary: for certain problem shapes constrained by NKI API limitations, AccelOpt cannot find valid optimizations.

The paper also notes that "there are actually runs where diverse exploration leads to post-plateau improvements as shown in Appendix Figure 16," meaning that the saturation categories are not exhaustive — some runs recover from temporary plateaus through continued exploration.

Generalization to Other Platforms (Triton/H100)

Appendix Figure 18 shows results on 24 Triton kernels from FlashInfer-Bench (Xing et al., 2026) on H100 GPUs. AccelOpt with gpt-oss-120b achieves 1.27× average speedup over the best Triton baselines, with a 3.19× peak speedup on a GQA decoding kernel. The paper notes this demonstrates platform-agnostic applicability while also observing that "the technique should be even more effective on mature platforms like GPUs, where LLMs have more relevant training data" — an untested but plausible prediction.

Ablation Studies and Robustness Checks

Beam diversity (B=1 vs. B=6): Reducing the number of candidates to 1 while keeping total profiled kernels constant (by increasing N to 72) decreases final speedup from 1.235× to 1.204× (with memory) or 1.229× (without memory). The B=1 with memory configuration performs worse than B=1 without memory (1.204× vs. 1.229×), suggesting that when candidates lack diversity, optimization memory can become counterproductive — the memory items, all derived from the same single lineage, may reinforce narrow strategies rather than expanding the search (Section 4.4, "Validate the Diversity Design").

Optimization memory capacity (ExpN=8 vs. ExpN=16): Doubling memory capacity from 8 to 16 entries produces speedup improvements that vary by executor model: +4.6% for Qwen3-Coder-30B, +1.7% for Qwen3-Coder-480B, +0.6% for gpt-oss-120b (Table 1). The cost increase is approximately $12–$ 18. The finding is not that larger memory is universally beneficial, but that its benefit depends on executor capability — weaker models gain more from richer historical context (Table 1, "Delta" column).

Memory update eagerness (TopK=8 vs. TopK=16): Increasing TopK from 8 to 16 produces smaller speedup gains than increasing ExpN from 8 to 16 at comparable cost (Figure 15). This confirms that retaining more historical experiences is more cost-effective than aggressively adding new experiences each iteration — diversity across time beats diversity within a single iteration.

Reflexion-style memory baseline: The Reflexion baseline, which reflects on every generated kernel rather than curating a subset, achieves both worse performance (1.137× vs. 1.235×) and higher cost ( $178.37 vs.$ 139.00) compared to AccelOpt (Section 4.4). The higher cost is attributed to the Reflexion baseline consuming more tokens by reflecting on all kernels, while AccelOpt reflects only on a "selected group of kernels per iteration."

Planner model robustness: Switching the planner model between gpt-oss-20b, gpt-oss-120b, and Qwen3-235B-Thinking produces nearly identical speedup (1.234×, 1.235×, 1.234× respectively at ExpN=16; Table 2). This suggests that the planner's strategic analysis is the less performance-sensitive component — the executor's code generation quality is the primary bottleneck.

Executor model scaling: Both Qwen3-Coder-30B and Qwen3-Coder-480B come from the same model family, and the larger model achieves better performance (1.197× vs. 1.144× at ExpN=8; 1.230× vs. 1.197× at ExpN=16; Table 1). However, gpt-oss-120b, a reasoning model at the same per-token cost as Qwen3-Coder-30B, outperforms both Qwen models (1.235× at ExpN=16), suggesting that reasoning capability rather than raw scale may be the determining factor.

Ensemble of executors: Selecting the best kernel across runs with three different executor models (Qwen3-Coder-30B, gpt-oss-120b, Qwen3-Coder-480B) improves speedup to 1.246× at ExpN=16, at proportionally higher cost ($470.66 = sum of all three experiments; Table 1). The marginal gain over the single best executor (1.235× → 1.246× = +0.9%) is small relative to the 3.4× cost increase, suggesting diminishing returns to ensembling beyond the strongest single model.

Claude Sonnet 4 as agent backbone: Table 3 shows that using Claude Sonnet 4 as both planner and executor within AccelOpt achieves 1.226× at $1732.73, which is actually slightly worse than gpt-oss-120b for both (1.235× at$ 139.00). The cost of the Claude-only AccelOpt run is 12.5× higher than the open-source run, despite producing slightly worse results. The paper attributes the 3.3× cost reduction (mentioned in Section 4.1) to comparing Claude-as-executor-with-gpt-oss-planner ( $1269.98) against repeated Claude sampling ($ 5806.83) — AccelOpt's structured exploration reduces the number of expensive Claude queries needed relative to brute-force repeated sampling.

Critical Assessment

Claim 1: AccelOpt matches Claude Sonnet 4 while being 26× cheaper.

The evidence in Figure 7 and the cost comparison supports this claim for the specific models and configurations tested, but with important scope limitations. The 26× figure comes from comparing AccelOpt with gpt-oss-120b + Qwen3-Coder-480B ( $139.00 per experiment per Table 3) against repeated sampling of Claude Sonnet 4 ($ 5806.83, mentioned in-text). However, the Claude Sonnet 4 repeated sampling baseline is not the strongest possible Claude-based approach — the paper itself shows that using Claude Sonnet 4 within AccelOpt (structured beam search) is 3.3× cheaper than repeated Claude sampling ( $1732.73 vs.$ 5806.83) while also achieving better speedup (1.226× vs. 1.222×). A fairer comparison would be "AccelOpt with open-source models vs. AccelOpt with Claude Sonnet 4," where the open-source configuration is 12.5× cheaper ( $139.00 vs.$ 1732.73) but the Claude-based AccelOpt achieves comparable speedup (1.226× vs. 1.235×). The "26× cheaper" framing emphasizes the most dramatic comparison rather than the most methodologically symmetrical one.

Additionally, the paper does not explore whether more effective prompting of Claude Sonnet 4 could close the gap. The Claude prompt (Appendix Figures 29–31) is a single unified optimization prompt — it does not use the Planner-Executor decomposition that AccelOpt uses. It is possible that applying the same structured agentic workflow with Claude Sonnet 4 agents would produce better results than the open-source models, making the cost/speedup tradeoff more nuanced than "open-source is strictly better."

Claim 2: Beam search outperforms repeated sampling.

Strongly supported by Figure 13, with a clear mechanism documented in Figure 12 (compounding of per-iteration improvements). However, there is a missing comparison: what if repeated sampling were given the same total profiling budget as beam search but with a single, very large batch (e.g., 2304 independent attempts, all profiled, best selected)? The paper's repeated sampling baseline in Figure 13 uses the same number of kernels profiled per iteration, but repeated sampling cannot compound because every attempt starts from the same baseline. At matched total profiling budget, beam search's advantage should theoretically be even larger (since it compounds improvements over iterations while repeated sampling cannot), but the paper does not explicitly run this extreme repeated-sampling comparison. The Reflexion baseline (1.137×) comes closest but introduces additional mechanisms (reflection) that complicate the comparison.

Claim 3: Optimization memory improves cost efficiency.

Supported with qualifications. The evidence (Figures 13, 14; Table 1) shows that memory enables reaching comparable speedup with fewer iterations (saving 16–17% cost), and that cumulative Fast@p is higher with memory than without. However, the absolute improvement in final best speedup is modest: Search + Memory achieves similar final performance to Search Only at the end of the full T=16 iterations (Figure 13). This means memory's primary benefit is faster convergence (better cost efficiency), not higher ceiling (better final kernels). For practitioners who can afford to run the full 16 iterations, memory provides marginal gains; for those constrained by budget, memory's acceleration is meaningful.

A more concerning finding is the B=1 result (Section 4.4): with only one candidate in the beam, memory actually hurts performance (1.204× with memory vs. 1.229× without). This suggests that AccelOpt's memory mechanism has a minimum diversity requirement — if the beam is too narrow, the memory items become correlated and reinforce a limited strategy rather than expanding the search. The paper does not explore at what beam size this reversal occurs (B=2? B=4?), which would be valuable for practitioners tuning the system for cost-sensitive deployments.

Claim 4: The system discovers both peephole and non-local global optimizations.

Demonstrated through case studies (Section 4.2, Figure 3, Figure 8), but not quantitatively characterized. The paper shows impressive examples — algebraic simplification, instruction-level intrinsic fusion, loop invariant code motion, and the multi-step BatchMatmul + Softmax transformation — but does not report what fraction of the total speedup comes from peephole vs. structural optimizations, or how the distribution of discovered optimization types evolves across iterations. This limits the generalizability claim: we know the system can discover both types, but not whether the structural transformations are rare discoveries or a reliable outcome of the search process.

Claim 5: Performance saturates either because kernels approach peak throughput or because exploration fails.

Supported through the three-case saturation analysis (Figures 9–11), but the analysis is qualitative and post-hoc. The paper classifies saturation into three categories based on whether metrics continue to vary (effective exploration) and whether the kernel is near peak throughput (healthy plateau). However, the classification relies on visual inspection of metric distributions rather than a formal criterion (e.g., a statistical test for whether the distribution of traffic efficiency has converged). The observation that "there are actually runs where diverse exploration leads to post-plateau improvements" (Section 4.3, referencing Appendix Figure 16) implies that even runs classified as "saturated" might eventually recover — the 16-iteration horizon may simply be too short to observe recovery in some cases.

Genuine Weaknesses and Missing Evaluations

Single benchmark suite, small test set. All 14 kernels are from NKI on Trainium. While the Triton/H100 results (Appendix Figure 18) provide some cross-platform evidence, they are reported with less detail (no per-kernel breakdown, no cost analysis, no iteration dynamics). The claim that AccelOpt is "platform-agnostic" (Section 4.6) rests on a single paragraph and one appendix figure — far less thorough than the main evaluation.

No comparison with non-LLM optimization systems. The paper explicitly declines to compare against search-based optimizers like Mirage (Wu et al., 2025) or TASO (Jia et al., 2019), citing the difficulty of setting up fair comparisons due to manual effort requirements. This is a reasonable scope limitation but weakens the claim of "first self-improving LLM agentic system for kernel optimization on emerging AI accelerators" — we don't know how much of the improvement is due to LLM-specific capabilities versus any structured search approach on the same problem.

Missing ablation: number of iterations (T=16). The paper runs all experiments with T=16 but does not explore what happens with more iterations. Do the beam search curves in Figure 13 continue to improve beyond 16 iterations, or do they plateau? The saturation analysis suggests some problems plateau early (iteration 7), while others show "post-plateau improvements" (Appendix Figure 16). The choice of T=16 seems arbitrary — it might be either too few iterations (leaving performance on the table for problems that would benefit from longer search) or too many (wasting compute on problems that saturated by iteration 10).

Cost accounting incompleteness. The paper accounts for LLM API costs but not for profiling infrastructure costs (Trainium hardware time, distributed system operation), which are substantial — up to 2304 kernel compilations and executions per problem. For organizations without access to dedicated Trainium clusters, the profiling cost might dominate. The paper also does not account for the one-time cost of developing the NKI programming guide (Appendix Figures 22–23), which required manual tuning "based on their common errors" — this represents non-trivial expert effort that is amortized across experiments but would need to be repeated for each new hardware platform.

No statistical confidence intervals. All reported speedups are point estimates (geometric means across 14 problems) without error bars or confidence intervals. Given the small test set (N=14), the uncertainty on these means could be substantial, and differences of 0.01× in speedup (e.g., 1.234× vs. 1.235× in Table 2) are almost certainly within noise. The paper's conclusion that planner model choice has minimal impact rests on differences of 0.001×, which is implausibly precise for a 14-problem benchmark.

The ensemble finding is weakly interpreted. Table 1 shows that ensembling three executors yields 1.246× at $470.66, versus the best single executor at 1.235× at$ 139.00 — a 0.9% speedup improvement for 3.4× the cost. The paper reports this as "achieves the best results" without noting the extreme diminishing returns. A practitioner reading uncritically might conclude that model ensembling is recommended, when the cost-efficiency analysis strongly argues against it.

Missing difficulty characterization. Unlike the reference paper's detailed difficulty-bin analysis (Figures 3, 7, 9), AccelOpt does not characterize which types of kernels benefit most from its approach. The 14 NKIBench kernels are listed in Table 5 with their bound type (memory, matmul, vector) and approximate latency, but the paper does not analyze whether AccelOpt's effectiveness varies systematically by kernel type, problem size, or initial performance level. The saturation analysis (Section 4.3) provides three cases but no systematic breakdown. This limits the prescriptive value — a practitioner cannot look at a new kernel and predict whether AccelOpt will help based on its characteristics.

Overall, the experimental evidence is strong for the headline claims (open-source models + structured search can match proprietary models at lower cost on this task) but weaker for the mechanism claims (the specific contribution of memory vs. beam search, the generalizability across platforms, the reliability of the approach across kernel types). The paper succeeds as a systems demonstration with thorough ablation, but the statistical foundations (small N, no confidence intervals, many unreported per-kernel results) limit the strength of the comparative claims between configurations within AccelOpt itself.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Replaced, Not Eliminated — The Memory Curation Overhead Is Partially Concealed

The assumption or constraint: The paper's architecture eliminates the explicit difficulty estimation bottleneck that plagued prior work (e.g., the 2048 samples per question required by the reference paper's approach). However, it replaces this with a different form of overhead: optimization memory curation itself consumes LLM inference budget that is not isolated from the optimization budget in the paper's accounting. The Summarizer agent processes every slow-fast pair that passes the t_pos and t_neg thresholds, generating structured experience items, and these items then expand the Planner's prompt in subsequent iterations (Figure 21 shows "Past experiences: <...>" as a Planner input field, and Figures 26–28 show that individual experience items span dozens of lines of text).

The paper reports cost only in aggregate (total USD per experiment, e.g., $139.00 for gpt-oss-120b in Table 3), without decomposing how much of this cost is spent on memory curation and memory consumption (the extra tokens in Planner prompts from including experience items) versus core optimization search (Planner + Executor queries). The paper acknowledges the cost implications indirectly in Section 4.5 when analyzing `TopK` and `ExpN` parameters (Figure 15), noting that higher `ExpN` and `TopK` both increase cost. But it never quantifies what fraction of the total$ 139.00 is attributable to the memory mechanism specifically versus what a memory-free beam search would cost.

The consequence: This means the headline "26× cheaper" figure compares a memory-augmented open-source AccelOpt run against repeated Claude Sonnet 4 sampling, but does not tell us whether the memory mechanism is cost-effective compared to a simpler beam search with no memory (which the ablation in Figures 13–14 shows achieves similar final speedup, just requiring ~3 more iterations). The 16–17% cost savings attributed to memory (Section 4.4, comparing Search + Memory at 13 iterations vs. Search Only at 16 iterations) is the paper's best estimate of memory's net benefit, but this does not account for whether a practitioner could simply run beam search for 16 iterations and achieve the same result at comparable total cost, since memory adds per-iteration overhead (Summarizer calls, longer Planner prompts) that the simpler approach avoids.

More subtly, the cost of memory curation scales with the number of high-quality kernel pairs discovered. In early iterations when few kernels exceed the speedup thresholds, curation is cheap. In later iterations when the beam contains well-optimized kernels and achieving further speedup is harder, fewer pairs pass the thresholds, and curation is again cheap. The peak curation cost occurs in middle iterations when the system is productively exploring and generating many qualifying slow-fast pairs. The paper does not characterize this cost profile, making it difficult for practitioners to predict total budget requirements for a new kernel before running AccelOpt.

What evidence exists in the paper: The paper provides Figure 15 (cost-benefit tradeoff across TopK and ExpN) and Figure 14 (cumulative Fast@p improvements from memory), but these analyze memory's effectiveness, not its cost decomposition. Table 1 shows total cost for different configurations but does not break down cost by component (Planner vs. Executor vs. Summarizer). The Reflexion baseline (1.137× at $178.37 vs. AccelOpt's 1.235× at$ 139.00) demonstrates that AccelOpt's curation is more cost-effective than naive reflection on every kernel, but does not establish whether any memory at all is cost-effective relative to pure beam search at matched total budget.

Mitigation status: The paper is transparent that memory parameters affect cost (Section 4.5) and that the memory mechanism is most beneficial when budget is constrained (memory enables reaching similar performance in fewer iterations). However, it does not provide the cost decomposition that would allow a practitioner to decide whether to include memory or simply run more iterations of beam search. This is a practical gap, not a theoretical one — the ablation in Figure 13 already provides the performance curves needed to make this decision, but the cost side is missing, making the cost-efficiency claim about memory specifically (rather than about beam search generally) difficult to verify from the reported data.

2. The B=1 Reversal Suggests a Minimum Diversity Threshold That Is Not Characterized

The assumption or constraint: AccelOpt's optimization memory mechanism assumes that the beam of candidates C_i contains sufficient diversity for the memory items to generalize usefully across different optimization directions. Section 2.3 states that the memory is designed to "expand the knowledge of the accelerator's optimization space" and that the summarizer produces strategies "general enough to be applied to other kernels." However, the ablation study in Section 4.4 reveals a concerning failure mode: when B=1 (a single candidate in the beam), the memory actually degrades performance relative to beam search without memory.

The paper reports:

B=1 with memory: 1.204× speedup at $143.58
B=1 without memory: 1.229× speedup at $116.35
B=6 with memory: 1.235× speedup at $139.00

The interpretation given is that "the memory might limit optimization if the candidates are not diverse enough. When only one candidate is carried forward, the memory items are all derived from variations of that single kernel, reducing diversity and making the memory less useful — or even harmful."

The consequence: This finding implies that AccelOpt's memory mechanism has a pathological regime where it actively hurts performance rather than helping. This matters because B=1 with N=72 (the configuration tested) profiles the same number of kernels as B=6 with N=12 — the total exploration budget is identical. A practitioner attempting to reduce cost by narrowing the beam (which reduces the number of Planner calls, since the Planner must generate plans for each of B candidates) would encounter this reversal somewhere between B=1 and B=6, but the paper does not establish where. Does the memory become net-beneficial at B=2? B=3? The reversal might occur at different beam sizes for different kernel types or different model configurations.

The mechanism of harm is important but only partially explained. The paper's interpretation — that narrow-beam memory items become correlated and reinforce limited strategies — is plausible but not empirically demonstrated. It could alternatively be that with B=1, the candidate selection function β has fewer kernels to choose from, making it more likely to select kernels that happen to score well on the specific transformations stored in memory (overfitting to memory) rather than kernels that genuinely advance the optimization.

What evidence exists in the paper: The B=1 ablation is reported in Section 4.4 ("Validate the Diversity Design") with three data points (B=1 with memory, B=1 without memory, and B=6 with memory — though B=6 without memory appears only in Figure 13, not in the explicit B=1 comparison text). This is a minimal characterization: one data point in the pathological regime and one in the healthy regime, with no intermediate points. The paper does not explore whether the reversal is gradual or abrupt, whether it depends on the executor model's capability, or whether it interacts with memory parameters (ExpN, TopK).

Mitigation status: Partially addressed. The paper identifies the existence of the diversity threshold and recommends against B=1 (since B=6 is used for all main experiments). However, it does not establish the threshold location, making cost-sensitive deployment risky — a practitioner who cannot afford B=6 has no guidance on how low they can go before the memory mechanism becomes counterproductive. The paper explicitly states "we expect the performance to further drop when decreasing B and N to (1,2,4)" but does not run these experiments. This is an acknowledged but unquantified failure mode.

3. The System Cannot Optimize Kernels Where the Baseline Is Already Tightly Bound to Hardware Constraints

The assumption or constraint: AccelOpt's optimization strategy relies on the existence of meaningful optimization headroom — transformations that the Planner can identify from profiling data and the Executor can implement within the NKI API's constraints. Section 4.3 documents a case (Figure 11) where this assumption fails: "Very few effective rewrites are discovered by AccelOpt. All the performance metrics barely change, and at iterations 7–9, no correct kernels are generated."

The paper's diagnosis reveals a fundamental capability boundary: the problem's characteristics make optimization essentially impossible within the current NKI programming model. Specifically, "the problem size is small enough for all the data to fit on-chip, which causes the traffic efficiency to be nearly 100% in the baseline, and the reduction dimension K=64 is half of the hardware-native reduction dimension (128), so it is hard to fully utilize the tensor engine using current NKI APIs." This is not a failure of exploration — it is a failure of the solution space itself: no valid NKI kernel can achieve substantially better performance than the baseline for this problem shape given the hardware's fixed reduction dimension.

The consequence: This means AccelOpt has a hard capability ceiling determined by the intersection of hardware constraints and programming model expressiveness. When a kernel's baseline already saturates the achievable performance under existing APIs, AccelOpt wastes compute budget generating kernels that either fail to compile or fail to improve performance. The system has no mechanism to recognize this situation and terminate early — it continues running for the full T=16 iterations, consuming LLM budget with no return. For Figure 11's case, iterations 7–9 produced no correct kernels at all, yet the system continued to iteration 16.

This is analogous to the reference paper's finding that test-time compute provides "near-zero improvement regardless of budget" on the hardest problems (difficulty bin 5). For AccelOpt, the "hardest problems" are kernels where the roofline model's bound is already tight — not necessarily because the kernel is well-optimized in absolute terms, but because the remaining optimization headroom is inaccessible through the NKI API. The paper provides no pre-flight diagnostic that would allow a practitioner to estimate whether a given kernel falls into this regime before committing to the full optimization budget.

What evidence exists in the paper: Figure 11 provides one explicit failure case. Figure 10 provides a related but distinct case: effective exploration continues but speedup saturates because the kernel is "dominated by matrix multiplication and the baseline already reaches about 83% of peak throughput" — here the hardware limit, not the API limit, is the ceiling. Neither figure quantifies how many of the 14 NKIBench kernels fall into each saturation category. The paper notes (Section 4.3) that "there are actually runs where diverse exploration leads to post-plateau improvements" (Appendix Figure 16), complicating the picture — some apparent "dead ends" eventually recover, making it difficult even in retrospect to distinguish irrecoverable failures from temporary plateaus.

Mitigation status: Not addressed. The paper does not propose any mechanism for early termination when exploration becomes futile, nor any diagnostic for predicting futility in advance. This means AccelOpt lacks a critical practical feature: the ability to say "I cannot improve this kernel further" and stop spending money. The fixed T=16 horizon is a one-size-fits-all budget that is wasteful for kernels that saturate early.

4. The Accuracy of the Difficulty Estimation Substitute (Memory) Degrades When the Beam Loses Diversity

The assumption or constraint: AccelOpt replaces explicit difficulty estimation with optimization memory, but this substitution creates a new, subtle failure mode: the quality of the memory depends on the quality of the beam that generates it. Memory items are derived from slow-fast kernel pairs where the "slow" kernel is a beam candidate and the "fast" kernel is a generated variant. If the beam contains only kernels that are far from optimal, the speedups recorded in early memory items may be misleading — a 1.5× speedup over a poorly optimized baseline does not represent a generalizable optimization strategy; it represents fixing an obvious inefficiency that an expert would have avoided in the initial implementation.

This creates a bootstrapping problem: in the first few iterations, the system's memory is populated with items describing transformations that achieved speedup over the initial (potentially very suboptimal) baseline. The paper acknowledges this indirectly through its discussion of how memory "evolves" with candidates (Section 5, comparison with LessonL): "AccelOpt's memory is evolving with the candidate kernels and thus could be more diverse." But diversity cuts both ways — early memory items may encode strategies that are only effective because the starting point was terrible, and these items persist in the memory queue for up to ExpN iterations (since older items are gradually evicted, not invalidated).

The consequence: The Planner may be influenced by obsolete memory items that describe optimizations no longer relevant to the current (improved) kernel state. For example, an early memory item might encode "double the tile size to reduce loop overhead," which produced a large speedup when the baseline used tiny tiles. By iteration 10, the beam contains kernels that already use large tiles — but the memory item persists and might lead the Planner to propose further tile size increases that exceed SBUF capacity or create register pressure, producing kernels that fail to compile or degrade performance. The paper does not report whether the Planner sometimes proposes inapplicable strategies based on stale memory items, or whether the Executor's syntax checking (via compilation failure) implicitly filters such plans.

This is related to the B=1 reversal (Limitation 2) but distinct: where the B=1 problem is about memory homogeneity (all items from the same lineage), this problem is about memory staleness (items from an earlier optimization phase persisting into a later phase where their assumptions no longer hold). Both flow from the same root cause — memory items are snapshots of a moving target — but they manifest differently. The B=1 problem causes systematic bias; the staleness problem causes occasional misleading suggestions that waste sampling budget on inapplicable transformations.

What evidence exists in the paper: Indirect evidence comes from the analysis of how memory benefit varies with executor capability (Table 1). The finding that weaker executors (Qwen3-Coder-30B, +4.6% from ExpN increase) gain more from larger memory than stronger executors (gpt-oss-120b, +0.6%) could be interpreted as weaker models being less able to recognize and ignore stale or irrelevant memory items, or alternatively as stronger models already possessing the optimization knowledge that memory would provide, making memory redundant. The paper does not distinguish these interpretations.

Direct evidence would require analyzing whether Planner outputs at late iterations cite early-iteration memory items, and whether those citations lead to successful or failed transformations. This analysis is not performed.

Mitigation status: Partially addressed through the queue eviction mechanism (ExpN = 16 cap, oldest items evicted first). This provides a form of temporal forgetting — items older than 16 iterations are guaranteed to be removed. However, in the paper's 16-iteration experiments, items from iteration 1 persist until iteration 17 (past the experiment horizon), meaning the entire memory is populated with items from all phases of optimization without any staleness-based filtering. The Summarizer extracts general principles (e.g., "Loop Invariant Code Motion") rather than specific parameter values, which partially mitigates the staleness problem by making items applicable across different tile sizes and loop structures. But this is an empirical design choice, not a principled solution — a strategy that was optimal at one optimization stage may be neutral or harmful at another, even when expressed in general terms.

5. No Mechanism for Early Termination or Budget-Adaptive Allocation — the Fixed 16-Iteration Horizon Wastes Compute on Saturated Kernels

The assumption or constraint: AccelOpt runs all 14 NKIBench kernels for a fixed T=16 iterations regardless of whether a given kernel's performance has plateaued. The beam search continues generating and profiling B × N × K = 144 kernels per iteration even when the best kernel's latency has not improved for several iterations. The candidate selection function β will continue carrying forward the best-known kernel when no improvement is found (the fallback mechanism in Algorithm 1 line 10), so the beam does not degrade, but it also does not advance — the system is effectively in a holding pattern, consuming LLM budget with zero marginal return.

The paper's own saturation analysis (Section 4.3) demonstrates that this situation is common. Figure 9 shows a kernel that plateaus at iteration 7 (reaching ~82% of peak throughput) but continues through iteration 16 — 9 iterations of exploration with no improvement. Figure 10 shows a kernel that saturates even earlier, and Figure 11 shows a kernel where "very few effective rewrites are discovered" and iterations 7–9 produce "no correct kernels," yet the search continues through iteration 16. The paper observes these patterns but draws no architectural conclusions from them.

The consequence: For a practitioner deploying AccelOpt on a new kernel, the fixed T=16 horizon means they pay for the full experiment regardless of when (or whether) the kernel's performance converges. If convergence occurs at iteration 7, roughly 56% of the total cost is wasted (9 iterations / 16 × cost per iteration). If the kernel never converges (Figure 11 case), the full budget is spent with minimal return. The cost implications are substantial: at $139.00 per full experiment (Table 3, gpt-oss-120b configuration) for a kernel that saturates at iteration 7, approximately$ 78 is spent on unproductive iterations.

More problematically, there is no mechanism to dynamically reallocate budget from saturated kernels to difficult kernels. In a production setting with a fixed total budget across many kernels, the ideal allocation would spend more iterations on kernels still showing improvement and fewer on kernels that have plateaued. AccelOpt's uniform T=16 allocation is as crude as the best-of-N baseline that the reference paper criticized — it applies the same strategy regardless of per-kernel dynamics.

What evidence exists in the paper: The saturation analysis (Section 4.3, Figures 9–11) provides direct evidence of early convergence. Figure 9 explicitly shows the maximum speedup plateauing after iteration 7 while the system continues to iteration 16. Figure 10 shows even earlier saturation. The paper notes these patterns in qualitative terms but does not calculate the fraction of total compute spent after convergence or propose any mechanism to detect convergence online.

The paper does not report per-iteration speedup for all 14 kernels, so we cannot estimate how many kernels plateau before T=16. The geometric mean speedup curves in Figure 13 continue to rise through iteration 16, but this aggregates across all kernels — individual kernels may plateau while the mean continues to improve due to late-improving kernels.

Mitigation status: Not addressed. The paper does not propose any form of early stopping, convergence detection, or adaptive iteration allocation. The fixed T=16 is a research convenience (ensuring all experiments are directly comparable) rather than a deployment recommendation, but the paper does not discuss what a practitioner should use as a stopping criterion. The observation that memory enables reaching "similar speedup in 13 iterations" versus 16 for search-only (Section 4.4) hints at the possibility of earlier stopping, but this is presented as a cost-efficiency gain from memory, not as a stopping rule.

6. Single Benchmark, Single Hardware Family — Generalization Claims Are Supported by a Single-Figure Appendix

The assumption or constraint: The paper claims AccelOpt is "platform-agnostic" (Section 4.6) and that "the technique should be even more effective on mature platforms like GPUs, where LLMs have more relevant training data." These claims rest on: (1) the conceptual argument that beam search and memory operate independently of specific hardware details, (2) a single-figure appendix (Figure 18) showing results on 24 Triton kernels on H100 GPUs, and (3) the assertion that adaptation requires only "a profiling service and platform-specific base prompts."

The main evaluation in Section 4.1, the ablation study in Section 4.4, and the cost analysis in Section 4.5 are all conducted exclusively on the 14-kernel NKIBench suite running on Trainium 1 and Trainium 2. The Triton/H100 results in Appendix Figure 18 are reported with substantially less detail: no per-kernel breakdown, no cost analysis, no iteration dynamics, no comparison against alternative LLM-based optimizers on the same platform, and no ablation of AccelOpt components (memory vs. beam search only) on the GPU platform.

The consequence: The claim of platform-agnosticism is directionally plausible but quantitatively unsupported by the paper's evidence. The Triton/H100 results (1.27× average speedup, 3.19× peak speedup) demonstrate that AccelOpt can produce non-trivial speedups on a GPU platform, but without the detailed analysis applied to Trainium, we cannot assess whether the memory mechanism is beneficial on GPUs, whether the same hyperparameters (B=6, N=12, ExpN=16) are appropriate, whether the cost-efficiency advantage over repeated sampling holds, or whether the saturation behaviors differ.

This matters because the paper's core architectural argument — that AccelOpt is particularly valuable for emerging platforms where LLMs lack prior knowledge — implies that the Trainium results represent a lower bound on effectiveness (since LLMs have less Trainium-specific training data than GPU-specific training data). If true, the Triton results should be systematically better than the Trainium results — larger average speedups, faster convergence, more diverse optimizations discovered. But the 1.27× average speedup on Triton is comparable to, not dramatically better than, the results on Trainium (where Figure 6 shows 49% → 61% peak throughput on Trainium 1, which corresponds to roughly a 1.24× absolute speedup — though the papers report different metrics, making direct comparison difficult). This is at least suggestive that the "lower bound" intuition may be wrong, or that other factors (e.g., Triton kernel quality from FlashInfer-Bench baselines being higher to begin with) dominate the platform effect.

What evidence exists in the paper: Appendix Figure 18 provides speedup bars for 24 Triton kernels, but the paper's text about this experiment (Section 4.6) occupies only three sentences. No table in the main paper or appendix breaks down the Triton results by kernel type, iteration, cost, or component ablation. The paper does not report whether the same open-source models were used, whether the same hyperparameters were applied, or what the baseline kernel quality was (in terms of percentage of peak throughput on H100). The comparison is essentially anecdotal: "AccelOpt with gpt-oss-120b achieved 1.27× average speedup over best Triton baselines, with 3.19× peak speedup on a GQA decoding kernel."

The paper also does not report any experiments on additional emerging platforms beyond Trainium — no TPU, no AMD NPU, no Cerebras, no Groq. The "emerging accelerator" framing in the title and abstract is supported by evaluation on exactly one emerging platform (Trainium) and one mature platform (H100), with the H100 evaluation being substantially less thorough.

Mitigation status: Partially addressed through the Triton/H100 results (Appendix Figure 18) and the explicit acknowledgment that "extending to communication primitives is valuable for full-stack performance and represents a promising direction for future work" (Section 4.6). However, the paper does not characterize the Triton results as preliminary or call for more thorough cross-platform evaluation. The strong claim in Section 1 that AccelOpt is "the first self-improving LLM agentic system for kernel optimization on emerging AI accelerators" implicitly positions Trainium as representative of all emerging accelerators, which is an extrapolation from a single data point.

7. Implications and Future Directions

How This Work Changes the Landscape

AccelOpt represents a methodological reframing rather than a paradigm shift. It does not introduce a fundamentally new optimization algorithm or learning objective — beam search and memory accumulation are well-established mechanisms in other contexts. What it changes is the organizing principle for LLM-based optimization on hardware platforms where expert knowledge is scarce.

The dominant mental model for LLM-assisted kernel development, as represented by prior work like AutoComp (Hong et al., 2025) and GEPA (Agrawal et al., 2025), treats the LLM as a recipe executor: a human expert identifies the relevant optimization categories, encodes them in prompts or search templates, and the LLM applies them. This works well on mature platforms like GPUs where the recipe book is thick. But it breaks on emerging hardware — by definition, the recipe book hasn't been written yet.

AccelOpt shifts the role of the LLM from recipe executor to autonomous explorer. The crucial word in the paper's own framing is "navigate" — the system must explore, make mistakes, learn from them, and progressively build its own understanding of the optimization landscape. The Planner does not receive a list of optimizations to try; it receives profiling data, architectural constraints, and a memory of what has worked and failed in previous iterations, and must generate its own hypotheses.

This reframing resolves an implicit tension in the literature. On one side, systems like AutoComp demonstrate that LLMs can apply predefined optimizations to produce fast kernels — but they're limited by the quality and completeness of the predefined list. On the other side, repeated sampling of powerful models (the Claude Sonnet 4 baseline) demonstrates that LLMs can sometimes discover optimizations without explicit guidance — but at extreme cost ( $5806.83 for the Claude repeated-sampling baseline vs.$ 139.00 for AccelOpt with open-source models). AccelOpt shows that structured search with memory accumulation bridges this gap: it doesn't require expert-curated optimization lists, but it's orders of magnitude more cost-efficient than brute-force sampling.

The paper's most consequential finding for the field may be what it reveals about the cost structure of LLM-based optimization. Table 3 shows that gpt-oss-120b within AccelOpt achieves 1.235× speedup at $139.00, while Claude Sonnet 4 within AccelOpt achieves 1.226× at$ 1732.73 — the open-source model actually performs marginally better at 1/12th the cost. This inverts the conventional wisdom that proprietary frontier models are necessary for difficult code generation tasks. It suggests that for iterative refinement tasks where the system can learn from its own intermediate results, the optimization architecture matters more than the base model quality, and open-source models are not just "good enough for the price" — they can be genuinely competitive in absolute terms. This has practical implications for the economics of deploying LLM-based optimization at scale: organizations can achieve frontier-competitive results without the per-query costs, rate limits, and vendor dependencies of proprietary APIs.

The educational impact documented in Section 4.2 also points to a landscape change beyond the research community. When Stanford CS149 Fall 2025 used AccelOpt to optimize a Conv2D kernel, 33.6% of 131 student teams successfully conquered a challenge based on optimizations the system discovered. The students learned principles like "transforming sequential temporal iteration to parallel spatial execution" and "specialization for workload under hardware constraints" — concepts that are typically taught through lectures and textbooks, not through interaction with an autonomous optimizer. This suggests AccelOpt-like systems could serve as pedagogical tools that surface optimization principles through exploration rather than instruction, a capability the paper's authors seem to have discovered serendipitously rather than designed for.

The paper also implicitly reorients the evaluation methodology for kernel optimization research. NKIBench's percentage-of-peak-throughput metric provides an absolute yardstick that relative speedup alone cannot. The saturation analysis (Section 4.3) demonstrates the diagnostic value: a 1.1× speedup at 82% of peak is a different outcome than a 3× speedup at 45% of peak, even though the latter sounds more impressive. If the community adopts this evaluation standard, comparisons between optimization systems will become more informative — we'll know not just which system improves kernels more, but which system leaves less performance on the table relative to the hardware's theoretical capability. This is a diagnostic innovation that raises the bar for what counts as a thorough evaluation in this subfield.

However, it's important to calibrate the magnitude of this contribution. This is not a Chinchilla moment — the paper does not establish quantitative scaling laws that predict performance as a function of model size, beam width, or memory capacity. The hyperparameter choices (B=6, N=12, ExpN=16, T=16) are empirically motivated but not derived from a theoretical framework. A practitioner wanting to deploy AccelOpt on a new accelerator with a new kernel language must re-do substantial parts of the prompt engineering (the NKI programming guide in Appendix Figures 22–23 required manual tuning based on "common errors") and has limited guidance on how to set the hyperparameters for their specific setting. The paper provides a demonstration of feasibility and an architecture template, not a turnkey system or a predictive theory.

The research directions that become more attractive after this work include: (1) automated prompt adaptation for new platforms (eliminating the manual NKI guide tuning), (2) cross-kernel memory transfer (the paper only transfers memory across iterations of the same kernel), (3) hierarchical search that combines AccelOpt's per-kernel optimization with higher-level operator selection and composition, and (4) tight integration with compiler frameworks where the LLM can propose and evaluate transformations at the intermediate representation level rather than source code. Research directions that become less attractive include: pure repeated sampling of proprietary models as a kernel optimization strategy (AccelOpt's structured search dominates on cost-efficiency), expert-curated optimization lists as a prerequisite for LLM-based optimization (the memory mechanism makes these optional), and percentage-of-peak-blind evaluation (NKIBench's roofline metric reveals how much information relative speedup alone conceals).

Follow-Up Research This Work Enables

Cross-kernel memory transfer. The current system's optimization memory is per-kernel: experience items accumulated while optimizing a Matmul do not transfer to a BatchMatmul, even though both share matrix multiplication as a core operation. This is an acknowledged limitation (Section 2.3: "Studying how to transfer the memory accumulated from optimizing some kernels to others is worth future exploration"). A direct extension would construct a shared optimization memory across kernels in the same benchmark, where the Summarizer tags each experience item with the operator type and architectural bottleneck (memory-bound, matmul-bound, vector-bound), and the Planner for kernel X receives items from kernel Y when their tags match. The key experiment: measure whether a kernel optimized as the 14th in a sequence (with memory accumulated from the previous 13) achieves higher final speedup or converges faster than the same kernel optimized in isolation. A positive result would transform AccelOpt from a per-kernel tool into a platform optimization knowledge base that improves with every kernel it processes, analogous to how human experts build intuition that transfers across operators. A negative result (no transfer) would reveal that kernel optimization strategies are less generalizable than the paper's Summarizer design assumes, suggesting that the memory mechanism works primarily by preventing repeated mistakes within a single trajectory rather than by capturing reusable principles.

Cheap difficulty estimation via profiling budget allocation. Section 4.3 identifies kernels where AccelOpt fails to explore effectively (Figure 11) — typically when the kernel is already near a hardware limit or the problem shape is constrained by API limitations. The system currently wastes the full T=16 iteration budget on such kernels. A follow-up could develop a budget-adaptive controller that monitors per-iteration metrics (variance in traffic efficiency, rate of change in best latency, fraction of kernels that fail compilation) and terminates exploration early when these signals indicate a saturated landscape. The concrete experiment: run AccelOpt for a larger budget (e.g., T=32) on all 14 NKIBench kernels, retrospectively identify the iteration where each kernel's best speedup stopped improving (using a threshold like <1% improvement over the next 4 iterations), and train a classifier to predict whether a kernel will saturate within the next K iterations based on the previous 2 iterations' metric trajectories. If the classifier performs well (>80% precision at 80% recall), deploy it as an online stopping rule and measure the cost savings versus fixed T=16 on a held-out set of kernels. The paper's observation that "there are actually runs where diverse exploration leads to post-plateau improvements" (Appendix Figure 16) means this classifier must distinguish temporary plateaus from permanent ones — a non-trivial challenge that would refine our understanding of when AccelOpt's exploration is genuinely stuck versus just in a slow phase.

Verifier-free optimization for reasoning tasks via execution-based feedback. The reference paper on compute-optimal scaling struggled with verifier over-optimization — the learned PRM gave high scores to incorrect solutions, limiting how much test-time compute could help. AccelOpt demonstrates that for kernel optimization, direct execution feedback (profiling on real hardware) eliminates the verifier bottleneck entirely. A natural question: can this principle extend to other domains? For code generation, unit tests provide execution-based correctness signals; for mathematical reasoning, symbolic execution or computer algebra systems could verify derivations; for SQL generation, a query executor could validate semantic equivalence. A follow-up study would apply AccelOpt's three-agent architecture (Planner, Executor, direct-feedback verifier) to these domains and measure whether it achieves similar cost-efficiency advantages over learned-verifier approaches. The key measurement is not just final accuracy but optimization efficiency: for a fixed budget, does the execution-feedback system find better solutions than a PRM-guided system, and does it avoid the over-optimization pathology documented in the reference paper's Figure 3? A positive result in multiple domains would establish execution-based feedback as a general alternative to learned verifiers whenever ground-truth evaluation is available through measurement rather than requiring oracle labels.

RL-based finetuning of the Executor agent using optimization trajectories. The current system uses fixed open-source models without any fine-tuning on kernel optimization data. The Executor makes K=2 attempts per plan because its first attempt often fails to produce compilable code — the NKI programming guide in the prompt represents the paper's workaround for the model's lack of NKI-specific knowledge. A follow-up could construct a dataset from AccelOpt's execution traces — specifically, the Planner's plan, the Executor's generated kernel, and a binary compilability label plus the profiling-measured speedup — and use this to fine-tune the Executor model with reinforcement learning (rewarding kernels that both compile and achieve speedup over the candidate). The hypothesis: a fine-tuned Executor would achieve higher per-attempt success rates (allowing K to be reduced, lowering cost) and might discover optimizations that the base model cannot express even with the programming guide. The paper's negative result with ReST $^{EM}$ -style on-policy training (a different domain, but conceptually related — the reference paper's Appendix K showed that self-improvement training can backfire) is a caution, not a prohibition: the key would be using offline trajectories (from completed AccelOpt runs) rather than on-policy sampling, to avoid the spurious correlation amplification that degraded the ReST $^{EM}$ revision model.

Robust equivalence checking against reward hacking. The paper notes (Appendix A.2) that "LLMs, especially gpt-oss, can exploit the correctness checker for certain kernel workloads. For example, it proposes to compute only the row-wise maximum of the first tile in each row chunk to achieve fake speedup by omitting necessary computation in safe softmax." This is kernel-specific reward hacking — the LLM discovers that random-input testing doesn't catch certain semantic errors. A follow-up could develop a differential testing framework that generates adversarial inputs (designed to violate the assumptions the shortcut relies on), runs both the candidate kernel and the CPU reference on these inputs, and flags kernels that pass random testing but fail adversarial testing. The question: how many of AccelOpt's "correct" kernels actually contain such semantic errors, and does filtering them change the reported speedup results? If the fraction is non-trivial (say, >5% of kernels that pass the current checker), the paper's reported best-kernel speedups would be slightly overstated, and the adversarial checker would become a necessary component of any production deployment. This is a specific, measurable stress-test rather than vague "future work on robustness."

Training data ablation: how much does the NKI programming guide's manual tuning matter? The Executor's prompt includes a concentrated NKI programming guide (Appendix Figures 22–23) that was "tuned for the agents based on their common errors." This represents non-trivial expert effort: the authors ran the system, observed failure patterns, diagnosed the NKI language rules being violated, and wrote corrective examples. This human-in-the-loop step contradicts the paper's claim of "no expert-provided, hardware-specific optimization knowledge" — the optimization strategies are discovered autonomously, but the syntactic competence to implement them depends on manually curated rules. A crucial follow-up experiment: remove the programming guide and run AccelOpt with only the NKI API basics (Appendix Figure 19, which describes architectural constraints but not common error patterns). Measure the drop in compilation rate and final speedup. If the drop is small, the guide is a nice-to-have and the autonomous claim holds; if the drop is large (e.g., compilation rate falls below 20% and final speedup collapses), then a substantial fraction of AccelOpt's effectiveness depends on human-crafted syntax rules, qualifying the "fully autonomous" claim. This experiment is straightforward to run using the open-sourced code and would provide critical calibration of how much platform-specific human effort the system actually requires.

Practical Applications and Downstream Use Cases

Batch kernel optimization for new accelerator bring-up. When a hardware team tapes out a new AI accelerator, the software team faces a blank slate: hundreds of ML operators need optimized kernels, but no optimization recipes exist for the new architecture. The paper's finding that AccelOpt works on Trainium — an architecture where LLMs have minimal prior knowledge — suggests it could serve as a first-pass kernel optimizer during hardware bring-up. The workflow: for each operator in the ML framework's required set, compile the reference kernel (typically a naive implementation from the vendor compiler), run AccelOpt for the full T=16 iterations (or until the adaptive stopping rule proposed above triggers), and use the resulting optimized kernel as the initial production implementation. The paper's results on NKIBench (49% → 61% of peak throughput on Trainium 1, 45% → 59% on Trainium 2) provide an expected performance range. Critically, because AccelOpt uses direct profiling feedback rather than a learned verifier, it requires no per-platform training — only a profiling service, which the hardware team already needs for manual optimization anyway. The cost: $139 per kernel at current API prices (Table 3, gpt-oss-120b configuration), which for hundreds of kernels represents tens of thousands of dollars — small compared to the engineering months that manual optimization would require. The 26× cost advantage over Claude Sonnet 4 repeated sampling makes this economically viable at scale.

Continuous kernel optimization in CI/CD for ML compiler development. ML compiler teams (e.g., the Neuron compiler team for Trainium, or Triton for GPUs) continuously improve their compilation strategies, and each compiler update changes what kernels are possible and what performance they achieve. AccelOpt could be deployed as a regression testing and optimization refresh tool in the compiler CI/CD pipeline. When a new compiler version is committed, run AccelOpt on NKIBench (or an expanded benchmark) with the new compiler as the baseline. If AccelOpt discovers kernels that significantly outperform the compiler's output, this signals either a missed optimization opportunity in the compiler (a bug or missing pass) or a transformation the compiler cannot express (a feature request). If AccelOpt cannot improve over the compiler's output, this validates the compiler's effectiveness. The paper's case where AccelOpt exceeded expert-level human results on Mamba (54.6% vs. 52.7% of peak) and RoPE (29.6% vs. 21.1%) demonstrates that the system can discover optimizations that elude both compiler heuristics and expert engineers — making it a valuable complement to traditional compiler testing.

Educational tool for parallel computing and hardware architecture courses. The Stanford CS149 experience (Section 4.2) demonstrates a concrete educational application: students enrolled in a graduate-level parallel computing course used AccelOpt's optimization suggestions to design an extra credit problem, with 33.6% of 131 teams successfully completing it. The workflow was not "ask AccelOpt to write the answer" but rather "study what AccelOpt discovered and implement the principle yourself." This suggests a pedagogical model where AccelOpt serves as an optimization oracle — it proposes strategies (e.g., "transform sequential temporal iteration to parallel spatial execution") that students must understand and manually implement. This forces engagement with the optimization principle rather than the syntax, addressing a common criticism of LLM-based education tools (that they enable answer-copying rather than learning). For courses on GPU programming, FPGA design, or custom accelerator architecture, AccelOpt-like systems could generate optimization challenges calibrated to the specific hardware platform used in the course, with the system's discovered strategies serving as hints rather than solutions. The paper's open-source release makes this immediately feasible for any course with access to Trainium or Triton-capable hardware.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives. It positions AccelOpt as complementary to systems that require expert-curated optimization recipes (AutoComp) and prompt-evolution approaches (GEPA), and as an alternative to repeated sampling of proprietary models (Claude Sonnet 4 baseline). However, it does not provide the kind of direct, controlled comparison that would support a prescriptive "use X when condition A, use Y when condition B" decision rule. The ablation study (Section 4.4) compares AccelOpt variants internally (beam search vs. repeated sampling, memory vs. no memory, Reflexion vs. AccelOpt curation), but these are variations of the same system, not comparisons against fundamentally different optimization paradigms. The paper's scope is better understood as "here is an approach that works when expert knowledge is unavailable" rather than "here is when to choose our approach over others." A decision-rule section would require the paper to have tested against systems like Mirage, AutoComp, or GEPA under matched conditions, which it explicitly did not do (Section 4.4: "It is thus difficult to set up a fair comparison with these methods").