SkillOpt: Executive Strategy for Self-Evolving Agent Skills

🎯 Pitch

Agent skills—the natural-language documents that control a frozen LLM’s tool use and reasoning—usually degrade or drift when iteratively revised from feedback, but SkillOpt shows that treating the skill as a trainable external state with bounded edits, minibatch trajectory reflection, and a strict held-out validation gate turns it into a reliably improvable artifact that adds zero inference cost at deployment.

1. Executive Summary

This paper introduces SkillOpt, a text-space optimizer that treats an external natural-language skill document as the trainable state for frozen LLM agents, applying deep-learning-style training controls—bounded edit budgets (textual learning rates), minibatch reflection over rollout trajectories, held-out validation gates, rejected-edit buffers, and epoch-wise slow/meta updates—to turn skill improvement into a controlled optimization process rather than ad hoc prompt revision. Evaluated across six benchmarks (SearchQA, SpreadsheetBench, OfficeQA, DocVQA, LiveMathematicianBench, ALFWorld), seven target models from frontier-scale GPT to small Qwen, and three execution harnesses (direct chat, Codex, Claude Code), SkillOpt is best or tied-best on all 52 evaluated (model, benchmark, harness) cells, lifting GPT–5.5 by +23.5 points on average over no skill in direct chat and by +24.8/+19.1 points under Codex and Claude Code harnesses respectively, while beating every per-cell competitor drawn from human-written, one-shot LLM, Trace2Skill, TextGrad, GEPA, and EvoSkill skills by +5.4 points on average. The optimized artifacts remain compact at 300–2,000 tokens assembled from only 1–4 accepted edits, and transfer positively across model scales, execution harnesses, and to nearby benchmarks without further optimization, establishing that a compact skill artifact can serve as a practical domain-adaptation layer for frontier agents only when the optimization loop enforces bounded, validation-gated updates rather than uncontrolled rewriting.

2. Context and Motivation

The Core Problem: Agent Skills Are Not Trainable, So They Cannot Reliably Improve

The fundamental gap this paper addresses is that agent skills today cannot be trained like model weights. When deploying a frozen LLM as an agent in a domain—say, to manipulate spreadsheets, answer document-based questions, or solve competition math problems—the agent's behavior depends heavily on its skill document: a natural-language artifact that packages procedures, tool-use policies, output formats, and failure-handling rules. Yet the dominant approaches for creating these skills—hand-writing by a human expert, generating once with a one-shot LLM prompt, or loosely revising from trajectory feedback—are all ad hoc. None behaves like an optimizer, and none reliably improves over its starting point under feedback.

This is a problem of control. In deep learning, when we want a model to improve on a task, we use an optimizer (SGD, Adam) with well-understood control mechanisms: gradient computation over minibatches to average out noise, learning rates and schedules to control step sizes, validation sets to gate updates and prevent overfitting, and momentum terms to carry stable directions across updates. These mechanisms are not decorative—they are what make weight-space optimization reproducible, stable, and generalizable. But no such discipline exists for text-space optimization of agent skills. Instead, the field has a collection of loosely controlled self-revision or feedback-driven rewriting methods where:

A single trajectory's failure can trigger a large, unbounded rewrite of the entire skill, erasing useful rules that worked for other examples.
There is no mechanism to average evidence across multiple rollouts before deciding what to change, so anecdotal failures drive edits.
Rejected edits are discarded rather than recorded as negative feedback for future updates, so the system can repeat harmful changes.
There is no validation gate to prevent plausible-sounding but harmful textual changes from accumulating in the deployed skill.
There is no separation between fast local updates and slower cross-epoch consolidation, so longer-horizon patterns are lost.

The paper's core argument is that the skill document itself should be trainable, with the same optimization discipline that makes weight-space training reliable. The paper formalizes this by placing the skill as an external state of a frozen agent model and building a separate optimizer model that edits this state under explicit training-style controls. This is not a minor engineering improvement—it is a conceptual reframing of what skills are: not static prompts, but trainable procedural artifacts that should be optimized through evidence, validation, and controlled step sizes.

Why This Problem Matters

The practical significance is immediate and multi-faceted.

Procedural benchmarks expose the brittleness of zero-shot frontier models. The paper's results make this vivid: on SpreadsheetBench, GPT–5.5 achieves only 41.8% with no skill; on OfficeQA, 33.1%; on LiveMathematicianBench, 37.6% (Table 1). These are not knowledge-gap failures—GPT–5.5 possesses the necessary factual knowledge and tool-use capability—but procedural failures: the model doesn't know to inspect workbook structure before writing formulas, to bind answers to exact document regions, or to prefer theorem-strength ordering in multiple-choice math questions. These are exactly the kinds of procedural rules that a well-optimized skill document could supply, but existing skill-creation methods cannot reliably discover and encode them. The gap between what the model can do and what it does zero-shot is large, and it is a gap that exists in the text artifact, not in the weights.

Weight adaptation is unavailable for closed frontier models and expensive for open ones. For deployments using API-accessed models (GPT, Claude, etc.), fine-tuning is either impossible or requires engineering effort that many practitioners cannot afford. For open-weight models, fine-tuning on each new domain incurs substantial computational cost and produces a new model artifact that must be versioned, served, and maintained. A compact skill document that can be optimized once, audited as text, and deployed with the frozen model avoids all of this infrastructure overhead while still achieving domain adaptation. The paper's finding that the optimized skill artifacts transfer positively across model scales (Table 4a), across execution harnesses (Table 4b), and to nearby benchmarks (Table 4c) makes this especially compelling: the cost of optimization is paid once and amortized across related deployment scenarios.

The adaptation layer for agents should be procedural, not parametric. As LLMs are increasingly deployed as agents with tools, files, and multi-step execution loops [1–4], domain adaptation requires improving the procedures by which the agent gathers evidence, calls tools, follows conventions, and formats outputs—not just changing what the model "knows." Agent skills provide a natural interface for this: a portable text artifact that packages procedural knowledge. But if skills are the adaptation layer, they need to be optimizable. The paper's central insight is that the recurring object of adaptation is the agent's procedure, and therefore the skill document should be trainable with the same discipline that applies to other trainable components. This reframes the problem: the skill is not a side artifact of prompting; it is the external state being optimized.

Self-improvement pipelines need reliable optimization, not just reflection. A growing body of work explores self-improving agents that learn from their own execution experience [9–13, 16, 17, 20–26]. These systems share a common pattern: execute tasks, reflect on successes and failures, and update some textual artifact (a skill, a prompt, a system design document). But without controlled optimization, this reflection loop is brittle—a single misleading trajectory can trigger a harmful rewrite, and there is no mechanism to prevent regression or average evidence across experiences. SkillOpt's contributions—bounded edit budgets, minibatch reflection, validation gating, rejected-edit buffers, epoch-wise consolidation—are exactly the controls that would make such self-improvement pipelines reliable. The paper doesn't just propose a better skill optimizer; it provides a template for turning any reflection loop into a controlled training process.

Where Existing Approaches Fall Short

The paper situates its contribution against several families of prior work, each of which addresses part of the skill optimization problem but lacks key control mechanisms.

Prompt auto-tuning and configuration search (TextGrad, GEPA). These methods treat prompts or system configurations as optimizable objects that can be refined from trajectory feedback [13, 40]. GEPA, for instance, demonstrates that reflective prompt evolution guided by Pareto-based selection can outperform reinforcement learning on language-agent tasks. But these methods target prompts—short instructional prefixes—rather than persistent, reusable skill documents that package domain procedures, tool policies, and failure modes. More critically, they lack the training-style controls that SkillOpt introduces: there is no bounded learning rate (the prompt can be rewritten arbitrarily at each step), no explicit validation gate (updates are applied based on training-trajectory scores rather than held-out performance), and no separation between fast and slow updates. The result is that these methods can improve prompts but cannot reliably produce a compact, transferable, validated skill artifact. The paper's evidence bears this out: on direct-chat GPT–5.5, TextGrad and GEPA achieve six-benchmark averages of 65.1 and 74.4 respectively, while SkillOpt reaches 82.3 (Table 1), and GEPA's gains on procedural benchmarks like SpreadsheetBench (+31.8) fall well short of SkillOpt's (+38.9).

Skill construction and evolution (Trace2Skill, EvoSkill, SkillForge, EvoSkills, AutoSkill). This family of work represents the most direct prior approach to the problem SkillOpt addresses. These systems construct skills from trajectory lessons, mine skill knowledge bases from heterogeneous resources, or evolve skill folders through failure analysis [9–12, 17–21]. Trace2Skill distills trajectory-local lessons into transferable skills. EvoSkill performs automated skill discovery through creation-evaluation-revision loops in multi-agent settings. SkillForge builds self-evolving skill libraries for cloud technical support. These works share the goal of producing reusable procedural artifacts, and they are closest to SkillOpt in spirit. But they lack the optimization discipline that SkillOpt introduces:

No bounded textual learning rate: Skills are rewritten without constraints on edit magnitude, so a single failure can overwrite useful rules. The paper's ablation shows that removing the learning rate drops performance (Table 3: without lr, SpreadsheetBench falls from 77.5 to 75.7).
No validation gate on separated data: Updates are accepted based on training-trajectory scores or qualitative analysis, not held-out performance. This is the most critical missing piece. Without a gate, plausible-sounding but harmful edits accumulate. The paper shows that the gate is strict—ties are rejected—and that this strictness is what makes the rejected-edit buffer useful.
No rejected-edit buffer: Failed updates are discarded, so the system can repeat harmful changes. SkillOpt records rejected edits and their score impact, then feeds them back as negative examples to later optimizer calls. Removing this buffer causes drops of 1.6–4.6 points across benchmarks (Table 3).
No epoch-wise consolidation: These systems update the skill continuously without aggregating cross-epoch patterns. SkillOpt's slow/meta update explicitly compares performance across epochs on the same tasks, identifying improvements, regressions, persistent failures, and stable successes. Removing this component causes the largest ablation drop: SpreadsheetBench falls from 77.5 to 55.0 (Table 3).

The paper's empirical comparison validates this analysis. EvoSkill—the strongest harness-side competitor—achieves 67.5 on SpreadsheetBench under Codex with GPT–5.5, a substantial +40.0 gain over no skill. But SkillOpt adds another +17.5 points on top of that (67.5 → 85.0). The gain is not from a better reflection model or more trajectories; it is from applying controlled optimization to the skill editing process itself.

One-shot skill generation (LLM-generated, human-written). A natural baseline is simply to ask a frontier model to generate a skill from a task description, or to have a human expert write one. Both approaches can work well when prior knowledge happens to match the benchmark. Human-written skills on GPT–5.5 lift SpreadsheetBench from 41.8 to 72.9 and OfficeQA from 33.1 to 66.9 (Table 1), and one-shot LLM skills achieve competitive results on DocVQA and ALFWorld. But these static skills have a key limitation: they cannot learn from execution experience. If the initial skill misses a procedural rule—e.g., that formula-style spreadsheet prompts require writing evaluated static values rather than formulas—no amount of execution will fix it without a feedback loop. SkillOpt's optimization loop discovers these missing rules from rollout evidence and validates them before deployment. The paper's results show that SkillOpt consistently outperforms both human and LLM skills, with the gap widest on the most procedural benchmarks: SpreadsheetBench +7.8 over human skill, OfficeQA +5.2, LiveMathematicianBench +28.5.

Iterative self-reflection (Reflexion, Self-Refine). These methods have the agent reflect on its own trajectory and revise its answer or plan [27, 28]. They are effective for single-task improvement but do not produce a persistent, transferable artifact. SkillOpt's output is not a revised answer for a specific question but an edited skill document that improves performance across all future tasks in the domain. The optimization loop operates on rollout batches, not single trajectories, and the learned rules generalize across the test set.

How SkillOpt Positions Itself

The paper draws an explicit analogy between text-space skill optimization and deep-learning weight optimization, and this analogy is operational, not merely decorative. The paper maps each component:

Parameters → the skill document (the external state being optimized)
Gradient direction → trajectory-derived edit direction (from minibatch failure/success analysis)
Learning rate → edit budget (how many atomic edits are allowed per step)
Validation check → held-out selection gate (only strictly improving candidates are accepted)
Batch/minibatch → rollout batch and reflection minibatch (controls evidence noise per update)
Schedule/gate → learning-rate schedule (cosine decay) and validation gate (ties rejected)
Momentum/stable training → epoch-wise slow/meta update (cross-epoch consolidation)

This mapping is not just a metaphor—it implies specific design choices that distinguish SkillOpt from prior work:

Bounded updates, not unbounded rewrites. The edit budget $L_t$ constrains how far the skill can move in one step. This preserves continuity: adjacent skill versions remain close enough that later optimizer calls can meaningfully learn from what changed, what helped, and what failed. Without this, "consecutive skill revisions move too far or in inconsistent directions, [so] rejected edits and previous accepted edits no longer provide a meaningful optimization history."
Evidence aggregation, not anecdotal fixes. The rollout batch provides multiple scored trajectories before any edit is proposed. The reflection minibatch further groups trajectories so the optimizer sees patterns rather than individual failures. The hierarchical merge (failure merge → success merge → final failure-prioritized merge) ensures that edits represent recurring evidence, not edge cases.
Validation gating, not unconditional self-editing. Every candidate skill is evaluated on a held-out selection split, and accepted only if it strictly improves performance. This is the paper's key departure from prior self-evolution methods, which accept edits based on training-trajectory analysis or qualitative assessment. The gate turns reflection into propose-and-test optimization, and its strictness (ties are rejected) ensures the deployed skill never silently drifts.
Negative feedback from rejected edits, not discarding failures. Rejected edits and observed failure patterns are stored in an epoch-local buffer and provided to future optimizer calls. This means the optimizer model can learn from what didn't work without the failed edits ever reaching the deployed skill.
Separation of fast and slow adaptation. Step-level edits learn from the current batch; epoch-wise slow/meta updates learn from cross-epoch comparisons of the same tasks under different skill versions. This prevents local edits from overwriting durable lessons, and the slow-update field is explicitly protected from step-level modifications.

The paper positions SkillOpt as a framework for training one compact domain skill with optimizer controls, explicitly narrowing the scope relative to systems that build large skill libraries or perform policy optimization. This is a deliberate choice: by focusing on optimizing a single skill artifact rather than growing a repository, the paper can isolate the effect of the optimization controls themselves. The paper acknowledges that extending to skill libraries and self-distillation back into model weights is natural future work (Section 5, "Outlook"), but the core contribution is demonstrating that the optimization discipline matters.

The relationship to the deep-learning analogy runs deeper than the design choices. The paper also frames the difficulty of text-space optimization in terms familiar from weight-space optimization. Just as training a neural network without validation leads to overfitting, training a skill without a held-out gate leads to overfitting to training trajectories. Just as large learning rates cause instability in SGD, large edit budgets cause skill revisions to oscillate or destroy useful content. Just as momentum smooths noisy gradients across batches, the slow/meta update smooths noisy edit directions across epochs. The paper's central claim is that these analogies are not coincidental—they reflect genuine structural similarities between weight-space and text-space optimization—and that respecting them is what makes SkillOpt work where prior approaches fail.

Finally, the paper emphasizes a practical deployment philosophy that distinguishes it from both weight-update methods and configuration-search methods: the optimizer model is used only during offline training. At deployment, the frozen target model receives only a compact, static best_skill.md file (300–2,000 tokens) with no additional inference-time model calls. This means the cost of optimization—training tokens, optimizer API calls, validation evaluations—is paid once and amortized across all future uses of the skill. The paper's cost analysis (Table 6) shows this cost ranges from 0.6M to 46.4M training tokens per absolute test-set point of gain, which is substantial but one-time. For applications where the skill is deployed repeatedly across many queries, this cost structure is favorable compared to per-query optimization approaches or fine-tuning.

In summary, SkillOpt positions itself as filling the gap between (a) the recognized need for trainable agent skills and (b) the absence of controlled optimization methods for text artifacts. Prior work either treats skills as static prompts, loosely revises them based on anecdotal evidence, or searches over them as black-box configuration parameters—none of which provides the stability, generalizability, and validation discipline that deep-learning training has shown is essential for reliable optimization. SkillOpt shows that by applying these controls to text-space skill editing, the resulting artifacts are not just better prompts but genuinely trainable procedural modules that improve performance, transfer across contexts, and remain compact and inspectable.

3. Technical Approach

3.1 Reader Orientation

What is being built: SkillOpt is a text-space optimizer that treats a natural-language skill document as the external trainable state for a frozen LLM agent, applying deep-learning-style training controls—bounded edit budgets, minibatch reflection over rollout evidence, held-out validation gating, rejected-edit buffers, and epoch-wise slow/meta updates—to turn skill improvement into a controlled optimization process rather than ad hoc prompt revision.

What problem it solves: When deploying a frozen LLM as an agent in a domain (e.g., spreadsheet manipulation, document QA, competition math), its behavior depends heavily on a skill document—a text artifact that packages procedures, tool-use policies, output formats, and failure-handling rules—but existing methods for creating and improving this artifact (hand-writing, one-shot generation, loosely controlled self-revision from feedback) are all ad hoc, cannot reliably improve over their starting point, and lack the stability and validation discipline of deep-learning weight-space optimization. SkillOpt solves this by making the skill document itself a trainable artifact that is edited by a separate optimizer model under explicit training-style controls, producing a compact, verified, reusable best_skill.md file without ever modifying the frozen agent's weights.

3.2 Big-Picture Architecture (Diagram in Words)

The SkillOpt system has five major components, arranged in a loop:

Frozen Target Model ( $M$ ) — the LLM agent whose behavior is being adapted. It executes tasks using the current skill document, produces trajectories and scalar scores, and is never updated. It is the "student" that the optimization serves.
Skill Document ( $s$ ) — a compact natural-language policy (markdown file) inserted into the agent's context before execution. It packages domain procedures, tool policies, output constraints, and failure-handling rules. This is the only state that changes during optimization.
Execution Harness ( $h$ ) — the environment in which the target model operates: direct single-turn chat, the Codex CLI agentic loop, or the Claude Code CLI loop. It runs the task, records trajectories (messages, tool calls, observations, outputs), and returns a scalar score $r \in [0,1]$ .
Optimizer Model ( $O$ ) — a separate frontier LLM (typically GPT–5.5) that reads scored trajectories, analyzes success and failure patterns in minibatches, proposes structured add/delete/replace edits, merges and ranks edits, and selects a bounded update. It runs only during offline training and is never invoked at deployment.
Validation Gate and Training-State Machinery — a held-out selection split $D_{\text{sel}}$ that evaluates every candidate skill before acceptance (only strictly improving candidates survive), a rejected-edit buffer that records failed proposals as negative feedback for future optimizer calls, and an epoch-wise slow/meta update mechanism that consolidates cross-epoch patterns.

Information flow (one optimization step): The current skill $s_{\text{cur}}$ is injected into the execution harness → the frozen target model $M$ runs a batch of tasks from the training split $D_{\text{tr}}$ , producing scored trajectories → the optimizer model $O$ groups trajectories into success/failure minibatches, analyzes each minibatch to propose structured edits, hierarchically merges the proposals (failure merge → success merge → final failure-prioritized merge), ranks the merged edits by importance, and clips to the current edit budget $L_t$ → the selected edits are applied to produce a candidate skill $\tilde{s}$ → the candidate is evaluated on the held-out selection split $D_{\text{sel}}$ ; if its score strictly exceeds the current selection score, it becomes the new current skill and, if it also beats the best-ever score, becomes best_skill.md → if rejected, the failed edits and observed failure patterns are stored in the rejected-edit buffer for future optimizer calls. Across epochs, the slow/meta update samples the same tasks under the previous and current epoch-end skills, compares performance, and writes longitudinal guidance into a protected slow-update field. The deployed output is a single best_skill.md file of roughly 300–2,000 tokens.

3.3 Roadmap for the Deep Dive

First, the formal problem setup (Section 3.1 of the paper): the mathematical definition of what SkillOpt optimizes—how the skill, model, harness, and score interact, what the train/selection/test splits mean, and what the optimizer's objective is. This grounds all subsequent design choices in a precise target.
Second, the forward pass: rollout evidence (Section 3.2): how the frozen target model generates scored trajectories from the training split using the current skill, why batch size and accumulation matter as evidence controls, and what metadata the harness records for the optimizer.
Third, the backward pass: minibatch reflection (Section 3.3): how the optimizer model converts trajectory batches into structured edit proposals, why minibatch partitioning—rather than single-trajectory analysis—exposes reusable procedural errors, and how success and failure analyses are hierarchically merged with failure-prioritized ranking.
Fourth, the bounded text update mechanism (Section 3.4): how the textual learning rate $L_t$ constrains edit magnitude, how the optimizer ranks proposed edits under this budget, the difference between patch mode and rewrite mode, and why schedules (constant, linear, cosine, autonomous) matter. This is the learning-rate analogue.
Fifth, the validation gate and rejected-edit buffer (Section 3.5): how the held-out selection split gates every candidate skill, why ties are rejected (strict improvement only), how rejected edits become structured negative feedback, and why this gate is essential for turning reflection into propose-and-test optimization.
Sixth, the epoch-wise slow/meta update (Section 3.7): how the optimizer compares the same tasks under consecutive epoch-end skills, identifies improvements, regressions, persistent failures, and stable successes, and writes longitudinal guidance into a protected skill field while maintaining a separate optimizer-side meta skill—the separation of fast and slow adaptation.
Seventh, the harness-agnostic deployment interface (Section 3.7): how the adapter pattern decouples the optimizer from any specific execution environment, enabling the same optimizer loop to work across direct chat, Codex, Claude Code, and embodied environments, and how this enables the cross-harness transfer experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical analysis paper whose core idea is that agent skill optimization should be treated as a controlled training loop—with explicit controls over evidence aggregation, step size, validation gating, negative feedback, and cross-epoch consolidation—operating on an external natural-language state (the skill document) with a frozen target model and a separate optimizer model.

Problem Setup: What Is Being Optimized and Under What Constraints

The paper formalizes skill optimization as a propose-and-test search over natural-language skill documents with a frozen task-execution model. The core definitions establish what the optimizer can control and what is held fixed.

The skill document. A skill $s$ is a natural-language policy (a markdown text file) that is inserted into the agent's context before execution. The paper notes this is "consistent with recent work treating skills as reusable procedural knowledge for agents." In direct-chat benchmarks, the skill is prepended to the system or developer instruction; in tool-use harnesses, it becomes persistent procedural memory. The skill is the only state variable that changes during optimization.

The execution function. For a given frozen target model $M$ , harness $h$ , task $x$ , and skill $s$ , execution produces two outputs:

$(\tau(s), r(s)) = h(M, x, s), \quad r(s) \in [0, 1]$

where $\tau(s)$ is the full trajectory (includes task metadata, messages, tool calls, observations, command outputs, final answers, verifier feedback, and benchmark-specific context such as spreadsheet previews, document references, or compact execution traces), and $r(s)$ is a scalar score between 0 and 1 representing task success (typically the benchmark's native hard score or exact-match accuracy). The notation $\tau(s)$ and $r(s)$ emphasizes that both outputs depend on the skill—the same model and task can produce different trajectories and scores under different skills.

What it computes: given a skill $s$ , the frozen model $M$ processes task $x$ through harness $h$ to produce a complete execution record $\tau(s)$ and a binary or continuous success score $r(s)$ . This is a black-box function: the optimizer cannot see inside $M$ or differentiate through it; it can only observe the input (skill + task), the trajectory output, and the scalar score.

Why this form: the formulation enforces a strict separation of concerns. $M$ and $h$ are fixed throughout optimization—the optimizer never modifies model weights, prompt templates, evaluators, or harness logic. The only controllable variable is $s$ . This is not just a design choice; it is the paper's central claim: that a single compact text artifact, when properly optimized, can serve as the adaptation layer for frozen agents without touching weights. The equation also sets up the score function as the optimization signal—the optimizer's goal is to find $s$ that maximizes $r(s)$ across the task distribution.

The data splits and optimization objective. SkillOpt uses three disjoint data splits, consistent with standard supervised learning practice:

$s^\star_{\text{sel}} = \arg\max_{s \in C(D_{\text{tr}})} \frac{1}{|D_{\text{sel}}|} \sum_{x \in D_{\text{sel}}} r(s)$

$\text{Test}(s^\star_{\text{sel}}) = \frac{1}{|D_{\text{test}}|} \sum_{x \in D_{\text{test}}} r(s^\star_{\text{sel}})$

where $C(D_{\text{tr}})$ is the set of candidate skills produced by the optimizer using the training split $D_{\text{tr}}$ as evidence, $D_{\text{sel}}$ is the held-out selection split used to gate updates and select the best skill, and $D_{\text{test}}$ is the locked test split used only for final reporting.

What this computes: the optimizer generates a set of candidate skills $C(D_{\text{tr}})$ by repeatedly proposing edits from training trajectories. The selection split $D_{\text{sel}}$ evaluates every candidate and picks the one with the highest average score—this is the final optimized skill $s^\star_{\text{sel}}$ . The test split $D_{\text{test}}$ measures the selected skill's generalization performance exactly once, at the end, and is never used during optimization.

Why this form: the three-way split is the standard machine learning protocol for preventing overfitting, but it is particularly important in text-space optimization because plausible-sounding textual edits can easily overfit to the training examples without improving generalization. The selection split acts as an explicit overfitting guard: the optimizer can propose as many candidates as it wants from training evidence, but only candidates that genuinely improve held-out performance survive. This is a critical departure from prior skill-evolution methods (Trace2Skill, EvoSkill) that implicitly use training performance for selection. The paper makes the gate strict—"ties are rejected"—to prevent silent drift, where a skill version that is equally good but differently worded could gradually accumulate changes that later backfire.

Optimizer state. The optimizer tracks five state variables: $s_{\text{cur}}$ (the current skill being edited), $s_{\text{best}}$ (the best validation-gated skill seen so far), a cache $C$ mapping skill hashes to their selection-split scores (to avoid redundant evaluation), an epoch-local rejected-step buffer $B$ (storing rejected edits and observed failure patterns), and optional slow/meta-update state (for epoch-wise consolidation). Only $s_{\text{best}}$ —exported as best_skill.md—is deployed; all other state is training-internal.

Forward Pass: Rollout Evidence (How Scored Trajectories Are Collected)

The forward pass is the data collection stage that provides the evidence for skill editing. It is analogous to computing the forward pass and loss in weight-space training: the frozen model processes tasks with the current skill, producing scored trajectories that will be analyzed to determine what to change.

Rollout batch as the evidence unit. At each optimization step, the target model runs a rollout batch—a set of tasks sampled from the training split $D_{\text{tr}}$ —with the current skill $s_{\text{cur}}$ . For each task $x$ in the batch, the harness records:

Task metadata (the question, problem instance, or goal specification)
The complete message history (system prompt with skill prepended, user input, assistant responses)
Tool calls and their results (e.g., openpyxl function calls for spreadsheets, grep or find for Codex/Claude Code)
Observations from the environment (spreadsheet previews, document text, command outputs)
The final answer or output and the benchmark's verifier feedback
Benchmark-specific context (e.g., spreadsheet cell ranges and formulas, document page images, step-by-step embodied-state transitions)

Why batch evidence matters. The paper explicitly contrasts batch-based evidence with single-trajectory analysis: "small batches update quickly but noisily, while larger batches expose more recurring patterns before the skill changes." This is the same rationale as minibatch SGD: a single trajectory may contain an idiosyncratic failure (e.g., the model happened to misread one cell in one spreadsheet), and editing the skill based on that isolated failure risks overfitting or introducing a rule that hurts other cases. A batch of trajectories reveals which failures are systematic (e.g., the model consistently writes formulas instead of evaluated values across many spreadsheet tasks) and which successes are stable patterns worth preserving.

Accumulation for throughput. The implementation supports an accumulation factor $A$ , where several rollout batches are collected and reflected on separately, then merged into a single update. This decouples execution throughput (how many tasks can be run in parallel) from update frequency (how often the skill changes). If $A=4$ and the rollout batch size is 40, the optimizer sees 160 trajectories before proposing edits for one update, but the reflection happens in four groups of 40 to keep the minibatch analysis tractable. The default setting in the experiments is $A=1$ (no accumulation), with rollout batch size 40 and reflection minibatch size 8.

Default hyperparameters (Section 4, default optimizer hyperparameters block). The paper states: "SkillOpt uses four epochs, rollout batch size 40 per step, reflection minibatch size 8 (with 16 analyst workers running reflections in parallel and a merge batch size of 8), textual learning rate $L_t=4$ with cosine decay (floor $L_t=2$ )." For benchmarks with tightly bounded training pools, per-benchmark configs scale batch sizes accordingly: "LiveMathematicianBench: 35 training items per epoch with rollout batch 200; ALFWorld: 39 training tasks." The rollout batch size thus controls how many scored trajectories are gathered before the optimizer is called to reflect.

Backward Pass: Minibatch Reflection (How Trajectories Are Converted to Edit Proposals)

The backward pass is the analysis and proposal stage where the optimizer model reads rollout evidence and proposes structured skill edits. It is the analogue of computing gradients in weight-space training: it determines the direction and magnitude of the update based on evidence.

Separation into successes and failures. The optimizer model first splits each rollout batch's trajectories into two groups: those where $r(s) > 0$ (successes) and those where $r(s) = 0$ (failures). This separation is critical because the two groups serve different purposes:

Failure analysis identifies what the skill is missing or getting wrong. Failure trajectories are grouped into reflection minibatches (default size 8), and the optimizer model is asked to identify common failure patterns across the minibatch, not individual mistakes. The paper's analyst prompt (Appendix C.2.1, analyst_error.md) explicitly instructs: "Identify the most important COMMON failure patterns across the batch... propose skill edits that address the COMMON patterns, not individual edge cases."
Success analysis identifies behaviors that already work and should be preserved. Success trajectories are similarly grouped into minibatches, and the optimizer model is asked to identify generalizable behavior patterns NOT already covered in the skill. The prompt (Appendix C.2.2, analyst_success.md) states: "Only propose patches for patterns NOT already covered in the skill... Prefer reinforcing existing sections over adding new top-level sections."

Why minibatch partitioning matters. The paper explains: "single trajectories often produce anecdotal fixes, while minibatches expose reusable procedural errors: the agent consistently searches the wrong source, writes an answer in the wrong format, or fails to verify a tool result." By forcing the optimizer to analyze multiple trajectories simultaneously, the minibatch analysis surfaces systematic patterns rather than one-off mistakes. A failure that appears in only one of eight trajectories is likely noise; a failure that appears in six of eight is a systematic gap in the skill. The minibatch size $B_m=8$ (the default) is chosen to be large enough to amortize noise but small enough that the optimizer model's context window can comfortably hold all trajectories.

Trajectory content for the optimizer. The optimizer model receives, for each trajectory in the minibatch, a compact representation that includes: the task description, the skill that was active during execution, the agent's reasoning steps (if visible), tool calls and their results, the final answer, the verifier's judgment (correct/incorrect and score), and the ground-truth answer where available for failure analysis. The paper notes that for harnessed settings (Codex, Claude Code), the optimizer receives a "compact execution trace (codex_trace_summary.txt) that is included in the teacher reflection context, so the optimizer learns from what the agent actually did, not just its final answer."

Structured edit representation. The optimizer model proposes edits in a patch format with four atomic operations, all expressed as JSON within the prompt contract (Appendix C.3):

append: add new content at the end of the skill document
insert_after: insert new content after a specified heading or text string in the existing skill
replace: replace a specified text substring with new content
delete: remove a specified text substring

Each proposed edit carries a support_count (how many source patches independently proposed similar edits) and a source_type ("failure" or "success"). The structured format is essential: it allows the optimizer to propose precise, localized changes rather than full-skill rewrites, and it enables the merge and ranking stages to operate on a machine-parseable edit list rather than free-text suggestions.

Hierarchical merge pipeline. The raw edit proposals from multiple minibatches are consolidated through a three-stage merge process:

Failure merge (Appendix C.2.3, merge_failure.md): consolidates all failure-driven edit proposals into a single non-redundant patch. The merge prompt instructs: "Deduplicate: keep the best-worded version of similar edits. Resolve conflicts: if patches contradict on the same point, choose the one with stronger justification or synthesize both. Prevalent-pattern bias: edits appearing consistently across multiple patches address systematic failures; preserve them with HIGH priority."
Success merge (Appendix C.2.4, merge_success.md): consolidates all success-driven edit proposals. The prompt includes: "Be conservative: success-driven patches reinforce existing behavior. Only include edits for patterns NOT already in the skill."
Final failure-prioritized merge (Appendix C.2.5, merge_final.md): combines the pre-merged failure and success patches. The prompt states: "FAILURE PATCHES TAKE PRIORITY: the primary goal of skill reflection is to fix failures. Failure-driven edits should be preserved unless they directly conflict with a well-supported success pattern."

Why hierarchical merging matters. The merge pipeline serves three functions beyond simple deduplication. First, it allows the optimizer to scale to many more trajectory minibatches than can fit in a single context window: each minibatch is analyzed independently (with up to 16 parallel analyst workers), and only the consolidated edit proposals are merged hierarchically. Second, the priority ordering (failure over success) encodes a design choice: the primary goal of skill optimization is to fix what's broken, not to over-specify what already works. Third, the support_count tracking provides a quantitative signal for the ranking stage: edits proposed independently by multiple analysts from different trajectory minibatches are more likely to represent genuine systematic patterns.

The optimizer is the teacher, the target model is the student. The paper explicitly refers to the optimizer model as the "teacher" and the frozen target model as the "student" in multiple places (e.g., "Both teacher and student calls default to a medium reasoning effort"). This framing is important: the optimizer model is typically a more capable frontier model (GPT–5.5 in the default experiments) than the target model it optimizes for, but the two are independent. The teacher proposes edits; the student executes tasks under the resulting skill. The teacher never sees the student's weights, and the student never sees the teacher's reasoning—the only communication channel is the skill document.

Bounded Text Updates: The Textual Learning Rate

The bounded text update is the learning-rate analogue in SkillOpt. Just as a learning rate $\eta$ in SGD controls how far the parameters can move in a single update, the edit budget $L_t$ controls how many atomic edits can be applied to the skill in a single optimization step. The paper treats this as the central control mechanism that distinguishes SkillOpt from ad hoc prompt rewriting.

The edit budget $L_t$ . At each optimization step $t$ , the optimizer is allowed to apply at most $L_t$ edits to the skill. After the hierarchical merge stage produces a pool of ranked candidate edits, these edits are clipped to the top $L_t$ edits by the optimizer's ranking judgment. The ranking prompt (Appendix C.2.6, ranking.md) provides explicit criteria:

Systematic impact (highest priority): edits addressing widespread, recurring failure patterns across many tasks
Complementarity: edits filling gaps in the current skill, not duplicating existing content
Generality: edits phrased as general principles rather than tied to specific question types
Actionability: edits with clear, concrete guidance rather than vague advice

What this computes operationally: given a ranked list of $E$ merged edit proposals, the optimizer selects the top $\min(E, L_t)$ edits. These edits are applied to the current skill $s_{\text{cur}}$ in the specified order (appends after inserts after replacements after deletions, with conflict resolution preferring edit operations to occur in order of proposal priority). The result is a candidate skill $\tilde{s}$ that differs from $s_{\text{cur}}$ by at most $L_t$ discrete changes.

Why a bounded budget matters. The paper argues this is "the key difference from ad hoc prompt rewriting." Without a budget, the optimizer could produce a complete rewrite of the skill in response to a single batch of trajectories. This creates three problems:

Useful content can be erased. If the skill contains rules that worked for 95% of tasks but failed on 5%, an unbounded rewrite could remove those successful rules to fix the 5%, causing a net regression.
Edit history becomes meaningless. If consecutive skill versions move too far apart, the rejected-edit buffer and previous accepted edits no longer provide a meaningful optimization history—the optimizer cannot learn from what changed because everything changed.
Instability and oscillation. Large jumps can cause the skill to oscillate between different regimes (e.g., one step adds a rule, the next step removes it, the next re-adds it in different wording), preventing convergence.

The paper's ablation (Table 3, "without lr" row) validates this: removing the learning rate (allowing unbounded edits) drops SpreadsheetBench from 77.5 to 75.7 and SearchQA from 87.1 to 84.6, confirming that the budget constraint matters even when the optimizer is otherwise identical.

Learning-rate schedules. SkillOpt supports four schedules for $L_t$ :

Constant: $L_t = L$ for all steps. The default $L=4$ .
Linear decay: $L_t$ decreases linearly from an initial value to a floor.
Cosine decay: $L_t$ follows a cosine curve from an initial value to a floor. This is the default schedule, with $L_0=4$ and floor $L_t = 2$ . The intuition is that early optimization steps benefit from larger edits to address major systemic gaps, while later steps should make small refinements to avoid destabilizing a mostly-correct skill.
Autonomous: the optimizer model itself decides the budget based on the current state of the skill and the severity of observed failures.

Table 2(e) shows that the constant schedule scores 87.3/80.7/62.1 on SearchQA/SpreadsheetBench/LiveMath, cosine scores 87.1/77.5/61.3, and linear scores 87.2/72.9/62.9. The constant schedule performs best on SpreadsheetBench (80.7), but all three schedules outperform baselines, indicating that the bounded-update principle matters more than the specific schedule.

Patch mode vs. rewrite mode. In patch mode (the default), edits are localized operations (append, insert_after, replace, delete) that modify specific parts of the skill while leaving the rest intact. In rewrite mode, the optimizer instead receives the ranked edit suggestions and uses them to condition a full skill rewrite, producing a new skill document from scratch that incorporates the suggested changes. Patch mode is preferred because it preserves continuity: the optimizer can trace which sections changed, and the rejected-edit buffer can identify which specific edits caused problems. Step-level edits also cannot overwrite the protected slow-update field (described below), separating fast local changes from slower epoch-wise consolidation.

The protected slow-update field. The skill document may contain a section delimited by  and  markers. This section is managed exclusively by the epoch-wise slow update process; all step-level reflection prompts (analyst_error, analyst_success, merge_failure, merge_success, merge_final) are explicitly instructed: "Do NOT propose any edits that target, modify, or delete content within these markers." This separation means that fast step-level edits cannot accidentally delete or overwrite durable cross-epoch lessons.

Validation Gate and Rejected-Edit Buffer: Gating Updates and Learning from Failures

The validation gate is the held-out evaluation checkpoint that prevents harmful edits from accumulating. It is the analogue of monitoring validation loss during weight-space training: the optimizer proposes edits based on training evidence, but the edits are only accepted if they improve performance on unseen data.

Gate operation. For each candidate skill $\tilde{s}$ proposed by the optimizer:

Compute the hash of $\tilde{s}$ and check the cache $C$ . If the hash exists in the cache (the skill has been evaluated before), use the cached score $\text{score}_{\text{cand}} = C[\text{Hash}(\tilde{s})]$ . This avoids re-evaluating skill versions that the optimizer has already tested, saving computation.
If the hash is new, evaluate $\tilde{s}$ on the selection split $D_{\text{sel}}$ : run the frozen target model $M$ with harness $h$ on all tasks in $D_{\text{sel}}$ under skill $\tilde{s}$ , compute the average score, and cache it.
Compare $\text{score}_{\text{cand}}$ to $\text{score}_{\text{cur}}$ (the selection-split score of the current skill $s_{\text{cur}}$ ). If $\text{score}_{\text{cand}} > \text{score}_{\text{cur}}$ (strictly greater), accept $\tilde{s}$ as the new current skill. If $\text{score}_{\text{cand}} > \text{score}_{\text{best}}$ (strictly greater than the best-ever score), update $s_{\text{best}}$ as well.
If $\text{score}_{\text{cand}} \leq \text{score}_{\text{cur}}$ , reject $\tilde{s}$ . The current skill remains unchanged.

Why the gate is strict about ties. The paper explicitly states: "the validation gate is intentionally strict: a candidate skill is accepted only when its selection-split score is strictly greater than the current selection score, so ties are rejected and the deployed skill never silently drifts." This design choice prevents a subtle failure mode: if ties were accepted, the skill could drift through a sequence of equally-good but differently-worded versions. Each version might introduce subtle changes that don't immediately hurt performance but accumulate over time. For example, a rule could be rewritten in a slightly different style, lose a key qualifier ("always check the header row" → "check the header row"), and still achieve the same selection-split score. But later edits might build on the weakened version, eventually causing a regression. The strict gate ensures that every accepted edit makes the skill strictly better on held-out data.

Why the gate is essential for the optimizer's framing. Without a held-out gate, the optimizer can fall into a plausibility trap: it proposes edits that sound reasonable based on training trajectories (e.g., "use XLOOKUP instead of INDEX/MATCH for all lookups"), but the edit actually hurts generalization because it introduces edge cases the optimizer didn't anticipate. In weight-space training, we accept that some gradient steps will increase validation loss—the optimizer corrects for this over multiple steps. But in text-space optimization, a single harmful edit can introduce a rule that persists and is built upon by future edits, causing compounding degradation. The gate prevents this by requiring every edit to prove itself on held-out data before it can become the foundation for future edits.

The rejected-edit buffer: turning failures into negative feedback. When a candidate skill is rejected, the optimizer does not simply discard the information. Instead, it records in an epoch-local buffer $B$ :

The specific edits that were proposed and caused the rejection
The score drop they produced ( $\text{score}_{\text{cand}} - \text{score}_{\text{cur}}$ )
The observed failure patterns from the training batch that motivated those edits

Later reflection calls within the same epoch receive this buffer as additional context. The optimizer model is instructed to avoid repeating edits that were previously tried and rejected, and to focus on unresolved failures rather than re-proposing solutions that already failed validation. The buffer is epoch-local (it resets at epoch boundaries) because edits that were harmful in an earlier epoch might become useful later after the skill has incorporated other changes—the buffer prevents immediate repetition without permanently banning edit directions.

Why the rejected-edit buffer matters. Table 3 quantifies its effect: removing the buffer (while keeping all other components fixed) drops SearchQA from 87.1 to 85.5 (−1.6 points), SpreadsheetBench from 77.5 to 72.9 (−4.6 points), and LiveMath from 61.3 to 58.9 (−2.4 points). The SpreadsheetBench drop is particularly significant: it suggests that on procedural benchmarks where many plausible-sounding but harmful edits are possible (e.g., "always use direct cell references" sounds reasonable but fails on dynamic-range problems), the buffer prevents the optimizer from cycling through variations of the same bad idea.

The cache $C$ for computational efficiency. The skill-hash cache avoids re-evaluating skill versions that the optimizer has already tested. This is important because the selection-split evaluation can be expensive: running 500+ tasks to evaluate one candidate skill. If the optimizer proposes a skill that is identical (by character-level hash) to the current skill or to a previously rejected skill, the cached score is used without re-executing. The paper notes that this "prevents redundant evaluation" and is a practical necessity for running the loop within reasonable compute budgets.

Epoch-Wise Slow/Meta Update: Separating Fast and Slow Adaptation

The epoch-wise slow/meta update is the momentum and cross-epoch consolidation mechanism in SkillOpt. While step-level edits learn from the current rollout batch (fast, local adaptation), the slow/meta update learns from across-epoch comparisons (slow, global consolidation). This separation prevents local edits from overwriting durable lessons and provides the optimizer with a longer-horizon view of what is improving, what is regressing, and what remains stuck.

When it runs. The slow/meta update runs at the end of each epoch (after all step-level updates in that epoch have completed). It requires at least two epochs to have completed (epoch $e \geq 2$ ), since it compares the previous epoch's end-of-epoch skill with the current epoch's end-of-epoch skill.

How it works: longitudinal comparison. The optimizer samples a fixed number of tasks (default: 20) from the training split. It executes each sampled task twice: once under the previous epoch's skill and once under the current epoch's skill. This paired evaluation produces four categories:

Improvements: tasks that failed under the old skill but succeed under the new skill
Regressions: tasks that succeeded under the old skill but fail under the new skill
Persistent failures: tasks that failed under both skills
Stable successes: tasks that succeeded under both skills

The optimizer model receives a longitudinal comparison report showing the task IDs and outcomes in each category, along with the previous slow-update guidance (if any) and both skill versions.

The slow update: protected skill-field guidance. The optimizer model writes a longitudinal guidance block into the protected slow-update field of the skill document (between the  and  markers). This guidance is written as direct, actionable instructions to the target model (the student), for example: "When you encounter X, always do Y." The slow-update prompt (Appendix C.2.7, slow_update.md) specifies:

Retain and strengthen parts of previous slow-update guidance that proved effective
Revise or remove parts that were ineffective or counterproductive
Add new instructions to address regressions and persistent failures
Prioritize: (1) preventing regressions, (2) fixing persistent failures, (3) reinforcing successful patterns
Do NOT duplicate content already in the main skill body; complement it

Validation of the slow update. The updated slow-update guidance is treated as a candidate change to the skill and is still passed through the validation gate:

$\text{If } \text{score}(s_{\text{cur}} + \text{slow\_update}) > \text{score}(s_{\text{cur}}), \text{ accept; else reject.}$

This means the slow update cannot silently introduce harmful guidance—it must strictly improve held-out performance to be included in the deployed skill.

Why the slow update is protected from step-level edits. The paper explains: "Step-level edits cannot overwrite the protected slow-update field, so fast local changes and slower epoch-wise consolidation remain separated." This separation prevents a common failure mode: a step-level edit motivated by a single batch's failures could delete a slow-update rule that was derived from across-epoch evidence. For example, a slow update might add: "Always verify the number of rows after a filter operation." A later step-level edit based on a batch where verification caused a timeout might delete this rule. By protecting the slow-update field, SkillOpt ensures that only the epoch-boundary process—which has the cross-epoch view—can modify long-horizon guidance.

The meta skill: optimizer-side memory (teacher-only). In addition to the slow update (which is student-facing, written into the skill document), SkillOpt maintains a separate optimizer-side meta skill $m_{\text{meta}}$ . This is a compact record of what editing strategies have helped or hurt during the optimization process itself. It is generated at epoch boundaries using the same longitudinal comparison, but its audience is the optimizer model, not the target model. The meta-skill prompt (Appendix C.2.8, meta_skill.md) specifies:

"Which kinds of edits tend to help in this environment"
"Which kinds of edits tend to be too vague, redundant, brittle, or harmful"
"What level of abstraction works best for rules here"
"What failure-repair patterns should be prioritized"
"What regression risks future optimizer calls should guard against"

The meta skill is prepended to future optimizer calls (for reflection, merging, ranking) but is never shipped with the target model. This separation of concerns is important: the deployed skill remains compact and portable (only the student-facing rules), while training benefits from a richer record of the editing process that guides the optimizer's future behavior.

Why the slow/meta update matters. Table 3 shows the ablation impact. Removing both meta skill and slow update drops SpreadsheetBench from 77.5 to 55.0 (−22.5 points), the largest degradation in the ablation suite. This is the clearest evidence that the slow/meta update provides something that step-level edits cannot: it catches regressions that individual batches cannot see (because any single batch only sees a subset of tasks, regressions on tasks not in the current batch are invisible to step-level analysis) and provides the optimizer with a record of what editing strategies have historically worked in this domain. Without it, the optimizer is effectively myopic—it reacts to each batch's failures without retaining any sense of what it has already tried and what has actually helped across the full training distribution.

The analogy with momentum in SGD. In weight-space training, momentum accumulates a running average of past gradients, smoothing noisy updates and carrying stable gradient directions across steps. The slow/meta update serves a similar role: the slow update accumulates cross-epoch evidence about what consistently helps (analogous to a low-frequency gradient signal), while the meta skill records what editing approaches are productive (analogous to optimizer hyperparameter tuning). Together, they provide the long-horizon stability that step-level batch updates lack.

Harness-Agnostic Deployment: The Adapter Interface

SkillOpt is designed to work across different execution environments—direct single-turn chat, multi-turn tool-use loops, and persistent embodied interaction—through a lightweight adapter pattern. This is not just an engineering convenience; it is what enables the cross-harness transfer experiments in Section 4.3.

The adapter interface. An adapter is a harness-specific module that implements three responsibilities:

Construct train/evaluation batches from the benchmark's native data format into a uniform task representation
Inject the current skill into the agent context in a harness-appropriate way (system prompt for direct chat, persistent procedural memory for tool-use harnesses, a per-task SKILL.md file for Codex/Claude Code)
Run the native harness to execute the task, capture the trajectory (messages, tool calls, observations, outputs), and compute the scalar score $r \in [0,1]$

How skills are injected differently across harnesses. The paper describes three injection modes:

Direct chat: The skill is prepended to the system or developer instruction. The target model receives a single chat completion call with the skill in the system message and the task in the user message.
Codex harness: SkillOpt renders the current skill to a per-task SKILL.md file alongside task files. The Codex CLI executes the task in a workspace-write sandbox, and SkillOpt reads back a compact execution trace (codex_trace_summary.txt) that captures what the agent actually did (code execution, file operations, tool calls), not just its final answer.
Claude Code harness: Mirrors the same workspace contract through the claude CLI, with the skill rendered as persistent procedural memory visible to the agent throughout its execution loop.

Both Codex and Claude Code modes use the same best_skill.md file format—the adapter handles the injection details—which is what enables the cross-harness transfer experiments where a skill trained inside Codex is evaluated inside Claude Code without modification.

Why harness agnosticism matters for the paper's claims. The harness-agnostic design supports three of the paper's central claims:

The optimizer is independent of the execution environment. The optimizer model never interacts with the harness directly; it only sees the trajectory traces and scores that the adapter produces. This means the same optimizer code, with the same hyperparameters, can optimize skills for spreadsheets, document QA, math, and embodied tasks without modification.
The optimized skill artifact is portable across harnesses. Because the skill is just a markdown file—not a harness-specific configuration—it can be deployed in a different execution environment than the one it was trained in. Table 4(b) validates this: a SpreadsheetBench skill trained inside Codex transfers to Claude Code with a +59.7 point gain.
The optimization loop costs are amortized. A skill trained once can be used with any harness that has an adapter, so the training cost (20M–213M tokens, per Table 6) is paid once and the best_skill.md is reused across deployment environments.

The deployed artifact. The final output of SkillOpt is a single file: best_skill.md. It contains only the skill content—no optimizer prompts, no training metadata, no rejected-edit history. The file is typically 300–2,000 tokens (Table 6) and can be read, audited, and edited by a human practitioner. Deploying it requires no optimizer calls, no weight updates, and no harness changes beyond prepending the skill to the agent's context through the existing adapter. This is the paper's key deployment advantage: "the optimizer model is used only during offline training, adding zero inference-time model calls at deployment."

4. Key Insights and Innovations

Innovation 1: Reframing Agent Skills from Static Prompts to Trainable External State

The most fundamental intellectual move in this paper is not any specific mechanism but the reframing of what an agent skill is. Prior work across the agent-skill ecosystem—from SkillsBench's benchmarking [7] and the SoK on agentic skills [8] to construction systems like Trace2Skill [9], SkillForge [11], and SkillFoundry [12] to evolution systems like EvoSkill [10] and EvoSkills [20]—all treated skills as artifacts to be constructed or discovered, not as state to be optimized. A skill was something you wrote (by hand), generated (by one-shot LLM call), distilled (from trajectories), or evolved (through loosely controlled revision loops). The optimization question—how should a skill be improved, under what controls, with what guarantees—was either unasked or answered only implicitly through ad hoc rewriting.

SkillOpt's reframing is to say: the skill document is the external state of a frozen agent, and it should be trainable with the same discipline that makes weight-space optimization reliable. This is not a metaphor. The paper operationalizes the mapping directly: the skill is the parameter vector; trajectory-derived edits are the gradient direction; the edit budget is the learning rate; the held-out selection split is the validation set; the rejected-edit buffer is negative gradient feedback; and the epoch-wise slow/meta update is momentum. Each of these is implemented as an explicit mechanism (described in Section 3), but the conceptual contribution is the mapping itself—the claim that text-space skill optimization has genuine structural similarities to weight-space optimization, and that honoring those similarities with training-style controls produces reliability gains that ad hoc rewriting cannot match.

What makes this distinct from prior work. Prompt optimization methods like TextGrad [40] and GEPA [13] already treat prompts as optimizable objects and use trajectory feedback to guide updates. But they do not conceptualize the prompt as a trainable state with an optimizer state machine. They have no analogue of learning rates, no validation gating on separate data, no rejected-update buffer, and no separation of fast and slow adaptation. The prompt is optimized, but not trained in the deep-learning sense. Skill evolution methods like EvoSkill and Trace2Skill go further—they maintain skill artifacts across iterations and reflect on failures—but they lack every control mechanism that SkillOpt introduces: bounded edits, validation gates, negative feedback from rejections, and epoch-wise consolidation. These systems perform evolution, not optimization: they revise skills and hope the revision helps, without any of the stabilizing infrastructure that makes gradient-based training converge.

The significance is in reliability, not just performance. The paper's headline results (52/52 cells best or tied-best, +23.5 average gain on GPT–5.5) are impressive, but the intellectual contribution would matter even if the gains were smaller. What SkillOpt demonstrates is that when you treat a skill as trainable state—with bounded step sizes, held-out validation, and negative feedback—the optimization process becomes reproducible in a way that prior skill-evolution methods are not. The paper's ablation (Table 3) makes this explicit: removing the learning rate, the rejected-edit buffer, or the slow/meta update each causes measurable degradation. These mechanisms are not decorations; they are load-bearing components of a controlled training loop, and their absence from prior work explains why those systems could not reliably improve over their starting point.

The framing also changes what counts as a valid skill improvement. In the pre-SkillOpt framing, a skill is "better" if it was produced by a more sophisticated process (better reflection, more trajectories, more capable generator). In the SkillOpt framing, a skill is better only if it demonstrably improves held-out performance after passing a validation gate. This is a subtle but profound shift: it moves evaluation from process quality to outcome quality, and it forces every edit to prove itself empirically. The paper's finding that only 1–4 edits survive the gate across six benchmarks (Table 6)—out of potentially dozens proposed—is evidence that the gate is doing real work: most edits that sound good based on training trajectories fail to improve generalization. Prior work without gates would have accepted those edits, and likely degraded performance.

Why this is fundamental, not incremental. The reframing of skills as optimizable external state is a category shift, not a refinement. It opens an entire design space: once you accept that the skill is a trainable state variable, you can ask questions that were incoherent under the old framing. What is the right learning-rate schedule for skill edits? Should validation gating be strict (ties rejected) or lenient? Should the optimizer have its own memory separate from the deployed artifact? Should there be separate fast and slow adaptation channels? These questions have no analogue in prior work because prior work didn't conceptualize skill improvement as optimization. SkillOpt doesn't just answer these questions—it shows that asking them at all leads to better skills, and it provides a framework within which future work can explore their answers.

Innovation 2: The Validation Gate as the Critical Divide Between Self-Reflection and Controlled Optimization

Many prior systems—Reflexion [27], Self-Refine [28], Trace2Skill [9], EvoSkill [10]—share a core loop: execute tasks, reflect on failures, and update some textual artifact based on that reflection. The prevailing assumption across these systems is that reflection quality is the bottleneck: if the reflection is detailed enough and clever enough, the resulting update will improve performance. SkillOpt challenges this assumption directly by introducing a held-out validation gate that tests every candidate edit before accepting it, and by showing that many plausible-sounding edits fail this test.

The conceptual move is separating proposal from acceptance. In prior self-evolution systems, the reflection step proposes an update, and that update is adopted unconditionally. In SkillOpt, the reflection step proposes an update, and the validation gate accepts or rejects it based on held-out performance. This separation transforms the optimization process from self-editing (where the system edits itself based on its own analysis) to propose-and-test optimization (where a separate evaluation mechanism validates proposals before they become state). This is the same conceptual move that separates training loss from validation loss in supervised learning: training loss tells you what to optimize, but validation loss tells you whether you're actually improving.

The evidence that this distinction matters is the strictness of the gate. The paper's gate is intentionally conservative: "ties are rejected and the deployed skill never silently drifts." This strictness creates a regime where most proposals are rejected—only 1–4 edits survive per benchmark (Table 6)—and the rejected-edit buffer becomes a valuable source of negative feedback. A lenient gate (accepting ties or small improvements) would produce a different optimization trajectory: the skill would drift, accumulate subtle regressions, and the optimizer would lose the ability to learn from rejections because rejections would be rare.

The rejected-edit buffer as negative feedback is a second conceptual contribution within this innovation. In weight-space optimization, a gradient step that increases validation loss is not wasted—it provides information about what direction not to move in, and the optimizer implicitly corrects for it in subsequent steps through momentum and adaptive learning rates. In text-space optimization without a rejected-edit buffer, a rejected edit is wasted: the optimizer proposed something that didn't work, learned nothing from the failure, and may propose the same edit again later. The buffer converts this waste into signal: future optimizer calls are explicitly told "these edits were tried and caused a score drop of X," so the optimizer can avoid repeating mistakes. The ablation (Table 3: removing the buffer drops SpreadsheetBench by 4.6 points) confirms that this mechanism prevents the optimizer from cycling through variations of the same bad idea—a failure mode that prior work without buffers would be vulnerable to.

Why this changes the field's understanding of what makes self-improvement work. The prevailing narrative in the agent self-improvement literature is that better reflection (more capable models, more detailed analysis, more examples) leads to better updates. SkillOpt's results suggest a different narrative: reflection quality is necessary but not sufficient; control mechanisms (validation gating, negative feedback) are what determine whether reflection translates into improvement or into drift. This is an important corrective because it redirects research investment: improving the optimizer's reflection capability has diminishing returns if the optimization loop lacks gates and buffers, while adding gates and buffers to an existing reflection loop may yield immediate gains. The paper's comparison with EvoSkill—which uses sophisticated failure analysis but lacks a validation gate—illustrates this: EvoSkill achieves strong results (67.5 on SpreadsheetBench under Codex), but SkillOpt adds another 17.5 points on top, and the additional gains come from gating and buffering, not from better reflection.

This is an incremental advance in mechanism but a fundamental shift in design philosophy. The validation gate and rejected-edit buffer are simple mechanisms (evaluate on held-out data, record failures). But their presence reflects a philosophical commitment—that optimization requires testing proposals against unseen data, not just analyzing training failures—that was absent from the prior self-evolution literature. The paper's contribution is not just the mechanisms but the argument that this philosophy is necessary for reliable skill improvement, and the empirical demonstration that it works where unconditional self-editing fails.

Innovation 3: The Difficulty of Text-Space Optimization as a Genuine Structural Challenge, Not Just a Prompt-Engineering Problem

A subtle but important intellectual contribution of this paper is its diagnosis of why text-space optimization is hard in a way that weight-space optimization is not, and its development of mechanisms that address this hardness. The paper does not state this diagnosis explicitly as a theoretical claim, but it is embedded in every design choice and validated by the ablation results.

The core diagnosis: text-space edits have non-local, irreversible effects. In weight-space optimization, a gradient update changes parameters continuously and (for small learning rates) approximately linearly. The effect of changing one parameter on the loss is largely independent of changes to other parameters, and reversing an update is as simple as subtracting it. In text-space optimization, none of this holds. An edit that adds a rule to a skill document can interact with existing rules in complex, non-local ways: a rule that says "always use XLOOKUP" can silently contradict a rule that says "verify formula results by cross-checking with INDEX/MATCH," and the contradiction may only manifest on specific edge cases that aren't in the current batch. Once an edit is applied, it changes the semantic context for all future edits—a later edit that modifies the XLOOKUP rule is operating on a different skill than the one the optimizer originally analyzed. And reversing an edit is not simply a matter of subtracting it: the optimizer would need to propose a new edit that restores the previous text, which may not be exactly what was changed.

The mechanisms SkillOpt introduces are direct responses to this diagnosis. The bounded edit budget (textual learning rate) limits the magnitude of change per step, analogous to how a small learning rate limits parameter movement in SGD—but for a different reason. In SGD, small learning rates prevent overshooting the loss minimum. In SkillOpt, small edit budgets prevent a single edit from interacting with too many existing rules at once, making the effects of each edit more predictable and making it easier for the rejected-edit buffer to identify which specific edit caused a regression. The validation gate ensures that edits with harmful non-local effects are caught before they become the foundation for future edits—a critical protection because, in text space, building on a harmful edit can compound the damage (the edit becomes embedded in the skill's context, and later edits may reinforce rather than correct it). The epoch-wise slow/meta update provides a separate channel for long-horizon patterns, preventing local edits from overwriting rules that address cross-batch regularities.

The ablation results support this diagnosis. The largest ablation drop—removing both meta skill and slow update, causing SpreadsheetBench to fall from 77.5 to 55.0 (Table 3)—is consistent with the non-local-effects diagnosis. Without the slow/meta update, step-level edits optimize greedily for the current batch, potentially introducing rules that help the current batch but harm tasks not in the batch. Over multiple steps, these harmful interactions accumulate, and the optimizer has no mechanism to detect them because it only sees each batch in isolation. The slow/meta update catches these regressions by re-evaluating the same tasks under consecutive epoch-end skills, providing the cross-batch view that individual steps lack.

The rejected-edit buffer addresses a specific instance of the non-local-effects problem: repeated harmful edits. In weight-space optimization, if a gradient step hurts validation loss, subsequent steps will naturally move away from that direction because the gradient on the next batch points elsewhere. In text-space optimization, there is no "gradient on the next batch" that automatically corrects a harmful edit—the optimizer might independently propose the same edit again on a different batch because the failure patterns that motivated it still appear. The buffer prevents this by explicitly marking edits as harmful, providing the negative feedback that weight-space optimization gets implicitly from the changing loss landscape.

Why this diagnosis matters beyond SkillOpt. The structural challenge of text-space optimization—non-local effects, semantic interactions, irreversibility—applies to any system that edits natural-language artifacts based on feedback: prompt optimization, system design refinement, curriculum generation, self-play instruction tuning. Prior work in these areas has largely treated the challenge as a prompt-engineering problem (design better reflection prompts, use more capable models) or an evolutionary search problem (generate many variants, select the best). SkillOpt's diagnosis suggests a different approach: the challenge is inherently about control—controlling step sizes, gating updates, recording negative feedback, and separating fast and slow adaptation—and the metaphor of text-space optimization as a training process provides a principled framework for designing these controls. This is a conceptual advance that applies well beyond skill optimization, and the paper's thorough ablation of each control mechanism provides a template for evaluating their importance in other text-optimization contexts.

This is a diagnostic contribution rather than a theoretical one. The paper does not prove theorems about text-space optimization or formally characterize its difficulty relative to weight-space optimization. But it makes a compelling empirical case that the difficulty is real and that the proposed controls address it. The diagnostic framing—identifying why existing approaches fail, not just that they fail—is what elevates this beyond a simple performance improvement and makes it a contribution to the field's understanding of text-space optimization.

Innovation 4: Compact, Transferable Skill Artifacts as the Unit of Domain Adaptation

The paper's fourth conceptual contribution is empirical rather than architectural: it demonstrates that a compact skill artifact (300–2,000 tokens, 1–4 accepted edits) can serve as an effective domain-adaptation layer that transfers across models, harnesses, and benchmarks. This finding challenges two implicit assumptions in the agent-adaptation literature: that domain adaptation requires model-weight updates (fine-tuning) to be effective, and that adaptation artifacts are inevitably task-specific and non-transferable.

The compactness result is the most counterintuitive finding in the paper. The optimized skills are tiny: 379 tokens for LiveMathematicianBench, 883 for OfficeQA, 1,995 for SpreadsheetBench (Table 6). And the number of edits that survive the validation gate is startlingly small: one edit each for LiveMathematicianBench and OfficeQA, two for ALFWorld, three for DocVQA, four for SearchQA and SpreadsheetBench. These are not long documents produced by concatenating many reflection outputs; they are concise procedural rules that the validation gate has verified as individually improving held-out performance. The implication is that the gap between what the model does zero-shot and what it can do with good procedures is small in information content but large in behavioral impact—a few hundred tokens of well-targeted procedural knowledge can produce 20–40 point accuracy gains.

The transfer results extend this compactness into a deployment argument. The cross-model transfer (Table 4a) shows that a skill optimized for GPT–5.4 transfers to GPT–5.4-mini and GPT–5.4-nano with positive gains, even though the smaller models have significantly lower zero-shot performance. The cross-harness transfer (Table 4b) shows that a SpreadsheetBench skill trained inside Codex transfers to Claude Code with a +59.7 point gain—nearly matching the in-domain SkillOpt reference. The cross-benchmark transfer (Table 4c) shows positive (though smaller) gains when a math skill trained on OlympiadBench is applied to Omni-MATH. These results collectively argue that the skill artifact encodes genuinely procedural knowledge—rules about how to approach a domain, what to verify, how to format outputs—rather than model-specific or harness-specific tricks.

What makes this a conceptual contribution rather than just a nice result. The field has a natural intuition that model-weight fine-tuning is the "real" way to adapt models to domains, and that prompt-based adaptation is a lightweight but inferior substitute. SkillOpt's results challenge this intuition on two fronts. First, the absolute magnitude of the gains—+23.5 average on GPT–5.5, +26.7 on GPT–5.4-nano—is in the range that practitioners would associate with fine-tuning, not with prompt engineering. Second, the transferability of the learned artifacts suggests that the knowledge they encode is at least partially separable from the model that was used to optimize them, which is a property typically associated with weight-based knowledge (model weights transfer across tasks) but not with prompt-based knowledge (a prompt optimized for one model often fails on another). The fact that a skill optimized on GPT–5.4 transfers to GPT–5.2 (a different model generation) with positive gains suggests that the optimization loop is discovering domain procedures that are genuinely model-independent.

The optimizer-strength analysis (Table 5) clarifies what is and isn't being claimed. The paper explicitly tests whether SkillOpt is just distilling knowledge from a stronger optimizer model (GPT–5.5) into a compact prompt. The results show that a target-matched optimizer (using the same model as the target to generate edits) recovers 56–74% of the gain of the strong optimizer. This means the optimization loop itself—the bounded edits, validation gating, negative feedback, epoch-wise consolidation—provides substantial value beyond what the strong optimizer contributes. The skill artifact is not just a "distilled prompt" from a smarter model; it is the product of a controlled training process that works even when the optimizer is no more capable than the target.

Why this matters for deployment practice. The paper's deployment philosophy—"the optimizer model is used only during offline training, adding zero inference-time model calls at deployment"—is only compelling if the resulting artifact is actually reusable. The transfer results validate this philosophy: a skill trained once can be deployed across model versions (as smaller/cheaper models become available), across execution environments (as the agent's tool infrastructure changes), and to nearby tasks (amortizing training cost). This is the practical argument for treating skills—not weights, not prompts, but compact procedural documents—as the domain-adaptation layer. The paper doesn't claim that skill optimization replaces fine-tuning in all cases (the FLOPs-matched comparison in Section 3 of the prior sections shows it fails on very hard problems), but it establishes that for the broad class of problems where the model's zero-shot behavior is suboptimal but not hopeless—which includes many real-world deployment scenarios—a compact optimized skill is a practical and transferable alternative to weight updates.

This is an empirical contribution that opens a research direction. The paper does not explain why the skills are so compact or why they transfer. The qualitative examples (Figure 4) suggest that the learned rules capture procedural discipline—binding answers to evidence, verifying tool outputs, tracking search state—that is general across models and harnesses because it encodes what the agent should do, not how the model should think. But the paper leaves open the question of what class of domain knowledge can be effectively encoded in compact text artifacts and what class requires weight updates, establishing a boundary that future work can explore.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on six benchmarks spanning diverse agent-interaction modes: SearchQA [29] (extractive QA with search-engine context), SpreadsheetBench [30] (multi-round spreadsheet manipulation with real openpyxl/pandas runtime, up to 30 turns, mode=multi), OfficeQA [31] (multi-turn tool-loop QA with up to 24 tool calls on enterprise documents), DocVQA [32] (document-image VQA requiring visual-textual binding), LiveMathematicianBench [33] (competition-level math multiple-choice reasoning with proof sketches), and ALFWorld [34] (persistent embodied decision-making with up to 50 steps per episode). Two additional benchmarks—OlympiadBench [41] and Omni-MATH [14]—are used only for cross-benchmark transfer experiments. Dataset-backed runs use deterministic train/selection/test splits derived from a fixed split seed (42), with a default 2:1:7 ratio when no benchmark-specific split is stated. The test split is locked until final reporting; the selection split is used exclusively to accept or reject candidate skill edits.
Base model(s). The paper evaluates across seven target models from two families: the GPT family (GPT–5.5, GPT–5.4, GPT–5.4-mini, GPT–5.4-nano, GPT–5.2) [35] and the Qwen family (Qwen3.5–4B, Qwen3.6–35B-A3B) [36, 37]. The GPT family spans frontier-scale to small models, enabling analysis of how optimization benefits scale with model capability; the Qwen models test generalization to a different architecture family. The optimizer model defaults to GPT–5.5; the target-matched optimizer ablation (Table 5) uses the same model as the target for optimization.
Metrics. The primary metric is the benchmark's native hard score or exact-match accuracy on the held-out test split, reported as a percentage. For each benchmark, this means exact string matching against ground-truth answers (SearchQA, OfficeQA, DocVQA, LiveMathematicianBench), execution-based correctness checks (SpreadsheetBench, ALFWorld), or the benchmark's published scoring function. All scores are scalar values in [0,1]; the paper reports them as percentages (multiplied by 100). The paper aggregates six-benchmark averages for headline summary statistics.
Baselines. Seven baselines span the no-adaptation through skill-evolution spectrum: no skill (frozen target model with the benchmark's default system prompt, no skill document), human skill (an expert-written skill document curated per benchmark, 145–516 tokens), one-shot LLM skill (a skill generated once by GPT–5.5 from a high-level task description, never updated), Trace2Skill [9] (trajectory-level skill distillation from training rollouts, producing a skill artifact without iterative validation), TextGrad [40] (gradient-style natural-language prompt optimization through text differentiation), GEPA [13] (Pareto reflective prompt evolution from trajectory feedback), and EvoSkill [10] (skill-folder evolution under failure analysis, evaluated only in harness-backed settings where a matched completed run is available). All baselines use the identical target model, test split, and scorer, isolating the adaptation procedure.
Generation budget / compute accounting. The paper reports training cost in total training tokens consumed by the optimization loop (rollout generation + optimizer calls), with per-benchmark breakdowns in Table 6. Deployment cost is zero additional inference-time model calls: the deployed best_skill.md is a static text file prepended to the agent context. The training cost is thus a one-time capital expenditure amortized across all future uses of the skill. The paper does not equate "generations" across methods since the baselines (Trace2Skill, TextGrad, GEPA) have fundamentally different training computation patterns.
Cross-validation / statistical protocol. The paper does not use cross-validation for hyperparameter selection. Instead, it uses a strict three-way split protocol: the training split provides rollout evidence for edit proposals, the selection split gates every candidate edit, and the test split is used exactly once at the end for final reporting. The optimizer never sees test-split data; the selection split is used repeatedly to evaluate candidate skills during training, but the paper argues this is analogous to how a validation set is used in standard ML training (to make accept/reject decisions) rather than to fit parameters. No confidence intervals or statistical significance tests are reported, which is a limitation discussed in the critical assessment.

Main Quantitative Results

The paper's results are organized around six axes of evidence: the main comparison table (Table 1) establishing SkillOpt's dominance across models, benchmarks, and harnesses; hyperparameter analysis (Table 2) characterizing sensitivity to optimizer settings; component ablations (Table 3) identifying which control mechanisms matter; transfer experiments (Tables 4a–4c) demonstrating artifact reusability; optimizer-strength analysis (Table 5) quantifying the teacher-student gap; and cost/edit-economy characterization (Table 6) documenting what the learned artifacts look like. We walk through each.

Main Comparison: SkillOpt vs. All Baselines Across 52 Cells

Table 1 is the central result matrix. It reports held-out test accuracy for every (target model, benchmark, harness) combination, with SkillOpt's scores in blue cells and absolute changes over the no-skill baseline shown as green/red subscripts. The paper defines a cell as one (model, benchmark, harness) combination, with the strongest of the seven baselines (no skill, human skill, LLM skill, Trace2Skill, TextGrad, GEPA, EvoSkill) considered the per-cell competition. Out of 52 evaluated cells, SkillOpt is best or tied-best on all 52.

Direct-chat results (GPT family). For GPT–5.5 in direct chat, SkillOpt achieves: SearchQA 87.3 (+9.6 over no skill), SpreadsheetBench 80.7 (+38.9), OfficeQA 72.1 (+39.0), DocVQA 91.2 (+12.4), LiveMath 66.9 (+29.3), ALFWorld 95.5 (+11.9). The six-benchmark average rises from 58.8 (no skill) to 82.3 (SkillOpt), a +23.5 point absolute improvement. The best per-cell baseline—an oracle that picks the strongest of the six competing methods per cell—averages 76.9, leaving SkillOpt +5.4 points clear.

Procedural benchmarks show the largest gains. SpreadsheetBench jumps from 41.8 → 80.7, a ×1.93 relative improvement, and OfficeQA from 33.1 → 72.1, a ×2.18 relative improvement. These are exactly the benchmarks the paper identified as exposing procedural gaps—workbook-structure inspection, answer-format constraints, tool-verification discipline—that zero-shot frontier models lack. In contrast, SearchQA (77.7 → 87.3, +9.6) and DocVQA (78.8 → 91.2, +12.4) start from higher baselines and show ceiling-compressed gains.

Direct-chat results across model scales. The pattern is remarkably uniform: SkillOpt is the best method on every benchmark for every model in direct chat. The magnitude of improvement is larger for smaller models: GPT–5.4-nano gains +26.7 points averaged across six benchmarks (including +49.4 on DocVQA, +35.1 on ALFWorld, +19.0 on SpreadsheetBench), while GPT–5.4 gains +12.7 points average. The paper notes this is "consistent with the view that a compact skill artifact can supply procedural knowledge that small models do not yet hold in weights." The cross-model generalization is striking: SkillOpt improves Qwen3.5–4B from 9.3 to 23.9 on SpreadsheetBench (×2.6) and from 30.6 to 81.3 on ALFWorld (×2.7), despite Qwen being a different architecture family from the GPT models that the default optimizer was designed around.

Harness-backed results (Codex and Claude Code). On the Codex harness with GPT–5.5, SkillOpt achieves average +24.8 points over no skill across five benchmarks (ALFWorld is excluded because it requires persistent embodied interaction not represented in the Codex/Claude Code adapters). It beats EvoSkill—the strongest harness-specific baseline—by +14.0 points on average. The SpreadsheetBench Codex cell is illustrative: no skill scores 27.5, EvoSkill lifts to 67.5 (+40.0), and SkillOpt adds another +17.5 to reach 85.0. On the Claude Code harness, SkillOpt achieves average +19.1 over no skill and +3.2 over EvoSkill, with SpreadsheetBench showing 22.1 → 80.4 (+58.3).

Headline aggregates. The paper provides four headline aggregates from Table 1: (i) 52/52 cells best or tied-best; (ii) average per-model improvement of approximately +17.6 points across the seven direct-chat target models; (iii) GPT–5.5 improvement of +23.5 (direct chat), +24.8 (Codex), +19.1 (Claude Code) over no skill; and (iv) GPT–5.5 oracle-baseline gap of +5.4 points, computed as the difference between SkillOpt's six-benchmark direct-chat average (82.3) and the best-per-cell baseline average (76.9).

Interpreting the baseline comparisons. Several comparisons in Table 1 illuminate why SkillOpt works. Human skills and LLM skills are competitive on benchmarks where prior knowledge matches the domain (SpreadsheetBench human skill: 72.9; DocVQA LLM skill: 89.6) but cannot correct failures after observing rollouts, so they plateau below SkillOpt everywhere. Trace2Skill mines trajectory lessons but lacks a validation gate, so it underperforms SkillOpt on every cell (e.g., GPT–5.5 direct chat SpreadsheetBench: 49.6 vs. 80.7). TextGrad and GEPA optimize prompts but not persistent skill artifacts, and they show inconsistent results: TextGrad actually hurts SpreadsheetBench (−0.7 on GPT–5.5) and ALFWorld (−0.8), while GEPA achieves strong SpreadsheetBench results (73.6) but weaker LiveMath results (43.2). The harness-backed EvoSkill comparison is the most informative: EvoSkill already achieves large gains on SpreadsheetBench under Codex (27.5 → 67.5), but SkillOpt's additional +17.5 points demonstrate that the gap is not just about having a skill evolution loop, but about having controlled optimization within that loop.

Hyperparameter Sensitivity Analysis

Table 2 sweeps six optimizer hyperparameters on three benchmarks (SearchQA, SpreadsheetBench, LiveMath) using GPT–5.5 as both target and optimizer. The paper argues that SearchQA has limited headroom and is therefore stable across most settings (cells fluctuate within ±1.5 points), while SpreadsheetBench and LiveMath expose the trade-off between learning useful procedures and over-editing.

Training set size (panel a). SearchQA saturates at 84–86 after seeing 20% of the training partition. SpreadsheetBench climbs monotonically from 47.5 (1 example) to 78.0 (100% train). LiveMath climbs from 59.1 to 70.5 with more data but shows a non-monotonic dip at 40% (65.9 → 64.8 → 67.0). This suggests procedural benchmarks reward more training evidence, while factual benchmarks with strong model priors benefit less from additional rollout data.

Reflection minibatch size (panel b). Sweeping Bm from 1 to 32 shows remarkable stability: SearchQA stays within 85.9–87.1, SpreadsheetBench within 75.4–77.9, LiveMath within 54.8–64.5. The default Bm=8 is at or near the top on all three benchmarks. The paper interprets this as evidence that "the headline gains are not the product of a fragile prompt-search batch size, but a genuine effect of having enough scored evidence per update."

Rollout batch size (panel c). Similarly stable: B=8 to full epoch keeps SearchQA within 85.1–87.2 and SpreadsheetBench within 75.0–77.5. The full-epoch setting (which exposes all training data before any edit) performs slightly worse on LiveMath (53.2) and SpreadsheetBench (75.0), suggesting that more frequent updates with smaller batches may help the optimizer course-correct before committing to edits based on noisy full-epoch evidence.

Textual learning rate Lt (panel d). Sweeping Lt ∈ {1, 2, 4, 8, 16} shows that any moderate edit budget works: Lt=4 achieves 86.5/78.2/56.5 on SearchQA/SpreadsheetBench/LiveMath; Lt=8 achieves the highest LiveMath score at 66.9; even the worst setting across all five values is 85.5 on SearchQA. The paper argues this supports "the bounded-update story does not depend on a single specific scheduler" or budget value.

Learning-rate schedule (panel e). Constant schedule scores 87.3/80.7/62.1, cosine scores 87.1/77.5/61.3, and linear scores 87.2/72.9/62.9. The constant schedule performs best on SpreadsheetBench (80.7) and SearchQA (87.3); linear produces the weakest SpreadsheetBench result (72.9). The paper notes that "any moderate, bounded edit budget already beats baselines that rewrite the skill without a budget." The constant schedule's strength on SpreadsheetBench may reflect that the optimizer benefits from consistently applying its full edit budget rather than decaying, perhaps because the domain has multiple independent procedural gaps that each require separate rules.

Slow-update samples (panel f). Varying the number of tasks sampled for epoch-wise comparison (5, 10, 20, 40) keeps all three benchmarks within ±2.7 points of the default (20). LiveMath shows the most sensitivity (64.5, 65.3, 61.3, 54.8), with the drop at 40 samples suggesting that including too many tasks in the slow-update comparison may blur the signal, perhaps because the optimizer becomes conservative when it sees too many regression cases simultaneously.

Component Ablations: What Makes SkillOpt Work

Table 3 tests three core mechanisms—textual learning rate, rejected-edit buffer, and epoch-wise slow/meta update—by removing or altering each while holding the rest of the optimizer fixed with GPT–5.5. The default setting is the top row in each group.

Learning-rate form. Three conditions: constant Lt=4 (the default), dynamic lr (the optimizer model autonomously decides the budget), and without lr (unbounded edits). SearchQA drops from 87.1 → 85.8 → 84.6; SpreadsheetBench drops from 77.5 → 71.8 → 75.7; LiveMath drops from 61.3 → 54.0 → 57.3. The "without lr" row (unbounded edits) does better than dynamic on two of three benchmarks (SpreadsheetBench: 75.7 vs. 71.8; LiveMath: 57.3 vs. 54.0), which is perhaps surprising—letting the model decide its own budget is worse than having no budget at all. The paper does not analyze this result further, but it suggests that the optimizer model, when given autonomy over edit magnitude, may be overly conservative or inconsistent, while a fixed moderate budget provides a reliability floor.

Rejected-edit buffer. Removing the buffer drops SearchQA from 87.1 to 85.5 (−1.6), SpreadsheetBench from 77.5 to 72.9 (−4.6), and LiveMath from 61.3 to 58.9 (−2.4). The SpreadsheetBench drop is the largest, consistent with the paper's earlier analysis that procedural benchmarks generate many plausible-sounding but harmful edit proposals—the buffer prevents the optimizer from cycling through variations of the same bad idea.

Slow/meta update. Three conditions: full meta skill and slow update (default), without meta skill (slow update still active but optimizer loses its editing-strategy memory), and without both meta skill and slow update. Removing only the meta skill drops SearchQA from 87.1 to 85.1 (−2.0), SpreadsheetBench from 77.5 to 75.7 (−1.8), LiveMath from 61.3 to 58.1 (−3.2). Removing both meta skill and slow update is catastrophic on SpreadsheetBench: 77.5 → 55.0 (−22.5), the largest single ablation drop in the suite. SearchQA drops to 86.3 (−0.8) and LiveMath to 59.7 (−1.6).

The paper interprets the SpreadsheetBench result as direct evidence that the slow/meta update "catches regressions that individual batches cannot see" and that "without it, the optimizer is effectively myopic—it reacts to each batch's failures without retaining any sense of what it has already tried and what has actually helped across the full training distribution." The SearchQA result is more muted, likely because the headroom is small and individual batches provide sufficient signal; the damage from removing long-horizon guidance is limited when most batches already point in the right direction.

Validation checkpoint trends (Figure 3). Figure 3 tracks training rollout score, selection-best score, and unseen test score across epoch checkpoints for SpreadsheetBench, SearchQA, and LiveMath. The key pattern: test-set performance generally tracks the selection-best score, meaning that the skill the validation gate prefers tends to be the skill that generalizes, not a skill that overfits to the selection split. On SpreadsheetBench (panel a), the test score climbs from 0.42 at epoch 1 to 0.82 at epoch 8, closely tracking selection-best. On SearchQA (panel b), test performance is essentially flat across epochs 1–16 (oscillating around 0.84–0.87), consistent with ceiling effects. On LiveMath (panel c), test performance shows more variance (0.55–0.80) but the selection-best checkpoint at epoch 8 aligns with the test peak. The paper does not report whether the test peak would have been selected under a different gating criterion, but the rough alignment supports the claim that the gate selects skills that generalize.

Transfer Experiments

Tables 4a–4c test whether an optimized skill behaves as a reusable artifact rather than a task-specific prompt. Each row shows: the source (model/benchmark/harness where the skill was trained), the target, the target's no-skill baseline, the in-domain SkillOpt reference (skill trained specifically for the target), the transferred score, and the gain over baseline.

Cross-model transfer (Table 4a). On SpreadsheetBench, a skill trained with GPT–5.4 transfers to GPT–5.4 (+10.7 over baseline), GPT–5.4-mini (+9.4), and GPT–5.4-nano (+3.0). On LiveMath, the same source skill transfers to GPT–5.4 (+10.4), GPT–5.4-mini (+4.5), and GPT–5.4-nano (+5.6). Two of the six rows show the transferred skill exceeding the in-domain SkillOpt reference: LiveMath GPT–5.4 (47.2 transferred vs. 44.0 in-domain) and LiveMath GPT–5.4-nano (28.8 transferred vs. 27.2 in-domain). The paper interprets this as evidence that "some learned procedures are target-model agnostic." The remaining rows recover a useful fraction of the in-domain gain. Crucially, no row falls below the target's no-skill baseline.

Cross-harness transfer (Table 4b). This is the strongest deployment signal. A SpreadsheetBench skill trained inside the Codex harness transfers to the Claude Code harness with a +59.7 point gain (22.1 → 81.8), slightly exceeding the in-domain Claude Code SkillOpt reference of 80.4. The symmetric Claude-Code→Codex transfer adds +43.6 over the Codex baseline (27.5 → 71.1), though it does not reach the in-domain Codex SkillOpt reference of 85.0. On LiveMath, the Codex→Claude Code transfer is modest (+1.6 over 40.8 baseline), while the Claude-Code→Codex transfer adds +12.8 (35.2 → 48.0). The paper argues that these positive transfers show "the learned rules are not only harness-specific command recipes," and specifically that the SpreadsheetBench skill encodes workbook-level procedures (structure-first inspection, formula-aware verification) that transcend the specific tool APIs of Codex vs. Claude Code.

Cross-benchmark transfer (Table 4c). The strictest test: an OlympiadBench skill is evaluated on Omni-MATH across three model scales. Gains are positive but small: +3.7 (GPT–5.4), +1.8 (GPT–5.4-mini), +1.3 (GPT–5.4-nano). The paper acknowledges these are "smaller than the in-domain and cross-harness transfers—unsurprisingly, since they require the optimized skill to retain useful procedural knowledge after both the test instances and the answer-format conventions change." The positive direction is notable given that the benchmarks share only the broad math family, but the magnitude is small enough that it could be noise from the test-set split.

Optimizer-Strength Analysis

Table 5 compares two optimizer regimes: a strong frontier optimizer (GPT–5.5) and a target-matched optimizer (the same model as the target) on SpreadsheetBench and SearchQA at two target scales (GPT–5.4-mini and GPT–5.4-nano). The strong optimizer always produces larger gains: on SpreadsheetBench, +11.4 vs. +7.1 (GPT–5.4-mini) and +19.0 vs. +11.9 (GPT–5.4-nano); on SearchQA, +4.3 vs. +2.4 (GPT–5.4-mini) and +19.0 vs. +14.1 (GPT–5.4-nano).

Two observations matter. First, the strong optimizer advantage is monotonic: the bounded-edit, validation-gated loop prevents a stronger optimizer from pushing larger but harmful rewrites. Second, the target-matched optimizer recovers 56–74% of the strong-optimizer gain (e.g., +11.9 of +19.0 on SpreadsheetBench GPT–5.4-nano), meaning "SkillOpt is not a distillation pipeline from a stronger teacher into a weaker student: the optimization loop itself contributes substantial value on top of whatever the optimizer can already do." The paper's deployment recommendation follows: use a strong frontier optimizer when available (it costs only training-time API calls), but the loop remains effective with a weaker optimizer if budget constraints require it.

Learned Skills: Compactness, Cost, and Qualitative Content

Table 6 characterizes the final skill artifacts from the GPT–5.5/GPT–5.5 runs. Three patterns stand out.

Compactness. Final skills range from 379 tokens (LiveMath) to 1,995 tokens (SpreadsheetBench), with a median around 920 tokens. Even the longest is far below typical system-prompt budgets for frontier models. The growth from initial to final skill is moderate (×2.5 to ×53 depending on whether the initial skill was a one-liner or a paragraph), but the final absolute size remains small enough for human audit.

Edit economy. The number of accepted edits surviving the validation gate is 1–4 across all six benchmarks. LiveMath's +29.3 point gain comes from a single accepted edit; OfficeQA's +39.0 gain similarly from one edit. The paper interprets this as "direct evidence that the validation gate is doing real work: the optimizer model proposes many more edits per epoch, but only a handful pass the held-out check and survive into the deployed skill." The bulk of text-space search effort is rejected and captured by the buffer for future use.

Cost per point. Training cost ranges from 0.6M tokens per test-set point (SpreadsheetBench: 21.4M total tokens, +38.9 point gain) to 46.4M tokens per point (DocVQA: 188.2M total tokens, +12.4 point gain). The paper splits these into two regimes: procedural benchmarks with short/cheap rollouts (SpreadsheetBench, OfficeQA, LiveMath: 0.6–3.6M tokens/point), and benchmarks with longer trajectories or richer context (SearchQA: 37.9M/point, DocVQA: 46.4M/point). These costs are paid once during training; the deployed artifact adds zero optimizer calls.

Qualitative content (Figure 4). One representative learned rule per benchmark, quoted verbatim from the deployed best_skill.md:

SearchQA: "Infer the expected answer type from clue wording, then choose the shortest canonical entity supported by co-occurring distinctive evidence."
SpreadsheetBench: "Inspect workbook structure and formulas, then write evaluated static values across the full requested target range instead of relying on Excel recalculation."
OfficeQA: "Treat oracle parsed pages as primary evidence, lock table/date/unit context, and output exactly the requested rounded value without extra labels."
DocVQA: "For tables, forms, charts, and legends, first bind the question to the exact visual row/header/field, then copy only the aligned answer span."
LiveMath: "In strongest-statement MCQs, rank choices by theorem strength and prefer a justified stronger-result option over true but weaker corollaries."
ALFWorld: "Keep a horizon-aware visited/frontier ledger, diversify search after repeated same-type failures, and avoid revisiting the destination until holding the target."

The paper notes three properties: the rules are procedural rather than instance-specific (no rule names a specific question or file); they encode discipline that frontier models lack zero-shot (answer-format constraints, evidence binding, search-frontier management); and they read like rules a thoughtful human would write after experience with the benchmark—except they are produced automatically and validated edit-by-edit on held-out data.

Qualitative Skill Evolution Case Studies

Section 4.5 walks through two representative optimization runs to show what the optimizer actually learns.

ALFWorld (GPT–5.4-nano student, GPT–5.5 teacher). The initial skill gives a generic household plan: search, pick up, transform, place. The accepted edits make this plan stateful: exact object-name matching (mugs ≠ cups), visited-location memory (prefer unvisited receptacles), destination memory, pick-two progress locks, and direct completion rules (once the agent can complete a subgoal, take that action instead of examining or verifying again). The skill evolves from a general search-transform-place strategy into a finite-state execution policy with object identity, search memory, progress locks, and loop breakers. The selected skill improves held-out test performance from 49.3 to 74.6.

SpreadsheetBench (GPT–5.5 student, GPT–5.5 teacher). The initial skill instructs the agent to use Python spreadsheet libraries and preserve unrelated content. The accepted edits turn this into workbook-forensics: inspect the actual workbook (not previews), locate headers and target ranges across sheets, normalize keys and cell types before lookup, preserve formatting during structural edits, and—crucially—when the grader reads cell values, write evaluated static values even if the prompt mentions formulas like INDEX/MATCH or XLOOKUP. Later edits require filling complete target ranges (including blanks), keeping helper computations in Python (not the workbook), and reopening the saved workbook to verify boundary rows. The selected skill improves held-out test performance from 40.4 to 78.9.

Both cases illustrate the paper's central claim about what controlled optimization produces: not a longer or more comprehensive skill, but a more procedurally precise one, where each accepted edit adds a constraint that prevents a specific recurring failure mode observed across multiple rollout batches.

Ablation Studies and Robustness Checks

The paper conducts an extensive ablation suite, largely summarized in Sections 4.2 and captured across Tables 2–3 and Figure 3. Beyond the component-level findings already discussed, several robustness checks deserve explicit mention.

Validation gate strictness and edit observability. The paper implements the gate as strictly greater-than (ties rejected) and records every candidate's accept/skip status in an edit_apply_report.json file for post-hoc auditability. This implementation choice—not evaluated in a separate ablation—is argued as load-bearing: it ensures the deployed skill never silently drifts through equally-performing but differently-worded versions.

Accumulation and parallelism. The paper notes that the optimizer supports an accumulation factor (multiple rollout batches reflected on separately and merged into one update) and parallel analyst workers (16 by default for reflection minibatch analysis), but these are engineering optimizations for throughput rather than quality parameters; no ablation tests varying the accumulation factor or worker count.

Cross-validation over skill splits. The paper does not use traditional k-fold cross-validation. Instead, it relies on the three-way train/selection/test split with a fixed seed (42). The paper acknowledges that some benchmarks have small training pools (LiveMath: 35 items/epoch; ALFWorld: 39 tasks) and adjusts batch sizes accordingly, but does not report variance across random seeds. This is a genuine limitation: with test splits of 500 questions (or fewer for some benchmarks) and selection splits proportionally small, the reported results may have non-trivial variance that is not characterized.

Rewite mode vs. patch mode. The paper mentions that patch mode is the default and rewrite mode is an alternative, but does not provide an ablation comparing the two. Given that the bounded-update mechanism operates at the level of individual edits in patch mode, a rewrite-mode ablation would clarify whether the edit operations or the edit budget is the primary mechanism—but this is not tested.

Oracle vs. predicted difficulty. Unlike the reference paper (which compared oracle and predicted difficulty bins for compute-optimal scaling), SkillOpt does not have a difficulty-prediction component; its gate is based purely on held-out task scores. This means there is no analogous oracle-vs-predicted comparison to test. The paper's claim about difficulty estimation is implicit: the validation gate is the difficulty-adaptive mechanism, accepting edits only when they help on the selection distribution.

Critical Assessment

The experiments in this paper are unusually comprehensive for a systems paper—52 evaluation cells, seven target models, three harnesses, two model families, six benchmarks, seven baselines, five hyperparameter sweeps, three component ablations, three transfer axes, and an optimizer-strength analysis. The breadth of evidence makes a strong prima facie case for SkillOpt's effectiveness. But several aspects of the experimental design constrain the strength of specific claims.

Claim 1: SkillOpt is best or tied-best on 52 of 52 evaluated cells. This claim is supported by the data in Table 1, but "tied-best" needs scrutiny. The paper counts a cell as "best or tied-best" if SkillOpt achieves the highest reported score among all baselines, but it does not report statistical significance or confidence intervals. With small test sets (e.g., LiveMath, ALFWorld, OfficeQA have test sizes on the order of 100–500 questions), a 1–2 point difference between SkillOpt and the next-best baseline may not be statistically reliable. The paper acknowledges this implicitly by not claiming "strictly best" on all cells—several cells show SkillOpt within a point of the next-best baseline (e.g., Qwen3.6–35B-A3B on OfficeQA: 47.1 vs. 45.9 no-skill, a 1.2 point gap). Without variance estimates, it is impossible to distinguish genuine improvement from sampling noise in these cases. This does not undermine the aggregate evidence—the pattern across 52 cells is clearly SkillOpt-favored—but the "52/52" claim overstates the precision of individual comparisons.

Claim 2: +23.5 average gain on GPT–5.5 direct chat, +5.4 over oracle baseline. This computation mixes benchmarks with very different score ranges and gain ceilings. SearchQA's gain (+9.6 on a 77.7 baseline) and DocVQA's gain (+12.4 on a 78.8 baseline) contribute much less to the average than SpreadsheetBench's (+38.9) and OfficeQA's (+39.0). The six-benchmark average is thus heavily weighted toward procedural benchmarks where SkillOpt excels. This is not a flaw—the benchmarks were chosen to span diverse domains, and the paper reports per-benchmark deltas transparently—but it means the +23.5 figure should be understood as an average over this particular benchmark mix, not as an expected gain on an arbitrary task.

Claim 3: Compact artifact (1–4 edits, 300–2,000 tokens). This is exactly supported by Table 6, but it raises a question the paper does not address: what is the lower bound on skill size? Could the gains be achieved with even fewer tokens? The paper's answer is implicit in the edit economy (1–4 edits), but there is no ablation testing whether a human could further compress the learned rules without loss, or whether training with a stricter token budget would produce even more compact artifacts. The compactness result is descriptive, not prescriptive—it says what SkillOpt produces, not what is minimal.

Claim 4: Transfer across models, harnesses, and benchmarks. The cross-model and cross-harness results in Tables 4a–4b are genuinely strong: every transferred skill beats the target's no-skill baseline, and some match or exceed the in-domain SkillOpt reference. The cross-benchmark result (Table 4c) is weaker: the gains are positive but small (+1.3 to +3.7 points), and with the small test sets involved, these could be noise. The paper acknowledges the weakness implicitly by describing these as "positive" rather than "substantial," but the headline claim that "optimized skill artifacts retain value when moved... to a nearby math benchmark" is narrowly supported by three positive rows on one source→target pair (OlympiadBench→Omni-MATH). This is the thinnest evidence in the paper, and further cross-benchmark experiments would be needed to establish general transferability.

Missing baselines and comparisons. Several experimental gaps limit the strength of the paper's claims:

No ensemble baselines. The paper compares against individual skill-construction methods but not against an ensemble of skills (e.g., majority voting across a human skill, LLM skill, and Trace2Skill artifact). Given that SkillOpt's validation gate implicitly ensembles over candidate skills by selecting the best on held-out data, a fair comparison might give baselines the same budget for generating and selecting among multiple skill candidates.
No prompt-length normalization. Human skills are 145–516 tokens; SkillOpt skills are 379–1,995 tokens. Longer skills might have an inherent advantage independent of their content. A length-matched baseline (e.g., concatenating the human skill with random task-irrelevant text, or asking the LLM to generate a skill of equivalent length) would control for this, but is not reported.
No comparison against fine-tuning on training trajectories. The paper's claim that skill optimization is a practical alternative to weight updates would be strengthened by a direct comparison against fine-tuning the target model on the training split's successful trajectories. This is a difficult comparison to make (closed models can't be fine-tuned, and fine-tuning protocols vary), but its absence means the claim that skill optimization "substitutes for weight adaptation" remains unverified.
No cost-matched comparison against baselines. The paper reports SkillOpt's training cost (Table 6) but does not report comparable cost figures for Trace2Skill, TextGrad, GEPA, or EvoSkill. A cost-matched comparison (e.g., "at equivalent training budget, which method performs best?") would strengthen the efficiency claim, but is missing.

Single optimizer model for main results. All main results use GPT–5.5 as the optimizer model. The optimizer-strength analysis (Table 5) shows that a target-matched optimizer recovers much of the gain, but the fact that the default optimizer is a frontier model raises a concern: is SkillOpt's performance advantage due to the optimization loop, or due to having a much stronger model in the loop? The paper's answer—56–74% of the gain remains with a matched optimizer—is reassuring but incomplete, because the matched-optimizer experiments cover only two benchmarks and two target scales. The claim that "the optimization loop itself contributes substantial value" would be more convincing with matched-optimizer results across all six benchmarks.

Test-set contamination risk. The paper uses deterministic splits with a fixed seed (42). The optimizer model (GPT–5.5) may have seen the test-set questions during its own pretraining, which could inflate SkillOpt's test-set performance if the optimizer proposes edits that inadvertently exploit this knowledge. The paper does not discuss this risk; human-written and LLM baselines share the same contamination exposure, so the relative comparison is valid, but the absolute gains (+23.5) may be inflated if the optimizer model has test-set familiarity.

Evaluation harness consistency. The three harnesses (direct chat, Codex, Claude Code) are described but not systematically compared as execution platforms. The paper does not report whether the no-skill baselines differ across harnesses due to harness-level prompt differences, tool availability, or error handling. The cross-harness transfer results partially address this (skills transfer despite harness differences), but the harness-specific baselines would be more interpretable if the paper reported the source of harness-level variance.

Qualitative evaluation of learned skills. The paper provides one representative rule per benchmark (Figure 4) and two qualitative case studies (Section 4.5). These are compelling illustrations but not systematic: there is no characterization of what fraction of learned rules fall into each category (procedural vs. instance-specific, general vs. benchmark-specific), no ablation testing whether individual rules are load-bearing (removing one rule and measuring the score drop), and no analysis of whether different optimizer runs produce similar rules (reproducibility of the learned artifact). The qualitative evidence supports the claim that learned skills encode procedural discipline, but it does not quantify how much of the gain comes from this discipline versus from less interpretable textual effects.

Statistical rigor. The paper reports no confidence intervals, no standard deviations, and no significance tests across any of its 52 cells. The ablation studies vary one hyperparameter at a time without reporting whether the observed differences exceed expected noise. The test sets range from moderate (500 for SearchQA) to small (LiveMath, ALFWorld have unclear test sizes in the main text), and with fixed single-seed splits, the reported numbers are point estimates with unknown variance. This is a common pattern in agent-systems papers—running full evaluations on 52 cells is expensive—but it means individual cell-level comparisons should be interpreted cautiously. The aggregate pattern across 52 cells, 3 harnesses, and 7 models is more robust than any single cell, but the paper's "best on all 52" framing invites scrutiny of individual comparisons that the experimental design does not fully support.

Summary of evidential strength. The paper's central empirical claim—that SkillOpt's controlled optimization loop produces better skills than existing approaches across a diverse set of benchmarks, models, and harnesses—is well-supported by the breadth and consistency of the evidence in Table 1, the hyperparameter stability in Table 2, and the component ablations in Table 3. The transfer claims are supported for cross-model and cross-harness settings, but the cross-benchmark evidence is thin. The compactness and cost-economy claims are exactly as reported in Table 6. The claim that controlled optimization (bounded edits, validation gating, negative feedback, epoch-wise consolidation) is the cause of SkillOpt's improvement—rather than, say, having a strong optimizer model or more computational budget—is supported by the component ablations (Table 3) and optimizer-strength analysis (Table 5), but the evidence would be stronger with: (1) matched-cost comparisons against baselines, (2) matched-optimizer experiments across all benchmarks, (3) statistical significance reporting, and (4) systematic qualitative analysis of what the learned rules contribute individually. These gaps do not undermine the paper's contribution—the evidence is more comprehensive than most systems papers—but they mean some subordinate claims should be held with appropriate uncertainty.

6. Limitations and Trade-offs

The Validation Gate Requires Scored Tasks with Reliable Automatic Verifiers

The assumption or constraint. SkillOpt's entire optimization loop depends on a held-out selection split D_sel with per-task scalar scores r(s) ∈ [0, 1] that can be computed automatically. The paper acknowledges this directly in Appendix B:

"it is most directly applicable when the target task has automatic verifiers, exact-match metrics, executable checks, or otherwise reliable feedback signals. For open-ended domains where success is subjective, multi-dimensional, or costly to judge, the validation gate may require stronger human or model-based evaluation."

The gate is the central control mechanism—the component that distinguishes SkillOpt from unconditional self-editing—and it requires evaluating every candidate skill on dozens to hundreds of held-out tasks. This evaluation must be fully automatic because the loop may test many candidate skills per epoch (at least one per optimization step, plus the slow-update candidate at epoch boundaries). The paper's experiments use benchmarks with deterministic verifiers: exact string matching (SearchQA, OfficeQA, DocVQA, LiveMath), execution-based checks (SpreadsheetBench), and environment success signals (ALFWorld). None of the benchmarks requires human judgment or LLM-as-judge evaluation.

The consequence. For domains where success is ambiguous—creative writing quality, dialogue helpfulness, code review usefulness, strategic advice soundness—the validation gate cannot operate as designed. An LLM-as-judge replacement introduces a new failure mode: the judge model may share biases with the target model, so both could be satisfied with a skill that produces fluent but incorrect outputs, and the gate would accept the edit because the judge (not the task) is fooled. Alternatively, human evaluation is too slow and expensive to gate every candidate skill edit (the optimizer may propose dozens of candidates per epoch). This means SkillOpt, in its current form, is restricted to domains with cheap, reliable, automatic success signals—a constraint that excludes many of the most commercially valuable agent applications (customer support quality, content generation, open-ended research assistance).

What evidence exists in the paper. None. The paper does not test SkillOpt on any domain without automatic verifiable ground truth. Appendix B mentions the limitation but does not experiment with LLM-as-judge gating, human-in-the-loop gating, or preference-driven selection. The six benchmarks all have clean correctness signals, so the paper provides no empirical guidance on how the method degrades when the gate signal becomes noisy, biased, or expensive.

Mitigation status. The paper acknowledges the limitation (Appendix B: "for open-ended domains where success is subjective... the validation gate may require stronger human or model-based evaluation") and suggests "reward-free or preference-driven validation gates for open-ended tasks" as future work in Section 5. No mitigation is attempted. This is a genuine scope boundary: SkillOpt is validated only for objectively verifiable tasks, and its applicability to subjective or open-ended domains is unproven speculation.

Training Cost Is Not Accounted for in the Headline Efficiency Claims, and Can Be Prohibitive for One-Off Tasks

The assumption or constraint. The paper reports training costs in tokens (Table 6: 21.4M–213.8M tokens per benchmark) and frames this as a one-time capital expenditure: "the cost is paid once during skill training; after export, the optimized best_skill.md adds no optimizer calls, no weight updates, and only a compact text artifact to the target agent." The paper's deployment philosophy—and its practical appeal—rests on this amortization argument. SkillOpt is efficient if the trained skill is reused many times; it is expensive if the domain changes frequently or the task is one-off.

The training process itself is computationally heavyweight. The default configuration runs four epochs with rollout batch size 40, reflection minibatch size 8, 16 parallel analyst workers, a merge pipeline, and an optimizer model (GPT–5.5) that is called repeatedly for reflection, merging, ranking, and slow/meta update. Each candidate skill is evaluated on the full selection split. The paper does not report wall-clock time or dollar cost in API calls, but the token counts suggest non-trivial expense: DocVQA consumed 188.2M training tokens for a +12.4 point test-set gain (46.4M tokens per point).

The consequence. For a practitioner deciding whether to deploy SkillOpt, the key question is: how many times will I use this skill? If the answer is thousands or millions (a production QA system, a spreadsheet automation service), the one-time training cost is negligible per query. If the answer is dozens (a one-off analysis task, a rapidly changing domain, an exploratory project), the training cost may exceed the value of the improved performance. The paper's headline "zero inference-time cost" framing is true but omits the training-cost break-even analysis that a practitioner needs. The paper also does not compare SkillOpt's training cost to the cost of alternative approaches: what does it cost to have a human expert write a skill? To run Trace2Skill or GEPA on the same data? To fine-tune the target model? Without these comparisons, the efficiency claim is one-sided—it counts inference savings but not training expenditures.

What evidence exists in the paper. Table 6 provides training token counts and cost-per-point for each benchmark. The hyperparameter sensitivity analysis (Table 2) shows that reducing training set size (panel a: "1 example," "20% train") and batch sizes (panels b, c) does not cause catastrophic drops—suggesting that cheaper training configurations are possible, though the paper does not frame this as a cost-ablation. The paper does not report dollar costs, API pricing, or wall-clock training time for any run.

Mitigation status. The paper explicitly acknowledges in Appendix B that "training the skill requires additional rollout computation and calls to an optimizer model; this cost is amortized when the same skill is reused, but may be less attractive for one-off tasks." The transfer experiments (Section 4.3) partially address this by showing that a trained skill can be reused across models, harnesses, and nearby benchmarks—amortizing the training cost across multiple deployment scenarios. But the fundamental tradeoff (training cost vs. reuse frequency) is not quantified. No ablation studies how performance varies with total training budget, and no comparison to the cost of alternative adaptation methods is made.

The Difficulty-Estimation Analogue Is Missing: SkillOpt Has No Mechanism to Detect When Problems Are Too Hard for Skill Optimization to Help

The assumption or constraint. SkillOpt optimizes a skill document by proposing edits based on training trajectories and gating those edits on held-out performance. It assumes that the training distribution contains correctable failures—trajectories where the model has the underlying capability but is making procedural errors that a skill rule can fix. The paper does not discuss, let alone address, what happens when this assumption is violated: what if the failures on the training split are due to fundamental capability gaps (the model genuinely cannot solve certain problem classes) rather than procedural gaps (the model could solve them but is using the wrong procedure)?

This is a structural limitation that the reference paper on compute-optimal test-time scaling addressed explicitly: its difficulty-binning mechanism identified the hardest problems (bin 5) where no test-time strategy helped, and routed compute away from them. SkillOpt has no analogous mechanism. The validation gate will simply reject edits that don't improve held-out performance, but it cannot distinguish between "no edit proposed so far has helped" and "no edit could possibly help because the model lacks the underlying capability." In the latter case, the optimizer will continue proposing edits, the gate will continue rejecting them, and the loop will consume compute without producing useful output.

The consequence. In domains with a mix of easy, medium, and hard problems, SkillOpt will spend optimization effort on hard problems that cannot benefit from skill editing. The rejected-edit buffer will record the failed proposals, but the optimizer has no way to learn the meta-pattern "stop trying to optimize for this class of problems; the model can't do them." The paper's results on the hardest problems are largely missing—the benchmarks do not have explicit difficulty bins, so we cannot see whether SkillOpt's gains are concentrated on easy/medium problems (as in the reference paper) or are uniform across difficulty levels. If the gains are concentrated, then SkillOpt's efficiency (cost per point of gain) is inflated by spending optimizer effort on problems it cannot improve.

What evidence exists in the paper. Indirect evidence exists in the performance ceilings. Several benchmarks show SkillOpt reaching high but not perfect accuracy: SearchQA plateaus around 87%, LiveMath around 67%, SpreadsheetBench around 81%. The residual errors could be fundamentally hard problems that no skill edit can fix. The paper does not analyze which test-set questions SkillOpt fails on or whether those failures share characteristics (e.g., requiring mathematical reasoning beyond the model's capability) that would make them identifiable in advance. The live training curves (Figure 3) show that test performance plateaus as epochs increase—suggesting diminishing returns—but do not decompose this plateau into "all correctable problems fixed" vs. "remaining problems are uncorrectable."

Mitigation status. The paper does not address this limitation. There is no difficulty-estimation component, no routing mechanism that allocates more optimizer budget to correctable problems and less to uncorrectable ones, and no analysis of whether residual errors are procedural or fundamental. The "Outlook" section mentions "skill libraries that share infrastructure across domains" and "self-distillation of optimized skills back into the target model" but does not mention difficulty-aware optimization. This is a notable gap because the reference paper demonstrated that difficulty-conditioned allocation is essential for efficient test-time compute scaling, and the same principle likely applies to training-time skill optimization.

The Optimized Skills Are Not Guaranteed to Be Robust, Interpretable, or Safe—Only That They Pass a Held-Out Accuracy Gate

The assumption or constraint. The paper's validation gate accepts a candidate skill if and only if it strictly improves held-out accuracy on the selection split. This criterion optimizes for one property—task performance—and does not consider robustness to distribution shift, interpretability, safety, or freedom from hidden failure modes. The paper presents the compactness and qualitative plausibility of the learned rules (Figure 4) as evidence of quality, but these are post-hoc observations, not constraints that the optimizer enforces during training.

The paper states that the learned rules "read like rules a thoughtful human practitioner would write after a day with the benchmark—except they are produced automatically by the optimizer and validated edit-by-edit on held-out data." The words "read like" are doing a lot of work. A rule that improves held-out accuracy could still be brittle (fails under slight distribution shift), overly specific (improves only on the exact test distribution and transfers poorly), or even harmful in edge cases that the selection split doesn't cover. The validation gate catches rules that hurt the selection-split distribution; it does not catch rules that help on the selection split but hurt elsewhere.

The consequence. For safety-critical deployments, SkillOpt provides no guarantees beyond accuracy on the specific held-out distribution it was validated on. A learned rule like "write evaluated static values across the full requested target range" (SpreadsheetBench, Figure 4) could be beneficial for the benchmark's grading rubric but harmful in a production spreadsheet where static values break downstream formula dependencies. A rule like "prefer a justified stronger-result option over true but weaker corollaries" (LiveMath, Figure 4) could produce overconfident answers in domains where weaker but more certain results are preferable. The optimizer has no way to encode these deployment-context constraints, and the validation gate has no way to enforce them.

The paper's transfer experiments (Tables 4a–4c) provide partial evidence of robustness: skills trained on one model/harness/benchmark transfer positively to others. But "positive transfer" only means the transferred skill beats the target's no-skill baseline—it does not mean the transferred skill is optimal or safe for the target context. A rule that helps on average could still cause catastrophic failures on specific inputs, and the paper does not report worst-case performance or failure mode analysis for transferred skills.

What evidence exists in the paper. The qualitative examples in Figure 4 and the case studies in Section 4.5 show that learned rules are procedural and general rather than instance-specific. This is evidence of interpretability but not of robustness. The paper does not test for adversarial robustness, distribution-shift robustness, or safety properties. It does not report per-task error analysis showing whether SkillOpt introduces new failure modes (errors that the no-skill model didn't make) or only fixes existing ones. The transfer results are positive on average but the paper does not report whether the transferred skill causes regressions on any individual tasks or task categories.

Mitigation status. None. The paper's philosophy—"the deployed artifact is correspondingly compact rather than the union of every reflection"—implies that compactness and gating provide a form of implicit regularization against overfitting, but this is not validated. The paper does not recommend human review of the skill before deployment, does not suggest additional safety testing protocols, and does not discuss how a practitioner should evaluate whether a SkillOpt-produced skill is safe for their specific deployment context. The "auditability" claim (the skill is inspectable text) is true but insufficient: a human can read the skill, but reading doesn't guarantee they can predict its failure modes. This is a fundamental tension in the paper's framing: it argues for the reliability of controlled optimization while the optimization criterion (held-out accuracy) is only loosely correlated with real-world reliability.

The Experimental Scope Is Broad but Shallow: Single Skill Document, Single-Domain Optimization, No Multi-Skill or Continual-Learning Evaluation

The assumption or constraint. SkillOpt is designed and evaluated as a single-skill, single-domain optimizer. Each run produces exactly one best_skill.md for exactly one benchmark. The paper acknowledges this explicitly in Appendix B:

"SkillOpt intentionally optimizes a single portable skill rather than growing a large skill library or changing model weights. This design improves deployment simplicity, but a single skill may be insufficient for highly heterogeneous domains that require many disjoint procedures."

The paper contrasts this with prior work that builds skill libraries (Trace2Skill, SkillForge, EvoSkills) or performs continual skill discovery (AutoSkill). It positions this narrowing as a deliberate choice to isolate the effect of optimization controls. But this narrowing also means the paper provides no evidence about how SkillOpt would behave in deployment scenarios that require more than one skill.

The consequence. Many real-world agent deployments involve heterogeneous task distributions that no single compact skill document can cover well. A customer support agent might need distinct procedures for billing inquiries (check payment records, issue refunds), technical troubleshooting (gather system logs, run diagnostic commands), and account management (verify identity, update preferences). Optimizing a single skill for this mixture would produce either a bloated document with conflicting rules or a compromise skill that underperforms specialized variants. The paper's design—optimizing one skill per domain—does not address the skill-selection problem: when a new task arrives, which skill should be applied? The paper mentions "skill libraries" as future work in the Outlook section but provides no mechanism for routing tasks to skills or for ensuring that independently optimized skills don't conflict when deployed together.

Additionally, the paper provides no evidence about continual learning: what happens if the domain shifts after a skill is optimized and deployed? Does SkillOpt support incremental updates (re-running the loop with new training data and the old skill as initialization)? If so, does the validation gate prevent catastrophic forgetting of procedures that were useful for old tasks but aren't represented in the new data? These questions are central to the paper's deployment philosophy (a skill artifact that can be reused and updated), but they are completely untested.

What evidence exists in the paper. The transfer experiments (Section 4.3) demonstrate that a single optimized skill can be reused across related contexts (models, harnesses, benchmarks), but they do not test multi-skill scenarios or skill composition. The paper evaluates exactly six skills (one per benchmark) for each target model; there is no experiment where multiple skills are trained and deployed together, or where a skill is updated incrementally with new data. The optimizer-state design (cache, rejected-edit buffer, slow/meta update) is capable of supporting warm-start incremental optimization (starting from an existing skill rather than from scratch), but the paper never tests this capability.

Mitigation status. The paper acknowledges the single-skill limitation (Appendix B: "a single skill may be insufficient for highly heterogeneous domains") and mentions skill libraries and meta-skill reuse as future work (Section 5: "natural extensions include skill libraries that share infrastructure across domains, reuse of optimizer-side meta skills across benchmarks"). But it provides no roadmap or preliminary evidence for how these extensions would work. The limitation is fundamental to the paper's scope—it studies whether controlled optimization can produce a better single-domain skill, and the answer is yes—but a practitioner deploying agents in heterogeneous environments would need to solve the multi-skill problem themselves, with no guidance from this paper.

Statistical Reliability of Individual Comparisons Is Unknown, and the "52 of 52 Cells Best" Claim Overstates Precision

The assumption or constraint. The paper's headline empirical claim—"SkillOpt is best or tied-best on all 52 evaluated cells"—is based on point estimates from single-seed evaluations on held-out test splits. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any comparison. Test-set sizes vary across benchmarks (SearchQA, DocVQA likely have hundreds of questions; LiveMath and ALFWorld have smaller test pools), and the fixed split seed (42) means the reported numbers are single point estimates with unknown variance.

This is a methodological limitation that affects the strength of the paper's claims, not the behavior of the method itself. But it matters because the paper's primary rhetorical move—SkillOpt is universally better than all competitors across all settings—relies on the precision of individual cell comparisons. If many cells show SkillOpt ahead by margins smaller than the test-set noise level, the "52 of 52" claim may overstate the method's reliability advantage.

The consequence. A practitioner reading Table 1 sees SkillOpt ahead of the next-best baseline by margins ranging from large (SpreadsheetBench GPT–5.5 direct chat: 80.7 vs. 73.6 for GEPA, a +7.1 gap) to small (Qwen3.6–35B-A3B OfficeQA: 47.1 vs. 46.5 for LLM skill, a +0.6 gap; Claude Code harness SearchQA: 85.9 vs. 84.0 for EvoSkill, a +1.9 gap). The large margins are almost certainly real effects. The small margins may not be: with test sets of 100–500 questions, a 1–2 point difference could be noise from which specific questions happened to land in the test split. The paper provides no way to distinguish reliable from unreliable comparisons.

This is not just a reporting issue. The paper's claim that SkillOpt is "best on 52 of 52" invites readers to treat the 52 cells as independent observations of SkillOpt's superiority. But if, say, 10 of those 52 cells have skill-baseline gaps within ±2 points, and the true differences (with confidence intervals) could be zero or even negative for some of them, then the "52 of 52" claim overstates the evidence. The aggregate pattern across 52 cells is still strongly SkillOpt-favored, but the per-cell precision is uncharacterized.

What evidence exists in the paper. The hyperparameter sensitivity analyses (Table 2) provide indirect evidence of stability: across multiple hyperparameter settings, SkillOpt's scores vary within bands of ~1–2 points on SearchQA and ~2–5 points on SpreadsheetBench. This suggests that test-set noise from hyperparameter choice is moderate, but it does not estimate test-set noise from split variation. Figure 3 shows test-set scores across epoch checkpoints; the curves provide a rough visual sense of plateau regions, but no error bars. The paper does not report results from multiple random seeds, does not bootstrap confidence intervals from the test set, and does not apply statistical tests (e.g., McNemar's test for paired accuracy comparisons).

Mitigation status. The paper does not address this limitation. It reports single-seed results throughout without caveats about statistical reliability. The fixed-seed protocol is standard in ML benchmarking, but the "52 of 52" framing—which is the paper's central empirical claim—demands a higher standard of evidence than a typical systems paper provides. A partial mitigation would be to report, for each cell, the magnitude of the gap to the next-best baseline and whether that gap exceeds a reasonable noise estimate (e.g., 2 standard errors from a bootstrap). The paper does not do this. Practitioners should treat the aggregate pattern (SkillOpt is consistently among the best methods) as well-supported, and individual cell rankings as suggestive rather than definitive, particularly where margins are small.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a category shift in how the field conceptualizes agent skill improvement, moving it from artifact construction to controlled optimization. The shift is not incremental—it redefines what it means for a skill to "improve" and what mechanisms are necessary to achieve reliable improvement.

The pre-SkillOpt landscape treated skills as products of a process. You could write a skill by hand, generate it with a one-shot LLM call, distill it from trajectory lessons (Trace2Skill), evolve it through failure analysis (EvoSkill, SkillForge), or refine it through reflective feedback (GEPA, TextGrad). The quality of the resulting skill was understood as a function of the sophistication of the process that produced it—better reflection, more trajectories, more capable generation models. Each system operated on an implicit assumption: if the process for analyzing failures and proposing improvements is good enough, the resulting artifact will be good. There was no notion of a "training loss" separate from a "validation loss" in skill space, no concept of step size control, and no mechanism for learning from rejected proposals.

SkillOpt reframes the problem entirely. The skill document is not the output of a generation process—it is the external state of a frozen agent undergoing optimization, with the same structural requirements as weight-space training. This reframing is operational, not metaphorical. The paper maps each component of deep-learning optimization onto a text-space analogue and demonstrates that removing any of these analogues degrades performance (Table 3). The implication is not just that SkillOpt's specific mechanisms work—it is that text-space optimization of agent artifacts has genuine structural similarities to weight-space optimization, and that ignoring these similarities (as all prior work implicitly did) is why those systems could not reliably improve over their starting point.

This shift has three concrete consequences for how the field should think about agent adaptation:

First, it redirects research investment from reflection quality to optimization control. The prevailing narrative in the self-improvement literature—from Reflexion to GEPA—has been that better reflection (more capable models, more detailed trajectory analysis, more sophisticated failure categorization) is the path to better artifacts. SkillOpt's results challenge this directly. The paper's comparison with EvoSkill is the clearest evidence: EvoSkill already performs sophisticated failure analysis on SpreadsheetBench under Codex (lifting from 27.5 to 67.5), but SkillOpt adds another +17.5 points on top. The gap is not from better reflection—it is from validation gating, bounded edit budgets, rejected-edit buffers, and slow/meta updates, none of which EvoSkill has. The ablation results (Table 3) reinforce this: removing the rejected-edit buffer drops SpreadsheetBench by 4.6 points; removing the slow/meta update drops it by 22.5 points. These control mechanisms—not the quality of the reflection prompting—are what determine whether reflection translates into improvement or into drift.

The practical implication: a research group building a self-improving agent system should invest at least as much effort in the validation and gating infrastructure as in the reflection and analysis prompting. The paper's design provides a template—a held-out selection split, strict improvement gating, rejected-edit memory, separate fast and slow update channels—that can be retrofitted onto any system that edits textual artifacts from feedback. Systems that lack these controls are implicitly operating with unbounded learning rates, no validation, and no negative feedback—a training regime that would be recognized as pathological in weight-space optimization but has been the default in text-space optimization.

Second, it establishes that compact procedural artifacts can substitute for weight updates across a broad class of adaptation problems. The paper does not claim that skill optimization replaces fine-tuning in all cases—the prior sections explicitly note that this paper lacks a FLOPs-matched comparison against fine-tuning that would establish the boundary. But the empirical results narrow the domain where fine-tuning is the clear winner. A compact best_skill.md file of 300–2,000 tokens (Table 6), assembled from only 1–4 accepted edits, produces gains of +23.5 points on average across six benchmarks (Table 1), transfers positively across model scales and harnesses (Tables 4a–4b), and works across two model families (GPT and Qwen). These are not prompt-engineering gains—they are in the range that practitioners typically associate with fine-tuning—and they come without modifying model weights, without serving a new model artifact, and with a static deployment cost of zero additional inference-time calls.

This changes the economics of domain adaptation. Before this paper, the default mental model for adapting a model to a new domain was: (1) collect task data, (2) fine-tune, (3) serve the new model. SkillOpt introduces an alternative path: (1) collect task data with a held-out split, (2) run the optimization loop to produce a skill, (3) deploy the frozen model with the skill prepended. The training cost is paid once (0.6M–46.4M tokens per absolute test-set point, Table 6), but deployment requires no model serving infrastructure changes. For organizations using API-accessed models where fine-tuning is unavailable or expensive, this is a qualitatively new capability. For organizations serving many different domains, the cost structure shifts from serving N fine-tuned models to serving one frozen model with N compact skill files.

Third, it resolves a latent contradiction in the skill-evolution literature. Prior work on skill construction and evolution had produced a confusing picture. Some papers reported substantial gains from trajectory-driven skill distillation (Trace2Skill, SkillForge), while others found that self-evolution loops were brittle or regressed (the paper notes that TextGrad actually hurts SpreadsheetBench by −0.7 points and ALFWorld by −0.8 on GPT–5.5, Table 1). The field lacked a framework for understanding when reflection-based skill editing works and when it fails. SkillOpt's ablation results provide a diagnostic: reflection works when it is embedded in a controlled optimization loop with validation gating and negative feedback; it fails when it is unconditional self-editing. The paper resolves the contradiction not by taking sides but by showing that the missing ingredient in the failure cases is control mechanisms, not reflection quality. This converts an empirical puzzle into a design principle: any system that edits skills from feedback should gate its edits on held-out performance and maintain a record of rejected proposals.

The magnitude of this shift should be understood precisely, not overstated. SkillOpt does not introduce a new theoretical framework for text-space optimization—it does not characterize the optimization landscape, prove convergence properties, or establish generalization bounds. It is an empirical demonstration that training-style controls matter for text artifacts, backed by a comprehensive ablation suite. The shift is in what the field considers the object of study. Before SkillOpt, the object of study was the skill artifact and the process that produced it. After SkillOpt, the object of study is the optimization loop—its controls, its stability properties, its failure modes—and the skill artifact is the output of that loop. This reframing opens a design space (learning-rate schedules for text edits, validation-gating strategies, negative-feedback mechanisms, fast-slow adaptation channels) that prior work didn't recognize as a design space at all.

Follow-Up Research This Work Enables

Difficulty-conditioned optimization: identifying when a problem is too hard for skill editing to help. SkillOpt currently optimizes a single skill for all problems in a domain, with no mechanism to detect that some problems are fundamentally beyond the target model's capability and cannot be fixed by procedural rules. The reference paper on compute-optimal test-time scaling showed that this distinction is critical—routing compute away from impossible problems and toward correctable ones yielded 4× efficiency gains. A direct extension of SkillOpt would add a difficulty-estimation component: run a small number of rollouts (say, 4–8) on each training task under the current skill, estimate whether the failures are procedural (the model gets close but misses a rule) or fundamental (the model has no idea), and allocate the optimizer's edit budget preferentially toward procedural failures. A strong follow-up would implement this on a benchmark with explicit difficulty labels (e.g., the MATH dataset used in the reference paper, or SpreadsheetBench with problem complexity tiers) and measure whether difficulty-conditioned allocation improves the cost-per-point efficiency reported in Table 6. The null result—difficulty conditioning doesn't help because the validation gate already filters out edits that overfit to fundamentally hard problems—would be informative in a different way, suggesting that the gate alone provides sufficient implicit difficulty routing.

Multi-skill libraries with cross-skill interference management. SkillOpt intentionally optimizes a single skill per domain, but the paper explicitly flags skill libraries as a natural extension (Section 5, "Outlook"). The challenge is not just training multiple skills independently—it is managing interference between them. If a deployment requires a billing skill and a technical-troubleshooting skill, and both are prepended to the agent context, do their rules conflict? Does the agent know which skill to apply to which task? A strong follow-up would extend SkillOpt to train 3–5 skills for a heterogeneous task distribution (e.g., a customer support benchmark with distinct subdomains), with each skill optimized on its subdomain's training data, and then test routing strategies: (1) include all skills in context and let the agent choose, (2) train a lightweight skill-classifier that selects which skill to include based on task features, (3) optimize skills jointly with a constraint that edits to one skill cannot regress performance on other skills' held-out splits. The key measurement would be whether joint optimization with interference constraints outperforms independent optimization per subdomain, and whether the rejected-edit buffer from one skill's training provides useful negative signal for other skills' optimization (analogous to multi-task learning where gradient conflicts are managed).

Self-distillation of optimized skills back into model weights. The paper's deployment philosophy separates the optimizer (used during training) from the deployed artifact (a static skill file). But the optimized skill encodes procedural knowledge that was learned from task feedback—knowledge that could, in principle, be distilled into the model weights through fine-tuning on successful trajectories generated under the optimized skill. A concrete follow-up experiment: take the best_skill.md from a SkillOpt run (e.g., SpreadsheetBench, which achieved 80.7 with skill vs. 41.8 without), generate new training trajectories using the frozen model with the optimized skill, fine-tune the model on those trajectories (with the skill removed, so the model must internalize the procedures), and measure how much of the skill's gain transfers into weights. The paper's cross-model transfer results (Table 4a) provide suggestive evidence that the procedural knowledge is model-agnostic enough for distillation to work, but the experiment would test a stronger claim: that the skill is a stepping stone toward weight-level adaptation, with the optimization loop serving as a data-generation engine for fine-tuning. A negative result—fine-tuning fails to recover the skill's gains because the procedures require explicit instruction rather than being learnable from examples—would establish an important boundary on what kind of knowledge can be encoded in text vs. weights.

Validation-gate design space: strictness, granularity, and surrogate metrics. The paper's validation gate is binary and strict: a candidate skill is accepted only if it strictly improves average held-out accuracy. This is a specific choice in a larger design space. A systematic follow-up would examine alternative gate designs: (1) lenient gating (accept ties) to allow skill drift that might enable later improvements, (2) per-category gating (accept if the edit improves performance on the specific failure category it targets, even if overall accuracy is flat), (3) confidence-based gating (accept if improvement exceeds a bootstrap confidence bound, to avoid accepting noise-level gains), and (4) cost-aware gating (accept if the improvement per training token exceeds a threshold). The SpreadsheetBench ablation (Table 3) shows that removing the gate entirely (the "without lr" row in a different context) causes measurable degradation, but the paper doesn't explore what happens if the gate is slightly relaxed—for example, accepting edits that don't hurt but enable future edits. The key measurement would be whether a more sophisticated gate enables the optimizer to reach higher final performance by accepting intermediate edits that are individually neutral but collectively enabling, versus the strict gate getting stuck in local optima.

Cross-optimizer transfer and optimizer curriculum. The paper's optimizer-strength analysis (Table 5) shows that a target-matched optimizer recovers 56–74% of a strong optimizer's gain, but this is measured only on two benchmarks. A more ambitious follow-up would ask: can we train a skill with a strong optimizer, then transfer the optimization process itself to a weaker optimizer through meta-skill transfer? The paper's meta skill (Appendix C.2.8) is an optimizer-side artifact that records what editing strategies work. A cross-optimizer experiment would train on Benchmark A with GPT–5.5 (producing a rich meta skill), then use that meta skill to initialize the optimizer for Benchmark B with a weaker optimizer model (e.g., GPT–5.4-mini), and measure whether the transferred meta skill accelerates convergence or improves the final skill quality relative to training from scratch. This tests whether the optimization knowledge encoded in the meta skill is about the domain, about the target model, or about editing strategies in general. A positive result would enable a deployment pattern: pay for a strong optimizer once to build meta skills for a class of domains, then use cheap optimizers for new domains initialized with the relevant meta skill.

Failure mode analysis and adversarial stress-testing of learned skills. The paper's evaluation measures average-case improvement on held-out test splits, but provides no analysis of whether the optimized skills introduce new failure modes—cases where the no-skill model was correct but the skilled model becomes incorrect. This is a critical deployment concern: a skill that improves average accuracy by +10 points but silently breaks 2% of previously-correct cases might be unacceptable in high-stakes settings. A thorough follow-up would perform per-instance diff analysis on the test set: for each question, compare the no-skill prediction, the skilled prediction, and the ground truth, categorizing every change as a fix (wrong → right), a regression (right → wrong), a persistent error (wrong → wrong), or a persistent success (right → right). The paper's slow/meta update mechanism already performs this analysis on training tasks—extending it to the test set would characterize the risk profile of the deployed skill. A second stress test would evaluate the skill on out-of-distribution inputs: if the SpreadsheetBench skill was trained on benchmarks with English headers, how does it perform on spreadsheets with non-English headers or different column naming conventions? These experiments would establish whether the paper's transfer results (positive on average) hide edge-case regressions that a practitioner would need to know about.

Practical Applications and Downstream Use Cases

API-accessed model customization for enterprise document workflows. A company using GPT–5.5 via API for spreadsheet automation (budgeting, financial reporting, inventory tracking) currently relies on the model's zero-shot capabilities, which the paper shows achieve only 41.8% accuracy on SpreadsheetBench (Table 1, direct chat). The company cannot fine-tune GPT–5.5—it's a closed API model—and hand-writing a skill requires domain expertise and iterative testing. SkillOpt provides a practical alternative: collect a few hundred example spreadsheet tasks with ground-truth outputs (the equivalent of the training split), run the optimization loop offline (cost: approximately 21.4M training tokens for a +38.9 point gain, per Table 6), and deploy the resulting ~2,000-token best_skill.md by prepending it to the system prompt. The deployed skill encodes workbook-forensics rules (inspect structure before writing formulas, write evaluated static values, verify boundary rows) that would take a human expert days to discover and codify. Cross-harness transfer (Table 4b) means the same skill works whether the company's execution environment is Codex-based or Claude-based, and cross-model transfer (Table 4a) means the skill remains useful when the company upgrades to the next model version.

On-device agent improvement for resource-constrained deployments. The paper's small-model results are striking: GPT–5.4-nano achieves only 23.5% on SpreadsheetBench zero-shot, but with a SkillOpt-optimized skill, it reaches 42.5% (Table 1). On ALFWorld, the improvement is from 34.3% to 69.4%—doubling the success rate. Qwen3.5–4B on ALFWorld goes from 30.6% to 81.3% (a 2.7× relative improvement). These gains are achieved without modifying the model weights, meaning a small on-device model (running on a phone, IoT device, or edge server) can be adapted to a specific domain by shipping a compact skill file alongside it. The use case: a robotics company deploys Qwen3.5–4B on an edge device for warehouse pick-and-place tasks. The zero-shot model's 30.6% success rate on ALFWorld-like tasks is too low for production. Running SkillOpt offline on a few hundred training episodes (cost: 59.3M tokens, or $O(100)$ API calls at deployment-planning time) produces a ~1,300-token skill that encodes visited-location memory, progress locks, and loop breakers—procedural knowledge the small model didn't have in its weights. The deployed system now achieves 81.3%, competitive with much larger models, without any on-device fine-tuning or additional inference cost.

Self-improving evaluation pipelines for benchmark development. Organizations that maintain internal benchmarks (e.g., a legal-tech company with a contract-review QA benchmark, or a medical-coding company with a diagnosis-coding accuracy benchmark) need to periodically improve their agent's performance as the benchmark evolves. SkillOpt provides an automated pipeline: when new training examples are added to the benchmark, re-run the optimization loop starting from the previous best skill as initialization. The validation gate ensures that new edits don't regress on old held-out data (the selection split includes legacy examples), while the rejected-edit buffer prevents the optimizer from re-proposing edits that failed in previous iterations. The cost-per-point figures (Table 6) provide budget guidance: procedural benchmarks like SpreadsheetBench cost ~0.6M tokens per point and produce ~2,000-token skills from 4 edits; more complex benchmarks like DocVQA cost ~46M tokens per point but still produce compact ~1,000-token skills. The organization can budget for periodic re-optimization as a line item in their ML operations cost, with predictable returns on held-out performance.

Skill marketplace and auditability for regulated domains. The paper's compact artifacts (300–2,000 tokens, human-readable, assembled from 1–4 verifiable edits) are well-suited for domains where adaptation decisions must be auditable. In healthcare, a clinical coding agent might use a SkillOpt-produced skill that encodes rules like "always verify the diagnosis code against the encounter notes before submitting." Unlike a fine-tuned model—where the adaptation is distributed across billions of weights and cannot be inspected—the skill document can be reviewed by a compliance officer, edited if necessary, and version-controlled in a regulatory audit trail. Each edit in the skill corresponds to a specific optimization step with a recorded selection-split score change (edit_apply_report.json, Appendix C); a regulator could trace why a particular rule was added by examining the trajectory evidence that motivated it. The cross-model transfer results mean a skill validated on one model version can be certified and then deployed on the next version with expected (but not guaranteed, per the paper's uncharacterized regression rate) continued benefit. This application is speculative—the paper provides no healthcare-domain experiments—but the compactness and edit-economy results (Table 6) make the auditability argument structurally, even if the paper doesn't make it explicitly.

When to Prefer This Method

The paper's experimental design implicitly defines the conditions under which SkillOpt should be preferred over alternatives, though it does not state them as an explicit tradeoff matrix. Based on the evidence in the paper, the decision boundary can be characterized as follows:

Prefer SkillOpt over human-written or one-shot LLM skills when: the target domain has systematic failure modes that are not obvious from a high-level task description—procedural rules that the model lacks zero-shot, such as workbook-structure-first reasoning (SpreadsheetBench), evidence-to-answer binding (DocVQA), or search-frontier management (ALFWorld). The paper's evidence: human skills achieve 72.9 on SpreadsheetBench vs. SkillOpt's 80.7, and LLM skills achieve 51.7 on OfficeQA vs. SkillOpt's 72.1 (Table 1). The gap is largest where the needed procedures are non-obvious—exactly the setting where feedback-driven optimization adds value over static prior knowledge.
Prefer SkillOpt over prompt-optimization methods (TextGrad, GEPA) when: (1) the adaptation need is procedural rather than instructional—the model needs rules about how to execute, not just what to focus on—and (2) the optimization can be done offline once and the resulting artifact reused. The paper's evidence: GEPA achieves competitive results on some benchmarks (73.6 on SpreadsheetBench) but is inconsistent (43.2 on LiveMath, 85.8 on ALFWorld) and can regress (TextGrad: −0.7 on SpreadsheetBench, −0.8 on ALFWorld). SkillOpt's validation gate prevents these regressions and produces consistently positive gains across all 52 evaluated cells.
Prefer SkillOpt over skill-evolution methods (Trace2Skill, EvoSkill) when: the cost of deploying a harmful skill edit is high—in production systems where regression on previously-working tasks is unacceptable. The paper's evidence: EvoSkill achieves strong average gains on SpreadsheetBench under Codex (67.5) but SkillOpt adds +17.5 points (to 85.0) while also providing the rejected-edit buffer as a regression-prevention mechanism (removing the buffer drops SpreadsheetBench by 4.6 points, Table 3). The validation gate provides a guarantee—no edit is accepted unless it strictly improves held-out performance—that unconditional evolution methods cannot offer.
Prefer fine-tuning or weight-update methods over SkillOpt when: (1) the target model's zero-shot performance on the domain is near zero—if pass@1 is essentially zero, there are no correct trajectories to learn from and skill optimization cannot create capability from nothing (analogous to the reference paper's finding that test-time compute doesn't help on difficulty bin 5), (2) the domain requires learning new factual knowledge rather than new procedures—a skill document can tell the model how to verify a spreadsheet, but it cannot teach the model what the tax code is—or (3) the deployment environment cannot accommodate a skill document in the agent context (extremely tight context budgets, or execution harnesses that strip system prompts). The paper does not test these boundary conditions directly; they are inferred from the method's design (optimization over procedural text artifacts) and the nature of the benchmarks (all require procedural reasoning, none require learning new facts).
Prefer the target-matched optimizer configuration over the strong-optimizer default when: API costs for the frontier optimizer model are prohibitive, or when the deployment requires demonstrating that the skill was produced without access to a model stronger than the deployed agent. The paper's evidence: the target-matched optimizer recovers 56–74% of the strong-optimizer gain (Table 5), meaning the optimization loop adds most of the value even when the optimizer is no more capable than the target. The remaining 26–44% of the gain is the value of a stronger optimizer and can be treated as an optional budget-dependent upgrade.