From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills

🎯 Pitch

Model-generated agent skills backfire 25% of the time, and a strong model that excels at writing skills can paradoxically fail when using them—or vice versa—with no reliable signal from model scale or baseline ability. This paper maps the full skill lifecycle to diagnose when and why skills fail, then distills the findings into a utility-grounded meta-skill prompt that consistently boosts extraction quality and slashes negative transfer.

1. Executive Summary

This paper conducts a systematic, utility-grounded study of model-generated, domain-level agent skills across the full lifecycle—experience generation, skill extraction, and skill consumption—spanning five diverse agentic task domains (ALFWorld, SpreadsheetBench, SWE-bench-Verified, SEAL-0, BFCL-v4) and multiple extractor and target models. The work introduces a comprehensive evaluation framework that disentangles two named roles: Extraction Efficacy (how reliably a fixed extractor produces helpful skills across targets, measured by averaging the performance delta ∆ across all targets for that extractor) and Target Evolvability (how much a fixed target benefits from skills extracted by different extractors from its own experience, measured by averaging ∆ across all extractors for that target). The paper finds that model-generated skills are beneficial on average—improving downstream performance in 75% of evaluated extractor–target pairings—but exhibit non-trivial negative transfer in the remaining 25% of cases, establishing that neither extractor capability nor target baseline strength reliably predicts skill utility and that skill benefit is jointly shaped by extractor, target, and domain. Through stage-by-stage analysis, the authors further validate a utility-grounded rubric of three dimensions—Failure Mechanism Encoding, Actionable Specificity, and High-Risk Action Blacklist—and operationalize it as a meta-skill that consistently improves extraction quality across all evaluated cells (average +1.55 pp) and substantially reduces negative transfer, establishing that empirical findings from lifecycle analysis can be translated into a drop-in improvement to skill extraction itself only when the guiding criteria are grounded in observed utility rather than textual plausibility.

2. Context and Motivation

The Core Gap: No Systematic Lifecycle-Wide Study of Model-Generated Agent Skills

The central problem this paper addresses is deceptively straightforward: model-generated agent skills are proliferating rapidly, but no one has systematically studied whether they actually work across their entire lifecycle. The field has produced a growing number of extraction methods—Trace2Skill [9], CoEvoSkills [10], AutoRefine [4], PRAXIS [5], ProcMem [6], and others—each claiming to convert agent execution trajectories into reusable procedural artifacts. Yet, as Section 1 observes, "while extraction methods continue to proliferate, understanding remains limited, with no comprehensive study spanning the full skill lifecycle—experience generation, skill extraction, and skill consumption—to ask whether such skills actually work, when they work, and what makes them succeed or fail."

This gap is not merely an academic oversight. It means that practitioners building agent systems cannot answer basic questions about the skill artifacts they deploy: Should I invest in better experience collection or better extraction prompting? Will a skill extracted by a strong model help a weaker one? Does the composition of my trajectory pool—how many successes vs. failures—matter more than the extraction method itself? The absence of lifecycle-wide empirical answers forces the field to rely on intuition and heuristics, which the paper's findings show can be actively misleading (Section 5.2 demonstrates that human judges and LLM judges both fail to identify which skills actually improve downstream performance).

Why This Problem Matters

The paper motivates its investigation through several converging practical concerns that the reader should understand as distinct:

1. Domain-level skills are becoming infrastructure. As the abstract notes, language agents "increasingly improve by reusing skills—structured procedural artifacts distilled from past experience." This is not speculative. Section 1 explicitly references that "skills have become a standard component in several commercial agent platforms" [3]. When skills become infrastructure—loaded at inference time across heterogeneous models and deployments—the cost of deploying ineffective or actively harmful skills scales with usage. A skill that causes negative transfer (degrading performance relative to a no-skill baseline) does not just waste the extraction compute; it silently makes every downstream agent that loads it less capable. The paper's finding that 25% of extractor–target pairings exhibit negative transfer (Table 1) is therefore not just a research observation—it is a practical engineering risk for any production system relying on skill libraries.

2. Hand-crafting skills cannot scale. Section 1 makes the economic motivation explicit: "hand-crafting skills is labor-intensive and cannot keep pace with the rapidly expanding scope of agent capabilities and deployment." This creates an unavoidable tension. The demand for skills (covering more domains, more edge cases, more model–environment combinations) grows faster than human annotators can supply them, making model-generated skills a necessity rather than a luxury. But if model-generated skills are neither understood nor reliably evaluated, the field is essentially deploying unverified code into production agent stacks—with no systematic way to predict which skills will help versus hurt. The paper's extraction framework (Section 3.2) is explicitly designed to be "intentionally minimal" to isolate extractor capability from scaffolding tricks—precisely because real-world deployment cannot depend on pipeline engineering that works for unknown reasons.

3. The training–specialization asymmetry in multi-model ecosystems. The extractor and target need not be the same model. An organization might use a powerful (and expensive) model to batch-extract skills from execution logs, then deploy those skills across a fleet of smaller, cheaper inference models. Table 1 reveals the practical consequence of this asymmetry: a model can be "a strong extractor yet a weak consumer, or vice versa" (Section 1 abstract). For example, on SpreadsheetBench, Gemini-3.1-FL achieves the highest Extraction Efficacy (EE = +5.86) despite having modest baseline task strength (25.00%), while GPT-5.4—the strongest baseline performer (37.17%)—ranks last as an extractor (EE = +1.67). This non-monotonicity means that choosing an extractor is fundamentally a compatibility problem, not a model-strength ranking problem. Without systematic lifecycle analysis, practitioners cannot make principled extractor–target pairing decisions.

4. The negative transfer problem is non-trivial and domain-dependent. The paper makes clear that average gains mask substantial risk. ALFWorld exhibits negative transfer in 47% of pairings (Table 1), making it the most fragile domain. Yet SpreadsheetBench and SWE-bench-Verified show negative transfer in only 13% of pairings. This domain variance means that a skill-development pipeline that works well in one domain (e.g., software engineering) cannot be assumed to transfer to another (e.g., embodied planning). Section 5.1 later provides a partial mechanistic explanation: the optimal success–failure composition of the experience pool is domain-specific, with ALFWorld favoring failure-heavy pools while SpreadsheetBench favors success-heavy ones. But the key motivational point is that without lifecycle-wide evaluation, this domain-dependence would remain invisible, and practitioners would likely apply uniform extraction strategies that are optimal for one domain but harmful for another.

Where Prior Approaches Fall Short

The paper identifies a specific landscape of partially overlapping but ultimately insufficient prior work, which it categorizes in Section 2 ("Related Work") along two axes: extraction method proposals and benchmarking efforts.

Extraction methods proliferate without a shared evaluation foundation. Section 2 enumerates a growing body of trajectory-to-skill methods: prompt-based distillation (Trace2Skill [9], AutoRefine [4], PRAXIS [5], MemP [15]), optimization and RL-based refinement (ProcMem [6], CoEvoSkills [10], SkillRL [8], EvoSkill [7]), and self-evolving lifecycle agents (EvolveR [17]). The paper acknowledges their individual innovations but identifies a structural limitation: "Despite their differences, all of these approaches rely on trajectory-to-skill extraction as the foundational step that turns raw agent experience into reusable knowledge. While these works propose effective extraction methods, they each operate under their own setup, and do not provide a systematic understanding spanning the full experience–extraction–consumption lifecycle." Each paper demonstrates that its method works under its chosen conditions, but no one has asked whether the choice of extractor model, target model, domain, or experience pool composition matters more than the extraction algorithm itself. The paper implicitly argues that without such an analysis, progress in extraction methods risks being overfitting to narrow experimental conditions rather than producing genuinely generalizable improvements.

Existing benchmarks illuminate single stages but leave the lifecycle opaque. Section 2 describes a landscape of recent skill-focused benchmarks, each examining a slice of the problem:

SkillsBench [11] and SWE-Skills-Bench [12] and Skills-in-the-Wild [13] focus exclusively on the skill consumption stage: they measure whether providing a skill to a target model improves downstream performance. SkillsBench uses "task-seeded, human-authored skills," while SWE-Skills-Bench and Skills-in-the-Wild draw skills from "existing public skill repositories." Critically, all three "leave the skill extraction stage outside the loop." They answer "does this skill help this model?" but cannot answer "does the extraction method that produced this skill reliably produce helpful skills across targets and domains?" This limitation matters because in practice, practitioners must decide how to extract skills—merely knowing that a particular hand-authored or repository-sourced skill helps provides no guidance on whether their own automated extraction pipeline will produce skills of comparable quality.
SkillCraft [14] takes a step toward studying extraction but has scope limitations. It "studies composition and accumulation via an extraction-and-reuse protocol at test time," making it the most lifecycle-adjacent prior work. However, the paper notes two specific constraints: (1) "it restricts skills to executable function compositions," i.e., skills that can be expressed as compositions of atomic tool calls, which is a representational limitation; and (2) "the benchmark's tasks are designed and scaled to admit such compositions, making it unclear whether the paradigm generalizes to broader domains whose tasks are not designed around function-style reuse." In other words, SkillCraft may be measuring its own benchmark's built-in assumptions rather than the general properties of skill extraction.
AgentSkillOS [18] and SkillFlow [19] study skill management at scale—retrieval, orchestration, ecosystem-level organization—but do not address the extraction–consumption pipeline itself. They assume skills exist and focus on selection, sidestepping the question of whether the skills were well-extracted in the first place.

The paper's diagnosis is that this fragmented landscape creates a specific, blockable gap: "the field still lacks a systematic understanding of the full trajectory-to-skill lifecycle across the raw experience generation, skill extraction, and skill consumption stages." No prior work systematically varies extractors, targets, and domains within a unified evaluation framework; no prior work disentangles extraction-side and consumption-side contributions to skill utility; and no prior work closes the loop by using lifecycle analysis to directly improve extraction.

How This Paper Positions Itself

The paper's positioning can be understood through four deliberate choices that distinguish it from what came before:

1. Lifecycle completeness as a core design constraint, not an afterthought. The paper structures its entire investigation around the three-stage lifecycle defined in Section 3.1: experience generation (target model M executes training tasks to produce a trajectory pool), skill extraction (extractor E distills the pool into a skill set), and skill consumption (the same target M consumes the skill on held-out tasks). This is not just an organizational scheme—it is an experimental protocol that loops the extraction stage back into evaluation, so that the skill being tested always originates from the target's own behavior. Section 3.1 explicitly describes this as "deployment-realistic, extractor-assisted single-step evolution: skills are distilled from M's own interaction logs and fed back to the same model on held-out tasks, grounding the skill source in M's actual behavior and failure modes." This ensures that observed performance changes are attributable to the extraction process, not to properties of externally sourced skills that may have been authored under different distributions.

2. Unified, minimal extraction framework to isolate model effects. The paper's extraction framework (Section 3.2) deliberately strips away domain-specific heuristics, filtering rules, and optimization tricks, leaving "all abstraction decisions to the extractor model itself." It borrows only the high-level decomposition of Trace2Skill [9]—per-trajectory analysis followed by hierarchical consolidation—while removing "its sub-agent fleet, conflict resolution, and skill-deepening mechanisms." This is a strategic choice: "this minimal design ensures that performance differences are attributable to extractor capability rather than pipeline engineering." In contrast to prior methods papers that optimize their pipelines heavily, this paper's goal is attribution, not optimization—at least in the main experiments, before Section 6 introduces the meta-skill improvement as a demonstration that the analysis can feed back into better extraction.

3. Utility as the sole evaluation criterion, with domain-specific generalization. Rather than evaluating extracted skills on any intrinsic text-quality metric (completeness, fluency, readability), the paper defines skill utility exclusively through downstream performance: $\Delta(E, M, D) = \text{Perf}(M \mid S_{E,M,D}, Q_{\text{test}}^D) - \text{Perf}(M \mid Q_{\text{test}}^D)$ . This is a deliberate epistemological stance—the paper argues implicitly that skills exist to improve agent performance, and any text-level metric that fails to predict performance improvement is not just insufficient but potentially misleading (as Section 5.2 demonstrates by showing that an LLM judge operating on text alone performs worse than chance at identifying the higher-utility skill in a pair). By spanning five qualitatively different domains (embodied interaction, productivity software, software engineering, web search, tool calling), the paper further tests whether utility patterns generalize or are domain-specific—a breadth that no prior skill benchmark achieves.

4. Disentangling extraction and consumption as separable phenomena. The Extraction Efficacy (EE) and Target Evolvability (TE) metrics (Equations 4 and 5) are the paper's key analytical contribution for decomposing skill utility into its two contributing factors. EE isolates the extractor's contribution by averaging ∆ across all targets for a fixed extractor, answering "how reliably does this extractor convert diverse experience pools into helpful skills?" TE isolates the target's contribution by averaging ∆ across all extractors for a fixed target, answering "how much does this target benefit from having its own experience distilled back to it, regardless of who does the distilling?" This decomposition is novel and important because prior work implicitly conflates the two: a negative result could reflect a bad extractor, a bad target (incapable of benefiting from skills), or a bad extractor–target mismatch. By reporting both EE and TE in Table 1, the paper reveals that these are empirically distinct—EE for a given extractor can be high while TE for a given target is low, or vice versa—confirming that skill utility is "jointly shaped by extractor, target, and domain" rather than being reducible to a single quality dimension.

5. Closed-loop validation as proof of analytical value. The paper does not merely diagnose problems—it demonstrates that its lifecycle analysis yields actionable improvements. Section 6 shows that the validated utility-grounded rubric (Failure Mechanism Encoding, Actionable Specificity, High-Risk Action Blacklist), derived from the cross-matrix analysis, can be operationalized as a meta-skill that "consistently improves extraction quality across all evaluated cells (average +1.55 pp) and substantially reduces negative transfer." This closed-loop design—from diagnosis to intervention to measured improvement—distinguishes the paper from pure benchmark studies that stop at measurement, and positions it as providing "a principled, utility-grounded discipline" for skill extraction rather than merely describing the current state of affairs.

In summary, the paper positions itself not as yet another extraction method or yet another benchmark, but as the missing empirical foundation that the growing ecosystem of skill-based agent systems requires: a systematic, lifecycle-wide account of when and why model-generated skills actually work, backed by controlled experiments that span extractors, targets, and domains, and that produce actionable guidance for improving extraction in practice.

3. Technical Approach

3.1 Reader Orientation

This paper builds an evaluation and analysis framework—not a new extraction method—that systematically measures how model-generated domain-level skills affect downstream agent performance across the full lifecycle of experience generation, skill extraction, and skill consumption. The problem it solves is that the field lacks principled ways to predict whether a skill will help or hurt a target agent: existing benchmarks test skills in isolation without linking them back to the extraction process that created them, and textual quality heuristics (fluency, completeness, readability) are actively misleading about utility (Section 5.2). The "shape" of the solution is a controlled cross-factor experimental design that systematically varies three factors—extractor model, target model, and domain—while holding the extraction protocol fixed, then decomposes the resulting performance deltas into extractor-side and target-side contributions through two derived metrics (Extraction Efficacy and Target Evolvability), and finally validates a set of text-level criteria that actually predict utility by feeding them back into the extractor as a meta-skill.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components organized around a three-stage pipeline:

Experience Generation Engine — a target model M executes tasks from a training split in a specific domain D, producing a trajectory pool containing both successes and failures alongside their outcomes. This is the raw material for skill extraction.
Unified Extraction Framework — an extractor model E (which may differ from M) processes the trajectory pool through a two-stage pipeline: per-trajectory pattern analysis followed by hierarchical consolidation, producing a single domain-level skill. The framework is deliberately minimal to isolate extractor capability from pipeline engineering.
Skill Schema and Storage — the consolidated patterns are converted into a structured skill artifact conforming to the Agent Skills open standard, with fixed fields (name, description, body in Markdown, optional references and scripts). This is the portable output consumed downstream.
Injection and Evaluation Harness — the extracted skill is provided to the same target M (usually via system prompt inlining for single-skill cases or a progressive-disclosure tool protocol for multi-skill cases), and M is evaluated on a held-out test split. The performance delta Δ relative to the no-skill baseline is recorded.
Utility Decomposition Layer — the cross-matrix of Δ values (extractors × targets × domains) is aggregated into Extraction Efficacy EE(E, D) and Target Evolvability TE(M, D), which separately quantify how much of the observed utility is attributable to the extractor's ability to produce helpful skills versus the target's ability to benefit from skills distilled from its own experience.

Information flows strictly left-to-right through the three lifecycle stages: a target model generates trajectories → an extractor distills them into a skill → the same target consumes that skill on new tasks → the performance change is measured and decomposed. The cycle is repeated systematically across all combinations of five extractors, six targets, and five domains to populate the full Δ matrix in Table 1.

3.3 Roadmap for the Deep Dive

First, the formal lifecycle formulation (Section 3.1 of the paper), which defines the three stages, the notation for models and domains, and crucially specifies which model does what at each stage—this determines what each experiment controls for.
Second, the extraction framework (Section 3.2), because it is the shared infrastructure across all experiments: the per-trajectory analysis step, the hierarchical consolidation step, the skill representation schema, and the design choices that make the framework minimal yet functional.
Third, the evaluation metric and the derived EE/TE metrics (Section 3.3), since Δ is the central dependent variable throughout the paper and the EE/TE decomposition is the key analytical innovation for disentangling extractor-side and target-side effects.
Fourth, the cross-factor experimental design (Section 4 setup), which defines the domain space, model space, data splitting strategy, and evaluation protocol—this is where all the numbers and configurations live.
Fifth, the meta-skill intervention design (Section 6), which closes the loop from diagnosis to improvement: how the validated rubric dimensions were discovered, validated, and operationalized into a generation-time prior.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis and evaluation methodology paper whose core idea is that skill utility is jointly determined by the extractor, the target, and the domain, and that systematic lifecycle-wide experimentation can identify which textual properties of skills actually predict utility, enabling guided extraction that consistently outperforms unguided extraction across diverse settings.

The Skill Lifecycle Formulation: Who Does What, and When?

The paper formalizes the entire skill generation and consumption process as a three-stage protocol with clearly assigned roles. The key insight is that the extractor E and the target M need not be the same model, and systematically varying both reveals their separable contributions to skill utility.

Notation convention. M denotes a target model (which both generates experience and consumes skills), E denotes a (possibly different) extractor model, and D denotes a domain. The protocol is domain-specific: everything described below runs independently per domain.

Stage 1: Experience Generation. Target model M executes tasks from the training split $\mathcal{Q}_{\text{train}}^D$ of domain D. For each task instance, M produces a trajectory—a step-by-step execution trace—and the environment provides an outcome (success or failure, based on the domain's task metric). The result is an experience pool:

$T_{M,D} = \{(\text{task}_i, \text{trajectory}_i, \text{outcome}_i)\}$

where the index i runs over all tasks in the training split. This pool contains both successful and failed trajectories, reflecting M's actual behavior and failure modes in the domain.

Stage 2: Skill Extraction. Extractor E processes the experience pool $T_{M,D}$ through the unified extraction framework (detailed in the next subsection). The extractor's output is a skill set:

$S_{E,M,D} = \mathcal{E}(T_{M,D})$

where $\mathcal{E}$ denotes the extraction procedure. The subscript $_{E,M,D}$ is deliberately triple-indexed to capture the dependence on all three variables: different extractors produce different skills from the same experience pool, different targets produce different experience pools for the same extractor, and the optimal extraction strategy may be domain-dependent. The skill set consists of structured procedural knowledge under a fixed schema and budget constraint (described below).

Stage 3: Skill Consumption. The same target model M is provided with the skill set $S_{E,M,D}$ and evaluated on the held-out test split $\mathcal{Q}_{\text{test}}^D$ . The evaluation measures how well the extracted skills generalize to unseen tasks within the same domain. The critical design choice here is that M consumes skills extracted from its own experience—the trajectory pool was generated by M in Stage 1, and the skill is fed back to M in Stage 3. This creates a "deployment-realistic, extractor-assisted single-step evolution" loop where the only variable introduced between baseline and skill-augmented evaluation is the extraction process itself, controlled by E.

Why this protocol isolates factors of interest. Holding M fixed while varying E enables controlled comparison of extraction quality: any difference in downstream performance must be attributable to the skill text produced by different extractors. Holding E fixed while varying M tests how well a given extractor generalizes across different target behavior distributions. Running the full cross-product (all E × all M within domain D) populates the Δ matrix analyzed in Section 4. The protocol also takes an implicit stance: by feeding skills back into the same M that generated the experience, the evaluation measures self-improvement potential—can a model benefit from having its own past experience distilled back to it? This is a more conservative test than measuring skills transferred across models (which Section 5.3 studies separately), but it isolates extraction quality from distribution-shift effects.

The Unified Extraction Framework: Minimal by Design

The extraction framework is the shared infrastructure that every extractor E uses to convert a trajectory pool into a skill set. Its defining property is intentional minimalism: the paper explicitly states it has "no domain-specific heuristics, filtering rules, or optimization tricks, leaving all abstraction decisions to the extractor model itself" (Section 3.2). This is not a methodological shortcut—it is a deliberate design to ensure that "performance differences are attributable to extractor capability rather than pipeline engineering." A complex pipeline with many tuned hyperparameters would confound the comparison: is Gemini-3.1-FL a better extractor than GPT-5.4, or does the pipeline just happen to be better tuned for Gemini's output distribution? The minimal framework eliminates this confound.

Structural borrowing from Trace2Skill. The framework adopts only the high-level two-stage decomposition from Trace2Skill [9]—per-trajectory analysis followed by hierarchical consolidation—while removing "its sub-agent fleet, conflict resolution, and skill-deepening mechanisms" (Section 3.2). This is important context for the reader: Trace2Skill uses multiple parallel sub-agents for trajectory analysis and sophisticated conflict resolution during merging. By stripping these away, the paper's framework is essentially the skeleton of Trace2Skill—enough structure to make extraction possible across diverse domains, but no more. The choice to retain the two-stage structure rather than a single-pass extraction is motivated by scalability: processing all trajectories in a single prompt would exceed context windows for realistic experience pools, while processing them independently (Stage 1) and then merging incrementally (Stage 2) is parallelizable and context-bounded.

Per-trajectory analysis (Stage 1 of extraction). The extractor E processes each trajectory $\tau_i$ from the experience pool $T_{M,D}$ independently. For each trajectory, E is prompted to extract up to K behavioral patterns (where K is a configurable parameter; the default is 3, per Appendix B.5 Table 7). Formally:

$\mathcal{E}: \tau_i \mapsto \mathbf{u}_i = \{p_1, \ldots, p_k\}, \quad \mathbf{U} = \{\mathbf{u}_1, \ldots, \mathbf{u}_n\}$

where each $p_j$ is either a success pattern (if the trajectory succeeded) or a failure pattern (if it failed). These two categories are never mixed: a pattern set from a successful trajectory contains only success patterns; a pattern set from a failed trajectory contains only failure patterns. Each pattern is a named behavioral insight with a 2–4 sentence description.

The prompt template for this step (reproduced in Appendix B.3, Table 2) specifies four quality requirements that each pattern must satisfy: (i) high-level and domain-general (not tied to specific file names, identifiers, or error messages), (ii) maximally broad in coverage (applicable across a broad class of tasks, not just the one trajectory it came from), (iii) information-dense with a concrete description (not vague platitudes like "be careful"), and (iv) self-contained and understandable without referencing the original trajectory. The prompt also includes per-type guidance: for success trajectories, the extractor is asked "What did this agent do RIGHT that other agents facing similar tasks should also do?"; for failure trajectories, "What should an agent AVOID doing when facing similar tasks?"

A crucial detail: the prompt explicitly tells the extractor to "Focus on genuinely novel, reusable patterns from THIS trajectory; do NOT try to be exhaustive" and to return an empty list if no useful patterns are found. This prevents extractors from generating boilerplate patterns just to fill a quota, which would dilute the signal in downstream consolidation. The maximum patterns per trajectory (K = 3) forces selectivity—the extractor must prioritize the most reusable insights.

Hierarchical consolidation (Stage 2 of extraction). After all trajectories are independently analyzed, the system has a collection of pattern sets U of size n (one per trajectory). These are merged in a tree-structured reduction with configurable group size G (default: 10, per Appendix B.5 Table 7):

$\mathbf{U}^{(0)} = \mathbf{U}, \quad \mathbf{U}^{(\ell+1)} = \left\{ \text{Merge}_\mathcal{E}\left( \mathbf{u}_{G(j-1)+1}^{(\ell)}, \ldots, \mathbf{u}_{Gj}^{(\ell)} \right) \right\}_j, \quad \text{until } |\mathbf{U}^{(L)}| = 1$

At each level ℓ, the extractor E receives G pattern sets and produces a single consolidated pattern set. The merge prompt (Appendix B.3, Table 3) instructs E to: (1) deduplicate overlapping patterns by combining them into one stronger pattern, (2) generalize to raise the abstraction level ("a single well-generalised pattern is worth more than several narrow ones"), (3) preserve type boundaries (success and failure patterns stay separate), (4) drop vague or low-value patterns, and (5) prioritize the most important patterns when the combined set would exceed a de facto budget.

The tree structure with G = 10 means that a typical experience pool of, say, 100 trajectories would require two levels of merging: 100 pattern sets → 10 consolidated sets → 1 final set. If the pool size is not an exact multiple of G, the last group at each level is smaller. This logarithmic reduction (ceil(log_G(n)) levels) keeps the number of LLM calls manageable—each call processes at most G inputs, so the prompt length is bounded.

Skill synthesis (final step). Once a single consolidated pattern set exists, the extractor converts it into the skill set via structured tool-calling operations against a writable store. The synthesis prompt (Appendix B.3, Table 4) instructs E to: integrate both success and failure patterns into each skill (so each skill includes both "what to DO" and "what to AVOID"), organize related patterns into coherent skills around shared themes, and structure the body with recommended approaches, common pitfalls, decision criteria, and verification methods. The system supports creation, update, and deletion of skills with schema validation. After all skills are added via tool calls, E signals completion with a finish tool.

Hyperparameters and budget constraints (from Appendix B.5 Table 7). The default extraction configuration uses:

Maximum patterns per trajectory: K = 3
Merge group size: G = 10
Maximum skill characters: 3000
Maximum skills per extraction: 1 (main experiments use single-skill output)
Extractor temperature: 1.0

The single-skill constraint for main experiments is a deliberate simplification that aids interpretability: when there is exactly one domain-level skill, the performance change can be attributed cleanly to that single artifact rather than to skill selection or composition decisions. Multi-skill protocols are supported (Appendix B.4 describes the progressive disclosure tool protocol) but used only in auxiliary experiments.

Design choice: zero-shot, single-pass extraction. The extractor E is prompted once per trajectory analysis, once per merge call, and once for synthesis. There is no iterative refinement, no verification feedback loop, no rejection sampling. This contrasts with methods like CoEvoSkills [10] that use co-evolutionary verification—repeatedly generating, testing, and refining skills. The paper's choice makes extraction cheaper and faster, but also means that extractor quality is measured "raw"—the skill produced is the first attempt at synthesis, not an optimized artifact. If an extractor produces a weak skill, that is a genuine capability limitation rather than a failure of an insufficient optimization budget. This is appropriate for a study aimed at understanding extractor differences rather than maximizing absolute performance.

Skill representation schema. Each skill follows the Agent Skills open standard with fixed fields: name (lowercase-hyphen slug, ≤64 characters), description (1–2 sentences summarizing the class of problems and when to apply), body (Markdown with strategies, pitfalls, decision criteria, and verification methods), and optional references and scripts. The schema is enforced via tool-calling validation: if a skill exceeds the character limit, the system returns an error and E must shorten and retry. This enforces information density—the extractor cannot simply dump the entire trajectory pool into the skill body; it must curate.

The Evaluation Metric: Performance Delta as Utility Proxy

The paper evaluates skill utility exclusively through downstream task performance, defining the metric $\Delta$ as:

$\Delta(E, M, D) = \text{Perf}(M \mid S_{E,M,D}, \mathcal{Q}_{\text{test}}^D) - \text{Perf}(M \mid \mathcal{Q}_{\text{test}}^D)$

where $\text{Perf}(M \mid S, \mathcal{Q})$ is the domain-specific task metric achieved by target model M on question set $\mathcal{Q}$ when equipped with skill set $S$ , and $\text{Perf}(M \mid \mathcal{Q})$ is the baseline (no-skill) performance on the same question set. The operator $\Delta$ returns a signed real number, typically in percentage points. Positive values indicate improvement (the skill helped), negative values indicate negative transfer (the skill harmed performance).

What it computes: the marginal performance change caused by injecting a specific extracted skill into a specific target on a specific domain's held-out test split, relative to the same target without any skill. The skill is extracted from M's own experience on the training split, so $\Delta$ measures self-improvement potential contingent on the extraction quality.

Why this form: this metric is deliberately minimal and utilitarian. It does not evaluate the skill's text quality, coverage, conciseness, or any other intrinsic property. The paper's stance is that the sole purpose of a skill is to improve downstream performance—if a skill reads beautifully but causes the agent to perform worse, it is a bad skill by definition. This stance is validated by the Section 5.2 finding that textual plausibility (as judged by an LLM) is negatively correlated with $\Delta$ at large performance gaps. Using a text-intrinsic metric (BLEU against a reference skill, completeness score from a judge, etc.) would have obscured this finding and given the false impression that extraction is easier than it is.

Domain-specific performance metrics. The paper uses each domain's native evaluation metric rather than a unified scoring function:

ALFWorld: task success rate (binary: did the agent complete the specified household task?).
SpreadsheetBench: cell-level accuracy against ground-truth spreadsheet values (grader checks whether specified output cells contain correct values/formulas).
SWE-bench-Verified: patch correctness rate (binary: does the generated patch pass all test cases?).
SEAL-0: answer accuracy (does the agent's final answer match the ground truth?).
BFCL-v4 (multi-turn subset): tool-calling accuracy (was the correct function called with correct parameters?).

These are not directly comparable across domains—a +1 pp gain on ALFWorld is not the same "amount" of improvement as +1 pp on SWE-bench-Verified—but the cross-factor analysis is performed within-domain, so the EE and TE metrics are interpretable per domain.

Repeated runs and aggregation. Each evaluation (both baseline and skill-augmented) is run three independent times (Appendix B.2), and the average is reported. This accounts for non-determinism in model sampling (temperature is non-zero at inference time for the target models, though the extraction temperature is fixed at 1.0). The three-run averaging reduces noise from sampling variation, though the paper does not report confidence intervals—a limitation noted in the prior section.

Skill injection protocol. The extracted skill is provided to the target M at inference time through one of two mechanisms depending on skill library size (Appendix B.4):

Single-skill protocol (main experiments): since all main-experiment extractions produce exactly one domain-level skill (max_skills = 1), the skill body is inlined directly into M's system prompt using the template in Appendix B.3 Table 5. The template includes a disclaimer: "This skill is an optional aid, not a mandatory procedure. Use your own judgment." This prevents the skill from overriding M's base capabilities when the skill's guidance is inappropriate for a specific task—which is important because the skill is domain-level (aggregate guidance for all tasks in D), not task-specific.
Multi-skill protocol (auxiliary): when multiple skills exist, M accesses them through progressive disclosure: it first calls list_skills to see names and descriptions, then view_skill for full bodies, then read_skill_file for attachments. For domains without native function calling (SpreadsheetBench), these are implemented as fenced ```skill ...``` blocks analogous to Python code execution blocks.

Why system-prompt inlining for single skills: the single-skill protocol ensures that M always has access to the skill without needing to decide whether to retrieve it. This eliminates retrieval quality as a confound—if a skill fails to improve performance, it is because the skill's content was unhelpful or harmful, not because M failed to locate or load it. For the paper's goal of isolating extraction quality, this is the right choice. In production, retrieval quality would be an additional source of variance, but the paper is not studying retrieval systems.

The EE and TE Decomposition: Disentangling Extractor and Target Effects

The cross-factor experimental design produces a matrix of $\Delta$ values: for a fixed domain D, each cell (i, j) is $\Delta(E_i, M_j, D)$ , where $E_i$ is one extractor and $M_j$ is one target. The paper aggregates these cell-level values into two derived metrics that decompose skill utility into extractor-side and target-side contributions.

Extraction Efficacy (EE). For a fixed extractor E and domain D, EE averages $\Delta$ across all target models:

$\text{EE}(E, D) = \frac{1}{|\mathcal{M}|} \sum_{M \in \mathcal{M}} \Delta(E, M, D)$

where $\mathcal{M}$ is the set of all target models (six in the main experiments: GPT-5.4, GPT-5.4-mini, Gemini-3.1-Pro, Gemini-3.1-FL, Qwen3.5-35B, Qwen3.5-9B).

What it computes: the average downstream performance improvement (in percentage points) that extractor E's skills produce when consumed by a random target from the population $\mathcal{M}$ . It answers: "averaging across diverse consumers, how good are this extractor's skills?"

Why this form: EE isolates the extractor's contribution by marginalizing over target identity. If an extractor produces consistently helpful skills across most targets, EE will be positive and large even if one particular target experiences negative transfer. Conversely, if an extractor's skills only help a narrow subset of targets, EE will be lower. The averaging assumes equal weight per target, which treats each target as equally important—a reasonable default for a study aimed at characterizing general extractor capability, but a limitation if one wanted to weight targets by deployment frequency.

What EE reveals that cell-level Δ does not: Consider Gemini-3.1-FL on SpreadsheetBench (Table 1). The cell-level Δ values range from +1.50 (on Gemini-3.1-Pro) to +7.33 (on Qwen3.5-9B). EE = +5.86 tells us that, averaging over the idiosyncrasies of individual targets, Gemini-3.1-FL is the strongest extractor for this domain. This ranking would not be obvious from scanning individual cells, where Gemini-3.1-FL and GPT-5.4-mini might appear comparable on some targets but diverge sharply on others.

Target Evolvability (TE). For a fixed target M and domain D, TE averages $\Delta$ across all extractor models:

$\text{TE}(M, D) = \frac{1}{|\mathcal{E}|} \sum_{E \in \mathcal{E}} \Delta(E, M, D)$

where $\mathcal{E}$ is the set of all extractor models (five: GPT-5.4, GPT-5.4-mini, Gemini-3.1-Pro, Gemini-3.1-FL, Qwen3.5-35B; Qwen3.5-9B is excluded because preliminary experiments showed it "cannot reliably follow the structured extraction protocol").

What it computes: the average downstream performance improvement (in percentage points) that target M achieves when consuming skills extracted by a random extractor from the population $\mathcal{E}$ , where all skills are distilled from M's own experience. It answers: "regardless of who does the extraction, how much does this target benefit from having its own past behavior patterns fed back to it?"

Why this form: TE isolates the target's capacity to benefit from skills—its "evolvability" or "skill-absorption capability." If a target consistently improves under most extractors, TE will be positive. If a target is brittle—skills frequently hurt it—TE will be negative. The key insight is that TE and the target's baseline performance are not the same thing: Table 1 shows Gemini-3.1-Pro has TE = -0.15 on ALFWorld despite having the highest baseline (87.56%), while GPT-5.4 has TE = +4.93 with a lower baseline (68.66%). A high-performing model can be resistant to skill-based improvement, and a moderate-performing model can be highly evolvable.

Why both EE and TE are needed: skill utility is a joint function of extractor and target. A high-Δ cell could reflect an excellent extractor, a highly evolvable target, or a lucky compatibility. EE and TE decompose the matrix into main effects, making it possible to ask focused questions: "Is Gemini-3.1-FL genuinely a better extractor, or does it just happen to pair well with the targets we tested?" (EE answers this by averaging across targets). "Is GPT-5.4 genuinely more evolvable, or is it just benefiting from one particular extractor's high-quality skills?" (TE answers this by averaging across extractors). The paper uses both metrics in parallel because they answer different questions, and the cross-matrix in Table 1 makes both visible simultaneously.

Note on exclusion of Qwen3.5-9B as extractor. Qwen3.5-9B is included as a target (it has TE values in Table 1) but excluded from $\mathcal{E}$ . The paper states this is because preliminary experiments found it "cannot reliably follow the structured extraction protocol" (Section 4.1). This means E has only five members, so TE averages over five extractors for most targets. The exclusion is a practical necessity—an extractor that cannot produce valid outputs under the extraction schema cannot be fairly compared to those that can—but it means TE for smaller models (like Qwen3.5-9B itself) is computed from a set of extractors that are all larger and more capable, which may introduce a systematic bias.

Cross-Factor Experimental Design: Domain Space, Model Space, and Protocol

The Δ matrix in Table 1 is populated through a systematic cross factor design. The choices of domains, models, splits, and evaluation protocol are not arbitrary—each is motivated by the paper's goal of achieving broad coverage while maintaining controlled comparisons.

Domain selection: five qualitatively different agentic task families. The paper selects domains to span "qualitatively different forms of agent behavior" (Section 4.1), ensuring that findings are not artifacts of a single task type:

ALFWorld [20]: embodied household tasks in a text-based simulator. Agents must navigate rooms, manipulate objects (pick up, clean, heat, cool), and reason about physical preconditions. The challenge is spatial reasoning, exploration, and multi-step planning with partial observability.
SpreadsheetBench [21]: spreadsheet manipulation tasks requiring table inspection, formula reasoning, filtering, and value editing. Agents write Python code (typically using openpyxl) to modify Excel files. The challenge is understanding workbook structure, performing correct computations, and writing values (not formulas, since headless Excel engines do not evaluate formulas).
SWE-bench-Verified [22]: real-world software engineering tasks requiring codebase understanding, fault localization, and patch generation. Agents must identify bugs in large codebases and produce minimal patches that fix the issue while passing all tests. The challenge is scalability (large codebases) and precision (minimal correct patches).
SEAL-0 [23]: web-search question answering tasks requiring retrieval, evidence synthesis, and multi-hop reasoning. Agents must formulate search queries, extract relevant information from results, and synthesize answers. The challenge is information retrieval quality and reasoning over retrieved evidence.
BFCL-v4 [24] (multi-turn subset): tool-calling tasks requiring function selection, parameter extraction, type matching, and multi-turn tool use. Agents must correctly interpret function signatures, extract parameters from user queries, and handle multi-turn conversations where context accumulates. The challenge is API understanding and accurate parameter grounding.

Why this breadth: by spanning embodied interaction, productivity software, software engineering, web search, and tool calling, the paper tests whether skill extraction patterns are domain-specific or general. The results show they are partially domain-specific—ALFWorld has 47% negative transfer while SpreadsheetBench has only 13% (Table 1)—which validates the multi-domain design. A single-domain study would have missed this heterogeneity.

Model selection: spanning families and scales. The paper selects six models that "span different families and scales" (Section 4.1):

GPT family: GPT-5.4 and GPT-5.4-mini [25]
Gemini family: Gemini-3.1-Pro [26] and Gemini-3.1-Flash-Lite [27]
Qwen family: Qwen3.5-35B and Qwen3.5-9B [28]

All six serve as targets. Five serve as extractors (Qwen3.5-9B excluded). The inclusion of both large and small models within families (GPT-5.4 vs. GPT-5.4-mini, Gemini-3.1-Pro vs. Gemini-3.1-FL, Qwen3.5-35B vs. Qwen3.5-9B) enables testing whether extraction or consumption capability scales with model size within a family—a question the main results answer negatively (e.g., the smaller Gemini-3.1-FL achieves higher EE than Gemini-3.1-Pro on SpreadsheetBench).

Data splits: 1:1 partition per domain. For each domain D, task instances are split 1:1 into an experience-generation split $\mathcal{Q}_{\text{train}}^D$ and a held-out test split $\mathcal{Q}_{\text{test}}^D$ (Appendix B.2). When the benchmark provides an official training split (as MATH does, and as SWE-bench does via its dev/test distinction), $\mathcal{Q}_{\text{train}}^D$ is sampled from it at the same 1:1 proportion relative to $\mathcal{Q}_{\text{test}}^D$ . Otherwise, instances are partitioned uniformly at random with a fixed seed. The same splits are used across all (extractor, target) combinations within a domain—this eliminates split variation as a confound, ensuring that differences between cells in the same column or row of Table 1 are attributable to extractor/target differences, not to which tasks happened to appear in training or test.

Evaluation protocol. Each target M is evaluated on $\mathcal{Q}_{\text{test}}^D$ both without skills (baseline) and with skills (skill-augmented). Both evaluations use the same test split and the same target configuration (temperature, etc.). Skill-augmented evaluation uses the injection protocol described earlier. Each evaluation is run three times independently and the average Perf is used to compute Δ (Equation 3). The three-run averaging is applied to both baseline and skill-augmented conditions separately before subtraction—so the base column in Table 1 is itself an average over three runs.

API access and inference configuration (Appendix B.2). For GPT models, the reasoning effort parameter is set to "medium." For Gemini models, the thinking level is set to "medium." GPT models access the Azure OpenAI API; Gemini models use the official Google Gemini API. Open-source Qwen models (35B and 9B) are served locally with vLLM [29] on a single node with 8 NVIDIA B200 GPUs. This configuration ensures sufficient throughput for the inference scales required (the paper does not report exact token counts, but each extraction requires multiple LLM calls per trajectory plus merge calls, and evaluation requires running all test instances).

Experience pool collection (Appendix B.1). For each (target, domain) pair, the target runs on $\mathcal{Q}_{\text{train}}^D$ for multiple rounds—the paper does not specify the exact number of rounds, but the implication is that multiple attempts per task are collected to ensure each pool contains both successes and failures. Pool sizes vary across domains due to differences in training-split size and per-task cost. The paper explicitly notes that the experience pool includes both successful and failed trajectories, which is crucial because Section 5.1 later manipulates pool composition to study its effect on skill quality.

Meta-Skill Guided Extraction: Closing the Loop from Diagnosis to Intervention

Section 6 of the paper operationalizes the empirical findings from the lifecycle analysis into a concrete, drop-in improvement to skill extraction. This is a closed-loop design: the same cross-factor experiments that revealed the gap between textual plausibility and utility also provide the data (high-gap skill pairs) used to discover which textual dimensions actually predict utility. The resulting criteria are then fed back into the extractor as a generation-time prior, and the improvement is measured on the same evaluation framework.

Step 1: Collecting high-gap skill pairs. The paper constructs pairs of skills where the $\Delta$ gap is large (δ = |Δ_A - Δ_B| > 0.5 pp) and both skills share the same (target, domain) context—so they were extracted from the same experience pool but by different extractors. This yields 151 pairs (Section 5.2). These pairs are the raw material for discovering utility-predictive textual features: the higher-Δ skill in each pair is the "ground truth" better skill, and the goal is to identify which textual properties systematically distinguish the higher-Δ from the lower-Δ skill.

Step 2: Automated rubric discovery pipeline. GPT-5.4 (serving as an analyzer, not as the extractor being evaluated) processes each of the 17 highest-gap pairs to extract per-pair differences—specific textual properties along which the higher-Δ skill outperforms the lower one. These per-pair differences are then iteratively merged and consolidated into seven candidate dimensions, called the raw rubric (Appendix H.2, Table 13):

Failure Mechanism Encoding
Actionable Specificity
Environment/Tool Semantics
Strategy Switching Conditions
Boundary Condition Coverage
High-Risk Action Blacklist
Benchmark-Aligned Priorities

Each dimension is a named quality axis with a definition. The rubric is "raw" because it is the unfiltered output of the contrastive analysis pipeline—some dimensions may genuinely predict utility, others may reflect spurious correlations in the training pairs.

Step 3: Validating dimensions against utility. To determine which raw dimensions actually predict utility, the paper operationalizes each dimension as a pairwise evaluation criterion. For each of the 151 high-gap pairs, GPT-5.4 judges which skill is better along each dimension separately. The better-rate for a dimension is the proportion of pairs where the higher-Δ skill receives more favorable judgments on that dimension. A dimension with a better-rate of 50% is uninformative (no better than random); a dimension above 50% carries genuine predictive signal.

Three dimensions emerge with better-rates consistently above 63%:

Failure Mechanism Encoding (65.5%): identifies why agents fail, not just that they fail—connects observed errors to their root causes and encodes countermeasures.
Actionable Specificity (66.0%): provides step-level procedures referencing domain objects and tools, not abstract advice—concrete enough that an agent can directly execute the guidance.
High-Risk Action Blacklist (64.6%): explicitly forbids specific harmful action patterns—not just "avoid mistakes" but "never do X because Y."

The remaining four dimensions have better-rates near or below 50% and are discarded. The three surviving dimensions form the validated rubric.

Why this validation step matters: the discovery pipeline could have identified dimensions that distinguish skills in the training pairs but do not generalize to new pairs. The better-rate measurement on held-out pairs (the 151-pair set includes pairs not used in discovery, though the paper does not specify the exact split) ensures that the validated dimensions carry genuine, cross-pair predictive signal. The paper implicitly argues that this is the key difference from the plausibility rubric (Section 6, "naive starting point"): the plausibility rubric is what an LLM believes should matter; the validated rubric is what actually correlates with measured performance improvement.

Step 4: Operationalization as a meta-skill. The validated rubric (three dimensions with definitions) is converted into a compact meta-skill—a generation-time prior inserted into the extractor's system prompt. The meta-skill consists of quality principles derived from each dimension, phrased as instructions for what the extractor should produce (e.g., "encode why agents fail, not just that they fail" for Failure Mechanism Encoding, "provide step-level procedures referencing domain objects/tools" for Actionable Specificity). This meta-skill is appended to the per-trajectory analysis prompt and the skill synthesis prompt (Appendix B.3), leaving all other extraction hyperparameters unchanged.

Step 5: Controlled comparison. The meta-skill guided extraction is compared against two baselines:

Original (un-guided): the standard extraction prompt described in Section 3.2, with no additional quality guidance.
Plausibility rubric: the same extraction prompt augmented with the seven-dimension plausibility rubric (Appendix H.1, Table 12) that GPT-5.4 generated without access to skill pairs or utility data. This tests whether any rubric helps versus specifically a utility-grounded rubric.

All three conditions use the same extractor (GPT-5.4-mini, per Figure 5) and the same evaluation protocol. The comparison span three domains (ALFWorld, SpreadsheetBench, SWE-bench) and three targets per domain (GPT-5.4, Gemini-3.1-Pro, Qwen3.5-35B), yielding nine cells.

Results (Figure 5, detailed in Appendix G Table 11). The plausibility rubric harms average performance (-0.59 pp relative to original, hurting in 6 of 9 cells), while the validated rubric improves all nine cells (+1.55 pp average, with gains ranging from +0.38 to +3.69 pp). This is the paper's strongest evidence that utility-grounded criteria are necessary for improving extraction: giving the extractor more guidance is harmful if that guidance is based on what looks good rather than what actually works.

Why this is a "drop-in" improvement: the meta-skill is a text string appended to the extractor's prompt. It does not require modifying the extraction pipeline, adding verification steps, changing hyperparameters, or collecting additional data. Any extraction framework that uses prompted LLMs for trajectory-to-skill distillation can incorporate the meta-skill with a one-line prompt change. This makes the improvement immediately practical, which distinguishes the paper from studies that propose new architectures or training procedures that require reimplementation.

4. Key Insights and Innovations

Innovation 1: Skill Utility as a Joint Function of Extractor, Target, and Domain — Not a Property of Skills Alone

The paper's most fundamental conceptual contribution is the decomposition of skill utility into separable extractor-side and target-side effects, operationalized through the Extraction Efficacy (EE) and Target Evolvability (TE) metrics. Prior to this work, the implicit assumption across both extraction-method papers and consumption-focused benchmarks was that skill quality is predominantly a property of the skill artifact itself — a good skill helps, a bad skill doesn't, and better extraction methods produce better skills. This paper systematically falsifies that assumption by showing that the same skill can help one target while hurting another, that the best task-executing model can be the worst skill extractor, and that a model's baseline task strength does not predict how much it benefits from skills.

The key evidence is not a single number but a pattern visible across Table 1: consider SpreadsheetBench, where Gemini-3.1-FL achieves the highest Extraction Efficacy (EE = +5.86) despite having the second-lowest baseline performance (25.00%), while GPT-5.4 — the strongest baseline performer (37.17%) — ranks last among extractors (EE = +1.67). This is not a marginal inversion; it is a qualitative reversal of the ranking one would predict from model capability alone. Similarly, on ALFWorld, GPT-5.4 has TE = +4.93 (benefits strongly from skills) while Gemini-3.1-Pro has TE = -0.15 (skills provide no average benefit) despite Gemini-3.1-Pro's substantially higher baseline (87.56% vs. 68.66%). The notion that a model can simultaneously be the best at solving tasks and the worst at extracting reusable knowledge from its own solutions — or the best at solving tasks and the most resistant to skill-based improvement — is counterintuitive and has no precedent in prior skill literature.

This finding reframes the entire problem of skill extraction. Before this work, one could reasonably ask "which extraction method works best?" The paper shows this is an ill-posed question without specifying the target, the domain, and the interaction between them. The field must now ask "which extraction method works best for this target on this domain, given that target's specific failure modes and behavioral patterns?" This is a fundamentally harder but more realistic framing — it converts skill extraction from an absolute optimization problem into a compatibility problem, analogous to how cross-lingual transfer depends on language similarity or how fine-tuning effectiveness depends on pretraining–downstream distribution alignment.

The significance extends beyond the empirical finding to the diagnostic framework itself. EE and TE are not just aggregate statistics; they are conceptual tools that allow future researchers to decompose any skill utility result into its two contributing factors. If a new extraction method produces negative average Δ, EE tells you whether the method is genuinely weak or merely tested on low-TE targets. If a target consistently fails to benefit from skills, TE tells you to investigate what makes that target resistant — perhaps its failure modes are not amenable to procedural guidance, or perhaps its experience pool contains patterns that extractors cannot effectively generalize. This decomposition is a conceptual advance that the field lacked, comparable in spirit to the bias-variance decomposition in machine learning or the precision-recall decomposition in information retrieval: it separates a composite signal into components that admit fundamentally different explanations and interventions.

Innovation 2: The Null Result That Textual Plausibility Does Not Predict Utility — and Why It Matters

The paper's Section 5.2 finding is not just a negative result; it is a foundational critique of evaluation-by-inspection that challenges the dominant intuition in the skill-extraction literature. The experiment is simple: take pairs of skills extracted from the same experience pool for the same target, where one skill substantially outperforms the other (δ = |Δ_A - Δ_B| > 0.5 pp), and ask a GPT-5.4 judge — acting as a proxy for human evaluation — to identify the higher-utility skill based on the text alone. The judge performs no better than random chance (46.4% overall accuracy), and critically, its accuracy decreases as the performance gap widens: on pairs with δ ≥ 5 pp, the judge picks the higher-Δ skill only 15.8% of the time. In other words, the LLM judge is not merely noisy — it is systematically biased toward skills that read better but perform worse.

This null result has profound implications because the field's default evaluation practices are built on the opposite assumption. When researchers present extracted skills in papers, the evaluation almost always includes qualitative examples where the reader is invited to inspect the skill text and judge its quality. Implicitly, this assumes that a skill that looks comprehensive, well-structured, and domain-appropriate will actually improve agent performance. The paper demonstrates that this assumption is not just unverified — it is actively harmful, because the textual features that make a skill look good (generic procedural advice, broad coverage, fluent structure) are precisely the features that correlate with lower downstream utility, while the features that actually predict utility (concrete failure mechanisms with executable remedies, domain-specific anti-patterns, explicit forbidding of specific high-risk actions) are less fluent and less impressive on casual inspection.

The contrastive case in Table 14 (Appendix H.3) crystallizes this pattern. The higher-Δ skill (+14.7 pp) encodes three domain-specific failure mechanisms tied to the headless Excel execution environment — the formula injection fallacy, index-shifting errors during deletion, and dynamic addressing to avoid hardcoded coordinates — each paired with an executable countermeasure. The lower-Δ skill (+4.3 pp) offers generic process-level directives: "resolve the contract before coding," "edit minimally and safely," "round-trip validate the saved result." These are reasonable, well-organized, and broadly applicable — they are exactly the kind of advice that an experienced engineer would endorse — but they fail to prevent the specific error modes that dominate SpreadsheetBench failures, because they operate at too high a level of abstraction to guide concrete action in the headless-execution environment.

This reframes the skill extraction problem as a specificity-fluency tradeoff that prior work did not recognize. The extractor must choose between producing a skill that reads well (general, broadly applicable, fluently structured — but too abstract to prevent concrete failures) and one that actually works (domain-specific, operationally precise, tied to concrete failure mechanisms — but less impressive on inspection). The paper's validated rubric (Innovation 3 below) provides criteria for steering extractors toward the latter, but the conceptual contribution is the identification of the tradeoff itself. For practitioners, this means that skill evaluation cannot rely on human or LLM inspection — it must be grounded in measured downstream performance, which is the paper's core methodological argument.

Innovation 3: Meta-Skill Guided Extraction as Closed-Loop Utility-Grounded Improvement

While the paper is primarily an evaluation and analysis study rather than a new extraction method, its Section 6 contributions form an intellectual innovation distinct from the diagnostic framework: the demonstration that empirical findings from lifecycle analysis can be translated into a drop-in extraction improvement, but only when the guiding criteria are validated against measured utility rather than assumed from intuition. This is a meta-methodological contribution about how to use evaluation infrastructure to improve the thing being evaluated.

The key experiment is the three-way comparison in Figure 5 (detailed in Appendix G Table 11). The plausibility rubric — seven dimensions of "good skill writing" generated by GPT-5.4 without access to utility data — harms average performance by -0.59 pp relative to unguided extraction, reducing accuracy in 6 of 9 evaluated cells. The validated rubric — the three dimensions (Failure Mechanism Encoding, Actionable Specificity, High-Risk Action Blacklist) that empirical pairwise evaluation showed actually correlate with downstream utility — improves all nine cells, with an average gain of +1.55 pp and maximum gains of +3.69 pp on SpreadsheetBench. The direction of the effect is what makes this finding significant: giving the extractor more guidance is harmful if that guidance encodes the wrong objective, and the wrong objective is precisely what an LLM would generate if asked naively "what makes a good skill?" The plausibility rubric represents the field's implicit evaluation criteria; the validated rubric represents what actually matters. That the two are anti-correlated in their effects on extraction quality is a finding with broad implications: it suggests that many existing extraction methods may be optimizing for the wrong signal, and that improving extraction requires grounding quality criteria in measured outcomes rather than human or LLM intuition.

What makes this innovative beyond a simple ablation study is the closed-loop architecture: the same cross-matrix experiments that revealed the plausibility-utility gap (Section 5.2) provided the data (high-gap skill pairs) used to discover which textual properties actually predict utility (Section 6, rubric discovery pipeline), and the resulting rubric was then validated and fed back into the same extraction framework to produce measurable improvements (Figure 5). This is not merely a "we fixed a problem" finding; it is a demonstration that utility-grounded evaluation infrastructure can serve as an optimization signal. The paper argues implicitly that the field should move toward a regime where skill extraction methods are iteratively improved using the same cross-factor evaluation framework that diagnosed their weaknesses — a form of meta-learning where the evaluation benchmark is also the training signal for extraction quality criteria.

The "drop-in" nature of the improvement is also conceptually significant for a different reason: it demonstrates that what the extractor is told to optimize for matters more than the extraction pipeline architecture, at least within the range of simple two-stage frameworks. The meta-skill is a text string appended to the extractor's prompt — no pipeline changes, no hyperparameter tuning, no additional model calls. That this minimal intervention produces consistent gains (all nine cells improve, with gains up to 3.69 pp) while a seemingly reasonable alternative rubric actively harms performance suggests that the bottleneck in skill extraction is not algorithmic sophistication but objective specification: extractors are capable of producing better skills with their existing architecture, but they need to be told what "better" actually means in terms that are grounded in utility rather than fluency. This is a fundamentally different diagnosis of the extraction problem than prior work assumed, and it redirects research attention from pipeline engineering toward utility-grounded objective design.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses five diverse benchmarks, each split 1:1 into an experience-generation split Q_train and a held-out test split Q_test. ALFWorld [20] provides embodied household tasks; SpreadsheetBench [21] provides spreadsheet manipulation tasks requiring Python-based Excel editing; SWE-bench-Verified [22] provides real-world software engineering bug-fixing tasks; SEAL-0 [23] provides web-search question answering tasks requiring retrieval and multi-hop reasoning; and BFCL-v4 [24] (multi-turn subset) provides tool-calling tasks requiring function selection and parameter extraction. When an official training split exists, Q_train is sampled from it at the same 1:1 proportion relative to Q_test; otherwise instances are partitioned uniformly at random with a fixed seed. The same splits are used across all (extractor, target) combinations within a domain to eliminate split variation as a confound (Appendix B.2).
Base model(s). Six models spanning three families and multiple scales serve as targets: GPT-5.4 and GPT-5.4-mini [25] from OpenAI, Gemini-3.1-Pro [26] and Gemini-3.1-Flash-Lite [27] from Google DeepMind, and Qwen3.5-35B and Qwen3.5-9B [28] from the Qwen family. Five of these serve as extractors (Qwen3.5-9B is excluded because preliminary experiments showed it "cannot reliably follow the structured extraction protocol," Section 4.1). The selection spans both large and small models within each family, enabling within-family scale comparisons. API-accessed models (GPT and Gemini) use medium reasoning effort and thinking level respectively; Qwen models are served locally with vLLM on 8 NVIDIA B200 GPUs (Appendix B.2).
Metrics. The primary metric is the performance delta Δ(E, M, D) = Perf(M | S_{E,M,D}, Q_test) − Perf(M | Q_test), measured in percentage points using each domain's native task metric: binary task success rate for ALFWorld, cell-level accuracy for SpreadsheetBench, patch correctness rate for SWE-bench-Verified, answer accuracy for SEAL-0, and tool-calling accuracy for BFCL-v4. Positive Δ indicates skill-augmented performance exceeding the no-skill baseline; negative Δ indicates negative transfer. Each evaluation (both baseline and skill-augmented) is averaged over three independent runs. From the full cross-matrix of Δ values, two derived metrics are computed: Extraction Efficacy EE(E, D), averaging Δ across all targets for a fixed extractor, and Target Evolvability TE(M, D), averaging Δ across all extractors for a fixed target (Equations 4 and 5).
Baselines. The baseline for all comparisons is the target model's performance on Q_test without any skill augmentation (the "No-Skill" or "Base" column in Table 1). This is a within-model comparison: the same target M is evaluated on the identical test split, with the only difference being whether the extracted skill is injected via system prompt inlining (single-skill protocol) or progressive-disclosure tool calls (multi-skill protocol, Appendix B.4). The paper does not compare against human-authored skills, skills from public repositories, or skills extracted by alternative methods — the goal is to characterize model-generated skill utility from a fixed extraction framework, not to benchmark extraction methods against each other.
Generation budget / compute accounting. The extraction framework uses a fixed compute budget per extraction, controlled by hyperparameters rather than a direct FLOPs count: maximum 3 patterns per trajectory in the per-trajectory analysis phase, merge group size G = 10 in the hierarchical consolidation phase, maximum 3000 characters per skill, and exactly 1 skill per extraction (Table 7). Temperature is fixed at 1.0 for all extraction calls. Target model inference uses default sampling parameters (medium reasoning effort for GPT, medium thinking level for Gemini). The paper does not account for difficulty estimation cost (the experience pool collection in Stage 1), which involves running all Q_train tasks at least once per target — this cost is treated as a sunk cost rather than amortized across downstream evaluations. Appendix B.2 notes that open-source models run on a single node with 8 NVIDIA B200 GPUs, which "is sufficient to run all open-source extractors and targets used in the study at the inference scales reported."
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (as might be done in a hyperparameter tuning study), since the main experiments do not select among competing strategies — they measure a fixed extraction pipeline across all factor combinations. The statistical protocol consists of three independent evaluation runs per condition, with the average reported in Table 1, to account for non-determinism in model sampling. The format normalization experiment (Appendix C) uses a Friedman test (non-parametric repeated-measures ANOVA) to test whether format type produces consistent ranking differences, treating each task instance as a block and each format as a treatment. The pairwise skill evaluation (Section 5.2, Appendix E) uses majority voting over 9 independent judge calls per pair, with randomized presentation order to mitigate position bias. The paper does not report confidence intervals or standard errors for the Δ values in Table 1, which is a limitation for assessing the statistical reliability of individual cell-level comparisons.

Main Quantitative Results

Cross-Matrix Skill Utility: The Full Extractor–Target–Domain Landscape

The paper's central empirical result is the full Δ matrix in Table 1, which reports the performance change (in percentage points) for every (extractor, target, domain) triple where the extraction framework produced a valid skill. The table covers 5 domains × 5 extractors × 6 targets = 150 possible cells, though not every cell is populated (e.g., Qwen3.5-9B is not an extractor, and some extractor–target combinations may have produced unparseable outputs in preliminary runs).

Aggregate pattern: skills help on average but with substantial variance. Across all populated cells in Table 1, approximately 75% show positive Δ (skill helps) and 25% show negative Δ (skill hurts). The domain-level negative transfer rates vary sharply: SpreadsheetBench and SWE-bench-Verified show the lowest negative rates (13% each), while ALFWorld is the most fragile domain (47% negative). This domain heterogeneity is a headline finding: the same extraction framework applied to the same models produces fundamentally different risk profiles depending on what kind of task the agent is performing.

Headline quantitative comparisons across domains (Table 1):

ALFWorld (embodied interaction): Average TE ranges from +4.93 (GPT-5.4, highest evolvability) to −1.69 (Qwen3.5-9B, lowest). Average EE ranges from +1.62 (Gemini-3.1-Pro) to −0.66 (GPT-5.4). Only two targets (GPT-5.4, GPT-5.4-mini) have positive TE; four targets have negative TE. The domain is unusually fragile — extraction routinely produces skills that harm performance, particularly for weaker targets.
SpreadsheetBench (productivity software): Average TE ranges from +9.66 (GPT-5.4) to +1.46 (Qwen3.5-9B), with all six targets showing positive TE. Average EE ranges from +5.86 (Gemini-3.1-FL, the highest EE in the entire table across all domains) to +1.67 (GPT-5.4). All extractors show positive EE, and 87% of individual cells are positive. This is the domain where model-generated skills are most reliably beneficial.
SWE-bench-Verified (software engineering): Average TE ranges from +2.91 (GPT-5.4-mini) to +0.37 (Qwen3.5-9B), with five of six targets positive. Average EE ranges from +3.11 (Gemini-3.1-FL) to +0.91 (Qwen3.5-35B). The extraction benefits are modest but consistent — gains are smaller than SpreadsheetBench but negative transfer is also rare (13%).
SEAL-0 (web search): Average TE ranges from +5.32 (Gemini-3.1-FL) to −1.04 (Gemini-3.1-Pro). Average EE ranges from +4.85 (GPT-5.4-mini) to −0.12 (Gemini-3.1-Pro). The domain shows high variance: some targets benefit substantially (GPT-5.4: +4.38 TE; Qwen3.5-9B: +3.38 TE), while others are harmed (Gemini-3.1-Pro: −1.04 TE). Gemini-3.1-Pro is notable as the only extractor with negative EE on this domain.
BFCL-v4 (tool calling): Average TE ranges from +5.15 (GPT-5.4-mini) to +1.12 (Qwen3.5-9B), with all six targets positive. Average EE ranges from +4.25 (GPT-5.4-mini) to +1.84 (Qwen3.5-35B). This domain shares SpreadsheetBench's pattern of universally positive average effects, suggesting tool-calling tasks are particularly amenable to skill-based improvement.

The extractor–target independence finding. Within each domain, the ranking of extractors by EE does not follow the ranking of models by baseline task strength. On SpreadsheetBench (Table 1), the baseline ranking (strongest to weakest target) is: GPT-5.4 (37.17%), Gemini-3.1-Pro (35.83%), GPT-5.4-mini (29.33%), Gemini-3.1-FL (25.00%), Qwen3.5-35B (23.83%), Qwen3.5-9B (13.67%). The EE ranking (strongest to weakest extractor) is: Gemini-3.1-FL (+5.86), Gemini-3.1-Pro (+4.11), Qwen3.5-35B (+3.33), GPT-5.4-mini (+3.22), GPT-5.4 (+1.67). The Spearman rank correlation between baseline strength and EE is approximately −0.7 — the relationship is inverted. The strongest task executor (GPT-5.4) is the weakest extractor; the second-weakest task executor (Gemini-3.1-FL) is the strongest extractor.

Similarly, TE does not follow baseline strength. On ALFWorld, the highest-baseline target (Gemini-3.1-Pro, 87.56%) has TE = −0.15, while a moderate-baseline target (GPT-5.4, 68.66%) has TE = +4.93. On SEAL-0, Gemini-3.1-Pro again shows the highest baseline (55.97%) yet negative TE (−1.04), while the lowest-baseline target (Gemini-3.1-FL, 14.93%) achieves the highest TE (+5.32).

Experience Composition: Success Ratio vs. Skill Quality

Section 5.1 addresses the question: given that experience is the raw material for extraction, does the success/failure composition of the trajectory pool matter for downstream skill quality?

Setup. The extractor is fixed to GPT-5.4-mini. For each tested domain (SpreadsheetBench, SWE-bench-Verified, ALFWorld), five experience pools are constructed from the same source trajectories with success ratios of 100%, 75%, 50%, 25%, and 0%. Each pool undergoes the identical extraction pipeline, and the resulting skills are evaluated on three targets (not explicitly named in Section 5.1 but implied to be the same targets as the main experiments). The average Δ across targets is reported.

Headline result (Figure 2). "All-failure pools consistently perform worst, highlighting successful trajectories as the foundation of skill extraction." Across all three domains, the 0% success pool produces the lowest or near-lowest average Δ. The optimal success–failure ratio is domain-specific: SpreadsheetBench favors more successful trajectories (peak at 75–100% success), SWE-bench-Verified peaks at 75% success, and ALFWorld performs best with failure-heavy pools (peak at 25% success, with 50% also performing well). The paper interprets this domain specificity as reflecting "how each domain weighs positive procedural signals against negative constraint signals." ALFWorld's inverted preference is particularly striking: the domain where skills are most likely to cause negative transfer (47% rate in Table 1) is also the domain where extraction benefits most from seeing failures — suggesting that ALFWorld's action space contains many invalid operations whose avoidance is more learnable than the correct sequence of actions.

Quantitative magnitudes (read from Figure 2, approximate): On SpreadsheetBench, the 100%-success pool achieves roughly +3.5 pp average Δ, while the 0%-success pool achieves roughly +1.0 pp. On SWE-bench-Verified, 75%-success peaks at roughly +3.5 pp while 0%-success falls below +1.0 pp. On ALFWorld, 25%-success peaks at roughly +2.0 pp — the highest absolute gain for that domain — while 100%-success achieves roughly +0.5 pp. The inter-domain variation in peak height mirrors the Table 1 TE rankings: SpreadsheetBench and SWE-bench-Verified (domains with high and consistent TE) reach higher peak Δ values than ALFWorld (the most fragile domain), even under optimal pool composition.

Textual Plausibility vs. Utility: The Pairwise Evaluation Null Result

Section 5.2 directly tests whether skill utility can be predicted from surface textual features — a question with immediate practical implications for skill library curation (can we screen out bad skills by reading them?).

Setup. From the cross-matrix experiments, 151 skill pairs are constructed where both skills share the same (target, domain) context but were extracted by different extractors, and where the absolute Δ gap exceeds 0.5 pp (excluding near-ties where the ground-truth ranking is unreliable). GPT-5.4, acting as a judge, evaluates each pair by reading only the two skill texts (without access to Δ values, trajectories, or domain-specific performance data) and selecting which skill it deems higher-quality and likely to produce better downstream performance. Each pair is evaluated 9 independent times with randomized presentation order, and the majority vote determines the judge's choice. Accuracy is measured as the proportion of pairs where the judge selects the higher-Δ skill.

Headline result (Figure 3, Section 5.2). "Without any evaluation criteria, overall LLM selection accuracy is 46.4%, indistinguishable from random." The judge performs slightly below chance, but the more striking pattern is the accuracy breakdown by Δ gap (δ, measured in percentage points). The gray bars in Figure 3 show:

δ in [0.5, 2) pp: accuracy approximately 55% (slightly above chance for small gaps)
δ in [2, 3) pp: accuracy approximately 64%
δ in [3, 5) pp: accuracy approximately 74%
δ ≥ 5 pp: accuracy approximately 16%

The accuracy decreases as the ground-truth performance gap widens. On the pairs where the utility difference is largest and most consequential (δ ≥ 5 pp, where choosing the wrong skill costs at least 5 pp in downstream performance), the judge systematically favors the worse-performing skill. The paper interprets this as evidence that "the skill that reads better is often the one that performs worse" and that "textual plausibility has come apart from downstream skill utility."

Why this matters beyond the null result. The judge's performance on large-gap pairs (15.8% accuracy when δ ≥ 5 pp) is not random error — it is systematic anti-correlation. This means that an LLM acting as a skill curator (filtering a skill library by reading skill texts and selecting the "best" ones) would actively discard the highest-utility skills and retain worse-performing ones. The finding validates the paper's methodological choice to define utility exclusively through downstream performance rather than any text-intrinsic metric: any evaluation practice based on reading skills is not just insufficient, but actively harmful when performance gaps are large.

Format normalization control (Appendix C, Table 8). To rule out the possibility that skill utility is driven by superficial formatting differences (e.g., bullet points vs. prose), the paper rewrites the strongest extractor's skill on SpreadsheetBench into four canonical formats (ordered list, unordered list, checklist, prose) and re-evaluates each. A Friedman test finds no significant format effect on any target (all p > 0.34, σ-ratios below 1), while the same test applied to extractor identity (different extractors producing different skills) yields significant effects for 5/6 targets (p < 0.005, σ-ratios well above 1). The σ-ratio (σ_factor / σ_round, where σ_factor is standard deviation across factor levels and σ_round is standard deviation across evaluation rounds) quantifies effect size relative to run-to-run sampling noise: format σ-ratios range from 0.11 to 0.77 (noise-dominated), while extractor σ-ratios range from 0.83 to 4.53 (signal-dominated). This confirms that the variance observed in Table 1 is driven by "what a skill says, not how it looks."

Cross-Model Skill Transfer: Same Skill, Different Consumers

Section 5.3 isolates the consumption side by holding the skill text fixed and varying only the target consumer. This addresses a practical question: if an organization extracts a skill from one model's experience, can that skill benefit other models?

Setup. A single extractor (GPT-5.4-mini) is fixed. Two skills are selected from its main-experiment outputs on SpreadsheetBench: a "strong-pool skill" extracted from the strongest target's experience pool (GPT-5.4, baseline 37.17%) and a "weak-pool skill" extracted from the weakest target's experience pool (Qwen3.5-9B, baseline 13.67%). Both skills are injected into all six targets and evaluated on the SpreadsheetBench test split. Because the skill text is identical across targets, any performance differences reflect the targets' varying ability to consume the same procedural guidance.

Headline result (Figure 4, Section 5.3).

The strong-pool skill (extracted from the best-performing target's experience) improves every target: gains range from +1.8 pp (on Gemini-3.1-Pro) to +9.5 pp (on Qwen3.5-35B). Even the weakest baseline target (Qwen3.5-9B, 13.67% base) benefits (+3.2 pp). The average gain is approximately +3.6 pp.
The weak-pool skill (extracted from the worst-performing target's experience) shows clear negative transfer on some targets: GPT-5.4 experiences −2.0 pp, while Gemini-3.1-Pro experiences −1.5 pp. Other targets see modest gains: Qwen3.5-35B gains +3.3 pp, Qwen3.5-9B gains +3.2 pp. The average gain is approximately +1.0 pp, but with high variance and sign changes.

Two separable findings. First, with skill text held fixed, per-target gains differ sharply — the strong-pool skill ranges from +1.8 to +9.5 pp, a spread of 7.7 pp, confirming that skill consumption ability is a per-target property. Second, the experience pool from which a skill is extracted strongly affects transferability: the strong-pool skill (from high-quality experience) consistently helps all targets, while the weak-pool skill (from low-quality experience) helps some targets but actively harms others. This echoes the Section 5.1 finding that experience quality (success composition) drives skill quality. The interaction — a weak-pool skill harms GPT-5.4 (−2.0 pp) but helps Qwen3.5-9B (+3.2 pp) — further demonstrates that skill utility cannot be predicted from the skill text alone; the same text that hurts a strong baseline model can help a weak one, presumably because the weak model's default behavior has more room for improvement in directions the skill addresses, while the strong model's default behavior is already closer to optimal and the skill's guidance may conflict with strategies the strong model already employs.

Behavioral impact analysis (Appendix D). The paper examines agent trajectories on two contrasting targets (GPT-5.4, which improves substantially, and Qwen3.5-9B, which regresses under certain skills) to characterize how skill consumption changes behavior. Three axes are analyzed:

Decision-making: GPT-5.4 shifts from writing spreadsheet formulas (which fail in headless Excel engines) to computing values in Python and writing back scalar results — a strategy correction. Qwen3.5-9B shifts toward workbook-native workflows (openpyxl-based in-place editing) that improve structural fidelity but increase execution complexity, leading to more frequent errors.
Exploratory behavior: Both models increase early inspection of sheet structure, headers, anchors, and target areas before editing. GPT-5.4 uses this to ground its edits more reliably; Qwen3.5-9B sometimes uses it to over-explore, leading to longer reasoning chains without better final results.
Tool-use behavior: Neither model makes new explicit "skill calls" — the consumed skill is absorbed into the prompt and changes how existing tools are deployed. GPT-5.4's tool use becomes more grounded (bounded write-back, anchor-based addressing, post-write verification). Qwen3.5-9B shifts from pandas-style dataframe operations to openpyxl-based editing, preserving workbook structure but creating more failure opportunities when the complex workflow cannot be reliably executed.

The paper summarizes: "Skill consumption acts by reshaping the target's default policy, so consumption ability is itself a per-target property that bounds achievable gains."

Meta-Skill Guided Extraction: Closing the Loop

Section 6 tests whether the diagnostic findings can be converted into an actual extraction improvement. The full pipeline is described in the Technical Approach (Section 3.4), but the experimental results form a key quantitative finding.

Three-way comparison (Figure 5, detailed in Appendix G Table 11). Using GPT-5.4-mini as the extractor, skills are extracted on three domains (ALFWorld, SpreadsheetBench, SWE-bench) for three targets each (GPT-5.4, Gemini-3.1-Pro, Qwen3.5-35B), yielding nine cells. Three extraction conditions are compared:

Original (un-guided): the standard extraction prompt with no additional quality guidance.
Plausibility rubric (7-dim): the original prompt augmented with the seven-dimension plausibility rubric generated by GPT-5.4 without access to utility data.
Validated rubric (3-dim): the original prompt augmented with the three-dimension utility-validated rubric (Failure Mechanism Encoding, Actionable Specificity, High-Risk Action Blacklist).

Headline result (Table 11).

Plausibility rubric vs. original: average Δ change = −0.59 pp. Performance decreases in 6 of 9 cells, with the largest drop on SpreadsheetBench/GPT-5.4 (−6.00 pp). Only 3 cells show improvement, all modest (+1.16 to +2.16 pp).
Validated rubric vs. original: average Δ change = +1.55 pp. Performance improves in all 9 cells, with gains ranging from +0.38 pp (ALFWorld/Qwen3.5-35B) to +3.69 pp (SpreadsheetBench/Qwen3.5-35B). The largest gains concentrate on SpreadsheetBench (+2.33, +3.58, +3.69 pp across the three targets), where the validated rubric dimensions most directly address the domain's characteristic failure modes (formula injection, index shifting, hardcoded coordinates).

The interpretation hinges on the direction of the plausibility effect. The plausibility rubric is not neutral — it actively harms extraction, reducing average downstream performance below the un-guided baseline. This means that telling an extractor "make your skill clear, complete, concise, well-structured, well-formatted, neutral-toned, and general" (the seven plausibility dimensions, Table 12) produces skills that are less useful than what the extractor produces without such guidance. Since these plausibility dimensions are precisely what a human evaluator or naive LLM judge would endorse as "good skill writing," the finding constitutes a direct critique of intuition-driven skill quality criteria. The validated rubric, in contrast, is "uglier" — it asks for concrete failure mechanisms, step-level procedures with domain referents, and explicit forbidding of specific actions — but produces consistently better downstream outcomes.

Ablation Studies and Robustness Checks

Format normalization (Appendix C, Table 8): Skill utility is not driven by output format. The strongest extractor's SpreadsheetBench skill is rewritten into four formats (ordered list, unordered list, checklist, prose). A Friedman test finds no significant format effect on any target (all p > 0.34), with σ-ratios (σ_factor / σ_round) below 1, indicating the format effect is smaller than run-to-run sampling noise. In contrast, the same test applied to extractor identity yields significant effects for 5/6 targets (p < 0.005) with σ-ratios well above 1 (range 0.83 to 4.53). This confirms that Table 1's variance is driven by content differences across extractors, not by superficial formatting choices.

Alternative agentic harnesses (Appendix F, Table 10): The SpreadsheetBench results are not artifacts of the Python-script evaluation harness used in main experiments. Re-evaluation of a subset of conditions using Claude Code (CC Opus 4.6, CC Sonnet 4.6) and Codex (GPT-5.4, GPT-5.4-mini) as interactive tool-use harnesses shows the same overall pattern: skill injection yields modest positive gains on average (Δ = +0.4 pp), with substantial variance across targets. Stronger targets (CC Opus, Codex GPT-5.4) show positive transfer from GPT-5.4-extracted skills, while the weakest target (Codex GPT-5.4-mini) shows no benefit. The small sample size (4 targets × 3 extractors = 12 cells) limits statistical power, but the qualitative consistency — average positive but with variance and consumption-ability gradient — corroborates the main findings.

Rubric dimension validation (Appendix H.2, Table 13): The seven raw rubric dimensions discovered by the contrastive pipeline have substantially different better-rates (proportion of high-gap pairs where the higher-Δ skill is judged favorably). Three dimensions exceed 63%: Failure Mechanism Encoding (65.5%), Actionable Specificity (66.0%), High-Risk Action Blacklist (64.6%). Four dimensions are near chance: Strategy Switching Conditions (47.5%), Benchmark-Aligned Priorities (56.2%), Boundary Condition Coverage (63.0% — borderline), and Environment/Tool Semantics (63.2% — borderline). The paper selects the three consistently best dimensions for the validated rubric, implicitly applying a threshold (roughly 64%+). The better-rate measurement itself uses the same GPT-5.4 judge as the pairwise evaluation, which could introduce judge-specific biases — a dimension might appear predictive only because the judge is better at applying that dimension's definition than others, not because it genuinely correlates with utility. However, the downstream improvement from the validated rubric (all nine cells improve, Table 11) provides independent confirmation that these three dimensions capture genuine utility-relevant properties.

Pairwise judge accuracy improvement with rubric guidance (Figure 3, guided bars): The validated rubric is fed back into the pairwise judge as explicit evaluation criteria: the judge scores each candidate on the three dimensions and aggregates them into a single preference. Under rubric guidance, overall judge accuracy rises from 46.4% (unguided) to 73.8%. Critically, the improvement extends to the hardest pairs (δ ≥ 5 pp), where the unguided judge achieved only 15.8% accuracy (systematically anti-correlated) and the guided judge now picks the higher-Δ skill "the majority of the time" — though the exact guided accuracy on δ ≥ 5 pp pairs is not explicitly stated, the green bars in Figure 3 show it substantially exceeding 50%. This is a non-trivial validation: the rubric not only improves extraction (when used as a meta-skill) but also enables reliable post-hoc evaluation of skill quality from text alone, closing the gap between what looks good and what works.

Plausibility rubric as negative control: The plausibility rubric's harmful effect (−0.59 pp average, Table 11) serves as a negative control establishing that not all prompt augmentation improves extraction. The fact that the plausibility rubric harms performance while the validated rubric helps — even though both are text strings appended to the same extractor prompt — demonstrates that the content of the guidance matters, not merely the presence of additional instruction. This rules out the alternative explanation that any "be more careful" style prompt augmentation would produce the observed gains, and strengthens the causal attribution of improvement to the specific validated dimensions.

Single-skill vs. multi-skill protocol: The main experiments use single-skill output (max_skills = 1, per Table 7) to ensure clean attribution of performance changes to a single skill artifact. The paper supports multi-skill protocols (Appendix B.4 describes the progressive-disclosure tool protocol) but does not report a systematic comparison of single-skill vs. multi-skill utility. This is a missing ablation: it is possible that some domains would benefit from multiple fine-grained skills (organized by task subtype or failure mode) rather than a single domain-level skill, and the paper's finding that skills help in some domains but cause frequent negative transfer in others might partly reflect the constraint of consolidating all patterns into one artifact.

Extractor temperature sensitivity: The extraction temperature is fixed at 1.0 (Table 7) for all experiments. No ablation varies extraction temperature to test whether lower-temperature (more deterministic) extraction produces more consistent skills or reduces negative transfer. Given that the pairwise evaluation shows extractor variance is the dominant source of utility differences (Table 8), temperature could interact with extractor identity — some extractors might produce more reliable skills at lower temperature, while others might benefit from the diversity of higher temperature. This is an unexplored dimension.

Critical Assessment

Claim: Model-generated skills are beneficial on average but exhibit non-trivial negative transfer (25% of pairings).

Holds, with domain-specific boundary conditions. Table 1 directly supports this claim: 75% positive rate across populated cells. The 25% negative rate is domain-dependent, ranging from 13% (SpreadsheetBench, SWE-bench-Verified) to 47% (ALFWorld). The claim is therefore true on average across the five domains, but a practitioner working only in ALFWorld would face nearly coin-flip odds of negative transfer — a substantially worse risk profile than the aggregate 25% suggests. The paper is transparent about this domain heterogeneity (the numbers are visible in Table 1), but the headline framing of "25%" could mislead readers into assuming uniform risk across domains. A more precise statement would be: "negative transfer rates range from 13% to 47% depending on domain, with embodied interaction tasks being the most fragile."

Weakness: The "75% beneficial" figure is computed from the cells in Table 1, but each cell represents a single extraction–evaluation run (averaged over three repeated evaluations of the same extracted skill). This measures whether a particular extractor–target pairing produced a beneficial skill, not whether the extraction process is reliable. If the same extractor–target pair were run 10 times with different random seeds (producing different skills due to sampling at temperature 1.0), would the same cell be positive in 75% of runs? The paper does not measure within-cell reliability, so the "75% beneficial" rate conflates extractor–target compatibility with sampling noise. A study measuring extraction stability (same extractor, same target, same pool, multiple extraction runs) would clarify whether the negative cells reflect systematic incompatibility or unlucky sampling.

Claim: A model can be a strong extractor yet a weak consumer, or vice versa, with skill utility independent of model scale or baseline task strength.

Strongly supported, with the caveat that the evidence is correlational, not mechanistic. Table 1 provides multiple clear examples: Gemini-3.1-FL is a strong extractor on SpreadsheetBench (EE = +5.86, ranks 1st) but a weak consumer (TE = +3.90, ranks 4th out of 6). GPT-5.4 is a strong consumer on ALFWorld (TE = +4.93, ranks 1st) but a weak extractor on the same domain (EE = −0.66, ranks 5th). The Spearman correlation between baseline task strength and EE on SpreadsheetBench is approximately −0.7 — effectively inverted. These patterns are consistent and visible in the raw data.

What the experiments do not establish is why extraction efficacy and consumption ability dissociate from task performance. The paper offers plausible interpretations (extraction requires generalization/abstraction skills distinct from task execution; consumption requires the ability to absorb procedural guidance without disrupting existing effective strategies), but the behavioral analysis (Appendix D) is qualitative and based on two targets in one domain. A mechanistic account would require controlled experiments manipulating specific extractor or target properties — for example, testing whether extractors that are better at instruction-following produce better skills, or whether targets with higher self-consistency benefit more from skills. The current evidence establishes the pattern convincingly but leaves the mechanism as informed speculation.

Strength: The cross-family design (GPT, Gemini, Qwen) strengthens the independence claim. If the pattern held only within one model family, it could be attributed to family-specific quirks. The fact that Gemini-3.1-FL (a small Gemini model) out-extracts both GPT-5.4 (a large GPT model) and Gemini-3.1-Pro (a larger Gemini model) on SpreadsheetBench shows the effect transcends within-family scaling. Similarly, Qwen3.5-35B shows low TE on ALFWorld (−1.34) while GPT-5.4 shows high TE (+4.93), confirming the pattern is not a GPT-specific phenomenon.

Claim: Neither skill format nor textual plausibility predicts utility, and an LLM judge performs no better than chance at identifying higher-utility skills.

Supported for textual plausibility as operationalized; the format claim has a minor caveat.

The pairwise evaluation result (46.4% unguided judge accuracy, Figure 3) is clear and well-designed: randomized order, majority voting over 9 calls, exclusion of near-ties (δ < 0.5 pp). The inversion on large-gap pairs (δ ≥ 5 pp dropping to 15.8% accuracy) is particularly convincing because it rules out the interpretation that the judge is merely noisy — it is systematically biased in the wrong direction. This is a strong negative result.

The format result (Table 8) is slightly weaker than the text suggests. The Friedman test finds no significant format effect (p > 0.34), but this is a test of whether any format is consistently ranked higher than others — it is a test of format preference, not format impact. If all four formats produce nearly identical performance (as the σ-ratios below 1 suggest), this supports the claim that format does not matter. But the test cannot rule out the possibility that format interacts with extractor identity — perhaps GPT-5.4-extracted skills work better in prose while Gemini-3.1-FL-extracted skills work better as ordered lists. The format normalization experiment uses only the "strongest extractor's skill on SpreadsheetBench" (singular), which limits generalizability. A full test would normalize skills from multiple extractors across multiple domains.

Caveat on "textual plausibility": The judge is GPT-5.4, which is one specific model. The paper uses it as a "proxy for human evaluation" but does not include actual human judges. It is possible that human domain experts would identify utility-relevant features that GPT-5.4 misses, or that a different LLM judge (e.g., one specifically fine-tuned for skill evaluation) would perform better. The paper's claim is about "textual plausibility" as operationalized by an LLM judge without utility-grounded criteria — this is a reasonable and practically relevant operationalization (since many practitioners would use LLMs to screen skills), but it is not a claim about the fundamental impossibility of text-based skill evaluation. The guided judge's improved accuracy (73.8%, Figure 3) demonstrates that text can predict utility when the evaluator knows what to look for — the problem is not that text is uninformative, but that naive text evaluation attends to the wrong features.

Claim: Experience pool composition (success–failure ratio) substantially affects downstream skill quality, with domain-specific optimal ratios.

Supported with the caveat of limited target coverage.

Figure 2 shows clear domain-dependent curves: SpreadsheetBench peaks at high-success, ALFWorld peaks at failure-heavy, SWE-bench-Verified at intermediate. The "all-failure pools consistently perform worst" finding is robust across all three tested domains. This is strong evidence that pool composition matters and that the optimal composition is not universal.

Weaknesses: The experiment fixes the extractor to GPT-5.4-mini (a single extractor) and reports average Δ across three targets. This conflates two questions: does pool composition affect skill quality, and does the optimal composition depend on the extractor? It is possible that GPT-5.4-mini is particularly good at extracting value from failures (or particularly bad at it, depending on domain), and a different extractor would show a different optimal ratio. A full design would cross pool composition with extractor identity. Additionally, the three domains tested are a subset of the five main-experiment domains — SEAL-0 and BFCL-v4 are excluded, so we do not know whether their optimal compositions follow the SpreadsheetBench pattern (success-favoring) or diverge.

The domain-specific optimal ratio interpretation is somewhat underspecified. The paper attributes ALFWorld's failure-favoring curve to "failed attempts often reveal invalid actions and dead-end states, making failures surprisingly informative." This is a plausible post-hoc explanation, but it is not tested. An experiment that would strengthen this claim: ablate by removing different types of failures (e.g., only navigation failures vs. only manipulation failures) to test whether specific failure categories drive the effect, rather than failures in general. Without such an ablation, the finding is an observed correlation (ALFWorld is both more fragile and benefits from failure-heavy pools) rather than an established mechanism.

Claim: The validated rubric improves extraction in all evaluated cells (average +1.55 pp), and the plausibility rubric harms extraction (−0.59 pp average).

Supported with the caveat of limited scale (9 cells, single extractor).

Table 11 shows the validated rubric improving all nine cells and the plausibility rubric harming six of nine. The largest gains concentrate on SpreadsheetBench (+2.33 to +3.69 pp), which is the domain where the validated rubric dimensions most directly address the documented failure modes (see the contrastive case in Table 14). This domain-specificity of improvement is internally consistent with the rubric's origin (derived from cross-matrix pairs spanning all domains, but with dimensions validated by their better-rates, which may have been driven disproportionately by domains where the dimensions are most discriminative).

Weaknesses: The experiment uses a single extractor (GPT-5.4-mini). We do not know whether the validated rubric would help other extractors to the same degree, or whether it might harm some extractors (as the plausibility rubric harmed GPT-5.4-mini). The meta-skill is derived from the same cross-matrix data that revealed the plausibility-utility gap, which introduces a potential circularity: the dimensions were validated on the pairs from Table 1 experiments, and the improvement is measured on the same domains and targets (though different skills are extracted, since the meta-skill changes the extraction output). A fully independent test would involve domains or models not used in the rubric discovery pipeline.

The 9-cell design is small for a "drop-in improvement" claim. Three domains × three targets = nine evaluations, which is sufficient to demonstrate a consistent pattern (all cells improve) but insufficient to characterize the improvement's distribution (variance, tail cases, failure modes). A production practitioner would want to know: does the meta-skill ever backfire? What fraction of extractor–target–domain triples would regress? With only nine cells, the lower bound on the failure rate could be zero (consistent with the data) or could be, say, 10% (which nine cells might miss). A larger-scale evaluation varying more extractors and domains would strengthen the claim of universal benefit.

The plausibility rubric's harmful effect is a critical finding that deserves more analysis. Table 11 shows it hurts across most cells, but the paper does not analyze why the plausibility rubric produces worse skills. Does it cause extractors to produce vaguer, more generic skills (optimizing for "clarity" and "conciseness" at the expense of specificity)? Does it cause extractors to drop domain-specific failure patterns in favor of broadly applicable but non-actionable advice? Qualitative analysis of plausibility-rubric skills compared to original and validated-rubric skills would illuminate the mechanism and strengthen the paper's argument that the specific dimensions matter. The contrastive case in Table 14 already shows a lower-Δ skill (extracted without rubric guidance) that exhibits the plausibility-preferred features (generic, broad, well-structured), so the plausibility rubric likely amplifies an existing tendency rather than introducing a new failure mode — but this is inference, not demonstration.

What Would Have Strengthened the Paper

Extraction stability measurement: Run the same (extractor, target, domain, experience pool) combination multiple times with different random seeds to measure within-cell variance in Δ. This would distinguish true negative transfer (systematically harmful extractor–target combinations) from unlucky sampling (a single extraction run that happened to produce a bad skill, where re-extraction would likely succeed). If negative cells are stable under re-extraction, the risk is structural. If they are unstable, the solution is ensemble extraction (extract multiple skills and select via validation) rather than avoiding certain extractor–target pairs.

Direct comparison with a simpler extraction baseline: The paper uses a two-stage extraction framework (per-trajectory analysis + hierarchical consolidation). How much does this structure contribute, versus a single-pass "look at all trajectories and produce a skill" prompt (for pools small enough to fit in context)? If single-pass extraction produces comparable Δ with lower cost, the two-stage structure might be unnecessary complexity. If single-pass is worse, the two-stage structure is justified but the paper should quantify the benefit.

Skill content ablation: The validated rubric has three dimensions. An ablation that tests each dimension individually (extracting with only Failure Mechanism Encoding guidance, only Actionable Specificity guidance, only High-Risk Action Blacklist guidance, and all three pairwise combinations) would reveal whether one dimension dominates the improvement, whether the dimensions interact synergistically, or whether any single dimension is sufficient. The current design only compares all-three against plausibility-all-seven and un-guided, leaving the contribution of individual dimensions unknown.

Human evaluation: Including human domain experts as judges in the pairwise evaluation would establish whether the plausibility-utility gap is specifically an LLM-judge phenomenon or whether human experts also fail at text-based skill evaluation. This matters for the paper's practical recommendations: if humans can identify good skills when LLMs cannot, the advice to practitioners changes from "never evaluate skills by reading them" to "use humans, not LLMs, for skill screening."

Larger-scale rubric validation: Validate the rubric dimensions on a held-out set of domains (e.g., discover dimensions from SpreadsheetBench, ALFWorld, and SWE-bench pairs, then test whether they predict utility on SEAL-0 and BFCL-v4 pairs). This would establish that the rubric captures domain-general properties of good skills rather than domain-specific artifacts of the discovery set. The current design validates on the same set of domains used for discovery (the 151 pairs come from the full cross-matrix, which spans all five domains), so the generalizability to new domains is untested.

Summary of Assessment

The paper's empirical design is well-motivated and its central claims are supported by the reported data, but the strength of support varies across claims. The dissociation of extraction efficacy from task performance and the failure of unguided textual evaluation are the strongest findings — they are based on the full cross-matrix (Table 1) and the well-controlled pairwise experiment (Figure 3), respectively, and both exhibit effect sizes large enough to be practically meaningful. The meta-skill improvement is promising but measured on a small scale (9 cells, single extractor). The experience composition finding is clear but tested on a subset of domains and a single extractor. The domain-heterogeneity of negative transfer (13% to 47%) is well-documented but its causes remain largely attributed rather than demonstrated — the paper identifies what varies across domains (optimal success–failure ratio, sensitivity to skill format, behavioral impact patterns) but does not isolate why ALFWorld is more fragile than SpreadsheetBench. Across all claims, the dominant limitation is within-cell reliability: the paper reports one extraction per (E, M, D) cell (averaged over three evaluation runs of that single skill), and the stability of those cell values under re-extraction is unknown. This limits how strongly the paper can claim that specific extractor–target combinations are fundamentally incompatible versus merely unlucky in the sampled extraction run.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation (Experience Collection) Cost Is Unaccounted For

The assumption or constraint. The entire compute-optimal framework depends on the target model M first executing the full experience-generation split to produce the trajectory pool $T_{M,D}$ from which skills are distilled. This cost—running the target on all tasks in $\mathcal{Q}_{\text{train}}^D$ , potentially multiple rounds (Appendix B.1 notes "multiple rounds" of collection)—is treated as a sunk cost and is not amortized into the headline performance deltas. The paper acknowledges this indirectly in Appendix A: "the observed performance change can be attributed as cleanly as possible to the skill itself rather than to retrieval policies, agentic scaffolding, or other confounding components." But the experience collection cost is a confounding component of a different kind: it is the prerequisite cost that must be paid before any skill benefit can be realized.

The consequence. A practitioner evaluating whether to adopt skill extraction in their pipeline must compare the total cost (experience collection + extraction + skill-augmented inference) against the baseline cost (no-skill inference). The paper reports $\Delta$ as the marginal improvement per skill-augmented inference on the held-out test split, but this $\Delta$ is only meaningful in regimes where the upfront collection cost is amortizable over many downstream evaluations. If the downstream evaluation volume is low—say, a small batch of new tasks, or a one-off evaluation—the experience collection cost could exceed any inference-time savings, making the net effect negative even when $\Delta$ is positive. The paper never quantifies this breakeven point.

The cross-model skill transfer experiment (Section 5.3, Figure 4) partially illuminates this issue: a skill extracted from one target's experience pool (the "strong-pool skill" from GPT-5.4) can transfer to other targets and still produce positive gains. This means the collection cost could be amortized across multiple consumer models—each new target does not need to regenerate its own experience pool. However, the paper also shows that transferred skills can cause negative transfer (the "weak-pool skill" harms GPT-5.4 by −2.0 pp in the same experiment), so transferred skills are not universally safe. A practitioner would need to validate transferred skills on each new target, which requires running the target on the test split anyway—partially defeating the amortization benefit.

What evidence exists in the paper. The paper reports no cost analysis, no FLOPs accounting for experience generation vs. extraction vs. skill-augmented inference, and no breakeven analysis. Table 1 reports the raw $\Delta$ per cell without cost discounting. The Appendix B.1 description notes that "pool sizes vary across domains, reflecting differences in training-split size and per-task cost" but does not provide absolute numbers that would enable a reader to estimate collection cost. The alternative harness experiment (Appendix F, Table 10) confirms that skill utility persists across different execution environments, but does not address cost amortization.

Mitigation status. The paper does not attempt to address this limitation quantitatively. Appendix A frames future work as "scaling to richer agent harnesses" and "scaling to substantially larger skill libraries," which are orthogonal directions. The paper's design choice to treat collection cost as sunk is acknowledged but not mitigated; the reported $\Delta$ values are best-case marginal gains assuming free experience generation, and a production deployment would see net gains that are strictly lower (potentially negative for small evaluation volumes).

6.2 Skills Are Evaluated Without Retrieval, Selection, or Multi-Skill Orchestration—The Simplest Possible Consumption Regime

The assumption or constraint. The paper's main experiments use a deliberately minimal consumption protocol: exactly one domain-level skill (max_skills = 1, Appendix B.5 Table 7) is inlined directly into the target's system prompt (Appendix B.4, single-skill protocol). There is no retrieval step (the skill is always present), no selection among multiple candidate skills, no skill composition, and no decision about when to consult the skill versus rely on the target's base knowledge. Section 3.2 explicitly states this is by design: "this minimal design ensures that performance differences are attributable to extractor capability rather than pipeline engineering." The multi-skill protocol exists in Appendix B.4 but is never systematically compared against the single-skill protocol.

The consequence. In a production agent system, a skill library might contain dozens or hundreds of domain-specific skills. The agent would need to retrieve relevant skills for each task, decide which to apply, and possibly compose multiple skills when a task spans sub-problems addressed by different skills. Each of these steps—retrieval, selection, composition—introduces failure modes that the paper's protocol systematically avoids. A skill that is beneficial when always-present might become harmful when retrieved at the wrong time (inappropriate application to a task it was not designed for), or might be correctly retrieved but incorrectly composed with other skills (conflicting guidance). Conversely, a skill that appears neutral or harmful when always-present might be beneficial when selectively retrieved only for tasks where it is applicable—the single-skill protocol forces the skill to be present even on tasks where its guidance is irrelevant, potentially interfering with the target's default behavior.

The paper's behavioral analysis (Appendix D) provides indirect evidence of this risk: it observes that skill consumption "reshapes the target's default policy" rather than triggering explicit skill calls. This means the skill is always active as a behavioral prior, even on tasks where its guidance is misaligned with the correct strategy. Qwen3.5-9B's regression under the weak-pool skill in Section 5.3 (Figure 4) is attributed to the skill pushing the model toward "workbook-native workflows that gain structural fidelity at the cost of execution robustness"—a tradeoff that might be beneficial on some tasks but harmful on others, yet the single-skill protocol forces the same tradeoff universally.

What evidence exists in the paper. The paper provides no comparison of single-skill vs. multi-skill vs. retrieval-based consumption protocols. The multi-skill injection template (Table 6) describes the protocol but is not evaluated. The cross-model transfer experiment (Section 5.3, Figure 4) shows that the same skill can help some targets while hurting others, which implies that skill consumption is context-dependent—and the context in a real deployment includes task-level applicability, not just target identity. The format normalization experiment (Appendix C) shows that format does not matter, but this is orthogonal to the retrieval question.

Mitigation status. Appendix A explicitly acknowledges this as future work: "scaling to richer agent harnesses (for example, with retrieval, planning, or tool-use scaffolds)." The paper frames the current minimal protocol as a strength for interpretability rather than a limitation, which is a defensible position for a controlled study. However, a practitioner reading this paper for deployment guidance should understand that the reported $\Delta$ values are measured under unrealistically favorable consumption conditions. In a retrieval-based system, skill utility would almost certainly be lower on average and more variable, because retrieval errors, inappropriate skill application, and skill interference are all unmeasured.

6.3 The Validated Rubric Is Derived and Tested on Largely Overlapping Domain–Target Space

The assumption or constraint. The three-dimension validated rubric (Failure Mechanism Encoding, Actionable Specificity, High-Risk Action Blacklist) is derived from the contrastive analysis of high-gap skill pairs from the same cross-matrix experiments that populate Table 1. The discovery pipeline processes "17 high-gap skill pairs" (Appendix H.2), and the better-rate validation is performed on "the 151 high-gap pairs" (Section 5.2) drawn from the identical extractor–target–domain space. The meta-skill guided extraction is then evaluated on a subset of the same domains (ALFWorld, SpreadsheetBench, SWE-bench) and some of the same targets (GPT-5.4, Gemini-3.1-Pro, Qwen3.5-35B), using the same extractor (GPT-5.4-mini) that contributed skills to the discovery pairs. Section 6 does not claim an independent test set of domains or models.

The consequence. The rubric dimensions may be domain-specific or model-family-specific rather than general properties of good agent skills. If the high-gap pairs from which the rubric was derived were dominated by SpreadsheetBench and ALFWorld contrasts (these are the domains with the largest Δ variance in Table 1—SpreadsheetBench has Δ ranges up to ~15 pp, ALFWorld up to ~10 pp), the rubric dimensions might capture what distinguishes good skills in these domains but fail to transfer to domains with different task structures. For example, "Actionable Specificity" and "High-Risk Action Blacklist" map naturally onto SpreadsheetBench's well-defined failure modes (formula injection in headless Excel, index-shifting during deletion, hardcoded coordinate fragility), but might be less diagnostic in SEAL-0 (web search QA), where the primary failure modes are retrieval quality and evidence synthesis rather than environment-specific anti-patterns. Similarly, the rubric was discovered using skills extracted by GPT-5.4-mini and evaluated on targets from GPT, Gemini, and Qwen families, but the GPT-5.4-mini extractor might have consistent textual tendencies that make certain dimensions artificially discriminative in the discovery set.

What evidence exists in the paper. The paper's validation of the rubric dimensions is internal to the discovery space. Table 13 reports better-rates on the same 151-pair set used for discovery (or a subset thereof—the exact split between discovery and validation pairs is not specified). The meta-skill evaluation (Figure 5, Table 11) tests three of the five main-experiment domains and three of six targets, all within the same model families. SEAL-0 and BFCL-v4 are excluded from the meta-skill evaluation, so we have no evidence of whether the rubric generalizes to those domains. Qwen3.5-9B is excluded from both the discovery (it is not an extractor) and the meta-skill evaluation (it is not among the three tested targets).

Mitigation status. The paper does not claim cross-domain generalization and does not test it. A conservative interpretation is that the validated rubric is a domain-robust finding for the three tested domains (ALFWorld, SpreadsheetBench, SWE-bench) and the tested model families (GPT, Gemini, Qwen), but its applicability to new domains or substantially different model architectures is unverified. The paper frames the meta-skill as a "drop-in" improvement, which implies broad applicability, but the evidence base for that claim is limited to the nine evaluated cells. A practitioner deploying in a novel domain should validate the rubric dimensions on their own skill pairs before adopting the meta-skill—the paper provides the methodology (pairwise evaluation with better-rate measurement) for doing so, but the specific three-dimension rubric may not transfer.

6.4 Hard Problems (Low-Baseline Targets, Fragile Domains) Show Persistent Negative Transfer That the Meta-Skill Does Not Eliminate

The assumption or constraint. The paper's analysis reveals that some domains and targets are systematically resistant to skill-based improvement. ALFWorld exhibits 47% negative transfer in Table 1, making it the most fragile domain. Qwen3.5-9B shows negative TE on ALFWorld (−1.69) and the lowest TE on SWE-bench-Verified (+0.37), indicating it consistently fails to benefit from skills. The experience composition experiment (Section 5.1, Figure 2) shows that even under optimal pool composition, ALFWorld's peak Δ is lower than SpreadsheetBench's or SWE-bench-Verified's. The meta-skill guided extraction (Section 6, Table 11) improves all nine evaluated cells, but the improvement on ALFWorld is modest (+0.38 to +1.00 pp) compared to SpreadsheetBench (+2.33 to +3.69 pp). The paper acknowledges this obliquely in the abstract: "model-generated skills are beneficial on average but exhibit non-trivial negative transfer."

The consequence. The meta-skill intervention improves skill quality in relative terms (all cells positive), but does not change the absolute ceiling that some domains and targets impose. On ALFWorld with Qwen3.5-35B, the validated-rubric skill achieves only +0.38 pp over the original skill (Table 11), and the original skill itself was extracted from a domain where 47% of pairings showed negative transfer. A practitioner working primarily in embodied interaction domains (robotics, game agents, physical reasoning) or deploying skills to weaker models (Qwen3.5-9B, Gemini-3.1-FL in fragile domains) cannot rely on the validated rubric to convert negative-transfer targets into positive-transfer ones—the rubric shifts the distribution of $\Delta$ upward, but the lower tail (negative transfer) may persist for fundamentally incompatible target–domain pairs.

The paper's own evidence suggests that some targets simply lack the consumption ability to benefit from skills. Section 5.3 shows that the same skill helps Qwen3.5-35B by +3.3 pp but hurts GPT-5.4 by −2.0 pp (weak-pool skill in Figure 4). This is not a skill quality problem—it is a compatibility problem where the target's default policy is already effective and the skill's guidance interferes. The meta-skill makes skills more specific and action-oriented (the three validated dimensions), but more specific guidance might actually increase the interference risk for targets that already have effective strategies, because the skill now competes more explicitly with the target's default behavior. The paper does not test whether the validated rubric reduces or increases the variance of $\Delta$ across targets—it only reports average improvement.

What evidence exists in the paper. The vulnerability is visible in the cross-matrix (Table 1), the cross-model transfer (Figure 4), and the meta-skill evaluation (Table 11). ALFWorld remains the most fragile domain even with validated-rubric extraction; Qwen3.5-35B's TE on ALFWorld is negative (−1.34, Table 1) and the validated-rubric gain for that target on that domain is the smallest observed (+0.38 pp, Table 11). The paper's behavioral analysis (Appendix D) attributes Qwen3.5-9B's regression to "workbook-native workflows that gain structural fidelity at the cost of execution robustness"—a target-specific limitation that no extraction improvement can fix without changing the target model itself.

Mitigation status. The paper does not propose a solution for fundamentally incompatible target–domain pairs. The meta-skill is an extraction-side intervention; it cannot fix consumption-side limitations. The paper's recommendation is implicit: use the cross-matrix evaluation framework (Table 1) to identify which targets and domains are amenable to skill-based improvement, and avoid skill extraction for combinations that show persistent negative transfer. This is a diagnostic recommendation, not a fix. The paper provides the diagnostic tools (EE, TE, cross-matrix analysis) but not the remediation for targets that score low on TE. Practitioners facing high-negative-transfer regimes have two unappealing options: (1) extract skills anyway and accept the risk, or (2) forgo skill extraction for those target–domain combinations. The paper's contribution is making this choice informed rather than blind—but it does not eliminate the choice itself.

6.5 The Extraction Framework's Compute Cost Is Opaque and May Rival Inference Cost for Small Evaluation Volumes

The assumption or constraint. The extraction pipeline requires multiple LLM calls per trajectory plus a tree-structured merge process. The per-trajectory analysis step (Section 3.2) makes one call per trajectory in the experience pool, extracting up to K = 3 patterns each. For a pool of n trajectories, this is n calls. The hierarchical consolidation step (Equation 2) makes ceil(log_G(n)) merge calls, where G = 10. For n = 100 trajectories, this is n + 2 = 102 extractor calls, each potentially processing thousands of tokens of trajectory text plus prompt overhead. The skill synthesis step adds one more call. For comparison, evaluating the same target on the test split requires running the target once per test task—typically a single inference pass per task, with no multi-stage pipeline. The paper never quantifies extraction cost in tokens, FLOPs, or wall-clock time relative to the inference cost that the skill is meant to improve.

The consequence. The headline finding—skills improve downstream performance by positive $\Delta$ in 75% of pairings—is measured in accuracy percentage points, not cost-adjusted utility. A +1.55 pp gain from the validated-rubric meta-skill (Table 11) might be eroded or reversed if the extraction cost is large relative to the inference volume. For a production system evaluating 1,000 tasks per day, the extraction cost (a one-time upfront cost amortized over many evaluations) might be negligible per evaluation. For a researcher evaluating a single 100-task benchmark, the extraction cost might exceed the inference-time savings, making skill extraction net-negative even if the per-task accuracy improves. The paper's framing implicitly assumes the high-volume amortization regime, but the breakeven point is never computed.

Additionally, the extraction cost varies with pool size, which varies across domains (Appendix B.1 notes that "pool sizes vary across domains"). A domain with a large training split (many trajectories to process) incurs higher extraction cost, potentially changing the cost–benefit calculus relative to a domain with a small training split. The paper's domain-level comparison of $\Delta$ values (Table 1) does not weight by extraction cost, so a practitioner comparing SpreadsheetBench (+3.58 pp validated-rubric gain, potentially high extraction cost due to large pool) against BFCL-v4 (+1.84 pp average EE, potentially lower extraction cost) cannot determine which domain yields better net utility per unit of total compute.

What evidence exists in the paper. The paper provides hyperparameter values that bound extraction cost (K = 3 patterns per trajectory, G = 10 group size, max_skill_chars = 3000, Appendix B.5 Table 7) but not absolute token counts or inference times. Appendix B.6 states that Qwen models run "on a single node equipped with 8 NVIDIA B200 GPUs, which is sufficient to run all open-source extractors and targets used in the study at the inference scales reported," but "sufficient" is not equivalent to "cheap" or "quantified." The paper does not report total extraction FLOPs, API costs for commercial models, or extraction latency.

Mitigation status. This limitation is unaddressed. The paper's design choice to treat extraction as a sunk cost (parallel to the experience collection cost in Limitation 6.1) is consistent with its goal of measuring skill utility conditional on extraction having already occurred, but it leaves a gap for practitioners who need to decide whether to invest in extraction at all. The meta-skill guided extraction (Section 6) adds no additional LLM calls—the meta-skill is a prompt modification—so it does not change the extraction cost relative to the un-guided baseline. This means the meta-skill improves quality per unit extraction cost, which is a genuine efficiency gain, but the absolute cost remains uncharacterized.

6.6 The Study Is Limited to a Single Extraction Framework and Does Not Benchmark Against Alternative Extraction Methods

The assumption or constraint. All skills evaluated in this paper are produced by the same two-stage extraction framework—per-trajectory analysis followed by hierarchical consolidation—described in Section 3.2. The paper explicitly acknowledges that this framework is a "minimal" design that "borrows the high-level structure of Trace2Skill [9] but strips away its sub-agent fleet, conflict resolution, and skill-deepening mechanisms." The paper does not compare skills extracted by its framework against skills extracted by alternative methods (Trace2Skill's full pipeline, CoEvoSkills' co-evolutionary verification, ProcMem's RL-based refinement, or even simpler baselines like single-pass summarization). The extractor model E is varied (five models tested), but the extraction method is held fixed.

The consequence. The paper's central findings—that model-generated skills exhibit non-trivial negative transfer, that textual plausibility does not predict utility, and that the validated rubric improves extraction—are all conditioned on this specific extraction framework. It is possible that a more sophisticated extraction method (e.g., with iterative refinement, verification feedback, or multi-agent debate) would produce skills that are consistently beneficial across all targets and domains, reducing or eliminating the negative transfer problem. Conversely, it is possible that a simpler method (single-pass summarization) would produce skills of comparable quality at lower cost, making the two-stage framework unnecessary overhead. The paper cannot distinguish between "model-generated skills have inherent limitations" (a claim about the skill lifecycle) and "this particular extraction framework produces skills with these limitations" (a claim about the extraction method).

The meta-skill finding is particularly sensitive to this confound. The validated rubric improves extraction quality when applied to the two-stage framework with GPT-5.4-mini as extractor. But the rubric dimensions were derived from skills produced by this exact framework—the high-gap pairs that powered the contrastive analysis pipeline are all framework-output skills. A different extraction framework might produce skills where different textual properties predict utility, or where the three validated dimensions are no longer discriminative. The rubric's effectiveness might be specific to the extraction framework's characteristic output style, not a universal property of agent skills.

What evidence exists in the paper. The paper provides no method-level comparison. Section 2 surveys related extraction methods but does not implement or evaluate any of them. The paper's scope is explicitly defined as "a systematic, utility-grounded study of model-generated, domain-level skills," not a benchmark of extraction methods. This is a legitimate scoping choice for an analysis paper, but it means that the reported failure modes (negative transfer, plausibility-utility gap) are upper bounds on the failure rate of the best available extraction method—a better method might eliminate some of these failures.

Mitigation status. The paper does not attempt to mitigate this limitation; it is inherent in the study design. Appendix A frames future work as "scaling to richer agent harnesses" and "larger skill libraries," not as comparing extraction methods. A follow-up study that fixes the evaluation framework (same domains, targets, metrics) and varies the extraction method—comparing single-pass summarization, the paper's two-stage framework, full Trace2Skill, and an iterative refinement method like CoEvoSkills—would isolate how much of the observed variance in $\Delta$ is attributable to extraction method versus extractor model versus target identity. Without such a study, practitioners choosing an extraction method for their own deployment must extrapolate from this paper's single-framework results, which may not represent the achievable frontier.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new extraction algorithm, a new benchmark, or a new model architecture. Instead, it provides something the field lacked entirely: a systematic, lifecycle-wide empirical account of when and why model-generated agent skills actually work. The contribution is primarily diagnostic and methodological—it establishes that skill utility is not a property of skills alone but is jointly determined by extractor, target, and domain in ways that are non-obvious, measurable, and actionable. The magnitude of this shift is best characterized as a reframing with practical diagnostic infrastructure, not a paradigm shift. The core components—lifecycle decomposition, cross-factor experimental design, Extraction Efficacy (EE) and Target Evolvability (TE) metrics, utility-grounded rubric validation—form a reusable evaluation methodology that any future extraction method can plug into, and the empirical patterns uncovered (the plausibility-utility gap, the domain-dependence of negative transfer, the independence of extraction capability from baseline task strength) reset assumptions that the field had implicitly held.

Several specific ways this work changes how researchers and practitioners should think about agent skills:

1. Skill evaluation can no longer rely on inspection. The paper's pairing of Section 5.2 (unguided LLM judge performs worse than chance at identifying higher-utility skills, with accuracy dropping to 15.8% on pairs with δ ≥ 5 pp) with Section 6 (the same judge achieves 73.8% accuracy when equipped with utility-grounded criteria) establishes a clear burden of proof: any claim that a skill is "good" based on reading its text is unreliable unless the evaluator can demonstrate that their evaluation criteria correlate with measured downstream performance. This is not a minor methodological footnote—it is a direct challenge to the common practice in extraction-method papers of presenting qualitative skill examples as evidence of extraction quality. The paper does not merely argue that text-based evaluation is insufficient; it demonstrates that naive text-based evaluation is systematically anti-correlated with utility when performance gaps are large, meaning that a practitioner screening a skill library by reading skills and selecting the most impressive-looking ones would actively discard the highest-utility skills. This finding redirects evaluation effort from human or LLM inspection toward utility-grounded measurement, and makes the paper's own evaluation framework (cross-matrix Δ measurement with EE/TE decomposition) a candidate standard for future work.

2. The extraction problem becomes a compatibility problem, not an absolute optimization problem. Prior work implicitly treated skill extraction as a capability question: better models produce better skills. The paper falsifies this by showing that the relationship between model strength and extraction quality is not monotonic—Gemini-3.1-FL (a lightweight model) achieves the highest EE on SpreadsheetBench (+5.86) while GPT-5.4 (the strongest baseline) ranks last among extractors on the same domain (+1.67), and on ALFWorld, GPT-5.4's EE is actually negative (−0.66). This means that choosing an extractor is not equivalent to choosing the strongest available model; it requires matching extractor characteristics to the target's behavioral patterns and the domain's failure modes. This reframing has immediate practical consequences: organizations building skill extraction pipelines should evaluate candidate extractors on their specific target–domain combinations rather than assuming that their most capable model is the best extractor. The EE metric provides the tool for doing so.

3. The plausibility-utility gap redirects extraction research from pipeline engineering toward objective specification. The meta-skill experiment (Section 6, Figure 5, Table 11) is the paper's most consequential intervention for how extraction should be improved. The finding is stark: giving an extractor more guidance based on what looks good (the 7-dimension plausibility rubric) harms average performance by −0.59 pp, while guidance based on what actually predicts utility (the 3-dimension validated rubric) improves all nine evaluated cells by an average of +1.55 pp. Both are text strings appended to the same prompt—the only difference is whether the criteria are grounded in measured utility or intuitive notions of quality. This suggests that the bottleneck in current extraction methods is not algorithmic sophistication but objective misspecification: extractors are capable of producing better skills within their existing architecture, but they need to be told what "better" means in utility-grounded terms. Prior work that invested heavily in pipeline complexity (multi-agent debate, iterative refinement, RL-based optimization) without first ensuring that the extraction objective was correctly specified may have been optimizing for the wrong signal—producing skills that are fluent, well-structured, and broadly applicable, but which fail to encode the concrete failure mechanisms and domain-specific anti-patterns that actually drive downstream improvement. This paper's validated rubric provides a candidate objective specification, and its methodology (contrastive analysis of high-gap pairs, dimension validation via better-rate measurement, closed-loop extraction improvement) provides a template for discovering better objectives in new domains.

4. Domain heterogeneity in skill transferability becomes a first-class concern. The paper documents that negative transfer rates range from 13% (SpreadsheetBench, SWE-bench-Verified) to 47% (ALFWorld), and that the optimal success–failure composition of the experience pool is domain-specific (success-favoring for SpreadsheetBench, failure-favoring for ALFWorld, Figure 2). This heterogeneity means that a one-size-fits-all extraction strategy—use the same pool composition, the same extractor, the same skill granularity across all domains—is suboptimal, and in fragile domains like ALFWorld, actively harmful nearly half the time. The paper does not fully explain why ALFWorld is more fragile (the behavioral analysis in Appendix D provides hints but not a mechanism), but establishing the fact of domain-dependence is itself a contribution: it means future extraction methods must be evaluated across multiple qualitatively different domains to establish generality, and practitioners must calibrate their extraction strategy per domain rather than assuming transfer. This finding also reconciles a latent tension in prior work: papers that tested skill consumption on different domains likely reached different conclusions about whether skills help because the domains themselves have different inherent transferability. A method that appears effective on software engineering tasks may fail on embodied planning tasks, not because the method is flawed, but because the domain imposes different demands on skill specificity and the target's consumption ability.

5. The training–specialization asymmetry becomes measurable and actionable. The paper's decomposition of skill utility into EE and TE is not just analytical convenience—it operationalizes a practical concern that arises whenever the extractor and target are different models. In multi-model deployments where a powerful model batch-extracts skills for a fleet of smaller inference models, the EE metric answers "which extractor should we use?" and the TE metric answers "which targets will actually benefit?" The cross-model transfer experiment (Figure 4) adds nuance: skills extracted from a strong target's experience pool consistently help all targets, while skills from a weak pool help some but harm others. This means the source of the experience pool matters as much as the extractor's identity—a finding with direct implications for skill library curation. If an organization collects experience from multiple models at different capability levels, not all experience pools are equally valuable for downstream skill extraction; the paper suggests prioritizing high-quality experience (from strong targets) even when the intended consumers are weaker models.

6. The closed-loop methodology—from diagnosis to intervention to measurement—provides a template for utility-grounded improvement cycles. The paper's architecture—cross-factor evaluation reveals a problem (plausibility-utility gap), contrastive analysis of high-gap pairs discovers candidate explanatory dimensions, pairwise evaluation with better-rate measurement validates which dimensions actually predict utility, validated dimensions are operationalized as a generation-time prior, and the resulting improvement is measured on the same evaluation framework—is generalizable. Any extraction method, domain, or model family could be subjected to the same cycle. This makes the paper's contribution extend beyond its specific findings: it provides a methodology for converting evaluation infrastructure into an improvement signal, turning benchmarks from passive measurement tools into active optimization drivers. This is a subtle but important shift in how the field should think about the relationship between evaluation and method development for agent skills.

7. The paper implicitly argues against over-engineering extraction pipelines before establishing a utility-grounded objective. The minimal extraction framework (Section 3.2) is not just a design choice for attribution—it is a methodological argument. By showing that a simple two-stage framework, combined with a utility-grounded meta-skill, can produce consistent improvements across diverse domains (all nine cells improve in Table 11), the paper suggests that the field's investment in pipeline complexity (sub-agent fleets, co-evolutionary verification, RL-based refinement) may be premature if the extraction objective has not been properly specified. A complex pipeline with a misaligned objective will efficiently produce skills that look good but perform poorly; a simple pipeline with a correctly specified objective may outperform it at lower cost. This is not a definitive claim—the paper does not compare against complex pipelines—but it is a hypothesis that the paper's results make testable: fix the utility-grounded objective (the validated rubric dimensions) and then add pipeline complexity to see whether the marginal benefit of sophisticated extraction architecture exceeds the marginal benefit of better objective specification.

Follow-Up Research This Work Enables

Utility-grounded objective discovery across domains and model families. The validated rubric (Failure Mechanism Encoding, Actionable Specificity, High-Risk Action Blacklist) was derived from skill pairs spanning five domains and multiple model families, but the rubric's generalizability to held-out domains is untested. A strong follow-up would: (1) run the full cross-matrix protocol on a new set of domains (e.g., code generation, dialogue, game playing) with models not used in this paper's discovery set, (2) apply the validated rubric as a meta-skill and measure whether it improves extraction in those new domains, and (3) if it fails, run the contrastive discovery pipeline on the new-domain pairs to identify domain-specific utility dimensions and test whether they subsume, complement, or contradict the original three dimensions. The hypothesis from this paper is that Failure Mechanism Encoding, Actionable Specificity, and High-Risk Action Blacklist are domain-general properties of useful skills, but the counter-hypothesis—that utility dimensions are partially domain-specific and the current rubric overfits to SpreadsheetBench and SWE-bench, which dominated the discovery pairs—is equally plausible given the concentration of large-Δ gaps in those domains. A negative result (the validated rubric fails to improve extraction in new domains) would be informative: it would mean utility-grounded criteria are domain-dependent and must be rediscovered per deployment context, which would make the discovery pipeline itself (not the specific three dimensions) the paper's durable contribution. A positive result (the rubric transfers) would strengthen the claim that these three dimensions capture fundamental properties of useful procedural guidance for language agents, independent of domain.

Decomposition of extraction vs. consumption failure modes through targeted behavioral experiments. The paper establishes that skills can fail either because the extractor produced a poor skill or because the target cannot consume even a good skill effectively, but does not isolate the mechanisms. A targeted follow-up would: (1) take a skill that helps target A but hurts target B (the weak-pool skill in Figure 4, which harms GPT-5.4 by −2.0 pp but helps Qwen3.5-9B by +3.2 pp, is a ready-made stimulus), (2) instrument the target's decision process at the token or reasoning-step level to identify where the skill interferes—does it override correct default strategies, introduce confusion at decision points, or add cognitive load that degrades execution, and (3) systematically ablate parts of the skill (remove Actionable Specificity passages, remove Failure Mechanism Encoding passages, remove High-Risk Action Blacklist passages) and measure which component drives the negative transfer for the harmed target. This would convert the paper's correlational finding ("skill consumption reshapes the target's default policy," Appendix D) into a mechanistic account with direct implications for how skills should be adapted per target. The prediction from this paper's behavioral analysis is that high-performing targets suffer from skill interference (the skill's guidance conflicts with the target's already-effective strategies), while low-performing targets benefit from skill supplementation (the skill provides guidance the target lacks). If confirmed, this would imply that skills should be deployed selectively: strong targets should receive skills only when they address specific known failure modes, while weak targets can benefit from broader skill injection.

Dynamic, task-level skill selection vs. static domain-level skill injection. The paper's single-skill protocol forces the same domain-level skill to be present on every task, including tasks where its guidance may be misaligned. A natural follow-up would compare the paper's static protocol against a retrieval-based protocol where: (1) multiple fine-grained skills are extracted per domain (using max_skills > 1 and organizing skills by task subtype or failure mode), (2) at inference time, the target retrieves skills relevant to the current task (using embedding similarity, keyword matching, or a learned router), and (3) only the retrieved skills are injected. The hypothesis is that selective injection would reduce negative transfer by avoiding skill interference on tasks where the skill is inapplicable, while preserving or increasing positive transfer by allowing task-specific guidance. The paper's behavioral analysis (Appendix D) supports this hypothesis: it observes that skill consumption reshapes default policy, which is beneficial when the default policy is wrong for the task at hand but harmful when it is already correct. Selective injection would decouple these cases. The experiment would require: a multi-skill extraction protocol (which the paper's framework already supports, per Appendix B.4), a retrieval mechanism, and an evaluation design that compares per-task accuracy under static vs. retrieved injection, with a breakdown by task subtype to test whether retrieval helps on tasks where the domain-level skill's guidance is known to be misaligned.

Extraction stability and ensemble extraction as a risk mitigation strategy. The paper reports one extraction per (E, M, D) cell, leaving open the question of whether negative-Δ cells reflect systematic incompatibility or unlucky sampling (extractor temperature is 1.0, so multiple extractions from the same pool would produce different skills). A follow-up would: (1) re-extract skills from the same experience pool multiple times (e.g., 10 independent extraction runs) for a subset of cells spanning positive-Δ, negative-Δ, and near-zero-Δ cases, (2) measure the within-cell variance in Δ to determine whether negative transfer is stable or stochastic, and (3) test an ensemble strategy that extracts N skills independently and selects the one that performs best on a small validation set (or, if no validation set is available, uses majority voting on the skill text via the guided rubric judge). If negative-Δ cells are unstable (high variance across extraction runs), ensemble extraction with even a small N could substantially reduce the negative transfer rate by rejecting unlucky draws, making skill extraction much safer for deployment. If negative-Δ cells are stable (low variance), then certain extractor–target–domain combinations are fundamentally incompatible, and the solution is to avoid those combinations rather than to ensemble-extract. The paper's Figure 3 (guided judge accuracy 73.8%) provides a candidate selection mechanism for ensemble extraction: the guided judge could serve as a cheap proxy for downstream evaluation during skill selection, though its 73.8% accuracy means it would misrank approximately one in four pairs.

Skill transferability across model architectures and training paradigms. The paper's models span three families (GPT, Gemini, Qwen) and multiple scales, but all are autoregressive language models with similar pretraining objectives. A stress-test would ask whether skills extracted from language model experience transfer to agents built on different architectures—e.g., vision-language models (VLMs) for embodied tasks, retrieval-augmented generation (RAG) pipelines for knowledge-intensive tasks, or fine-tuned specialist models. The prediction from this paper is that skill consumption ability (TE) depends on the target's capacity to interpret and execute procedural guidance, which may be lower for models that were not pretrained on extensive procedural text. If skills extracted from GPT-5.4's ALFWorld experience fail to help a VLM-based embodied agent, this would establish a boundary condition: skill extraction for a target likely requires the target to have generated the experience pool, or at minimum to share the same action space and behavioral priors. This has direct implications for the vision of universal skill libraries that transfer across heterogeneous agents—a vision the paper's cross-model transfer experiment (Figure 4) partially supports (skills do transfer, but with variable success and occasional negative transfer) but which may break down across larger architectural gaps.

The interaction between extraction and the target's self-reflection or self-critique capability. Section 5.3 observes that skill consumption reshapes the target's default policy rather than triggering explicit skill calls, and Appendix D notes that for Qwen3.5-9B, this can increase exploration (more workbook inspection) without improving final accuracy. This suggests an interaction between skill injection and the target's metacognitive abilities: a target that can recognize when the skill's guidance is leading it astray might benefit more than a target that blindly follows the skill's implied strategy. A follow-up would: (1) pair skill injection with explicit self-critique instructions (e.g., "after reading the skill, explain whether and why its guidance applies to the current task"), (2) measure whether self-critique reduces negative transfer (by helping the target reject misaligned guidance) and whether it preserves or reduces positive transfer (by potentially causing the target to reject helpful guidance), and (3) compare against the paper's current disclaimer-based approach ("This skill is an optional aid, not a mandatory procedure. Use your own judgment.") to test whether more structured metacognitive scaffolding outperforms a simple disclaimer. The hypothesis is that stronger targets (which already have good judgment) would benefit from self-critique because they can discriminate between applicable and inapplicable guidance, while weaker targets might be confused by the additional metacognitive task and perform worse.

Practical Applications and Downstream Use Cases

Skill library quality screening in multi-model deployment pipelines. An organization that maintains a fleet of agents at different capability tiers (e.g., a strong cloud model for complex tasks, a lightweight on-device model for latency-sensitive tasks) and uses model-generated skills faces a concrete problem this paper solves: how to screen extracted skills before deployment to avoid silently shipping skills that degrade performance. The paper's guided evaluation protocol (pairwise judge with the validated rubric, 73.8% accuracy) provides a cheap, automated screen: before deploying a newly extracted skill, compare it via the guided judge against a set of known-good skills for the same domain, and only deploy if it ranks favorably. The 73.8% accuracy is not perfect—it would incorrectly advance roughly one in four bad skills—but it is substantially better than unguided screening (which would preferentially advance bad skills when the quality gap is large, given the 15.8% accuracy on δ ≥ 5 pp pairs). For a production pipeline processing hundreds of skill extractions, automated screening with the guided rubric could catch the worst skills (the large-negative-Δ cases that cause the most damage) and flag them for human review or re-extraction, while allowing high-confidence positive skills through. The compute cost of the guided pairwise evaluation is minimal (two LLM judge calls per pair) relative to the cost of deploying a harmful skill to a production fleet, making the screening step a net-positive investment.

Calibrated extraction for domain-specific risk tolerance. The paper's finding that negative transfer rates are domain-dependent (13% for SpreadsheetBench and SWE-bench-Verified, 47% for ALFWorld) provides a risk calibration that practitioners can use to decide whether to deploy model-generated skills at all in a given domain. In low-risk domains like spreadsheet manipulation (where even unguided extraction helps 87% of the time and the validated rubric pushes this higher), automated skill extraction with minimal human oversight is defensible: the expected benefit is positive, the failure rate is low, and failures are unlikely to be catastrophic (a spreadsheet formula error is typically detectable and correctable). In high-risk domains like embodied planning (where nearly half of extractions cause negative transfer), automated extraction without human validation is irresponsible unless the deployment includes fail-safes (e.g., skill deactivation when task success rate drops below a threshold, or confidence-based selective skill application). The paper's per-domain breakdown (Table 1, TE and EE columns) gives practitioners the data to make this calibration. A new domain not covered by the paper can be analyzed using the same cross-matrix protocol, with a small pilot study (a few extractors, a few targets, a modest pool of test tasks) sufficient to estimate the domain's negative transfer rate and determine whether extraction should proceed with or without human-in-the-loop validation.

Experience pool curation guided by the success–failure ratio finding. Section 5.1 (Figure 2) demonstrates that the composition of the experience pool substantially affects downstream skill quality, and that the optimal ratio is domain-specific. A practitioner collecting experience for skill extraction can use this finding directly: rather than naively collecting all available trajectories, they should curate the pool to match the success–failure ratio that maximizes skill utility for their domain. For a spreadsheet manipulation domain (analogous to SpreadsheetBench), prioritize successful trajectories (75–100% success). For an embodied interaction domain (analogous to ALFWorld), deliberately include failure trajectories (25–50% success), since failures encode the action-space constraints and dead-end states that are most informative for skill extraction. The paper's domain-specific optimal ratios are not transferable directly to new domains (the optimal ratio for a new domain would need to be discovered by running the Section 5.1 experiment on that domain), but the principle—that pool composition matters and is domain-dependent—is transferable, and the paper's methodology for discovering the optimal ratio (sweep success ratios, fix the extractor, measure average Δ) is replicable with modest compute (five extractions per tested ratio, assuming one extractor and domain).

Extractor selection for cost-constrained deployments. In a deployment where the extractor model can be chosen independently of the target (e.g., batch extraction on cloud GPUs for skills deployed to on-device models), the paper's EE metric enables principled extractor selection. On SpreadsheetBench, Gemini-3.1-FL (a lightweight, presumably cheaper model) achieves EE = +5.86, while GPT-5.4 (a larger, more expensive model) achieves EE = +1.67. Choosing Gemini-3.1-FL as the extractor yields higher average downstream gains at lower extraction cost—a rare case where the cheaper option is also the better one. The paper's cross-matrix (Table 1) enables this analysis per domain: a practitioner can identify the extractor with the highest EE for their specific domain and target family, potentially saving substantial extraction compute while improving skill quality. The caveat is that EE averages across targets; if the deployment uses only a single target model, the practitioner should look at the specific Δ cell for that (extractor, target, domain) combination rather than the EE aggregate, since an extractor with high EE might perform poorly on a particular target (as GPT-5.4 does on ALFWorld for some targets, despite having moderate EE overall).

When to Prefer This Method

The paper does not position its approach against named alternative extraction methods or consumption protocols—it provides an evaluation framework and a meta-skill improvement, not a competing method that trades off against alternatives. The decision facing a practitioner is therefore not "should I use this extraction method versus that one?" but rather "given that I'm using model-generated skills, how should I evaluate them and improve their extraction?" The paper's guidance is:

Use the cross-matrix evaluation framework (Section 3, Table 1) when deploying model-generated skills to a new domain or with a new target model, to estimate the domain's negative transfer rate and identify which extractor–target combinations are most reliable. This is diagnostic infrastructure, not a method choice.
Apply the validated rubric as a meta-skill (Section 6, the three dimensions: Failure Mechanism Encoding, Actionable Specificity, High-Risk Action Blacklist) when extracting skills in domains with structured procedural tasks and well-defined failure modes (analogous to SpreadsheetBench, SWE-bench-Verified, BFCL-v4). The meta-skill consistently improves extraction quality in the tested domains and adds no extraction cost (it is a prompt modification). The improvement is largest in domains where concrete failure mechanisms and anti-patterns drive performance gaps.
Be cautious with the validated rubric in fragile domains (analogous to ALFWorld) where negative transfer rates are high. The meta-skill helps (all nine cells improve in Table 11) but the absolute gains are smaller in fragile domains (+0.38 to +1.00 pp on ALFWorld vs. +2.33 to +3.69 pp on SpreadsheetBench), and the rubric does not address the consumption-side limitations that cause some targets to regress even with improved skills.
Prefer high-quality experience pools (strong target models, high success ratios for domains analogous to SpreadsheetBench, failure-inclusive pools for domains analogous to ALFWorld) as the source of trajectories for extraction, regardless of the intended consumer model. The cross-model transfer experiment (Figure 4) shows that skills from strong-pool experience help all targets, while skills from weak-pool experience harm some.
Do not rely on textual inspection to evaluate skill quality. Use downstream performance measurement (the Δ metric) or, if measurement is infeasible, use the guided rubric judge (73.8% accuracy) as a screening proxy—but never use unguided human or LLM inspection, which is anti-correlated with utility for large performance gaps.

The paper does not articulate a tradeoff between its approach and a specific alternative method (e.g., "use our framework instead of Trace2Skill"), so a comparative preference matrix would be the paper's fabrication rather than its argument. The contribution is a methodology and a set of empirical findings that apply on top of any extraction method, not a method that competes with alternatives.