ArXiv: 2604.25727

🎯 Pitch

Training terminal agents doesn't just need more tasks—it needs tasks that force the agent to apply different skills in fundamentally different intermediate scenarios. SkillSynth achieves this by constructing a skill graph where paths encode real-world workflows, and fine-tuning on its synthesized trajectories lets a 32B model beat a 480B coder on Terminal-Bench.

1. Executive Summary

This paper introduces SkillSynth, an automated framework for synthesizing diverse terminal task instances by constructing a scenario-mediated skill graph — a directed multigraph where scenarios serve as nodes and skills as transitions — and sampling compositional paths from it as abstractions of real-world workflows. Using a multi-agent harness built around a planner–constructor architecture with dual verification (execution-based oracle checks and rubric-based quality evaluation), SkillSynth instantiates 3,560 verified task instances from 3,721 sampled paths at a 95.7% oracle pass rate and an average cost of $27.3 per verified instance. Fine-tuning Qwen3-8B, Qwen3-14B, and Qwen3-32B on trajectories collected from these synthesized tasks yields consistent gains over single-skill and randomly composed multi-skill baselines on Terminal-Bench 1.0 and 2.0, with Qwen3-32B + SS reaching 33.8% and 29.6% accuracy respectively — outperforming even the 480B-parameter Qwen 3 Coder on TB 2.0. Critically, the paper establishes that training effectiveness for terminal agents depends on the diversity of execution trajectories along both scenario and skill dimensions, not merely on task volume, with graph-guided synthesis producing 31% higher unique scenario–skill coverage than single-skill baselines and driving corresponding downstream performance improvements.

2. Context and Motivation

The Core Problem: Terminal Agent Training Is Bottlenecked by Trajectory Diversity, Not Just Task Count

The fundamental problem this paper addresses is that terminal agents — LLMs that execute commands through a command-line interface — are constrained not merely by how many training tasks exist, but by how diverse the execution trajectories through those tasks actually are. This distinction matters because of how terminal agents operate and learn.

When an LLM-based terminal agent solves a task, it does not produce a single atomic output. It takes dozens or hundreds of sequential actions — running commands, inspecting outputs, diagnosing errors, trying alternative approaches — and each decision point is conditioned on the specific intermediate state of the environment at that moment. The paper formalizes this in Section 2 by abstracting each trajectory into a sequence of scenarios (decision-relevant states) and skills (action subsequences that transition between scenarios). Training such an agent amounts to learning a policy over skills conditioned on scenarios (Equation 3):

J(π)=Eξ∼D∑t=1Llog⁡π(κt∣σt−1,g)J(\pi) = \mathbb{E}_{\xi \sim \mathcal{D}} \sum_{t=1}^{L} \log \pi(\kappa_t \mid \sigma_{t-1}, g)

Decomposing this objective over the support of the empirical distribution D\mathcal{D} (Equation 4) reveals the crux of the problem:

J(π)=∑σ∈ΩpD(σ∣g)∑κ∈KσpD(κ∣σ,g)log⁡π(κ∣σ,g)J(\pi) = \sum_{\sigma \in \Omega} p_{\mathcal{D}}(\sigma \mid g) \sum_{\kappa \in \mathcal{K}_\sigma} p_{\mathcal{D}}(\kappa \mid \sigma, g) \log \pi(\kappa \mid \sigma, g)

The learnable region of the agent's policy is confined to the support of D\mathcal{D} along both factors. Scenarios with pD(σ∣g)=0p_{\mathcal{D}}(\sigma \mid g) = 0 are never observed during training. Skills with pD(κ∣σ,g)=0p_{\mathcal{D}}(\kappa \mid \sigma, g) = 0 are never exercised. The consequence is stark: if your training data covers the same narrow set of (scenario, skill) pairs repeatedly, you cannot learn beyond that set, regardless of how many total examples you have. Maximizing the agent's learned capability requires training data whose empirical distribution D\mathcal{D} densely covers the conditional product space {(σ,κ):σ∈Ω,κ∈Kσ}\{(\sigma, \kappa) : \sigma \in \Omega, \kappa \in \mathcal{K}_\sigma\}.

This is not an abstract theoretical concern. The paper's motivating empirical observation appears in Figure 1, where the authors measure the diversity of existing trajectory datasets by counting unique scenarios, skills, and (scenario, skill) pairs after semantic canonicalization. The finding is that current datasets exhibit significant redundancy: different task instances repeatedly expose the agent to overlapping intermediate states and reuse similar skills. The problem is therefore: how do we synthesize terminal task instances that explicitly maximize trajectory diversity along both scenario and skill dimensions, rather than merely scaling task count?

Why This Problem Matters: Practical and Theoretical Significance

Practical impact on open-source terminal agent capability. The results in Table 3 reveal a substantial capability gap between proprietary and open-source terminal agents. GPT-5.3-Codex achieves 64.7% on Terminal-Bench 2.0, while Qwen 3 Coder 480B — an open-source model with 480 billion parameters — reaches only 23.9%. Even fine-tuned open-source models lag significantly: Qwen3-32B + SS reaches 29.6% on TB 2.0. This gap exists because proprietary models benefit from massive, diverse training data that open-source efforts cannot easily replicate through manual curation. Automated synthesis of diverse terminal tasks is therefore the primary lever for closing this capability gap — and doing so requires intentionally designing synthesis to maximize trajectory diversity, not just task count.

The scaling law for terminal agent training is diversity, not volume. The paper's ablation study (Table 4) demonstrates this empirically: Qwen3-32B trained on SkillSynth's graph-guided trajectories (33.8% TB 1.0, 29.6% TB 2.0) significantly outperforms the same model trained on the same number of randomly composed multi-skill trajectories (30.8%, 25.8%) — a gap of 3.0 and 3.8 points respectively — even though both use the identical multi-agent harness, identical base model, and identical training recipe. The difference is purely in how the seed workflows were constructed. This has direct economic implications: if NN diverse tasks produce a better agent than M>NM > N redundant tasks, then synthesis budgets should be allocated to diversity-maximizing generation rather than volume-maximizing generation.

Terminal agents as a universal interface. Terminal agents are not a narrow application. The command line is a universal action space: it provides programmatic access to file systems, package managers, version control, containers, databases, compilers, network tools, and virtually every software tool. An agent that can operate effectively in the terminal can theoretically perform any computational task. The paper's skill graph spans categories from coding agents and DevOps through audio/speech processing, 3D simulation, and IoT workflows (Figure 5, Appendix D), underscoring the breadth of what terminal agents could accomplish with sufficient training diversity.

Theoretical significance: formalizing trajectory diversity. Prior work on synthetic data for agents largely operated on intuition — "more data is better" — without a formal framework for understanding what makes some trajectories more valuable than others. The paper's formulation in Section 2 provides a precise criterion (the coverage of the Ω×K\Omega \times \mathcal{K} product space) that connects directly to the training objective. This transforms trajectory diversity from a vague desideratum into a measurable, optimizable quantity — analogous to how the original scaling laws work (Hoffmann et al., 2022) transformed pretraining from "use more data" into "optimally allocate compute between parameters and tokens."

Where Existing Approaches Fall Short

The paper identifies specific shortcomings in prior terminal task synthesis methods, organized along two dimensions: what they scale and what they control.

Scaling task count without controlling trajectory diversity. The dominant paradigm in prior work is to generate a large number of task instances and hope that diversity emerges as a side effect. The paper catalogs several variations of this approach:

  • LLM-generated taxonomies for domain expansion (Gandhi et al., 2026, Endless Terminals; Zhu et al., 2026, TermiGen; Pi et al., 2026, Nemotron-Terminal): These approaches prompt LLMs to produce hierarchical taxonomies of terminal tasks, then sample from the taxonomy to generate instances across diverse domains. The taxonomy expands the types of tasks but provides no mechanism for controlling which intermediate scenarios or skills appear in the solving trajectories. A task about "configuring a web server" might be superficially different from one about "configuring a database," but both could produce trajectories that share the same substrata of file editing, package installation, and configuration parsing — contributing little additional coverage of the Ω×K\Omega \times \mathcal{K} space beyond the first few examples.

  • Repository-derived task instances (Wu et al., 2026; Lin et al., 2026, CLI-Gym; Chen et al., 2026, SWE-Universe): These approaches collect Docker environments from real GitHub repositories and derive task instances from them, either directly or by inverting healthy environments into buggy states. While grounded in real-world usage, these methods are narrowly scoped to software engineering domains — issue resolution, feature development, bug fixing (Yang et al., 2025b; Zhang et al., 2025; Wang et al., 2025a). The paper notes that these tasks "provide limited explicit control over the scenario or skill composition underlying the resulting trajectories." An agent trained exclusively on repository-derived tasks sees a narrow slice of the Ω\Omega space (software project states) and a narrow slice of the K\mathcal{K} space (debugging and patching workflows), missing entire categories like system administration, data processing pipelines, and multimedia workflows that appear in the paper's skill graph (Figure 5).

  • No mechanism for compositional difficulty. Tasks synthesized from single-skill seeds or random multi-skill compositions (the paper's baselines in Section 4.4) fail to produce genuinely challenging workflows. Randomly composed skills lack the sequential dependencies that make real terminal tasks difficult — the multi-agent harness, when given incoherent skill sequences, tends to "generate simplified task instances that contain multiple fine-grained requirements but require few execution steps." The result is a collection of tasks that look complex on the surface (many sub-requirements) but produce trivial trajectories (few decision points, no error recovery, no environment state reasoning).

Failure to model the scenario dimension explicitly. A deeper limitation cuts across all prior approaches: they treat tasks as atomic units defined by their initial state and goal, without modeling the intermediate states that the agent actually encounters during execution. This is the insight formalized in Equation 2 — a trajectory is not just an initial state and a final answer, but a sequence of intermediate scenarios and the skills applied at each one. Prior work implicitly assumes that controlling the diversity of task definitions (instructions and initial environments) is sufficient to control the diversity of training trajectories. The paper demonstrates this assumption is false: tasks that differ in their surface descriptions can produce overlapping trajectories, and tasks that are superficially similar can produce divergent trajectories depending on how the agent navigates them.

Validation-focused rather than training-focused synthesis. Terminal-Bench itself (Merrill et al., 2026), which the paper uses for evaluation, provides hand-crafted tasks for benchmarking but not for training. These tasks are diverse and high-quality, but only 89 exist in TB 2.0 — far too few for supervised fine-tuning or reinforcement learning at scale. The gap between "enough tasks to evaluate" and "enough tasks to train" is enormous, and prior work has not bridged it in a way that preserves trajectory diversity.

How SkillSynth Positions Itself

SkillSynth's positioning relative to prior work can be understood through three design commitments that distinguish it:

1. Trajectory diversity as an explicit optimization target, not a hoped-for byproduct. Rather than generating tasks and hoping their trajectories are diverse, SkillSynth inverts the process: it first constructs a graph whose structure encodes potential trajectory diversity (scenarios as nodes, skills as edges), then samples diverse paths from it, and finally instantiates task instances whose intended solutions follow those paths. This means the diversity of the resulting trajectories is designed in at the graph construction and path sampling stages, not left to chance during task instantiation. The inverse-frequency path sampling algorithm (Algorithm 1) explicitly pushes the empirical distribution toward uniform coverage of the Ω×K\Omega \times \mathcal{K} space, directly implementing the coverage criterion from Equation 4.

2. Scenarios as first-class objects in task synthesis. The paper's central architectural innovation is making scenarios — not just skills or tasks — into explicit, first-class objects in the synthesis framework. The skill graph is "scenario-mediated": scenarios are the nodes, and skills are transitions between them. This design choice directly follows from the theoretical framing in Section 2, which identifies scenarios as a joint factor in the learning objective alongside skills. By constructing a graph where both scenarios and skills are explicitly represented and connected, SkillSynth can sample paths that jointly specify which scenarios will be traversed and which skills will be exercised — providing control over both dimensions of trajectory diversity.

This contrasts fundamentally with prior skill organization work (Li et al., 2026, AgentSkillOS; Liang et al., 2026, SkillNet), which organizes skills into hierarchies or relational graphs but does not model the intermediate states (scenarios) that connect them. AgentSkillOS arranges skills into DAG-based orchestration graphs where edges represent execution dependencies, not state transitions. SkillNet models inter-skill relationships but does not track the environmental states between skill executions. SkillSynth's scenario-mediated structure is more expressive for trajectory synthesis because it captures compatibility constraints — skill B can follow skill A only if B's precondition scenario is semantically compatible with A's postcondition scenario — that purely skill-to-skill graphs cannot represent.

3. Grounding in real-world terminal expertise without manual curation. The skill graph is not constructed from scratch by prompting an LLM to imagine plausible terminal workflows. It is grounded in existing, human-authored skills from ClawHub (OpenClaw, 2026) and public GitHub repositories — skills that represent "practical experience distilled from real terminal usage." The graph construction pipeline (Section 3.2, Figure 3) takes these raw skills, infers their pre- and postcondition scenarios via LLM prompting, deduplicates and aligns scenarios across skills, and constructs the unified graph. This grounding provides two advantages: (1) the skills are realistic and executable, not hallucinated by an LLM, and (2) the graph can grow organically as the community contributes more skills to ClawHub, making the task synthesis capacity scale with the ecosystem.

The paper explicitly frames this as scalable infrastructure: "As the community contributes more skills, the graph continues to expand, enabling continual synthesis of diverse terminal tasks." This positions SkillSynth not as a one-time dataset (like Nemotron-Terminal or Endless Terminals) but as a living synthesis framework whose diversity and coverage grow with the underlying skill ecosystem.

4. Bridging the gap between synthesis and training. A subtle but important positioning: SkillSynth is not just a task synthesis method — it is a pipeline from skill graph → task instances → execution trajectories → fine-tuned agents. The paper validates this end-to-end by fine-tuning Qwen3 models on trajectories collected from SkillSynth tasks and demonstrating gains on Terminal-Bench. This is distinct from prior synthesis work that focuses primarily on generating tasks and leaves trajectory collection and model training as separate, unvalidated steps. The full pipeline demonstrates that diversity designed into the synthesis stage propagates through to measurable downstream performance improvements.

5. Addressing the hardness gap through compositional paths. The paper observes that prior synthesis approaches produce tasks that are either too easy (single-skill tasks with trivial trajectories) or incoherent (random multi-skill compositions that the harness simplifies into easy tasks). SkillSynth addresses this by sampling paths from a graph whose edges represent real compatibility relationships, ensuring that multi-skill workflows are coherent — each skill's postcondition genuinely enables the next skill's precondition. As shown in Figure 4, a sampled path like "Video Analyzer → Frame Extractor → GIF Generator" forms a coherent video-processing pipeline where each skill's output (video session with metadata, extracted frames with timestamps) naturally feeds into the next skill's input. The resulting task instances require agents to chain these skills sequentially through progressively changing intermediate states, producing longer, more diverse trajectories with genuine compositional difficulty.

The evidence for this positioning appears in Table 2, where 38% of SkillSynth tasks receive a 0/3 success rate from Hy3 Preview (a strong proprietary agent), and in the trajectory-level finding that Claude Opus 4.6 requires an average of 37 steps to solve SkillSynth tasks — indications that graph-guided synthesis produces genuinely challenging problems that stress agent capabilities.

3. Technical Approach

3.1 Reader Orientation

SkillSynth is an end-to-end pipeline that automatically constructs diverse, executable terminal command-line tasks by first building a massive graph of executable skills and their compatible intermediate system states, then sampling coherent multi-step workflows from it, and finally using LLM-based agents to flesh those workflows out into fully containerized, verified task instances with instructions, test suites, and oracle solutions. The system solves the problem that simply generating more terminal tasks does not guarantee diverse training data — because different tasks can exercise the same narrow set of intermediate scenarios and skills — by explicitly designing trajectory diversity into the synthesis process from the start, using the graph structure to control which scenarios an agent will encounter and which skills it must apply to solve each task.

3.2 Big-Picture Architecture (Diagram in Words)

SkillSynth operates in three sequential stages, whose information flow is: raw human-authored skills → constructed skill graph → sampled workflow paths → verified executable task instances → collected agent trajectories → fine-tuned models.

  1. Skill Graph Construction (Section 3.2, Figure 3): Takes a pool of real terminal skills from ClawHub and GitHub, prompts an LLM to infer each skill's precondition and postcondition scenarios (descriptions of system states before and after execution), semantically deduplicates and clusters these scenarios, aligns compatible pre/post-condition scenario pairs across different skills using embedding similarity followed by LLM verification, and constructs a directed multigraph $G = (\Omega, \mathcal{K})$ where nodes are scenarios and directed edges are skills. The result is a graph with 82,073 scenario nodes, 57,214 skill-labeled transitions, and 185,529 LLM-verified bridges, spanning domains from coding agents through audio processing and IoT workflows.

  2. Graph-Guided Path Sampling (Section 3.3, Algorithm 1): Samples directed paths from the constructed graph, where each path $\mathcal{P} = (\sigma_0, \kappa_1, \sigma_1, \ldots, \kappa_L, \sigma_L)$ specifies a coherent multi-step workflow — a sequence of skills to be applied in order, together with the intermediate scenarios they traverse. Uses inverse-frequency weighting to push the empirical distribution toward uniform coverage of the $\Omega \times \mathcal{K}$ product space, directly implementing the diversity criterion from Equation 4. The output is a set of unique, diversity-maximizing workflow paths (3,721 paths in the paper's run, with lengths from 1 to 7 skills).

  3. Multi-Agent Harness (Section 3.4, Figure 2, panels b and c): Takes each sampled path and instantiates it into a fully executable terminal task instance through a two-stage planner–constructor architecture with dual verification. A planner first transforms the abstract path into a structured plan of sub-objectives and expected outputs; a constructor then generates all five task components (natural-language instruction, initial filesystem snapshot, Dockerfile-based containerized environment, verification scripts, and oracle solution). Each task instance passes through execution-based oracle verification (does the oracle solution actually pass the tests inside the container?) and rubric-based evaluation (are the tests aligned with the instruction, and is the instruction self-contained?), with up to 3 repair cycles for failed instances. The output is a validated, executable task instance ready for trajectory collection.

The downstream pipeline then uses these task instances as environments: a teacher agent (MiniMax M2.7) solves each task 3 times, producing execution trajectories; both successful and failed trajectories are retained; and Qwen3 base models are fine-tuned on the collected trajectories using standard supervised fine-tuning, yielding the +SS models evaluated on Terminal-Bench.

3.3 Roadmap for the Deep Dive

  • First, the mathematical foundation from Section 2 in full detail: the formalization of scenarios and skills, the trajectory decomposition in Equation 2, the training objective in Equation 3, and the decomposition in Equation 4 that establishes why coverage of the $\Omega \times \mathcal{K}$ product space is the target — because this theoretical framing drives every subsequent design decision.
  • Second, the skill graph construction pipeline end-to-end: skill filtering criteria, scenario inference via LLM prompting, the two-stage hierarchical clustering procedure for scenario deduplication, the bidirectional cross-skill alignment process, and the final merging and filtering step that produces the unified graph.
  • Third, the inverse-frequency path sampling algorithm: the motivation for avoiding uniform random walks, the mathematical form of the sampling probabilities, the monotone progression constraint, and how the algorithm progressively steers the empirical distribution toward uniform coverage.
  • Fourth, the multi-agent harness in full operational detail: the planner–constructor decoupling and why it matters, the five task components generated, the dual verification mechanism (execution-based oracle check and rubric-based LLM-as-Judge evaluation), and the interactive repair loop with its specific hyperparameters (up to 3 repair cycles, at most 20 tool calls per cycle).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and data synthesis paper whose core idea is that terminal task diversity can be explicitly engineered by constructing a scenario-mediated skill graph, sampling diversity-maximizing paths from it, and using those paths as compositional blueprints for multi-agent task instantiation — inverting the usual relationship where task synthesis drives trajectory diversity into one where desired trajectory diversity drives task synthesis.

Formal Foundation: Scenarios, Skills, and Why Coverage Matters

The paper's entire technical approach is motivated by a precise mathematical framing of what makes terminal agent trajectories diverse and why that diversity matters for training. This framing appears in Section 2 and serves as the theoretical justification for the graph construction and path sampling design choices.

What is a terminal agent task? The paper starts with a standard formulation. A terminal agent task is a tuple $\tau = (\mathcal{E}, s_0, g, \mathcal{V})$, where $\mathcal{E}$ is the executable environment (a Docker container with a filesystem and installed tools), $s_0 \in \mathcal{S}$ is the initial state of that environment, $g$ is a natural-language goal description, and $\mathcal{V}$ is an external verifier that checks whether the final state $s_T$ satisfies the goal. At each time step $t$, the agent receives an observation $o_t \in \mathcal{O}$ (typically the terminal output from the previous command), and samples an action $a_t \sim \pi(\cdot \mid o_{\leq t}, a_{<t}, g)$ — the next shell command to execute. The resulting low-level trajectory $\zeta = (o_0, a_0, o_1, a_1, \ldots, o_T)$ is a sequence of alternating observations and actions. The trajectory is labeled successful if $\mathcal{V}(s_T) = 1$.

The problem with low-level trajectories. This formulation captures everything the agent does, but it obscures the semantic structure of the agent's problem-solving strategy. A trajectory might contain hundreds of tokens — ls, cat, pip install, python script.py, grep, git commit — but knowing which commands were typed does not directly reveal what high-level subgoal the agent was pursuing at each stage or what partial state of the environment prompted that subgoal. This is the gap the scenario–skill abstraction fills.

Lifting to scenarios and skills. The paper defines two abstractions that lift the low-level trajectory into a semantically meaningful structure:

Scenario $\sigma_t \in \Omega$: "a decision-relevant abstraction of the observation at a decision point of execution, and by construction serves as a sufficient statistic of the interaction history up to that point for the agent's next decision."

In plain language: a scenario is a semantic description of "where the agent is" in the problem-solving process — not the raw terminal output (which might be 500 lines of compilation errors), but the meaningful state that output implies (e.g., "Python project with failing unit tests," "empty Git repository," "server container with missing dependency"). The scenario captures what the agent needs to know to decide what to do next.

Skill $\kappa_t \in \mathcal{K}$: "an action subsequence applied at one scenario that produces a predictable transition to the next."

A skill is a coherent multi-step workflow — a sequence of low-level commands — that the agent executes to move from one scenario to the next. For example: "run pytest and capture output," "install missing package via pip," "create a Dockerfile for the service." Mathematically, a skill $\kappa_t$ is a function from one scenario to the next:

κt:σt−1→σt,κt=(ait,ait+1,…)\kappa_t : \sigma_{t-1} \rightarrow \sigma_t, \quad \kappa_t = (a_{i_t}, a_{i_t+1}, \ldots)

This is important because it captures the idea that skills are state transformers — they take the environment from a known state (the precondition scenario) to a new state (the postcondition scenario) through a predictable sequence of actions.

The trajectory decomposition (Equation 2). Under these definitions, any low-level trajectory $\zeta$ can be lifted to a higher-level execution trajectory $\xi$:

ξ=(σ0,κ1,σ1,κ2,…,κL,σL)∈(Ω×K)L×Ω\xi = (\sigma_0, \kappa_1, \sigma_1, \kappa_2, \ldots, \kappa_L, \sigma_L) \in (\Omega \times \mathcal{K})^L \times \Omega

where $\sigma_0$ is the initial scenario, $\kappa_1$ is the first skill applied, $\sigma_1$ is the resulting intermediate scenario, $\kappa_2$ is the second skill, and so forth, through $L$ skill applications ending in the final scenario $\sigma_L$.

What it computes: Given a raw trajectory of observations and actions, this decomposition segments it into a sequence of $L$ (scenario, skill) transitions. Each scenario captures the semantic state at a decision boundary; each skill captures the multi-step action sequence that transitions between scenarios. The output is a sequence of $2L+1$ elements (scenario–skill–scenario–skill–...–scenario).

Why this form: This decomposition separates the what (the semantic state) from the how (the action sequence), making it possible to analyze trajectory diversity along two independent axes — which scenarios were encountered and which skills were exercised. Without this abstraction, diversity analysis would operate on raw token sequences, where semantically identical states with different surface forms (e.g., different error messages indicating the same type of failure) would be counted as distinct, and semantically different states with similar surface forms would be conflated.

The learning objective (Equation 3). Under this abstraction, the terminal agent reduces to a policy over skills conditioned on scenarios. Training such an agent means maximizing the likelihood of selecting the correct skill $\kappa_t$ given the current scenario $\sigma_{t-1}$ and the goal $g$:

J(π)=Eξ∼D∑t=1Llog⁡π(κt∣σt−1,g)J(\pi) = \mathbb{E}_{\xi \sim \mathcal{D}} \sum_{t=1}^{L} \log \pi(\kappa_t \mid \sigma_{t-1}, g)

where $\mathcal{D}$ is the empirical distribution induced by the training trajectories, $\xi$ is a trajectory sampled from $\mathcal{D}$, $\kappa_t$ is the $t$-th skill in that trajectory, and $\sigma_{t-1}$ is the scenario before applying that skill.

What it computes: For each trajectory in the training set, sum the log-probabilities the agent assigns to each skill given the scenario it was applied in and the overall goal. Take the expectation over all training trajectories. The result is a scalar objective to maximize through training.

Why this form: This is the standard maximum-likelihood objective for sequential decision-making under the abstraction. It is equivalent to the standard next-token-prediction loss on the raw low-level interaction trajectories (as proven in Appendix A), meaning that no additional training machinery is needed — the abstraction is purely for analysis and design, not for implementation. The proof works by applying the chain rule of conditional probability: under the assumptions that the mapping $\zeta \mapsto \xi$ is deterministic, each scenario is a sufficient statistic, and each skill is executed autoregressively, the sum of skill-level log-probabilities telescopes down to the sum of token-level log-probabilities used in standard supervised fine-tuning.

The coverage decomposition (Equation 4) — the key insight. The critical step is decomposing this objective to reveal what the training data must cover. For a fixed goal $g$:

J(π)=∑σ∈ΩpD(σ∣g)∑κ∈KσpD(κ∣σ,g)log⁡π(κ∣σ,g)J(\pi) = \sum_{\sigma \in \Omega} p_{\mathcal{D}}(\sigma \mid g) \sum_{\kappa \in \mathcal{K}_\sigma} p_{\mathcal{D}}(\kappa \mid \sigma, g) \log \pi(\kappa \mid \sigma, g)

where $p_{\mathcal{D}}(\sigma \mid g)$ is the empirical probability of encountering scenario $\sigma$ during training (conditioned on goal $g$), $\mathcal{K}_\sigma \subseteq \mathcal{K}$ is the set of skills admissible at scenario $\sigma$, and $p_{\mathcal{D}}(\kappa \mid \sigma, g)$ is the empirical probability of applying skill $\kappa$ when in scenario $\sigma$.

What it computes: The expected log-likelihood decomposes into an outer sum over all possible scenarios (weighted by how often each scenario appears in training) and an inner sum over all skills admissible at each scenario (weighted by how often each skill is exercised when that scenario is encountered). The decomposition separates the contribution of scenario coverage ($p_{\mathcal{D}}(\sigma \mid g)$) from the contribution of skill coverage conditioned on scenario ($p_{\mathcal{D}}(\kappa \mid \sigma, g)$).

Why this form matters — the diversity criterion: This decomposition makes explicit the paper's central claim about diversity. The agent's policy $\pi(\kappa \mid \sigma, g)$ can only be learned for $(\sigma, \kappa)$ pairs that actually appear in the training data. If $p_{\mathcal{D}}(\sigma \mid g) = 0$ for some scenario $\sigma$, that scenario is never observed, and the agent cannot learn what to do when it encounters that state — regardless of how many total training examples exist. If $p_{\mathcal{D}}(\kappa \mid \sigma, g) = 0$ for some skill $\kappa$ at an observed scenario $\sigma$, that skill is never practiced in that context, creating a blind spot. Maximizing the agent's learned capability therefore requires training data whose empirical distribution $\mathcal{D}$ densely covers the conditional product space $\{(\sigma, \kappa) : \sigma \in \Omega, \kappa \in \mathcal{K}_\sigma\}$.

This formal criterion is what drives every subsequent design choice in SkillSynth. The skill graph is constructed to make the $\Omega \times \mathcal{K}$ space traversable. The path sampling algorithm is designed to push the empirical distribution toward uniformity over this space. The multi-agent harness is designed to faithfully instantiate sampled paths so that the intended $(\sigma, \kappa)$ coverage is realized in the collected trajectories. And the evaluation (Figure 1, Section 4.5) measures diversity precisely in terms of unique scenarios, skills, and scenario–skill pairs.

Skill Graph Construction: From Raw Skills to Traversable State Space

The skill graph is the foundational data structure of SkillSynth. It is a directed multigraph $G = (\Omega, \mathcal{K})$ where nodes $\Omega$ are scenarios (semantic descriptions of system states) and directed edges $\mathcal{K}$ are skills (each edge points from a skill's precondition scenario to its postcondition scenario). A path through this graph represents a coherent multi-step workflow — each skill's output state becomes the next skill's input state. The construction pipeline (Figure 3) has five sequential stages, each addressing a specific challenge in transforming raw, human-authored skill descriptions into a clean, traversable graph.

Stage 1: Skill filtering. The raw skill pool is sourced from ClawHub (OpenClaw, 2026), a community registry of AI agent skills, and from public GitHub repositories. ClawHub skills are particularly valuable because they represent "practical experience distilled from real terminal usage" — they are designed by practitioners to accomplish specific terminal tasks and include structured specifications (Markdown descriptions, code, and usage examples). However, not all skills in these sources are suitable for terminal task synthesis. The paper applies four filtering criteria, each motivated by the downstream goal of generating executable, safe, and verifiable task instances:

  1. Executability on a Linux terminal: Skills must actually interact with a Linux command-line environment. Skills that are purely prompt-engineering templates, web API wrappers, or GUI automation scripts are excluded because the downstream task instances run in Docker containers with only terminal access.

  2. Defined by a structured workflow, not prompt engineering alone: Skills must have a concrete implementation — a sequence of commands, scripts, or tool invocations — rather than being merely "describe how to solve X" prompts. This ensures that when the multi-agent harness instantiates a task requiring a skill, there is an executable oracle solution to verify against.

  3. Free of adversarial or jailbreak content: Skills that download files from unknown IPs, exfiltrate environment keys, or perform other potentially harmful actions are excluded. The paper gives specific examples: "downloading files from unknown IPs, exfiltrating environment keys." This is a safety constraint for the synthesized task environments.

  4. Producing deterministic, objectively verifiable outputs: The outcome of executing the skill must be checkable by an automated test script. Skills whose success is subjective (e.g., "improve code readability" without objective metrics) are excluded because the oracle verification step in the multi-agent harness requires a ground-truth pass/fail signal.

The filtered skill pool $\mathcal{K}$ contains 57,214 skills after this stage, each with its full specification (description, implementation code, usage examples) intact.

Stage 2: Scenario inference. For each retained skill $\kappa \in \mathcal{K}$, the system must determine what states it can start from (preconditions) and what states it produces (postconditions). These scenarios are not part of the raw skill specifications — ClawHub skills describe what they do but not the systematic pre/post-condition state space. The paper extracts this information by prompting an LLM (DeepSeek Reasoner v3.2) with the skill's full specification:

For each retained skill $\kappa \in \mathcal{K}$, we prompt an LLM with its full specification (Markdown description, code, and usage examples) to infer plausible precondition scenarios $\Omega^{\text{pre}}_\kappa$ and postcondition scenarios $\Omega^{\text{post}}_\kappa$, yielding atomic transitions $\{\kappa : \sigma \rightarrow \sigma' \mid \sigma \in \Omega^{\text{pre}}_\kappa, \sigma' \in \Omega^{\text{post}}_\kappa\}$.

The key design choice here is that a single skill can have multiple precondition and postcondition scenarios — not just one each. For example, a "git commit" skill might apply when in a scenario like "Git repository with staged changes" or "Git repository with unstaged changes" (with different preconditions implying different initial steps within the skill), and it might produce scenarios like "committed changes with clean working tree" or "committed changes with remaining unstaged files." Capturing this multiplicity is essential for the graph to represent the true variety of real-world state transitions.

The LLM infers these scenarios as natural-language descriptions — typically 1–2 sentence summaries like "Python project with a virtual environment and a failing test suite" or "Docker container with a web server running on port 8080."

Why DeepSeek Reasoner v3.2 specifically? The paper states that all LLM calls in graph construction use this model "to ensure extraction quality and reliability." The choice of a reasoning model (rather than a standard instruction-tuned model) is motivated by the need for careful semantic analysis — determining whether two natural-language state descriptions refer to the same real-world condition requires reasoning about equivalence, not just surface-form matching.

Stage 3: Scenario deduplication. The scenario inference stage produces a large collection of natural-language scenario descriptions across all skills. Many of these describe the same underlying state but in different words — for instance, "Python project with failing unit tests" and "Python codebase where pytest returns errors" refer to essentially the same scenario. If these duplicates are kept separate, the graph becomes fragmented: skills that should connect (because their postcondition matches another skill's precondition) fail to do so because the textual descriptions differ.

The paper approaches this as a semantic clustering problem: embed all scenarios into a vector space, then group semantically equivalent descriptions. However, the scale is substantial (the final graph contains 82,073 scenarios after merging), and naive clustering approaches have specific failure modes:

  • Flat clustering (e.g., k-means): Requires prespecifying the number of clusters, which is unknown a priori for a scenario space that spans dozens of domains with varying granularity.

  • Global hierarchical clustering (e.g., complete-linkage agglomerative clustering on all pairs): Has $O(n^2)$ memory cost, which is prohibitive for $n$ in the tens of thousands. Moreover, complete linkage is desirable (because it ensures every pair within a cluster is mutually close, preventing "chain-drift" where A is close to B, B is close to C, but A and C are semantically distant), but naively it scales poorly.

  • Embedding similarity thresholding: Risk of both over-merging (different states that happen to have similar embeddings get collapsed) and under-merging (same state with slightly different wording gets kept separate).

The paper's solution is a two-stage scalable hierarchical procedure that combines the scalability of community detection with the quality guarantees of complete-linkage clustering:

Stage 3a: Coarse bucketing via Louvain community detection. First, the system constructs a sparse semantic similarity graph over the normalized scenario embeddings (the paper uses Microsoft/Harrier-OSS-v1-27B for embedding). Edges connect scenario pairs whose cosine similarity exceeds a threshold, creating a graph where dense sub-communities correspond to groups of semantically similar scenarios. The Louvain community detection algorithm (Blondel et al., 2008) partitions this graph into coarse buckets. Louvain is chosen because it scales well to large graphs (near-linear time complexity in practice) and produces communities that reflect the natural modular structure of the similarity graph.

Stage 3b: Fine-grained clustering via complete-linkage agglomerative clustering within buckets. Within each Louvain bucket, the system runs complete-linkage agglomerative clustering with cosine distance. Because each bucket is much smaller than the full scenario set (typically hundreds rather than tens of thousands of scenarios), the $O(n^2)$ memory cost of complete linkage is manageable. Complete linkage is specifically chosen because it enforces that every pair of scenarios within a cluster must be mutually close — preventing the chain-drift artifact where transitivity across intermediate neighbors pulls semantically distant scenarios into the same cluster.

Hyperparameter tuning. The agglomerative clustering has a distance threshold hyperparameter that controls how aggressively scenarios are merged. The paper describes the tuning process:

"For the clustering hyperparameters, we sweep the agglomerative distance threshold on held-out scenario samples and manually inspect the resulting clusters. We choose the final threshold by balancing merge quality and over-fragmentation: the selected value should merge clear paraphrases and near-equivalent states while keeping semantically distinct states, especially negations and pre/post condition changes, in separate clusters."

This manual inspection is critical because the semantics of scenarios are subtle: "Docker container running" and "Docker container stopped" have high embedding similarity (they share many words) but represent opposite pre/post conditions that must stay in separate clusters. The distance threshold must be set conservatively enough to avoid merging such negations.

Why this two-stage approach over alternatives? The paper states they "evaluated nine common clustering algorithms and found that a hierarchical agglomerative clustering method with Louvain-based coarse bucketing performs best empirically." The two-stage design solves the scalability–quality tension: Louvain provides scalable coarse grouping, and complete-linkage within groups provides the quality guarantee that prevents semantic drift.

The output of this stage is a set of deduplicated scenarios, where each cluster has been merged into a single canonical scenario description.

Stage 4: Cross-skill alignment. At this point, the system has a large collection of atomic transitions — individual skills pointing from their (deduplicated) precondition scenarios to their (deduplicated) postcondition scenarios. However, these transitions are isolated: there is no connection between skill A's postcondition and skill B's precondition unless the system explicitly identifies that the two scenarios are semantically compatible.

The alignment stage builds these connections. For each postcondition scenario of any skill, the system retrieves the top 1,000 most similar precondition scenarios across all skills (using embedding similarity), and an LLM (DeepSeek Reasoner v3.2 again) judges semantic compatibility — does this postcondition describe a state from which this precondition could plausibly follow?

Crucially, the paper runs this process bidirectionally:

"We repeat this process in reverse (precondition → top-1,000 postconditions) using separately designed prompts to ensure bidirectional alignment quality."

The bidirectional check is a quality safeguard: a postcondition might be similar to a precondition by embedding similarity (e.g., both mention "Python project") but actually represent incompatible states (e.g., one is "project with passing tests" and the other requires "project with failing tests" as a starting point). Running the alignment in both directions with separately designed prompts makes the compatibility judgment more robust than a single-direction embedding-similarity threshold.

At scale, this produces 185,529 LLM-verified bridges — connections between scenarios across different skills that indicate compatibility for sequential execution.

Why top-1,000 and not all pairs? The full cross-product of postconditions $\times$ preconditions would be $O(|\Omega|^2)$, which for 82,073 scenarios is billions of pairs — prohibitively expensive to run through an LLM. The top-1,000 retrieval by embedding similarity is a computationally efficient filter that captures the vast majority of genuinely compatible pairs (since semantically compatible scenarios will have similar embeddings) while dramatically reducing the number of LLM calls needed.

Stage 5: Scenario merging and filtering. The aligned pre- and postcondition pairs that pass the compatibility check are then merged into unified scenario nodes. An LLM takes the pair of scenario descriptions and produces a single merged description that represents the shared state. For example, if skill A's postcondition is "Python project with pytest installed and passing test suite" and skill B's precondition is "Python codebase ready for deployment with all tests green," the merged scenario might be "Python project with all tests passing and ready for production deployment."

Finally, the system performs an LLM-based filtering pass over all resulting (scenario, skill, scenario) triples, "retaining only those that form valid transitions." This is a final quality check that catches edge cases where the merging or alignment stages produced nonsensical connections. The paper notes:

"Through manual review of sampled graph cases, we find that each stage is necessary to ensure the overall quality of the constructed graph."

Graph statistics. The constructed graph (Table 6, Appendix D) has the following properties:

  • 82,073 scenario nodes after deduplication and merging.
  • 57,214 skill-labeled transitions (directed edges). Note that the number of edges is smaller than the number of nodes — this is a sparse graph where many scenarios are sources or sinks with limited connectivity.
  • Scenario role distribution: 18,749 scenarios (22.8%) are source-only (they appear only as preconditions and are never anybody's postcondition — these are "entry points" into the graph). 12,299 scenarios (15.0%) are sink-only (only postconditions, "exit points"). 46,699 scenarios (56.9%) are bridge scenarios that serve as both preconditions and postconditions — these are intermediate states that can appear mid-workflow.
  • Degree distribution is heavy-tailed (Figure 6): The mean degree is 4.32, the median is 2, and the maximum is 752. A small number of "hub" scenarios have extremely high connectivity — these correspond to generic intermediate states that are compatible with many different skills (e.g., "clean Ubuntu environment with standard system tools installed").
  • Connected components: The graph has 6,251 connected components. One giant component contains 118,806 nodes (85.6% of all scenarios), confirming that the cross-skill alignment successfully chained the majority of skills into a single traversable subgraph. The remaining 6,250 components are small (mostly 2–10 scenarios), representing specialized, self-contained workflows that do not connect to the broader graph.
  • Domain coverage (Figure 5): The skill categories span 26 domains, from common ones like "Coding Agents & IDEs" (11.9% of skills), "General Automation & Utilities" (9.4%), and "DevOps & Cloud Infrastructure" (7.8%) through long-tail domains like "Audio & Speech" (1.2%), "Games, 3D & Simulation" (1.0%), and "IoT, Hardware & Robotics" (0.4%). This breadth confirms that the graph captures real terminal usage beyond software engineering — encompassing multimedia processing, scientific computing, system administration, and creative workflows.

Why build the graph this way rather than generating it end-to-end with an LLM? The paper explored "several alternative graph construction strategies, including embedding-based scenario alignment and single-pass subgraph generation" and found that "the LLM-based pairwise alignment approach yields higher-quality graphs." End-to-end generation would require an LLM to produce a coherent graph with thousands of nodes and edges in a single prompt — a context-length and coherence challenge that is likely to produce hallucinated connections, missing transitions, and inconsistent scenario semantics. The staged approach — infer, deduplicate, align, merge, filter — decomposes the problem into manageable subtasks, each verifiable independently.

Graph-Guided Path Sampling: Maximizing Coverage of the Scenario–Skill Space

With the skill graph constructed, the next stage is to sample directed paths from it. Each path $\mathcal{P}$ serves as a compositional blueprint for the multi-agent harness: it specifies exactly which skills must be applied in which order, and which intermediate scenarios the agent should encounter. A path is formally:

P=(σ0,κ1,σ1,κ2,…,κL,σL)\mathcal{P} = (\sigma_0, \kappa_1, \sigma_1, \kappa_2, \ldots, \kappa_L, \sigma_L)

where $\sigma_0$ is the starting scenario, $\kappa_1$ is the first skill (directed from $\sigma_0$ to $\sigma_1$ in the graph), $\sigma_1$ is the scenario after skill 1, and so forth through $L$ skills. Paths with length $L$ in the range $[L_{\text{min}}, L_{\text{max}}]$ are retained; the paper sets $L_{\text{min}} = 1$ and $L_{\text{max}} = 7$, covering single-skill tasks ($L \in \{1, 2, 3\}$) through compositional multi-step ones ($L \geq 4$).

Why not just sample uniformly at random? A uniform random walk on the graph — start at a random node, pick a random outgoing edge, repeat — has a fundamental flaw: it concentrates on high-degree nodes and frequently traversed edges. Because the degree distribution is heavy-tailed (Figure 6), a uniform walk will repeatedly visit the same hub scenarios (e.g., "clean Ubuntu environment") and the same popular skills, producing redundant paths that cover a tiny fraction of the $\Omega \times \mathcal{K}$ space. The sampled paths would exhibit exactly the redundancy the paper identifies as problematic in existing datasets (Figure 1).

Inverse-frequency path sampling (Algorithm 1). The paper introduces a sampling procedure that explicitly pushes the empirical distribution toward uniform coverage of the scenario–skill space. The core idea is to track how often each scenario and each skill has been sampled, and then to bias future sampling toward rarely-visited elements.

The algorithm maintains two counters:

  • $\nu(\sigma)$: the number of times scenario $\sigma$ has appeared in any accepted path so far.
  • $\mu(\kappa)$: the number of times skill $\kappa$ has appeared in any accepted path so far.

Both counters are initialized to zero. For each sampling attempt (up to a budget of $N$ attempts, where $N = 3,721$ in the paper's run):

Step 1: Sample the source scenario with inverse-frequency weighting. The starting node $\sigma_0$ is drawn with probability:

p(σ)∝(ν(σ)+1)−1p(\sigma) \propto (\nu(\sigma) + 1)^{-1}

The $+1$ prevents division by zero for unseen scenarios (where $\nu(\sigma) = 0$). This means scenarios that have never been sampled have the highest probability of being chosen, while frequently-visited hub scenarios are progressively downweighted.

Step 2: Walk the graph with inverse-frequency edge sampling and monotone progression. Starting from $\sigma_0$, at each step $l$, the algorithm looks at all outgoing edges from the current node $\sigma_l$ that point to unvisited scenarios (within this path) and represent unvisited skills (within this path):

N(σl)={κ∈K:κ:σl→σ′ for some σ′∈Ω∖Vσ,κ∉Vκ}\mathcal{N}(\sigma_l) = \{\kappa \in \mathcal{K} : \kappa: \sigma_l \rightarrow \sigma' \text{ for some } \sigma' \in \Omega \setminus \mathcal{V}_\sigma, \kappa \notin \mathcal{V}_\kappa\}

where $\mathcal{V}_\sigma$ is the set of scenarios already visited in this path, and $\mathcal{V}_\kappa$ is the set of skills already used in this path. If $\mathcal{N}(\sigma_l)$ is empty, the walk terminates (dead-end).

From the available outgoing edges, the next skill $\kappa_{l+1}$ is sampled with probability:

p(κ)∝(μ(κ)+1)−1p(\kappa) \propto (\mu(\kappa) + 1)^{-1}

This biases toward skills that have been rarely used across all sampled paths so far. The next scenario $\sigma_{l+1}$ is then chosen from the postconditions of $\kappa_{l+1}$ that have not yet been visited in this path, with probability:

p(σ)∝(ν(σ)+1)−1p(\sigma) \propto (\nu(\sigma) + 1)^{-1}

Step 3: Check path validity and update counters. The walk continues until it reaches $L_{\text{max}}$ or hits a dead-end. If the resulting path length $l$ is within $[L_{\text{min}}, L_{\text{max}}]$ and the set of skills in the path $\text{skills}(\mathcal{P})$ has not been seen before in any previously accepted path, the path is accepted. The counters are incremented:

ν(σ)←ν(σ)+1for each σ in P\nu(\sigma) \leftarrow \nu(\sigma) + 1 \quad \text{for each } \sigma \text{ in } \mathcal{P} μ(κ)←μ(κ)+1for each κ in P\mu(\kappa) \leftarrow \mu(\kappa) + 1 \quad \text{for each } \kappa \text{ in } \mathcal{P}

If the path is rejected (too short, too long, or duplicate skill set), the counters are not updated, and the next attempt starts fresh.

What it computes: Given the skill graph, a length range, and a sampling budget, this algorithm produces a set $\Pi$ of unique paths where each path is a sequence of scenario–skill–scenario transitions. The inverse-frequency weighting ensures that the paths collectively cover a wide range of scenarios and skills rather than concentrating on common ones. The monotone progression constraint (no revisiting scenarios or skills within a single path) ensures that each path represents a genuine forward workflow rather than cycling within a small subgraph.

Why this form — five design choices explained:

1. Inverse-frequency rather than uniform: A uniform sampler would produce paths dominated by hub scenarios and popular skills because these nodes have the most incoming and outgoing edges. Inverse-frequency weighting actively fights this concentration, steering the sampler toward the long tail of the distribution. This directly implements the coverage criterion from Equation 4: to maximize the learned policy's coverage, the training data's empirical distribution should be as close to uniform over $\Omega \times \mathcal{K}$ as possible.

2. Global counters rather than within-path counters: The counters $\nu$ and $\mu$ are incremented across all accepted paths, not reset per path. This means the algorithm learns and adapts over the course of the sampling run — early paths will sample from a relatively uniform initial distribution, while later paths will be increasingly biased toward elements that have been rarely or never sampled. This progressive steering is what generates the diversity gain over a static sampling distribution.

3. Monotone progression constraint: Within a single path, the algorithm excludes already-visited scenarios ($\Omega \setminus \mathcal{V}_\sigma$) and already-used skills ($\kappa \notin \mathcal{V}_\kappa$) from future steps. This prevents the path from looping — revisiting a scenario or reapplying a skill that has already been used — which would produce non-progressive workflows where the agent retreads ground rather than making forward progress. In real terminal tasks, workflow steps are monotone: you don't install the same package twice or configure the same setting repeatedly within a single task. The constraint ensures the sampled paths are plausible abstractions of real workflows.

4. Dead-end handling: If a node has no outgoing edges that lead to unvisited scenarios and unvisited skills, the walk terminates early rather than forcing a continuation. Paths that terminate below $L_{\text{min}}$ are discarded without updating counters, so they do not penalize the nodes they visited. This means "bad" partial paths do not consume sampling budget from the counters, preserving the diversity incentive.

5. Uniqueness constraint on skill sets: The condition $\text{skills}(\mathcal{P}) \notin \mathcal{S}$ (where $\mathcal{S}$ is the set of skill sets seen in previously accepted paths) ensures that no two accepted paths have exactly the same combination of skills, even if they traverse different scenarios. This prevents redundancy at the skill-composition level — two paths that use the same three skills in the same order but with different intermediate scenarios would exercise the same skill transitions from the agent's perspective, contributing less marginal diversity than completely novel skill combinations.

Scale of the sampling space. The paper enumerates 16,632,220 paths requiring seven or more skills from the constructed graph, confirming a "vast combinatorial space for task synthesis." The 3,721 sampled paths in the experimental run represent a tiny fraction of this space, highlighting the importance of the inverse-frequency sampling to select a maximally diverse subset rather than a random sample that would concentrate on common substructures.

Length distribution. The sampled paths range from $L = 1$ (single-skill tasks) to $L = 7$ (seven-skill compositional tasks). Shorter paths ($L \in \{1, 2, 3\}$) produce simple tasks suitable for basic skill practice, while longer paths ($L \geq 4$) produce genuinely compositional tasks that require chaining multiple skills through progressively changing intermediate states — the kind of workflow that real terminal users encounter when, for example, setting up a development environment (install dependencies → configure tools → clone repository → run tests → fix failures → deploy).

Multi-Agent Harness: From Sampled Path to Verified Executable Task Instance

The multi-agent harness is the final stage of SkillSynth's synthesis pipeline. It takes a sampled path $\mathcal{P}$ (an abstract sequence of scenarios and skills) and produces a fully executable terminal task instance that an agent can actually solve. A task instance consists of five concrete components:

  1. Natural-language instruction (instruction.md): A textual description of the goal $g$, written to be self-contained and unambiguous, telling the agent what to accomplish without revealing the solution strategy.

  2. Initial filesystem snapshot (environment/Files/): The starting state of the workspace, including any files, directories, configuration, or data the agent needs to begin the task.

  3. Containerized environment (environment/Dockerfile): A Docker specification that defines the execution environment — which operating system, which packages are installed, which tools are available. This ensures reproducibility and isolation.

  4. Verification scripts (test/test.py and test/test.sh): Automated tests that check whether the agent's solution satisfies the task requirements. These are the implementation of the external verifier $\mathcal{V}$.

  5. Oracle solution (solve.sh): A reference implementation that correctly solves the task, used to verify that the task is solvable (by running it through the verification scripts).

The planner–constructor architecture. A natural approach would be to prompt an LLM with the sampled path and ask it to generate all five components in a single pass. The paper identifies two problems with this naive approach:

"Directly prompting an LLM to generate all components in a single pass leads to two issues: 1) long-context generation produces outputs of inconsistent quality, and 2) the model focuses on implementation details rather than designing a coherent task instance, resulting in tasks that lack sufficient complexity."

The first issue is a standard challenge with long-form LLM generation — as the output grows, quality degrades, details get dropped, and consistency across components (e.g., instruction aligning with tests) becomes unreliable. The second issue is more subtle: when an LLM is asked to simultaneously design what the task is and implement its components, it tends to collapse into implementation mode, producing tasks that are easy to implement (few steps, straightforward file operations) rather than tasks that require genuine multi-step reasoning.

The paper's solution is to decouple planning from implementation:

Planner (🧠 in Figure 2b). Given the sampled path $\mathcal{P}$, the planner's job is to design the task at a semantic level without getting bogged down in implementation details. The paper describes its output as "a structured plan of sub-objectives and expected outputs." For the video-processing example in Figure 4, the planner might specify: (1) the task is about converting a raw video into a GIF summary; (2) sub-objective 1: analyze the video and extract metadata; (3) sub-objective 2: extract representative frames at chosen intervals; (4) sub-objective 3: compose frames into an animated GIF with metadata; (5) the final deliverable is a directory containing the GIF, sampled frames, and a metadata file. The planner's output is a mid-level specification — concrete enough to guide implementation but abstract enough to leave implementation freedom.

Constructor (⚙️ in Figure 2b). The constructor takes the planner's structured plan and generates all five task components. Because the plan already specifies the structure, the constructor can focus on implementation quality without needing to simultaneously design the task architecture. The constructor is a tool-augmented LLM — it has access to file manipulation tools (shown in Figure 2b as read_file, overwrite_file, str_replace, delete_file, create_file) that allow it to interact with a workspace to create the initial filesystem snapshot, write Dockerfiles, and implement test scripts. This tool use is essential because many task components are concrete artifacts (files, scripts) that cannot be generated purely as text — the constructor must actually write files, test that they execute, and iterate.

Dual verification with repair loop. After the constructor produces a candidate task instance, it passes through two independent verification checks, implemented in sequence:

Verification 1: Execution-based oracle check (Harbor Oracle in Figure 2b). The system runs the oracle solution (solve.sh) inside the task's Docker container and executes the verification scripts (test.py or test.sh). If the verification passes (exit code 0, all assertions satisfied), the task is confirmed solvable — there exists at least one correct solution, and the verification scripts correctly recognize it. If the verification fails, it indicates either (a) the oracle solution is buggy, (b) the environment is misconfigured (e.g., missing dependencies), or (c) the verification scripts are incorrect. In any case, the task is not usable for training (an agent cannot succeed if even the reference solution fails), so it must be repaired or discarded.

Verification 2: Rubric-based LLM evaluation (Rubric Eval in Figure 2b). Even if the oracle passes, the task might have quality issues that would contaminate training. The rubric evaluation uses an LLM-as-Judge to assess two specific quality dimensions:

  • Instruction–test alignment: Do the verification scripts test exactly what the instruction asks for, and nothing else? Tests that are too lenient (missing required functionality — the agent could pass without actually solving the task) produce false positives that reward incorrect behavior. Tests that are too strict (imposing unstated constraints — checking for things the instruction never mentioned) produce false negatives that penalize correct solutions. Both types of misalignment produce inaccurate evaluation of agent trajectories, which is especially harmful for reinforcement learning (where the reward signal comes from the verifier).

  • Instruction self-containedness: Does the instruction stand alone without leaking hints about the oracle solution? If the instruction says "use ffmpeg to extract frames" when the oracle solution uses ffmpeg, it is giving away the solution strategy rather than letting the agent discover it. Self-contained instructions are essential for training agents that learn to reason about what tools to use, rather than pattern-matching specific tool names from the instruction.

The paper reports that 77% of rubric failures stem from misaligned test scripts, indicating that test generation is the harder quality challenge.

The repair loop. If either verification check fails, the task instance is not immediately discarded. Instead, "diagnostic feedback is returned to the constructor for repair via multi-turn tool use." The constructor receives the specific failure information — which tests failed, which rubric criteria were violated, and why — and can modify the task components to address the issues. This repair cycle is bounded:

  • Maximum repair cycles: $R = 3$. If a task instance still fails after three repair attempts, it is either accepted as-is (if only rubric checks fail — the paper retains these for supervised fine-tuning to preserve trajectory diversity but excludes them from reinforcement learning to avoid erroneous reward signals) or discarded (if oracle checks fail — unsolvable tasks are never usable).
  • Maximum tool calls per cycle: $N_{\text{tool}} = 20$. Each repair cycle costs at most 20 tool interactions (file reads, writes, edits, test executions), bounding the computational cost of repair.

The statistics in Table 1 demonstrate the repair loop's effectiveness: 721 task instances (out of 3,721 attempted, or 19.4%) were recovered after failing the first round. The average repair cycle count was 2.31, and the average tool calls per task was 11. Without the repair loop, these 721 instances would have been lost, reducing the usable task yield from 95.7% to approximately 76.3%.

Why repair is necessary at scale. The paper notes that "poor first-round generations are hard to recover" — the dominant cause of oracle failures is "corrupted filesystem snapshots" where "due to the randomness of LLM generation, the first synthesis round may produce buggy stages that remain unrecoverable even after multiple repair cycles." This suggests a floor on repair effectiveness: some initial generations are so flawed that no amount of iterative fixing can salvage them, because the constructor's subsequent repair attempts build on a corrupt foundation. For these cases, the paper suggests "re-running failed paths with a higher sampling temperature" as a simple recovery strategy — essentially re-rolling the initial generation rather than trying to fix a hopelessly broken one.

Yield and cost. A single fully automated run of SkillSynth:

  • Samples 3,721 paths from the skill graph.
  • Passes them through the multi-agent harness.
  • Produces 3,560 usable task instances (after removing oracle-failed instances among the 3,423 that pass both checks and the 137 that pass oracle but fail rubric).
  • Achieves a 95.7% oracle pass rate (3,423 + 137 = 3,560 oracle-passing instances out of 3,721 attempts).
  • At an average cost of $27.3 per verified task instance.

The cost figure requires interpretation because the paper does not break down which model API calls dominate. Given that the multi-agent harness uses LLMs for planning, construction, rubric evaluation, and repair, the $27.3 likely reflects the cumulative API cost across all stages, with the repair cycles adding to the cost for the 721 instances that required them.

Task difficulty distribution (Table 2). The 3,560 usable instances are categorized by giving each to Hy3 Preview (a strong proprietary terminal agent from Tencent) for three independent solution attempts, then binning by success count:

  • 0/3 success (38% = 1,352 tasks): The most difficult tier. Even a strong agent fails on all three attempts. These tasks represent challenging problems that are currently beyond state-of-the-art capabilities — useful as stretch goals for future training.
  • 1/3 success (18% = 637 tasks): Difficult but occasionally solvable. These are in the "learnable range" where an agent sometimes succeeds, making the trajectories valuable for training (successful trajectories show what works; failed trajectories show error recovery patterns).
  • 2/3 success (19% = 679 tasks): Moderately difficult — solvable with some consistency but not trivially. Also in the learnable range.
  • 3/3 success (25% = 892 tasks): Easy tasks that the agent consistently solves. While less valuable for pushing capability boundaries, these provide positive examples that reinforce correct skill application.

The fact that 38% of tasks receive 0/3 success and another 37% are partially solvable (1/3 or 2/3) confirms that graph-guided synthesis produces genuinely challenging problems — not just trivial single-step tasks that any agent can solve. This difficulty distribution stands in contrast to single-skill baselines (Table 4), where only 16% of tasks are in the 0/3 tier and 34% are in the 3/3 tier, indicating that single-skill synthesis produces predominantly easy tasks.

An end-to-end example (Figure 4). The paper provides a concrete illustration of the full pipeline for a video-processing task. The sampled path from the skill graph specifies three skills in sequence:

  • Skill 1: Video Analyzer — takes a scenario "raw video footage" and transitions to "video session with metadata"
  • Skill 2: Frame Extractor — takes the video session and transitions to "extracted frames with timestamps"
  • Skill 3: GIF Generator — takes the extracted frames and transitions to "generated GIF with output metadata"

Each skill expands into a multi-step internal workflow (5 steps each in this example). The multi-agent harness translates this abstract path into the concrete instruction: "Turn the provided raw video into a GIF summary package with sampled frames, metadata, and the final animated GIF." This is a genuinely compositional task — the agent cannot skip to making a GIF without first analyzing the video and extracting frames, because the intermediate outputs of each step are required inputs for the next.

Why this architecture over single-stage generation? The planner–constructor split addresses the two identified failure modes of single-stage generation. The planner ensures task coherence by designing the task structure before anyone writes a line of code — preventing the "collapse into implementation" problem where the LLM focuses on easy-to-implement details rather than task design. The constructor ensures implementation quality by focusing exclusively on executing a pre-specified plan, with tool access that enables iterative file creation and testing. And the dual verification with repair ensures that every shipped task instance is both solvable (oracle check) and well-specified (rubric check), which is essential when the synthesized tasks are used to generate training trajectories — a buggy task environment would produce garbage trajectories that degrade, rather than improve, the trained agent.

Design choice: why retain rubric-failed instances for SFT? The paper retains task instances that pass oracle verification but fail rubric checks for supervised fine-tuning. The rationale is that for SFT, the training signal comes from the collected trajectories themselves (the agent learns to imitate the teacher's actions), not from the verifier's reward signal. If the tests are slightly misaligned with the instruction but the task is still solvable and the teacher agent produces reasonable trajectories, the training data is still valuable for exposing the model to diverse scenarios and skills. For reinforcement learning, however, the verifier IS the reward signal — misaligned tests would provide erroneous rewards that could actively degrade the policy. Hence the bifurcation: rubric-failed tasks go to SFT only; fully-passing tasks go to both SFT and RL.

Design choice: why 3 repair cycles and 20 tool calls? These hyperparameters represent a cost–quality tradeoff. More repair cycles might recover additional failed instances, but at diminishing returns (the paper notes that corrupted filesystem snapshots are often unrecoverable regardless of cycles). More tool calls per cycle give the constructor more iterative refinement capacity but increase the per-task synthesis cost. The specific values (R=3R = 3, Ntool=20N_{\text{tool}} = 20) were likely determined empirically to balance yield against API cost, though the paper does not report an ablation over these values.

4. Key Insights and Innovations

Innovation 1: Trajectory Diversity as an Optimizable Quantity Rather Than a Hoped-For Side Effect

The paper's most fundamental intellectual contribution is transforming trajectory diversity from a vague desideratum into a precise, optimizable quantity with a direct connection to the training objective. This is not a new mechanism — it is a new framing that changes how the field should think about synthetic data generation for agents.

What was the prior assumption? The dominant paradigm in synthetic data for terminal agents assumed that if you generate enough diverse task instances (by expanding domain taxonomies, collecting varied repositories, or inverting environments), the resulting execution trajectories will naturally be diverse as well. Endless Terminals (Gandhi et al., 2026) generates tasks across diverse domains; TermiGen (Zhu et al., 2026) aims for high-fidelity environments; Nemotron-Terminal (Pi et al., 2026) scales task count. All implicitly operate on the assumption that task diversity → trajectory diversity. Figure 1 empirically demolishes this assumption: existing datasets exhibit significant redundancy in both scenario coverage and skill usage despite containing superficially different tasks. Different task instances repeatedly expose the agent to the same narrow set of intermediate states and the same subset of skills.

What is the new framing? The paper's Section 2 derivation — specifically the decomposition in Equation 4 — reveals why the prior assumption fails and what should be optimized instead. The training objective decomposes into two multiplicative factors: how often each scenario appears in training, and how often each skill is exercised in each scenario. The agent's policy is literally incapable of learning for any (scenario, skill) pair that does not appear in the training distribution, regardless of how many total examples exist. This is not an empirical observation — it follows directly from the maximum-likelihood objective. The implication is profound: scaling task count without explicitly controlling the coverage of the Ω × K product space is fundamentally inefficient, because redundant (scenario, skill) pairs contribute zero additional learning capacity. This is the inference-time analog of a well-understood pretraining principle — that duplicate training examples provide diminishing returns — but applied to the two-dimensional space of agent states and actions rather than the one-dimensional space of text tokens.

Why this is fundamental rather than incremental. This framing is not a refinement of existing synthesis methods; it redefines the synthesis target. Prior work asked "how do we generate more varied task descriptions?" SkillSynth asks "how do we generate trajectories whose empirical distribution densely covers the Ω × K product space?" The shift from task diversity to trajectory diversity — and specifically to coverage of the scenario–skill joint space — is a conceptual reorientation that makes diversity optimizable rather than emergent. The inverse-frequency path sampling algorithm (Algorithm 1) is not an arbitrary diversity heuristic; it is a direct implementation of the coverage criterion derived from Equation 4, progressively steering the empirical path distribution toward uniformity over Ω × K. The downstream performance gains in Tables 3 and 4 are evidence that this theoretical framing translates into practical improvements, but the framing itself is the innovation — it provides a language and a target for future work on synthetic agent data that did not previously exist.

Evidence anchor. Figure 1 shows the measurement of this quantity (unique scenarios, skills, and scenario–skill pairs) in existing datasets and SkillSynth, demonstrating both the problem and the solution in the same metric. Table 4 shows that controlling for total task count and synthesis harness, graph-guided sampling produces 3.0–8.4 point improvements on Terminal-Bench over random composition — confirming that diversity explicitly engineered into the Ω × K space produces better agents than diversity that emerges by chance.

Innovation 2: Scenarios as First-Class Architectural Objects in Task Synthesis

The paper makes a distinctive architectural choice that separates it from both the terminal agent synthesis literature and the broader skill organization literature: scenarios — descriptions of intermediate system states — are elevated to first-class nodes in the graph, co-equal with skills as edges. This is not an obvious design choice, and understanding why it matters requires contrasting it with how prior work handles intermediate states.

What did prior work do with intermediate states? In the terminal synthesis literature (Wu et al., 2026; Lin et al., 2026; Pi et al., 2026; Zhu et al., 2026), the intermediate states an agent encounters during task execution are an emergent property of the task definition — they are whatever states happen to occur when an agent attempts to solve the task. The system designer specifies the initial state (via a Dockerfile and filesystem snapshot) and the goal (via a natural language instruction), and the intermediate states are whatever the agent produces along the way. There is no mechanism for constraining or designing which intermediate states the agent will encounter.

In the skill organization literature, the situation is different but equally revealing. AgentSkillOS (Li et al., 2026) organizes skills into DAG-based orchestration graphs where edges represent execution dependencies — skill B depends on skill A's output. SkillNet (Liang et al., 2026) models skills as nodes in a relational graph with explicit inter-skill connections. In both cases, the graph's structure represents relationships between skills, not between states. The intermediate states that connect skills are implicit — they are whatever state must exist for one skill to follow another — but they are not modeled as explicit objects with semantic descriptions, deduplication, or compatibility verification.

What does SkillSynth do differently? The skill graph G = (Ω, K) places scenarios as nodes and skills as directed edges between them. This is a conceptual inversion: rather than organizing skills and having scenarios emerge implicitly, the graph explicitly represents the state space and treats skills as transitions through it. The practical consequence is that compatibility between skills is determined not by skill-to-skill similarity but by scenario compatibility — can skill B's precondition scenario be aligned with skill A's postcondition scenario? This is a more expressive and verifiable compatibility criterion because it captures the actual semantic relationship that matters for sequential execution: the output state of one operation must match the input state of the next.

Why this matters beyond the graph structure. The scenario-first design enables three capabilities that skill-first graphs cannot provide:

  1. Explicit control over trajectory diversity along both dimensions. If the graph only modeled skills, a sampled path would specify which skills but not which intermediate states. The resulting training trajectories would exercise the right skills but could traverse arbitrary intermediate scenarios — losing control over the scenario-coverage factor in Equation 4. By making scenarios explicit nodes, the sampled path jointly specifies both axes, and the multi-agent harness can instantiate the task to realize both the skill sequence and the intermediate scenarios.

  2. Compatibility verification through semantic state matching. When the cross-skill alignment stage (Section 3.2, Stage 4) judges whether one skill can follow another, it operates on scenario descriptions — "does this postcondition describe a state from which this precondition could plausibly follow?" — rather than on skill names or categories. This is a richer compatibility signal that can identify connections between skills in entirely different domains that happen to share an intermediate state (e.g., a "data export" skill producing a CSV file could connect to a "data visualization" skill expecting a CSV input, even though the skills belong to different categories). A skill-to-skill graph would miss such cross-domain connections unless they were explicitly encoded.

  3. Deduplication of semantically equivalent states across skills. The scenario deduplication stage (Stage 3) collapses different textual descriptions of the same underlying state into a single canonical node. This is what enables the graph to be traversable — if "Python project with passing tests" from skill A and "Python codebase with all tests green" from skill B remain as separate nodes, they cannot connect despite representing the same real-world state. This deduplication is only possible because scenarios are explicit objects that can be embedded, clustered, and merged. In a skill-only graph, there is nothing to deduplicate.

Is this a fundamental shift or an incremental refinement? This is a fundamental architectural shift, not a refinement. Prior work organized skills; SkillSynth organizes states. The difference is analogous to organizing a transportation network around intersections (states) rather than around vehicles (skills) — intersections constrain which routes are possible, deduplicating intersections reveals the true connectivity, and planning a route means specifying which intersections you'll pass through, not just which vehicles you'll use. The paper's empirical evidence that this matters comes from the ablation in Table 4: randomly composed multi-skill baselines (which lack scenario-mediated compatibility constraints) produce lower-quality tasks than graph-guided paths, not because the skills themselves differ, but because the random composition ignores state compatibility — producing incoherent workflows that the multi-agent harness simplifies into trivial tasks.

Evidence anchor. The connected component analysis (Table 6, Appendix D) demonstrates the consequence of scenario-mediated construction: 85.6% of scenarios belong to a single giant component, meaning the cross-skill alignment successfully chained the majority of skills into a traversable state space. The 15.0% sink-only and 22.8% source-only scenarios represent natural entry and exit points. This structural property — a highly connected state space with identifiable boundaries — is a direct result of making scenarios first-class objects and would not emerge from skill-only organization.

Innovation 3: Inverse-Frequency Path Sampling as a Coverage-Maximizing Mechanism with Theoretical Motivation

The inverse-frequency path sampling algorithm (Algorithm 1) is more than an engineering convenience for getting diverse paths out of a graph. It represents a specific theoretical commitment — that the empirical distribution of training trajectories should approach uniformity over the Ω × K space — and an algorithmic realization of that commitment that connects directly to the learning objective.

What did prior work do for diversity in synthetic data? The standard approach to diversity in LLM-based data synthesis is temperature sampling: increase the sampling temperature of the generation model, or prompt it with "generate diverse examples," and hope that diversity emerges stochastically. This is how Nemotron-Terminal (Pi et al., 2026) generates varied task instances across taxonomy categories, and how Endless Terminals (Gandhi et al., 2026) produces domain-diverse tasks. The problem is that temperature sampling has no memory and no explicit diversity objective — it can produce the same output twice (or highly similar outputs) because each generation is independent. Diversity is a property of the collection, not of any individual sample, and temperature sampling optimizes neither.

Some prior work in other domains has used deterministic diversity mechanisms like clustering-based selection or maximum marginal relevance, but these typically operate post-hoc on an already-generated candidate set — filtering for diversity after generation rather than steering generation toward diversity.

What is distinctive about Algorithm 1? Three properties make this algorithm conceptually novel in the context of agent data synthesis:

1. It explicitly targets the coverage criterion from Equation 4. The inverse-frequency weighting p(σ) ∝ (ν(σ) + 1)^(-1) and p(κ) ∝ (μ(κ) + 1)^(-1) is not an arbitrary heuristic — it is a direct algorithmic implementation of the requirement that the empirical distribution D should uniformly cover Ω × K to maximize the learnable region of the policy. The counters ν and μ track how far the current sampling run is from uniformity, and the probabilities actively push toward filling coverage gaps. This connects the abstract theoretical criterion to concrete algorithmic choices in a way that prior diversity heuristics (temperature, post-hoc filtering) do not.

2. It is adaptive rather than static. The counters accumulate across the entire sampling run, meaning the algorithm's behavior changes as it learns which scenarios and skills are over-sampled. Early paths are drawn from a relatively flat distribution (all counters start at zero, so all elements have uniform probability). As the run progresses and counters diverge, later paths are increasingly steered toward under-sampled regions. This adaptivity means the algorithm automatically adjusts to the graph's connectivity structure — hub scenarios that would dominate a uniform random walk are progressively suppressed as their visit counts grow, while rare scenarios in the long tail are progressively amplified as they remain unvisited.

3. The monotone progression constraint enforces workflow coherence. Within a single path, visited scenarios and used skills are excluded from future steps. This might seem like a detail, but it encodes a substantive assumption about what makes a trajectory useful for training: agents learn to make forward progress, not to cycle. If a sampled path allowed revisiting scenarios, the multi-agent harness might produce a task where the agent loops between a small set of states — producing repetitive trajectories that exercise the same few (scenario, skill) pairs many times. Such trajectories would contribute little marginal learning value (because the pairs are already covered) while consuming the sampling budget. The monotone constraint ensures each path represents a genuine forward workflow, maximizing marginal coverage per accepted path.

Is this a fundamental contribution or an incremental technique? This is more fundamental than a typical sampling heuristic because it operationalizes the paper's core theoretical claim. Equation 4 says "to maximize learned capability, cover Ω × K uniformly." Algorithm 1 is the mechanism that makes this operational given the constraint that you can only sample paths (not arbitrary (σ, κ) pairs) from a graph whose structure determines which paths exist. The algorithm is therefore a bridge between theory and practice — it solves the constrained optimization problem of selecting a set of paths from a graph such that the induced empirical distribution over nodes and edges approaches uniformity. This problem (diverse path sampling from a graph) has not been previously studied in the context of agent training data synthesis, and the inverse-frequency solution, while drawing on established ideas from importance sampling, is applied here to a novel objective.

Evidence anchor. The downstream impact of this sampling strategy can be seen by comparing SkillSynth to the multi-skill baseline in Table 4. Both use the same multi-agent harness, the same skill pool, and the same number of synthesized tasks (3,721). The only difference is how the skill sequences were composed: graph-guided inverse-frequency sampling versus random composition. SkillSynth trajectories exhibit 19% higher unique scenario–skill coverage than the multi-skill baseline, and the fine-tuned model achieves 3.0 points higher on TB 1.0 and 3.8 points higher on TB 2.0. These gains are attributable purely to the path sampling strategy, isolating its contribution from the harness quality or training recipe.

Innovation 4: Verifier Over-Optimization Avoidance Through Structural Grounding Rather Than Adversarial Training

The paper does not frame this as a primary contribution, but a distinctive pattern emerges when comparing SkillSynth's synthesis quality control to how other synthetic data pipelines handle the problem of LLM hallucination and quality degradation: SkillSynth avoids verifier over-optimization and quality collapse not through adversarial filtering or RL-based alignment, but through structural grounding in an external, human-authored knowledge graph.

What is the problem? When LLMs are used to generate synthetic training data — whether task instances, trajectories, or solutions — they inevitably produce outputs of inconsistent quality. The standard solutions in the literature are (a) filtering: use a verifier (another LLM, a reward model, or execution-based checks) to discard low-quality outputs, or (b) iterative refinement: use the verifier's feedback to improve outputs through multi-turn generation, as in rejection sampling or RLHF-style optimization. Both approaches have a well-documented failure mode: the generator learns to exploit the verifier — producing outputs that score highly under the verifier's metric but are actually low-quality by the true objective. This is the same over-optimization phenomenon the SkillSynth paper's companion literature has documented for test-time compute scaling, and it applies equally to synthetic data generation pipelines.

What does SkillSynth do differently? SkillSynth's quality assurance does not rely primarily on filtering or iterative refinement against a learned verifier. Instead, it grounds the synthesis process in an external, structured knowledge source — the skill graph constructed from real, human-authored skills — that provides hard constraints on what constitutes a valid task. The graph ensures that:

  • Skills are real and executable because they come from ClawHub and GitHub, not from LLM imagination. An LLM cannot hallucinate a nonsensical skill into the graph because skills are filtered at the source (Section 3.2, Stage 1) before graph construction begins.

  • Workflow coherence is structurally enforced because the path must be a valid walk through the graph. The multi-agent harness cannot simplify a sampled path into a trivial task (as it does with random multi-skill compositions) because the graph edges encode real compatibility constraints — the planner must design a task that genuinely requires transitioning through the specified scenarios using the specified skills.

  • Semantic consistency across task components is verified against the sampled path, not against a learned quality metric. The rubric evaluation checks whether tests align with instructions and whether instructions are self-contained, but the underlying task structure (which skills, in which order, starting from which state) is pre-specified by the graph path — the LLM cannot drift into incoherence because the path is a fixed blueprint.

The dual verification with repair loop does use LLM-as-Judge for quality assessment, but this is a secondary quality check on implementation details (test alignment, instruction clarity), not on task structure (which is graph-grounded). The paper reports that 77% of rubric failures are test-instruction misalignment, and that corrupted filesystem snapshots (from buggy first-round LLM generation) are the dominant cause of unrecoverable oracle failures. These failure modes are implementation quality issues, not structural coherence issues — the underlying path is sound, but the LLM's instantiation of it is buggy. This separation of concerns (graph guarantees structure; LLM handles implementation) means SkillSynth avoids the feedback loop where an LLM both designs the task structure and evaluates its quality, which in purely LLM-driven synthesis pipelines can lead to self-consistent but nonsensical tasks that exploit the verifier.

Why this matters beyond SkillSynth. This design pattern — using an external knowledge structure to constrain LLM generation toward valid outputs, rather than relying on post-hoc verification of unconstrained generation — is broadly applicable to synthetic data pipelines where quality has a structural component. For code synthesis, the knowledge structure might be a type graph or API specification. For mathematical reasoning, it might be a theorem dependency graph. For dialogue, it might be a task-oriented ontology. The key insight is that when the space of valid outputs has combinatorial structure (which workflows do), representing that structure explicitly and sampling from it is more reliable than generating freely and filtering — because the structure provides hard constraints that no amount of filtering can recover once violated.

Distinguishing incremental from fundamental. This is a design pattern contribution rather than a fundamental theoretical advance. The specific mechanisms (graph construction, path sampling, harness with repair) are novel, but the underlying principle — constrain generation with external structure — is well-established in other fields (e.g., grammar-constrained decoding, template-based generation). What makes it notable here is the scale and domain: constructing a graph with 82,073 nodes and 57,214 edges from real-world skills and using it to synthesize 3,560 verified task instances without structural coherence failures is a demonstration that this approach scales to complex, real-world domains where purely generative approaches struggle with quality consistency.

Evidence anchor. The 95.7% oracle pass rate and 92.0% dual-pass rate (Table 1) demonstrate the structural reliability of the approach — nearly all paths that survive graph sampling can be successfully instantiated into solvable tasks, with repair recovering most implementation-quality failures. This contrasts with what one would expect from purely generative synthesis at this scale, where hallucinated task structures would produce higher failure rates. The comparison to multi-skill baselines in Table 4 provides indirect evidence: random skill compositions (which lack graph-grounded structure) produce simplified, lower-quality tasks because the harness, when given an incoherent skill sequence, "tends to generate simplified task instances that contain multiple fine-grained requirements but require few execution steps" — essentially reverting to easy tasks when structural constraints are absent.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use Terminal-Bench 1.0 and 2.0 (Merrill et al., 2026). TB 1.0 comprises 80 community-curated terminal tasks; TB 2.0 is a harder successor with 89 tasks. Both are hand-crafted benchmarks covering diverse domains, requiring agents to complete end-to-end workflows within containerized Docker environments. The paper evaluates on both to assess training effectiveness and generalization difficulty.

  • Base models. The primary base models are the Qwen3 dense series (Yang et al., 2025a) at three scales: 8B, 14B, and 32B parameters. The paper also reports a separate scaling experiment with Qwen3-14B. The choice is explicitly motivated to study the effect of model scale on training gains from SkillSynth trajectories. For evaluation context, the paper reports baseline performance (zero-shot, no fine-tuning) of numerous proprietary and open-source models on TB 1.0 and TB 2.0 in Table 3, including GPT-5.3-Codex (64.7% TB 2.0), Claude Opus 4.6 (62.9%), Gemini 3 Pro (56.9%), and Qwen 3 Coder 480B (23.9%), establishing the capability landscape. For the primary experiments, the models are trained with full-parameter supervised fine-tuning using a learning rate of 2 × 10⁻⁵ for 5 epochs, with AdamW (β₁ = 0.9, β₂ = 0.95), cosine learning rate schedule, 10% warmup ratio, weight decay 1 × 10⁻⁴, gradient clipping at 1.0, and bfloat16 precision. Training uses a micro-batch size of 1 per GPU with gradient accumulation.

  • Metrics. The primary metric is Terminal-Bench accuracy (%) — the fraction of tasks on which the agent's final environment state passes the task-specific verifier scripts. All reported results are the mean accuracy over three independent runs, reported with 95% confidence intervals. This triple-run protocol is important because terminal agent performance is inherently stochastic (different trajectories through non-deterministic LLM sampling produce different outcomes), and single-run results can be misleadingly optimistic or pessimistic. The paper also reports task difficulty distribution as a secondary metric: each synthesized task instance is attempted three times by Hy3 Preview, and tasks are binned by how many of the three attempts succeed (0/3, 1/3, 2/3, 3/3), providing a measure of how challenging the synthesized tasks are for a strong agent. Claude Opus 4.6 step counts on solved tasks are reported as an additional difficulty signal (average 37 steps).

  • Baselines. The paper uses several carefully constructed baselines to isolate different contributions of SkillSynth. For the main results (Table 3), the baseline is the untrained base Qwen3 model at each scale, evaluated zero-shot on TB 1.0 and TB 2.0. The paper also reports performance of numerous proprietary and open-source models for context (Table 3, upper portion), though these are not direct baselines in the ablation sense — they establish the capability ceiling and the open-source gap that SkillSynth aims to close. For the ablation study (Table 4), two synthesis baselines are constructed that use the identical multi-agent harness and identical downstream training pipeline as SkillSynth, differing only in how the seed workflows are composed: (1) Single-skill: 3,721 skills randomly drawn from the skill pool, each used as a standalone seed for task synthesis — producing 3,721 task instances each requiring only a single skill. (2) Multi-skill: 3,721 randomly composed combinations of 2–7 skills from the same pool, without graph-guided ordering — producing the same number of task instances as SkillSynth but with skill sequences assembled randomly rather than sampled from the graph. These baselines are critical because they control for the multi-agent harness quality, the skill pool, the number of tasks, the trajectory collection protocol, and the fine-tuning recipe, isolating the effect of graph-guided workflow composition.

  • Generation budget / compute accounting. The paper measures compute in two distinct contexts. For task synthesis, the cost is reported in dollars per verified task instance ($27.3 on average), reflecting the cumulative API cost of all LLM calls across graph construction, path sampling, and multi-agent harness operation. The internal compute budget of the multi-agent harness is measured in repair cycles (up to R = 3) and tool calls per cycle (up to N_tool = 20). For trajectory collection, the teacher model (MiniMax M2.7) generates 3 trajectories per task instance across 3,560 usable instances, producing 10,680 total trajectories. The evaluation infrastructure uses Harbor (Harbor Framework Team, 2026) to parallelize trajectory sampling across 128 concurrent Docker environments. The paper does not report the total FLOPs or wall-clock time for trajectory collection, nor does it compare the computational cost of trajectory collection across the SkillSynth, single-skill, and multi-skill conditions (which might differ if task difficulty affects average trajectory length).

  • Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional machine learning sense for the main supervised fine-tuning experiments — models are trained once on the full trajectory set and evaluated on the separate Terminal-Bench test sets. For task difficulty categorization (Table 2), each of the 3,560 instances is attempted exactly three times by Hy3 Preview, and the paper reports the distribution of success counts — this is a descriptive statistic of the synthesized task set, not a cross-validated model evaluation. For the diversity analysis (Figure 1, Section 4.5), the paper samples 1,000 trajectories independently from each dataset and uses DeepSeek Reasoner (v3.2) to extract scenarios and skills through the same prompt template across all strategies, ensuring "statistically comparable extraction granularity." The values are averaged over three independent samples of 1,000 trajectories each, with counts reported after embedding-based semantic deduplication using Harrier-OSS-v1-27B and clustering. For the error analysis (Table 5), three authors independently analyzed 20 failed trajectories each to design an error analysis taxonomy, which was then applied by an LLM agent to all failed trajectories — an inter-annotator calibration step followed by automated scaling.

Main Quantitative Results

The experiments are organized around three nested questions: (1) Does the multi-agent harness produce high-quality, solvable task instances at scale? (2) Does training on SkillSynth-synthesized trajectories improve terminal agent performance, and if so, by how much relative to baselines? (3) Does the graph-guided path composition specifically (as opposed to single-skill or random multi-skill composition) drive the performance improvements?

Harness Yield and Task Quality

The multi-agent harness demonstrates high synthesis reliability. Table 1 reports the outcomes for all 3,721 attempted task instances:

"95.7% of synthesized instances pass the oracle check, with 92.0% passing both quality checks."

Specifically: 3,423 instances (92.0%) pass both oracle and rubric verification; an additional 137 instances (3.7%) pass oracle but fail rubric checks (making them usable for SFT but not RL); and 161 instances (4.3%) fail entirely (oracle failures, making them unusable). The total yield of usable task instances (for SFT purposes) is 3,560 out of 3,721, or 95.7%.

The repair loop proves essential to achieving this yield. 721 task instances (19.4% of attempts) were recovered after failing in the first synthesis round. Without repair, the usable yield would have dropped from 95.7% to approximately 76.3%. The average repair required 2.31 cycles and 11 tool calls per task, demonstrating that iterative refinement corrects a substantial fraction of initial failures without excessive computational overhead.

The oracle failure analysis reveals a fundamental limitation: "the dominant cause is corrupted filesystem snapshots: due to the randomness of LLM generation, the first synthesis round may produce buggy stages that remain unrecoverable even after multiple repair cycles." This suggests that some initial generations are structurally broken in ways that iterative repair cannot fix — the constructor builds on a corrupt foundation. The paper's recommended mitigation is pragmatic: "re-running failed paths with a higher sampling temperature is a simple yet effective strategy to help recover these paths and preserve the diversity of the synthesized task set."

The rubric failure analysis identifies test-instruction misalignment as the dominant quality issue: "Of failed rubric checks, 77% stem from test scripts that either over-specify or under-specify relative to the instruction." Over-specification means the tests check for things the instruction never asked for — penalizing correct but differently-structured solutions. Under-specification means the tests miss required functionality — passing agents that didn't actually complete the task. The paper explicitly notes that these misaligned instances "potentially produce inaccurate evaluation of agent trajectories and erroneous reward signals for reinforcement learning," motivating the bifurcation: use for SFT only, not RL.

The task difficulty distribution (Table 2) confirms that graph-guided synthesis produces genuinely challenging problems. Across the 3,560 usable instances:

Successes (out of 3)PercentageNumber of Tasks
0/338%1,352
1/318%637
2/319%679
3/325%892

38% of tasks receive zero successes from Hy3 Preview in three attempts, indicating they are beyond the current capability of a strong proprietary agent. Another 37% are partially solvable (1/3 or 2/3), falling in the "learnable range" where an agent sometimes succeeds — these trajectories are particularly valuable for training because they contain both successful strategies (positive examples) and failure-and-recovery patterns (diagnostic examples). Only 25% are consistently solvable. Claude Opus 4.6, a frontier proprietary model, requires an average of 37 steps to solve SkillSynth tasks, further confirming the compositional complexity. Additionally, 121 tasks remain unsolved after three independent rollouts by Claude Opus 4.6 — these represent stretch goals that may require future capability advances.

Main Fine-Tuning Results

Table 3 reports the central experimental results: fine-tuning Qwen3 models at three scales on SkillSynth trajectories and evaluating on Terminal-Bench 1.0 and 2.0, compared against zero-shot baselines and reported proprietary/open-source performance.

At Qwen3-8B scale: The SkillSynth-trained model (+SS) achieves 17.1 ± 1.8% on TB 1.0 and 13.5 ± 2.8% on TB 2.0. The paper does not report the Qwen3-8B zero-shot baseline in Table 3, but the implicit comparison is against the untrained base model.

At Qwen3-14B scale: +SS achieves 22.9 ± 1.8% on TB 1.0 and 19.9 ± 1.6% on TB 2.0. Again, the zero-shot baseline is not reported for this specific model size in Table 3 (the ablation in Table 4 uses a 14B model, but the main results table focuses on 8B and 32B with 14B as a secondary data point).

At Qwen3-32B scale (primary model): +SS achieves 33.8 ± 3.1% on TB 1.0 and 29.6 ± 1.6% on TB 2.0. This is the headline result.

Three patterns emerge from the full leaderboard context in Table 3:

  1. Gains scale with model size. The absolute improvement over base model capability increases from 8B to 32B, consistent with the interpretation that larger models have greater capacity to absorb diverse trajectory data.

  2. SkillSynth-trained models substantially close the open-source gap. The strongest SkillSynth model (Qwen3-32B + SS) at 29.6% TB 2.0 outperforms Qwen 3 Coder 480B (23.9%) — a model with ~15× more parameters — and approaches Claude Haiku 4.5 (28.3%) and Claude Opus 4.1 (38.0%). This is the paper's central practical claim: that targeted data construction can make a 32B model competitive with or superior to much larger models and entry-level proprietary offerings. The paper explicitly states this interpretation:

"Qwen3-32B + SS outperforms the larger Qwen 3 Coder 480B on TB 2.0, suggesting that targeted data construction and domain-specific training can effectively improve the terminal agentic capabilities of smaller models."

  1. A substantial gap to frontier proprietary models remains. GPT-5.3-Codex achieves 64.7% on TB 2.0, and Claude Opus 4.6 achieves 62.9% — roughly double the best SkillSynth-trained model's accuracy. This gap likely reflects both model scale differences (proprietary models are presumably much larger than 32B) and training data volume/quality differences (proprietary models likely have access to much larger and more diverse terminal interaction data than 10,680 synthesized trajectories). The paper presents this gap as motivation rather than failure — SkillSynth demonstrates a scalable path toward closing it.

Ablation: Single-Skill vs. Multi-Skill vs. Graph-Guided Composition

Table 4 reports the critical ablation that isolates the contribution of graph-guided path composition. All three conditions use the identical multi-agent harness, the identical skill pool (all skills appearing in the sampled paths), the identical number of synthesized task instances (3,721), and the identical downstream fine-tuning pipeline (Qwen3-32B, same hyperparameters). The only difference is how the seed workflows are composed:

StrategyTask Difficulty DistributionTB 1.0 AccuracyTB 2.0 Accuracy
0/31/32/3
Single-skill16%23%34%
Multi-skill27%24%21%
SkillSynth (Ours)38%18%19%

Headline finding: Graph-guided composition produces substantially better agents than either baseline. SkillSynth outperforms single-skill by 8.4 points on TB 1.0 and 8.3 points on TB 2.0, and outperforms multi-skill by 3.0 points on TB 1.0 and 3.8 points on TB 2.0. These gaps are substantial relative to the overall accuracy scale (where 30% represents the state of the art for open-source models).

Task difficulty shifts with composition strategy. The difficulty distribution tells a revealing story. Single-skill synthesis produces predominantly easy tasks: only 16% are 0/3 difficulty, while 34% are 3/3 (consistently solvable) and the modal category is 2/3. Multi-skill random composition shifts the distribution leftward (harder): 27% 0/3, but still 28% 3/3. SkillSynth produces the hardest distribution: 38% 0/3, and only 25% 3/3, with the modal category being the hardest tier. This confirms that graph-guided paths produce genuinely more challenging tasks — not merely different tasks.

Why random multi-skill composition underperforms. The paper identifies the specific failure mode:

"Randomly composed skills lack sequential dependencies and cannot form coherent workflows. The multi-agent harness tends to generate simplified task instances that contain multiple fine-grained requirements but require few execution steps."

This is a critical mechanistic insight. When given an incoherent skill sequence (skills that don't naturally chain because their pre/postconditions are incompatible), the multi-agent harness cannot produce a task that genuinely requires coherent sequential execution. Instead, it falls back to generating a task that checks multiple independent requirements — the agent must do all the listed things, but the order doesn't matter and the intermediate states are independent. Such tasks are multi-requirement but not compositional — they exercise skills in isolation rather than in sequence through progressively changing states. The resulting trajectories are effectively single-skill trajectories concatenated together, missing the state-transition richness that makes SkillSynth trajectories valuable.

The incremental value of graph guidance over random composition. The 3.0–3.8 point gap between multi-skill and SkillSynth isolates the effect of graph-guided composition specifically, controlling for the shift from single-skill to multi-skill. This gap demonstrates that it's not enough to simply combine multiple skills — the combination must follow semantically coherent transitions for the resulting tasks to produce diverse training trajectories. The graph provides these transitions through scenario-mediated compatibility constraints; random composition does not.

Claude Opus 4.6 step counts. The paper mentions that Claude Opus 4.6 requires an average of 37 steps to solve SkillSynth tasks, and 121 tasks remain unsolved after three rollouts. Comparison figures for single-skill and multi-skill tasks are not provided, preventing a direct difficulty comparison via step counts. However, the difficulty distribution in Table 4 provides a proxy: more 0/3 tasks and fewer 3/3 tasks imply greater difficulty.

Diversity Analysis

Section 4.5 quantifies diversity at two levels: the structural diversity of the skill graph and sampled paths, and the diversity of the resulting execution trajectories.

Skill graph diversity (structural). The constructed graph's domain coverage (Appendix D, Figure 5) spans 26 categories, including long-tail domains such as Audio & Speech (1.2%), Games, 3D & Simulation (1.0%), IoT, Hardware & Robotics (0.4%), and Education & Learning — confirming that the skill pool captures terminal usage beyond software engineering. The degree distribution (Figure 6) is heavy-tailed with mean 4.32, median 2, and maximum 752, indicating a sparse graph where most nodes have limited connectivity but a small number of hub scenarios are highly connected. This structure motivates the inverse-frequency sampling: a naive random walk would over-sample hub scenarios. The connected component analysis (Table 6) shows one giant component of 118,806 nodes (85.6% of all scenarios) plus 6,250 smaller components — confirming that cross-skill alignment successfully chains the majority of skills into a traversable state space while preserving specialized, self-contained workflows in isolated components.

Combinatorial space for path sampling. The paper enumerates 16,632,220 paths requiring seven or more skills from the constructed graph, establishing that the graph supports a "vast combinatorial space for task synthesis." The 3,721 paths sampled in the experimental run represent approximately 0.02% of the enumerated 7+ skill paths, indicating substantial headroom for scaling.

Trajectory diversity (execution-level). Using the extraction + deduplication protocol described in Section 4.5 and Appendix E, the paper compares the trajectory diversity of SkillSynth against baselines:

"Trajectories sampled from SkillSynth tasks exhibit 31% higher unique scenario-skill coverage than single-skill baselines and 19% higher than randomly composed multi-skill baselines on average."

These percentages refer to the number of unique (scenario, skill) pairs after semantic canonicalization, per 1,000 randomly sampled trajectories, averaged over three independent samples. The 31% improvement over single-skill is expected (single-skill tasks by definition traverse fewer distinct scenarios), but the 19% improvement over multi-skill is the key finding — it demonstrates that graph guidance produces genuinely more diverse execution trajectories even when controlling for the number of skills per task. This diversity gain translates directly into downstream performance, as shown in Table 4.

Why this matters beyond the ablation. The diversity analysis closes the loop on the paper's theoretical argument from Section 2. Equation 4 established that the agent's learnable region is confined to the support of D over the Ω × K product space. Figure 1 demonstrated that existing datasets leave large gaps in this support. The trajectory diversity analysis demonstrates that SkillSynth's graph-guided synthesis significantly increases coverage of this product space. And the downstream performance gains (Tables 3 and 4) demonstrate that this increased coverage translates into improved agent capability — completing the chain from theory → measurement → intervention → outcome that the paper set out to establish.

Error Analysis of Failed Trajectories

Table 5 reports an error analysis of trajectories where the fine-tuned agent failed to solve the task. Three authors manually analyzed 20 failed trajectories each to design an error taxonomy, which was then applied by an LLM agent to all failed trajectories. The dominant failure modes reveal systematic weaknesses in current terminal agents that training on SkillSynth trajectories partially addresses but does not eliminate:

Partial Implementation (42.2%): The agent completes some but not all of the instruction's requirements, runs tests that pass (on the implemented subset), and submits without verifying the unimplemented requirements. Example: "Agent runs pytest on self-generated tests, gets all tests passing, and submits but silently skips one or more clauses from the instruction." This is a specification-following failure — the agent loses track of the full instruction scope during extended execution.

Inline Self-Test Over-trust (29.5%): The agent verifies its solution using hand-crafted checks (python3 -c assertions on happy paths) rather than the provided test suite, missing edge cases that the real tests would catch. "Agent never runs pytest; instead verifies with hand-picked python3 -c assertions on happy paths, missing edge cases." This is a testing discipline failure — the agent substitutes self-narrated verification for specification-grounded testing.

Premature Termination (12.9%): The agent finishes writing code and exits without running any form of verification. This is a task-completion discipline failure — the agent treats "code written" as equivalent to "task completed."

API/Flag Hallucination (7.0%): The agent repeatedly invokes non-existent modules or command flags, triggering errors such as No module named or unexpected kwarg, and fails to recover. This is a tool-use reliability failure — the agent confidently uses incorrect interfaces.

Debug Fixation Loop (5.1%): The agent repeats the same command 10+ times with minor edits, never switching strategy, and exhausts the episode budget. This is an exploration failure — the agent cannot recognize when its current approach is unproductive and cannot pivot to alternatives.

Error Rationalization (3.3%): The agent observes failing tests but rationalizes them as "pre-existing" or "flaky" and completes anyway. This is a verification-override failure — the agent invents explanations to dismiss contradictory evidence.

What these failure modes reveal about trajectory diversity. The two dominant failure modes (Partial Implementation and Inline Self-Test Over-trust, together 71.7%) are both forms of specification-verification misalignment — the agent substitutes its own narrower interpretation of the task for the actual specification. The paper's argument that trajectory diversity matters is relevant here: training on diverse scenarios and skills exposes the agent to a wider range of edge cases and failure patterns, potentially teaching it to be more thorough in verification. However, these failure modes also suggest that trajectory diversity alone is insufficient — the agent also needs stronger alignment with task instructions and better meta-cognitive strategies for recognizing when its verification is incomplete.

The paper interprets these findings as evidence that "terminal agents require stronger alignment with task instructions and more flexible exploration strategies during execution" — pointing toward future work on instruction-following penalties, curiosity-driven exploration, and meta-verification (having the agent explicitly check its own verification completeness).

Ablation Studies and Robustness Checks

The paper does not report traditional hyperparameter sweeps or component ablations of the kind common in model architecture papers. Instead, the ablation analysis focuses on the central design dimension: graph-guided path composition versus alternative seed composition strategies. Additional ablation-like analyses are distributed across the diversity analysis, error analysis, and discussion sections.

  • Seed composition strategy (Table 4): The primary ablation. Single-skill vs. multi-skill vs. graph-guided composition, described in detail above. The finding is that graph-guided synthesis produces harder tasks and 3.0–8.4 point improvements on TB 1.0 and TB 2.0 compared to the baselines, with corresponding gains in trajectory diversity. This ablation controls for task count, harness quality, skill pool, trajectory collection protocol, and fine-tuning recipe — isolating the effect of graph-guided composition specifically.

  • Model scale robustness (Table 3): The paper demonstrates that SkillSynth trajectories improve performance across three model scales (8B, 14B, 32B), with gains increasing with model size. This is not a controlled ablation (the base model capabilities differ), but it demonstrates that the training signal from diverse trajectories is useful across a range of model capacities and is not specific to a particular scale.

  • Retaining failed trajectories for training: The paper retains both successful and failed trajectories for supervised fine-tuning, citing Pi et al. (2026) for the rationale that "failed trajectories contain useful reasoning patterns for agentic problem-solving, such as error diagnosis and recovery strategies across diverse scenarios." This is a methodological choice rather than an ablation — the paper does not report a comparison of SFT with and without failed trajectories — but it represents a deliberate design decision motivated by the diversity argument: failed trajectories expose the agent to scenarios and skills that may not appear in successful trajectories (because the agent fails before reaching certain states or because the failure mode itself exercises a different skill set).

  • Rubric-failed tasks retained for SFT but excluded from RL: The paper's bifurcation — tasks that pass oracle but fail rubric checks are used for SFT but would be excluded from RL — is a design choice motivated by the different roles of the verifier in these training paradigms. For SFT, the training signal comes from the teacher's trajectory, not the verifier, so mild test misalignment does not corrupt the training objective. For RL, the verifier IS the reward signal, so misalignment would provide erroneous rewards. The paper does not empirically validate this bifurcation (no RL experiments are reported), but the reasoning is sound.

  • Bidirectional cross-skill alignment (Appendix B): The paper explored alternative graph construction strategies and found that "the LLM-based pairwise alignment approach yields higher-quality graphs" than "embedding-based scenario alignment and single-pass subgraph generation." No quantitative comparison is reported, but the qualitative finding is noted. The bidirectional alignment (postcondition → preconditions AND precondition → postconditions, with separately designed prompts) is presented as a quality safeguard rather than an ablation — the paper does not report graph quality with unidirectional alignment.

  • Two-stage hierarchical clustering (Appendix B): The paper evaluated "nine common clustering algorithms" and selected the Louvain + complete-linkage agglomerative approach. No quantitative comparison is reported, but the selection was based on manual inspection of resulting clusters on held-out scenario samples. The distance threshold was tuned by sweeping and manually inspecting clusters to balance "merge quality and over-fragmentation."

  • Repair loop effectiveness (Table 1): The repair loop recovers 721 task instances that failed in the first round. The paper does not ablate the number of repair cycles (R = 3) or tool calls per cycle (N_tool = 20), so the sensitivity of yield to these hyperparameters is unknown. The average of 2.31 repair cycles suggests that most recovered instances required 2–3 attempts, but the marginal value of the third cycle versus the second is not quantified.

  • Teacher model choice: The paper uses MiniMax M2.7 for trajectory collection "due to its cost efficiency." No comparison of trajectories collected with different teacher models is provided — the diversity and quality of the trajectories are functions of both the task instances and the teacher model's capabilities, but this interaction is not explored.

  • Negative result: repair has a floor (Section 4.2): The paper reports that corrupted filesystem snapshots from buggy first-round generations "remain unrecoverable even after multiple repair cycles." This is a genuine negative finding — the repair loop cannot fix all failures, and some initial generations are structurally broken in ways that iterative refinement cannot correct. The paper's practical mitigation (re-running with higher temperature) is suggested but not empirically validated.

  • Negative result: difficulty estimation via synthesis (implied): The paper does not claim to predict which paths will produce hard tasks. The difficulty distribution (Table 2) shows substantial variance even within graph-guided synthesis, and the paper does not report correlations between path properties (length, node degrees, domain diversity) and resulting task difficulty. This is an unaddressed limitation — the system can generate hard tasks on average but cannot target specific difficulty levels.

Critical Assessment

The paper's central claims, as established in the Executive Summary, are: (1) SkillSynth synthesizes diverse terminal task instances using a scenario-mediated skill graph and multi-agent harness; (2) the synthesized tasks enable effective supervised fine-tuning of terminal agents, improving performance on Terminal-Bench; and (3) the graph-guided composition specifically produces harder, more diverse tasks and better downstream agents than single-skill or random multi-skill baselines. How well do the reported experiments support these claims?

Claim 1 — synthesis quality at scale — is well-supported but with important caveats. The 95.7% oracle pass rate at $27.3 per verified task instance (Table 1) demonstrates that the pipeline reliably produces solvable tasks without human intervention. The difficulty distribution (Table 2) and Claude Opus 4.6 step counts (average 37) confirm that the tasks are non-trivial. However, several aspects of synthesis quality are not directly measured:

  • Instruction quality beyond rubric checks. The rubric evaluation checks instruction–test alignment and instruction self-containedness, but does not assess whether instructions are clear, unambiguous, or realistically phrased. An instruction could be perfectly aligned with tests and self-contained yet be confusing or unnatural in ways that confound agents. Human evaluation of instruction quality is not reported.

  • Oracle solution quality. The oracle check verifies that the oracle solution passes the tests, but does not verify that the oracle solution is a reasonable or efficient way to solve the task. An oracle solution that works but is convoluted or unnatural would still train agents to imitate suboptimal strategies. The paper does not assess oracle solution quality beyond solvability.

  • Generalizability of graph quality to new skills. The paper claims the graph is "naturally scalable infrastructure" as the community contributes more skills. But the graph construction relies heavily on LLM-based inference, alignment, and filtering — stages whose error rates may compound as the graph grows. The paper does not analyze how graph quality scales with size or how robust the construction pipeline is to distribution shifts in contributed skills.

Claim 2 — downstream performance improvements — is supported but the magnitude should be interpreted carefully. The fine-tuning results (Table 3) show consistent improvements over base models, and the 32B model reaches 33.8% TB 1.0 and 29.6% TB 2.0. However:

  • No comparison to training on equivalent-volume non-synthesized data. The paper demonstrates that SkillSynth trajectories improve over base models, but does not compare against training on the same number of trajectories collected from, e.g., existing terminal benchmarks, manually curated tasks, or other synthetic datasets. The only comparison is SkillSynth vs. its own baselines (single-skill and multi-skill), which use the same harness and differ only in composition strategy. This is appropriate for isolating the graph-guided composition effect, but it leaves open the question of whether SkillSynth trajectories are better than what could be obtained from alternative synthesis approaches at the same cost.

  • The absolute performance remains low relative to frontier models. The best SkillSynth model (29.6% TB 2.0) achieves less than half the accuracy of the best proprietary model (GPT-5.3-Codex at 64.7%). This gap could reflect insufficient trajectory volume (10,680 trajectories), insufficient trajectory diversity (the graph covers Ω × K partially, not exhaustively), model capacity limitations (32B parameters vs. presumably much larger proprietary models), or fundamental capability gaps that supervised fine-tuning on trajectories cannot address (e.g., the error analysis reveals systematic reasoning failures that SFT alone may not correct). The paper does not disentangle these factors.

  • No reinforcement learning experiments. The paper mentions that rubric-failed tasks should be excluded from RL to avoid erroneous reward signals, but does not report any RL experiments. Given that RL is a common and often effective training paradigm for agents (enabling exploration and reward-driven policy improvement beyond imitation), the absence of RL results means the paper demonstrates only the SFT value of SkillSynth trajectories. Whether these trajectories are also effective for RL — and whether the diversity gains persist under RL optimization — is an open question.

Claim 3 — graph-guided composition specifically drives the gains — is the paper's strongest and best-supported claim. The ablation in Table 4 is clean, well-controlled, and convincing. All three conditions use the identical harness, skill pool, task count, trajectory collection protocol, and fine-tuning recipe, differing only in seed composition strategy. The 3.0–8.4 point gaps on TB 1.0 and TB 2.0, combined with the 19–31% diversity gains, provide strong evidence that graph-guided composition produces more useful training data than random or single-skill composition. The mechanistic explanation — that random multi-skill composition lacks sequential dependencies, causing the harness to simplify tasks — is plausible and consistent with the observed difficulty shifts.

Genuine weaknesses and missing experiments:

  • Test set size. Terminal-Bench 2.0 has 89 tasks. While community-curated and diverse, this is a small evaluation set by machine learning standards. A 3.8 point improvement (SkillSynth vs. multi-skill on TB 2.0) represents roughly 3–4 additional correctly solved tasks. Confidence intervals (±1.6–3.1%) span a range of roughly 2–3 tasks at this scale. The statistical reliability of these comparisons depends on the difficulty distribution of the 89 tasks — if a few tasks are disproportionately influential (very hard or very easy), the results could be sensitive to task sampling.

  • Single model family. All fine-tuning experiments use Qwen3 dense models. The paper argues these are representative, but different model families have different base capabilities, different in-context learning behaviors, and different capacities to absorb trajectory data. The SkillSynth trajectory diversity gains might interact with model architecture in unknown ways.

  • No comparison with other synthetic datasets. The paper compares SkillSynth trajectories to its own baselines but not to trajectories collected from Nemotron-Terminal, Endless Terminals, or TermiGen tasks. Such a comparison would be confounded by different task counts, harness differences, and teacher model differences, but it would provide external validation of the diversity claim beyond internal baselines.

  • Teacher model dependence. All trajectories are collected using MiniMax M2.7. The diversity and quality of the collected trajectories depend on both the task instances AND the teacher model's exploration behavior. A weaker teacher might produce less diverse trajectories (getting stuck, making similar errors) even on diverse tasks. A stronger teacher might produce more diverse trajectories even on simpler tasks. The paper does not ablate the teacher model, so the extent to which trajectory diversity is task-driven vs. teacher-driven is unclear.

  • Cost and scale analysis is incomplete. The $27.3 per task figure includes API costs for synthesis but not the compute cost of trajectory collection (10,680 trajectories across 128 parallel Docker environments) or the training cost (full-parameter fine-tuning of up to 32B models). A complete cost accounting would enable comparison against alternative approaches — e.g., is it cheaper to synthesize 10,680 diverse trajectories and fine-tune, or to simply use a larger base model with fewer trajectories?

  • Difficulty estimation cost is unaddressed. Unlike the companion paper on compute-optimal test-time scaling which explicitly flags difficulty estimation cost as an open problem, SkillSynth does not address how to predict task difficulty before synthesis. The difficulty distribution in Table 2 is measured post-hoc. A system that could target specific difficulty levels (e.g., generating more learnable-range tasks and fewer impossible tasks) would be more efficient, but this capability is not demonstrated.

  • No dynamic or iterative graph expansion experiments. The paper claims the graph can grow with the ClawHub ecosystem, but does not demonstrate that adding new skills to an existing graph improves downstream model performance. The graph is constructed once from a static skill snapshot. Whether incremental graph expansion produces diminishing returns or sustained diversity gains is an open question.

  • No analysis of path length vs. task quality. Paths range from 1 to 7 skills, but the paper does not analyze how path length correlates with task difficulty, trajectory diversity, or downstream training value. Longer paths might produce more diverse trajectories (more intermediate scenarios) but also higher synthesis failure rates (more opportunities for incompatibility). This tradeoff is unexplored.

  • Trajectory diversity measured but not optimized in the training objective. The paper demonstrates that trajectory diversity correlates with downstream performance (Tables 4, diversity analysis), but the fine-tuning objective (Equation 3) does not include any explicit diversity penalty or reweighting. Trajectories are used as-is, with both successful and failed trajectories retained. Whether further gains could be achieved by explicitly reweighting trajectories to maximize coverage of Ω × K — e.g., through prioritized sampling or data augmentation — is not explored.

Conditions under which claims hold:

  • Graph-guided composition outperforms baselines: This holds for the specific graph constructed from ClawHub and GitHub skills, the specific multi-agent harness with planner–constructor architecture and dual verification, the specific teacher model (MiniMax M2.7), and the specific base models (Qwen3 8B–32B). Generalization to other skill sources, harness architectures, teacher models, or base model families is plausible but unvalidated.

  • Trajectory diversity improves downstream performance: This holds for supervised fine-tuning of Qwen3 models on Terminal-Bench tasks. Whether it holds for reinforcement learning, for other agent benchmarks, or for other model families is not tested.

  • SkillSynth scales to large graphs: Demonstrated for a single graph of 82,073 nodes and 57,214 edges. Whether the construction pipeline degrades at larger scales (more skills, more domains) or with noisier skill specifications is unknown. The scalability claim is aspirational — the infrastructure exists, but the scaling behavior of the construction pipeline is not empirically characterized.

6. Limitations and Trade-offs

6.1 The Skill Graph Is Constructed from a Static Skill Snapshot; Dynamic Expansion Behavior Is Unvalidated

The assumption or constraint. The paper positions the skill graph as a "naturally scalable infrastructure" and "living synthesis framework" that expands as the community contributes more skills to ClawHub. Section 3.2 states that "as the community contributes more skills, the graph continues to expand, enabling continual synthesis of diverse terminal tasks." However, the entire experimental validation uses a single static graph snapshot constructed once from the current ClawHub and GitHub skill pools. The paper does not demonstrate incremental graph construction — adding new skills to an existing graph, re-running alignment and deduplication, and verifying that the expanded graph produces incremental diversity gains rather than introducing noise, incompatibilities, or quality degradation.

The consequence. This matters for two reasons. First, the scalability claim is aspirational rather than empirically grounded. The graph construction pipeline involves five LLM-dependent stages (scenario inference, semantic deduplication via clustering, bidirectional cross-skill alignment, scenario merging, and filtering), each of which has a non-zero error rate. As the number of skills grows, these errors may compound: a single poorly-inferred scenario could create spurious edges that pollute the compatibility structure; a single incorrectly merged scenario cluster could collapse semantically distinct states and degrade the graph's ability to enforce coherent workflows. The paper provides no characterization of how error rates scale with graph size.

Second, even if the construction pipeline remains reliable, the marginal diversity gain from additional skills is unknown. The graph already spans 26 domains and contains 82,073 scenarios. New skills contributed to ClawHub may disproportionately belong to already-dense regions of the graph (e.g., additional coding or DevOps skills) while leaving long-tail domains (Audio & Speech, IoT) sparse — producing diminishing returns in Ω × K coverage without the explicit coverage-maximizing sampling strategy being able to compensate for structural gaps in the graph itself.

What evidence exists in the paper. The only evidence is the single-graph structural statistics (Table 6, Appendix D), which describe the constructed graph but not its scaling behavior. The enumeration of 16,632,220 paths requiring 7+ skills demonstrates that the current graph supports vast combinatorial diversity, but this is a static snapshot. No experiment constructs a second graph from a larger or different skill pool and compares the resulting trajectory diversity or downstream model performance. The paper explicitly frames future work as "scaling SkillSynth to larger data regimes" (Section 4.7), implicitly acknowledging that current-scale validation is preliminary.

Mitigation status. The paper does not attempt to mitigate this limitation. The claim that the graph scales with the ecosystem is presented as a conceptual property of the architecture (graph nodes and edges can be added incrementally) rather than an empirically validated feature. A minimal validation would involve taking a subset of the current skills, constructing a graph, measuring trajectory diversity, then adding held-out skills, reconstructing the graph, and measuring the incremental diversity gain — establishing a scaling trend. No such experiment is reported.

6.2 Difficulty Is Measured Post-Hoc; There Is No Mechanism for Targeting Specific Difficulty Levels During Synthesis

The assumption or constraint. SkillSynth synthesizes task instances whose difficulty is discovered after synthesis through black-box evaluation by a strong proprietary agent (Hy3 Preview, Table 2). The system has no mechanism for predicting, controlling, or targeting the difficulty of the tasks it generates during the path sampling or harness stages. A practitioner wanting to generate only "learnable-range" tasks (1/3 or 2/3 success rate, comprising 37% of the current set) or wanting to avoid unsolvable tasks (0/3, at 38%) has no way to steer the synthesis process toward those difficulty regimes.

The consequence. This creates a significant efficiency problem for training pipelines. From a training perspective, tasks that are impossible for any current agent (0/3 success, 38% of SkillSynth instances) produce trajectories that are exclusively failures. While the paper argues these contain useful error-recovery patterns, the marginal value of a trajectory on an unsolvable task — where the agent never reaches success and only explores dead ends — is likely lower than a trajectory on a task where the agent sometimes succeeds and sometimes fails, providing both positive and negative examples. If a training budget is fixed (e.g., collect 10,000 trajectories), the presence of 38% unsolvable tasks means 38% of the trajectory collection budget is spent on tasks that may contribute less learning signal. Being able to filter or deprioritize these tasks during synthesis would improve training data efficiency.

Conversely, the 25% of tasks that are consistently solvable (3/3 success) may be too easy to push the frontier of agent capability — they reinforce existing skills but do not expose failure modes or require novel strategies. An ideal synthesis pipeline would target the middle difficulty tier where trajectories exhibit both successful and failed attempts, maximizing the information density per collected trajectory.

The paper does not analyze what properties of a sampled path (length, node degrees, domain diversity, presence of specific skill types) correlate with resulting task difficulty. Without such a correlation model, difficulty targeting is impossible.

What evidence exists in the paper. The difficulty distribution in Table 2 provides the only relevant data: 38% 0/3, 18% 1/3, 19% 2/3, 25% 3/3. The paper reports that Claude Opus 4.6 requires an average of 37 steps to solve SkillSynth tasks, and 121 tasks remain unsolved after three rollouts — confirming compositional difficulty on average — but these are aggregate statistics that provide no per-path difficulty signal. The ablation in Table 4 shows that difficulty shifts with composition strategy (single-skill tasks are easier, graph-guided tasks are harder), but this is a distributional shift, not a per-instance prediction capability.

Mitigation status. Not addressed. The paper does not claim difficulty targeting as a capability, and it does not suggest it as future work. The difficulty distribution is presented as a descriptive statistic characterizing the synthesized task set. This is a genuine open problem: given a graph path, can one predict whether instantiating it will produce a task that is trivially easy, learnably difficult, or impossibly hard? Solving this would likely require features of the path (skill count, graph centrality measures, domain diversity) combined with features of the multi-agent harness output (instruction length, test complexity, environment complexity).

6.3 Trajectory Diversity Is Measured but Not Directly Optimized in the Training Objective; The Full Causal Chain from Graph Coverage to Model Performance Is Incomplete

The assumption or constraint. The paper's core theoretical argument (Section 2, Equation 4) establishes that maximizing the learned policy's capability requires training data whose empirical distribution D densely covers the Ω × K product space. The inverse-frequency path sampling algorithm (Algorithm 1) is designed to produce paths whose collective coverage of Ω × K approaches uniformity. The multi-agent harness instantiates these paths into tasks, and the collected trajectories are measured to have 19–31% higher unique scenario–skill coverage than baselines (Section 4.5). The fine-tuned models achieve 3.0–8.4 point improvements on Terminal-Bench (Table 4).

However, the paper does not establish the full causal chain connecting graph coverage improvements to downstream performance through the intermediate variables. Specifically: (1) Does higher Ω × K coverage in the sampled paths directly produce higher Ω × K coverage in the collected trajectories? (2) Does higher trajectory Ω × K coverage directly improve the model's policy coverage (the set of (σ, κ) pairs for which π(κ | σ, g) is accurate)? (3) Are the Terminal-Bench gains attributable specifically to improved coverage of the benchmark's scenario–skill space, or do they arise from other factors (e.g., higher task difficulty causing longer trajectories, which provide more training tokens)?

The consequence. Without establishing this chain, the paper's theoretical framing (Section 2, Equation 4) remains a motivating intuition rather than a validated mechanism. It is possible that the downstream gains arise not from improved Ω × K coverage per se but from correlated factors: graph-guided tasks might be harder (as Table 4 shows), producing longer trajectories with more decision points, more error-recovery sequences, and simply more training tokens — and the diversity gains in Ω × K are a side effect of task difficulty rather than the causal driver of performance improvement. Alternatively, the improvements might come from the graph ensuring workflow coherence, which makes tasks more realistic and their trajectories more transferable to Terminal-Bench, independent of diversity metrics.

This matters for practitioners because the paper's prescription is to maximize trajectory diversity along the scenario and skill dimensions. If the actual driver of performance is task difficulty or workflow coherence (with diversity as a correlated byproduct), then a practitioner might achieve similar gains by focusing on generating hard, coherent tasks without explicitly optimizing for Ω × K coverage — a simpler engineering goal.

What evidence exists in the paper. The paper provides two pieces of correlational evidence: (1) SkillSynth trajectories have higher Ω × K coverage than baselines, AND (2) SkillSynth-trained models outperform baselines. The paper also provides a mechanistic explanation for why random multi-skill baselines underperform ("randomly composed skills lack sequential dependencies and cannot form coherent workflows. The multi-agent harness tends to generate simplified task instances"), which suggests workflow coherence — not diversity per se — is the differentiator. The paper does not perform a mediation analysis or partial-correlation study that would isolate the contribution of Ω × K coverage from the contribution of task difficulty or trajectory length.

Table 4 shows that multi-skill baselines have fewer 0/3 tasks (27% vs. 38%) and more 3/3 tasks (28% vs. 25%) compared to SkillSynth — meaning SkillSynth tasks are harder overall. If harder tasks produce longer trajectories (more steps, more tokens), and more training tokens improve performance, then the performance gap could be partially or fully explained by trajectory length differences rather than diversity differences. The paper does not report average trajectory lengths for SkillSynth vs. baselines, so this confound cannot be evaluated.

Mitigation status. Not addressed. The paper presents the diversity measurement (Figure 1, Section 4.5) as confirmatory evidence for the theoretical framework, but does not attempt to control for alternative explanations or establish the causal pathway. The paper does not reweight trajectories by Ω × K coverage during training, does not ablate trajectory length, and does not compare against a baseline that controls for task difficulty while varying diversity. This is a gap between the theory (Section 2) and the empirical validation (Section 4) that weakens the paper's central mechanistic claim.

6.4 The Synthesis Pipeline's Computational Cost Is Partially Accounted For; Trajectory Collection and Training Costs Are Omitted

The assumption or constraint. The paper reports the synthesis cost of $27.3 per verified task instance (Section 4.2), which covers the API calls for graph construction, path sampling, and multi-agent harness operation (planning, construction, dual verification, repair). However, three substantial additional costs are not quantified:

  1. Trajectory collection cost: The paper collects 3 trajectories per task instance across 3,560 usable tasks, yielding 10,680 trajectories using MiniMax M2.7 as the teacher model. The API cost of 10,680 trajectory rollouts — each potentially dozens of steps (Claude Opus 4.6 averages 37 steps on SkillSynth tasks) — is not reported. If each trajectory costs a similar order of magnitude to the task synthesis itself (given the multi-step, tool-using nature of terminal agent execution), trajectory collection could dominate the total pipeline cost.

  2. Training cost: Full-parameter supervised fine-tuning of Qwen3-8B, Qwen3-14B, and Qwen3-32B on 10,680 trajectories for 5 epochs involves substantial GPU compute. The paper reports training hyperparameters (learning rate 2e-5, bfloat16 precision, gradient clipping) but not the total GPU-hours, model FLOPs, or dollar cost of training. For the 32B model, full-parameter fine-tuning is expensive and may dominate the synthesis cost.

  3. Difficulty estimation cost (buried in Table 2): The difficulty distribution in Table 2 requires 3,560 tasks × 3 attempts = 10,680 rollouts by Hy3 Preview — a cost comparable to the trajectory collection itself. This difficulty estimation is performed for evaluation purposes, but any practical deployment wanting to filter by difficulty would need to replicate this cost or develop a cheaper difficulty predictor (which the paper does not provide).

The consequence. The headline cost figure of $27.3 per verified task instance is misleading as a total-cost estimate for deploying SkillSynth end-to-end. It captures the synthesis stage but not the trajectory collection, model training, or (if desired) difficulty filtering stages. For a practitioner deciding whether to adopt SkillSynth, the relevant metric is the total cost to go from raw skills to a fine-tuned agent: synthesis + trajectory collection + training. Without this full accounting, the economic case for SkillSynth (vs. alternative data sources or simply using a larger base model) cannot be evaluated.

What evidence exists in the paper. The paper reports: synthesis cost ($27.3/task, Section 4.2), number of trajectories collected (10,680, Section 4.3), and training hyperparameters (Appendix C). Missing: trajectory collection cost, training cost, total end-to-end cost. The paper does not compare the total cost of the SkillSynth pipeline against the cost of alternative approaches (e.g., purchasing additional proprietary model API calls vs. fine-tuning a smaller model on SkillSynth data).

Mitigation status. Not addressed. The paper frames the $27.3 figure as the cost of the multi-agent harness specifically, not the total pipeline cost. However, Section 4.2 introduces this figure without contextualizing what fraction of the total pipeline cost it represents, which could mislead readers into thinking this is the dominant cost.

6.5 Evaluation Is Limited to a Single Benchmark (Terminal-Bench) with 89 Tasks; Generalization to Other Terminal Domains and Task Types Is Unvalidated

The assumption or constraint. All downstream performance evaluation is conducted on Terminal-Bench 1.0 (80 tasks) and 2.0 (89 tasks). While Terminal-Bench is a well-regarded community benchmark with hand-crafted tasks spanning diverse domains, the evaluation set is small (169 total tasks across both versions) and represents a specific distribution of task types curated for benchmarking rather than a random sample of real-world terminal usage.

The consequence. Small test sets amplify variance and limit the statistical reliability of comparisons. The 3.0-point gap between SkillSynth and multi-skill on TB 1.0, and the 3.8-point gap on TB 2.0 (Table 4), represent roughly 2–3 and 3–4 tasks respectively at this test set size. The reported 95% confidence intervals (±1.6–3.1 percentage points) span ranges of 1–3 tasks. This means the observed gaps, while consistent across both benchmarks, are based on a small number of task-level successes, and the statistical significance is sensitive to the difficulty distribution of the specific 89 tasks in TB 2.0. If a few tasks happen to align particularly well (or poorly) with SkillSynth's training distribution, the results could shift meaningfully.

More importantly, the paper does not evaluate on tasks from other terminal agent benchmarks or on held-out tasks synthesized by SkillSynth itself. The 3,560 synthesized tasks represent a distribution of terminal problems — do the fine-tuned models perform better on a held-out split of these tasks? Does the diversity gain generalize to terminal tasks that are stylistically different from both SkillSynth's synthesis distribution and Terminal-Bench's curation? Without multi-benchmark evaluation, the claim that SkillSynth-trained agents have "enhanced agentic capabilities in terminal-based settings" (Abstract) is supported only for the specific Terminal-Bench task distribution.

What evidence exists in the paper. The paper evaluates exclusively on Terminal-Bench 1.0 and 2.0 (Table 3, Table 4). No evaluation on other terminal benchmarks (e.g., SWE-Bench, CLI-Bench, InterCode), on held-out SkillSynth tasks, or on real-world terminal tasks collected from user logs is reported. The paper does acknowledge using Terminal-Bench as the evaluation standard, but does not discuss this as a limitation.

Mitigation status. Not addressed. The paper does not claim generalization beyond Terminal-Bench, but the abstract and conclusion describe the contributions in general terms ("terminal-based settings," "terminal agentic capabilities") that imply broader applicability. A held-out evaluation on a subset of SkillSynth's own synthesized tasks (or on an independent terminal benchmark) would strengthen the generalization claim.

6.6 The Revision Model for Iterative Task Improvement During Synthesis Is Absent; Only Supervised Fine-Tuning Is Validated

The assumption or constraint. The paper's training experiments (Section 4.3, Table 3) use only supervised fine-tuning (SFT) on collected trajectories. The multi-agent harness includes a repair loop that iteratively improves task instances during synthesis (Section 3.4), but the downstream agent training is standard imitation learning — the agent learns to mimic the teacher model's actions from static trajectories. The paper explicitly notes that tasks failing rubric checks should be excluded from reinforcement learning (Section 4.2), but does not report any RL experiments.

The consequence. SFT has a known limitation for agent training: it trains the agent to imitate the teacher's specific action sequence rather than to achieve the goal through any valid path. If the teacher's trajectories contain suboptimal decisions, irrelevant exploration, or teacher-specific stylistic patterns, the agent learns these as well. More importantly, SFT does not provide a mechanism for the agent to learn from its own mistakes during training — it never practices solving tasks and receiving feedback on its own attempts. Reinforcement learning (RL), where the agent interacts with environments and receives verifier-based rewards, can overcome these limitations by enabling exploration and reward-driven policy improvement.

The SkillSynth pipeline produces verified task instances with working oracle solutions and automated test suites — precisely the infrastructure needed for RL. The paper's bifurcation (rubric-passed tasks for RL, rubric-failed tasks for SFT only) indicates the authors considered RL, but no RL results are reported. Without RL validation, the paper demonstrates that SkillSynth tasks are useful for imitation learning but leaves open whether they are useful for the more powerful RL paradigm — and whether the diversity gains observed in SFT transfer to RL settings, where the agent's own exploration behavior (rather than a fixed teacher) determines which (σ, κ) pairs are actually experienced.

What evidence exists in the paper. The only mention of RL is the bifurcation in Section 4.2: "We retain these task instances for supervised fine-tuning to preserve trajectory diversity, but discard them for reinforcement learning to avoid erroneous reward signals." No RL experiments are reported. The paper does not discuss this absence.

Mitigation status. Not addressed. The paper does not frame SFT-only validation as a limitation, and Section 4.7 (Discussion and Future Work) does not mention RL as a future direction. Given that the companion literature on terminal agents and the broader agent training literature increasingly emphasizes RL (e.g., STaR, ReST^EM, RLHF variants), the absence of RL validation is a significant gap in the demonstrated utility of SkillSynth tasks.

7. Implications and Future Directions

How This Work Changes the Landscape

SkillSynth does not introduce a new model architecture, a new training objective, or a new reinforcement learning algorithm. What it introduces is a methodological reframing of synthetic data generation for agents — from "generate diverse task descriptions and hope trajectories follow" to "explicitly construct the state space you want the agent to experience, then generate tasks that force traversal through it." This is not a paradigm shift in the Kuhnian sense — the field is not abandoning existing synthesis methods — but it is a substantive reorientation of what synthetic data pipelines should optimize, with downstream consequences for how researchers allocate synthesis budgets.

The reframing: trajectory diversity as a design target rather than an emergent property. Prior to this work, the terminal agent synthesis literature operated on an implicit assumption that task diversity implies trajectory diversity. The paper's Figure 1 empirically demolishes this assumption by measuring what actually matters — the coverage of the $\Omega \times \mathcal{K}$ product space — and showing that existing datasets leave large gaps. The theoretical decomposition in Equation 4 provides the formal justification: because the agent's learnable policy is confined to the support of the training distribution over scenario–skill pairs, generating 10,000 tasks that exercise the same 100 (scenario, skill) pairs is fundamentally less valuable than generating 1,000 tasks that exercise 1,000 distinct pairs. This reframing changes the optimization target for synthetic data pipelines from task count to coverage of the joint state–action space, and provides a measurable proxy (unique scenarios, skills, and pairs after semantic canonicalization) for evaluating diversity.

What this reframing changes about research priorities. The paper makes certain research directions more attractive and others less so:

  • More attractive: structured knowledge sources as diversity constraints. The skill graph demonstrates that grounding synthesis in an external, human-authored knowledge structure (ClawHub skills, scenario compatibility relationships) produces coherent, diverse workflows that purely generative approaches struggle to match. This suggests a broad research program: what other external knowledge structures — API documentation, package dependency graphs, filesystem hierarchies, configuration file formats — can serve as diversity-constraining scaffolds for synthetic data generation? The key property is that the structure provides hard compatibility constraints (this skill can follow that skill only if their scenarios align) that prevent the generated data from collapsing into a narrow region of the state space. This contrasts with the dominant LLM-as-generator paradigm, where diversity is controlled through soft mechanisms (temperature, prompting) that lack structural guarantees.

  • More attractive: coverage-maximizing sampling from structured spaces. The inverse-frequency path sampling algorithm (Algorithm 1) is a specific solution to a general problem: given a graph whose structure constrains what sequences are valid, how do you sample a set of sequences whose empirical distribution over nodes and edges approaches uniformity? This problem appears in many synthetic data contexts — sampling diverse API call sequences, diverse database queries, diverse dialogue flows — and the inverse-frequency approach provides a simple, adaptive, theoretically-motivated baseline. Future work can improve on this baseline with more sophisticated coverage objectives (e.g., maximizing coverage of rare transitions rather than uniform coverage) or with learned samplers that predict which unexplored regions are most valuable for downstream training.

  • More attractive: disentangling task difficulty from task diversity. The paper's difficulty distribution (Table 2) reveals a practical tension: graph-guided synthesis produces tasks that are more diverse AND harder than baselines, but the two properties are confounded. Does diversity drive the downstream performance gains, or does difficulty (which produces longer trajectories with more decision points and error-recovery sequences)? Disentangling these factors — perhaps by synthesizing tasks with similar difficulty but varying diversity, or vice versa — would sharpen the field's understanding of what makes training data valuable. The paper's difficulty binning methodology (0/3, 1/3, 2/3, 3/3 success from a strong agent) provides a practical tool for this disentanglement.

  • Less attractive: scaling task count without a diversity metric. If the paper's core claim is correct — that trajectory diversity along the $\Omega \times \mathcal{K}$ dimensions is what drives learning, not raw task volume — then synthesis efforts that report only the number of generated tasks without measuring trajectory diversity are providing an incomplete picture. The field should adopt standardized diversity metrics (unique scenarios, skills, and scenario–skill pairs after semantic canonicalization, as in Figure 1) alongside task count when reporting synthesis results. A paper claiming to generate "1 million terminal tasks" would need to additionally report how many distinct scenario–skill pairs those tasks actually exercise to be evaluated against the SkillSynth baseline.

  • Less attractive: purely generative synthesis without structural grounding. The paper provides evidence that when the multi-agent harness is given randomly composed skill sequences (the multi-skill baseline), it produces simplified, low-quality tasks because the skills lack sequential dependencies. This suggests that purely generative approaches — prompting an LLM to produce diverse tasks without an external structure constraining what valid tasks look like — may produce superficially diverse but structurally redundant outputs. The 3.0–3.8 point gap between SkillSynth and multi-skill baselines on Terminal-Bench quantifies the value of structural grounding over random composition, establishing a benchmark that future purely generative approaches would need to match or exceed.

Reconciling conflicting evidence in the literature. The paper does not resolve a direct empirical contradiction in the way that, for example, a paper showing that self-correction works on easy problems but not hard ones reconciles conflicting prior findings. However, it does resolve a methodological tension: prior work on terminal agent synthesis (Endless Terminals, TermiGen, Nemotron-Terminal, CLI-Gym, SWE-Universe) has scaled task instances along different dimensions — domain coverage, environment fidelity, repository diversity — with each approach claiming to improve agent training through increased data. SkillSynth's contribution is to show that these approaches, despite their surface differences, share a common limitation: they do not explicitly control trajectory diversity, and their trajectories exhibit redundancy in scenario coverage and skill usage (Figure 1). The paper thus provides a unified diagnostic for why prior synthesis efforts may be less efficient than their task counts suggest, and a unified framework (the $\Omega \times \mathcal{K}$ coverage criterion) for evaluating and improving them.

A new diagnostic for evaluating agent training data. Perhaps the paper's most transferable contribution is the scenario–skill pair coverage metric as a diagnostic for training data quality. This metric can be applied to any agent training dataset — not just terminal agents — by extracting scenarios and skills from trajectories using an LLM (as in Section 4.5, Appendix E) and measuring unique counts after semantic deduplication. A dataset with high task count but low scenario–skill pair coverage is likely to produce diminishing returns from additional training, regardless of the agent architecture or training algorithm. This diagnostic provides a practical tool for data engineers deciding whether to invest in collecting more trajectories or in improving the diversity of the collection process.

Follow-Up Research This Work Enables

1. Cheap difficulty prediction from graph path features. The paper measures task difficulty post-hoc using Hy3 Preview rollouts (Table 2), at substantial computational cost. A natural follow-up is to train a difficulty predictor that takes a sampled graph path $\mathcal{P}$ as input and predicts the resulting task's difficulty tier (0/3, 1/3, 2/3, or 3/3) before synthesis. Features could include: path length $L$, average node degree along the path, domain entropy (how many distinct skill categories appear), presence of specific high-difficulty skill types, and graph-theoretic properties like betweenness centrality of the traversed nodes. Training data already exists from the paper's 3,560 categorized instances. A strong predictor would enable targeted synthesis: generate more learnable-range tasks (1/3 or 2/3 difficulty) where trajectories are information-rich (both successes and failures appear), and fewer impossible tasks (0/3) where all trajectories are failures. The key evaluation metric would be the downstream model performance when trained on difficulty-targeted vs. uniformly-sampled task sets at equal trajectory count — quantifying the efficiency gain from targeting.

2. Reinforcement learning on SkillSynth environments with verifier-based rewards. The paper validates SkillSynth only for supervised fine-tuning, explicitly noting that rubric-failed tasks should be excluded from RL to avoid erroneous reward signals. The obvious extension is to run RL fine-tuning on the 3,423 dual-passing task instances (which have aligned tests) and compare against SFT-only performance. A strong experiment would compare: (a) SFT on all 3,560 usable tasks, (b) RL on the 3,423 dual-passing tasks initialized from the SFT checkpoint, and (c) SFT + RL combined. The key question is whether RL amplifies the diversity advantage — does the agent's exploration during RL cause it to encounter novel (scenario, skill) pairs that SFT trajectories missed, producing additional coverage-driven gains? Terminal-Bench 2.0 accuracy for the best 32B model after RL would be the headline metric. A negative result (RL provides no benefit over SFT on SkillSynth tasks) would suggest that the teacher trajectories already cover the relevant $\Omega \times \mathcal{K}$ space for the benchmark, or that current RL algorithms are unstable on long-horizon terminal tasks.

3. Incremental graph expansion and diversity scaling laws. The paper claims the skill graph is "naturally scalable infrastructure" but constructs it from a static skill snapshot. A direct test of the scalability claim would add a held-out set of ClawHub skills to the existing graph, reconstruct affected subgraphs (re-running alignment and deduplication for new and neighboring nodes), and measure the incremental gain in trajectory $\Omega \times \mathcal{K}$ coverage and downstream Terminal-Bench performance. The experiment would characterize a scaling law for graph-guided synthesis: as the number of skills in the graph grows from $N$ to $N + \Delta N$, what is the marginal gain in unique scenario–skill pair coverage? A sublinear scaling curve (diminishing returns) would suggest that the current graph already saturates the practically useful skill space and additional skills are redundant. A near-linear curve would validate the scalability claim and motivate continued investment in community skill contributions. A negative result — graph quality degrades with size due to compounding alignment errors — would reveal a fundamental limitation of LLM-dependent graph construction and motivate research into more robust alignment methods.

4. Subgraph sampling for parallel-skill tasks. The paper samples linear paths from the skill graph, producing sequential workflows. Section 4.7 explicitly proposes extending this to sampling subgraphs rather than chains, which would require the agent to execute multiple skills in parallel — for example, simultaneously setting up a database AND configuring a web server AND preparing test data, then integrating them. This would increase task complexity by introducing concurrency, resource contention, and interleaved execution. The skill graph already contains the necessary structural information (multiple outgoing edges from a scenario indicate skills that could be parallelized), but the path sampling algorithm, multi-agent harness, and evaluation infrastructure would all need modification. The key experiment would measure whether subgraph-sampled tasks produce trajectories with higher $\Omega \times \mathcal{K}$ coverage than linear paths at equal synthesis cost, and whether models trained on subgraph tasks show improved performance on Terminal-Bench tasks that require parallel execution. A negative result — subgraph tasks are too hard for current agents and produce exclusively failed trajectories — would indicate that the capability ceiling for current models is below the difficulty level that parallel-skill tasks demand.

5. Cross-domain transfer from SkillSynth-trained terminal agents to non-terminal agent tasks. The paper demonstrates that SkillSynth trajectories improve performance on Terminal-Bench, but does not evaluate whether the trained agents generalize to non-terminal settings. A strong test of the diversity claim would evaluate SkillSynth-trained models on software engineering benchmarks (SWE-Bench, SWE-Bench++), code generation benchmarks (HumanEval, MBPP), and general reasoning benchmarks (ARC, MMLU) . The hypothesis is that training on diverse terminal trajectories — which expose the agent to filesystem operations, package management, debugging, configuration, and multi-step planning across 26 domains (Figure 5) — develops general-purpose computational reasoning skills that transfer beyond the terminal. The key comparison would be SkillSynth-trained vs. base Qwen3 models on these benchmarks, controlling for any general degradation from domain-specific fine-tuning. Positive transfer would substantially increase the practical value of SkillSynth; negative transfer (terminal fine-tuning degrades non-terminal performance) would suggest catastrophic forgetting and motivate research into multi-task training strategies.

6. Coverage-weighted trajectory sampling during training. The paper's theoretical framework (Equation 4) establishes that uniform coverage of $\Omega \times \mathcal{K}$ is optimal for maximizing the learnable region of the policy, but the training procedure (Section 4.3, Appendix C) treats all trajectories equally — both successful and failed, from common and rare scenario–skill pairs. A direct test of the theory would reweight trajectories during SFT based on their marginal contribution to $\Omega \times \mathcal{K}$ coverage, upweighting trajectories that contain rare (scenario, skill) pairs and downweighting trajectories that contain only common pairs. The implementation would require extracting scenarios and skills from all training trajectories (using the prompt in Appendix E), computing coverage statistics, and applying importance weights to the SFT loss. The hypothesis is that coverage-weighted training achieves higher Terminal-Bench accuracy than uniform training at equal trajectory count, because it explicitly optimizes the coverage criterion that Equation 4 identifies as theoretically optimal. A null result — coverage weighting provides no benefit — would suggest either that the SFT loss is already near-optimal under uniform sampling, or that the theoretical framing does not capture an important aspect of what makes trajectories useful (e.g., trajectory length, error-recovery richness, or solution quality matter more than raw coverage).

7. Stress-test: what if the skill graph is randomly rewired? The paper claims that scenario-mediated compatibility — where skill B can follow skill A only if their post/precondition scenarios are semantically aligned — is what makes graph-guided paths coherent and useful. A clean ablation would randomly rewire the skill graph while preserving the degree distribution: shuffle the edges so that skills point to random postcondition scenarios rather than their inferred postconditions, then run the full SkillSynth pipeline (path sampling, multi-agent harness, trajectory collection, fine-tuning) on the rewired graph. If SkillSynth's performance drops to the multi-skill baseline level (Table 4), it confirms that semantic compatibility — not graph structure per se — is the active ingredient. If performance remains above the multi-skill baseline, it suggests that any graph structure, even random, provides useful constraints that prevent the harness from simplifying tasks. This experiment would cleanly separate the contribution of the graph topology from the contribution of the semantic content of the edges.

Practical Applications and Downstream Use Cases

1. Fine-tuning on-premise terminal agents for enterprise-specific workflows. An organization with a large internal collection of shell scripts, deployment playbooks, and system administration procedures could construct a private skill graph from these assets (analogous to the ClawHub ingestion in Section 3.2), run the SkillSynth pipeline to synthesize task instances, collect trajectories using an existing strong agent (or human operators), and fine-tune a smaller open-source model for enterprise-specific terminal operations. The value proposition is that the synthesized tasks would exercise the organization's specific toolchain, file layouts, and operational conventions — producing an agent that understands not just generic terminal commands but the organization's particular infrastructure. The paper's cost of $27.3 per verified task instance, combined with the 95.7% pass rate, suggests that a few thousand organization-specific tasks could be synthesized for tens of thousands of dollars in API costs — comparable to a few engineer-days of manual task curation, but with the diversity guarantees of graph-guided sampling. The fine-tuned model would then run on-premise without sending sensitive operational data to external API providers.

2. Scaling data generation for open-source terminal agent development. The paper demonstrates that a 32B model fine-tuned on 10,680 SkillSynth trajectories achieves 29.6% on Terminal-Bench 2.0, outperforming a 480B model (Qwen 3 Coder) at 23.9%. For open-source AI organizations aiming to close the gap with proprietary terminal agents (GPT-5.3-Codex at 64.7%), this suggests a concrete scaling path: invest in expanding the skill graph (more ClawHub skills, more domains), synthesize an order of magnitude more task instances (tens of thousands rather than thousands), collect proportionally more trajectories, and train larger models (70B+ parameters). The paper provides evidence that trajectory diversity — not model size alone — is a key lever, meaning that data investment may yield higher returns than compute investment for open-source terminal agent development. The 16,632,220 enumerated 7+ skill paths from the current graph alone provide substantial headroom for scaling synthesis without even expanding the graph.

3. Curriculum learning for terminal agent training. The difficulty distribution in Table 2 (38% impossible, 37% learnable, 25% easy) suggests a natural curriculum: train first on 3/3-success tasks to establish basic skills, then on 2/3 and 1/3 tasks to push capability boundaries, and finally evaluate on 0/3 tasks as stretch goals. A practitioner could implement this by binning SkillSynth's synthesized tasks by difficulty (using a strong agent for post-hoc binning, as the paper does), then fine-tuning in stages. The paper does not evaluate curriculum training, but its difficulty-binned dataset provides the necessary infrastructure. The key practical benefit would be training stability — avoiding the situation where a model early in training encounters mostly unsolvable tasks and learns degenerate behaviors from exclusively failed trajectories. The bifurcation in Section 4.2 (rubric-failed tasks for SFT only, dual-passing for RL) provides an additional quality filter for the curriculum: RL training, if pursued, would use only the highest-quality task subset.

4. Bootstrapping terminal agent evaluation benchmarks. Terminal-Bench 2.0 contains 89 hand-crafted tasks — too few for robust evaluation and with unknown overlap with any training distribution. SkillSynth enables the creation of large-scale, automatically-generated terminal evaluation suites with controllable properties. By sampling paths from held-out regions of the skill graph, synthesizing task instances, and verifying them with the same dual verification pipeline, an organization could create evaluation sets of arbitrary size with known difficulty distributions (via post-hoc binning as in Table 2), specific domain coverage (by filtering sampled paths to desired skill categories from Figure 5), and freshness guarantees (tasks generated after model training cannot have been leaked). The 95.7% oracle pass rate ensures that synthesized evaluation tasks are solvable, and the 92.0% dual-pass rate ensures that the evaluation rubric is aligned with the task specification — both essential properties for a reliable benchmark. The paper's error analysis (Table 5) suggests specific evaluation dimensions (instruction following, verification thoroughness, exploration flexibility) that a SkillSynth-generated benchmark could systematically probe by designing tasks that stress each capability.