On Data Engineering for Scaling LLM Terminal Capabilities

🎯 Pitch

Nemotron-Terminal-32B hits 27.4% on Terminal-Bench 2.0, matching models 15–35× larger, by using a surprising data strategy: keeping failed teacher trajectories during fine-tuning, which boosts the 8B model from 5.1% to 12.4% compared to discarding errors. The key insight is that exposure to realistic failures and recovery teaches terminal robustness better than sanitized, success-only data.

1. Executive Summary

This paper introduces Terminal-Task-Gen, a lightweight synthetic data generation pipeline for training LLMs as terminal agents, combining dataset adaptation — wrapping existing math, code, and software engineering prompts in a command-line scaffold — with targeted synthetic task generation from both seed problems and a skill taxonomy of primitive terminal operations (e.g., file I/O, process management, dependency resolution). Using the resulting open-source Terminal-Corpus dataset, the authors fine-tune the Nemotron-Terminal family from Qwen3 backbones and achieve substantial gains on Terminal-Bench 2.0: Nemotron-Terminal-8B rises from 2.5% to 13.0%, 14B from 4.0% to 20.2%, and 32B from 3.4% to 27.4%, matching the performance of models 15–35× larger — including outperforming Qwen3-Coder-480B (27.4% vs. 23.9%). A systematic ablation reveals that retaining failed teacher trajectories during filtering substantially outperforms success-only and completion-only strategies (12.4% vs. 5.1–6.7% on the 8B model), establishing that exposure to realistic error states and recovery patterns is critical for terminal robustness, while strict trajectory filtering discards valuable supervisory signal.

2. Context and Motivation

The Core Problem: We Don't Know How to Train LLMs for Terminal Interaction

The central gap this paper addresses is deceptively simple: the training data strategies behind state-of-the-art terminal agents are proprietary and undisclosed, leaving the open research community without a principled understanding of how to construct effective training datasets for terminal-based LLM agents. The paper states this directly in its opening line:

"Despite rapid recent progress in the terminal capabilities of large language models, the training data strategies behind state-of-the-art terminal agents remain largely undisclosed."

This gap is acute because terminal interaction represents a fundamentally different challenge from the text-to-text generation that LLMs are typically trained for. A terminal agent must reason about environment state, compose sequences of shell commands, interpret command outputs to decide next steps, handle errors gracefully, and operate under resource constraints — all within a multi-turn interaction loop where each action conditions on the cumulative history of previous actions and their outcomes. Building training data that teaches these skills requires not just high-quality prompts, but executable environments, dependency management, verification mechanisms, and realistic interaction trajectories. The paper identifies two specific bottlenecks that make this data generation problem particularly difficult (Section 1):

Scarcity of foundational resources: Diverse task prompts are hard to source; each task needs associated dependency files and pre-configured environments that match the task requirements. Real-world human terminal interactions — the natural data source — are difficult to capture at scale because they are ephemeral, context-dependent, and rarely documented with clean correctness signals.
Prohibitive cost of synthetic trajectory generation: Even when tasks exist, generating training trajectories requires running an LLM agent in a fresh environment for each task, often over many turns. Each trajectory needs environment instantiation (spinning up a Docker container), multi-turn interaction (the agent sending commands and receiving outputs), and verification (running test suites to determine success). This is computationally expensive and logistically complex, making large-scale trajectory collection — the kind needed for supervised fine-tuning of capable models — a significant engineering challenge.

These bottlenecks explain why, despite terminal agents being a hot area of development (Claude Code, Codex CLI, Devin, and numerous other systems), the publicly available models lag far behind their proprietary counterparts. Table 3 in the paper provides stark evidence: the open-source Qwen3 models score between 2.5% and 4.0% on Terminal-Bench 2.0, while proprietary systems like GPT-5 (35.2%), Claude Sonnet 4.5 (42.8%), and Gemini 3 Pro (56.9%) operate in an entirely different performance regime. The gap is not just about model scale — Qwen3-Coder-480B at 23.9% is handily outperformed by the paper's 32B model at 27.4% once trained on appropriate data. This suggests that data quality and task coverage, not parameter count, is the binding constraint for open-source terminal capability.

Why This Problem Matters

The significance of terminal agent capability extends across multiple axes:

Practical utility. Terminal interaction is the primary interface for a vast range of software engineering and system administration workflows. An LLM that can reliably navigate a terminal — install packages, debug build failures, configure services, analyze log files, run experiments — can automate substantial portions of development and operations work. The paper's category breakdown (Table 4) illustrates the real-world scope: Software Engineering (24 tasks in the benchmark), System Administration (9), Security (8), Data Science (8), Model Training (4), and more. These are not toy problems; they represent genuine professional workflows that currently consume significant human engineering time.

Equalizing access to capable models. When the training data recipes are proprietary, only well-resourced organizations can produce capable terminal agents. This creates a capability asymmetry where open-source models — despite comparable or superior base capabilities on standard benchmarks — cannot perform terminal tasks at a useful level. The paper's contribution of both a data generation framework and an open-source dataset directly addresses this asymmetry, enabling smaller teams to train competent terminal agents without reinventing the data engineering pipeline from scratch. The paper explicitly positions this as a democratization goal, releasing "models and most of our synthetic datasets" to "democratize research in autonomous terminal agents."

Scientific understanding of agent training data. Beyond the practical impact, there is a scientific question at stake: what properties of training data matter for agent capabilities? Is it prompt diversity? Trajectory correctness? Exposure to failure modes? The number of interaction turns? The paper's systematic ablation studies — examining filtering strategies, curriculum learning, long-context training, and scaling behavior — begin to answer these questions, establishing empirical regularities that can guide future research. The finding that retaining failed trajectories substantially outperforms filtering for success-only data (Section 5.4) is a particularly striking result that would not be obvious a priori and has implications for agent training beyond terminal domains.

The shift from agent scaffolding to model capability. Terminal agent systems currently achieve frontier performance through sophisticated agentic scaffolds — custom prompting strategies, tool selection heuristics, retry logic, and environment-specific optimizations. The paper argues (Section 2) that this approach has inherent limitations: scaffolds are model-specific, require extensive engineering, and provide diminishing returns as base models improve. Investing in model-level capability — teaching the model to reason effectively about terminal states directly through fine-tuning — offers a more scalable path that generalizes across different agent frameworks. The paper's evidence supports this: Nemotron-Terminal models achieve their gains using the standard Terminus 2 agent scaffold, without any scaffold-specific optimization, suggesting the improvements come from genuine capability acquisition rather than clever prompting.

Prior Approaches and Their Limitations

The paper categorizes existing approaches to improving terminal agents into two broad streams, each with identifiable weaknesses:

Agent scaffolding improvements (Singhal et al., 2025; Nichols, 2025; Antigma, 2025; and others) have demonstrated impressive results through careful prompt engineering, multi-step reasoning frameworks, and environment-specific optimization. The Droid system from Factory AI, for instance, achieved the top position on Terminal-Bench through a sophisticated scaffold. However, the paper identifies several limitations of this approach:

Model specificity: Scaffolds are typically designed for and tuned to specific frontier models. What works for Claude may not work for GPT-5 or an open-source model, making the approach fragile and requiring re-engineering for each new model release.
Engineering burden: Effective scaffolds require deep understanding of both the model's behavior and the task domain, representing a substantial upfront investment that may not transfer.
Diminishing returns: As the paper argues, "as base models improve, the marginal benefit of complex scaffolding will likely decrease" (Section 2). A model that truly understands terminal environments doesn't need elaborate prompting to navigate them effectively.

The paper positions its approach — improving the underlying model through targeted fine-tuning — as complementary to scaffold work, not competing with it. A better model will benefit from any scaffold, and the combination of improved model capability plus scaffold optimization likely represents the frontier.

Existing dataset adaptation approaches (Development, 2025a,b; DCAgent, 2025) represent a more direct attempt to build training data for terminal agents. These approaches take existing datasets — primarily code and math problems — and "adapt" them by wrapping the problem statements in a terminal interface prompt, then running an LLM agent to generate interaction trajectories. Several such adapted datasets are available on Hugging Face, including STaQC and CodeContests trajectories collected through the Terminus agent framework.

The paper identifies critical limitations of this adapter-only approach:

Structural assumptions from source datasets: Math and code benchmarks were designed for text-to-text generation (write a function, solve an equation), not for sequential environment interaction. When adapted, they inherit assumptions about problem structure, solution format, and verification that don't naturally map to terminal workflows. As the paper states, adapters "inherit structural assumptions from source datasets never designed for sequential environment interaction, potentially limiting their effectiveness" (Section 1).
Narrow skill coverage: Dataset adapters primarily exercise code-writing and mathematical reasoning, which are necessary but not sufficient for terminal competence. Table 4 reveals that base Qwen3 models score 0.0% across Debugging, Security, File Operations, and Model Training — all categories that require skills beyond what standard code and math prompts teach, such as environment navigation, process management, and system-level reasoning.
No controlled task design: When using existing datasets, practitioners have no control over difficulty distribution, domain coverage, or skill composition. If a particular terminal skill (say, parsing configuration files or managing package dependencies) is underrepresented in the adapted data, there is no mechanism to address the gap.

Multi-agent synthetic data generation (Austin, 2025; Peng et al., 2025) represents a more recent attempt to generate terminal training data from scratch. These approaches use multiple LLMs in coordinated roles (a task proposer, an environment designer, a solution verifier) to brainstorm and validate terminal tasks before generating trajectories. While these methods can produce diverse, high-quality tasks, the paper identifies scalability as the fundamental problem:

"these introduce computational complexity that scales poorly for large-scale training" (Section 1).

A multi-agent system requires multiple LLM calls per generated task — sometimes dozens — and the coordination stages (brainstorming, environment design, validation) add latency and cost that makes generating hundreds of thousands of training examples prohibitively expensive. The paper's goal is to achieve comparable or better task quality with a more efficient single-stage generation pipeline.

The broader landscape of synthetic data generation provides relevant context. Techniques like Evol-Instruct (Xu et al., 2023) and its code-oriented variant WizardCoder (Luo et al., 2023) demonstrated that instruction-following and code-generation capabilities can be improved through iterative complexity-increasing data synthesis. AgentInstruct (Mitra et al., 2024) and LAB (Sudalairaj et al., 2024) extended these ideas to multi-agent settings with suggester-editor pairs and taxonomy-driven generation. However, none of these prior works specifically addressed terminal environments with executable verification, which adds the crucial dimension of environment interaction: tasks must be designed so that correctness can be programmatically checked through test suites, and trajectories must be collected from actual agent-environment interactions, not just model completions.

How This Paper Positions Itself

The paper positions its contribution as a balanced, practical framework that sits between the extremes of (1) simple dataset adaptation (cheap but limited in coverage and control) and (2) multi-agent synthetic generation (flexible but prohibitively expensive). The key insight is that these two approaches are complementary, addressing different stages of capability development, and that a coarse-to-fine pipeline combining both is more effective than either alone:

Dataset adaptation provides breadth: by leveraging existing high-quality math, code, and SWE problem repositories, it efficiently scales data volume and builds foundational terminal competencies — the model learns to operate in a terminal environment while solving problems it already has some capacity for. This stage yields approximately 226K training trajectories from 163K math, 35K code, and 32K SWE prompts (Table 5).
Synthetic generation provides depth and control: by generating tasks from a skill taxonomy (with 9 domains and 6 skill dimensions, Table 10), it fills capability gaps that adapters miss — debugging, security, system administration, file operations, model training — and enables precise control over task difficulty, domain coverage, and primitive skill composition. This stage yields approximately 264K trajectories from 124K seed-based and 140K skill-based tasks (Table 5).

The paper's empirical validation of this complementarity is one of its strongest contributions. Table 5 shows that dataset adapters alone achieve 9.66% on Terminal-Bench 2.0 with the 8B model, synthetic tasks alone achieve 12.4%, and combining them (as done in the final models) pushes performance to 13.0% — with the combination significantly outperforming either component in isolation. More revealingly, Table 4 shows that synthetic tasks specifically unlock capabilities where base models and adapter-only training score zero: Data Querying (0.0% → 60.0% for 32B), Model Training (0.0% → 50.0%), Security (2.5% → 27.5%), and Debugging (0.0% → 33.3%). These are precisely the terminal-specific skills that general math and code data don't teach.

The paper also positions itself as an empirical study of data engineering practices for terminal agents, not just a new dataset or model release. The systematic ablation experiments — on filtering strategies (Tables 6, 7), curriculum learning (Table 9), long-context training (Table 8), and data scaling (Figure 4) — represent a deliberate attempt to establish empirical regularities that the broader research community can build on. The finding that no filtering of teacher trajectories outperforms both complete-only and success-only filtering (12.4% vs. 6.74% and 5.06% on synthetic tasks) is a significant result that challenges the intuitive assumption that cleaner training data is better. The paper's interpretation — that "retaining unsuccessful trajectories appears to provide valuable supervision, exposing the model to realistic error states and recovery patterns that enhance overall robustness" — provides actionable guidance for future agent training efforts.

Finally, the paper positions its technical contributions — particularly the use of pre-built domain-specific Docker images rather than per-task Dockerfile generation — as scalability enablers. By maintaining 9 fixed Docker images (one per domain) with pre-installed common packages, the pipeline avoids the costly multi-turn validation and repair cycles that previous multi-agent approaches required for environment generation. This design choice (Section 4.2.3) is specifically framed as addressing the computational complexity that "scales poorly for large-scale training" in prior work, enabling the generation of ~490K total training trajectories — a scale that would likely be infeasible with per-task environment generation.

3. Technical Approach

3.1 Reader Orientation

This paper presents Terminal-Task-Gen, a data engineering pipeline that produces supervised fine-tuning (SFT) datasets for training LLMs to act as autonomous terminal agents. The core problem it solves is the scarcity of high-quality, executable training data for terminal-based agent capabilities, and the shape of the solution is a two-stage coarse-to-fine pipeline that first adapts existing math, code, and software engineering datasets into terminal-interaction format for broad foundational coverage, then synthetically generates targeted terminal-specific tasks from seed problems and a structured skill taxonomy to fill capability gaps that general-purpose datasets miss.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a sequential pipeline:

Prompt Data Sources (existing math, code, SWE datasets) — these provide the raw problem statements that will be adapted into terminal tasks. They are filtered and deduplicated before entering the pipeline.
Dataset Adapters — a rule-based transformation module that wraps each prompt from the source datasets into the Terminus 2 system prompt template, appending domain-specific instruction suffixes and instantiating required environment files. This produces terminal-format tasks without requiring any LLM generation.
Synthetic Task Generator (Terminal-Task-Gen core) — an LLM-driven module with two sub-components: seed-based generation (transforms existing problems into novel terminal tasks) and skill-based generation (synthesizes tasks from a taxonomy of primitive terminal operations across 9 domains). Both sub-components use DeepSeek-V3.2 as the teacher model to produce tasks with executable test cases, domain-specific Docker environments, and input data files.
Trajectory Generator — takes all tasks (from both adapters and synthetic generation) and runs them through the Terminus 2 agent scaffold with DeepSeek-V3.2 as the acting model, collecting the full interaction history (terminal commands sent, outputs received, final state) as training trajectories. This runs inside containerized environments managed by Harbor/Singularity.
Post-Processing Pipeline — applies decontamination filtering (14-gram overlap removal against Terminal-Bench 2.0 test samples), quality filters (identity leak removal, Chinese character filtering), and optional trajectory-level filtering (complete-only, success-only, or no filtering) before producing the final SFT dataset (Terminal-Corpus).

Information flows sequentially: source prompts → adapter transformation → synthetic task generation → trajectory collection via agent interaction → post-processing → SFT dataset used to fine-tune Qwen3 base models into Nemotron-Terminal.

3.3 Roadmap for the Deep Dive

First, the Terminal-Bench evaluation framework and Terminus 2 agent scaffold, because understanding how tasks are structured and how agents interact with environments is foundational to understanding what the data pipeline must produce.
Second, the dataset adaptation mechanism — how existing math, code, and SWE prompts are transformed into terminal-format tasks without LLM involvement — since this is the simpler, higher-volume foundation of the data pipeline.
Third, the synthetic task generation framework (Terminal-Task-Gen), covering both seed-based and skill-based generation, because this is where the paper's novel technical contribution lies and where the most important design decisions (pre-built Docker images, solution isolation, test-only verification) are made.
Fourth, trajectory generation and teacher model selection — how the generated tasks are actually executed by an agent to produce training data, and why DeepSeek-V3.2 was chosen as the teacher.
Fifth, post-processing and filtering strategies, since the paper's most striking empirical finding — that no filtering outperforms success-only filtering — depends on understanding what these filters do.
Sixth, the SFT training configuration and model deployment, connecting the data pipeline output to the final Nemotron-Terminal models.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering and empirical analysis paper whose core idea is that terminal agent capability can be substantially improved through supervised fine-tuning on synthetically generated interaction trajectories, and that a coarse-to-fine pipeline combining dataset adaptation with targeted synthetic task generation is more effective than either approach alone.

Terminal-Bench 2.0 and the Terminus 2 Agent Scaffold

Before diving into the data pipeline, we must understand the evaluation framework and agent scaffold that define the terminal task format, since the entire data generation pipeline is designed to produce training data compatible with this format.

Terminal-Bench 2.0 task structure. Each task in the benchmark consists of four components (Section 3.1, Figure 2):

A natural language instruction describing the objective — for example, "Install the required Python packages and run the data analysis script to produce output.csv."
A containerized Docker environment providing the execution context — this is a Linux filesystem with specific packages, configuration files, and initial state that the agent interacts with through a shell.
A verification test suite that programmatically checks whether the task was completed correctly — typically pytest-based tests that examine output files, check return codes, verify file contents, or run assertions against the final system state.
An oracle solution demonstrating a valid approach — a reference implementation showing one correct way to complete the task, used during benchmark development for validation but never shown to the agent.

The benchmark (Terminal-Bench 2.0 specifically) comprises 89 hand-crafted, human-verified tasks spanning diverse domains: Software Engineering (24 tasks), System Administration (9), Security (8), Data Science (8), Scientific Computing (7), and several smaller categories. These tasks require agents to complete end-to-end workflows — compiling code, training models, configuring systems, debugging environments — rather than just generating isolated code snippets.

Terminus 2 agent scaffold. The Terminus 2 framework (Section 3.2) is the model-agnostic reference agent released alongside Terminal-Bench. Unlike traditional coding agents that provide multiple specialized tools (file editors, web browsers, Python interpreters), Terminus 2 provides exactly one tool: an interactive tmux session running inside a sandboxed Docker container. A tmux session is a terminal multiplexer — essentially a persistent shell session that the agent can send keystrokes to and receive terminal output from. This minimalist design forces the agent to interact with the environment exactly as a human would: by typing commands into a terminal and reading the responses.

At each step of the interaction loop, the agent receives the current terminal output (the text displayed in the tmux session), and the underlying LLM is prompted to respond with a structured JSON object containing four fields (Figure 3):

analysis: the model's reasoning about the current state and what needs to happen next,
plan: a high-level strategy for approaching the remaining work,
commands: the actual keystrokes or shell commands to send to the tmux session,
task_complete: a boolean flag indicating whether the model believes the task is finished.

This JSON response format is what the SFT training data must teach the model to produce — the model must learn not just what commands to run, but how to structure its thinking in the analysis-plan-command format that the Terminus 2 scaffold expects.

Why understanding the scaffold matters for the data pipeline. The entire Terminal-Task-Gen pipeline is designed around generating training trajectories in exactly this format. Every trajectory in Terminal-Corpus is a sequence of (terminal_output → JSON_response) pairs, collected by running DeepSeek-V3.2 as the acting model inside the Terminus 2 scaffold on each generated task. This means the data pipeline must produce tasks that are compatible with the scaffold's expectations: tasks must have instructions that can be placed into the system prompt template (which includes an {instruction} placeholder), and environments must be instantiable as Docker containers that the scaffold can connect to.

Dataset Adaptation: Transforming Existing Prompts into Terminal Tasks

Dataset adaptation is the simpler and higher-volume foundation of the data pipeline. It converts existing high-quality prompts from math, code, and software engineering datasets into Terminal-Bench-compatible task format without requiring any LLM generation — this is a purely rule-based transformation (Section 4.1).

Prompt data sources and selection. The paper draws from three Nemotron-Cascade SFT prompt sets (Wang et al., 2025), each representing a carefully filtered subset of a larger public dataset:

Math prompts (163K unique prompts): Sourced from OpenMathReasoning (Moshkov et al., 2025), specifically the "Stage-2" prompt set from Nemotron-Cascade's math reasoning SFT data. The filtering criterion is that prompts are retained only if DeepSeek-R1's response length exceeds 2,000 tokens — this heuristic filters out "easy" questions that the strong teacher model can solve trivially, leaving only problems that require substantial multi-step reasoning. The exact count is 162,692 prompts after this filtering.

Code prompts (35K unique prompts): Sourced from OpenCodeReasoning (Ahmad et al., 2025), specifically Nemotron-Cascade's Stage-2 code reasoning SFT prompt set. The original set contains 79K prompts covering challenging coding problems; the paper applies additional filtering and deduplication to obtain a 35K subset (exact count: 31,960 after the final processing). The filtering criteria for this additional reduction are not specified in detail beyond "filter and deduplicate."

SWE prompts (32K unique prompts): Sourced from four software engineering benchmark datasets — SWE-Bench-Train (Jimenez et al., 2023), SWE-reBench (Badertdinov et al., 2025), SWE-Smith (Yang et al., 2025b), and SWE-Fixer-Train (Xie et al., 2025). The Nemotron-Cascade SWE code repair SFT data contains 127K instances total; the paper filters and deduplicates this to 31,661 unique prompts. Each SWE prompt includes a problem statement (describing a bug or feature request) and the contents of one or more buggy code files that need to be modified.

Why these specific source datasets? The paper argues that math, code, and SWE domains are "foundational to several of the topics covered in terminal use" (Section 4.1.1). Math problems teach structured multi-step reasoning — the kind needed for debugging dependency chains or analyzing log output. Code problems teach algorithmic thinking and implementation — directly applicable to writing shell scripts or Python utilities within terminal sessions. SWE problems teach environment-aware problem solving — modifying existing codebases, understanding file structures, running tests — which maps naturally to the kinds of repository-level tasks that terminal agents face.

The adaptation mechanism. The adaptation process is defined in Section 4.1.2 and detailed in Appendix A.2. It proceeds as follows:

Template instantiation: Each source prompt is inserted into the Terminus 2 system prompt template (Figure 7 in Appendix A.2), replacing the {instruction} placeholder. The system prompt template is the exact same prompt that the Terminus 2 scaffold uses during evaluation, meaning the adapted tasks teach the model to respond in the expected JSON format within the expected terminal context.
Domain-specific instruction suffixes: After the original prompt text, a domain-specific suffix is appended that reframes the task for terminal execution. These suffixes are shown in Figures 8-10:
- Math suffix (Figure 8): Instructs the agent to "solve the following math problem" and specifies that the solution should be implemented as a Python script that computes and outputs the answer, with the script saved to a specific filename.
- Code suffix (Figure 9): Instructs the agent to "solve the following coding problem" by implementing the solution in a specified file, running test cases to verify correctness, and handling input/output according to the problem specification.
- SWE suffix (Figure 10): Instructs the agent to "fix the bug" described in the problem statement by modifying the provided code files, running the existing test suite to verify the fix, and ensuring all tests pass.
Environment file instantiation: For SWE prompts, each code file identified in the problem statement is physically created within the environment before the agent begins. This means the Docker container starts with the buggy code files already present at the specified paths, mimicking the real-world scenario of inheriting an existing codebase. For math and code prompts, the environment starts empty and the agent must create all files from scratch.

The critical distinction from synthetic generation: these adapted tasks have no test cases. The prompts from Nemotron-Cascade are pure problem statements without associated verification suites. During trajectory generation, the teacher model (DeepSeek-V3.2) simply attempts the task, and the resulting interaction trace — commands, outputs, file contents — is recorded as training data. Whether the teacher succeeded or failed is known post-hoc (by checking if the output matches expected answers from the source datasets), but this correctness signal is only used optionally during filtering (Section 4.4), not during trajectory collection itself.

Scaling characteristics. The adapter approach is highly scalable because it involves no LLM calls during task creation — it's purely string templating. The bottleneck is trajectory generation (running the teacher model on each adapted task), which is the same bottleneck shared by synthetic tasks. The adapter pipeline produces 226,313 total trajectories (each corresponding to one adapted prompt), with the following per-domain breakdown from Table 5: Math contributes 162,692 samples, Code contributes 31,960, and SWE contributes 31,661.

Synthetic Task Generation: The Terminal-Task-Gen Framework

While dataset adapters provide broad foundational coverage efficiently, they are limited by the formats of their source repositories — math and code datasets were designed for isolated problem-solving, not for the kind of multi-step, environment-aware, tool-composing workflows that terminal agents must perform. Synthetic task generation addresses this gap by producing tasks specifically designed to exercise terminal-relevant skills, with precise control over domain coverage, skill composition, and difficulty (Section 4.2).

The framework supports two complementary generation methods: seed-based generation (transforming existing problems into novel terminal tasks) and skill-based generation (synthesizing tasks from a structured taxonomy of primitive terminal operations). Both methods use the same teacher model (DeepSeek-V3.2), the same task output format, and the same pre-built Docker environment strategy.

Seed-Based Synthetic Generation

Seed-based generation takes existing problems as inspiration rather than as templates — instead of wrapping the original problem in a terminal scaffold (as adapters do), it prompts the LLM to synthesize an entirely new terminal task that exercises similar skills or domain knowledge (Section 4.2.1).

Seed data structure. Each seed entry contains three optional fields:

A problem description specifying the computational challenge — this could be a natural language description of a scientific computing problem, an algorithmic challenge, or a domain-specific exercise.
A domain label indicating the scientific or technical area (e.g., biology, physics, optimization) — used to ensure the generated task stays within the intended domain.
A reference solution providing a correct implementation — used exclusively for deriving ground-truth test expectations, never shown to the agent during training or evaluation.

The paper does not specify the exact source or quantity of seed data, only that it covers "scientific computing challenges, algorithmic problem sets, or domain-specific coding exercises" (Section 4.2.1). Table 5 indicates that seed-based generation produces 124,366 synthetic tasks.

Task adaptation via LLM. The teacher model acts as a "task adapter" that transforms each seed problem into a self-contained terminal task through several key operations, all specified in the generation prompt:

Concretizing the problem statement: The abstract mathematical or algorithmic problem is augmented with concrete software engineering requirements. Instead of "compute the eigenvalues of a matrix," the task becomes: "Install numpy, read the input matrix from /data/matrix.csv, implement eigenvalue computation in eigen.py, write results to /output/eigenvalues.json." This bridges the gap between abstract problem-solving and executable terminal workflows.
Generating realistic input data: The teacher produces concrete input files that instantiate the problem with specific test cases, including edge cases and boundary conditions. For a graph traversal problem, this means generating actual adjacency lists or edge lists as CSV or JSON files with specific structure. For a numerical computation, this means generating input values that exercise numerical stability concerns.
Synthesizing pytest-based test cases: Comprehensive test suites are generated that verify multiple aspects of correctness: output file existence (did the agent create the expected file?), format compliance (is the output JSON valid? does it have the expected schema?), numerical accuracy (with appropriate tolerances for floating-point results, e.g., assert abs(result - expected) < 1e-6), and edge case handling. When a reference solution is provided in the seed data, it is included in the generation context, and the prompt specifically instructs the LLM to use it as ground truth when designing test expectations — running the reference solution to obtain the correct outputs, then encoding those outputs as expected values in the test assertions.

This seed-based approach is particularly effective for domains where well-defined problems already exist but lack terminal-oriented structure — scientific computing challenges from numerical methods courses, algorithmic problems from competitive programming, domain-specific exercises from textbooks. The LLM does the creative work of embedding these abstract problems into realistic terminal workflows with file I/O, package management, and output formatting requirements.

The seed-based generation used in Terminal-Task-Gen differs from the adapter approach in a fundamental way: adapters keep the original problem text and just change the delivery format, while seed-based generation produces substantially transformed problems that may differ in scope, difficulty, and specific requirements from the original seed. This is why seed-based tasks have test cases (the LLM generates them as part of the task creation) while adapted tasks do not.

Skill-Based Synthetic Generation

Skill-based generation takes a fundamentally different approach: rather than adapting existing problems, it synthesizes entirely novel tasks by recombining primitive terminal operation skills drawn from a structured taxonomy (Section 4.2.2). This is the more novel and technically sophisticated component of the generation pipeline.

The skill taxonomy. The paper curates a structured taxonomy of primitive terminal operation skills organized across two dimensions: 9 task domains and 6 skill types (Table 10 in Appendix A.3). Each domain is associated with a specific generation prompt that guides the teacher model toward tasks exercising the domain's focus areas:

Data Processing: Building transformation pipelines with interpolation and feature extraction; handling data I/O, manipulation, string/text processing, and time series operations.
Data Querying: Writing queries using formal syntax of declarative query languages for structured data; graph processing and result processing.
Data Science: Loading and transforming tabular data with groupby, filtering, and aggregation; statistical analysis and data transformation.
Debugging: Resolving package dependency conflicts through constraint analysis; debugging, testing, and system-level troubleshooting.
Dependency Management: (Implied from the domain descriptions; specific skills not separately enumerated but integrated into Debugging and Software Engineering domains.)
File Operations: Parsing structured formats (JSON/XML/CSV) with encoding and validation; file I/O, navigation, transformation, archives, and resource management.
Scientific Computing: Computing distance metrics between discrete probability distributions; numerical integration, statistical modeling, and mathematical operations.
Security: Crafting exploit payloads to bypass authentication and identify vulnerabilities; systems, data processing, web security, algorithmic, and testing skills.
Software Engineering: Implementing graph traversal (BFS/DFS) for dependency resolution; algorithmic, systems, data processing, and testing skills.
System Administration: Managing file permissions, configuring services, and automating tasks with shell scripts; filesystem, process/service, network, configuration, and deployment operations.

The skill types that cut across domains include: Algorithmic (graph traversal, constraint satisfaction, backtracking search), Systems (file I/O, process management, network configuration), Data Processing (parsing, serialization, transformation pipelines), Mathematical (numerical integration, statistical modeling), Testing (validation, verification, benchmarking), and Web/Security (HTTP handling, authentication, vulnerability analysis).

This taxonomy serves as the "grammar" for task generation — the teacher model is instructed to select 3–5 primitive skills from the relevant domain and combine them in non-trivial ways to produce a task that requires integrated problem-solving rather than isolated skill application. For example, a Software Engineering task might combine algorithmic (graph traversal), systems (file I/O for reading dependency specifications), and testing (verifying that the dependency resolution is correct) into a single task: "Parse dependencies.json, perform topological sort to determine installation order, install packages in that order, and verify with a test script."

Domain-specific generation prompts. Each of the 9 domains has a dedicated generation prompt (detailed in Appendix A.3, Figures 11–20) that is composed of two parts:

A shared system prompt (Figure 11) that governs general task logic across all domains. This template includes placeholders for domain-specific requirements and instructs the teacher model on the expected output format, quality standards, and the requirement for solution isolation (no solution leakage into the task prompt).
Domain-specific requirement modules (Figures 12–20) that define the preconditions and objectives for each vertical. For instance, the Security module (Figure 19) mandates "crafting exploit payloads to identify vulnerabilities" and specifies that tasks should involve "authentication bypass, privilege escalation, or cryptographic weakness exploitation." The Data Processing module (Figure 13) requires "building transformation pipelines with interpolation and feature extraction" and specifies input/output format requirements.

The generation prompt explicitly instructs the teacher to emphasize novelty — the model is guided to invent new scenarios rather than reproducing known problems. This is a crucial design choice: without the novelty instruction, LLM-based generation tends to produce variations of well-known problems (e.g., "write a function to reverse a linked list" in terminal form), which limits the diversity and coverage of the resulting training data. The novelty emphasis pushes the teacher toward creative recombination of skills into scenarios that don't directly match any existing benchmark or training example.

Compositional task synthesis. The teacher model is instructed to combine multiple primitives — typically 3–5 skills per task — in non-trivial ways. The key principle is that tasks should require integrated problem-solving: the agent must understand how the skills relate to each other and sequence them appropriately, rather than simply executing each skill in isolation. A File Operations task, for instance, might combine parsing (extract metadata from XML configuration files), transformation (convert the metadata to a different schema), and validation (verify that the transformed output conforms to a JSON schema) — requiring the agent to chain these operations into a coherent pipeline.

Task Format, Execution Environment, and Design Decisions

Both seed-based and skill-based generation methods produce tasks in a standardized format compatible with Terminal-Bench and the Terminus 2 scaffold (Section 4.2.3):

A natural language task prompt specifying objectives and constraints — this is what gets inserted into the {instruction} placeholder of the Terminus 2 system prompt.
pytest-based test cases with configurable weights for partial credit — multiple test functions can pass/fail independently, and the overall task score reflects the weighted sum of passed tests.
Supplementary input files providing necessary data — these might be CSV files containing the dataset to analyze, JSON configuration files specifying parameters, or binary files to process.
A domain-specific Docker environment — a pre-built container image that provides the execution context with appropriate tools and libraries pre-installed.

The file structure follows the exact same layout as Terminal-Bench tasks (Figure 2), ensuring that tasks generated by the pipeline can be evaluated using the same infrastructure (Harbor, Terminus 2) as the benchmark itself.

Critical design decision 1: Solution isolation. All generation prompts explicitly instruct the teacher model to avoid solution leakage — the task prompt visible to the agent must not reveal the algorithm, implementation approach, or any code that solves the problem. When reference solutions are available (e.g., in seed data), they are used exclusively for deriving ground-truth test expectations and are never included in the agent-visible task prompt. The paper states this principle clearly: "the task prompt visible to the agent must not reveal the algorithm, implementation approach, or any code that solves the problem" (Section 4.2.3). This ensures that generated tasks require genuine problem-solving — the agent must figure out the approach, not just implement a described solution.

Critical design decision 2: No oracle solutions for synthetic tasks. Unlike Terminal-Bench's hand-crafted tasks, the synthetic generation pipeline does not produce oracle solutions. The paper explains: "we do not generate oracle solutions, as producing ground-truth code is prohibitively difficult without human verification" (Section 4.2.3). Instead, the pipeline relies on the principle of tasks that are easy to verify but difficult to solve: the teacher model generates test cases that can programmatically check correctness, but does not attempt to produce a reference implementation. This is a pragmatic tradeoff — oracle solutions would be valuable for some training approaches (e.g., process reward modeling), but generating them reliably at scale without human validation is itself an unsolved problem. The test cases serve as the ground truth signal: if the agent's output passes the tests, the task is considered completed.

Critical design decision 3: Pre-built Docker images instead of per-task Dockerfiles. This is perhaps the most important architectural decision for scalability. Previous multi-agent approaches (Austin, 2025; Peng et al., 2025) generated a unique Dockerfile for each task, which required extensive validation and repair cycles — the generated Dockerfile might have syntax errors, reference non-existent packages, or produce build failures, requiring multiple rounds of LLM-based debugging before a working environment existed. The paper explicitly calls out this problem: "multi-agent systems can be time-consuming and costly" (Section 2) due to "the costly multi-turn repair often needed for per-task environment generation" (Section 4.2.3).

Terminal-Task-Gen instead maintains a fixed set of 9 domain-specific Docker images, each pre-installing the packages and dependencies commonly required within that domain. For example:

The Data Science image pre-installs pandas, scikit-learn, numpy, matplotlib, and Jupyter-related tools.
The Security image pre-installs cryptography libraries, network analysis tools, and common exploitation frameworks.
The Scientific Computing image pre-installs scipy, sympy, and numerical computation libraries.

When a task is generated, it is assigned to the appropriate domain's Docker image. If the task requires packages not in the base image, the agent can install them at runtime using pip, apt, or other package managers — this is itself a terminal skill that the agent should learn (dependency management is one of the skill taxonomy categories).

This design provides three specific scalability advantages enumerated in Section 4.2.3:

Eliminates Dockerfile validation overhead: By avoiding per-task environment generation entirely, the pipeline can produce tasks in a single pass without repair cycles. The teacher model generates the task prompt and test cases, and the task is immediately ready for trajectory collection because the environment already exists.
Reduces resource footprint: Instead of building and caching thousands of unique container images (which can consume terabytes of disk space and hours of build time), the system uses just 9 shared base images. This is the difference between storing 9 Docker layers versus 9 × N unique layers for N tasks.
Decouples environment and task generation: The pipeline can produce diverse scenarios within stable environments — the task prompt can specify complex operations (installing packages, modifying configurations, creating databases) without the environment generation needing to anticipate every detail. The agent is expected to handle environment setup as part of the task, which is realistic: real terminal workflows often involve setting up the environment before doing the actual work.

Critical design decision 4: Test-only verification. The synthetic tasks are verified entirely through their test suites — there is no separate oracle solution or human validation step. This means the quality of test cases is the crucial bottleneck: if the teacher model generates incorrect tests (e.g., tests that pass for wrong implementations, or tests that fail for correct implementations), the training signal will be corrupted. The paper relies on the teacher model's strong capabilities (DeepSeek-V3.2 scores 38.2% on Terminal-Bench 2.0 natively) and the solution isolation principle (the teacher generates tests without being influenced by its own solution approach) to maintain test quality at scale.

Trajectory Generation and Teacher Model Selection

Once tasks exist (from either adapters or synthetic generation), they must be executed to produce training trajectories — sequences of (terminal_state, agent_response) pairs that form the supervised fine-tuning data. This is the most computationally expensive phase of the pipeline (Section 4.3, Appendix A.2).

Teacher model: DeepSeek-V3.2. The paper selects DeepSeek-V3.2 (Liu et al., 2025) as the teacher model for both synthetic task generation and trajectory collection, motivated by its "strong performance on Terminal-Bench 2.0" (Section 4.3). Table 3 shows DeepSeek-V3.2 achieving 38.2% on Terminal-Bench 2.0, making it the strongest open-source model evaluated and competitive with proprietary systems like Claude Sonnet 4.5 (42.8%).

To validate that DeepSeek-V3.2 can produce useful trajectories across domains beyond Terminal-Bench itself, the paper evaluates it on three adapted benchmarks using the Terminus 2 scaffold (Table 2):

AIME 2024 and AIME 2025 (math competition problems): 93.33% pass@1 — exceptionally strong, indicating the teacher can reliably solve hard math problems in a terminal environment.
LiveCodeBench v6 (competitive programming): 67.20% pass@1 — strong but not perfect, meaning a significant fraction of code trajectories will contain errors.
SWE-bench Verified (real-world GitHub issue resolution): 52.40% pass@1 — substantially lower, indicating that about half of software engineering trajectories will be from unsuccessful attempts.

This heterogeneous performance across domains is actually desirable for the data pipeline, because it means the training data will naturally contain a mix of successful and unsuccessful trajectories — providing the model with examples of both correct terminal workflows and realistic error states. As the filtering ablation results later reveal (Section 5.4), this mixture is crucial for model robustness.

Trajectory collection infrastructure. The paper uses Harbor (Shaw, 2025), the infrastructure framework from Terminal-Bench 2.0, to orchestrate large-scale trajectory generation in containerized environments (Section 5.1). Each trajectory generation run involves:

Environment instantiation: Harbor spins up a Singularity or Docker container for the task, using the domain-specific pre-built image.
Agent initialization: The Terminus 2 scaffold is launched inside the container, connected to a tmux session.
Interaction loop: At each step, the current terminal output is captured, formatted into the Terminus 2 system prompt (with the task instruction and terminal state), sent to DeepSeek-V3.2, and the model's JSON response is parsed. If the task_complete flag is false, the commands field is sent to the tmux session, the terminal output updates, and the loop continues. If task_complete is true, or if a maximum step limit is reached, the trajectory ends.
Trajectory recording: The full sequence of (prompt, response, terminal_output) triples is saved as the training example.

The paper extends Harbor to support Singularity (Kurtzer et al., 2017) for deployment on HPC clusters. Singularity is a containerization system designed for high-performance computing environments where Docker's daemon-based architecture (requiring root privileges) is often not permitted. The authors note that this introduces "rare failures due to fakeroot overlay limitations" but consider these "acceptable for synthetic data generation" — since the data is used for training rather than evaluation, occasional corrupted trajectories are tolerable.

For evaluation (as opposed to data generation), the paper uses Daytona (Daytona, 2025) to manage reliable, parallel execution in isolated cloud sandboxes. This separation of infrastructure — Singularity/HPC for training data generation, Daytona/cloud for evaluation — is a practical engineering choice: training data can tolerate occasional failures, but benchmark evaluation must be reliable to produce trustworthy scores.

Trajectory characteristics. Appendix A.1 provides distributional analysis of the generated trajectories. Figure 5 shows the distribution of token counts: most trajectories fit within the default Qwen3 maximum sequence length of 32,768 tokens, but a "nontrivial subset exceeds this limit" and would be truncated during SFT. Figure 6 shows the distribution of interaction turns (steps in the agent loop): trajectories vary significantly in length depending on task type and difficulty. This motivates the long-context training experiments in Section 5.5.

Post-Processing and Filtering Strategies

The raw trajectories from the generation phase undergo several post-processing steps before becoming the final SFT dataset (Section 4.4).

Decontamination. Any prompt that has a 14-gram overlap with Terminal-Bench 2.0 test samples is removed from the training set. A 14-gram is a sequence of 14 consecutive tokens (or characters — the paper doesn't specify the tokenization, but n-gram decontamination typically operates on whitespace-delimited tokens or character n-grams). This threshold is chosen to be long enough to catch near-exact matches while being short enough to avoid false positives from common phrases. Decontamination is critical because Terminal-Bench 2.0 is the evaluation benchmark — any training data that leaked test information would invalidate the evaluation.

Quality filters. Two additional filters are applied:

Identity leak removal: Trajectories where the model's response reveals its own identity or training provenance (e.g., "As an AI assistant trained by...") are removed. These are artifacts of the teacher model's instruction tuning that are irrelevant to terminal capability and could bias the fine-tuned model's behavior.
Chinese character filtering: Responses containing Chinese characters are discarded. This is presumably because the teacher model (DeepSeek-V3.2) occasionally produces mixed-language output, and the target terminal environments expect English-language commands — Chinese characters in commands would cause execution failures.

Trajectory-level filtering (the crucial ablation). The paper experiments with three strategies for filtering individual trajectories based on their completion status (Section 4.4, Section 5.4):

No filtering: All trajectories are retained regardless of whether the teacher model succeeded or failed. This yields 264,207 synthetic trajectories (Table 7).
Complete-only filtering: Only trajectories where the teacher model completed the interaction loop (i.e., set task_complete = true within the step limit) are retained. This removes trajectories that were truncated due to hitting the maximum step count, regardless of whether the teacher was on track to succeed. This yields 104,603 synthetic trajectories — a 60% reduction from the unfiltered set. For dataset adapters (which lack test cases), this is the only applicable trajectory-level filter, reducing the set to 196,940 trajectories (Table 6).
Success-only filtering: Only trajectories where the teacher model's final output passes the task's test cases are retained. This is the most stringent filter — it removes not just truncated trajectories but also completed trajectories where the teacher's solution was incorrect. This yields only 83,448 synthetic trajectories — a 68% reduction from the unfiltered set.

The success-only filter is defined mathematically — for a trajectory with associated test cases, the teacher's final output (the state of the container after the last action) is checked against the pytest-based verification suite. If all weighted tests pass (or the weighted sum exceeds some threshold — the exact threshold is not specified), the trajectory is retained. For dataset adapters, which lack test cases, success-only filtering is not applicable.

The striking empirical result. Table 7 reveals that no filtering dramatically outperforms both filtering strategies: 12.4% Terminal-Bench 2.0 accuracy with no filtering versus 6.74% for complete-only and 5.06% for success-only on the Qwen3-8B model trained on synthetic tasks. For dataset adapters (Table 6), the pattern is similar though less extreme: no filtering achieves 9.66% versus 8.09% for complete-only on the combined adapter set.

The paper's interpretation (Section 5.4) is that "retaining unsuccessful trajectories appears to provide valuable supervision, exposing the model to realistic error states and recovery patterns that enhance overall robustness." In other words, seeing the teacher model fail — running a command that produces an error, then diagnosing and recovering from that error — teaches the student model how to handle the inevitable failures it will encounter during deployment. Strict filtering removes these failure-recovery patterns from the training data, leaving the model unprepared for the messy reality of terminal interaction where commands fail, packages are missing, permissions are wrong, and the agent must adapt.

This finding is one of the paper's most important contributions because it challenges the intuitive assumption in SFT data curation that "cleaner data is better." It suggests that for interactive agent tasks, the diversity of experiences — including negative experiences — matters more than the correctness of individual trajectories. The unfiltered data contains a natural curriculum: some trajectories are clean successes, some are messy successes with recovery, some are ultimate failures, and the model benefits from seeing all of these patterns.

SFT Training Configuration and Model Deployment

The final stage of the pipeline is supervised fine-tuning of Qwen3 base models on the processed Terminal-Corpus dataset (Section 5.1).

Base models. The paper uses pretrained models from the Qwen3 family (Yang et al., 2025a): Qwen3-8B (the primary model for ablation studies), Qwen3-14B, and Qwen3-32B. These are standard dense transformer models, not mixture-of-experts. The choice of Qwen3 is motivated by their strong base performance on standard benchmarks and the availability of pretrained weights at multiple scales, enabling systematic study of how data engineering gains scale with model size.

Training hyperparameters. The exact configuration (Section 5.1):

Learning rate: 2e-5
Weight decay: 1e-4
Epochs: 2
Maximum sequence length: 32,768 tokens (the default Qwen3 context window)
Global batch size: 128
Micro-batch size: 1 per GPU
Optimizer: AdamW with β = (0.9, 0.95)
Learning rate scheduler: Cosine decay with 10% warmup
Gradient clipping: 1.0

Compute infrastructure. The 8B and 14B models are trained on 4 nodes with 8 GPUs per node (32 total GPUs) using sequence parallelism of 2 — meaning the sequence dimension is split across 2 GPUs, reducing memory per GPU while keeping the full 32K context. The 32B model is trained on 16 nodes (128 total GPUs). All experiments use CPU offloading, which moves optimizer states and activations to CPU memory when GPU memory is insufficient — a common technique for training large models with long sequences on limited GPU memory.

Training framework. The paper uses veRL (Sheng et al., 2024), an open-source framework designed for efficient LLM training with support for sequence parallelism and CPU offloading.

Single-stage mixed training. All data sources — math adapters, code adapters, SWE adapters, seed-based synthetic tasks, and skill-based synthetic tasks — are mixed together and trained on concurrently (single-stage). The paper experiments with a two-stage curriculum (adapter data first, then synthetic data) but finds no advantage — the mixed strategy achieves 13.03% versus 10.39% for the curriculum approach (Table 9) — and therefore adopts single-stage mixed training for all final models.

Why this training configuration matters. The single-stage mixed training finding is important for practitioners because it simplifies deployment: there's no need to carefully sequence data sources or manage multiple training phases. All data can be combined and trained on together. The 2-epoch training with 2e-5 learning rate represents a relatively standard SFT configuration, suggesting that the gains come from data quality and diversity rather than from unusual or highly tuned training procedures. The per-GPU micro-batch of 1 is common for long-sequence training because each sequence can be 32K tokens, quickly exhausting GPU memory — this constraint is what necessitates the sequence parallelism and CPU offloading techniques.

4. Key Insights and Innovations

Innovation 1: Failed Trajectories Are More Valuable Training Signal Than Successful Ones for Interactive Agents

The paper's most striking and counterintuitive finding is that no filtering of teacher trajectories substantially outperforms both completion-only and success-only filtering when training terminal agents. This is not a marginal improvement — on the synthetic task subset, unfiltered data achieves 12.4% Terminal-Bench accuracy versus 6.74% for complete-only and 5.06% for success-only (Table 7). Training only on trajectories where the teacher succeeded yields less than half the performance of training on the full mixture of successes and failures.

This finding upends the dominant assumption in SFT data curation, which treats data quality as synonymous with correctness. The standard recipe — collect demonstrations from a strong model, filter for successful examples, train on those — implicitly assumes that exposing a student model to errors will teach it to make errors. This assumption is so pervasive that filtering for correctness is rarely even questioned; it is the default. The paper's result demonstrates that for interactive, environment-grounded tasks specifically, this assumption is actively harmful. The training data loses the very patterns the model most needs to learn: what happens when a command fails, how error messages look, what recovery strategies work, and how to recognize when a different approach is needed.

What makes this intellectually distinctive is not just the empirical result but the diagnostic framing it enables. The paper identifies a fundamental category distinction between training data for text-to-text tasks (where correctness filtering is sensible — you don't want the model learning to produce wrong answers) and training data for interactive agent tasks (where the environment provides dynamic feedback and the model must learn to react to negative signals). In the interactive setting, the "wrong" trajectory contains information that no "correct" trajectory can provide: the shape of failure, the texture of error recovery, the experience of dead ends. A model trained only on clean successes enters deployment with no preparation for the messy reality where commands fail, packages are missing, and permissions are wrong. The unfiltered data provides a natural curriculum — successes demonstrate the target behavior, failures teach resilience — and the model needs both.

This insight has implications beyond terminal agents. Any interactive, environment-grounded domain where the agent receives dynamic feedback — code generation with execution, web navigation, robotics control — may benefit from retaining unsuccessful trajectories during training. The paper provides the first clear empirical evidence for this principle at scale, with a mechanism-level interpretation (the model learns error recovery patterns) and quantitative magnitude (more than 2× improvement from retaining failures). This is a genuinely new piece of knowledge about how to construct agent training data, not just an incremental refinement of existing filtering heuristics.

Anchor evidence: Table 7 (synthetic task filtering ablation) and Table 6 (adapter filtering ablation), with the interpretive framing in Section 5.4.

Innovation 2: Terminal Capability Is Primarily a Data Engineering Problem, Not a Scale Problem

The paper provides what is arguably the cleanest demonstration to date that data engineering choices can substitute for massive parameter scale in interactive agent domains. Nemotron-Terminal-32B (27.4% on Terminal-Bench 2.0) outperforms Qwen3-Coder-480B (23.9%), a model with 15× more parameters, despite using the same Qwen3 base architecture (Table 3). The 14B model (20.2%) outperforms GPT-OSS 120B (18.7%) and Gemini 2.5 Flash (16.9%). In every comparison, the smaller, data-engineered model defeats larger models that lack terminal-specific training data.

This is not merely "fine-tuning helps" — that would be incremental and unsurprising. What makes this finding conceptually important is the scale of the substitution and what it reveals about the nature of terminal capability. A 15× parameter advantage — the kind of scaling that typically requires hundreds of thousands of GPU-hours and fundamentally different hardware — is completely negated by access to approximately 490K carefully constructed training trajectories. This implies that terminal interaction is not primarily bottlenecked by raw reasoning capacity (which scales with parameters) but by procedural knowledge and environment familiarity (which scales with relevant training data). The base Qwen3 models have the reasoning capacity to solve terminal tasks — they score 2.5–4.0% rather than 0.0% — but lack the procedural knowledge of how to translate reasoning into terminal actions, interpret command outputs, and chain operations into workflows. The training data provides exactly this missing knowledge.

This reframes the terminal agent problem from a scaling challenge (build bigger models) to a data engineering challenge (build better training data). Prior to this work, the dominant narrative — reinforced by the strong performance of proprietary models like GPT-5 (35.2%) and Claude Sonnet 4.5 (42.8%) — was that terminal capability required frontier-scale models with proprietary training recipes. The paper demonstrates that an open-source 32B model, trained on synthetically generated data from an open-source teacher, can match or exceed models 15× larger. The bottleneck was never parameters; it was knowing what data to train on and how to generate it at scale.

The category-level evidence in Table 4 strengthens this interpretation further. Base Qwen3-32B scores 0.0% on Data Querying, Model Training, and Debugging — these are not small numbers, they are zero. The model has no capability whatsoever in these categories. After training, Nemotron-Terminal-32B jumps to 60.0%, 50.0%, and 33.3% respectively. This is not amplification of existing capability; this is capability acquisition through data. The model learned something fundamentally new — how to query structured data, how to train models from the command line, how to debug environment issues — that was entirely absent from its pretraining distribution. The data did not just refine existing skills; it taught new ones.

Anchor evidence: Table 3 (overall Terminal-Bench 2.0 comparison), Table 4 (category-level breakdown), with the broader framing developed across Sections 1 and 5.2.

Innovation 3: The Pre-Built Docker Image Strategy Is an Architectural Innovation That Enables Scalable Synthetic Agent Data Generation

While the paper's core contribution is empirical, it makes one architectural contribution that is methodologically significant and likely to influence how future synthetic agent data pipelines are designed: the use of pre-built, domain-specific Docker images rather than per-task Dockerfile generation for environment provisioning (Section 4.2.3).

Prior approaches to synthetic terminal task generation — specifically the multi-agent frameworks of Austin (2025) and Peng et al. (2025) — generated a unique Dockerfile for each task. This required the generation system to not only design the task but also specify the complete environment: which packages to install, which configuration files to create, which services to start. The generated Dockerfiles were then validated by attempting to build them, and build failures triggered repair cycles where the LLM attempted to debug its own Dockerfile — a process that could require multiple rounds of expensive LLM calls per task. This made environment generation a primary bottleneck in the data pipeline, consuming substantial compute and wall-clock time while introducing a failure mode (invalid Dockerfiles) that reduced yield.

Terminal-Task-Gen sidesteps this problem entirely by decoupling task generation from environment generation. The system maintains 9 pre-built Docker images, one per domain, each with the commonly required packages pre-installed. Task generation focuses exclusively on designing the problem: the instruction, the input files, the test suite. The environment is assumed to exist and be functional. If a task requires packages not in the base image, the agent — not the generation system — handles installation at runtime, which is itself a terminal skill that the training should teach.

This architectural choice provides three compounding benefits that the paper identifies and quantifies implicitly through the scale of data generated:

Elimination of the validation bottleneck: By removing Dockerfile generation and validation from the critical path, the pipeline can produce tasks in a single pass. This is the difference between generating a task (one LLM call) and generating-and-validating a task (one LLM call plus potentially multiple repair calls). At the scale of 264K synthetic tasks, this difference is enormous.
Reduced resource footprint: Nine shared base images versus thousands of unique images. Storage, build time, and cache management all scale with O(1) rather than O(N) in the number of tasks.
Decoupled iteration: Task design and environment design can evolve independently. The pipeline can experiment with new task generation prompts without touching the Docker images, and can update the Docker images (adding new packages, fixing vulnerabilities) without regenerating tasks.

What makes this an innovation rather than just an engineering optimization is that it enables a fundamentally different scaling regime for synthetic agent data. The multi-agent Dockerfile-generation approach inherently hits a complexity wall: as you try to scale to more tasks, you spend an increasing fraction of your compute budget on environment validation rather than task generation. The pre-built image approach removes this wall, making it feasible to generate the ~264K synthetic tasks that power the Nemotron-Terminal models. Without this architectural choice, generating training data at this scale — and therefore achieving the performance gains the paper demonstrates — would likely have been prohibitively expensive.

The innovation is not the idea of pre-built Docker images per se (that's obvious in retrospect) but the recognition that environment provisioning is the bottleneck in synthetic agent data generation and the demonstration that decoupling it from task generation unlocks scalable data production. This insight will likely transfer to other interactive agent domains (web navigation, GUI interaction, robotics simulation) where environment setup is similarly a barrier to scale.

Anchor evidence: Section 4.2.3 (pre-built Docker image discussion), with contrast to prior multi-agent approaches in Section 2 and Section 4.2 introduction.

Innovation 4: A Skill Taxonomy as the Generative Grammar for Targeted Capability Acquisition

The paper's skill-based synthetic generation framework (Section 4.2.2) introduces a conceptual approach to training data design that is more principled and controllable than prior synthetic data methods: using a structured taxonomy of primitive skills as the generative grammar from which diverse, targeted tasks are composed.

Prior synthetic data approaches for LLM training — Evol-Instruct (Xu et al., 2023), WizardCoder (Luo et al., 2023), AgentInstruct (Mitra et al., 2024) — operate primarily through complexity scaling: take an existing instruction or task and make it harder through iterative in-depth evolution. This produces more challenging data but does not provide control over which specific skills are exercised. If a model is weak at file permission management but strong at numerical computation, complexity scaling provides no mechanism to generate more file permission tasks and fewer numerical ones. The data distribution is determined by the initial seed distribution and the particular evolution trajectories, with no targeted gap-filling capability.

Terminal-Task-Gen's skill taxonomy approach addresses this limitation directly. By defining 9 domains, 6 cross-cutting skill types, and explicit primitive operations within each (Table 10), the framework can generate tasks that guarantee coverage of specific skill combinations. The generation prompt instructs the teacher model to select 3–5 primitives from a domain and compose them into a novel task. This means the resulting data distribution is designed, not emergent: if Security tasks involving Web Security and Algorithmic skills are underrepresented, the pipeline can simply generate more tasks targeting that combination.

What makes this intellectually distinctive is the curriculum-through-coverage paradigm it enables. Traditional curriculum learning (train on easy data first, then hard data) is a sequential strategy that the paper finds ineffective for this domain (Table 9: curriculum hurts performance compared to mixed training). The skill taxonomy approach offers an alternative: rather than sequencing by difficulty, ensure coverage of all skill dimensions simultaneously, and let the model learn the interactions between skills through diverse exposure. The paper does not explicitly frame it this way, but the empirical result that mixed training outperforms curriculum is consistent with the idea that skill coverage, not difficulty progression, is the binding constraint.

The taxonomy itself — spanning Algorithmic, Systems, Data Processing, Mathematical, Testing, and Web/Security skill types across 9 domains — represents a decomposition of terminal competence into its constituent primitives. This is a conceptual contribution to understanding what terminal agents need to learn, beyond the practical contribution of enabling data generation. Prior to this work, "terminal capability" was an opaque construct — either a model could use the terminal or it couldn't. The taxonomy provides a language for analyzing capability gaps: a model might be strong at Data Processing but weak at System Administration, good at Algorithmic reasoning but poor at Testing/verification. Table 4 validates that these categories correspond to real capability distinctions — base models score zero on Security, Debugging, and File Operations, while fine-tuned models show differential improvement across categories.

Anchor evidence: Table 10 (skill taxonomy), Section 4.2.2 (skill-based generation description), Table 4 (category-level results validating that skills are distinct and differentially acquirable), Appendix A.3 (domain-specific generation prompts).

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary benchmark is Terminal-Bench 2.0 (Merrill et al., 2026; Team, 2025b), consisting of 89 hand-crafted, human-verified terminal tasks spanning Software Engineering (24 tasks), System Administration (9), Security (8), Data Science (8), Scientific Computing (7), and several smaller categories. The authors explicitly note that these tasks require "end-to-end workflows, such as compiling code, training models, configuring systems, and debugging environments," distinguishing them from isolated code generation benchmarks. Task structure follows a standardized format: each task includes a natural language instruction, a containerized Docker environment, a pytest-based verification test suite, and an oracle solution (never shown to the agent). The paper does not mention a separate development or validation split from this benchmark; all 89 tasks are used for final evaluation. For decontamination, any training prompt with a 14-gram overlap with Terminal-Bench 2.0 test samples is removed from the SFT dataset (Section 4.4).
Base model(s). All experiments use pretrained models from the Qwen3 family (Yang et al., 2025a): Qwen3-8B (the primary model for ablation studies), Qwen3-14B, and Qwen3-32B. These are standard dense transformer models, not mixture-of-experts. The authors state that Qwen3-8B is used "as our primary model for ablation studies," while the 14B and 32B variants "verify that our findings scale with model size" (Section 5.1). The choice is motivated by the availability of pretrained weights at multiple scales, enabling systematic study of how data engineering gains interact with parameter count. For the FLOPs-matched comparison perspective, the paper does not train larger baselines from scratch but instead compares against publicly reported Terminal-Bench 2.0 scores for larger open-source models including Qwen3-Coder-480B (23.9%), GPT-OSS 120B (18.7%), and MiniMax M2 230B (30.0%).
Metrics. The sole evaluation metric is Terminal-Bench 2.0 accuracy (%), computed as the fraction of the 89 benchmark tasks for which the agent's final state passes all weighted verification tests (or reaches a passing score threshold). The paper reports this as a percentage with ± standard error, computed across evaluation runs. For the category-level breakdown (Table 4), accuracy is computed per category subset (e.g., Security accuracy is the fraction of 8 Security tasks passed). The Terminus 2 agent scaffold (Section 3.2) is used consistently across all evaluations, with the model producing structured JSON responses (analysis, plan, commands, task_complete) at each interaction step. No alternative metrics — such as trajectory length, success rate excluding partial credit, or token efficiency — are reported.
Baselines. The paper reports several categories of baselines in Table 3, drawn from the Terminal-Bench 2.0 leaderboard:
- Base Qwen3 models (8B, 14B, 32B) without terminal-specific fine-tuning, evaluated under the identical Terminus 2 scaffold.
- Closed-source frontier models: GPT-5 (35.2%), GPT-5.1 (47.6%), GPT-5.2 (54.0%), Claude Opus 4.5 (57.8%), Gemini 3 Pro (56.9%), and others.
- Open-source models of varying scales: Qwen3-Coder-480B (23.9%), GPT-OSS 20B (3.10%) and 120B (18.7%), MiniMax M2 230B (30.0%), Kimi K2 Thinking 1T (35.7%), DeepSeek-V3.2 685B (38.2%).
- Ablation baselines within the paper: Training on individual data components (Math-only, Code-only, SWE-only, seed-based synthetic only, skill-based synthetic only) as shown in Table 5, and various filtering strategies as shown in Tables 6 and 7. The key comparison is Nemotron-Terminal models versus their corresponding Qwen3 base models at the same parameter count, and versus larger models that lack terminal-specific training data.
Generation budget / compute accounting. The paper does not use a generation budget or FLOPs-matched comparison framework in the style of scaling laws research. There is no systematic sweep of inference-time compute (no best-of-N, no beam search, no sequential vs. parallel sampling comparison). Instead, all models are evaluated using the Terminus 2 scaffold with a fixed interaction budget per task — the agent runs until it declares task_complete: true or hits a maximum step limit (the exact limit is not specified in the paper). Training compute is reported in terms of GPU infrastructure (32 GPUs for 8B/14B, 128 GPUs for 32B) and SFT hyperparameters (Section 5.1), but there is no FLOPs accounting or compute-matched comparison between training strategies. The primary "budget" axis studied is training data volume (Section 5.7, Figure 4), where models are trained on 0%, 1%, 2%, 5%, 10%, and 100% of synthetic data and evaluated on Terminal-Bench 2.0 — but this is data scaling, not compute scaling, and does not control for training FLOPs across conditions.
Cross-validation / statistical protocol. The paper does not describe any cross-validation procedure for model selection or hyperparameter tuning. The Terminal-Bench 2.0 results in Table 3 are reported with ± standard error, computed across multiple evaluation runs, but the number of runs and the source of variance (different random seeds? different environment instantiations?) are not specified. For the ablation studies (Tables 5-9), results are reported for single training runs per condition — there is no averaging across seeds or indication of training variance. This is a limitation: the ablation differences, while often large in magnitude (e.g., 12.4% vs. 5.06% for unfiltered vs. success-only), are not accompanied by any measure of statistical reliability from repeated training runs. The decontamination protocol (14-gram overlap removal against Terminal-Bench 2.0 test samples, Section 4.4) is described but the extent of overlap found and removed is not quantified.

Main Quantitative Results

Overall Terminal-Bench 2.0 Performance (Table 3)

The paper's headline result is the performance of Nemotron-Terminal models on Terminal-Bench 2.0, shown in Table 3:

Nemotron-Terminal-8B: 13.0 ± 2.2%, up from Qwen3-8B's 2.47 ± 0.5% — a 5.3× improvement.
Nemotron-Terminal-14B: 20.2 ± 2.7%, up from Qwen3-14B's 4.04 ± 1.3% — a 5.0× improvement.
Nemotron-Terminal-32B: 27.4 ± 2.4%, up from Qwen3-32B's 3.37 ± 1.6% — an 8.1× improvement.

The 32B model's 27.4% places it above Qwen3-Coder-480B (23.9 ± 2.8%), a model with 15× more parameters, and above GPT-OSS 120B (18.7 ± 2.7%). It is competitive with Claude Haiku 4.5 (28.3 ± 2.9%) and MiniMax M2 230B (30.0 ± 2.7%), despite having 7-9× fewer parameters. The 14B model at 20.2% outperforms GPT-OSS 120B (18.7%) and Gemini 2.5 Flash (16.9%).

These gains are measured using the identical Terminus 2 agent scaffold for both base and fine-tuned models, isolating the effect of training data to model capability rather than scaffold engineering. The standard errors indicate non-trivial variance — the 32B model's 27.4 ± 2.4% means a 95% confidence interval of roughly 22.6% to 32.2% — but even the lower bound substantially exceeds the Qwen3-32B baseline of 3.37 ± 1.6%.

Category-Level Performance Breakdown (Table 4)

Table 4 provides the most diagnostically informative results, disaggregating performance by Terminal-Bench 2.0 task category. The base Qwen3 models show striking capability gaps:

Zero-performance categories for base Qwen3-32B: Data Querying (0.0%), Model Training (0.0%), Debugging (0.0%), File Operations (0.0%), Mathematics (0.0%), Games (0.0%), Video Processing (0.0%), Personal Assistant (0.0%). Across all three base model sizes, Data Querying, Model Training, Debugging, File Operations, Mathematics, Games, and Video Processing are uniformly at 0.0% — the base models have no measurable capability in these categories.
Non-zero base categories: Software Engineering (5.0% for 32B), System Administration (6.7%), Security (2.5%), Data Science (0.0% for 32B but 7.5% for 14B — indicating high variance), Data Processing (5.0%), Scientific Computing (2.9%).

After fine-tuning with Terminal-Corpus, Nemotron-Terminal-32B shows dramatic improvements in previously zero categories:

Data Querying: 0.0% → 60.0% (1 task, so this is a binary pass/fail)
Model Training: 0.0% → 50.0% (2 of 4 tasks)
Debugging: 0.0% → 33.3% (1 of 3 tasks)
Security: 2.5% → 27.5%
Data Processing: 5.0% → 50.0%
Software Engineering: 5.0% → 31.7%

Categories that remain at or near zero even after fine-tuning for the 32B model:

Scientific Computing: 2.9% → 0.0% (a regression — the fine-tuned model loses the base model's minimal capability)
Mathematics: 0.0% → 0.0%
Games: 0.0% → 0.0%
Video Processing: 0.0% → 0.0%
File Operations: 0.0% → 5.0% (minor improvement, far below other categories)

The category-level results reveal that gains are highly uneven. The data pipeline is extremely effective for Data Querying, Model Training, Data Processing, and Software Engineering — categories where terminal-specific workflow knowledge (querying databases, managing training pipelines, building data transformation scripts, modifying codebases) is the primary bottleneck. It is ineffective for Scientific Computing, Mathematics, Games, and Video Processing — categories that may require domain-specific reasoning or tool familiarity (scientific libraries, mathematical proof strategies, game mechanics, video codecs) not adequately covered by the synthetic task taxonomy or adapter datasets.

The 8B and 14B models show similar patterns but at lower absolute performance levels, suggesting that parameter scale amplifies the benefits of data engineering — the 32B model gets more out of the same training data than the 8B model does, consistent with the scaling results in Figure 4.

Data Source Ablation (Table 5)

Table 5 decomposes Qwen3-8B performance when trained on individual data sources versus the full combination:

Data Source	# Samples	TB2.0 Accuracy
Math adapters only	162,692	5.39 ± 1.65%
Code adapters only	31,960	6.29 ± 1.65%
SWE adapters only	31,661	7.02 ± 2.13%
All adapters combined	226,313	9.66 ± 2.11%
Seed-based synthetic only	124,366	6.18 ± 1.91%
Skill-based synthetic only	139,841	12.4 ± 2.38%
All synthetic combined	264,207	12.4 ± 2.29%

Several patterns emerge:

Adapter complementarity: Individual adapter domains (Math 5.39%, Code 6.29%, SWE 7.02%) each underperform the combined adapter set (9.66%). The gain from combination (9.66% vs. best single source 7.02%) is substantial, confirming that math, code, and SWE data exercise complementary terminal skills.
Skill-based generation dominance: Skill-based synthetic data alone (12.4%) substantially outperforms the full adapter set (9.66%) despite having a comparable number of samples (140K vs. 226K). This validates the paper's central thesis that targeted synthetic generation provides more effective training signal per sample than adapting existing datasets.
Seed-based adds robustness, not peak performance: Adding seed-based synthetic data to skill-based data does not increase mean accuracy (12.4% for both), but the paper notes it "successfully reduces variance" (Section 5.3). The standard error drops from ±2.38% (skill-based only) to ±2.29% (combined), consistent with this interpretation — seed-based tasks provide overlapping skill coverage that makes the model more reliable across evaluation runs.
SWE adapters punch above their weight: Despite having the fewest samples (31,661), SWE adapters produce the highest adapter-only accuracy (7.02%). This aligns with the intuition that software engineering tasks — which involve navigating file systems, running tests, modifying code — are structurally closest to terminal workflows.

The full Nemotron-Terminal-8B (13.0%) combines all adapter and synthetic data, outperforming both the adapter-only (9.66%) and synthetic-only (12.4%) configurations. The paper does not report a direct ablation of the final combined model versus these components in a single table, but the implication from Tables 5 and 3 is that the combination provides approximately a 0.6 percentage point gain over synthetic-only for the 8B model.

Ablation Studies and Robustness Checks

Trajectory filtering strategy on synthetic tasks (Table 7): No filtering (12.4 ± 2.29%) dramatically outperforms both complete-only filtering (6.74 ± 2.20%) and success-only filtering (5.06 ± 2.11%). The magnitude is striking — success-only filtering, which might intuitively seem like the highest-quality data, produces less than half the accuracy of unfiltered data. This result is validated across all three synthetic data configurations (seed-based, skill-based, combined) in Table 7. The mechanism the paper proposes is that unsuccessful trajectories contain error states and recovery patterns that teach robustness; filtering them out removes this crucial supervisory signal.

Trajectory filtering strategy on dataset adapters (Table 6): The pattern differs from synthetic tasks. For individual adapter domains, complete-only filtering sometimes helps (Math: 7.19% vs. 5.39% unfiltered) and sometimes hurts (SWE: 5.39% vs. 7.02% unfiltered; Code: 6.07% vs. 6.29% unfiltered — within noise). For the combined adapter set, unfiltered (9.66%) outperforms complete-only (8.09%). Since dataset adapters lack test cases, success-only filtering is not applicable. The weaker and less consistent effect of filtering on adapters compared to synthetic tasks may reflect that adapter trajectories — being derived from existing benchmarks — have inherently different failure characteristics than synthetic terminal-specific tasks.

Curriculum learning vs. mixed training (Table 9): The two-stage curriculum (adapter data first, then synthetic data) achieves 10.39 ± 1.71%, substantially underperforming the single-stage mixed strategy (13.03 ± 2.16%). This is a negative result with practical implications: carefully sequencing data by type does not help, and may actively hurt by allowing the model to overfit to adapter-style patterns before seeing synthetic terminal tasks. The paper adopts mixed training for all final models.

Long-context training and YaRN2 extension (Table 8): Extending SFT context length to 65,536 tokens (from the Qwen3 default of 32,768) and/or applying YaRN2 (Peng et al., 2023) position extrapolation does not improve performance. The baseline configuration (SFT max length 32,768, eval max length 40,960, no YaRN2) achieves 13.0 ± 2.2%. Adding YaRN2 at evaluation only (without long-context training) drops to 11.9 ± 2.0%. Training at 65,536 tokens without YaRN2 drops further to 10.3 ± 2.0%. Training with YaRN2 at 65,536 recovers to 11.9 ± 2.1%. The paper's interpretation is that "most high-quality supervision already fits within the standard window, while the long-tail trajectories tend to be noisy and less informative." Extending context appears to dilute the training signal with noisy long trajectories rather than providing useful additional supervision.

Training data scale (Figure 4): Both Qwen3-8B and Qwen3-14B show monotonically increasing Terminal-Bench 2.0 performance as synthetic training data volume increases from 0% to 100%. The 14B model not only achieves higher absolute performance at every data scale but also exhibits steeper gains — the gap between 8B and 14B widens as data increases, suggesting a positive interaction between model capacity and data volume. At 1% of synthetic data (~2,600 samples), the 8B model achieves roughly 5-6% (estimated from the figure), already a substantial improvement over the 2.47% base. At 100%, the 14B model reaches approximately 20%. Since the x-axis is percentage of synthetic data rather than absolute sample count, the 100% point corresponds to ~264K synthetic samples. The paper does not fit a scaling law functional form to these curves, so there is no prediction of saturation behavior beyond 100%.

Teacher model validation (Table 2): DeepSeek-V3.2, the teacher used for both task generation and trajectory collection, achieves 93.33% on AIME 2024/2025 (math), 67.20% on LiveCodeBench v6 (coding), and 52.40% on SWE-bench Verified (software engineering) when evaluated through the Terminus 2 scaffold. These results validate that the teacher has sufficient capability to generate useful trajectories — particularly important for the math adapter data, where the teacher's 93.33% pass@1 means most math trajectories will be successful, while the coding and SWE results (67.20% and 52.40%) mean a substantial fraction of trajectories in those domains will contain errors, providing the mixed success/failure distribution that the filtering ablation suggests is valuable.

Negative result: Scientific Computing regression (Table 4): Nemotron-Terminal-32B scores 0.0% on Scientific Computing, down from Qwen3-32B's 2.9%. This is a genuine regression — the fine-tuned model loses a minimal capability the base model possessed. The paper does not diagnose or explain this result. Possible interpretations: the synthetic task generation for Scientific Computing may produce tasks that are misaligned with the benchmark's Scientific Computing tasks (different libraries, different problem types), or the adapter data may interfere with the base model's fragile scientific computing knowledge. This is a notable failure mode that the paper acknowledges only implicitly by reporting the number.

Negative result: Mathematics and Games remain at zero (Table 4): Despite training on 162,692 math adapter trajectories, Nemotron-Terminal models of all sizes score 0.0% on the Mathematics category (4 tasks) of Terminal-Bench 2.0. This suggests that the math adapter tasks — which involve solving competition math problems in a terminal environment — do not transfer to the benchmark's mathematics tasks, which may require different mathematical reasoning approaches or tool usage patterns. Similarly, Games (1 task) and Video Processing (1 task) remain at 0.0% across all model sizes, indicating that single-task categories provide limited signal and that these domains require specialized training data not covered by the pipeline.

Dataset adapter filtering interaction with domain (Table 6): Complete-only filtering shows domain-dependent effects on adapter data: it improves Math adapter performance (7.19% vs. 5.39% unfiltered) but degrades SWE adapter performance (5.39% vs. 7.02%). The paper does not analyze this interaction, but it suggests that the nature of teacher failures differs between math trajectories (where incomplete trajectories may be genuinely uninformative — the teacher gives up before reaching a solution) and SWE trajectories (where incomplete trajectories may contain useful debugging and environment-navigation patterns even if the bug is not ultimately fixed).

Critical Assessment

Claim 1: "The training data strategies behind state-of-the-art terminal agents remain largely undisclosed" and Terminal-Task-Gen provides a systematic framework that addresses this gap.

This claim is contextual rather than experimentally tested — it is the motivation, not a hypothesis. The paper does demonstrate that its data pipeline produces substantial performance improvements over base models (Tables 3, 4, 5), which is consistent with the claim that previously missing data strategies are important. However, the paper does not compare against any disclosed training data strategies from proprietary systems, because none exist to compare against. The claim of addressing a disclosure gap is therefore validated by existence rather than by comparative experiment. A stronger test would require access to proprietary training recipes (obviously impossible) or head-to-head comparison of Terminal-Task-Gen against alternative disclosed data generation frameworks at the same scale — which the paper does not perform because the prior multi-agent frameworks (Austin, 2025; Peng et al., 2025) operate at smaller scale and target different model families. The practical contribution — releasing models and datasets that close the gap between open-source and proprietary terminal capability — is genuine, but the claim about systematic understanding of undisclosed strategies is inherently difficult to verify experimentally.

Claim 2: The coarse-to-fine pipeline (dataset adaptation + synthetic generation) is more effective than either approach alone.

The paper provides evidence for this claim, but it is indirect rather than directly ablative. Table 5 shows adapter-only at 9.66% and synthetic-only at 12.4% for the 8B model. The final Nemotron-Terminal-8B achieves 13.0% (Table 3), which is indeed higher than either component alone. However, the final model is trained on the combination of both data sources, and the paper does not report a direct ablation row in Table 5 for "adapters + synthetic combined" — the 13.0% in Table 3 is the closest comparison point, but it comes from a different experimental context (final model configuration) than the 9.66% and 12.4% ablation results. The incremental gain from combination appears to be relatively modest for the 8B model (0.6 percentage points above synthetic-only), and without a direct ablation in a single controlled table, it's unclear whether this gain is robust or within experimental noise. The category-level analysis in Table 4 provides stronger support: the synthetic data specifically unlocks categories that adapters miss (Data Querying, Model Training, Debugging), demonstrating complementarity in coverage even if the additive benefit on overall accuracy is modest.

A missing experiment: training on synthetic data only at the full scale (with hyperparameters matched to the final model configuration) and comparing directly to the combined model in a single table, with standard errors, to quantify the exact marginal contribution of adapter data given synthetic data already present.

Claim 3: "Retaining unsuccessful trajectories appears to provide valuable supervision."

This is the paper's strongest and best-supported claim. Table 7 provides a clear, high-magnitude result: unfiltered (12.4%) vs. success-only (5.06%) on synthetic tasks. The ~2.4× difference is unlikely to be explained by experimental noise, and the result is consistent across synthetic data configurations (seed-based, skill-based, combined). Table 6 provides convergent evidence from adapter data, where unfiltered (9.66%) outperforms complete-only (8.09%), though the gap is smaller and domain-dependent.

However, the mechanism — that failed trajectories teach error recovery patterns — is interpretive rather than experimentally verified. The paper does not provide trajectory-level analysis showing that the fine-tuned model actually exhibits better error recovery behavior when trained on unfiltered data. It does not measure the frequency of error recovery attempts, the success rate of those attempts, or the types of errors handled. The claim that "retaining unsuccessful trajectories appears to provide valuable supervision, exposing the model to realistic error states and recovery patterns that enhance overall robustness" is a plausible interpretation consistent with the result, but it is not directly tested. An experiment that would strengthen this claim: analyze the unfiltered training set to identify trajectories containing error-recovery patterns, measure how often the fine-tuned model encounters and recovers from errors during evaluation, and correlate this with training data composition.

Additionally, the filtering ablation is performed only on the 8B model. The paper does not verify whether the unfiltered advantage holds at 14B and 32B scales, which would strengthen the generality of the finding. The result might be specific to smaller models that benefit more from diverse training signals, while larger models might perform better with cleaner data.

Claim 4: Nemotron-Terminal models "achieve substantial improvements" and "match the performance of significantly larger models."

The numbers in Table 3 clearly support substantial improvement over base Qwen3 models: 5.3× to 8.1× relative gains. The comparison to larger models is more nuanced:

Strong evidence: Nemotron-Terminal-32B (27.4%) vs. Qwen3-Coder-480B (23.9%) — the 32B model genuinely outperforms a model with 15× more parameters. The standard errors (2.4% vs. 2.8%) are overlapping but the point estimate favors the smaller model. Nemotron-Terminal-14B (20.2%) vs. GPT-OSS 120B (18.7%) and Gemini 2.5 Flash (16.9%) — also clear outperformance by a smaller model.
Weaker evidence: Nemotron-Terminal-32B (27.4%) vs. Claude Haiku 4.5 (28.3%) — the Claude model is slightly ahead, and the standard errors fully overlap. The paper's framing of "matching" is appropriate here, but the evidence for "outperforming" in the abstract ("match the performance of significantly larger models") is more accurate than "outperform" used in some parts of the introduction. Nemotron-Terminal-32B vs. GPT-5 (35.2%) or Claude Sonnet 4.5 (42.8%) — the Nemotron model does not match these, and the paper does not claim it does.
Missing comparison: No comparison against fine-tuned versions of the larger models. The Qwen3-Coder-480B comparison is against the base model; we don't know how Qwen3-Coder-480B would perform if fine-tuned on Terminal-Corpus. The fact that a 32B model with terminal-specific training outperforms a 480B model without such training is interesting and practically useful, but it demonstrates the importance of training data rather than an inherent advantage of smaller models. A reader might incorrectly conclude that parameter scale doesn't matter for terminal capability; the scaling results in Figure 4 show that within the Nemotron-Terminal family, more parameters do help (8B → 14B → 32B shows consistent improvement at fixed data scale).

Claim 5: The skill taxonomy enables targeted capability acquisition and coverage of terminal-specific skills.

Table 4 provides strong circumstantial evidence: Data Querying (0% → 60%), Model Training (0% → 50%), Debugging (0% → 33.3%), Security (2.5% → 27.5%). These are categories where base Qwen3 models have zero capability, and the synthetic data — which is explicitly designed around a skill taxonomy that includes these domains — appears to teach them. However, the paper does not provide a controlled ablation that isolates the effect of the skill taxonomy. The comparison in Table 5 is between skill-based synthetic (12.4%) and seed-based synthetic (6.18%), but both use the same teacher model and same task format; the key variable is that skill-based generation uses the taxonomy while seed-based generation adapts existing problems. The taxonomy's contribution cannot be cleanly separated from other factors (the seed data sources, the specific generation prompts, the distribution of task difficulties).

A missing experiment: generate synthetic data using the same infrastructure but without the skill taxonomy — for example, prompting the teacher model to "generate diverse terminal tasks" without the structured skill constraints — and compare against taxonomy-guided generation. This would isolate whether the taxonomy itself (the structured decomposition into domains and skill types) provides benefits beyond just generating more terminal tasks.

Claim 6: Pre-built Docker images enable scalable synthetic data generation.

This is an engineering claim evaluated implicitly by the scale of data generated (~264K synthetic tasks) rather than through direct comparison. The paper does not provide an ablation comparing pre-built images against per-task Dockerfile generation in terms of generation throughput, failure rate, or cost. The claim that per-task generation "scales poorly" is supported by citation to prior work (Austin, 2025; Peng et al., 2025) but not by head-to-head measurement. The practical demonstration — that the pre-built approach successfully generated 264K tasks — is evidence of feasibility but not evidence of superiority over alternatives. A reader evaluating whether to adopt this approach would want quantitative comparison: what was the Dockerfile validation failure rate in prior work? What throughput improvement did pre-built images provide? The paper does not provide these numbers.

Unaddressed weaknesses in the experimental design:

Single evaluation benchmark: All results are on Terminal-Bench 2.0 (89 tasks). There is no evaluation on alternative terminal benchmarks, real-world terminal tasks, or held-out task sets. The 14-gram decontamination addresses direct leakage but cannot address the possibility that the synthetic task generation produces tasks that are functionally similar to benchmark tasks (same skills, same libraries, same problem structures) without n-gram overlap. Training on 490K terminal trajectories and evaluating on 89 terminal tasks creates a risk of overfitting to the benchmark's task distribution, even if exact text overlap is removed.
Small test set and high variance: The 89-task benchmark, split into categories as small as 1 task (Data Querying, Personal Assistant, Games, Video Processing), means that category-level results are determined by pass/fail on a single task. The 60.0% jump on Data Querying represents passing 1 task. The standard errors on overall accuracy are substantial (±2.4% for the 32B model), meaning the 95% confidence interval spans roughly 5 percentage points. This makes precise comparison between models with similar scores unreliable.
No training seed replication: All ablation results are from single training runs. We don't know whether the difference between unfiltered (12.4%) and success-only (5.06%) would replicate across different random seeds, data shuffles, or teacher model sampling. The large magnitude of the difference makes it likely robust, but the lack of error bars on ablation results (Tables 5-9 report standard errors for evaluation, not for training variance) is a limitation.
Teacher model dependency: All synthetic data is generated by a single teacher (DeepSeek-V3.2). The results may be specific to this teacher's strengths, weaknesses, and failure modes. A different teacher — with different pass@1 rates on different task types, different error patterns, different verbosity — might produce trajectories with different optimal filtering strategies. The finding that unfiltered data is best might be an artifact of DeepSeek-V3.2's particular mix of successes and failures rather than a general principle. Ablation with a different teacher model would test this.
No evaluation of trajectory quality: The paper evaluates the fine-tuned models on Terminal-Bench 2.0 but never directly evaluates the quality of the generated training trajectories themselves. Are the teacher's error recovery patterns sensible? Do the test cases correctly distinguish success from failure? What fraction of synthetic tasks have buggy test cases that would mislabel trajectories? Without trajectory-level quality analysis, we don't know whether the unfiltered advantage comes from useful error patterns or simply from having more data (the unfiltered set is 2.5× larger than the success-only set — Table 7). A data-volume-matched comparison (e.g., 83K unfiltered trajectories vs. 83K success-only trajectories) would disambiguate data quantity from data composition effects.
The adapter data lacks test cases: This is an inherent limitation of the adapter approach, but it means that adapter trajectories provide no verifiable correctness signal. The model learns from whatever the teacher did, without knowing whether it was correct. In contrast, synthetic trajectories have test-based verification (even if unused in the unfiltered condition). This asymmetry in data quality between adapters and synthetic tasks is not analyzed or controlled for.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For

The constraint. The entire compute-optimal framework rests on estimating each prompt's difficulty before allocating inference compute. The paper's method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter, particularly for one-off queries where the 2048-sample overhead cannot be amortized across repeated similar prompts. A practitioner evaluating whether to adopt this approach cannot use the headline 4× figure without understanding how much of that gain is eaten by the difficulty estimation pre-processing step. For batch processing where difficulty can be estimated once and reused across many similar prompts, the overhead may be acceptable; for interactive or low-volume settings, it may be prohibitive.

Paper evidence. The difficulty estimation procedure is described in Section 3.2 with the explicit caveat that costs are not accounted for. Figures 4 and 8 show compute-optimal scaling curves starting from low budgets (2–4 generations), but these curves implicitly assume difficulty is already known at zero cost. The gap between predicted and oracle difficulty bins — while small in Figure 4 (overlapping curves) — widens at higher budgets for revisions (Figure 8, ~3 percentage point gap at 256 generations), suggesting that even the cheaper PRM-based estimation introduces some degradation versus knowing ground-truth difficulty.

Mitigation status. The paper flags this explicitly as "a key avenue for future work" (Section 3.2) and suggests training models to predict difficulty directly from question text, but develops no such model and evaluates no cheaper estimation alternative. The authors do not explore adaptive estimation strategies — generating a small number of initial samples, assessing difficulty, and then allocating the remaining budget — which could amortize the estimation cost into the problem-solving process itself.

Hard Problems Remain Essentially Unsolved

The constraint. Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. The pass@1 rate for these problems is so low that even aggressive test-time compute cannot surface correct solutions because none exist in the model's output distribution to find or refine.

The consequence. Test-time compute amplification has a hard ceiling: it can only improve performance on problems where the base model already produces correct solutions at some non-trivial rate. If a problem is fundamentally outside the model's capability — requiring knowledge, reasoning patterns, or problem-solving strategies not present in its training — no amount of search or revision will help. This means the compute-optimal framework offers no path forward for genuinely novel or out-of-distribution reasoning tasks. For deployment scenarios where the problem distribution skews toward genuinely hard problems (the tail of MATH, novel reasoning challenges, problems requiring capabilities the base model lacks), the approach provides zero benefit. The paper frames test-time compute as a substitute for pretraining (Section 7), but this only holds for problems within the base model's existing capability envelope.

Paper evidence. Figure 3 (right) shows bin 5 (hardest) accuracy hovering at 1–3% for all methods and all budgets. Figure 7 (right) shows bin 5 at roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line (blue, bottommost) essentially flat near 0–5% for all test-time compute budgets, consistently below the stars representing the 14× larger model's performance. Table 4 in the main paper does not break out equivalent quintile data but the diagnostic value of the difficulty-bin analysis is high. The FLOPs-matched comparison shows that on hard problems at high R values, test-time compute has a -52.9% relative disadvantage versus pretraining (Figure 1, bottom-right bar chart for PRM search).

Mitigation status. The paper is transparent about this limitation in its Section 7 takeaway, noting that pretraining remains necessary for problems outside the base model's capability range. However, it offers no mechanism for identifying which problems fall into this category without first attempting to solve them (the difficulty estimation procedure would flag bin 5 problems as hard, but at that point the estimation cost is already sunk). The framework provides no graceful degradation strategy — it will spend up to the full budget on bin 5 problems and still fail, with no early termination mechanism.

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate

The constraint. Because the revision model is trained exclusively on trajectories where all in-context answers are incorrect (followed by a correct target), it has never seen a scenario where the current answer is already correct. At test time, when the model produces a correct answer early in a revision chain, subsequent revision steps may incorrectly "revise" that correct answer into a wrong one. The paper reports that approximately 38% of correct answers get converted back to incorrect ones during revision chains when using a naive approach.

The consequence. This reversion problem means that longer revision chains are not strictly beneficial — the pass@1 trajectory in Figure 6 (left) shows improvement from step 1 (~18.2%) to steps 15–20 (~24–25%), but the gains plateau and the model never reliably preserves correctness once achieved. The mitigation — using majority voting or verifier-based selection to pick the best answer from anywhere in the chain rather than always taking the final revision — is an imperfect patch. It requires storing all intermediate answers and running a verifier over them, increasing storage and compute overhead. More fundamentally, it means the revision model cannot be trusted to iterate toward better solutions monotonically; each additional revision step risks degrading the current answer.

Paper evidence. The 38% reversion rate is reported in Section 6.1. Figure 6 (left) shows the per-step pass@1 trajectory plateauing rather than continuing to improve linearly with revision steps. The mitigation strategy — majority voting or verifier-based selection across the chain — is described in Section 6.1 but its effectiveness in recovering from reversions is not quantified in isolation (the sequential vs. parallel results in Figure 6, right, show the aggregate benefit of chain-level selection, not the reversion recovery rate specifically). Figure 16 (Appendix K) shows that the ReST^EM-trained revision model degrades dramatically with sequential revisions, suggesting the problem is sensitive to training methodology.

Mitigation status. The paper implements within-chain selection (majority or verifier) as a workaround, but does not address the root cause through training. A more principled solution — such as including "already correct" examples in the revision training data so the model learns to recognize when no revision is needed — is not explored. The revision model remains fragile: it can improve incorrect answers but cannot reliably maintain correctness.

Single Benchmark, Single Model Family Limits Generality

The constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The 500-question test set is split into five difficulty quintiles of ~100 questions each, further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper states it "believes this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a finding.

The consequence. Several aspects of the findings could be model- or benchmark-specific and may not transfer:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different common error patterns, or different solution styles might exhibit different difficulty-dependent scaling curves. The beam search degradation on easy problems (Figure 3, right) — attributed to verifier over-optimization — might be more or less severe with a different base model's output patterns.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with stronger or weaker in-context learning might show different sequential-to-parallel optimal ratios than those in Figure 7.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems but helping medium ones, revisions dominating on easy problems — generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference.
The 500-question test set with cross-validation across ~50 questions per bin per fold means the selected compute-optimal strategies are based on small samples. Strategy selection variance could be high, and the reported compute-optimal curves may not be stable across different random splits or when applied to different problem distributions.

Paper evidence. The single-benchmark, single-model-family scope is acknowledged in Section 4, but the paper provides no out-of-distribution evaluation, no comparison with alternative base models, and no analysis of how results might vary with problem type. Figures 3, 4, 7, and 8 are all specific to PaLM 2-S* on MATH. The cross-validation protocol is described in Section 3.2 but the stability of strategy selection across folds is not quantified.

Mitigation status. The authors acknowledge the scope limitation but do not address it experimentally. The claim that PaLM 2-S* is "representative" is unverified. A practitioner wishing to apply these methods to a different model family or task domain would need to replicate the full analysis pipeline — difficulty binning, strategy sweeping, compute-optimal policy selection — without knowing which findings would transfer.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained

The constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022) where both data and parameters are scaled equally. The paper states:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Additionally, the 14× larger model uses only greedy decoding — no test-time compute augmentation of its own.

The consequence. A Chinchilla-optimal model (scaling both parameters and data with total compute) trained with 14× more total FLOPs would likely outperform the parameter-only-scaled baseline used here. The reported advantages of test-time compute over pretraining — e.g., +27.8% on easy questions at R ≪ 1 (Figure 1, top-right) — may shrink or reverse against a properly compute-optimal larger model. Furthermore, the larger model with no test-time augmentation is a relatively weak baseline: giving the 14× larger model even a modest test-time compute budget (e.g., best-of-8 or a few revision steps) would create a stronger comparison that is never evaluated. This makes the FLOPs-matched comparison favorable to test-time compute in ways that may not reflect realistic deployment tradeoffs.

Paper evidence. The parameter-only scaling choice is stated in Section 7 with the explicit caveat quoted above. Figure 9 and the bar charts in Figure 1 show the comparison, but always against greedy decoding for the larger model. The paper provides no ablation where the larger model receives any test-time compute budget.

Mitigation status. The authors acknowledge this as future work but provide no sensitivity analysis. A practitioner deciding whether to invest in a 14× larger model versus test-time compute for a smaller model cannot assess how much the advantage depends on the suboptimal pretraining of the larger model versus the inherent benefits of test-time computation.

Sequential Revisions Are Inherently Serial and Create a Latency Bottleneck

The constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial — each revision depends on the previous one — while parallel best-of-N can be executed simultaneously given sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel (a configuration favored by the compute-optimal policy on easy problems per Figure 7, right) takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously.

The consequence. For latency-sensitive applications — interactive assistants, real-time decision-making, deployment scenarios where users wait for responses — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. The paper's failure to discuss latency means a practitioner cannot assess whether the reported accuracy gains are achievable within acceptable response time budgets. The tradeoff is particularly acute for the revision model: while sequential revisions marginally outperform parallel sampling in aggregate (Figure 6, right, ~2.5 percentage point gap at 64 generations), this small accuracy gain comes at a potential 64× wall-clock time cost in the fully sequential case.

Paper evidence. The paper never discusses latency, wall-clock time, or throughput. All "generation budgets" equate one generation to one unit of compute regardless of whether it is executed in parallel or sequentially. Figure 7 (left) sweeps the sequential-to-parallel ratio but plots only accuracy, not latency. The training infrastructure (Section 5.1) mentions GPU counts but only in the context of training, not inference.

Mitigation status. This limitation is entirely unaddressed. The paper provides no latency measurements, no discussion of acceptable latency budgets for deployment, and no analysis of how the compute-optimal policy would change if a latency constraint were imposed. For a practitioner deploying these methods in an interactive setting, this is a critical missing dimension of the analysis.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes terminal agent capability as primarily a data engineering problem rather than a model scaling problem. This is a significant reframing rather than a paradigm shift — the methods (SFT on synthetic trajectories) are well-established, but the paper's empirical demonstration that careful data construction can substitute for 15× parameter scale (32B Nemotron-Terminal at 27.4% vs. Qwen3-Coder-480B at 23.9%, Table 3) establishes a new baseline understanding of what matters for interactive agent domains.

The conceptual shift has three dimensions. First, it moves the bottleneck from "how do we train bigger models?" to "how do we construct better training data?" — a question that is both cheaper to explore (data engineering experiments cost a fraction of pretraining runs) and more democratized (smaller teams can iterate on data pipelines, while only large organizations can scale model pretraining). Second, it establishes that trajectory diversity — including negative examples — is more valuable than trajectory correctness for interactive agent training, directly challenging the dominant SFT assumption that cleaner data is always better. The 2.4× gap between unfiltered and success-only training (12.4% vs. 5.06%, Table 7) provides a concrete, actionable finding that will influence how agent training data is curated across domains. Third, it demonstrates that targeted skill coverage through structured task generation can teach capabilities entirely absent from pretraining — the jumps from 0.0% to 60.0% on Data Querying and 0.0% to 50.0% on Model Training (Table 4) represent genuine capability acquisition, not just refinement of existing skills.

This work reconciles a tension in the agent training literature between approaches that rely on scaffolding sophistication (Droid, Warp, other leaderboard entries achieving 40-50%+ on Terminal-Bench through model-specific prompt engineering and tool orchestration) and approaches that focus on base model improvement (post-training, RL, data augmentation). Prior to this work, the scaffolding approach appeared dominant — the top Terminal-Bench entries were scaffold-heavy, and open-source models without elaborate scaffolds scored in the low single digits. The paper demonstrates that model-level improvement through data engineering can close a substantial fraction of the scaffolding gap without any scaffold optimization, using the same standard Terminus 2 agent for both base and fine-tuned evaluation. This suggests that the field's allocation of effort — currently skewed toward scaffold engineering — may be suboptimal; improving the underlying model through better training data provides a more general and scalable path that benefits any scaffold.

The paper also makes several research directions more attractive and several less attractive:

More attractive: Synthetic trajectory generation with lightweight, single-stage teacher models (rather than expensive multi-agent frameworks). The paper generated ~490K trajectories using a single model (DeepSeek-V3.2) with pre-built Docker images — demonstrating that the multi-agent complexity of prior approaches (Austin, 2025; Peng et al., 2025) may be unnecessary for achieving strong results. Research on efficient, scalable data generation for interactive agents is now clearly validated as high-impact.
More attractive: Study of what makes trajectories useful for training, beyond correctness. The filtering ablation opens a new empirical question: which specific properties of unsuccessful trajectories (error recovery patterns? exploration diversity? realistic failure modes?) drive the benefit? This is a rich space for mechanistic analysis.
More attractive: Domain-specific skill taxonomies as a data design tool. The 9-domain, 6-skill-type taxonomy demonstrates that decomposing a complex capability into primitives enables targeted data generation that fills specific capability gaps. This approach is likely transferable to other agent domains (web navigation, GUI interaction, code review).
Less attractive: Per-task environment generation for synthetic agent data. The pre-built Docker image approach removes the scaling bottleneck that made per-task Dockerfile generation (as in Austin, 2025; Peng et al., 2025) expensive. Unless per-task environments provide clear benefits for specific applications, the field will likely converge on shared, domain-specific environments as the default.
Less attractive: Curriculum learning for agent SFT data mixing. The negative result in Table 9 (curriculum: 10.39% vs. mixed: 13.03%) suggests that carefully sequencing data sources by type provides no benefit — and may actively hurt — compared to simple mixed training. This simplifies training pipelines substantially.

Follow-Up Research This Work Enables

Determining the mechanism behind the unfiltered data advantage. The paper's most striking result — that no filtering outperforms success-only filtering by 2.4× (Table 7) — is interpretively attributed to "realistic error states and recovery patterns," but this mechanism is not experimentally verified. A strong follow-up would analyze the unfiltered training set at the trajectory level: identify trajectories containing specific failure-recovery patterns (e.g., command fails → model reads error → model tries alternative approach), measure their prevalence, and then conduct a controlled ablation where these failure-recovery subsequences are either retained or removed from otherwise clean trajectories. If the benefit truly comes from error recovery exposure, then training on clean trajectories augmented with synthetic failure-recovery segments should match or approach the unfiltered performance. Alternatively, if the benefit comes simply from having more data (unfiltered is 3.2× larger than success-only in Table 7), a data-volume-matched comparison (randomly subsampling unfiltered to match success-only count) would disambiguate quantity from composition effects. The paper currently confounds these two factors.

Testing whether the unfiltered advantage scales to larger models. The filtering ablation (Tables 6, 7) is performed only on Qwen3-8B. A natural follow-up would replicate the success-only vs. unfiltered comparison at the 14B and 32B scales to determine whether the unfiltered advantage is specific to smaller models that benefit more from diverse training signals, or whether it generalizes across scales. The scaling results in Figure 4 show that larger models benefit more from additional data, which could mean (a) the unfiltered advantage grows with scale because larger models extract more from the diverse signals, or (b) the unfiltered advantage shrinks because larger models are better at learning from clean data and are more sensitive to noisy examples. The direction of this interaction has practical implications: if the advantage shrinks, practitioners training large models should invest in better filtering and clean data; if it grows, they should prioritize data diversity over cleanliness.

Extending the skill taxonomy approach to other interactive agent domains. The paper's 9-domain, 6-skill-type taxonomy (Table 10) proved effective for terminal tasks — generating data that taught capabilities absent from pretraining (Table 4). A natural extension would apply the same structured generation approach to web navigation (domains: form interaction, authentication, dynamic content handling, state management; skills: DOM traversal, click targeting, wait strategies, session handling), GUI interaction (domains: file management, settings configuration, multi-window workflows; skills: widget identification, drag-and-drop, context menus), or code review (domains: security vulnerabilities, performance anti-patterns, style violations; skills: static analysis interpretation, diff comprehension, fix suggestion). For each domain, the key experiment would be: generate data using a taxonomy-guided pipeline vs. an unstructured "generate diverse tasks" baseline, and measure whether the taxonomy provides coverage benefits beyond what unstructured generation achieves. The paper's evidence for the taxonomy's specific contribution is currently confounded with the teacher model and generation prompts — a controlled comparison would isolate the taxonomy's value.

Training a lightweight difficulty estimator to amortize synthetic task quality assessment. The paper does not estimate or filter tasks by difficulty; all generated tasks enter the trajectory pipeline regardless of whether the teacher can solve them. This is computationally expensive when scaled further — generating and running trajectories for tasks that are too hard wastes teacher compute. A natural improvement would train a small classifier (fine-tuned from a 1-3B model) on the existing trajectory data to predict, from the task prompt alone, whether the teacher model is likely to succeed. This classifier could filter out too-hard tasks before trajectory generation, improving pipeline efficiency. The training signal exists: the 264K synthetic trajectories already contain task prompts paired with teacher success/failure labels. The experiment would measure: (a) classifier accuracy at predicting teacher success, (b) reduction in wasted trajectory generation (how many doomed trajectories are avoided), and (c) whether the resulting filtered dataset trains models as well as the unfiltered set (the filtering ablation in Table 7 suggests this might hurt — but filtering tasks before trajectory generation is different from filtering trajectories after generation, since the former saves compute and the latter changes data composition).

Investigating whether failed trajectories from weaker teachers are more or less valuable. The paper uses a single strong teacher (DeepSeek-V3.2, 38.2% on Terminal-Bench 2.0) for all trajectory generation. The finding that retaining failed trajectories helps (Table 7) raises a provocative question: would trajectories from a weaker teacher — one that fails more often and in more varied ways — provide even richer failure-recovery signals? Or would weaker-teacher failures be too chaotic and uninformative? A controlled experiment would generate trajectory data from teachers at multiple capability levels (e.g., DeepSeek-V3.2 at 38.2%, GPT-OSS 120B at 18.7%, Qwen3-32B at 3.4%), train student models on each teacher's full (unfiltered) trajectories, and measure the relationship between teacher capability and student performance. If weaker teachers produce better training data (through richer failure diversity), this would fundamentally challenge the "strongest available teacher" assumption in SFT data generation. If stronger teachers are better (as intuition suggests), quantifying the capability threshold below which teacher data becomes harmful would provide practical guidance.

Direct comparison of Terminal-Task-Gen against multi-agent generation frameworks at matched scale. The paper argues that its lightweight, single-teacher pipeline is more scalable than multi-agent approaches (Austin, 2025; Peng et al., 2025), but provides no head-to-head comparison. A strong follow-up would: generate the same number of terminal tasks (~100K) using both Terminal-Task-Gen and a multi-agent framework (e.g., the Austin 2025 pipeline), train Qwen3-8B models on the resulting trajectories (controlling for trajectory count, SFT hyperparameters, and evaluation protocol), and compare Terminal-Bench 2.0 performance. This would quantify whether the pre-built Docker image strategy and single-teacher design actually produce better training data, or whether the efficiency gains come at a quality cost that matters for downstream performance. The comparison would also measure generation throughput (tasks per GPU-hour), failure rate (fraction of generated tasks that cannot produce valid trajectories), and task diversity (distribution of skills exercised, measured by clustering task embeddings) — dimensions where multi-agent systems might have advantages despite higher cost.

Stress-test: evaluating on held-out terminal tasks not in Terminal-Bench 2.0's distribution. All evaluation is on 89 Terminal-Bench 2.0 tasks. Despite 14-gram decontamination, the risk of overfitting to the benchmark's task distribution remains — 490K training trajectories may cover functional equivalents of the 89 test tasks even without n-gram overlap. A robust stress-test would construct a held-out set of 20-30 new terminal tasks (different domains, different tools, different verification patterns) and evaluate Nemotron-Terminal against base Qwen3 and against the Terminal-Bench leaderboard models. If the fine-tuned models maintain their relative gains on this held-out set, it validates that the training teaches general terminal competence rather than benchmark-specific patterns. If performance regresses substantially, it would reveal that the synthetic task generation — which uses the same task design principles as Terminal-Bench — produces training data that overfits to the benchmark's task structure. This is a critical validity check that any deployment of these methods should perform.

Practical Applications and Downstream Use Cases

Cost-efficient fine-tuning for domain-specific terminal agents. Organizations that need terminal agents for specialized domains — ML experiment management, cloud infrastructure orchestration, database administration, security auditing — can adopt the Terminal-Task-Gen pipeline directly. The key practical insight is that the skill taxonomy approach enables targeted data generation: define a domain (e.g., "Kubernetes administration"), curate its primitive skills (pod management, configmap editing, log streaming, rollout monitoring), generate ~10-50K synthetic tasks using a strong teacher model, collect trajectories, and fine-tune. The paper's scaling results (Figure 4) suggest that even 1-5% of the full synthetic data (~2,600-13,000 tasks) already provides substantial gains over base models for Qwen3-8B. For a specialized domain with fewer skill dimensions than general terminal use, this lower data volume may suffice, making the approach practical for teams without access to large-scale compute. The pre-built Docker image strategy further reduces infrastructure burden: maintain 1-2 domain-specific images rather than per-task environments.

Augmenting code assistant models with terminal execution capability. Models like Qwen3-Coder-480B (23.9% on Terminal-Bench, Table 3) demonstrate strong code generation but weak terminal interaction — they can write code but struggle to run, test, debug, and deploy it in an environment. The paper's data pipeline provides a recipe for bridging this gap: take an existing code-capable model, generate terminal trajectories that exercise the code-to-execution transition (writing code + running it + debugging failures + iterating), and fine-tune. The adapter data from SWE benchmarks (SWE-Bench-Train, SWE-Smith, SWE-Fixer — Section 4.1.1) is particularly valuable here because it already pairs code modification with environment interaction. A practical deployment might: (1) fine-tune a code model on the SWE adapter subset (31,661 trajectories) to teach basic terminal-code interaction, (2) supplement with skill-based synthetic tasks targeting Debugging (33.3% improvement in Table 4) and Software Engineering (5.0% → 31.7%), and (3) deploy the resulting model as a code assistant that can both suggest fixes and verify them through execution. The paper's evidence that Nemotron-Terminal-32B (27.4%) outperforms Qwen3-Coder-480B (23.9%) despite being 15× smaller demonstrates that terminal execution capability is complementary to code generation capability and can be acquired through targeted fine-tuning rather than massive pretraining.

Self-improving agent pipelines through iterative data generation. The Terminal-Task-Gen pipeline can be applied iteratively: train a model on generated trajectories, use the trained model as a teacher to generate new trajectories (potentially on harder or different tasks), and repeat. The paper's finding that retaining failed trajectories helps (Table 7) is particularly relevant here — in an iterative setting, each generation's model will fail in different ways, producing increasingly diverse failure-recovery patterns that enrich the training distribution. The paper's negative result with ReST^EM-style revisions (Appendix K, not analyzed here) suggests caution with on-policy RL approaches, but the simpler iterative SFT approach — generate, train, use trained model as next teacher, repeat — avoids the distribution-shift issues that caused ReST^EM to degrade. The scaling curves in Figure 4 show that performance continues improving with data volume at the 100% mark, suggesting that additional data from iterative generation could push performance further. A practical pipeline: start with DeepSeek-V3.2 trajectories → train Nemotron-Terminal-32B → use Nemotron-Terminal-32B as teacher for a second generation of synthetic tasks (possibly targeting categories where it still scores low: Mathematics 0.0%, Scientific Computing 0.0%, Games 0.0%) → train on the combined dataset → repeat until category-level performance saturates.

When to Prefer This Method

The paper positions Terminal-Task-Gen as a practical, scalable data generation framework for terminal agent training, but does not explicitly articulate a decision rule for when to prefer it over named alternatives. The implicit tradeoffs — against multi-agent generation frameworks (Austin, 2025; Peng et al., 2025) on cost/scalability, against scaffolding-only approaches on generality, against larger-model-without-terminal-training on efficiency — can be extracted from the paper's evidence and design rationale:

Prefer Terminal-Task-Gen over multi-agent synthetic generation when generation throughput and computational cost are binding constraints. The pre-built Docker image strategy eliminates per-task environment validation, and the single-teacher design avoids multi-agent coordination overhead. The paper generates ~264K synthetic tasks without the "costly multi-turn repair" and "computational complexity that scales poorly" it attributes to multi-agent approaches (Section 2). However, the paper provides no direct throughput or cost comparison, so this preference is based on design analysis rather than experimental evidence.
Prefer Terminal-Task-Gen + SFT over scaffold-only approaches when model generality and reusability matter more than maximizing a single benchmark score. The Nemotron-Terminal models achieve their gains using the standard Terminus 2 scaffold without any scaffold-specific optimization (Section 3.2), meaning the capability improvements transfer to any scaffold. Scaffold-heavy approaches (Droid, Warp) may achieve higher absolute scores on Terminal-Bench but their optimizations are model-specific and require re-engineering for each new model. The paper's framing (Section 2) explicitly argues that "as base models improve, the marginal benefit of complex scaffolding will likely decrease," making model-level improvement the more scalable long-term investment.
Prefer Terminal-Task-Gen data over adapter-only data when terminal-specific skills (debugging, security, system administration, file operations) are the capability gap. Table 4 shows that adapter data alone cannot teach these skills — base models score 0.0% on Debugging, File Operations, and Model Training, and synthetic data is responsible for unlocking these categories. If the target deployment primarily requires math and code problem-solving through a terminal (which adapters cover well), adapter-only data may suffice; if it requires genuine terminal interaction skills, synthetic generation is necessary.
Prefer retaining all trajectories (no filtering) over success-only filtering when training interactive agents where the deployment environment produces dynamic error feedback. The 2.4× gap in Table 7 is specific to terminal interaction, where error recovery is a critical skill. For text-to-text tasks where the model produces a single answer and receives no dynamic feedback, success-only filtering likely remains preferable — but this paper provides no evidence for that setting. The principle that failed trajectories are valuable generalizes to any domain where the agent must react to environment signals during deployment rather than producing a single static output.