Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Text

ArXiv: 2601.10355

🎯 Pitch

You don't need predefined APIs to train tool-using agents—raw text already contains the procedural knowledge. By mining web corpora for implicit multi-step workflows, this work synthesizes training trajectories that boost a model's multi-turn tool-use score by 16.5%, even matching in-domain baselines on specialized benchmarks.

1. Executive Summary

This paper proposes a text-based paradigm for synthesizing multi-turn tool-use trajectories by extracting implicit procedural knowledge from unstructured text corpora, bypassing the dependency on predefined API sets. The core contribution is GEM, a four-stage data synthesis pipeline—relevance filtering, workflow and tool extraction, trajectory grounding, and complexity refinement—that transforms raw text segments into structured agentic training data, and a distilled Trajectory Synthesizer that internalizes this mapping into an efficient end-to-end generator. When fine-tuned on GEM-synthesized data, Qwen3-32B achieves a 16.5% improvement on the BFCL V3 Multi-Turn benchmark (overall accuracy reaching 44.88%) and matches or exceeds models trained on in-domain τ-bench data on the out-of-domain τ²-bench, with the 8B Trajectory Synthesizer matching the full pipeline's quality while significantly reducing inference cost. The paper establishes that text corpora serve as a scalable, authentic source of agentic training data, demonstrating that out-of-domain text-derived trajectories can rival in-domain synthetic data on specialized benchmarks—though only when the refinement and hallucination-filtering stages are applied, as ablations show a 12-point accuracy drop without refinement.

2. Context and Motivation

The Core Problem: We Don't Have Enough Realistic Multi-Turn Tool-Use Data

The fundamental challenge this paper tackles is a data scarcity problem that sits at the intersection of two critical trends in AI. First, Large Language Models are increasingly being deployed as autonomous agents—systems that don't just answer questions but actively interact with environments, make tool calls, track state across multiple turns, and recover from errors. Second, the primary bottleneck preventing these agents from becoming more capable is not architecture or scale, but the availability of high-quality training data that reflects the messy, realistic, multi-step interactions that agents encounter in practice.

The paper frames this problem with striking specificity: existing LLMs "still struggle in realistic multi-turn interactions, particularly when faced with ambiguous instructions, long-context dependencies, and unexpected errors" (Section 1). These aren't marginal issues—they represent the difference between a demo that works in controlled conditions and a system that functions in production. An agent that fails when a user provides an incomplete request, forgets a constraint mentioned five turns ago, or can't recover from a tool execution error is not practically useful, regardless of how well it performs on single-turn benchmarks.

The scarcity is structural. Multi-turn tool-use trajectories—sequences of alternating user queries, assistant reasoning (including tool calls), and environment responses—are rarely found in naturally occurring data. Unlike single-turn question-answering, where the internet provides billions of implicit training examples, or code generation, where repositories contain millions of function calls, realistic agent-environment dialogues with proper error handling, clarification, and constraint enforcement simply don't exist at scale in the wild. The paper emphasizes this directly: such trajectories "are rarely found in real-world scenarios" (Section 1).

Why This Problem Matters: The Gap Between Benchmarks and Reality

The paper's motivation goes deeper than simply "we need more data." The issue is that the kind of data matters enormously for what the model learns. Consider three scenarios that a competent agent must handle:

1. Ambiguous user requests: A user says "I want to cancel my order" without specifying which order or providing a reason. A well-trained agent should ask clarifying questions rather than hallucinating an order ID.

2. Constraint conflicts: A user asks for something that violates domain rules (e.g., font size 150 when the maximum is 96). The agent should recognize the violation, explain the constraint, and offer alternatives—not silently comply or fail.

3. Error recovery: A tool call fails (e.g., printer unavailable, API timeout). The agent should diagnose the issue and adapt its strategy, not abandon the task or loop indefinitely.

These interaction patterns don't emerge from training on single-turn function-calling data. They require explicitly demonstrated trajectories where the assistant reasons about constraints, asks clarifying questions, handles errors gracefully, and maintains coherent task state across many turns. The paper's experiments bear this out: models trained on simpler datasets like APIGEN-MT (which averages only 18.5 turns and 4.3 tool calls per trajectory) achieve substantially lower BFCL V3 scores than models trained on GEM data (averaging 46.1 turns and 16.3 tool calls, Section 4.5).

The practical stakes are significant. As the paper notes (Section 1), the goal of agentic training is "exposure to a sufficiently broad range of scenarios during training to enable agents to generalize effectively to unseen environments and scenarios." Without diverse, complex training trajectories, agents overfit to narrow patterns and fail catastrophically when deployed in novel situations—a problem that grows more acute as agents are integrated into customer service, healthcare, finance, and other high-stakes domains.

Where Existing Approaches Fall Short

The paper identifies a dominant paradigm that it calls "tool-centered simulation" (Section 1, Figure 1), and systematically diagnoses its limitations.

The prevailing approach: predefined API sets. Prior work on multi-turn tool-use data generation—encompassing ToolBench (Qin et al., 2023), APIGEN-MT (Prabhakar et al., 2025), TOUCAN (Xu et al., 2025), MAGNET (Yin et al., 2025), ToolACE-MT (Zeng et al., 2025), and MUA (Zhao et al., 2025)—operates under a shared assumption: you start with a predefined set of APIs, then synthesize user tasks and simulate interactions within that environment. The pipeline typically involves designing or collecting tool definitions, generating task blueprints that can be solved with those tools, and then simulating multi-turn dialogues where an assistant invokes the prescribed APIs.

This approach has produced valuable datasets and measurable progress. APIGEN-MT, for instance, generates structured task blueprints with ground-truth action sequences and simulates realistic human-agent dialogues grounded in executable APIs. TOUCAN crawls MCP servers to synthesize 1.5M tool-use examples from real-world environments. These are genuine contributions. But the paper argues they share a fundamental limitation that caps their utility for training general agents.

Limitation 1: Narrow coverage from predefined APIs. The paper states the problem directly: "gathering a sufficiently diverse and comprehensive tool set is inherently expensive and difficult. The resulting tool-use training data is often limited by the scope of the predefined APIs" (Section 1). This is not merely a cost complaint—it's a conceptual limitation. When an agent is trained exclusively on data synthesized from a fixed set of APIs, it learns patterns specific to those APIs' structures, constraints, and interaction dynamics. The diversity of the training data is bounded by the diversity of the APIs you can collect and document.

The corollary is that agents trained this way may struggle with unseen tools and domains at deployment time. The paper frames this as a generalization problem: "the ultimate goal of agentic training is the exposure to a sufficiently broad range of scenarios during training to enable agents to generalize effectively to unseen environments and scenarios" (Section 1). Predefined-API approaches inherit a sampling bias—you train on the tools you have, not on the tools your agent will encounter.

The paper's τ²-bench results provide empirical evidence for this claim. Models trained on in-domain τ-bench synthetic data (APIGEN-MT, SIMIA) are fine-tuned on trajectories generated within the exact environment they're tested on. Yet GEM-trained models—which never saw the τ²-bench APIs during data generation—achieve comparable or superior performance (e.g., Qwen3-32B-GEM achieves 55.48% Avg@4 on Retail vs. 49.56% for MUA, which uses in-domain data; Figure 5). This out-of-domain generalization is precisely what the predefined-API paradigm struggles to deliver.

Limitation 2: The scalability ceiling. The paper identifies a more subtle problem: even if you could collect an arbitrarily large set of APIs, the density of procedural knowledge in API documentation is fundamentally lower than in human-written text. An API specification tells you what a function does and what parameters it takes. A real-world text—a tutorial, a user manual, a forum post—captures how humans actually use tools in context: the edge cases they encounter, the workarounds they develop, the constraints they discover through experience. The paper's core insight is that this richer, experiential knowledge is latent in the text corpora that LLMs are already pretrained on, but it has never been systematically extracted and transformed into agent training data.

Consider the paper's running example about photo frame ordering (Appendix F). A raw text about custom framing contains not just API-like function descriptions ("calculate frame size given photo dimensions and mount width") but also implicit business rules ("maximum size is length + width ≤ 170cm"), conditional logic ("if using non-glare glass, maximum dimensions are different"), error patterns ("odd sizes must be ordered via email with remarks"), and clarification requirements ("indicate whether you need plain or non-glare glass"). A predefined-API approach might define a calculate_frame_size function, but it would struggle to automatically surface the constellation of real-world constraints, exceptions, and interaction patterns that make the task genuinely complex.

Limitation 3: Inauthenticity of simulation. While the paper is diplomatic about this point, there is an implicit critique of purely simulated trajectories. When a multi-agent system simulates a user and an assistant interacting through a predefined toolset, the resulting dialogues can lack the grounded messiness of real human problem-solving. The paper's approach, by contrast, derives its workflows from text that documents actual human procedures—people explaining how they actually solved problems, complete with the constraints and edge cases they encountered. This grounding in authentic human experience is what the paper means when it describes text corpora as providing "authentic human problem-solving behaviors" (Section 2). The preliminary analysis (Section 3.1) confirms this: 14% of randomly sampled UltraFineWeb segments contain explicit multi-step operational procedures spanning domains from customer support to education to data analysis, representing a "substantial reservoir of procedural knowledge" that simulation-based approaches cannot replicate from scratch.

How This Paper Positions Itself

The paradigm shift: text-to-trajectory. The paper's central conceptual move is to reframe the problem entirely. Rather than asking "given these APIs, what tasks can we synthesize?", it asks "given this massive corpus of human-written text, what procedural knowledge can we extract and operationalize?" The paper calls this the "text-based extraction paradigm" (Section 1) and positions it as a fundamental alternative to tool-centered simulation, not an incremental improvement.

This shift has several important properties that the paper leverages:

• Scale: Text corpora are essentially unlimited. UltraFineWeb alone provides billions of tokens. If even 14% contain usable procedural content, the potential training data pool is enormous and growing with web-scale pretraining corpora.

• Diversity: The preliminary analysis reveals that procedural content spans dozens of domains (Figure 2, Appendix E)—from customer support and developer tools to education, healthcare, and gaming. This diversity is an emergent property of the corpus, not something that must be engineered through API collection.

• Authenticity: The workflows extracted from text are grounded in real human problem-solving, not synthetic task generation. This authenticity manifests in the natural inclusion of edge cases, constraints, and failure modes that might be overlooked in top-down task design.

Positioning against prior work. The paper explicitly distinguishes itself from simulation-based approaches (APIGEN-MT, TOUCAN, ToolACE-MT, MAGNET) while acknowledging their contributions. The key differentiator is the data source: "Unlike prior works that rely on pre-defined tools, our work introduces a novel paradigm that directly extracts multi-turn trajectories from text, thereby unlocking an authentic and scalable source of tool-use agentic data" (Section 5, Related Work). This isn't positioned as "our synthesis pipeline is better" but rather as "our data source is fundamentally different and complementary."

Within this paradigm, the paper's GEM pipeline plays a specific role: it's the proof-of-concept implementation that demonstrates the viability of text-to-trajectory synthesis. The four stages (filtering, extraction, generation, refinement) are not presented as the optimal pipeline but as a working instantiation that validates the paradigm's core claim—that text corpora can be transformed into high-quality agentic training data that yields measurable benchmark improvements. The Trajectory Synthesizer further demonstrates that this mapping can be distilled into an efficient model, making the paradigm practical at scale.

The implicit research agenda. While the paper is primarily an empirical contribution, it opens a significant research direction. If text corpora are indeed a viable data source for agent training, then:

• How do we improve extraction quality? The ablation results (Figure 6) show a 12-point accuracy drop without refinement, suggesting substantial room for better extraction methods.
• Can we combine text-derived trajectories with simulation-based data for complementary benefits? The paper's τ²-bench results show text-derived data competing with in-domain simulation data—combining both might yield further gains.
• Does text-derived training generalize beyond the specific benchmarks tested? The paper demonstrates generalization from arbitrary web text to specialized domains like airline and retail—how far does this generalize?

The paper positions these as open questions, not solved problems, while establishing the "text-to-trajectory" paradigm as a legitimate and productive alternative to the dominant simulation-based approach.

3. Technical Approach

3.1 Reader Orientation

The GEM system is an automated pipeline that reads unstructured text documents, identifies procedural knowledge embedded within them, and transforms that knowledge into structured multi-turn tool-use dialogues suitable for training AI agents. The core problem it solves is the scarcity of realistic agent training data: GEM bypasses the traditional requirement of manually defining APIs and simulating interactions by instead mining the vast reservoir of human problem-solving experience already captured in web text, converting tutorials, guides, and procedural descriptions into executable training trajectories complete with tool definitions, system constraints, and rich interaction patterns.

3.2 Big-Picture Architecture (Diagram in Words)

The GEM pipeline consists of six major components arranged in two phases:

Phase 1: Knowledge Extraction

1. Text Filtering Module — scans raw text corpora and classifies each segment as containing multi-step operational procedures or not, using a prompted LLM classifier. This acts as a coarse gate, retaining only the ~14% of web text that describes actionable workflows.
2. Workflow & Tool Extraction Module — takes filtered text segments and produces two structured artifacts: (a) abstract workflow descriptions enumerating sequential steps, dependencies, and conditional logic, and (b) a set of functional API tool definitions in OpenAI JSON-schema format, synthesized to support the extracted workflows.

Phase 2: Trajectory Generation and Quality Assurance 3. Trajectory Generation Module — combines the original text, extracted workflows, and tool definitions and produces a complete multi-turn dialogue (system prompt, user queries, assistant responses with tool calls, and simulated tool responses) in a single forward pass using a strong teacher model. 4. Refinement Module — takes the initial trajectory and rewrites it to increase complexity across multiple dimensions: expands the toolset, adds environmental constraints, increases user request ambiguity, and ensures the inclusion of non-trivial interaction patterns (clarification, error recovery, rule enforcement). 5. Validation Module — applies two-stage filtering: (a) a rule-based checker that verifies structural correctness (valid JSON schemas, correct function names, proper turn ordering), and (b) an LLM-based hallucination detector that verifies every tool call parameter value is grounded in the dialogue context rather than fabricated. 6. Trajectory Synthesizer — a distilled model trained via supervised fine-tuning on (text, full trajectory) pairs produced by the full pipeline, which internalizes the entire text-to-trajectory mapping into a single end-to-end forward pass, dramatically reducing inference cost while maintaining quality.

Information flows sequentially: raw text → filtered segments → abstract workflows + tools → initial trajectory → refined trajectory → validated trajectory. The Trajectory Synthesizer short-circuits this by learning the direct mapping from text to validated trajectory from the pipeline's outputs.

3.3 Roadmap for the Deep Dive

• First, the preliminary corpus analysis, because it establishes the empirical foundation for the entire paradigm—quantifying that procedural knowledge exists in text, how much of it there is, and what domains it spans—before any pipeline engineering begins.
• Second, Stage 1 (Text Filtering), the gating mechanism that determines which text segments enter the pipeline and which are discarded, since all downstream stages depend on its precision.
• Third, Stage 2 (Workflow & Tool Extraction), the core extraction step that converts unstructured procedural descriptions into structured workflows and functional API definitions, since these artifacts are the bridge between raw text and executable trajectories.
• Fourth, Stage 3 (Trajectory Generation), which transforms the structured intermediate representations into concrete multi-turn dialogues with system prompts, user queries, assistant reasoning, tool calls, and simulated tool responses.
• Fifth, Stage 4 (Refinement and Validation), because the paper's ablation experiments show these stages are critical: refinement alone contributes over 12 percentage points to final accuracy, and hallucination filtering provides further gains—understanding their mechanisms is essential to understanding the pipeline's performance.
• Sixth, the Trajectory Synthesizer, which learns to replicate the entire pipeline in a single model forward pass, representing both a practical contribution (cost reduction) and a scientific one (demonstrating that the text-to-trajectory mapping is learnable).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data synthesis pipeline paper whose core idea is that unstructured web text contains implicit, multi-step procedural knowledge—documented human problem-solving experiences—that can be systematically extracted, operationalized as tool-use trajectories, and used to train general-purpose autonomous agents, bypassing the coverage limitations of predefined-API approaches. The pipeline is not theoretically complex; its innovation lies in the novel data source (text corpora rather than API specifications), the careful four-stage extraction-refinement-validation architecture that progressively increases trajectory quality, and the empirical demonstration that text-derived training data enables out-of-domain generalization to specialized benchmarks.

Preliminary Corpus Analysis: Establishing Feasibility

Before committing engineering effort to the full pipeline, the paper conducts a quantitative survey to determine whether unstructured text actually contains the necessary ingredients for tool-use trajectory synthesis. This analysis is conducted on UltraFineWeb (Wang et al., 2025), a high-quality web-scale corpus, using approximately 250,000 randomly sampled raw text segments.

The annotation procedure. Each segment is processed through a sequential labeling pipeline. First, a classifier (specifically Qwen3-8B, prompted with the annotation template provided in Appendix A.1) determines whether the text "contains multi-step operations involving the use of an APP, website, computer, or other machine (such as robot, elevator, etc.)." This is a binary gate: segments without procedural content are discarded; segments with procedural content proceed to metadata annotation.

For the procedural segments, the same model annotates rich metadata including:

• Platform category: one of {operator, computer, phone, machine, other}
• Domain category: one of 25 predefined domains including arts_and_entertainment, business_and_industrial, computers_and_electronics, finance, health, shopping, sports, travel_and_transportation, etc.
• Task category: one of 34 fine-grained categories including databases, multimedia_processing, cloud_platforms, search, file_systems, ecommerce_and_retail, research_and_data, education_elearning, robot_control, website_control, etc.

The prompt templates for this annotation are fully specified in Appendix A.1, using a structured output format with XML-like tags (<multi_step>, <summary>, <domain>, <platform>, <task>).

Key finding 1: Prevalence of procedural content. The paper reports that approximately 14% of the sampled segments contain explicit multi-step workflows. Given UltraFineWeb's massive scale (the full corpus contains billions of tokens), this 14% represents an enormous absolute volume of procedural knowledge—"a substantial reservoir" of training data that has never before been systematically exploited for agent training.

Key finding 2: Domain and task diversity. The identified procedural segments span a remarkably wide spectrum (Figure 2 and Appendix E). Customer support leads at 22.4%, followed by research & data (15.5%), education & e-learning (12.1%), developer tools (7.6%), e-commerce & retail (6.3%), and dozens of other categories with representation from 1-5% each. The domain distribution (Appendix E, Figure 9) further confirms breadth: computers & electronics (578,494 segments), science (355,244), shopping (348,893), business & industrial (283,763), health (198,491), and many more.

This diversity is not engineered—it emerges naturally from the corpus. The paper emphasizes that this confirms "textual sources can provide the necessary variety in tasks, tools, and environments required for robust agent training" (Section 3.1). No API collection effort could match this breadth because it would require manually defining tools for domains as disparate as educational platforms, cloud infrastructure, smart home systems, and financial trading platforms.

Key finding 3: Inherent structural alignment with agent trajectories. Through qualitative examination of text cases, the paper identifies that unstructured documents naturally contain three core components essential for constructing agentic trajectories (Figure 3):

1. User queries: Goals or problems stated in the text (e.g., "Creating a Music Visualizer in Adobe After Effects" serves as the high-level user intent).

2. Environmental tools: Descriptions of software functions, API capabilities, or machine operations embedded within explanatory contexts (e.g., "Click 'Composition' > 'New Composition'. Set the width to 1920 pixels" implicitly defines a create_composition operation with width and height parameters).

3. Multi-step workflows: Step-by-step procedures or operational narratives (e.g., "1. Prepare your audio file. 2. Create a new composition. 3. Import the Audio File...") that map directly to sequential tool-call sequences.

The paper uses a concrete example: a tutorial about creating music visualizers in Adobe After Effects (Figure 3). This single text segment contains the user's goal (create a music visualizer), explicit tool interactions (create composition, import audio, configure settings), sequential ordering constraints (audio must be imported before it can be added to the timeline), and implicit tool parameters (frame width 1920, frame height 1080, frame rate 30fps).

What this analysis establishes. The preliminary study is not merely descriptive—it serves as a proof of concept for the entire paradigm. It answers the question "is this even possible?" with quantitative evidence: yes, procedural knowledge exists in web text at meaningful scale (~14% of a random sample), spans sufficient diversity to train general-purpose agents, and is structurally amenable to conversion into tool-use trajectories. This finding justifies the substantial engineering effort of building the GEM pipeline and provides the empirical foundation for the paper's claim that text corpora represent an "untapped, scalable, and authentic data source for multi-turn tool-use tasks" (Section 1).

Stage 1: Text Filtering — The Gating Mechanism

The first operational stage of the GEM pipeline filters incoming text segments to retain only those describing multi-step operations. Without this filter, the pipeline would waste computation on text segments that lack procedural content (narrative fiction, opinion pieces, descriptive encyclopedia entries, etc.) and risk generating nonsensical or hallucinated trajectories from inappropriate source material.

Filtering mechanism. The filtering procedure uses the same annotation prompt and model configuration as the preliminary analysis (Section 3.1). Specifically, each raw text segment c ∈ C from the input corpus is submitted to Qwen3-8B with a prompt that asks: "Determine whether the following text contains multi-step operations involving the use of an APP, website, computer, or other machine." The model outputs a binary classification: a <multi_step>True</multi_step> or <multi_step>False</multi_step> tag.

Design rationale. The paper reuses the annotation pipeline from the preliminary analysis rather than designing a separate filtering mechanism. This is a pragmatic choice that ensures consistency: the same model, same prompt, and same classification criteria are used both for the corpus-wide feasibility study and for the operational pipeline. It also means the 14% yield rate observed in the preliminary analysis directly informs expectations about the pipeline's throughput—for every 1,000 raw segments, approximately 140 will pass the filter and enter the extraction stage.

What passes through. The filter is deliberately broad in scope. The key criterion is not that the text describes tool usage that maps neatly to API calls, but rather that it contains "multi-step operations" involving some form of machine interaction. This includes:

• Software tutorials (e.g., "How to create a pivot table in Excel")
• Hardware configuration guides (e.g., "Setting up a home router")
• Procedural documentation (e.g., "Hospital reimbursement claim procedure")
• Online service workflows (e.g., "How to return an item on Amazon")
• Programming guides (e.g., "Building a REST API with Flask")

The filter's recall is more important than its precision at this stage. False positives (texts that are classified as procedural but don't actually contain useful workflows) will be handled downstream by the generation and validation stages. False negatives (procedural texts that are incorrectly discarded) represent permanent data loss and should be minimized. The paper does not report precision/recall metrics for the filter, which would be a useful diagnostic—but given the massive scale of input corpora, a conservative filter that over-admits is reasonable.

Implicit assumption. The filtering stage assumes that the classifier's notion of "multi-step operations involving machines" correlates well with "text that can produce useful tool-use trajectories." The paper validates this assumption indirectly through downstream results—if the filter were admitting inappropriate content, the generation and validation stages would produce low-quality trajectories that fail to improve model performance on benchmarks. The strong empirical results on BFCL V3 and τ²-bench (Section 4) suggest the filter is working adequately, though a direct analysis of filtered-out vs. retained segments would be informative.

Stage 2: Workflow & Tool Extraction — From Unstructured Text to Structured Representations

This is the core knowledge extraction stage where unstructured prose is transformed into two structured artifacts that will drive trajectory generation: (1) abstract workflow descriptions and (2) formal tool definitions. The transformation is performed by a powerful LLM (the paper uses GLM-4.6 for this stage, though the specific model choice is not justified in detail) prompted with detailed extraction instructions (Appendix A.2).

Input specification. For each text segment that passes the filtering stage, the model receives the complete text as its "Workflow Description" input. No additional preprocessing or chunking is performed. The text may describe a single workflow or multiple distinct workflows; the extraction prompt instructs the model to handle both cases.

Workflow extraction: capturing procedural structure. The model is instructed to identify all intermediate steps within each workflow and enumerate them sequentially. This extraction is not a simple copy-paste of the text's step numbering—the model must abstract away from narrative prose and extract the logical structure of the procedure. For a text about hospital reimbursement claims, the extracted steps might be:

Step 1: Verify insurance coverage and eligibility. Step 2: Collect required documentation (medical records, itemized bills). Step 3: Complete claim form with patient information and procedure codes. Step 4: Submit claim to insurance provider. Step 5: Track claim status and respond to any requests for additional information.

The prompt explicitly encourages the model to recognize workflow complexity beyond linear sequences. Three types of structural complexity are targeted:

• Sequential dependencies: "X must happen before Y"—capturing that certain operations are prerequisites for others (e.g., authentication must precede any data access).

• Uniqueness/limits: "Only one Admin allowed", "Name must be unique"—constraints that restrict the state space and create realistic error conditions.

• Conditionals: "If user is X, they cannot do Y"—branching logic that introduces context-dependent behavior.

The model outputs workflows in a structured XML format with tags for <description>, <steps>, and <execution_graph>. The execution graph represents the workflow as a directed acyclic graph of API calls, e.g., (authenticate_user)->(get_user_profile, search_products)->(place_order), capturing which calls can be made in parallel and which must be sequential.

Tool definition: operationalizing procedural knowledge as APIs. Concurrently with workflow extraction, the model synthesizes a set of functional API tools in standard OpenAI JSON-schema format. Each tool is designed to support one or more steps in the extracted workflows. The design instructions are detailed and reflect several non-obvious engineering choices:

1. Single-function principle: "Each tool should implement a single, coherent capability. It should not bundle multiple unrelated or multi-stage workflows into one tool." This encourages decomposition—e.g., creating separate plan_trip and book_trip tools rather than a monolithic plan_and_book_trip.

2. Descriptive naming: "Each tool's name should be short and readable, semantically clear and general, reusable (e.g., 'flight_search' rather than 'flight_detailed_search_for_tom_2025')." The emphasis on reusability matters for generalization—an agent trained on flight_search can adapt to any flight search scenario, while one trained on overly specific tool names overfits to particular task instances.

3. Realistic parameter design: "The required parameters of a tool need to be carefully considered and designed, mirroring the logic of the real world. For example, viewing system data typically requires authorization authentication, and providing user ID, product ID, etc." This instruction pushes the model to include authentication parameters, identifiers, and other realistic constraints that create authentic multi-step interaction patterns (authentication → query → action).

4. Read-write pairing: "It mimics a database structure and provides read and write tools." The model is encouraged to generate pairs of tools—e.g., get_user_profile (read) and update_user_profile (write)—that require agents to track state and understand that read operations are safe while write operations may have side effects.

5. Self-explanatory parameters: "Parameter names should be self-explanatory rather than cryptic (e.g., use 'check_in_date' with type 'string' and a short description, rather than a vague parameter named 'd1')." This ensures that the tool definitions themselves provide sufficient context for the trajectory generation stage to produce correct tool calls.

Output format. The model outputs each identified workflow as an XML block containing:

<workflow>
  <description>short task description</description>
  <steps>Step1: ...\nStep2: ...</steps>
  <execution_graph>(api_name1)->(api_name2, api_name3)->..</execution_graph>
  <actions>[{"name":"api_name", "arguments": {...}}, ...]</actions>
  <tools>[{OpenAI schema JSON}, ...]</tools>
</workflow>

The <actions> field contains a concrete example of the tool-call sequence with realistic parameter values, providing a reference for the trajectory generation stage. The <tools> field contains the complete JSON-schema definitions that will be passed forward.

Why this stage is necessary: bridging unstructured and structured. The extraction stage performs a critical translation function. Raw text about the hospital reimbursement process contains implicit knowledge about what information is needed, in what order, with what constraints—but in a form that a language model cannot directly use for tool-call training. The extracted workflows make the procedural logic explicit (step dependencies, conditionals, uniqueness constraints). The synthesized tools create a concrete action space (named functions with typed parameters). Together, they convert "this is how humans solve this problem" into "this is how an agent should interact with tools to solve this problem."

Design choice: separate extraction and generation. The paper could have designed a single-stage pipeline where the model directly converts text into trajectories without the intermediate workflow-and-tool representation. The two-stage design (extraction → generation) has several advantages that the paper implies rather than explicitly states:

• Modularity: Each stage can use different models optimized for different tasks. The extraction stage benefits from strong reasoning capabilities (abstracting workflows from prose), while the generation stage benefits from strong dialogue and formatting capabilities.

• Debuggability: Intermediate artifacts can be inspected and validated before committing to full trajectory generation. If the extracted tools are nonsensical, the trajectory will be poor regardless of generation quality.

• Reusability: A single text might support multiple distinct trajectories (different user personas, different edge cases). The abstract workflow and tool definitions can be reused across multiple generation runs, amortizing extraction cost.

Stage 3: Trajectory Generation — Grounding Workflows in Conversational Interactions

Given (1) the original text segment, (2) the extracted abstract workflows, and (3) the synthesized tool definitions, the trajectory generation stage produces a complete multi-turn dialogue that demonstrates an AI assistant helping a user complete tasks while navigating the domain-specific rules and constraints implicit in the source material.

Generation model. The paper employs GLM-4.6 as the teacher model for this stage, citing it as a "strong teacher model." The choice is pragmatic—any sufficiently capable LLM could serve this role—but the paper does not ablate different teacher models to quantify their impact on downstream trajectory quality, which would be a useful robustness analysis.

Single-pass generation. Unlike simulation-based approaches that iteratively step through a multi-agent process (user simulator generates a query, assistant responds, environment provides tool output, repeat), GEM generates the entire trajectory in a single forward pass. The prompt (Appendix A.3) is comprehensive—it specifies the structure of every component (system prompt, user queries, assistant responses, tool responses), enumerates required interaction patterns, specifies formatting constraints, and provides detailed examples. The model receives all tools and the workflow description as input and produces the complete trajectory as output.

This design choice has important efficiency implications. Multi-agent simulation is computationally expensive because it requires multiple LLM calls per trajectory turn. GEM's single-pass approach generates the full trajectory (averaging 46.1 messages and 16.3 tool calls) in one inference, making it dramatically faster and cheaper. The quality tradeoff—whether single-pass generation produces less realistic or less coherent trajectories than iterative simulation—is not directly evaluated, but the strong benchmark results suggest the teacher model is capable enough to maintain consistency across the full trajectory in a single generation.

Trajectory components. Each generated trajectory T comprises four distinct element types:

System prompt s: A domain-specific preamble extracted from the source text that establishes the assistant's persona and enumerates the rules, constraints, and operational boundaries it must respect. The generation prompt instructs: "Extract and explicitly state ALL important domain-specific rules and constraints from the source text document." For the photo frame ordering example (Appendix F), the system prompt establishes authentication requirements, frame specification constraints (plain glass max 100×150cm, non-glare max 60×70cm), mount options, and a seven-step order process. This prompt becomes part of the training data, teaching the fine-tuned model to attend to system-level constraints—a capability that directly transfers to benchmarks like τ²-bench where domain rules are a primary evaluation criterion.

User queries (u₁, ..., uₙ): A series of progressively complex natural-language requests designed to test the assistant's rule enforcement and constraint handling. The generation prompt specifies several properties that user queries should exhibit:

• Naturalness: "Requests should reflect real-world use cases" and "may include colloquial language, implied context, vague references to prior steps, or real-world motivations." The goal is to avoid the stilted, instructional language that characterizes synthetic data and instead capture how actual users communicate.

• Ambiguity: "User requests are often incomplete, requiring the assistant to analyze or clarify them." Examples include a user asking to cancel an order without specifying the reason (requiring clarification) or requesting a desktop recommendation while mentioning they "often go out" (requiring the assistant to infer portability as an implicit preference).

• Complexity: "At least in one turn, the user's request is very complex and requires assistant to handle it carefully." The examples illustrate multi-constraint requests: "I need to make several changes to my order... Can I change the E-Reader to a different size, swap the Garden Hose color, and also update my shipping address?" This requires the assistant to check order status, verify each item modification independently, handle the address change separately, and remind about one-time modification limits.

• Consistency: "User's intention, persona, and their behavior should be consistent across the dialogue." The user shouldn't contradict earlier statements or switch personas mid-conversation.

Assistant responses (a₁, ..., aₙ): These serve as the primary training signal—they demonstrate correct tool-use behavior in context. The generation prompt specifies an extensive set of requirements for assistant responses:

• Precondition checks: "Before executing tasks, the Assistant should validate any necessary preconditions (e.g., authenticating identity, verifying the status of an order)." The assistant should not assume preconditions are met.

• Rule enforcement: "The Assistant must follow domain-specific rules at all times" and "ensure the assistant's tool call and response genuinely addresses those requirements."

• Reasoning and planning: The assistant should reason through problems and plan appropriate steps, including determining parameter values from context when the user doesn't explicitly provide them.

• No hallucination: Specifically, tool call arguments "must only use argument values that are explicitly provided or implied by the user. It must not fabricate IDs, names, or other parameters; if any required value is missing or unclear, the Assistant should ask the user to supply it before calling the tool."

• Consequence awareness: "Before executing any write operation that modifies the environment, the Assistant must actively think and assess its impact. For changes that are irreversible, the assistant should obtain explicit user confirmation before proceeding."

• Capability limitation disclosure: If a request exceeds the assistant's capabilities, it "must communicate this limitation clearly" while offering alternatives: "I cannot do A, but I can do B. Should I proceed with B?"

Tool responses (o₁, ..., oₙ): Simulated environment feedback that completes each tool-call turn. The generation prompt specifies:

• Success responses must return complete, structured information (e.g., JSON with order IDs, statuses, product details).
• Error responses should return only the error message—"no additional information should be provided that directly aids the Assistant in recovering from the error." This is a deliberate choice that forces the assistant to diagnose failure causes and adapt strategies independently, rather than being spoon-fed recovery instructions.

Turn ordering constraints. The generation prompt explicitly enforces correct dialogue structure:

• A user message or tool response must be followed by an assistant message.
• If the assistant message includes a tool call, it can be followed by a tool response.
• Otherwise, it should be followed by a user message.
• A tool response must never be followed directly by a user message (since the assistant needs to process the tool output first).

These constraints prevent malformed trajectories that would confuse the fine-tuning process by teaching incorrect turn-taking patterns.

Interaction pattern requirements. To ensure the trajectories capture the diverse practical challenges of real-world multi-turn tool use, the prompt requires the inclusion of multiple interaction patterns, with at least three distinct patterns appearing and each used at most twice per trajectory. The specified patterns are:

• Pattern 1: Domain Rules & User Need conflicts — the user requests something that violates domain constraints, forcing the assistant to recognize the conflict, explain the limitation, and offer compliant alternatives.

• Pattern 2: Error Recovery — a tool call fails (e.g., invalid parameters, unavailable resource), and the assistant must diagnose the error and adapt its approach rather than abandoning the task.

• Pattern 3: Clarification and Disambiguation — the user provides an ambiguous or incomplete request, and the assistant recognizes the missing information and asks targeted clarifying questions.

• Pattern 4: Assistant's Multi-hop Reasoning — the assistant must chain information across multiple tool calls, using outputs from earlier calls to determine parameters for later calls or to answer a question that requires synthesizing multiple data sources.

• Pattern 5: Assistant's Awareness of Domain Rules — the assistant proactively applies domain knowledge without being prompted, e.g., checking constraints before executing a request rather than waiting for an error.

The pattern diversity requirement ensures that each trajectory teaches multiple distinct agentic capabilities, increasing the information density of the training data and preventing the fine-tuned model from learning only narrow patterns.

Tool call format. All tool calls use a specific XML-like format:

<func>{"name": "exact_tool_name", "arguments": {"arg": "value"}}</func>

The tool name must exactly match one of the tools in the synthesized toolset, and the arguments must match the schema definition. This strict formatting is essential for downstream parsing and validation—the rule-based checker in Stage 4 depends on being able to extract and verify function names and argument types programmatically.

Design choice: why single-pass over iterative simulation. The paper explicitly notes that it "synthesizes the full trajectory directly in a single pass instead of simulating turn-by-turn conversations via a multi-agent system to ensure efficiency." This is an important engineering tradeoff. Multi-agent simulation (where a user model and an assistant model interact turn-by-turn, with an environment simulator providing tool feedback) can potentially produce more realistic dynamics—the assistant can genuinely respond to unexpected user behavior or tool errors in ways that a single-pass generation might script rather than discover. However, single-pass generation is dramatically cheaper and faster, making it feasible to generate the 10K trajectories used in the paper's experiments. The paper does not directly compare single-pass against multi-agent quality, leaving open the question of whether simulation-based generation would produce trajectories that lead to even better fine-tuning results despite higher generation cost.

Stage 4: Refinement and Validation — Increasing Complexity and Ensuring Quality

Motivation for refinement. The paper makes a critical empirical observation: "although the initial multi-turn dialogue trajectories T are complete, they often lack sufficient complexity and tend to be relatively straightforward." This isn't a failure of the generation stage—the initial trajectories are structurally correct and demonstrate basic tool-use patterns—but they underutilize the richness available in the source text and the tool definitions. The refinement stage is designed to unlock this latent complexity.

The paper quantifies the impact of refinement in Section 4.4 (Figure 6): removing the refinement stage drops overall BFCL V3 accuracy from 30.25% to 26.00% for the 8B model and from 44.88% to 32.50% for the 32B model—a gap of over 12 percentage points for the larger model. Appendix D (Table 5) further breaks down the structural impact: without refinement, trajectories average 30.05 messages, 5.01 tools, and 7.83 tool calls; with refinement, these increase to 46.1, 8.6, and 16.3 respectively. This dramatic difference explains the large performance gap: refined trajectories are substantially longer, use more tools, and involve more complex tool-call chains, providing richer training signals.

Refinement mechanism. Given an initial trajectory T and its associated toolset, the refinement stage uses the same teacher model (GLM-4.6) to produce a rewritten trajectory T'. The refinement prompt (Appendix A.4) is extensive—substantially longer than the generation prompt—and provides detailed guidelines organized into five categories:

1. System prompt complexity. The refiner is instructed to "refine and upgrade the constraints of the system prompts to make them more structured, systematic, and consistent with real-world logic." This often involves defining an explicit database schema within the system prompt (tables, fields, relationships) that the tools will operate on, creating a more principled and internally consistent constraint environment.

2. User request complexity and naturalness. Beyond the complexity requirements of the initial generation, the refinement prompt specifies several additional dimensions:

• User diversity: "Create a user profile and maintain the user's personality and characteristics throughout the conversation history." The refiner establishes a consistent persona (e.g., budget-conscious, technically unsophisticated, impatient) and maintains it across all turns.

• Deep reasoning requirements: "The user's request must necessitate careful analysis and multi-step reasoning to identify the correct tool(s) and determine appropriate parameter values."

• Tool dependency chaining: "The request should force the assistant to understand dependencies between tools and use outputs from previous steps to decide which tool to invoke next."

• Multi-constraint turns: At least one turn must contain "multiple constraints, including explicit constraints, implicit requirements that require the assistant to infer."

• Cross-turn synthesis: At least one turn must ask a question "that can only be answered by reasoning across outputs from multiple tool calls in the long context."

• Challenging pitfalls: "MUST INCLUDE AT LEAST 1-2 PITFALLs for one trajectory." These are "traps to test the assistant's ability to correctly make robust tool calls based on rules, constraints or user preferences." An example might be a user asking for an operation that seems valid but violates a subtle constraint only mentioned earlier in the system prompt—the assistant must identify the pitfall and navigate it correctly.

3. Assistant intelligibility. The refinement prompt enumerates three dimensions of assistant behavior to strengthen:

• Communication skills: Intent understanding, proactive clarification, capability limitation awareness, result explanation and summarization, and proactive assistance (anticipating needs).

• Robust tool-calling capability: Tool selection, sequential tool usage, parameter handling (including complex nested structures), result analysis, state tracking, constraint analysis, error handling, and context management.

• Reasoning and execution ability: Planning and task decomposition, prerequisite management, and verification/validation before critical operations.

4. Realistic and complex environment. The refinement prompt addresses the tool response side:

• Tool responses should include structured inputs (lists, dicts, nested objects) with meaningful constraints, increasing the difficulty of correct parameter construction.

• Success responses must return "complete data structure in a well-structured format (e.g., JSON)," not partial or placeholder data.

• Error responses should simulate "non-simple errors that might occur in the real world" while avoiding telling the assistant how to recover.

• Unique identifiers (user IDs, product IDs, order IDs) should be used throughout to mimic real database logic.

5. Trajectory diversity. The refinement prompt explicitly warns against repetitive tool usage: "Reduce the frequency of repeatedly using certain tools to solve problems, avoiding the reduction of trajectory diversity, and retain only the most valuable trajectories for learning." This prevents the refiner from defaulting to simple patterns (e.g., query-query-query instead of query-write-query-write) when increasing trajectory length.

The refinement process is a full rewrite, not an edit. The refiner does not simply insert additional turns or modify parameters—it receives the initial trajectory as input and produces a completely rewritten version that preserves the core task while upgrading every component (system prompt, user requests, tool responses, interaction patterns). This is evident from the transformation described in Table 5: going from 7.83 to 16.3 tool calls and from 5.01 to 8.6 tools requires substantially restructuring the trajectory, not just tweaking it.

Design choice: why refinement over better initial generation. An implicit design question is: if the initial generation produces trajectories that are "relatively straightforward," why not improve the generation prompt to produce complex trajectories directly? The paper's two-stage approach (generate then refine) has several advantages:

• Separation of concerns: The generation stage focuses on getting the basic structure right—correct tool definitions, valid turn ordering, coherent task progression. The refinement stage focuses on enrichment—adding complexity, diversity, and challenge. Each stage can be optimized independently.

• Incremental improvement visibility: By keeping the initial and refined trajectories, the paper can quantify exactly how much refinement contributes (Table 5, Figure 6), providing scientific insight into what makes training data effective.

• Reusability of initial trajectories: Even without refinement, the initial trajectories provide useful training signals (the ablation shows they still improve over the base model). Refinement becomes an optional quality upgrade rather than a requirement for the pipeline to function.

Validation: two-stage quality assurance. Before a refined trajectory enters the final training dataset, it must pass through two validation gates:

Rule-based structural check. This is a deterministic, programmatic verification that ensures:

• All tools are correctly defined according to OpenAI format (valid JSON schemas with required fields).
• Each tool call in the trajectory corresponds to a valid function within the designated toolset (no hallucinated function names).
• Argument names and types match their definitions in the schema.
• The conversation format meets all requirements: tool responses follow tool calls, role tags are properly opened and closed, and turn ordering constraints are satisfied (no <tool> directly followed by <user>).

This check is fast and catches the most common formatting errors that could corrupt fine-tuning or cause evaluation failures.

LLM-based hallucination detection. This is the more sophisticated validation layer. The paper employs Qwen3-32B as a judge model, prompted with a hallucination detection rubric (Appendix A.5) that evaluates three categories:

• R1: Tool-call hallucination — "Check whether any tool call uses argument values that are not provided or reasonably derivable from the dialogue context." The judge examines every tool call's parameters against the preceding conversation to verify that IDs, names, dates, quantities, and other values either appear explicitly in user queries or tool responses, or can be legitimately inferred from context.

• R2: Capability hallucination — Two sub-types are checked. R2-a ("False inability"): "The user request IS solvable using the available tools, but the assistant claims it cannot be done or refuses without justification." R2-b ("Missing limitation disclosure"): "The user request is NOT solvable with the available tools, but the assistant proceeds as if it is solvable, or fails to clearly explain the limitation."

• R3: Context hallucination — Checks for cross-turn inconsistencies: "Wrongly referencing previous user constraints, preferences, or decisions," "cross-turn inconsistency: changing entities/values (IDs, counts, dates, constraints) without new evidence or tool output," and "conflicting summaries: later summary contradicts earlier established facts."

The judge outputs binary scores (0 or 1) for each rubric. The paper applies strict criteria: "If any single round does not meet (the condition), the corresponding rubric should be scored as 0." Only trajectories that score 1 on all three rubrics (R1=1, R2=1, R3=1) are retained in the final dataset T_final.

Ablation results. The ablation study (Figure 6, Table 4) quantifies the contribution of LLM-based checking:

• For Qwen3-8B-GEM on BFCL V3, removing the LLM-based check drops overall accuracy from 30.25% to 27.38% (a 2.87 percentage point loss).
• For Qwen3-32B-GEM, the drop is smaller: 44.88% → 44.25% (only 0.63 percentage points). The paper does not explain this asymmetry, but it may reflect the larger model's greater robustness to imperfect training data—it can better distinguish useful signal from hallucinated noise.

On τ²-bench (Table 4), removing the LLM-based check has mixed effects: at 8B, Retail Pass@4 drops from 75.44% to 71.05%, while at 32B, Airline Avg@4 drops from 35.50% to 35.00% (minimal) but Retail Avg@4 actually increases from 55.48% to 56.80%. This suggests the hallucination filter may be overly aggressive for some domain-specific evaluations, potentially removing challenging-but-valid trajectories that would benefit training. The paper does not analyze this counterintuitive result, but it highlights a tension between purity (removing all hallucinations) and diversity (retaining edge cases that stress-test the model).

Data yield. The pipeline's sequential filtering stages progressively reduce the dataset size. Starting from the UltraFineWeb corpus:

• Stage 1 (filtering): ~14% of segments pass (the 250K-segment preliminary sample represents a subset).
• Stage 2-3 (extraction + generation): the paper does not report exact yield rates, but presumably some extractions fail to produce coherent workflows or tools.
• Stage 4 (refinement + validation): some trajectories fail the rule-based or LLM-based checks and are discarded.

The final output from the full pipeline is 10K validated trajectories, which are used for all fine-tuning experiments (both the 8B and 32B models, and to train the Trajectory Synthesizer). The paper does not report how many initial trajectories were generated to yield these 10K, making it difficult to estimate the pipeline's computational efficiency end-to-end.

Trajectory Synthesizer — Distilling the Pipeline into an End-to-End Generator

Motivation. The full GEM pipeline involves multiple LLM calls per trajectory: one for filtering, one for extraction, one for generation, one for refinement, and one for hallucination checking—each potentially using a different model (Qwen3-8B for filtering, GLM-4.6 for extraction/generation/refinement, Qwen3-32B for validation). This multi-stage architecture, while modular and debuggable, is computationally expensive and slow. Generating 10K trajectories is feasible for a research paper; generating millions would be prohibitively costly.

The Trajectory Synthesizer addresses this by learning to replicate the entire pipeline in a single model forward pass. This is a form of knowledge distillation: the full pipeline serves as the teacher, producing high-quality (text, trajectory) pairs, and the synthesizer is trained to map directly from text to trajectory, bypassing all intermediate stages.

Training data construction. The synthesizer is trained on the same 10K trajectories produced by the full pipeline. For each data instance, the input x consists of an instruction string ("Turn the following text into multi-turn tool-use trajectories") concatenated with the original text segment. The target output y is the complete validated trajectory, including both the tool definitions (in JSON-schema format) and the full multi-turn dialogue (system prompt, user queries, assistant responses with tool calls, and tool responses).

This input-output pairing is notable: the model must learn to perform filtering, extraction, generation, refinement, and validation implicitly in a single forward pass. It never sees the intermediate artifacts (abstract workflows, initial trajectories, refinement diffs)—only the original text and the final validated trajectory.

Model architecture and training. The synthesizer is initialized from Qwen3-8B and fine-tuned using supervised fine-tuning (SFT). The training configuration mirrors the agent training setup: learning rate 5 × 10⁻⁶, two epochs, full-parameter fine-tuning using LLaMA-Factory (Zheng et al., 2024) with DeepSpeed ZeRO-3, BF16 precision, batch size 64, max sequence length 32K tokens, cosine learning rate schedule with 0.1 warmup ratio, and weight decay 0.05. The tool-call template is "Hermes."

Evaluation of synthesizer quality (Table 2). The paper evaluates the Trajectory Synthesizer by using it to generate a new set of 10K trajectories from the same UltraFineWeb text segments, then fine-tuning Qwen3-8B on these trajectories and evaluating on BFCL V3 and τ²-bench. The comparison is against Qwen3-8B fine-tuned on the original GEM-GLM (full pipeline) trajectories:

• BFCL V3 overall accuracy: GEM-Synthesizer achieves 28.38% vs. GEM-GLM's 30.25%. The gap is small (1.87 percentage points), demonstrating that the synthesizer largely matches the full pipeline's quality. In some subcategories, the synthesizer actually outperforms: Miss Func (41.50% vs. 40.00%) and Long Context (27.50% vs. 28.00% for GEM-GLM, with GEM-Synthesizer's 27.50% being very close).

• τ²-bench Retail Pass@4: GEM-Synthesizer achieves 73.68% vs. GEM-GLM's 75.44%—again, close but slightly lower.

• τ²-bench Airline Avg@4: GEM-Synthesizer achieves 26.00% vs. GEM-GLM's 22.00%—an interesting case where the synthesizer outperforms the full pipeline, possibly because the synthesizer learns to filter out noise that the multi-stage pipeline inadvertently introduces.

Cross-domain generalization of the synthesizer. To test whether the synthesizer generalizes beyond the training corpus, the paper applies it to WikiHow (Koupaee and Wang, 2018), a different text source consisting of how-to guides. The synthesizer—trained only on UltraFineWeb-derived trajectories—is used to generate trajectories from WikiHow text segments, which are then used to fine-tune Qwen3-8B. Results (Table 2):

• BFCL V3 overall accuracy: 28.50% (comparable to UltraFineWeb-based trajectories at 28.38%).
• τ²-bench Airline Pass@4: 42.00% (vs. 40.00% for UltraFineWeb-based).

This is a significant finding: the synthesizer has learned a general text-to-trajectory mapping that transfers to out-of-distribution text sources, not merely memorizing patterns from UltraFineWeb. This suggests the underlying mapping from procedural text to tool-use trajectories is learnable and does not depend on corpus-specific features.

Cost implications. The paper does not provide quantitative latency or cost comparisons between the full pipeline and the synthesizer, but the architecture implies dramatic savings. The full pipeline requires at minimum four LLM forward passes (filtering, extraction, generation, refinement) plus the hallucination detection pass, potentially using multiple different models. The synthesizer requires a single forward pass using a single 8B model. This makes large-scale data generation—millions of trajectories from web-scale corpora—computationally tractable in a way the full pipeline is not.

Why the synthesizer works: the learnability hypothesis. The synthesizer's strong performance implies an important property of the data generation task: the mapping from text to trajectory is, to a significant degree, learnable through supervised imitation. The full pipeline encodes a complex sequence of reasoning steps (identify procedural content → abstract workflows → design APIs → generate dialogue → increase complexity → verify correctness), but the synthesizer demonstrates that an 8B model can internalize this process sufficiently to produce trajectories that fine-tune agents nearly as effectively as those from the teacher pipeline. This doesn't mean the synthesizer is performing the same internal reasoning—it may be learning heuristic shortcuts that produce similarly-structured outputs—but the end result is functionally equivalent for downstream training.

Unanswered questions about the synthesizer. The paper leaves several aspects of the synthesizer unexplored:

• Does the synthesizer handle edge cases where the input text contains no usable procedural content? The full pipeline's filtering stage explicitly discards such texts; the synthesizer must either learn to produce empty or minimal outputs, or it might hallucinate trajectories from non-procedural text.
• What is the quality distribution of synthesizer outputs compared to the full pipeline? The paper reports mean benchmark performance but not per-trajectory quality metrics.
• Can the synthesizer be iteratively improved by generating trajectories, fine-tuning agents on them, evaluating, and using agent performance as a reward signal to refine the synthesizer? This would close the loop between data generation and agent training.

Summary of Design Choices and Their Justifications

The GEM pipeline embodies several deliberate engineering decisions that collectively enable text-to-trajectory synthesis:

• Sequential extraction + generation over direct text-to-trajectory: The intermediate workflow-and-tool representation makes the generation task easier by providing structured guidance, enables modular optimization (different models for different stages), and produces inspectable intermediate artifacts for debugging.

• Single-pass trajectory generation over multi-agent simulation: Prioritizes computational efficiency to enable generating 10K trajectories at manageable cost, accepting potential quality tradeoffs that the refinement stage subsequently addresses.

• Full-trajectory refinement over targeted editing: Rather than identifying specific weaknesses and patching them, the refinement stage performs a complete rewrite, which enables holistic improvements across all trajectory components (system prompt, user requests, assistant behavior, tool responses) without being constrained by the initial trajectory's structure.

• Dual-stage validation (rule-based + LLM) over single-method filtering: Rule-based checks guarantee structural correctness deterministically; LLM-based hallucination detection catches the subtler semantic errors that rule-based methods cannot detect. Each method addresses failure modes the other misses.

• Diversity requirements enforced in prompts over post-hoc filtering: By requiring multiple interaction patterns, pitfall inclusions, and constraint complexity directly in the generation and refinement prompts, the pipeline produces diverse trajectories by construction rather than filtering for diversity after generation, which would waste compute on trajectories that would be discarded.

• Read-write tool pairing over read-only tools: Including both query and modification tools forces trajectories to include state-changing operations, which teach the agent about consequence awareness, user confirmation, and error recovery—capabilities evaluated on benchmarks like τ²-bench.

• Distillation into Trajectory Synthesizer over scaling the full pipeline: Rather than optimizing the multi-stage pipeline for throughput, the paper demonstrates that the entire mapping can be compressed into a single model, making the paradigm practical for web-scale data generation while providing scientific evidence that text-to-trajectory mapping is a learnable capability.

4. Key Insights and Innovations

Innovation 1: Reframing the Data Source — From API-Centric Simulation to Text-Centric Extraction

The paper's most fundamental intellectual contribution is not a specific algorithmic improvement but a paradigm-level reframing of where agent training data comes from. The dominant approach in prior work—spanning ToolBench (Qin et al., 2023), APIGEN-MT (Prabhakar et al., 2025), TOUCAN (Xu et al., 2025), MAGNET (Yin et al., 2025), ToolACE-MT (Zeng et al., 2025), and MUA (Zhao et al., 2025)—operates under a shared, largely unquestioned assumption: to generate tool-use trajectories, you must start with a predefined set of APIs and simulate interactions within that environment. The paper calls this the "tool-centered simulation paradigm" (Section 1, Figure 1) and systematically argues that it imposes a coverage ceiling: the diversity of training data is bounded by the diversity of APIs you can collect, document, and maintain. No matter how many APIs you gather, you are sampling from a manually curated space, and the resulting agents inherit that curation's blind spots.

GEM's conceptual move is to ask a different question entirely. Instead of "given these APIs, what tasks can we synthesize?", it asks "given this massive corpus of human-written text documenting real problem-solving, what procedural knowledge can we extract and operationalize?" The data source shifts from API specifications (structured, narrow, manually collected) to text corpora (unstructured, broad, naturally occurring). This is not an incremental improvement—it is a change in what counts as a legitimate starting point for agent data generation.

The significance of this reframing extends beyond the immediate empirical results. Prior work treats text corpora as a source of linguistic knowledge (pretraining) and APIs as a source of tool-use knowledge (fine-tuning). GEM dissolves this boundary: the same text that teaches a model grammar and facts also contains implicit procedural knowledge—step-by-step problem-solving experiences—that can be converted into executable agent trajectories. The paper's preliminary analysis (Section 3.1) provides the empirical foundation for this claim: ~14% of randomly sampled UltraFineWeb segments contain explicit multi-step operational procedures spanning dozens of domains, from customer support to education to healthcare. This 14% represents an enormous absolute volume of procedural knowledge that has never been systematically exploited for agent training.

This reframing has a specific intellectual consequence: it changes what "generalization" means for tool-use agents. In the API-centric paradigm, generalization is typically measured by how well an agent handles unseen compositions of known APIs or unseen parameter values. In the text-centric paradigm, generalization is measured by how well an agent trained on arbitrary web text performs on completely unseen domains with entirely novel APIs—the exact scenario tested by τ²-bench (Airline and Retail). The paper's results on τ²-bench (Figure 5) are the strongest evidence for this distinction: Qwen3-32B-GEM, trained on trajectories synthesized from general web text with no exposure to airline or retail APIs, achieves 55.48% Avg@4 on Retail and 35.50% on Airline—competitive with or exceeding models fine-tuned on in-domain τ-bench synthetic data (SIMIA, APIGEN-MT). This is not "training on API X and testing on API X." It is "training on text about photo editing, hospital claims, and music visualizers and testing on airline reservation systems." The fact that this works at all is the paradigm's validation.

This contribution is fundamental rather than incremental because it redefines the input space for a whole subfield. It does not improve an existing pipeline; it argues that the pipeline should start from a different kind of data altogether. The downstream implications—if text corpora are indeed a viable source for agent training—are substantial: the scalability ceiling imposed by API collection is removed, replaced by the effectively unbounded scale of web text.

However, the paradigm is not a complete solution. The paper's own ablation shows a 12-point accuracy drop on BFCL V3 when the refinement stage is removed (Figure 6, 32B model), indicating that raw text-extracted trajectories are not immediately sufficient. The paradigm provides the data source; the GEM pipeline provides the extraction method. The paradigm's generality will depend on whether extraction quality can be improved to the point where complex, engineering-heavy refinement becomes less necessary. The paper opens this research direction without closing it.

Innovation 2: Trajectory Complexity as a First-Class Design Objective — The Refinement Stage's Role and the Empirical Case for "Harder Is Better"

A less visible but equally important contribution is the paper's explicit treatment of trajectory complexity as a quantifiable, optimizable design objective rather than an emergent property of the generation process. Prior work on multi-turn tool-use data synthesis typically focuses on correctness (are the tool calls valid? are the turn structures correct?) and diversity (do trajectories cover different task types?). GEM adds a third dimension that the paper demonstrates is empirically decisive: complexity, measured along multiple axes—number of tools used, number of tool calls per trajectory, depth of tool-call chains, presence of challenging interaction patterns (clarification, error recovery, constraint conflicts), and ambiguity of user requests.

The paper's commitment to complexity is not merely aspirational—it is operationalized through a dedicated refinement stage (Stage 4, Section 3.2) that takes structurally correct but "relatively straightforward" initial trajectories and systematically rewrites them to increase difficulty. The refinement prompt (Appendix A.4) does not simply ask the model to "make the trajectory better." It enumerates specific complexity dimensions: expanding the toolset, defining explicit database schemas, adding multi-constraint user requests, requiring cross-turn reasoning, including at least 1–2 "pitfalls" (deliberate traps testing the assistant's ability to identify subtle constraint violations), making tool responses return structured nested data rather than simple values, and introducing non-trivial error responses that don't directly tell the assistant how to recover.

This is intellectually distinctive because it inverts a common assumption in data synthesis: that the goal is to produce representative training data that matches the distribution of deployment scenarios. GEM's approach is closer to adversarial curriculum design—deliberately generating trajectories that are harder than most real-world interactions, on the hypothesis that training on hard cases produces more robust agents. The paper provides strong but indirect evidence for this hypothesis through the ablation study (Figure 6): removing refinement drops Qwen3-32B's BFCL V3 overall accuracy from 44.88% to 32.50%, a 12.38 percentage point gap that dwarfs the impact of removing hallucination filtering (0.63 points). Appendix D (Table 5) quantifies the structural difference: refinement nearly doubles the average tool calls per trajectory (7.83 → 16.3) and increases average messages by over 50% (30.05 → 46.1).

The significance of this finding extends beyond the specific GEM pipeline. It suggests that trajectory complexity is a trainable axis of data quality, not merely an attribute of the data source. Even if the text corpus provides rich procedural knowledge, extracting that knowledge into maximally informative training trajectories requires deliberate engineering. The paper's refinement stage is a specific implementation of this principle, but the principle itself—that difficulty should be explicitly designed, not passively sampled—is the conceptual contribution. It implies that future work on agent training data should measure and report trajectory complexity metrics (tool call depth, interaction pattern diversity, constraint density) as standard quality indicators alongside downstream benchmark performance.

This contribution is incremental in mechanism (refinement is an additional LLM call in the pipeline) but fundamental in implication: it establishes that how you process extracted procedural knowledge matters as much as where you extract it from. The text-to-trajectory paradigm provides the raw material; complexity-targeted refinement transforms it into effective training data. The paper does not claim to have solved the complexity optimization problem—the refinement prompt is hand-crafted and likely suboptimal—but it identifies the problem as worth solving and provides quantitative evidence that solving it matters.

Innovation 3: Distillation as a Proof of Learnability — The Trajectory Synthesizer Demonstrates Text-to-Trajectory Is a Coherent Learned Capability

The Trajectory Synthesizer (Section 3.2, evaluated in Section 4.3) might appear at first glance to be an engineering contribution—a way to make the pipeline cheaper. That framing undervalues it. The synthesizer's true intellectual contribution is that it serves as an existence proof: the mapping from unstructured text to structured multi-turn tool-use trajectories is sufficiently coherent and systematic that an 8B model can learn it through supervised fine-tuning on 10K examples and generalize it to unseen text sources.

This is not obvious a priori. The full GEM pipeline involves multiple distinct reasoning steps performed by different models with carefully engineered prompts: binary classification (filtering), structured extraction with constraint reasoning (workflow and tool design), creative generation with pattern enforcement (trajectory generation), complexity-targeted rewriting (refinement), and hallucination verification (validation). Each stage encodes domain knowledge about what makes a good trajectory—the extraction prompt encodes knowledge about API design principles (single-function, descriptive naming, read-write pairing), the generation prompt encodes knowledge about interaction patterns (clarification, error recovery, constraint conflicts), the refinement prompt encodes knowledge about complexity dimensions. The fact that an 8B model can absorb all of this implicit knowledge from input-output pairs—without seeing the intermediate reasoning steps, without access to the prompts that encode the design principles, without any explicit instruction on what makes a trajectory "good"—is a non-trivial finding about the learnability of the underlying mapping.

The evidence for this claim comes from Table 2. The Trajectory Synthesizer, trained on UltraFineWeb-derived (text, trajectory) pairs, generates trajectories that fine-tune Qwen3-8B to within 1.87 percentage points of the full GEM pipeline on BFCL V3 overall accuracy (28.38% vs. 30.25%). More importantly, when applied to WikiHow—a completely different text source with different stylistic conventions, domain distributions, and procedural description patterns—the same synthesizer produces trajectories that yield comparable performance (28.50% on BFCL V3, 42.00% Pass@4 on τ²-bench Airline). This cross-corpus generalization is the critical result: the synthesizer has not memorized patterns specific to UltraFineWeb but has learned a transferable text-to-trajectory capability.

This finding has implications for the research agenda the paper opens. If text-to-trajectory mapping is learnable through SFT, then:

• The mapping can potentially be improved through better training data (more diverse text sources, higher-quality pipeline outputs) rather than better prompts.
• Iterative self-improvement becomes possible: the synthesizer generates trajectories → agents are fine-tuned and evaluated → high-performing trajectories are fed back as additional training data → the synthesizer improves.
• The mapping might be learnable by even smaller models, enabling on-device trajectory generation from user-provided procedural text.

This contribution is fundamental rather than incremental because it demonstrates that the text-to-trajectory paradigm—which the GEM pipeline establishes as possible through a complex, multi-stage process—is actually a unified capability that can be compressed into a single model. This transforms the paradigm from a research pipeline (useful for producing a fixed dataset) into a deployable capability (useful for on-demand trajectory generation at scale). The paper does not fully explore this transformation—the synthesizer is evaluated only on its ability to replicate the pipeline's output quality, not on its ability to scale to truly web-scale generation—but the existence proof is the key contribution.

Innovation 4: Out-of-Domain Generalization as the Primary Evaluation Criterion — A Distinctive Empirical Strategy

The paper's evaluation strategy embodies a conceptual claim about what constitutes meaningful progress in agent training that differs from standard practice in the tool-use literature. Most prior work on multi-turn tool-use data synthesis evaluates by training on the synthesized data and testing on benchmarks that either use the same APIs (in-domain) or evaluate general function-calling capability (BFCL). The paper goes further by making out-of-domain generalization the central empirical demonstration: the τ²-bench evaluation (Figure 5) tests models trained on text-derived trajectories against completely unseen domains (Airline, Retail) with novel APIs, and explicitly compares against baselines that were trained on in-domain synthetic data generated within those exact environments (APIGEN-MT, SIMIA).

This is a distinctive empirical strategy because it changes what the experiment is testing. Standard evaluation asks: "Does training on this dataset improve performance on related tasks?" The τ²-bench evaluation asks: "Does training on trajectories derived from arbitrary web text produce an agent that can handle specialized domain tasks with unfamiliar tools?" The answer—GEM-trained models match or exceed in-domain baselines despite having zero exposure to the target APIs—is the paper's strongest evidence that text-derived training teaches something more fundamental than API-specific patterns. It teaches general tool-use reasoning: how to attend to system constraints, how to chain tool calls based on outputs, how to clarify ambiguous requests, how to recover from errors, how to verify preconditions before executing state-changing operations.

The paper makes this claim explicit: "our text-based synthesis pipeline instills a fundamental understanding of tool-use reasoning that transfers effectively to unseen, real-world domains" (Section 4.2). This is not a claim about data scale or diversity—though those matter—but about the kind of knowledge that text-derived trajectories encode. Because they are extracted from real human problem-solving documentation, they naturally include the edge cases, constraints, and interaction patterns that make tool use genuinely difficult. An agent trained on these trajectories has practiced navigating ambiguity, constraint conflicts, multi-hop dependencies, and error recovery in dozens of different domains and tool environments. When dropped into a new domain (airline reservations), it already possesses the meta-skills needed to operate effectively.

This evaluation strategy is significant beyond the specific results because it sets a higher bar for the field. If the goal of agentic training is, as the paper states, "exposure to a sufficiently broad range of scenarios during training to enable agents to generalize effectively to unseen environments and scenarios" (Section 1), then evaluating on seen environments is insufficient. The paper demonstrates that out-of-domain generalization is achievable and measurable, and it provides a specific methodology for testing it: train on data derived from general-domain text, test on specialized domain benchmarks, and compare against models trained on in-domain synthetic data. This methodology could—and arguably should—become standard practice for evaluating data synthesis approaches.

This contribution is incremental in mechanism (it's an evaluation choice, not a technical innovation) but fundamental in implication: it reframes what "good" training data means. Data is not good because it matches the test distribution; data is good because it teaches generalizable capabilities. The paper's results suggest that text-derived trajectories, despite being out-of-domain for every benchmark tested, achieve this better than in-domain synthetic data for certain evaluations—a finding that should shift how the field thinks about data quality.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two benchmarks: (1) BFCL V3 (Berkeley Function Calling Leaderboard, Patil et al.), specifically its multi-turn scenarios comprising four categories—Multi Turn Base, Miss Func, Miss Param, and Long Context—with 200 tasks per category, where agents interact with a Python-based API environment; and (2) τ²-bench (Barres et al., 2025), which evaluates user-agent interactions in specialized real-world domains (Airline and Retail) using a dual-control environment where both agent and user can invoke tools, with GPT-4.1 serving as the user simulator. BFCL V3 measures general function-calling capability across diverse interaction patterns; τ²-bench measures domain-specific task success with unseen APIs, directly testing out-of-domain generalization.

• Base model(s). The paper fine-tunes Qwen3-8B and Qwen3-32B (both from the Qwen3 family) on the synthesized trajectories. These models are chosen as representative open-weight LLMs at two distinct scales (8B and 32B parameters), enabling analysis of how data quality interacts with model capacity. For trajectory generation (the teacher model), the paper employs GLM-4.6, described as a "strong teacher model" (Section 3.2), though the specific rationale for this choice over alternatives is not provided.

• Metrics. On BFCL V3, the paper reports accuracy (%) for each of the four multi-turn subcategories and an overall accuracy aggregated across all 800 tasks. Accuracy measures whether the model's function calls match the expected API invocations, evaluated using AST-based checks as specified by the BFCL framework. On τ²-bench, the paper reports Avg@4 (average task success rate across 4 independent runs) and Pass@4 (fraction of tasks where at least one of 4 runs succeeds), following the original benchmark protocol. These metrics capture both average performance and best-case capability, which is important for agent tasks where stochasticity in tool selection can produce high variance.

• Baselines. The paper compares against four open-source multi-turn tool-use datasets used to fine-tune the same base models: (1) APIGEN-MT (Prabhakar et al., 2025), an in-domain τ-bench synthetic dataset generated via simulated agent-human interplay; (2) TOUCAN (Xu et al., 2025), synthesized from real-world MCP server environments, with 50K multi-turn samples randomly drawn; (3) MUA (Zhao et al., 2025), a multi-turn user-interacting agent dataset; and (4) SIMIA (Li et al., 2025, referred to as "Simia-Tau" and "Simia" interchangeably), sampled at 50K multi-turn instances. Critically, APIGEN-MT and SIMIA are in-domain for τ²-bench—they are generated within the τ-bench environment (Airline and Retail) using the same or similar APIs that appear in the τ²-bench test sets. GEM-synthesized data is strictly out-of-domain for τ²-bench, having no exposure to airline or retail APIs. The paper also reports proprietary model baselines for BFCL V3: GPT-4.1 (38.88% overall), DeepSeek-V3.2-Exp (37.38%), and Gemini-2.5-Flash (36.25%). TOUCAN and MUA results at 8B are reported but SIMIA is excluded from BFCL V3 because "its scores [are] generally low, likely due to its reliance solely on in-domain τ-bench data" (Section 4.2).

• Generation budget / compute accounting. The primary resource measure for data synthesis is the number of trajectories generated: the full GEM pipeline produces 10K validated trajectories from UltraFineWeb, which are used to fine-tune all models (8B and 32B) and to train the Trajectory Synthesizer. The paper does not report the total LLM inference cost (number of forward passes, token counts, or FLOPs) to produce these 10K trajectories, nor the yield rate (how many initial text segments were processed to produce 10K validated trajectories). For the Trajectory Synthesizer, cost is measured qualitatively: the synthesizer "provides a cost-effective, end-to-end solution for large-scale data generation, matching the quality of the multi-stage synthesis pipeline while significantly reducing costs" (Section 1), but no quantitative cost comparison (e.g., tokens per trajectory, wall-clock time per trajectory) is provided.

• Cross-validation / statistical protocol. None is reported. The paper does not describe cross-validation, multiple random seeds, or confidence intervals for any experimental results. All fine-tuning runs appear to use a single training run per configuration. The 10K trajectory dataset is a fixed set produced once by the pipeline; there is no analysis of variance across different random samples of training data or different teacher model generations. For τ²-bench, the Avg@4 and Pass@4 metrics inherently capture some stochasticity through multiple evaluation runs, but training-side variance is not assessed.

Main Quantitative Results

BFCL V3: GEM Data Dramatically Outperforms Existing Open-Source Datasets and Rivals Proprietary Models

The headline result from Table 1 is that Qwen3-32B fine-tuned on GEM data achieves 44.88% overall accuracy on BFCL V3 Multi-Turn, exceeding all open-source baselines by wide margins and outperforming proprietary models including GPT-4.1 (38.88%), DeepSeek-V3.2-Exp (37.38%), and Gemini-2.5-Flash (36.25%). The improvement over the Qwen3-32B base model (28.35%) is +16.53 percentage points—a 58% relative improvement.

Comparison by model scale (Table 1). At the 8B level, Qwen3-8B-GEM achieves 30.25% overall accuracy, compared to:

• Qwen3-8B base: 18.00% (+12.25 points)
• APIGEN-MT: 21.00% (+9.25 points over APIGEN-MT)
• TOUCAN: 21.88% (+8.37 points)
• MUA: 21.13% (+9.12 points)

At the 32B level, Qwen3-32B-GEM achieves 44.88% overall accuracy, compared to:

• Qwen3-32B base: 28.35% (+16.53 points)
• APIGEN-MT: 29.50% (+15.38 points)
• TOUCAN: 35.00% (+9.88 points)
• MUA: 26.25% (+18.63 points)

The gap between GEM and the best open-source baseline (TOUCAN at 35.00%) is 9.88 percentage points at 32B scale—GEM data provides nearly 10 absolute points of additional accuracy over the next best publicly available dataset.

Per-category breakdown reveals where GEM excels (Table 1). The four BFCL V3 categories test different failure modes:

• Multi Turn Base (standard multi-turn interactions): Qwen3-32B-GEM achieves 52.00% vs. TOUCAN's 41.00% and Qwen3-32B base's 34.00%. GEM-trained models excel at basic multi-turn coordination.
• Miss Func (missing function scenarios): Qwen3-32B-GEM achieves 40.00% vs. TOUCAN's 37.50% and APIGEN-MT's 27.00%. The advantage over APIGEN-MT is substantial (+13 points), suggesting GEM trajectories—which include explicit "capability limitation awareness" patterns where the assistant must recognize unavailable functions—directly teach models to handle missing tools.
• Miss Param (missing parameter scenarios): Qwen3-32B-GEM achieves 38.50% vs. TOUCAN's 26.00% and Qwen3-32B base's 25.50%. The +12.5-point improvement over TOUCAN is the largest per-category gap among all subcategories. GEM trajectories explicitly include clarification patterns where the assistant requests missing parameters, which appears to transfer directly to this evaluation scenario.
• Long Context (extended dialogues): Qwen3-32B-GEM achieves 49.00% vs. TOUCAN's 35.50% and Qwen3-32B base's 29.50%. GEM trajectories average 46.1 messages per dialogue (Section 4.5), providing extensive training on long-context state tracking. The +13.5-point improvement over TOUCAN—whose trajectories average only 6.24 messages—strongly suggests that trajectory length during training directly impacts long-context performance.

Qwen3-8B-GEM vs. proprietary models. At the 8B scale, Qwen3-8B-GEM (30.25% overall) does not match proprietary models on overall accuracy, but it substantially closes the gap relative to the base model. On Miss Func specifically, Qwen3-8B-GEM (30.00%) approaches DeepSeek-V3.2-Exp (39.50%) and Gemini-2.5-Flash (36.00%), suggesting that even an 8B model can learn robust missing-function handling from GEM data.

τ²-Bench: Out-of-Domain GEM Data Matches In-Domain Synthetic Data on Specialized Domains

The paper's most striking empirical claim is that GEM-trained models—fine-tuned on trajectories synthesized from general web text with zero exposure to airline or retail APIs—achieve performance comparable to or exceeding models trained on in-domain τ-bench synthetic data (APIGEN-MT, SIMIA). Table 5 (Figure 5 in the paper) presents these results.

Retail domain (Table 5). Qwen3-32B-GEM achieves 55.48% Avg@4 and 86.84% Pass@4 on τ²-bench Retail. This outperforms:

• APIGEN-MT (in-domain): 44.52% Avg@4, 74.56% Pass@4 (+10.96 Avg@4, +12.28 Pass@4)
• SIMIA (in-domain): 48.03% Avg@4, 73.68% Pass@4 (+7.45 Avg@4, +13.16 Pass@4)
• MUA: 49.56% Avg@4, 80.70% Pass@4 (+5.92 Avg@4, +6.14 Pass@4)
• Qwen3-32B base: 43.20% Avg@4, 70.18% Pass@4 (+12.28 Avg@4, +16.66 Pass@4)

At the 8B scale on Retail, Qwen3-8B-GEM achieves 44.52% Avg@4 and 75.44% Pass@4, exceeding APIGEN-MT (42.54% Avg@4, 69.30% Pass@4) and being competitive with SIMIA (43.20% Avg@4, 70.18% Pass@4). The Pass@4 score of 75.44% means that in at least one of four attempts, the 8B GEM-trained model successfully completes over three-quarters of Retail tasks—despite never having seen retail-specific APIs during training.

Airline domain (Table 5). Qwen3-32B-GEM achieves 35.50% Avg@4 and 56.00% Pass@4 on τ²-bench Airline. This is:

• Slightly below SIMIA (38.00% Avg@4, 62.00% Pass@4) by 2.5 Avg@4 and 6.0 Pass@4 points
• Comparable to APIGEN-MT (36.00% Avg@4, 52.00% Pass@4)
• Substantially above Qwen3-32B base (21.00% Avg@4, 40.00% Pass@4)

At the 8B scale on Airline, Qwen3-8B-GEM (22.00% Avg@4, 40.00% Pass@4) matches APIGEN-MT (23.50% Avg@4, 42.00% Pass@4) but trails SIMIA (35.50% Avg@4, 52.00% Pass@4) significantly. This is the one domain-scale combination where GEM data shows a clear deficit against in-domain baselines, suggesting that Airline domain—which involves complex scheduling, pricing rules, and multi-party coordination—may be harder to learn from general web text patterns than Retail.

The generalization claim (Section 4.2). The paper frames these results as evidence that "our text-based synthesis pipeline instills a fundamental understanding of tool-use reasoning that transfers effectively to unseen, real-world domains." The critical comparison is GEM vs. APIGEN-MT/SIMIA on τ²-bench: GEM models have never seen the target domain's APIs, yet they match or exceed models that were trained on data synthesized within that exact environment. This is the strongest evidence in the paper for the text-based paradigm's advantage over API-centric simulation—it demonstrates that the diversity and authenticity of text-derived trajectories produce more generalizable tool-use capabilities than domain-specific synthetic data.

However, this comparison has an asymmetry: APIGEN-MT and SIMIA models were trained on the specific tool environments they're tested on, while GEM models were trained on trajectories from arbitrary domains. For GEM to match them is impressive, but the comparison doesn't control for the total volume or diversity of training data—GEM's 10K trajectories span dozens of domains, while APIGEN-MT's trajectories are concentrated in two domains. The performance parity might reflect GEM's domain diversity advantage rather than the text-based paradigm per se.

Trajectory Synthesizer: Near-Pipeline-Quality at Lower Cost

The paper's Trajectory Synthesizer—an 8B model trained to map text directly to trajectories—is evaluated by generating a new set of 10K trajectories from UltraFineWeb text segments, fine-tuning Qwen3-8B on these trajectories, and comparing against Qwen3-8B fine-tuned on the original GEM-GLM (full pipeline) trajectories (Table 2).

Main comparison (Table 2, rows "GEM-GLM" vs. "GEM-Synthesizer" on UltraFineWeb). On BFCL V3 overall accuracy:

• GEM-GLM (full pipeline): 30.25%
• GEM-Synthesizer: 28.38%
• Gap: 1.87 percentage points

On per-category BFCL V3:

• Multi Turn Base: GEM-Synthesizer (28.38%) slightly exceeds GEM-GLM (30.25% overall, but per-category breakdown for synthesizer shows 28.38% overall; the paper does not break out synthesizer per-category accuracy in Table 2 beyond the overall score)
• Miss Func: GEM-Synthesizer achieves 23.50% vs. GEM-GLM's 30.00% (this is a notable gap—the synthesizer underperforms on missing-function scenarios by 6.5 points)
• Miss Param: GEM-Synthesizer achieves 27.50% vs. GEM-GLM's 28.00% (nearly identical)
• Long Context: GEM-Synthesizer achieves 21.00% vs. GEM-GLM's 23.00% (close)

Wait—the numbers don't reconcile. Table 2 shows GEM-Synthesizer (UltraFineWeb) with "Overall Acc" of 28.38% and per-category scores: Base 28.38? No—the table structure is ambiguous. Let me re-read. Table 2 columns are: Model, BFCL V3 Multi-Turn (with sub-columns: Overall Acc, Base, Miss Func, Miss Param, Long Context), then Tau2 (Airline Avg@4, Airline Pass@4, Retail Avg@4, Retail Pass@4). The row "GEM-Synthesizer (Ultrafineweb)" shows: 28.38, 41.50, 23.50, 27.50, 21.00. So:

• Overall Acc: 28.38%
• Multi Turn Base: 41.50%
• Miss Func: 23.50%
• Miss Param: 27.50%
• Long Context: 21.00%

Compared to the full pipeline (GEM-GLM): 30.25, 40.00, 30.00, 28.00, 23.00. The synthesizer actually outperforms the full pipeline on Multi Turn Base (41.50% vs. 40.00%) and is close on Miss Param (27.50% vs. 28.00%) and Long Context (21.00% vs. 23.00%). The largest gap is Miss Func (23.50% vs. 30.00%, a 6.5-point difference). On τ²-bench, GEM-Synthesizer achieves Retail Pass@4 of 73.68% vs. GEM-GLM's 75.44% (gap of 1.76 points) and Airline Pass@4 of 40.00% vs. 40.00% (identical). Airline Avg@4: 26.00% vs. 22.00%—the synthesizer outperforms the full pipeline by 4 points on this metric.

Cross-domain generalization (Table 2, "GEM-Synthesizer (Wikihow)"). The synthesizer, trained only on UltraFineWeb-derived trajectories, is applied to WikiHow text. The resulting trajectories fine-tune Qwen3-8B to achieve:

• BFCL V3 overall: 28.50% (vs. 28.38% on UltraFineWeb, essentially identical)
• τ²-bench Airline Pass@4: 42.00% (vs. 40.00% for UltraFineWeb-based synthesizer; beats the full pipeline's 40.00%)
• τ²-bench Retail Pass@4: 68.42% (lower than UltraFineWeb-based at 73.68%, suggesting domain shift affects Retail more than Airline)

The critical finding is that BFCL V3 overall accuracy is nearly identical whether the synthesizer processes UltraFineWeb or WikiHow text (28.38% vs. 28.50%), demonstrating that the learned text-to-trajectory mapping generalizes across corpora with different stylistic conventions. This is evidence that the synthesizer has captured the abstract mapping from "text containing procedural knowledge" to "valid tool-use trajectory," not merely memorized UltraFineWeb-specific patterns.

Data Complexity Analysis: GEM Trajectories Are Substantially Richer Than Existing Datasets

Section 4.5 and Figure 7 present statistical profiles of the synthesized trajectories across three dimensions, compared implicitly against prior datasets:

• Number of distinct tools per trajectory: mean 8.6, median 8.0 (Figure 7, left). This means each trajectory requires the model to meaningfully select from and combine approximately 8–9 different API functions. APIGEN-MT averages 18.5 turns total but the paper doesn't report its tool count; the comparison point is the tool-call count rather than the tool count.

• Number of messages per dialogue: mean 46.1, median 41.0 (Figure 7, center). This includes all user messages, assistant responses, and tool responses. The paper explicitly compares: "existing open-source datasets such as APIGEN-MT average around 18.5 turns, while TOUCAN contains only about 6.24 turns." GEM trajectories are 2.5× longer than APIGEN-MT and 7.4× longer than TOUCAN in terms of conversational turns. The right-skewed distribution (median 41.0, mean 46.1) indicates that most trajectories cluster around 40-50 messages but some extend much further.

• Number of tool calls per trajectory: mean 16.3, median 13.0 (Figure 7, right). GEM trajectories average nearly 4× more tool calls than APIGEN-MT (4.3 tool calls per trajectory, reported in Section 4.5). The distribution shows that a typical trajectory involves 10-20 tool calls, with some extending beyond 50. This high tool-call density forces models to learn state tracking across many operations, parameter propagation between dependent calls, and error recovery when mid-sequence calls fail—precisely the capabilities evaluated by BFCL V3's Long Context and τ²-bench's multi-step task scenarios.

These statistics are presented as evidence of trajectory complexity, but they also raise a question the paper does not address: is there a point of diminishing returns where additional length and tool-call density no longer improve downstream performance, or where the training signal becomes diluted by repetitive or low-information interactions? The ablation showing a 12-point accuracy gain from refinement (which increased messages from 30.05 to 46.1, Table 5) suggests that within this range, more complexity helps—but the optimal complexity ceiling is unexplored.

Ablation Studies and Robustness Checks

Refinement stage (Section 4.4, Figure 6 and Table 5). Removing the refinement stage causes the largest performance degradation of any ablation. For Qwen3-32B-GEM on BFCL V3 overall accuracy: with refinement, 44.88%; without refinement, 32.50%—a drop of 12.38 percentage points. For Qwen3-8B-GEM: with refinement, 30.25%; without refinement, 26.00%—a drop of 4.25 points. The asymmetry is notable: the 32B model benefits roughly 3× more from refinement than the 8B model in absolute terms (12.38 vs. 4.25 points), suggesting that larger models are better able to exploit the additional complexity that refinement introduces. On τ²-bench (Table 4), without refinement: Qwen3-32B Retail Avg@4 drops from 55.48% to 40.35% (−15.13 points), while Airline Avg@4 drops from 35.50% to 31.00% (−4.50 points). The Retail domain appears particularly sensitive to trajectory complexity, possibly because retail interactions involve more nuanced constraint handling (return policies, inventory management, pricing rules) that benefit from the extended training trajectories.

The structural impact of refinement is quantified in Table 5: trajectories go from averaging 30.05 messages, 5.01 tools, and 7.83 tool calls (before refinement) to 46.1 messages, 8.6 tools, and 16.3 tool calls (after refinement). Every dimension approximately doubles: messages increase 53%, tools increase 72%, tool calls increase 108%. This suggests the refinement stage is not merely polishing edges but fundamentally restructuring trajectories to incorporate more tool interactions and longer dialogue chains.

LLM-based hallucination check (Section 4.4, Figure 6 and Table 4). Removing the LLM-based validation filter (which detects fabricated parameter values, capability misrepresentations, and context inconsistencies; Section 3.2, Appendix A.5) produces:

• Qwen3-8B-GEM BFCL V3 overall: 30.25% → 27.38% (−2.87 points)
• Qwen3-32B-GEM BFCL V3 overall: 44.88% → 44.25% (−0.63 points)

The 8B model is more sensitive to hallucinated training data than the 32B model, which is consistent with the hypothesis that larger models have greater capacity to distinguish useful signal from noise—they can "look past" occasional hallucinations in training trajectories that would mislead a smaller model. On τ²-bench (Table 4), the effect is mixed: Qwen3-8B Retail Pass@4 drops from 75.44% to 71.05% (−4.39 points), but Qwen3-32B Retail Avg@4 actually increases from 55.48% to 56.80% (+1.32 points) when the hallucination filter is removed. This counterintuitive result suggests the filter may be removing some trajectories that, despite containing marginal hallucinations, provide valuable training diversity. The hallucination detector is conservative by design ("If any single round does not meet (the condition), the corresponding rubric should be scored as 0," Appendix A.5), potentially filtering out edge cases that would improve robustness.

Data source generalization (Table 2, WikiHow experiment). When the Trajectory Synthesizer—trained exclusively on UltraFineWeb-derived (text, trajectory) pairs—is applied to WikiHow text, the resulting trajectories fine-tune Qwen3-8B to BFCL V3 overall accuracy of 28.50%, compared to 28.38% when the same synthesizer processes UltraFineWeb text. The 0.12 percentage point difference is negligible, indicating that the synthesizer's learned text-to-trajectory capability transfers across corpora with no measurable degradation. On τ²-bench, WikiHow-based trajectories yield Airline Pass@4 of 42.00% (higher than UltraFineWeb-based at 40.00%) but Retail Pass@4 of 68.42% (lower than UltraFineWeb-based at 73.68%). This domain asymmetry may reflect differences in corpus composition: WikiHow's how-to guides may contain more structured instructional content that maps well to airline-type procedural tasks but less of the customer-interaction patterns that characterize retail scenarios.

Comparison against in-domain baselines is not a controlled experiment. While not a formal ablation, an important caveat is that the comparison between GEM-trained models and APIGEN-MT/SIMIA-trained models on τ²-bench (Figure 5) does not control for total training data volume, domain diversity, or trajectory complexity. APIGEN-MT and SIMIA trajectories are generated within the τ-bench environment, so they are inherently limited to the APIs and task types of those specific domains. GEM trajectories span dozens of domains with higher average complexity (46.1 messages, 16.3 tool calls vs. APIGEN-MT's 18.5 turns and 4.3 tool calls). The performance parity could result from GEM's higher trajectory complexity and domain diversity rather than the text-based paradigm per se. A controlled experiment would generate in-domain τ-bench trajectories with matched complexity (similar length, tool call density, interaction pattern diversity) and compare—but this experiment is not run.

Critical Assessment

Claim 1: "Our proposed GEM method demonstrates clear improvements over baseline models at both the 8B and 32B scales" on BFCL V3 (Section 4.2). This claim is well-supported by Table 1. Qwen3-8B-GEM (30.25%) outperforms all 8B baselines (APIGEN-MT 21.00%, TOUCAN 21.88%, MUA 21.13%) and the base model (18.00%). Qwen3-32B-GEM (44.88%) outperforms all 32B baselines (APIGEN-MT 29.50%, TOUCAN 35.00%, MUA 26.25%) and the base model (28.35%). The margins are substantial (4–19 absolute percentage points) and consistent across all four BFCL V3 subcategories.

However, the evaluation does not establish whether the improvement comes from the text-based paradigm specifically or from higher trajectory complexity generally. GEM trajectories are substantially longer and more tool-call-dense than any baseline (46.1 messages, 16.3 tool calls vs. TOUCAN's 6.24 messages and APIGEN-MT's 4.3 tool calls). An additional baseline training on equivalently complex trajectories from a simulation-based approach would isolate the text-based paradigm's contribution, but no such baseline exists. The improvement over baselines is therefore a joint effect of data source and data complexity, not attributable to either factor alone.

Claim 2: "Qwen3-32B-GEM attains an accuracy of 44.88%... outperforms proprietary large-scale models, including GPT-4.1 (38.88%) and DeepSeek-V3.2-Exp (37.38%)" (Section 4.2). This claim is supported with qualifications. The comparison is asymmetric: Qwen3-32B is fine-tuned on GEM data specifically for tool-use tasks, while the proprietary models are evaluated off-the-shelf (presumably without task-specific fine-tuning on tool-use data). The proprietary models may not have been optimized for BFCL V3's specific evaluation format or interaction patterns. The comparison demonstrates that a 32B open model + domain-specific fine-tuning on GEM data can exceed the zero-shot or few-shot performance of larger proprietary models on this specific benchmark—but it does not demonstrate that GEM data is superior to whatever training data the proprietary models received. A fairer comparison would be proprietary models fine-tuned on their own tool-use data vs. GEM-fine-tuned models, but this is not feasible with closed APIs.

Additionally, the specific proprietary model versions (GPT-4.1, DeepSeek-V3.2-Exp, Gemini-2.5-Flash) are not described with training cutoffs or tool-use-specific optimization details, making it unclear what capabilities are being compared. The BFCL V3 scores for these models (38.88%, 37.38%, 36.25%) are clustered within a narrow range, suggesting potential ceiling effects in the benchmark rather than clear model hierarchy.

Claim 3: "Our text-based synthesis pipeline instills a fundamental understanding of tool-use reasoning that transfers effectively to unseen, real-world domains" (Section 4.2), evidenced by τ²-bench results matching in-domain baselines. This is the paper's most important and most nuanced claim. The evidence from Figure 5 is partially supportive with important asymmetry.

On the Retail domain, GEM-trained models clearly exceed in-domain baselines at 32B scale (55.48% Avg@4 vs. APIGEN-MT's 44.52% and SIMIA's 48.03%). At 8B scale, GEM outperforms APIGEN-MT (44.52% vs. 42.54% Avg@4) and is competitive with SIMIA. This is strong evidence that text-derived trajectories teach transferable retail-domain skills.

On the Airline domain, the picture is less clear. At 32B, GEM (35.50% Avg@4) trails SIMIA (38.00%) and matches APIGEN-MT (36.00%). At 8B, GEM (22.00%) substantially trails SIMIA (35.50%) but matches APIGEN-MT (23.50%). The Airline domain—with its complex scheduling constraints, multi-leg itineraries, and coordination across multiple reservation systems—appears harder for text-derived training to cover than Retail. This may reflect genuine domain differences: retail interactions (returns, order modifications, inventory queries) appear frequently in general web text (e-commerce tutorials, customer service guides), while airline-specific workflows (fare class rules, rebooking policies, interline agreements) may be less represented in UltraFineWeb.

The paper does not acknowledge this asymmetry in its claims, presenting the results as uniformly supporting out-of-domain generalization. A more precise claim would be: text-derived trajectories enable strong out-of-domain generalization to retail domains and competitive (but not superior) generalization to airline domains, with the gap potentially reflecting domain representation in the source corpus.

Claim 4: "Our Trajectory Synthesizer matches the quality of the full pipeline while significantly reducing inference latency and costs" (Section 1, Section 4.3). The quality-matching claim is supported with qualifications. Table 2 shows GEM-Synthesizer (UltraFineWeb) achieves BFCL V3 overall accuracy of 28.38% vs. GEM-GLM's 30.25%—a gap of 1.87 percentage points. The paper presents this as "close" or "matching," but it is a measurable degradation. On τ²-bench Retail Pass@4, the synthesizer achieves 73.68% vs. 75.44% (gap of 1.76 points). These gaps are small enough that the synthesizer is clearly a viable alternative to the full pipeline, but "matches" overstates the evidence—the synthesizer is slightly worse on most metrics.

The cost-reduction claim is asserted but not quantitatively supported. The paper states the synthesizer "significantly reduces costs" and "provides a cost-effective, end-to-end solution" but provides no measurements: tokens per trajectory, GPU-hours per 1K trajectories, wall-clock time comparisons, or dollar cost estimates for the full pipeline vs. the synthesizer. The architectural argument (single forward pass vs. multiple stages with different models) makes the cost reduction plausible, but without quantitative evidence, the magnitude of savings is unknown. A trajectory from the full pipeline requires at minimum: one Qwen3-8B forward pass for filtering, one GLM-4.6 forward pass for extraction, one GLM-4.6 forward pass for generation, one GLM-4.6 forward pass for refinement, and one Qwen3-32B forward pass for hallucination checking—five LLM calls, three of which use a larger model (GLM-4.6). The synthesizer requires one Qwen3-8B forward pass. The cost reduction is likely significant, but the paper should report it.

Missing analysis: how many trajectories are needed? The paper uses exactly 10K trajectories for all experiments. No data scaling curve is presented—would 5K trajectories achieve similar performance? Would 50K trajectories yield substantial further gains? The choice of 10K appears to be arbitrary or budget-constrained rather than data-optimal. This matters for assessing the scalability claim: if trajectory quality saturates at 10K, the paradigm's value is in data efficiency; if quality continues improving with scale, the Trajectory Synthesizer's cost reduction becomes critical for generating much larger datasets.

Missing analysis: quality comparison between initial and refined trajectories as training data. The ablation (Figure 6) shows that removing refinement hurts performance, but the paper does not train models on only initial (unrefined) trajectories at a larger volume to test whether volume can compensate for per-trajectory complexity. If 30K unrefined trajectories produce performance comparable to 10K refined trajectories, the refinement stage's value would be primarily in data efficiency, not final model quality. This experiment is not run.

Missing baseline: text-extracted trajectories without tool synthesis. The paper extracts both workflows and tools from text. An alternative baseline would extract workflows from text but use a fixed, general-purpose toolset (a standard set of read/write/search tools) rather than synthesizing domain-specific APIs. This would test whether the benefit comes from the extracted procedural knowledge (workflows) or from the domain-specific tools that force diverse API interactions. The paper does not run this ablation, making it difficult to attribute gains to the text-to-tool mapping specifically versus the text-to-workflow mapping.

Benchmark limitations. BFCL V3 evaluates function-calling correctness in a Python API environment—it captures syntactic and semantic accuracy of tool invocations but may not fully capture the interaction quality dimensions (naturalness of clarification, appropriateness of error recovery) that GEM's trajectory patterns are designed to teach. τ²-bench is more holistic, evaluating end-to-end task success with a simulated user, but it covers only two domains (Airline and Retail). Neither benchmark directly tests the specific interaction patterns (constraint conflicts, multi-hop reasoning, error recovery) that GEM's refinement stage explicitly targets, so the paper's claim that these patterns improve agent robustness is indirect—inferred from benchmark improvements rather than demonstrated through pattern-specific evaluation.

Single training run per configuration. All results are reported from single fine-tuning runs. Without multiple random seeds or cross-validation, the stability of the reported improvements is unknown. The τ²-bench Avg@4 metric provides some robustness by averaging over 4 evaluation runs, but training-side variance could affect which specific trajectories are generated by the pipeline and which model checkpoint is selected.

Test set size. BFCL V3 Multi-Turn contains 800 total tasks (200 per subcategory × 4 subcategories). This is a reasonable size for detecting large effects (the 12–16 point gaps observed between GEM and baselines are well above noise level), but smaller differences (the 1.87-point gap between synthesizer and full pipeline, or per-subcategory differences of 1–2 points) may not be statistically reliable. τ²-bench test sets are not described with task counts in the paper, making it difficult to assess statistical power.

6. Limitations and Trade-offs

Difficulty Estimation Cost Dominates the Pipeline and Is Unaccounted For

The assumption or constraint. The GEM pipeline synthesizes trajectories from raw web text, but this synthesis is performed offline, once, to produce a fixed training dataset (10K trajectories). The paper does not account for the computational cost of running the full pipeline—filtering, extraction, generation, refinement, and validation—in its headline efficiency claims. The Trajectory Synthesizer is presented as reducing this cost (Section 3.2, "Generating such trajectories is costly and time-consuming"), and the paper asserts it "provides a cost-effective, end-to-end solution for large-scale data generation, matching the quality of the multi-stage synthesis pipeline while significantly reducing costs" (Section 1).

The consequence. Without quantitative cost reporting, a practitioner cannot assess whether the GEM pipeline is practical for their scale of data generation. The full pipeline requires at minimum five LLM forward passes per trajectory: Qwen3-8B for filtering, GLM-4.6 for extraction, GLM-4.6 for generation, GLM-4.6 for refinement, and Qwen3-32B for hallucination checking. Three of these passes use GLM-4.6, a model significantly larger than the 8B synthesizer. The trajectory yield rate—how many raw text segments must be processed to produce one validated trajectory—is also unreported. If the yield is low (e.g., only 1 in 20 filtered segments produces a passing trajectory after refinement and validation), the per-trajectory cost multiplies accordingly. The Trajectory Synthesizer simplifies this to one Qwen3-8B forward pass, but the paper provides no tokens-per-trajectory, GPU-hours, or dollar-cost estimates for either approach. The claim of "significantly reducing costs" is therefore qualitative and unverifiable.

What evidence exists in the paper. The paper briefly states the pipeline is "costly and time-consuming" (Section 3.2) but reports zero quantitative cost metrics. The synthesizer evaluation (Table 2, Section 4.3) measures output quality (BFCL V3 accuracy, τ²-bench scores) but not generation cost. The 10K trajectory dataset size is reported (Section 4.1) but the input volume required to produce these 10K is not. Given that only ~14% of raw text segments pass the initial filtering stage (Section 3.1), and additional segments are lost during extraction, generation, refinement, and validation, the total number of raw segments processed is likely 2–5× larger than 10K, but this is undocumented.

Mitigation status. The paper acknowledges the cost implicitly by developing the Trajectory Synthesizer as a cheaper alternative, stating in Section 1 that it "match[es] the quality of the multi-stage synthesis pipeline while significantly reducing costs." However, the paper treats cost reduction as qualitative motivation rather than a measured outcome. No ablation compares the cost of generating 10K trajectories via the full pipeline vs. the synthesizer, nor does the paper establish a cost-quality Pareto frontier that would guide practitioners in choosing between approaches.

Single Benchmark Family and Single Base Model Family Limit Generality Claims

The assumption or constraint. All experimental results are derived from fine-tuning exactly two model sizes (Qwen3-8B and Qwen3-32B) from a single model family (Qwen3) and evaluating on two benchmarks (BFCL V3 and τ²-bench). The τ²-bench evaluation covers only two domains (Airline and Retail). The paper asserts general claims: GEM "instills a fundamental understanding of tool-use reasoning that transfers effectively to unseen, real-world domains" (Section 4.2) and that the Trajectory Synthesizer "generaliz[es] across corpora" (Section 4.3). The UltraFineWeb corpus is the sole source for pipeline-generated trajectories; WikiHow is used only for the Trajectory Synthesizer generalization test.

The consequence. A practitioner cannot determine whether GEM's benefits are specific to Qwen-family models or transfer to other architectures (LLaMA, Mistral, DeepSeek, etc.). Different model families have different pretraining data mixtures, which could affect how well they absorb text-derived trajectories. Models pretrained on code-heavy corpora might already possess strong function-calling priors, making GEM data less impactful; models pretrained primarily on narrative text might benefit more. Similarly, the two τ²-bench domains may not represent the full range of deployment scenarios. Retail and airline are both structured customer-service domains with clear success criteria—GEM's effectiveness on open-ended creative tasks, technical support, or domains lacking clean task-completion signals is entirely untested. The paper's claim of "fundamental understanding of tool-use reasoning" cannot be separated from the specific evaluation domains tested.

What evidence exists in the paper. All quantitative results in Tables 1, 2, 5, and 6 use Qwen3 models exclusively. The paper states in Section 4.1 that it employs "GPT-4.1 as the user simulator" for τ²-bench without testing other user simulators that might interact differently with GEM-trained agents. The WikiHow experiment (Table 2) demonstrates cross-corpus generalization for the synthesizer but still uses Qwen3-8B as the fine-tuned agent model. The BFCL V3 results in Table 1 include proprietary model baselines (GPT-4.1, DeepSeek-V3.2-Exp, Gemini-2.5-Flash), but these are evaluated off-the-shelf without GEM fine-tuning, so they serve only as performance references, not as tests of GEM's model-agnostic effectiveness. The paper acknowledges in Section 4.2 that SIMIA was excluded from BFCL V3 "as we observed its scores to be generally low, likely due to its reliance solely on in-domain τ-bench data"—this acknowledges a domain-specific limitation of a baseline but does not test whether GEM suffers from analogous restrictions in other domains.

Mitigation status. The paper does not address this limitation explicitly. The two-scale experiment (8B and 32B) provides some evidence that GEM data scales with model size—the 32B model gains more absolute improvement from GEM data than the 8B model (16.53 vs. 12.25 percentage points on BFCL V3 overall, per Table 1)—but this is within-family scaling, not cross-family generalization. The paper does not discuss model family as a variable, propose multi-family experiments, or qualify its generalization claims with caveats about model specificity.

Refinement Is Both Essential and Unexplained—A 12-Point Black Box

The assumption or constraint. The refinement stage (Stage 4, Section 3.2) is the single largest contributor to GEM's performance, responsible for over 12 percentage points of BFCL V3 accuracy on the 32B model (Figure 6: 32.50% without refinement vs. 44.88% with refinement). The mechanism of refinement is described only at the level of the prompt given to the teacher model (Appendix A.4)—it instructs GLM-4.6 to increase complexity, add pitfalls, expand toolsets, define database schemas, and enhance interaction patterns. But the paper provides no analysis of what specific changes the refinement stage makes that drive its outsized impact.

The consequence. Without understanding why refinement works, practitioners cannot optimize it or determine whether simpler alternatives would suffice. Several hypotheses are plausible but untested: (1) refinement increases trajectory length and tool-call density, and the benefit is purely from more training tokens; (2) refinement introduces specific interaction patterns (error recovery, constraint conflicts) that the initial generation misses, and these patterns are the active ingredient; (3) refinement improves the quality of tool responses (more realistic errors, structured outputs) rather than trajectory structure; (4) refinement eliminates subtle correctness issues that the validation stage doesn't catch. Each hypothesis implies a different optimization strategy. If length is the key factor, generating more initial trajectories might be more cost-effective than refining a smaller set. If specific patterns matter, targeted prompting to include those patterns in initial generation might obviate refinement. The paper's ablation shows that refinement matters enormously but not which aspects of refinement matter, leaving the practitioner to either replicate the full refinement prompt exactly or guess at simplifications.

What evidence exists in the paper. Table 5 shows that refinement approximately doubles all complexity metrics: messages increase from 30.05 to 46.1, tools from 5.01 to 8.6, and tool calls from 7.83 to 16.3. This demonstrates that refinement makes trajectories structurally more complex, but it does not isolate which complexity dimension(s) drive performance. The case study (Figure 8) shows a refined trajectory that includes clarification, correct tool sequences, rule adherence, and error recovery—but it is a single illustrative example, not a systematic analysis of how refinement changes trajectory content across the dataset. The hallucination check ablation (Figure 6) shows a smaller effect (0.63 points for 32B) than refinement (12.38 points), suggesting that correctness filtering alone cannot substitute for refinement's contribution—but the mechanism remains opaque.

Mitigation status. The paper does not analyze the refinement mechanism beyond the aggregate metrics in Table 5. Section 4.4 states that "even the original trajectories extracted directly from the original text (though relatively simpler) still provide valuable training signals and contribute to improved tool-calling capability," but this is an observation about the initial trajectories, not an analysis of refinement. The paper suggests in Section 4.4 that "more effectively leveraging information from the original text to synthesize high-quality tool-calling trajectories is a promising research direction," which implicitly acknowledges that the current refinement approach is a specific (and likely suboptimal) instantiation rather than a solved design.

Hard Problems: The Pipeline's Performance Ceiling on Structurally Complex Domains

The assumption or constraint. The GEM pipeline extracts procedural knowledge from web text and converts it into tool-use trajectories. This paradigm assumes that the procedural knowledge required for a target domain is present in the source corpus—either explicitly (tutorials, guides, documentation) or implicitly (patterns of reasoning that transfer across domains). When this assumption fails, GEM-trained models degrade relative to in-domain approaches.

The consequence. On the Airline domain of τ²-bench, GEM-trained models trail the in-domain SIMIA baseline significantly at the 8B scale (22.00% vs. 35.50% Avg@4, Table 5) and are competitive but not superior at the 32B scale (35.50% vs. 38.00%). The Airline domain involves specific reasoning patterns—multi-leg itinerary optimization, fare class rules, rebooking policies with change fees, interline agreements—that may not appear frequently in general web text. A practitioner deploying GEM for a specialized domain (healthcare scheduling, legal procedure automation, financial compliance) cannot assume that UltraFineWeb or similar general corpora contain sufficient domain-specific procedural knowledge to produce effective training trajectories. The paradigm's scalability advantage over predefined-API approaches rests on the breadth of the source corpus, but breadth does not guarantee depth in any particular domain.

The paper also notes that the hardest difficulty tier produces no improvement in the prior sections, but the analogy here is different: it's not that some problems are inherently too hard for any agent, but that the text corpus may lack coverage for specific problem types. This is a coverage ceiling rather than a capability ceiling—the base model could potentially learn the domain if appropriate training data existed, but GEM cannot synthesize it from unavailable source material.

What evidence exists in the paper. Table 5 and Figure 5 show the Airline-Retail asymmetry. Qwen3-32B-GEM achieves 55.48% Avg@4 on Retail (exceeding APIGEN-MT by 10.96 points and SIMIA by 7.45 points) but only 35.50% on Airline (trailing SIMIA by 2.50 points). At 8B, the gap is stark: Retail Avg@4 is 44.52% (competitive with SIMIA at 43.20%) but Airline Avg@4 is 22.00% (vs. SIMIA at 35.50%). The paper does not analyze this asymmetry, treating both domains as evidence of out-of-domain generalization. The domain distribution analysis in Figure 2 and Appendix E (Figure 9) shows that the source corpus covers domains like "Computers & Electronics," "Science," and "Shopping" in large volumes, but specialized transportation/logistics categories are not broken out separately. Without knowing the corpus's airline-specific content volume, the asymmetry's cause—corpus coverage vs. domain complexity—cannot be determined.

Mitigation status. The paper does not acknowledge the Airline-Retail performance gap as a limitation. Section 4.2 presents the τ²-bench results as uniformly demonstrating "that our text-based synthesis pipeline instills a fundamental understanding of tool-use reasoning that transfers effectively to unseen, real-world domains" without discussing domain-specific performance variation. The conclusion (Section 6) similarly makes unqualified claims about "the potential of leveraging open-world textual knowledge as a scalable source for advancing autonomous agents." No analysis is offered of which domains UltraFineWeb covers well vs. poorly, and no guidance is provided for practitioners assessing whether their target domain is likely to benefit from text-derived trajectories.

The In-Domain Baseline Comparison Privileges GEM and Does Not Isolate the Text-Based Paradigm

The assumption or constraint. The paper's headline τ²-bench result—that GEM-trained models match or exceed models trained on in-domain τ-bench data (APIGEN-MT, SIMIA)—is used as evidence that text-derived trajectories teach more generalizable tool-use capabilities than API-centric simulation (Section 4.2). This comparison assumes rough equivalence between the training data being compared: similar trajectory volume, similar trajectory complexity, differing primarily in data source (text-derived vs. simulation-derived).

The consequence. The comparison is confounded by unequated variables. GEM trajectories average 46.1 messages and 16.3 tool calls per trajectory (Section 4.5). APIGEN-MT trajectories average approximately 18.5 turns and 4.3 tool calls. TOUCAN trajectories average 6.24 turns. The GEM trajectories are 2.5× longer and 3.8× more tool-call-dense than the baseline datasets they're compared against. Performance differences could result from trajectory complexity rather than the text-based extraction paradigm. A practitioner choosing between approaches cannot determine whether to invest in (a) text-based extraction with its expensive refinement pipeline or (b) improving simulation-based data to match GEM's complexity metrics. The comparison tests "GEM data vs. existing simulation datasets" but not "text-based extraction vs. simulation-based extraction at matched complexity."

The training data volume is also unequated. GEM uses 10K trajectories. APIGEN-MT's dataset size is not reported in the paper; SIMIA and TOUCAN are sampled at 50K multi-turn instances. If APIGEN-MT's full dataset is larger than 10K, GEM's performance parity at lower data volume would be evidence of data efficiency, but the paper does not standardize training data volume across baselines. Conversely, if SIMIA's 50K sample is needed to achieve its reported performance, GEM's 10K trajectories are dramatically more data-efficient—but this interpretation is not explored.

What evidence exists in the paper. Section 4.5 provides complexity statistics for GEM trajectories. Section 4.1 reports that TOUCAN and SIMIA are sampled at 50K instances but does not report APIGEN-MT's training data volume. The paper compares models fine-tuned on these datasets as if they were equivalent training configurations. Nowhere does the paper run a controlled experiment where simulation-based trajectories are generated at matched complexity (similar length, tool-call density, interaction pattern diversity) to isolate the text-based paradigm's contribution. The BFCL V3 comparison (Table 1) suffers from the same confound: APIGEN-MT and TOUCAN trajectories are substantially simpler than GEM trajectories, so performance gaps may reflect complexity differences rather than data source differences.

Mitigation status. The paper does not address this confound. Section 4.2 states that GEM demonstrates "clear improvements over baseline models" without qualifying that those baselines use structurally different training data. Section 5 positions the work against prior approaches by emphasizing the novel data source ("Unlike prior works that rely on pre-defined tools") rather than acknowledging that the data processing (complexity-targeted refinement) may be as important as the data source. The ablation in Figure 6 showing a 12-point drop without refinement provides indirect evidence that complexity matters enormously, but it does not test whether equivalently complex simulation-based data would close or reverse the gap.

Validation Filtering May Be Overly Aggressive and Counterproductive at Scale

The assumption or constraint. The hallucination detection stage (Stage 4 validation, Section 3.2) uses a binary rubric: trajectories must score 1 on all three hallucination categories (R1: tool-call hallucination, R2: capability hallucination, R3: context hallucination) to be retained. The judge model (Qwen3-32B) applies a strict criterion: "If any single round does not meet (the condition), the corresponding rubric should be scored as 0" (Appendix A.5). This all-or-nothing filtering assumes that any trajectory containing any hallucination—even a single parameter value not traceable to dialogue context—is harmful for training and should be discarded entirely.

The consequence. The counterintuitive τ²-bench ablation result in Table 4—where removing the LLM-based check increases Qwen3-32B Retail Avg@4 from 55.48% to 56.80% (+1.32 points)—suggests the filter may be discarding trajectories that, despite containing superficial hallucinations, provide valuable training diversity. A trajectory might have one fabricated parameter value (failing R1) but otherwise demonstrate excellent error recovery, multi-hop reasoning, and constraint handling—patterns that would benefit the fine-tuned model. By discarding the entire trajectory for a single violation, the filter potentially reduces the dataset's pattern diversity and eliminates edge cases that stress-test the model's robustness.

At larger scale, this tradeoff becomes more consequential. If the full pipeline were used to generate 100K trajectories and the hallucination filter rejected 30% of them (the paper does not report rejection rates), 30K trajectories would be discarded—representing substantial wasted computation from the filtering, extraction, generation, and refinement stages. A less aggressive filter (e.g., scoring trajectories on a continuous quality scale and training on all but the worst) might yield better overall model performance by retaining partially-flawed-but-informative trajectories. The paper's design choice of binary filtering optimizes for training data purity but potentially sacrifices both diversity and computational efficiency.

What evidence exists in the paper. Table 4 shows the asymmetric ablation results: removing the LLM-based check helps Qwen3-32B on Retail Avg@4 (55.48% → 56.80%, +1.32 points) while hurting Qwen3-8B on Retail Pass@4 (75.44% → 71.05%, −4.39 points). On BFCL V3 (Figure 6), removing the check hurts Qwen3-8B substantially (30.25% → 27.38%, −2.87 points) but Qwen3-32B minimally (44.88% → 44.25%, −0.63 points). This pattern suggests that larger models are more robust to training data imperfections and may benefit from retaining borderline trajectories that the filter would discard. The paper does not analyze these asymmetric effects or discuss the filter's rejection rate. The hallucination detection prompt (Appendix A.5) is presented as the rubric, but no examples of accepted vs. rejected trajectories are shown, making it impossible for a practitioner to calibrate whether the filter's strictness is appropriate for their use case.

Mitigation status. The paper does not address the potential over-aggressiveness of the hallucination filter. The ablation is reported as validating the filter's contribution (Section 4.4: "This stage consistently improves results by filtering out samples with hallucinations or inconsistencies"), but the 32B Retail result in Table 4 directly contradicts "consistently improves." The paper does not discuss this contradiction, explore alternative filtering strategies (continuous scoring, partial retention, targeted hallucination correction), or provide guidance on when strict filtering is beneficial vs. harmful. The Trajectory Synthesizer is trained on the full pipeline's filtered output, so it inherits whatever bias the filter introduces without the possibility of recovering discarded trajectory patterns.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a paradigm-level reframing of where agent training data comes from. The dominant assumption in the multi-turn tool-use literature—spanning ToolBench, APIGEN-MT, TOUCAN, MAGNET, ToolACE-MT, and MUA—has been that synthesizing tool-use trajectories requires starting from a predefined set of APIs and simulating interactions within that environment. This assumption is so deeply embedded that it is rarely questioned: if you want an agent to learn tool use, you must first collect APIs, then generate tasks solvable with those APIs, then simulate dialogues. GEM challenges this assumption at its root by asking a different question: what if the procedural knowledge needed for tool-use training already exists in the unstructured text that LLMs are pretrained on, and the challenge is extraction rather than simulation?

The magnitude of this reframing is fundamental, not incremental. It changes what counts as a legitimate data source for a whole subfield. The paper does not propose a better simulation pipeline—it proposes that simulation pipelines are solving the wrong problem. The bottleneck is not API collection; it is procedural knowledge extraction. Text corpora contain millions of documented human problem-solving experiences—tutorials, how-to guides, troubleshooting forums, procedural documentation—that encode exactly the interaction patterns (clarification, error recovery, constraint enforcement, multi-hop reasoning) that agents need to learn. The preliminary analysis (Section 3.1) quantifies this claim: 14% of randomly sampled UltraFineWeb segments contain explicit multi-step operational procedures spanning dozens of domains. This 14% represents an enormous absolute volume of training data that has never before been systematically converted into agent trajectories.

The reframing has a specific conceptual consequence: it dissolves the boundary between "pretraining data" (text corpora, used for language understanding) and "agent training data" (tool-use trajectories, used for tool-use capability). In the API-centric paradigm, these are distinct data types with distinct generation pipelines. GEM demonstrates that they are two views of the same underlying knowledge—the procedural expertise embedded in human writing. A tutorial about creating music visualizers in Adobe After Effects is simultaneously natural language (useful for pretraining) and a latent multi-turn tool-use trajectory (useful for agent fine-tuning). This unification has practical implications: the same web-scale corpora that drive pretraining improvements can also drive agent training improvements, with no API collection bottleneck.

The paper also reconciles a latent tension in the agent training literature. Prior work on multi-turn tool-use data synthesis consistently reported improvements over single-turn baselines, but the improvements came with an uncomfortable caveat: the training data was generated within the same API environment (or a closely related one) as the evaluation benchmark. This raised the question of whether the models were learning general tool-use reasoning or merely memorizing API-specific patterns. GEM provides the strongest evidence to date for the former interpretation. Qwen3-32B-GEM, trained on trajectories synthesized from general web text with zero exposure to airline or retail APIs, achieves 55.48% Avg@4 on τ²-bench Retail—exceeding APIGEN-MT (44.52%) and SIMIA (48.03%), both of which were trained on in-domain τ-bench synthetic data (Figure 5). This out-of-domain generalization is the paper's most important empirical contribution: it demonstrates that text-derived trajectories teach transferable tool-use capabilities, not narrow API-matching heuristics.

The landscape implications extend to how the field should think about scalability. The API-centric paradigm has an inherent scalability ceiling: the diversity of training data is bounded by the diversity of APIs you can collect, document, and maintain. Collecting a thousand diverse APIs is a major engineering effort; collecting a hundred thousand is infeasible. The text-centric paradigm replaces this ceiling with the effectively unbounded scale of web text. If 14% of UltraFineWeb contains usable procedural content, and UltraFineWeb itself contains billions of tokens, the potential training data pool is massive and growing with web-scale corpora. This scalability argument does not rely on GEM's specific extraction quality—even if only a fraction of that 14% can be converted into high-quality trajectories, the absolute volume still dwarfs what API-centric simulation can produce.

However, the paper's own results establish an important boundary condition on this landscape shift. The ablation (Figure 6) showing a 12.38 percentage point accuracy drop on BFCL V3 when refinement is removed (Qwen3-32B: 44.88% → 32.50%) demonstrates that raw text-extracted trajectories are not immediately sufficient. The paradigm provides the data source; the refinement stage provides the data quality. Extracting procedural knowledge from text and converting it into maximally informative training trajectories requires deliberate engineering—complexity-targeted rewriting, toolset expansion, interaction pattern enforcement, hallucination filtering. The landscape has shifted from "how do we collect enough APIs?" to "how do we extract and refine procedural knowledge from text at quality and scale?" This is a different research problem with different bottlenecks.

The paper also redirects research attention away from increasingly sophisticated multi-agent simulation architectures. If a single-pass generation followed by complexity-targeted refinement can produce trajectories that rival or exceed simulation-based approaches on out-of-domain benchmarks, the marginal value of more realistic simulation (with turn-by-turn user-agent-environment interaction) becomes questionable. The efficiency argument is stark: GEM's single-pass generation produces an entire 46-message, 16-tool-call trajectory in one forward pass, while multi-agent simulation requires an LLM call for every user turn, assistant turn, and tool response—for an equivalent trajectory, that could be 40+ separate LLM calls. The paper does not directly compare single-pass against multi-agent quality, but the strong results suggest that for training data generation, the efficiency-quality Pareto frontier favors single-pass approaches with post-hoc refinement over iterative simulation.

Follow-Up Research This Work Enables

Scaling curves for text-derived training data: how many trajectories are needed, and where does performance saturate? The paper uses exactly 10K trajectories for all experiments—a single, unreplicated data point on what is presumably a scaling curve. This leaves open the most natural follow-up question: if 10K text-derived trajectories produce a 16.5 percentage point improvement on BFCL V3 for Qwen3-32B (28.35% → 44.88%), what would 50K trajectories achieve? 100K? Is there a regime where additional text-derived data produces diminishing returns, or does performance continue to improve with corpus scale? The Trajectory Synthesizer makes this experiment newly tractable: generating 100K trajectories with the full pipeline would be prohibitively expensive, but the synthesizer can produce them at roughly the cost of a single 8B model forward pass per trajectory. A strong follow-up would generate trajectories at logarithmically spaced volumes (1K, 2K, 5K, 10K, 20K, 50K, 100K), fine-tune Qwen3-8B and Qwen3-32B at each volume, and fit a scaling law relating trajectory count to downstream accuracy on both BFCL V3 and τ²-bench. This would answer whether the paradigm's value is primarily in data efficiency (good results from small datasets) or in scalability (continuing improvements from larger datasets), and would give practitioners concrete guidance on data generation budgets.

What specific properties of refinement drive the 12-point accuracy gain? The ablation in Figure 6 shows that refinement contributes over 12 percentage points to Qwen3-32B's BFCL V3 accuracy, and Table 5 shows that refinement doubles all complexity metrics (messages, tools, tool calls). But complexity is not a unitary variable—it bundles together trajectory length, tool-call density, interaction pattern diversity, constraint complexity, and tool response realism. A targeted follow-up would generate ablated variants of the refinement stage, each targeting a single complexity dimension: (a) refinement that only extends trajectories (more messages, same tool diversity, no new interaction patterns), (b) refinement that only increases tool variety (more distinct tools, same trajectory length), (c) refinement that only adds interaction patterns (clarification, error recovery, constraint conflicts without increasing tool count or length), and (d) refinement that only improves tool response realism (structured nested outputs, non-trivial error messages). Fine-tuning models on each variant and measuring per-category BFCL V3 performance would decompose the refinement benefit. If trajectory length alone accounts for most of the gain, the paradigm simplifies dramatically—generate longer initial trajectories rather than refining them. If interaction pattern diversity is the active ingredient, future work should focus on extraction prompts that surface these patterns directly rather than relying on post-hoc refinement.

Combining text-derived and simulation-derived trajectories: complementary or redundant? The paper demonstrates that text-derived trajectories enable out-of-domain generalization to τ²-bench, while simulation-derived trajectories (APIGEN-MT, SIMIA) provide in-domain expertise. These data sources are tested independently. A natural follow-up tests whether they are complementary: fine-tune a model on a mixture of GEM trajectories (general procedural knowledge from diverse domains) and in-domain simulation trajectories (domain-specific API expertise for Airline or Retail), and measure whether the combination exceeds either source alone. If the skills are complementary—GEM teaches general tool-use reasoning (clarification, error recovery, constraint handling) while simulation teaches domain-specific API patterns—the combination should outperform either source alone, particularly on the structurally complex Airline domain where GEM trails SIMIA at the 8B scale (22.00% vs. 35.50% Avg@4, Table 5). If the skills are redundant, performance will saturate at the level of the stronger individual source. This experiment would clarify whether the text-based paradigm should replace or complement simulation-based approaches.

Corpus coverage analysis: which domains does UltraFineWeb support well, and where does text-derived training fail? The paper's τ²-bench results reveal a domain asymmetry: GEM-trained models strongly exceed in-domain baselines on Retail but trail on Airline (particularly at 8B). The paper does not analyze whether this reflects corpus coverage (UltraFineWeb contains more retail-related procedural text than airline-related text) or domain complexity (airline tasks are inherently harder to learn from general procedural patterns). A follow-up would perform a systematic domain coverage analysis: sample procedural text segments from UltraFineWeb at large scale (e.g., 1M segments), use the Stage 1 classifier to identify those with multi-step operations, and categorize them by domain using the annotation taxonomy from Appendix A.1. Then correlate per-domain corpus frequency with per-domain τ²-bench or BFCL V3 performance. If Airline performance is low because airline-related text is rare in UltraFineWeb, the fix is corpus supplementation—add airline-specific documentation, travel forums, or booking tutorials to the source corpus before extraction. If Airline performance is low despite adequate corpus coverage, the limitation is in the extraction pipeline's ability to handle domain-specific reasoning patterns, requiring domain-adapted extraction prompts or refinement strategies. This analysis would transform the paradigm from "extract from whatever text is available" to "curate the source corpus for target domain coverage."

Iterative self-improvement: can the Trajectory Synthesizer be improved by training on its own high-quality outputs? The Trajectory Synthesizer is trained once on the full pipeline's 10K outputs and evaluated statically (Table 2). But the synthesizer's outputs are themselves evaluated through downstream agent performance. This opens a self-improvement loop: (1) generate trajectories with the synthesizer, (2) fine-tune an agent on those trajectories, (3) evaluate the agent on BFCL V3 and τ²-bench, (4) select the highest-performing trajectories (e.g., those where the agent's tool-call accuracy was highest), (5) add those trajectories to the synthesizer's training set, retrain, and repeat. This is analogous to rejection sampling fine-tuning or STaR-style self-improvement, but applied to the data generator rather than the agent. The key question is whether the synthesizer can learn to produce higher-quality trajectories by observing which of its outputs produce better agents—essentially learning a quality function over the trajectory space without explicit quality annotations. The paper's finding that the synthesizer generalizes to WikiHow (Table 2) suggests it has learned a robust text-to-trajectory mapping; self-improvement could further boost quality while maintaining the efficiency advantage over the full pipeline.

Stress test on adversarial or out-of-distribution text: does the synthesizer hallucinate when given non-procedural input? The full pipeline's Stage 1 explicitly filters out non-procedural text before extraction and generation. The Trajectory Synthesizer receives no such filter—it learns to map from text to trajectory end-to-end, and must implicitly determine whether the input contains usable procedural content. A stress-test would feed the synthesizer deliberately non-procedural text—narrative fiction, opinion essays, sports commentary, conversational dialogue—and measure whether it (a) correctly produces minimal or empty outputs, (b) hallucinates tool-use trajectories from non-procedural content, or (c) produces degraded trajectories that would poison downstream training. If the synthesizer hallucinates aggressively on non-procedural input, the filtering stage remains essential and the synthesizer's end-to-end claim is weakened—it can only safely process pre-filtered text. If the synthesizer correctly handles non-procedural input, it has learned an implicit procedural-content detector as part of the text-to-trajectory mapping, making it a true end-to-end replacement for the full pipeline. This experiment would also reveal the synthesizer's failure modes, guiding robustness improvements (adversarial training on non-procedural text, explicit procedural-content classification as an auxiliary training objective).

Practical Applications and Downstream Use Cases

Rapid agent prototyping for niche domains without API documentation. A startup building a customer-support agent for a specialized industry—medical equipment troubleshooting, agricultural supply chain management, legal document processing—typically faces a cold-start problem: there is no existing API specification or simulation environment for their domain. The conventional approach requires manually defining APIs, writing task templates, and simulating interactions—a months-long engineering effort before any training data exists. With GEM, the startup can feed domain-specific documentation (user manuals, troubleshooting guides, procedural SOPs, forum discussions) directly into the Trajectory Synthesizer and produce thousands of training trajectories in hours. The paper's results suggest these trajectories will teach general tool-use reasoning (clarification, error recovery, constraint enforcement) that transfers to the domain's specific APIs. The BFCL V3 improvement of 16.5 points over the base model (Table 1) provides a quantitative anchor for expected gains: a base model with no domain-specific training can gain substantial tool-use capability from text-derived trajectories alone, even before any in-domain fine-tuning on real interactions. For a startup with limited resources, this dramatically lowers the barrier to building a competent domain-specific agent.

Data augmentation for existing simulation pipelines. Organizations that already have simulation-based data generation pipelines (using APIGEN-MT, TOUCAN, or custom multi-agent simulators) face a quality-diversity tradeoff: simulation produces domain-specific trajectories with correct tool calls, but the diversity is bounded by the predefined task templates and API environments. GEM trajectories can serve as an augmentation dataset—added to the simulation-derived training mix to increase domain diversity and introduce interaction patterns (ambiguous requests, constraint conflicts, non-trivial error recovery) that template-based simulation may underrepresent. The paper's complexity statistics (Section 4.5: 46.1 messages per dialogue, 16.3 tool calls, 8.6 distinct tools) demonstrate that GEM trajectories are substantially richer than existing datasets (APIGEN-MT: 18.5 turns, 4.3 tool calls; TOUCAN: 6.24 turns). A practitioner could blend 10K simulation trajectories with 10K GEM trajectories and expect improvements on both in-domain metrics (from simulation data) and out-of-domain generalization (from GEM data), with the τ²-bench results (Figure 5) providing evidence that GEM data does not interfere with domain-specific performance—Qwen3-32B-GEM matches or exceeds APIGEN-MT on τ²-bench Retail despite having no retail-specific training.

Cost-effective large-scale training data generation for open-source agent models. The open-source LLM community (models like Qwen, LLaMA, Mistral) typically lacks access to the proprietary tool-use training data that powers commercial agents (GPT-4, Claude, Gemini). GEM's Trajectory Synthesizer provides a path to generating large-scale, high-quality agent training data at manageable cost. A single Qwen3-8B Trajectory Synthesizer forward pass replaces the five LLM calls of the full pipeline, making it feasible to generate 100K+ trajectories from web-scale corpora. The synthesizer's cross-corpus generalization (Table 2: BFCL V3 accuracy of 28.50% on WikiHow vs. 28.38% on UltraFineWeb) means the same trained synthesizer can process diverse text sources without retraining. An open-source project could run the synthesizer over CommonCrawl dumps, filter for procedural content, and produce millions of training trajectories—creating a public-domain agent training dataset that rivals proprietary data in diversity and complexity. The paper's results provide a performance floor: even 10K trajectories from a single corpus produce a 16.5-point BFCL V3 improvement over the base model. Scaling to 100K+ trajectories from diverse corpora is likely to further improve performance, though the scaling curve remains to be measured.

Bootstrapping agents for low-resource languages and domains. The text-based paradigm is inherently language-agnostic: if a language has web text containing procedural descriptions, GEM can extract trajectories from it. For languages where API documentation and simulation environments are scarce (most languages other than English and Chinese), the API-centric paradigm is essentially unusable—there are no predefined APIs to simulate against. GEM bypasses this barrier. A practitioner building a tool-use agent for Thai, Swahili, or Finnish can feed native-language how-to guides, technical documentation, and procedural forum posts into the Trajectory Synthesizer (or a translated version of the English-trained synthesizer, if cross-lingual transfer works) and produce training trajectories without any language-specific API engineering. The paper does not test this directly—all experiments use English text—but the paradigm's logic extends naturally: procedural knowledge encoded in any language's text is extractable. A follow-up experiment training the Trajectory Synthesizer on multilingual corpora and evaluating on multilingual BFCL V3 equivalents would quantify cross-lingual transfer, but the conceptual path is clear.

When to Prefer This Method

The paper does not explicitly position GEM against named alternatives with a structured decision rule. It presents the text-based paradigm as a new capability—extracting trajectories from text—rather than as a replacement for simulation-based approaches that should be preferred under specific conditions. The τ²-bench comparison (Figure 5) shows GEM-trained models matching or exceeding simulation-trained models on out-of-domain evaluation, but the paper does not claim GEM should replace simulation for in-domain use cases. The Trajectory Synthesizer is presented as a cost-reducing alternative to the full pipeline, not as a method to prefer over simulation.

A practitioner reading the paper can infer the following approximate guidance from the results, though the paper itself does not state it:

• Prefer text-based extraction (GEM) when the target domain lacks comprehensive API documentation or simulation environments, when out-of-domain generalization to diverse unseen tools is the primary objective (as in general-purpose agent training), or when training data volume and domain diversity are the bottlenecks (the 14% procedural content rate in web text provides effectively unlimited scale). The BFCL V3 results (44.88% overall for 32B, exceeding all open-source baselines and proprietary models, Table 1) and the τ²-bench Retail results (55.48% Avg@4, exceeding in-domain baselines, Figure 5) provide the quantitative evidence.

• Prefer simulation-based approaches (APIGEN-MT, SIMIA, TOUCAN) when maximum in-domain performance on a specific, well-documented API environment is required and the cost of API collection and simulation infrastructure is acceptable. The Airline domain results (Table 5) show SIMIA achieving 38.00% Avg@4 vs. GEM's 35.50% at 32B and 35.50% vs. 22.00% at 8B, suggesting simulation data provides an edge in structurally complex domains with specialized reasoning patterns.

• Prefer the Trajectory Synthesizer over the full GEM pipeline when generation throughput and cost are primary concerns—the synthesizer produces trajectories in one 8B forward pass vs. the full pipeline's five LLM calls—or when generating trajectories from diverse text corpora at scale (the WikiHow results in Table 2 demonstrate cross-corpus generalization with negligible quality degradation). The full pipeline should be preferred when maximum per-trajectory quality is critical and generation cost is secondary, or when the pipeline's intermediate artifacts (abstract workflows, initial trajectories) are valuable for inspection and debugging.