Towards General Agentic Intelligence via Environment Scaling

ArXiv: 2509.13311

🎯 Pitch

Training language agents on a massive, automated diversity of 30,000 APIs organized into over 1,000 domains yields performance rivaling trillion-parameter models—yet their accuracy still linearly degrades as the number of required tool calls increases.

1. Executive Summary

This paper proposes a scalable framework for advancing general agentic intelligence by systematically constructing heterogeneous, fully simulated tool-use environments and learning agentic capabilities from the resulting interaction trajectories. Using a corpus of over 30,000 real-world APIs organized into more than 1,000 domains via Louvain community detection, the authors programmatically materialize tools as executable code grounded in read–write databases, then generate verifiable agentic tasks through simulated agent–human interplay — addressing both the diversity of function-calling scenarios and the reliability of supervision signals. The framework employs a two-phase agent fine-tuning strategy: first endowing agents with fundamental tool-calling capabilities across general domains, then specializing them for domain-specific contexts. The resulting model family, AgentScaler (4B, 8B, 30B-A3B), achieves state-of-the-art performance among open-source models under 1T parameters on τ-bench, τ2-Bench, and ACEBench, with the 30B-A3B model delivering results comparable to trillion-parameter and closed-source systems. A systematic analysis of long-horizon tool calling reveals a clear negative correlation between the number of tool calls and task accuracy, establishing that extended tool-use chains remain a fundamental challenge even for models trained with extensive environment scaling.

2. Context and Motivation

The Core Problem: Function-Calling Intelligence Is Bottlenecked by Training Data Scarcity

This paper tackles a specific, practical bottleneck in building capable language agents: the scarcity of high-quality agentic training data. By "agentic data," the authors mean something very specific — not just text corpora or human-written dialogs, but trajectories generated by autonomous agents interacting with environments through explicit function calls (tool invocations). Each trajectory consists of alternating turns of user instructions, assistant reasoning, function calls, tool responses, and natural-language replies. This is the kind of data that teaches a model when to invoke a tool, which tool to use, how to parameterize the call, and how to integrate the tool's output into a coherent response.

The importance of this problem stems from a fundamental asymmetry in how language agents learn compared to how they are evaluated. In deployment, an agent's competence is measured by its ability to navigate real-world APIs — flight booking systems, retail databases, telecom service platforms — where each tool invocation must be correct in both selection and argumentation, and where mistakes compound across multi-turn interactions. But producing the training data for this competence requires the agent to actually interact with these environments, which is expensive, slow, and difficult to supervise at scale. As the authors put it in Section 1:

"language agents must experience these interactions themselves in a predefined environment, which makes both data collection and reliable supervision highly challenging."

This is not merely an inconvenience — it is a structural limitation on how far function-calling capabilities can advance. Without a way to generate diverse, verifiable interaction trajectories at scale, models are constrained by whatever limited human-curated data exists for a given domain, and the breadth of their tool-use competence is directly tied to the narrowness of their training distribution.

Why This Matters: The Gap Between Research and Deployment

The practical significance of this problem becomes clear when considering what real-world deployment demands of a language agent. A flight-booking agent, for instance, must handle not just the obvious tool calls (search_flights, book_flight) but also edge cases: authenticating users, looking up stored preferences, handling cancellations, resolving conflicts when a selected seat is no longer available, and escalating to a human when the tools cannot satisfy the user's request. Each of these sub-tasks involves different tool compositions, different argumentation patterns, and different conversational dynamics.

The τ-bench and τ2-Bench benchmarks used in this paper's evaluation (Yao et al., 2024; Barres et al., 2025) are designed precisely to test this kind of multi-turn, tool-mediated interaction in the domains of retail, airline, and telecom. The ACEBench benchmark (Chen et al., 2025) further stratifies difficulty into Normal, Special, and Agent categories, with the Agent category specifically targeting complex multi-step tool compositions. Performance on these benchmarks reveals that even the largest models struggle with extended tool-use chains — a finding the paper itself confirms in Section 5, where a clear negative correlation appears between the number of tool calls in a trajectory and task accuracy.

The theoretical significance of the paper's approach goes beyond benchmark scores. The authors frame function calling through a unifying abstraction (Section 2): every tool invocation can be understood as a read–write operation over an underlying environmental database. This abstraction is powerful because it transforms the problem of training an agent from one of collecting heterogeneous interaction traces into one of constructing the environments that generate those traces programmatically. If tools are executable code grounded in a database state, then the environment can be fully simulated — tool responses are deterministic given the state, state transitions are verifiable, and the entire interaction can be validated against ground-truth database states without human judgment. This insight is what makes scaling feasible in principle, even if prior work had not fully realized it in practice.

Prior Approaches and Where They Fall Short

The paper identifies two broad categories of prior work on synthetic agentic data generation, both of which have fundamental limitations that motivate the proposed approach.

The Reverse Paradigm: Query Generation from Tool Calls

The first category, which the authors call the "reverse paradigm," starts with observed assistant function calls and works backward to generate matching user queries (Yin et al., 2025). The idea is straightforward: if you have examples of correct tool invocations, you can synthesize the conversational context that would naturally lead to those invocations. This approach has the advantage of guaranteed tool-call correctness — the tool calls are pre-existing, so they are valid by construction.

However, the authors identify a critical weakness: the resulting trajectories "may exhibit limited realism" (Section 1). The reason is subtle but important. When you backward-generate a user query to match a given tool call, the query tends to be unnaturally direct or transparently aligned with the tool's function — the kind of interaction where a user says "look up flight BA123" and the assistant calls get_flight_status("BA123"). Real users are messier: they provide partial information, change their minds, ask follow-up questions, and sometimes don't know which tool is appropriate. The reverse paradigm cannot capture this messiness because it starts from neat, deterministic tool executions and works backward to construct a sanitized conversational wrapper around them.

The Forward Paradigm: Simulated Agent–Human Interplay

The second category, which the paper calls the "forward paradigm" or "simulated agent–human interplay," takes the opposite approach. Here, a high-level user intent is first formulated ("the user wants to return an order and get a refund"), and then the system simulates a conversation where a user and agent interact to fulfill that intent, with the agent making tool calls and the user responding to the agent's outputs (Chen et al., 2024a; Liu et al., 2024b; Prabhakar et al., 2025a; Barres et al., 2025; Zeng et al., 2025). This produces more natural, multi-turn interactions with authentic conversational dynamics.

But this forward paradigm introduces a different — and, the authors argue, more fundamental — limitation: the environment is not scalable. The problem is that constructing the environment in which the simulated interaction takes place — the tools, their implementations, the underlying database, the rules governing state transitions — requires substantial manual effort. As the authors note in Section 6.1:

"constructing a reliable tool suite and a high-fidelity execution environment typically requires substantial manual effort. Furthermore, it is difficult to automatically validate the quality of such environments without human involvement, making scalability a significant challenge."

This is the crux. Prior work in the forward paradigm (including the very benchmarks the paper evaluates on, τ-bench and τ2-Bench) has demonstrated that simulated agent–human interplay can produce high-quality training trajectories, but only for the small number of manually constructed domains where human designers have painstakingly built the tools and environments. The τ-bench retail domain, for example, involves a few dozen tools operating over a structured database of products, orders, and users — and building this environment required domain experts to define the schema, implement each tool's behavior, and verify that the state transitions were correct. This simply does not scale to the thousands of domains and tens of thousands of APIs that would be needed for genuinely general agentic intelligence.

The Unaddressed Gap

The paper thus identifies a specific, well-defined gap in the literature: prior work has either sacrificed naturalness for scalability (the reverse paradigm) or sacrificed scalability for naturalness (the forward paradigm). No existing approach simultaneously achieves:

1. Fully simulated environments — no calling real APIs, which are slow, expensive, and unreliable for training at scale.
2. Automated environment construction — no manual definition of tool implementations or database schemas per domain.
3. Verifiable trajectory correctness — supervision signals that can be automatically checked without human judgment.
4. Broad domain coverage — spanning thousands of diverse tool-use scenarios, not just a handful of hand-crafted domains.

The absence of such an approach means that agentic training data remains scarce, and agentic capabilities remain narrow. Models can be trained to perform well on the specific domains that humans have invested in building (retail, airline, telecom), but their competence does not generalize to novel tool compositions or unseen domains because the training distribution is too narrow.

How This Paper Positions Itself

The paper positions itself as solving exactly this gap through a specific architectural insight: if tools are defined as executable code that reads from and writes to a database, then the entire environment construction pipeline can be automated given only a collection of API specifications.

This is not merely an engineering contribution — it is a conceptual reframing of what constitutes an "environment" for agent training. Prior work treated environments as something to be built (manually defining tool behaviors and database schemas). This paper treats environments as something to be induced from the structure of the tool space itself. The key steps in Section 2 make this concrete:

• Tool dependency graph modeling (Section 2.1) discovers natural domain boundaries by analyzing which tools share compatible parameter structures. The Louvain community detection algorithm identifies clusters of tools that naturally operate together — tools for project management cluster separately from tools for e-commerce, which cluster separately from tools for financial transactions. This is an unsupervised, data-driven alternative to humans manually defining domain boundaries.

• Function schema programmatic materialization (Section 2.1) automatically generates database schemas from the parameters of all tools within a domain, then instantiates each tool as executable Python code that performs read/write operations on that schema. This eliminates manual tool implementation — the code is generated, not written by domain experts.

• Agentic task construction (Section 2.2) samples tool sequences from the dependency graph (via directed walks), generates arguments, initializes database states, and verifies that the generated trajectories produce correct final states. This provides the verifiability that prior forward-paradigm approaches lacked.

The paper explicitly notes an intriguing validation of this approach: when the system generates database structures and tool implementations for domains that happen to overlap with τ-bench's manually constructed domains, the automated outputs "exhibit a high degree of consistency with the official implementations provided by τ-bench" (Section 2.1). This suggests the automated approach is not just scalable but also capable of producing environments of comparable quality to human-designed ones.

On the learning side, the paper positions its two-stage agent experience learning framework (Section 3.2) as addressing a different gap: the observation that general tool-use competence and domain-specific expertise benefit from different training regimes. Stage 1 focuses on breadth — exposing the agent to a diverse set of tools and tasks across general domains to develop fundamental capabilities like knowing when to use a tool versus responding directly, how to parameterize calls, and how to integrate tool outputs into responses. Stage 2 focuses on depth — grounding the agent in the specific tools, user intents, and conversational patterns of target vertical domains. This two-phase approach is motivated by the intuition (validated in the ablation in Figure 3) that domain specialization builds on and refines general foundations rather than replacing them.

The Training-Inference Deployment Context

While the paper's primary framing is about data generation, there is an important subtext about model scale that shapes its positioning. The authors explicitly note in the Limitations section that their method has been validated only up to the 30B-parameter scale, and they endorse the position (citing Belcak et al., 2025) that "small language models are the future of agentic AI." This is not an arbitrary choice — it reflects a specific deployment motivation. A 30B-parameter model that can match the tool-use competence of a 1T-parameter model (as the paper claims to achieve) is dramatically cheaper to serve, can run on edge devices, and has lower latency. The environment scaling approach is thus positioned not just as a way to improve agentic capabilities in general, but specifically as a way to make agentic capabilities accessible in resource-constrained settings — a use case where the alternative of "just use a bigger model" is not feasible.

This deployment-oriented motivation explains why the paper invests so heavily in the verifiability and filtering of training data (Section 3.1). When training a smaller model, data quality matters more — a 30B-parameter model cannot compensate for noisy supervision through sheer parameter count the way a much larger model might. The three-stage funnel-based filtering (validity control → environment state alignment → function calling exact match) is designed to ensure that every training trajectory is a reliable supervision signal, which is critical for efficient learning at smaller model scales.

3. Technical Approach

3.1 Reader Orientation

The system being built is a pipeline that automatically constructs thousands of fully simulated tool-use environments and then trains language models to be capable agents within those environments — essentially, a factory for generating diverse, verifiable agentic training data and then using it to produce competent function-calling models. The core problem it solves is the dual bottleneck of environment scarcity and supervision unreliability in agentic training: prior approaches either used real APIs (slow, expensive, unstable) or required manual environment construction (not scalable), and they struggled to verify that generated trajectories were actually correct. The solution's shape is a two-stage architecture: first, induce environments automatically from API specifications by modeling tools as executable read–write operations over induced database schemas; second, learn agentic behavior from the resulting trajectories through a two-phase fine-tuning process that builds general competence before specializing to target domains.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, organized into two sequential stages:

1. API Corpus & Tool Dependency Modeling — Takes a large collection of real-world API specifications (over 30,000) and organizes them into coherent domains by analyzing parameter compatibility. Uses Louvain community detection on a graph where edges represent parameter similarity, producing over 1,000 tool domains.

2. Environment Materialization Engine — For each domain, automatically generates a database schema from the parameters of all tools in that domain, then instantiates each tool as executable Python code that performs read/write operations on that schema. This produces a fully simulated environment where tool calls have deterministic, database-grounded responses.

3. Agentic Task Constructor — Samples coherent tool sequences from each domain's dependency graph (via directed walks), generates arguments, initializes diverse database states, and synthesizes a high-level user intent that would naturally require that tool sequence. The result is a verifiable task specification with known ground-truth tool sequences, arguments, and final database states.

4. Simulated Interaction Engine — Instantiates a simulated user (tasked with fulfilling the intent) and a task agent, then runs multi-turn interactions where the agent makes tool calls, the environment responds, and the simulated user provides feedback. Collects full interaction trajectories as agentic experience data.

5. Two-Stage Training Pipeline — In Stage 1, fine-tunes the base model on trajectories from general domains to develop fundamental tool-usage capabilities. In Stage 2, fine-tunes further on domain-specific trajectories to specialize the agent for target vertical contexts. The loss function masks user instructions and tool responses from supervision, training only on assistant-generated tool calls and natural-language responses.

Information flows as follows: raw API specifications enter → tool dependency graph is constructed and partitioned → each domain's tools are materialized as executable code with an induced database → agentic tasks are constructed by sampling tool sequences and initializing states → simulated interactions produce trajectories → trajectories are filtered through a three-stage validation pipeline → validated trajectories train the model in two phases → the resulting AgentScaler model performs function calling at test time.

3.3 Roadmap for the Deep Dive

• First, the formal design principle and abstraction that unifies all tool calls as read–write database operations, since this is the conceptual foundation that makes automated environment construction feasible.
• Second, the environment construction pipeline in detail — scenario collection, tool dependency graph modeling (including the Louvain community detection), and function schema programmatic materialization — because the environments are the source of all training data and must be understood before the learning process.
• Third, the agentic task construction process — how tool sequences are sampled, arguments generated, database states initialized, and user intents synthesized — since this transforms static environments into concrete, verifiable training tasks.
• Fourth, the human–agent interplay and trajectory collection mechanism, including user simulation, since this is where raw experience data is generated.
• Fifth, the three-stage funnel-based filtering framework, since this is what ensures the training data is reliable and why the approach works at scale without human verification.
• Sixth, the training objective and the two-stage experience learning framework, since this is where the model actually acquires agentic capabilities from the constructed data.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and data engineering paper whose core idea is that general agentic intelligence can be advanced by scaling the diversity and verifiability of synthetic environments, and that a principled pipeline for automated environment construction — combined with a two-phase training strategy — can produce compact models (4B–30B parameters) that match or exceed the function-calling performance of much larger models.

The Unifying Abstraction: Tools as Read–Write Database Operations

Before any component is built, the paper establishes a formal design principle (Section 2) that governs the entire pipeline. The key insight is that any function call can be interpreted as a read or write operation over an underlying environmental database.

Let $D$ represent the database that encodes the state of an environment. Each tool (function) $func$ is assigned an operator type:

$\text{op}(func) \in \{\text{read}, \text{write}\}$

where a read-type function queries $D$ without modifying it (e.g., retrieving a user's order history, checking flight availability, inspecting a project timeline), and a write-type function induces a state transition in $D$ (e.g., creating a new order, booking a flight, updating a project milestone). The response of calling tool $func$ with arguments $\alpha$ is then equivalent to evaluating the assigned operator on the database:

$\text{API}(func, \alpha) \equiv \text{op}(func)(\alpha; D)$

where $func$ is the tool being invoked, $\alpha$ represents the input arguments provided to that function call, $D$ is the current database state, and $\text{op}(func)$ is the read or write operator assigned to that tool.

What this equation computes: given a tool name, a set of arguments, and the current database state, it produces the tool response by executing the tool's assigned operator (either querying $D$ if it is a read, or modifying $D$ and returning the result if it is a write). The output is a deterministic, database-grounded response rather than an LLM-generated hallucination. This enables downstream verifiability because the expected final database state after a sequence of tool calls can be computed exactly, without relying on an external judge model or human evaluation.

Why this form: the alternative — having tools return LLM-simulated responses — introduces hallucinations and inconsistent variability across calls (the same get_order_status("order_123") might return different results on different invocations). The read–write abstraction eliminates this uncertainty by grounding every tool response in a ground-truth database state. Furthermore, the binary read/write categorization cleanly separates tools that only provide information from tools that change the environment state, which matters for verification: a trajectory containing only read operations cannot be validated by checking final state (since the state never changed), requiring a different verification strategy (exact sequence matching) that the filtering pipeline handles in its third stage.

A further structural property: tools within the same domain $d$ — meaning they operate on the same conceptual entity like "project management" or "e-commerce" — typically exhibit structurally similar read–write patterns. This similarity can be captured by a common database schema $S_k$. Consequently, the entire design problem reduces to two sub-problems: (1) partitioning the tool space into domains $\{T_1, \ldots, T_M\}$ where tools in each domain share compatible data structures, and (2) assigning to each domain a database schema $S_k$ that specifies the environment for that domain. This reduction is what enables the automated pipeline: if tools can be automatically clustered into domains and schemas can be automatically induced from their parameters, then environment construction becomes a programmatic process rather than a manual one.

Scenario Collection: Assembling a Diverse API Corpus

The pipeline begins (Section 2.1) by assembling a large corpus of real-world API specifications. The authors collect more than 30,000 APIs from three sources:

• ToolBench (Qin et al., 2023; Guo et al., 2024), a large-scale benchmark for tool learning that includes APIs from diverse categories.
• API-Gen (Prabhakar et al., 2025b), a dataset for generating verifiable function-calling data.
• Their internal tool repository at Alibaba's Tongyi Lab, which provides production APIs from real deployed services.

After collection, the authors apply "rigorous filtering, including the removal of low-quality APIs and subsequent refinement." The paper does not specify the exact filtering criteria in detail, beyond noting that low-quality APIs are removed and some API descriptions are rewritten "to incorporate explicit input–output specifications," following the approach of Fang et al. (2025). This rewriting step is significant: raw API documentation often describes what a function does in natural language without specifying the precise format of arguments and return values. By enhancing descriptions with explicit input–output specifications, the pipeline creates a more structured foundation for later stages that rely on parameter matching and schema generation.

The authors also note that they "further constructed tool compositions by systematically exploiting the input–output relationships among APIs." This suggests that beyond collecting individual API specifications, they analyzed which tools could be chained together — for instance, a search_flights tool returning flight IDs that could be fed into a book_flight tool. The result is an API pool $\Theta_F$ of size $N$ (over 30,000 APIs), with explicit parameter-level information about each tool and some understanding of inter-tool composability.

Tool Dependency Graph Modeling: Automatically Discovering Domains

Given the API pool, the next step is to organize these 30,000+ tools into coherent domains without human labeling. The paper solves this through graph-based community detection, treating the API pool as a network where edges represent compatibility between tools.

Construction of the tool graph. Each node in the graph represents a tool $func$. A tool has a natural-language description and a list of parameters $P_{func}$. For a pair of tools $i$ and $j$, the authors extract their respective parameter lists and convert them into vector representations $\phi$ using an embedding function (the paper does not specify which embedding model is used, though it is presumably a standard text embedding model such as a sentence transformer). They then compute the cosine similarity between these parameter embeddings:

$\text{sim}(\phi(P_{func_i}), \phi(P_{func_j}))$

where $\phi$ is the embedding function that maps a tool's parameter list to a dense vector, and $P_{func_i}$ denotes the parameter specification of tool $i$.

An edge $E$ is inserted between tools $i$ and $j$ if this similarity exceeds a predefined threshold $\tau$:

$E = \{(i, j) \mid \text{sim}(\phi(P_{func_i}), \phi(P_{func_j})) > \tau, i \neq j\}$

where $\tau$ is the similarity threshold (the paper does not specify its exact value), and the condition $i \neq j$ ensures no self-loops.

What this edge criterion means in practice: if two tools have parameters that are vectorially similar — for example, both have parameters like project_id, task_name, deadline, and assignee — they likely operate on the same kind of underlying data and belong in the same domain. The edge captures the notion that these tools are "compatible" or "related" in terms of the data structures they manipulate. The resulting graph is a network where clusters of densely connected nodes represent natural tool domains.

Domain partitioning via Louvain community detection. With the tool graph constructed, domain partitioning reduces to a graph clustering problem. The authors apply Louvain community detection (Blondel et al., 2008), a widely-used algorithm for identifying communities in large networks by optimizing modularity — a measure of how much denser the connections within communities are compared to what would be expected in a random graph. The algorithm is chosen for its scalability to large graphs (30,000+ nodes) and its ability to produce a hierarchical partitioning without specifying the number of communities in advance.

The output is a partition of the tool space into $M$ domains, where the paper states $M$ exceeds 1,000. This means the algorithm discovers over 1,000 distinct tool-use contexts — project management, travel booking, financial transactions, customer support, inventory management, etc. — each corresponding to a different "environment" in which agents can be trained. Critically, this partitioning is completely unsupervised: no human labeled which tools belong to which domain; the structure emerges from the parameter compatibility patterns encoded in the API specifications.

Refining edges with an LLM. The authors identify a limitation of purely parameter-similarity-based edge construction: "since parameter matching relies solely on vectorization and considers only individual parameter information, the overall inter-tool dependencies may be difficult to capture." In other words, two tools might share parameter names (high cosine similarity) but actually serve unrelated functions in different domains, or conversely, two tools might have differently named parameters but be functionally dependent (e.g., one tool outputs a booking_reference and another takes a reservation_code — semantically identical but textually different).

To address this, for tools within each already-segmented domain, the authors further employ an LLM to "systematically examine the dependencies between each pair of tools, thereby further improving the accuracy of edges in the tool graph." The paper does not specify the LLM used for this refinement, nor the exact prompt structure. However, the intended effect is clear: the LLM reads the natural-language descriptions and parameter specifications of both tools and determines whether a genuine functional dependency exists (e.g., the output of tool A is semantically compatible with an input of tool B), augmenting the purely embedding-based edges with semantic understanding.

Why this two-stage edge construction: the initial parameter similarity provides a computationally cheap way to narrow down candidate edges — exhaustively evaluating all 30,000² ≈ 900 million tool pairs with an LLM would be prohibitively expensive. By first clustering tools into domains and only then applying the LLM within each domain, the computational cost is dramatically reduced while still capturing the nuanced dependencies that pure embedding similarity misses. The result is a tool graph with high-quality, semantically meaningful edges that serves as the foundation for sampling coherent tool sequences in the task construction phase.

Function Schema Programmatic Materialization: Generating Executable Environments

Once domains are identified and tool relationships are established, the next step is to transform each domain's tool set into a fully executable environment. This involves two sub-steps: schema generation and code instantiation.

Schema generation. For a given domain $d$ containing a set of tools $T_d$, the authors "leverage the parameters of all tools within a domain to generate a domain-specific database structure." This means they analyze the parameter specifications of every tool in the domain — all the fields that any tool can read or write — and synthesize a unified database schema that can store all this information. The schema serves as the underlying state $D$ for that domain. For example, in an e-commerce domain with tools for looking up products, creating orders, and processing returns, the schema would include tables for products, orders, users, and returns, with columns corresponding to the parameters that the domain's tools reference.

The paper notes that the generated schemas are implemented as Python data structures, not as actual SQL databases — the tools operate on in-memory Python dictionaries and lists that represent the database state. This choice is pragmatic: it avoids the overhead of setting up database servers for each of the 1,000+ domains, and it makes state transitions trivially serializable for verification.

Code instantiation. With the schema defined, each tool in the domain is "formalized in Python code, enabling it to perform read–write operations over the database schema." This means the tool's description and parameter specification are translated into an executable Python function. A read-type tool takes the database state and some arguments as input, queries the appropriate part of the database structure, and returns the result (e.g., get_order_status(data, order_id) looks up data["orders"][order_id]["status"] and returns it). A write-type tool takes the database state and arguments, modifies the database structure according to its specification, and returns a confirmation or the updated state (e.g., cancel_order(data, order_id) sets data["orders"][order_id]["status"] = "cancelled" and updates a timestamp).

A concrete example is given in Figure 1 for a ProjectSchedulecreate tool:

class ProjectSchedulecreate(Tool):
    @staticmethod
    def __call__(
        data: Dict[str, Any],
        projectDetails: Dict[str, Any]
    ) -> str:
        projects = data["projects"]
        # tool-specific creation logic
        return ...

This tool takes the database state data and a dictionary of project details as arguments, creates a new project entry in the projects table, and returns a result. The important property is that the code is generated, not manually written — the system produces it from the tool specification.

Validation against human-designed environments. The paper reports an interesting external validation of this approach: "when generating database structures and formalizing code within specific domains of τ-bench, we observe through manual inspection that our outputs exhibit a high degree of consistency with the official implementations provided by τ-bench" (Yao et al., 2024). This suggests that the automated pipeline, when applied to domains that happen to overlap with τ-bench's human-designed retail and airline environments, produces schemas and tool implementations that closely match what domain experts manually constructed. This is non-trivial: it indicates that the parameter structure of real APIs carries enough information to recover the underlying domain model.

Why this automated approach over manual construction: the alternative — having domain experts manually define schemas and implement tools for each of 1,000+ domains — would be prohibitively expensive in human time (the τ-bench retail domain alone required substantial expert effort for a few dozen tools). The programmatic materialization approach makes environment construction a function of the API specifications alone, which scale with the availability of API documentation rather than with the availability of domain experts. Additionally, automated generation ensures consistency: every tool response is guaranteed to be grounded in the database state, avoiding the human errors that might arise from manually implementing complex state transition logic.

Agentic Task Construction: Generating Verifiable Training Scenarios

With executable environments in place, the pipeline now needs to construct specific tasks — concrete scenarios where a user has a goal that requires the agent to invoke a particular sequence of tools. The challenge is to generate tasks that are both realistic (the tool sequence should correspond to a naturally motivated user need) and verifiable (the correct sequence of tool calls and final database state should be known in advance).

State initialization. The first step is to initialize the environment state $D$ "based on the domain-specific database schema, while encouraging as much diversity as possible in the initial state." This means populating the database tables with realistic example data — fake users, products, orders, flights, etc. — that provide a rich context for the task. Diversity is important because it prevents the agent from overfitting to a narrow set of database configurations (e.g., always seeing the same five products in every simulated retail interaction).

Tool sequence sampling. The critical sub-step is generating a logically coherent tool sequence that the agent should execute to fulfill some user intent. The approach exploits the tool dependency graph constructed earlier. Specifically, the authors "construct a directed dependency graph over APIs" — this is a refinement of the earlier undirected similarity graph, now with directed edges indicating that tool A produces outputs that can serve as inputs to tool B. They then "traverse it to obtain valid sequences" through a directed walk:

1. Start from a randomly selected initial node (tool) in the dependency graph.
2. Follow outgoing edges to select the next tool in the sequence.
3. Continue this directed walk until either (a) the maximum execution steps are reached, or (b) a node with no outgoing edges is encountered (a "sink" in the dependency graph).
4. The result is a sequence of tools $[func_1, func_2, \ldots, func_k]$ that forms a logically coherent workflow — each step's outputs are compatible with the next step's inputs by construction.

Argument generation and ground-truth execution. For each step in the sampled tool sequence, the system "generates the corresponding arguments and performs the actual tool call, grounding the operations directly on the database and continuously tracking the evolving database state." This means the system does not just generate a list of tool names — it generates specific parameter values (which user ID to look up, which flight to book, what refund amount to process) and actually executes these function calls against the initialized database, producing the ground-truth database state at each step and the final database state $D_{final}$ after the complete sequence.

User intent synthesis. With the ground-truth tool sequence, arguments, and final state established, the system then synthesizes a high-level user intent that would naturally require this sequence of tool calls. The paper does not specify exactly how this synthesis is performed (presumably via an LLM prompted with the tool sequence and database context), but the key property is that the intent is formulated to be realistic: a human user might genuinely express something like "I want to return my order and get a refund" without specifying the exact tool calls required, and the agent must determine that this requires get_user_info → get_order_info → process_return → issue_refund.

Dual verifiability. The resulting task specification provides verifiability at two complementary levels:

1. Database-level state consistency: the final database state after the agent's interaction must match the ground-truth final state $D_{final}$. This catches errors in write operations — if the agent booked the wrong flight or issued the wrong refund amount, the database state will differ from the expected state.
2. Exact matching of tool sequences: the sequence of tools invoked by the agent can be compared against the ground-truth sequence. This is particularly important for tasks involving only read operations — if the agent is supposed to look up a user's order history and summarize it, there are no state changes to verify, so the only way to check correctness is to confirm that the right tools were called with the right arguments in the right order.

This dual verifiability is what enables the filtering pipeline (Section 3.1) to operate without human judgment. Every component of the task — the intent, the gold tool sequence, the gold arguments, the gold initial state, the gold final state — is known in advance, so any trajectory produced through interaction can be automatically scored against ground truth.

Human–Agent Interplay for Experience Collection

Once a verifiable agentic task is constructed, the system performs simulated human–agent interplay to collect interaction trajectories (Section 3.1). This is where the forward paradigm's naturalness is achieved while retaining the verifiability that prior forward-paradigm approaches lacked.

The three simulated entities. The interplay involves three components, all simulated within the system:

• Simulated User: an LLM-based user agent that is given the overall user intent and instructed to interact with the assistant to fulfill it. The simulated user does not know the gold tool sequence — it behaves like a real user, providing information when asked, responding to the agent's questions, and only declaring the task complete when its needs are satisfied.
• Task Agent: the model being trained (or a stronger teacher model, in the case of offline data generation) that has access to the domain's tools and must determine which tools to call and how to respond to the user.
• Environment: the fully simulated environment (executable tools + database) that provides deterministic tool responses based on the current database state.

Interaction dynamics. The interplay proceeds in turns (Figure 2). The simulated user initiates the conversation with a query that expresses the high-level intent (e.g., "Can you return my order?"). The agent may ask clarifying questions, invoke tools to look up information, and provide natural-language responses. Each tool call is executed by the environment, which reads or writes to the database and returns a response. The simulated user responds to the agent's outputs, potentially providing additional information or confirming actions. The interaction continues until the simulated user deems the task complete.

An example from Figure 2 illustrates the pattern:

• User: "Can you return my order?"
• Agent: "Sure, but I would need your email to authenticate you and order info."
• User: "My email is … and the order is about …"
• Agent calls get_user_info(…) → obtains user info
• Agent calls get_order_info(…) → obtains order info
• Agent calls get_..._info(…) → obtains more info
• Continues through more interaction turns.

Collecting experience trajectories. Each completed interaction constitutes an agentic experience — a full trajectory of alternating user instructions and assistant turns, with each assistant turn decomposed into three parts:

$a_t = (\tau_t, \rho_t, y_t)$

where $a_t$ is the assistant's turn at interaction round $t$, $\tau_t$ represents the function call tokens (the specific tool name and arguments the model chose to invoke), $\rho_t$ represents the tool response tokens (the output returned by the environment after executing the function call), and $y_t$ represents the assistant's natural-language response tokens (what the model says to the user after processing the tool output). The trajectory also includes the human instructions $h_t$ at each turn and the initial system prompt $h_0$.

Why simulated users rather than static scripts: prior work in the reverse paradigm generated static user queries that directly mapped to tool calls, producing unnaturally transparent interactions. The simulated user approach produces more realistic conversational dynamics because the user agent can be surprised by the assistant's responses, ask follow-up questions, provide incomplete information initially, and only reveal additional details when prompted — all behaviors that real users exhibit and that agents must learn to handle. The paper's adoption of this approach is motivated by Yao et al. (2024), where the τ-bench authors demonstrated that simulated users produce valid, diverse interactions for agent evaluation; this paper extends that idea from evaluation to large-scale training data generation.

Scalability of the interplay. Critically, because the simulated user, the agent, and the environment are all fully simulated, the interplay can be run at arbitrary scale without human involvement. The system can generate thousands of interaction trajectories per domain, covering diverse user behaviors and database configurations, simply by varying the random seeds for state initialization, tool sequence sampling, and user agent prompting. This is what the paper means by "a highly scalable framework" — the bottleneck is compute, not human time.

Three-Stage Funnel-Based Trajectory Filtering

Not all interaction trajectories are suitable for training. The simulated user might accept a task as complete even when the agent made errors (e.g., the user simulation might not notice that the wrong refund amount was processed), or the interaction might devolve into unproductive loops. The paper addresses this with a three-stage funnel-based filtering framework that progressively applies stricter correctness criteria (Section 3.1).

Stage 1: Validity Control. The first and broadest filter removes "invalid interaction trajectories to ensure well-formed alternating user assistant exchanges." This catches structural problems: trajectories where the conversation format is broken, turns are missing, or the interaction ended prematurely. Additionally, the authors "apply an n-gram-based filtering procedure to eliminate severely repetitive reasoning segments" — a common failure mode where the agent gets stuck in a loop, generating the same function call or thinking pattern repeatedly. Trajectories exhibiting such repetitive patterns are discarded entirely at this stage.

Stage 2: Environment State Alignment. The second filter is more stringent: it "retains only those trajectories whose final database state matches the golden state after the interplay, thereby validating the effectiveness of write operations." This is the database-level verifiability check. After the simulated interaction completes, the system compares the final database state $D_{final}^{agent}$ (the state after all the agent's tool calls) against the ground-truth final state $D_{final}^{gold}$ (the state that should have resulted from the correct tool sequence). If they match, the trajectory passes — the agent's write operations were correct in aggregate. If they differ, the trajectory is discarded.

The filtering granularity at this stage is the database/environment level — the entire database state must match, not just individual records. This is a holistic correctness check: the agent might have called tools in a different order than the gold sequence, or used different intermediate arguments, but as long as the final state is correct, the trajectory is retained. This leniency is intentional — it allows the training data to include diverse but valid solution paths rather than forcing the agent to learn a single canonical sequence.

Stage 3: Function Calling Exact Match. The third and most stringent filter addresses a specific limitation of Stage 2: "a tool sequence consisting entirely of read operations without any write operations would cause state-based filtering to fail." If the task is purely informational (e.g., "summarize my order history"), the database state never changes, so state alignment is trivially satisfied regardless of what the agent did. To handle such cases, Stage 3 applies exact match filtering at the tool sequence level: "a trajectory is preserved only if the sequence of invoked tools and arguments exactly matches the overall intent." Both the tool names and the arguments must match the gold specification.

The filtering granularity here is the individual tool call and argument, not just the final state. This is a much stricter criterion that ensures the agent not only achieved the right outcome but followed the intended procedure. It is applied to read-only trajectories where state-based verification is impossible, and potentially also as an additional quality filter for write-containing trajectories.

Retaining error-recovery trajectories. An important design choice: "we do not filter out trajectories in which tool calls return errors." The rationale is that "such trajectories may still accomplish the intended goal despite intermediate failures" — an agent might call a tool with incorrect arguments, receive an error, recognize the mistake, correct the arguments, and successfully complete the task. Retaining these trajectories "helps improve the robustness of the model" by teaching it to recover from tool-call errors, a critical skill for real-world deployment where API calls can fail for many reasons. The filtering pipeline validates the final outcome (state alignment and, for read-only cases, sequence matching), not the smoothness of the intermediate steps.

Why a three-stage funnel: each stage filters at a different granularity and catches different types of errors. Stage 1 removes structurally broken trajectories cheaply (n-gram analysis is computationally light). Stage 2 removes trajectories with incorrect outcomes using the strongest available signal (database state, which is the ground truth for write operations). Stage 3 catches the remaining cases where Stage 2 is blind (read-only trajectories) and provides an additional precision check. By ordering stages from cheapest to most expensive and from broadest to most specific, the funnel maximizes the efficiency of the filtering process — expensive LLM-based or full-trajectory comparisons are only performed on trajectories that have already passed the cheaper structural and state-based checks.

Agentic Fine-Tuning Objective

With validated trajectories in hand, the model is fine-tuned using a specific objective that selectively supervises different parts of the trajectory (Section 3.2).

Trajectory structure for training. An experience trajectory $H$ from the interplay is represented as:

$H = (h_0, a_1, \ldots, a_{n-1}, h_n, a_0)$

where each human instruction is denoted by $h_t$ at interaction round $t$, and each assistant turn $a_t$ decomposes as $a_t = (\tau_t, \rho_t, y_t)$ — function call tokens, tool response tokens, and assistant response tokens.

Selective supervision with token masking. The training objective is to optimize only the assistant's outputs — the function calls and the natural-language responses — while conditioning on everything else. Human instructions $h_t$ represent the task specification (what the user wants) and should not be modified by the model. Tool responses $\rho_t$ represent ground-truth environment feedback and should not be optimized (the model cannot change how the environment responds). The model should learn to produce the right $\tau_t$ (what tool to call and with what arguments) and the right $y_t$ (what to say to the user after processing the tool output).

Formally, given an autoregressive model $p_\theta(x_k \mid x_{<k})$ that predicts each token $x_k$ conditioned on all preceding tokens $x_{<k}$, the loss is:

$\mathcal{L}(\theta) = -\frac{1}{\sum_{k=1}^{|H|} \mathbb{I}[x_k \in \mathcal{T}]} \sum_{k=1}^{|H|} \mathbb{I}[x_k \in \mathcal{T}] \cdot \log \pi_\theta(x_k \mid x_{<k})$

where $x_k$ is the $k$-th token in the trajectory $H$, $\pi_\theta$ is the model's predicted probability distribution over the vocabulary at position $k$, $\mathbb{I}[\cdot]$ is the indicator function (1 if the condition is true, 0 otherwise), and $\mathcal{T}$ is the set of tokens belonging to tool calls $\tau$ or assistant responses $y$ — that is, the tokens the model is supposed to learn to generate.

What this equation computes: it is a standard autoregressive language modeling loss (next-token prediction cross-entropy) applied selectively. The indicator function $\mathbb{I}[x_k \in \mathcal{T}]$ acts as a mask: for tokens belonging to human instructions $h$ or tool responses $\rho$, the indicator is 0 and the token contributes nothing to the loss. For tokens belonging to function calls or assistant responses, the indicator is 1 and the model is penalized for predicting those tokens incorrectly. The denominator $\sum_{k=1}^{|H|} \mathbb{I}[x_k \in \mathcal{T}]$ normalizes the loss by the number of supervised tokens rather than the total sequence length, ensuring the loss magnitude is comparable across trajectories with different ratios of supervised-to-unsupervised tokens.

Why this masking: if the loss were applied uniformly across all tokens, the model would be penalized for "incorrectly predicting" the user's instructions (which are external inputs, not things the model should generate) and the tool responses (which are deterministic environment outputs, not model decisions). This would waste model capacity on predicting noise and could create undesirable behaviors, such as the model learning to hallucinate tool responses rather than actually calling tools. By masking out everything except the model's own decisions ($\tau_t$ and $y_t$), the training signal focuses exclusively on the skills that matter: choosing the right tool, parameterizing it correctly, and communicating results effectively.

Crucially, while $\rho_t$ and $h_t$ tokens are masked from the loss, they remain visible in the context $x_{<k}$. This means the model conditions on the full interaction history — the user's instructions and the environment's feedback — but only receives gradient updates for the tokens it actually produces. This is the standard approach for instruction-following fine-tuning with tool calls, and it ensures the model learns to attend to tool responses and user requests without being forced to reproduce them.

Two-Stage Agent Experience Learning

The paper proposes a two-phase training strategy (Section 3.2) motivated by the intuition that general tool-use competence and domain-specific expertise benefit from different training distributions and objectives.

Stage 1: General Foundation Learning. In the first phase, "the agent is trained to acquire fundamental skills for tool usage and user interaction." The training data for this phase consists of trajectories from general domains — a broad set of tools and tasks drawn from across the 1,000+ domains constructed by the environment pipeline. The emphasis is on breadth and generality: the agent encounters many different types of tools, argument structures, conversational patterns, and task categories, allowing it to develop a robust understanding of:

• When to invoke a function call versus responding directly in natural language — distinguishing between queries that require tool access ("what's my order status?") and those that don't ("what's your return policy?").
• How to parameterize calls across diverse argument types — handling required vs. optional parameters, understanding parameter constraints, and dealing with missing information.
• How to integrate tool outputs into coherent user-facing responses — not just dumping raw tool output but synthesizing it into helpful natural language.
• How to handle multi-turn interactions — asking clarifying questions when needed, maintaining context across turns, and recognizing when the task is complete.

This stage "ensures that the agent builds a versatile foundation of agentic behaviors before domain-specific specialization." The key design choice is that Stage 1 is domain-agnostic: it does not target any specific vertical (retail, airline, telecom) but instead exposes the model to the full diversity of the constructed environments.

Stage 2: Domain-Specific Specialization. In the second phase, "the agent undergoes fine-grained training in vertical domains, where tasks, tools, and user intents exhibit domain-specific characteristics." The training data for this phase is drawn from the specific target domains that the model will be evaluated on — for the AgentScaler models, this means the retail, airline, and telecom domains that align with τ-bench and τ2-Bench, as well as the domains tested by ACEBench.

The rationale for this two-phase approach is twofold. First, learning transfer: the fundamental skills acquired in Stage 1 (parameterization, tool selection, response integration) transfer across domains, so Stage 2 can focus on domain-specific nuances — the particular tools available, the typical user intents, the common argument patterns, and the conversational dynamics of that domain. Second, efficiency: training directly on domain-specific data from scratch might lead to overfitting to narrow patterns, whereas Stage 1 provides a broad prior that regularizes Stage 2 training, enabling the model to generalize within the domain rather than memorizing surface-level patterns.

Ablation evidence (Figure 3). The paper validates this design through an ablation on ACEBench-en using the Qwen3-Thinking-30B-A3B base model. The base model achieves some performance, Stage 1 training substantially improves performance across all subsets (Normal, Special, Agent, Overall), and Stage 2 training provides further improvements, particularly on the Agent subset which requires complex multi-step tool compositions. The sequential improvement pattern — base → Stage 1 → Stage 2 — supports the claim that general foundation learning is critical for establishing tool-usage competence, and that subsequent domain specialization consolidates and contextualizes these capabilities.

Design choice: why not single-stage on all data? The paper does not explicitly compare against a single-stage approach that mixes general and domain-specific data, but the two-phase design reflects a common curriculum learning intuition: the model benefits from first mastering the fundamentals in a diverse but lower-stakes setting before tackling the specific complexities of target domains. In a single-stage approach, the domain-specific data might dominate the gradient signal (if it is overrepresented relative to the diversity of general data), causing the model to converge to a narrower competence that does not generalize. The two-stage approach enforces that broad foundations are established first, with specialization as a refinement rather than a replacement.

Summary of Design Choices and Their Justifications

• Read–write database abstraction over LLM-simulated tool responses: eliminates hallucination and enables deterministic verifiability by grounding every tool response in a ground-truth database state. This is the key enabler for automated filtering without human judgment.
• Louvain community detection over manual domain labeling or k-means clustering: automatically discovers the natural number of domains from the data structure, handles the scale of 30,000+ nodes, and produces a hierarchical partitioning without requiring a pre-specified number of clusters.
• Parameter-similarity edges refined by LLM over pure embedding-based edges: the embedding step provides computational efficiency by narrowing candidate pairs, while the LLM refinement captures semantic dependencies that pure vector similarity misses (different parameter names for the same concept, functional dependencies beyond parameter overlap).
• Programmatic code generation for tool materialization over manual implementation: scales to 1,000+ domains without human effort, and the generated code is guaranteed to be consistent with the induced schema, avoiding human implementation errors.
• Directed walk on dependency graph for tool sequence sampling over random sampling: ensures that sampled sequences are logically coherent (each tool's inputs are compatible with previous outputs), which random sampling of tool combinations would not guarantee, producing unrealistic or impossible sequences that would confuse the agent.
• Simulated user for interaction over static scripts or backward query generation: produces realistic conversational dynamics (incomplete information, follow-up questions, surprise) that teach the agent to handle the messiness of real user interactions, while still operating within the fully simulated framework that enables verification.
• Three-stage funnel filtering (validity → state alignment → exact match) over single-stage filtering: each stage catches different failure modes at different computational costs. Ordering from cheapest to most expensive minimizes total filtering cost. State alignment is lenient (allowing diverse valid paths), while exact match provides precision for cases where state alignment is insufficient (read-only trajectories).
• Selective token masking in the loss over uniform next-token prediction: focuses the training signal on the skills that matter (choosing and parameterizing tools, responding to users) while preventing the model from wasting capacity on predicting external inputs (user instructions, environment responses) that it should condition on but not reproduce.
• Two-stage training over single-stage: Stage 1 provides broad foundational competence through diverse domain exposure, establishing skills that transfer. Stage 2 refines these skills for specific target domains, enabling efficient specialization without overfitting. The curriculum prevents domain-specific data from dominating the gradient signal before fundamentals are established.
• Retention of error-recovery trajectories over filtering them out: teaches the model to recover from tool-call failures (incorrect arguments, unavailable resources), which is critical for real-world robustness. The filtering validates final outcomes, not intermediate smoothness.

4. Key Insights and Innovations

Innovation 1: Environment Construction as an Inductive Process from API Structure, Not Manual Design

The paper's most conceptually original contribution is the reframing of what it means to "build an environment" for agent training. Prior work in the forward paradigm — including the very benchmarks the paper evaluates on (τ-bench, τ2-Bench) — treats environments as artifacts that domain experts must manually design: define the database schema, implement each tool's behavior, specify state transition rules, and verify correctness through human inspection. This is expensive, slow, and inherently limits the number of domains that can be constructed. The implicit assumption in the field has been that high-fidelity tool-use environments require human domain knowledge to build, and that automated approaches would produce environments too simplistic or error-prone to serve as reliable training signals.

This paper challenges that assumption by demonstrating that environment structure can be induced from API specifications alone, without any human labeling of domain boundaries, tool behaviors, or database schemas. The core insight is the read–write database abstraction (Section 2): if every function call is understood as a read or write over an underlying database, then the entire environment — schema, tool implementations, state transitions — can be derived programmatically from the parameters and descriptions of the tools themselves. The Louvain community detection on the parameter-similarity graph automatically discovers over 1,000 coherent tool domains without any human specifying what domains exist. The programmatic materialization step automatically generates executable Python code for each tool, grounded in an induced database schema, without any human implementing tool behaviors.

What makes this contribution fundamental rather than incremental is its reversal of the dependency between environments and agents. In prior work, environments are prerequisites for agent training — you must build the environment first, which requires human effort, and only then can you train agents within it. In this paper's framing, environments are outputs of a data-driven pipeline — they emerge from the collected API corpus through unsupervised structure discovery and programmatic code generation. This shifts the scaling bottleneck from human domain expertise to API documentation availability, which is a fundamentally different — and much more scalable — resource constraint. API documentation for thousands of real-world services already exists; the paper shows how to convert that documentation into executable, verifiable training environments without additional human annotation.

The paper provides suggestive external validation of this claim by noting that the automated pipeline produces database structures and tool implementations that "exhibit a high degree of consistency with the official implementations provided by τ-bench" (Section 2.1) — meaning that the induced environments match what human experts manually constructed for the same domains. This is not a quantitative result but a qualitative one with significant implications: it suggests the information content of API parameter specifications is rich enough to recover the domain model that human designers would have built, and that the automated approach is not merely scalable but also capable of producing environments of comparable fidelity to human-designed ones.

The significance of this contribution extends beyond the paper's immediate empirical results. It provides a template for scaling agentic training to arbitrary tool ecosystems: if a new set of APIs becomes available (for instance, a new enterprise software suite or a new web service platform), the pipeline can automatically construct the corresponding training environments without domain experts studying the API documentation and manually designing schemas and tool implementations. This is a genuine conceptual advance in how the field thinks about the environment bottleneck in agentic AI.

Innovation 2: Verifiability as a First-Class Design Constraint in Trajectory Generation

A second contribution — more methodological than conceptual, but with important practical implications — is the paper's elevation of verifiability from an afterthought or a heuristic to a first-class design constraint that shapes every stage of the data generation pipeline. Prior work on synthetic agentic data generation has struggled with a fundamental tension: simulated interactions can produce naturalistic trajectories, but verifying that those trajectories are correct typically requires either human evaluation (not scalable) or reliance on the same LLM that generated the data (circular, since the verifier shares the generator's biases and errors). The reverse paradigm sidesteps this by generating queries from known-correct tool calls, sacrificing naturalness for guaranteed correctness. The forward paradigm produces natural interactions but lacks reliable correctness signals, leading to noisy training data where agents may learn from trajectories that contain subtle errors.

This paper resolves this tension through a specific architectural commitment: every trajectory is verifiable against two independently computed ground truths — the database state and the tool sequence — which are established before the interaction takes place. The task construction process (Section 2.2) first computes the gold tool sequence and the gold final database state by sampling from the dependency graph and executing the tools with generated arguments. Only after this ground truth is established does the simulated human–agent interplay begin. When the interplay produces a trajectory, it can be automatically scored against the gold state and gold sequence without any LLM-as-judge or human evaluation. The three-stage filtering funnel operationalizes this verifiability: Stage 1 catches structural failures cheaply, Stage 2 validates write operations against the gold database state, and Stage 3 validates read-only trajectories against the gold tool sequence.

The innovation here is not just having verification — prior work has used programmatic checks for specific tasks — but rather designing the entire data generation pipeline so that ground-truth verification is guaranteed by construction for every trajectory. The read–write database abstraction is what makes this possible: because tool responses are deterministic functions of the database state, the "correct" final state is computable independently of the interaction dynamics. The simulated user can behave unpredictably, the agent can make mistakes and recover, the conversation can take unexpected turns — but as long as the trajectory eventually reaches the gold database state (or, for read-only tasks, executes the gold tool sequence), it is known to be correct. This design means that the filtering pipeline has access to an oracle correctness signal that is never noisy or ambiguous — a property that no prior forward-paradigm approach has achieved at scale.

The practical significance of this is hard to overstate for a data engineering paper. The three-stage filtering produces training data where correctness is guaranteed, not estimated. This matters enormously for training smaller models (the paper's 4B and 8B AgentScaler variants), which cannot compensate for noisy supervision through parameter count the way much larger models might. It also means that the approach can be extended to new domains without designing new verification procedures — the same state-alignment and sequence-matching checks apply regardless of what the tools do, because they are grounded in the generic read–write abstraction rather than domain-specific correctness criteria.

Innovation 3: Separating General Tool-Use Competence from Domain Specialization as a Curriculum Design Principle

The two-stage training framework (Section 3.2) represents an insight about the structure of agentic capability that goes beyond the specific models trained in this paper. The field has largely approached tool-use training as a monolithic problem: collect data where agents use tools, train on it, and evaluate. Whether the data comes from human annotations, reverse-paradigm generation, or forward-paradigm simulation, the training recipe is typically single-stage — all available data is mixed together and the model learns from it simultaneously.

This paper argues, and provides supporting evidence, that general tool-use competence and domain-specific expertise are distinct capabilities that benefit from distinct training phases. Stage 1 exposes the agent to a broad diversity of tools, tasks, and conversational patterns across over 1,000 domains, developing transferable skills: knowing when a function call is needed versus a direct response, how to parameterize calls across varied argument structures, how to integrate tool outputs into natural language, how to handle multi-turn interactions. Stage 2 then refines these skills within specific target domains, where the agent encounters the particular tools, user intents, and conversational dynamics of retail, airline, telecom, and other vertical contexts.

The significance of this distinction lies in what it implies about efficient use of training data. If general tool-use skills are genuinely transferable — if learning to parameterize a search_flights call helps with parameterizing a lookup_product call, because both involve understanding required vs. optional parameters, handling missing information, and formatting arguments correctly — then the broad Stage 1 data provides a foundation that makes Stage 2 data more valuable per example. The Stage 1 training acts as a regularizer that prevents the model from overfitting to narrow domain-specific patterns in Stage 2 data. Conversely, if all training were done in a single stage with mixed general and domain-specific data, the domain-specific data might dominate the gradient signal (particularly if it is overrepresented in the batch), causing the model to converge to a narrower competence that does not leverage the diversity of the general data.

The ablation in Figure 3 provides evidence for this claim: on ACEBench-en, the sequential progression from base model to Stage 1 to Stage 2 shows monotonic improvement, with Stage 2 providing gains particularly on the Agent subset (which tests complex multi-step tool compositions). This pattern — where general pre-training on diverse tool-use data improves the effectiveness of subsequent domain-specific fine-tuning — is analogous to the now-standard observation in NLP that language model pre-training on broad corpora improves downstream task performance, but applied to the specific capability of function calling. The paper is, to my knowledge, the first to explicitly articulate and validate this two-phase curriculum as a design principle for agentic training.

However, it is worth noting that this contribution is more methodological than theoretical. The paper does not provide a mechanistic account of why the two stages transfer — it does not analyze which specific skills are acquired in Stage 1, which are refined in Stage 2, or whether the same benefits could be achieved by carefully balancing the data mixture in a single stage. The two-stage design is validated by the ablation, but the ablation compares against a baseline with no general-domain training, not against a single-stage model trained on the same total data with a carefully tuned domain-general mixing ratio. This leaves open the question of whether the two-stage architecture is necessary or whether the key insight is simply that general-domain data is valuable — a question the field will likely explore in follow-up work.

Innovation 4: Diagnostic Characterization of the Long-Horizon Tool-Calling Challenge

The paper's final contribution is an empirical diagnosis of where current agentic models fail, rather than a novel method for fixing those failures. The scatter plot in Figure 5 and the accompanying analysis in Section 5 establish a clear pattern: there exists a negative correlation between the number of tool calls in a trajectory and task accuracy, and this holds for AgentScaler models as well as for prior models. The paper plots this relationship on τ-bench data, showing that as the number of tool calls increases, the accuracy drops — a finding that is intuitive but had not been systematically quantified in prior work.

The significance of this finding lies in what it reveals about the limitations of environment scaling as a solution to agentic capability. The paper's entire framework is designed to scale the diversity and volume of tool-use training environments. If the primary bottleneck on agent performance were simply insufficient exposure to diverse tools and tasks, then environment scaling would directly address it. But the long-horizon challenge suggests a deeper bottleneck: even with extensive training on multi-step interactions, the model's competence degrades as the interaction length grows. This degradation is not (or not entirely) a data scarcity problem — it persists in models trained with the paper's extensive pipeline — but rather a fundamental difficulty with maintaining coherent reasoning, tracking state, and recovering from errors over extended interaction chains.

This finding serves a valuable function for the research community by redirecting attention from data scaling to architectural or algorithmic improvements for long-horizon reasoning. The paper itself acknowledges this in Section 5, noting that "handling extended tool-use chains is still an open problem that we plan to address in future work." The implication is that environment scaling, while necessary, is not sufficient for general agentic intelligence — the next frontier is developing models that can maintain reliable performance as the number of sequential tool calls grows.

The diagnostic is also practically informative for deployment. If an organization is considering deploying an AgentScaler-like model, Figure 5 provides a concrete signal about when to expect failures: trajectories requiring more than a certain number of tool calls (the exact threshold varies by domain) are at elevated risk of errors, and might warrant escalation to a human or a more capable backup system. This kind of calibrated understanding of failure modes — knowing not just that a model fails sometimes but under what conditions it is likely to fail — is essential for safe deployment and is often missing from benchmark-focused evaluations.

Compared to the other innovations, this one is primarily empirical rather than conceptual — it does not introduce a new method or reframe the problem, but it provides quantitative evidence for a limitation that the community has suspected but not systematically measured. The paper's contribution here is making the challenge explicit and measurable rather than anecdotal, which enables future work to target it directly and to benchmark progress not just on aggregate accuracy but specifically on long-horizon robustness.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three established agentic benchmarks: τ-bench (Yao et al., 2024), covering the retail and airline domains; τ2-Bench (Barres et al., 2025), spanning retail, airline, and telecom domains; and ACEBench-en (Chen et al., 2025), which stratifies tasks into Normal, Special, and Agent categories. For τ-bench and τ2-Bench, the authors adopt the passˆ1 metric and additionally analyze the trend of passˆk, following the evaluation protocols in Yao et al. (2024) and Barres et al. (2025). For ACEBench-en, results are reported using accuracy on the Normal, Special, and Agent subsets as well as Overall. An additional out-of-distribution evaluation is conducted on ACEBench-zh, the Chinese-language variant of ACEBench. No explicit information is provided about the number of test examples in each benchmark split, but τ-bench and τ2-Bench use standardized test sets from their respective publications.

• Base model(s). The paper trains three model variants — AgentScaler-4B, AgentScaler-8B, and AgentScaler-30B-A3B — all built on the Qwen-3 model family (Team, 2025b). Specifically, AgentScaler-4B and AgentScaler-30B-A3B are trained on Qwen3-Thinking-4B-2507 and Qwen3-Thinking-30B-A3B-2507 respectively, while AgentScaler-8B is trained on the base Qwen3-8B (non-thinking variant). The choice of Qwen-3 is justified implicitly: the paper needs strong base models at multiple scales to demonstrate that the training pipeline improves agentic capabilities across model sizes, and Qwen-3 provides a consistent architecture family spanning 4B to 30B active parameters (the 30B-A3B is a Mixture-of-Experts model). The authors also provide baseline performance for Qwen3-Thinking-4B, Qwen3-8B, Qwen3-14B, and Qwen3-Thinking-30B-A3B as untrained reference points.

• Metrics. The primary metric across all benchmarks is task accuracy — the fraction of test questions for which the model's trajectory achieves the correct outcome. For τ-bench and τ2-Bench, this is operationalized as the passˆ1 metric: in a single attempt, does the model successfully complete the task? The passˆk metric (reported in Figure 4 for stability analysis) measures the fraction of tasks where the model succeeds in all k independent trials, providing a stricter measure of reliability. For ACEBench-en and ACEBench-zh, the paper reports accuracy on each subset (Normal, Special, Agent) and an Overall aggregate. The paper does not provide detailed information about how task completion is evaluated on each benchmark — whether it requires exact match of database state, satisfaction of a user simulator, or some other criterion — but follows the standard evaluation protocols of each benchmark.

• Baselines. The paper compares against an extensive set of both closed-source and open-source models. Closed-source baselines include Gemini-2.5-pro (Comanici et al., 2025), Claude-Sonnet-4 (Anthropic, 2025), GPT-o3, GPT-o4-mini (OpenAI, 2025b), and GPT-5 with thinking (OpenAI, 2025a). Open-source large-scale models include GPT-OSS-120B-A5B (Agarwal et al., 2025), Deepseek-V3.1-671B-A37B (DeepSeek-AI, 2024), Kimi-K2-1T-A32B (Team et al., 2025), Qwen3-Thinking-235B-A22B (Team, 2025b), Seed-OSS-36B (Team, 2025a), and Qwen-Coder-30B-A3B (Hui et al., 2024). Open-source agentic models include the xLAM-2 model series at 8B, 32B, and 70B scales (Prabhakar et al., 2025a). Additionally, the base Qwen-3 models (4B, 8B, 14B, 30B-A3B) serve as untrained reference baselines. This baseline selection is comprehensive, covering the strongest available models at the time of writing across closed API access, large open-source parameter counts, and comparable-scale agentic models.

• Generation budget / compute accounting. The paper does not explicitly define a unit of inference compute (e.g., FLOPs, tokens generated, or number of function calls) for fair comparison between models. Since all models are evaluated using the same benchmark protocols — given a task description and access to domain-specific tools, produce a trajectory of function calls and responses — the implicit unit of comparison is one complete interaction trajectory per task. Models are evaluated on their ability to produce correct trajectories in a single attempt (passˆ1). The paper does not compare models at matched inference budgets or report inference-time compute costs, which means the comparison between, say, AgentScaler-30B-A3B and GPT-5 is on accuracy alone, not on accuracy-per-FLOP or accuracy-per-dollar. This is a standard practice for benchmark evaluation but means that the paper's claims about "efficiency" and "deployment suitability" are based on model size (parameter count) as a proxy for computational cost, rather than on direct measurement of inference costs.

• Cross-validation / statistical protocol. The paper does not report any cross-validation or statistical significance testing. Results in Table 1 are presented as point estimates (single accuracy scores per model per domain) without confidence intervals, standard deviations, or information about the number of evaluation runs. The passˆk analysis in Figure 4 inherently involves multiple trials (each point at passˆk requires k independent evaluations per task), but the paper does not specify how many total trials were run or how variance across trials was handled. For the ablation study in Figure 3, only point estimates are reported without statistical characterization. This absence of statistical rigor is a limitation — particularly for the fine-grained comparisons where a 1–2 percentage point difference between similarly performing models cannot be confidently attributed to model quality rather than sampling variance.

Main Quantitative Results

Aggregate Benchmark Performance

The headline result appears in Table 1: AgentScaler-30B-A3B achieves scores that place it as state-of-the-art among open-source models under 1T parameters and competitive with both much larger open-source models and closed-source systems.

On the τ-bench benchmark:

• Retail domain: AgentScaler-30B-A3B achieves 70.4, matching GPT-o4-mini (70.4) and Seed-OSS-36B (70.4), and exceeding GPT-o3 (70.4 — actually the same score, the decimal precision suggests these are identical values), Qwen3-Thinking-235B-A22B (67.8), and the base Qwen3-Thinking-30B-A3B (67.8). It trails GPT-5-think (78.3) and Claude-Sonnet-4 (73.9) but surpasses Gemini-2.5-pro (68.7). Among open-source models, it is outperformed only by Kimi-K2-1T-A32B (73.9) and ties with GPT-OSS-120B-A5B at a score of 67.8 — wait, the table actually shows AgentScaler-30B-A3B at 70.4, GPT-OSS at 67.8, so AgentScaler exceeds GPT-OSS.

• Airline domain: AgentScaler-30B-A3B achieves 54.0, matching Claude-Sonnet-4 at 40.0 — no, the value is 54.0, substantially exceeding Claude-Sonnet-4 (40.0) and GPT-5-think (44.0). This is a notable finding: on the airline domain, the 30B-A3B model outperforms several closed-source models by a significant margin. It also exceeds GPT-OSS-120B-A5B (49.2), Deepseek-V3.1-671B-A37B (40.0), Qwen3-Thinking-235B-A22B (46.0), and the base Qwen3-Thinking-30B-A3B (48.0). It trails GPT-o3 (52.0) and ties with — wait, let me re-read the actual numbers. The table shows AgentScaler-30B-A3B Airline at 54.0, while GPT-o3 is at 52.0, GPT-o4-mini at 46.0, Gemini-2.5-pro at 44.0. So AgentScaler-30B-A3B actually exceeds all closed-source models on the Airline domain except — checking again — GPT-o3 is 52.0, so AgentScaler (54.0) beats it. GPT-5-think is 44.0. Claude-Sonnet-4 is 40.0. Gemini-2.5-pro is 44.0. This means AgentScaler-30B-A3B achieves the highest Airline score in the entire table, including all closed-source models. This is a striking result. It also exceeds Kimi-K2-1T-A32B (51.2) and all other open-source models. However, AgentScaler-4B achieves 54.0 on Airline as well — identical to the 30B model — which is unusual and raises questions about whether this score reflects a ceiling effect or some property of the Airline domain evaluation. Similarly, AgentScaler-8B achieves only 42.0 on Airline, well below the 4B model, suggesting potential variance issues or differences in training data composition across model scales.

On the τ2-Bench benchmark:

• Retail domain: AgentScaler-30B-A3B achieves 70.2, exceeding Kimi-K2-1T-A32B (70.6? — no, that's actually higher: Kimi-K2 at 70.6 vs. AgentScaler at 70.2, so Kimi-K2 is slightly ahead). It exceeds GPT-o3 (80.2? — no, GPT-o3 at 80.2 is actually much higher, I'm reading the wrong column). Let me re-examine: τ2-Bench Retail column shows AgentScaler-30B-A3B at 70.2, GPT-o3 at 80.2, GPT-5-think at 81.1, Gemini-2.5-pro at 67.5, Claude-Sonnet-4 at 67.5, GPT-o4-mini at 70.2, GPT-OSS at 57.0, Deepseek-V3.1 at 64.9, Kimi-K2 at 70.6, Qwen3-Thinking-235B at 71.9, Seed-OSS-36B at 68.4, Qwen-Coder-30B at 60.5, and the base Qwen3-Thinking-30B-A3B at 58.8. So AgentScaler-30B-A3B (70.2) is competitive with Kimi-K2 (70.6) and GPT-o4-mini (70.2) but trails GPT-o3 (80.2), GPT-5-think (81.1), and Qwen3-Thinking-235B (71.9). This is a more moderate result than the Airline domain. AgentScaler-4B achieves 62.3 and AgentScaler-8B achieves 58.8 on τ2-Bench Retail.

• Airline domain: AgentScaler-30B-A3B achieves 60.0, which ties with — wait, let me check: GPT-o3 is 64.8, GPT-5-think is 62.6, Gemini-2.5-pro is 56.0, Claude-Sonnet-4 is 54.0, GPT-o4-mini is 56.0. AgentScaler at 60.0 exceeds most closed-source models except GPT-o3 and GPT-5-think (which is at 62.6, only 2.6 points ahead). Among open-source models, it exceeds Kimi-K2 (56.5), Qwen3-Thinking-235B (58.0), and the base Qwen3-Thinking-30B-A3B (58.0). This is a strong result.

• Telecom domain: AgentScaler-30B-A3B achieves 55.3. Closed-source models show: GPT-5-think at 96.7 (dramatically higher — this enormous gap on the Telecom domain is noteworthy), GPT-o3 at 58.2, Claude-Sonnet-4 at 47.4, GPT-o4-mini at 46.5, Gemini-2.5-pro at 27.2. So AgentScaler exceeds GPT-o4-mini (46.5), Claude-Sonnet-4 (47.4), and Gemini-2.5-pro (27.2), but trails GPT-o3 (58.2) and is far behind GPT-5-think (96.7). Among open-source models, Kimi-K2 achieves 65.8 (exceeding AgentScaler), while Qwen3-Thinking-235B achieves 45.6 (below AgentScaler), and the base Qwen3-Thinking-30B-A3B achieves only 26.3. The jump from 26.3 (base) to 55.3 (AgentScaler) on Telecom represents a gain of 29.0 points — the largest absolute improvement of any model variant on any domain.

• Weighted overall (across Retail, Airline, and Telecom, presumably weighted by the number of test cases in each domain): AgentScaler-30B-A3B achieves — the passˆk figure (Figure 4) shows the passˆ1 score numerically, with the "Weighted Overall" at passˆ1 for AgentScaler at 62.5 and for Qwen3-Thinking-30B-A3B at 45.3. This represents a 17.2 percentage point improvement from base to trained model. However, Table 1 does not provide a weighted overall for the full model set — the weighted overall only appears in Figure 4 for the passˆk analysis comparing AgentScaler to its base model, not in the main results table comparing against all baselines.

On ACEBench-en (Table 1):

• Overall: AgentScaler-30B-A3B achieves 75.7. Closed-source comparisons: Gemini-2.5-pro at 78.2, Claude-Sonnet-4 at 76.1, GPT-o3 at 78.2, GPT-o4-mini at 77.9, GPT-5-think at 72.2. So AgentScaler (75.7) slightly trails the leading closed-source models (~78) but exceeds GPT-5-think (72.2), which is notable since GPT-5-think is considered a top-tier reasoning model. Among open-source models, AgentScaler-30B-A3B (75.7) slightly trails GPT-OSS-120B-A5B (76.0) and Kimi-K2-1T-A32B (77.4), and is essentially tied with Seed-OSS-36B (76.7 — that's actually 1 point higher). The base Qwen3-Thinking-30B-A3B achieves 67.2, so the training pipeline yields an 8.5 point improvement.

• Agent subset (most challenging): AgentScaler-30B-A3B achieves 60.0. This is significantly above the base model (42.8, a 17.2 point gain) and competitive with GPT-o3 (63.3), GPT-o4-mini (60.0, tie), and Kimi-K2 (65.0). It substantially exceeds Claude-Sonnet-4 (42.5) and Gemini-2.5-pro (63.4 — wait, that's higher). So on the hardest ACEBench subset, AgentScaler-30B-A3B is competitive with but generally a few points behind the best closed-source and the largest open-source models.

• AgentScaler-4B achieves 65.9 Overall on ACEBench-en, which exceeds the base Qwen3-8B (65.9 — interestingly the same score), Qwen3-Thinking-4B (49.5, a 16.4 point gain), and all xLAM-2 variants except — actually, xLAM-2-70B-fc-r achieves only 36.5 Overall, so AgentScaler-4B dramatically exceeds all xLAM models at much smaller parameter counts. This is the core efficiency claim: a 4B model achieving competitive or superior performance to 30B+ models.

• AgentScaler-8B achieves 67.4 Overall, which is slightly above AgentScaler-4B (65.9) but below AgentScaler-30B-A3B (75.7). The performance ordering across scales (4B→8B→30B) is monotonic in Overall score (65.9 → 67.4 → 75.7), which provides validation that the training pipeline benefits from scale and is not saturating at the smaller model sizes.

Cross-Lingual Generalization (Table 2)

The paper evaluates AgentScaler models on ACEBench-zh, the Chinese-language variant of ACEBench, to test out-of-distribution generalization. Since the training data is presumably English-dominant (given the API sources are English-language documentation from ToolBench, API-Gen, and internal repositories), ACEBench-zh represents a genuinely OOD test scenario — different language, different cultural context for tool-use patterns, different phrasing of user intents.

Key findings from Table 2:

• AgentScaler-4B vs. Qwen3-Thinking-4B: Overall score improves from 43.9 to 65.6 (+21.7 points), a dramatic gain. The Agent subset jumps from 6.7 to 38.4 (+31.7 points), suggesting that the training pipeline imparts rudimentary agentic capabilities even to a 4B model on Chinese-language tasks where the base model was almost completely unable to handle multi-step tool use (6.7 is near-random for the Agent subset). The Normal subset improves from 34.7 to 70.8 (+36.1 points). However, the Special subset decreases from 85.3 to 70.0 (-15.3 points) — a notable regression on the Special tasks that the paper does not analyze or explain.

• AgentScaler-8B vs. Qwen3-8B: Overall improves from 71.3 to 73.7 (+2.4 points), a much smaller gain than the 4B model. The Agent subset improves from 35.0 to 58.4 (+23.4 points), showing significant gains on complex tasks. The Normal subset decreases from 80.3 to 75.2 (-5.1 points), another regression on simpler tasks. The Special subset improves from 72.7 to 79.3 (+6.6 points).

• AgentScaler-30B-A3B vs. Qwen3-Thinking-30B-A3B: Overall improves from 74.2 to 81.5 (+7.3 points). Normal improves from 73.4 to 85.3 (+11.9). Agent improves from 55.8 to 64.1 (+8.3). Special decreases from 86.7 to 83.3 (-3.4 points).

A consistent pattern emerges across all scales: the training pipeline produces large gains on the Agent subset (complex multi-step tool use) and on the Normal subset (for 4B and 30B scales), but regresses on the Special subset at two of three scales (4B and 30B). The Special subset regression is unexplained in the paper but is an important finding — it suggests that the training pipeline may trade off some capability on specialized or unusual tool-use patterns in exchange for improved general agentic competence. Without further analysis, it's unclear whether this represents catastrophic forgetting of the base model's Special-subset capabilities or some other phenomenon.

The absolute gains are largest for the smallest model (4B: +21.7 overall), moderate for the 30B model (30B: +7.3 overall), and smallest for the 8B model (8B: +2.4 overall). This pattern — where the weakest base model benefits most from the training — is consistent with the interpretation that the training pipeline imparts foundational agentic capabilities that the 4B base model lacked entirely but that the 8B and 30B base models already partially possessed.

Consistency and Stability Analysis (Figure 4)

The paper analyzes model stability through the passˆk metric on τ2-Bench, where k ranges from 1 to 4 (Figure 4). The passˆk metric measures the fraction of tasks where the model succeeds in all k independent attempts — it is a stricter measure than passˆ1 because it requires consistent correctness rather than occasional success.

Key findings:

• AgentScaler-30B-A3B consistently outperforms Qwen3-Thinking-30B-A3B at every passˆk level across all domains and the weighted overall. At passˆ1 (standard accuracy), the weighted overall is 62.5 for AgentScaler vs. 45.3 for Qwen3-Thinking-30B-A3B (a 17.2-point gap). At passˆ4 (succeeding on all 4 attempts), the weighted overall drops to 30.6 for AgentScaler vs. 27.7 for Qwen3-Thinking-30B-A3B (a much narrower 2.9-point gap). This suggests that while AgentScaler has substantially higher success rates on individual attempts, its consistency across repeated attempts converges toward the base model's performance as k increases.

• A clear downward trend in scores is observed as k increases for both models across all domains. For AgentScaler-30B-A3B on the weighted overall: passˆ1 = 62.5 → passˆ2 = 48.6 → passˆ3 = 38.5 → passˆ4 = 30.6. This represents roughly a halving of the score from k=1 to k=4, indicating that the model's failures on any given task are not purely random — if a task is failed once, it has a substantial probability of being failed again, suggesting systematic failure modes rather than stochastic errors. The baseline Qwen3-Thinking-30B-A3B shows a similar proportional decline: 45.3 → 34.1 → 30.9 → 27.7, a flatter curve suggesting that the base model's errors are more consistent (already failing most tasks, so repeated attempts don't reveal much additional inconsistency).

• Domain-specific patterns: The Retail domain shows the highest passˆ1 for AgentScaler (70.2) and retains the highest passˆ4 (41.2), suggesting retail tasks are both easier and more consistently solvable. The Telecom domain shows the lowest passˆ1 (55.3) and the steepest decline to passˆ4 (22.8 — less than half the passˆ1 score), indicating telecom tasks are not only harder but also less consistently solvable when they are solved.

The authors note that "the stability of existing LLMs remains a considerable challenge" — the fact that even a specialized agent model sees its passˆ4 score drop by more than 50% compared to passˆ1 indicates substantial room for improvement in reliability. This finding connects to the long-horizon analysis (Figure 5) in suggesting that current models have difficulty maintaining consistent performance, whether measured across repeated attempts on the same task or across extended tool-use chains within a single task.

Long-Horizon Tool-Calling Analysis (Figure 5)

The paper provides a scatter plot and trend analysis of tool call count vs. task accuracy on τ-bench, broken out by domain (Retail vs. Airline).

Key findings:

• There exists a clear negative correlation between the number of tool calls in a trajectory and the corresponding trajectory accuracy. The dashed trend line for Retail and the dotted trend line for Airline both show a downward slope. This is consistent with the intuitive expectation that longer tool-use chains are more difficult — more steps mean more opportunities for error, more complex state tracking requirements, and more cumulative probability of a mistake in tool selection or argumentation.

• The relationship is quantified through bubble size in the scatter plot, where bubble size indicates sample size. The paper provides specific accuracy values for different tool call counts: for Retail, accuracy ranges from ~82% at 2 tool calls, ~78% at 4 tool calls, ~54% at — the numbers in the figure legend indicate specific data points: for Retail, accuracies of 82% (2 calls), 78% (?), 54% (6 calls?), 90% (some call count), 80%, 82%, 40%, 57%, 0%, 100%, 100%, 0%, 100%, 67%, 70%, 33%, 0%, 0%, 33%, 33%, 67%, 50%, 0%, 0%, 100%. For Airline: 82%, 78%, 54%, 90%, 80%, 82%, 40%, 57%, 0%, 100%, 100%, 0%, 100%, 67%, 70%, 33%, etc. — the data is presented as raw points in the figure rather than aggregated statistics, making precise characterization difficult from the text alone. The overall negative trend is visually clear but not reduced to a correlation coefficient or regression slope.

• Small sample size bubbles appear predominantly at higher tool call counts, which introduces a caveat: the negative correlation may be partially explained by the fact that very long tool-use trajectories are rare in the test set, and the few that exist may be unusually difficult for reasons beyond their length (e.g., they involve rare edge cases or complex multi-domain interactions).

The paper acknowledges this finding as evidence that "handling extended tool-use chains is still an open problem." This is one of the strongest negative results in the paper: despite the extensive training pipeline and strong aggregate benchmark performance, the models still exhibit systematic degradation as interaction complexity increases. This finding is not ablated — the paper does not compare long-horizon performance of AgentScaler against baseline models to determine whether the training pipeline improves long-horizon capabilities (even if they remain imperfect) or whether the slope of degradation is the same across models.

Ablation Studies and Robustness Checks

Two-stage training ablation on ACEBench-en (Figure 3): The paper ablates the two-stage training framework by comparing the base model (Qwen3-Thinking-30B-A3B), Stage 1 training only, and Stage 2 training (full pipeline) on ACEBench-en. Across all subsets:

• Normal subset: Base = 64.7 → Stage 1 ≈ 74 (estimated from bar chart) → Stage 2 ≈ 77 (estimated). Stage 1 provides a large gain (~9 points), Stage 2 adds a further ~3 points.
• Special subset: Base = 86.7 → Stage 1 ≈ 83 (estimated, a slight decrease) → Stage 2 ≈ 83 (flat). The Special subset shows little improvement and possibly a small regression from base, consistent with the cross-lingual finding that Special tasks are not well-served by the training pipeline.
• Agent subset: Base = 42.8 → Stage 1 ≈ 52 (estimated) → Stage 2 ≈ 60. The largest absolute gain is on the Agent subset (~17 points from base to Stage 2), with both stages contributing meaningfully.
• Overall: Base = 67.2 → Stage 1 ≈ 72 (estimated) → Stage 2 = 75.7 (from Table 1). The Overall score shows monotonic improvement, with Stage 1 providing the majority of the gain and Stage 2 providing additional refinement.

The ablation supports the paper's claim that "both Stage 1 and Stage 2 training substantially improve performance over the base model" and that "general foundation learning is critical for establishing tool-usage competence, and subsequent domain-specialization further consolidates and contextualizes these capabilities." However, the ablation does not compare against a single-stage approach that mixes general and domain-specific data in comparable proportions, which would be the proper test of whether the two-stage architecture is necessary or merely a convenient curriculum. The paper does not report a "Stage 2 only" ablation (domain-specific training without general foundation), which would isolate the contribution of Stage 1. The Special subset's lack of improvement (and slight regression) is not discussed in the text, representing an unexplained negative result.

Model scale ablation (implicit in Table 1 and Table 2): The paper does not explicitly label this as an ablation, but the training of three model sizes (4B, 8B, 30B-A3B) using the same pipeline serves as a scale ablation. Key observations:

• On ACEBench-en Overall, the 8B model (67.4) achieves only marginally better performance than the 4B model (65.9), a 1.5-point gain for 2× the parameters. The jump to 30B-A3B (75.7) provides a much larger 8.3-point gain over 8B. This suggests diminishing returns at the small-to-medium scale and a possible threshold effect where substantial agentic competence requires a minimum parameter count that 4B and 8B models approach but do not fully cross.
• On τ-bench Airline, the unusual pattern (AgentScaler-4B at 54.0, AgentScaler-30B-A3B also at 54.0, but AgentScaler-8B at only 42.0) suggests non-monotonic scaling behavior that could indicate variance in training data quality or composition across model sizes, or genuine non-monotonicity in how agentic capabilities emerge with scale.
• On ACEBench-zh Overall, the gains from training decrease with model size: +21.7 for 4B, +2.4 for 8B, +7.3 for 30B-A3B. This inconsistent pattern (8B gains less than 30B) suggests that the effectiveness of the training pipeline is not a simple function of model scale.

Base model variant ablation (implicit in model selection): The paper uses Qwen3-Thinking variants for the 4B and 30B-A3B models but the base Qwen3-8B (non-thinking) for the 8B model. The choice is not explicitly ablated or justified beyond the authors' selection of specific Qwen3 checkpoints. This introduces a confound: the 8B model's relatively weak performance compared to the 4B model on some metrics might partially reflect the difference between thinking and non-thinking base models, not just the training pipeline's interaction with model scale. A proper ablation would compare thinking vs. non-thinking variants at the same parameter count.

Cross-lingual transfer (Table 2): While not explicitly framed as an ablation, the evaluation on ACEBench-zh serves as a robustness check for out-of-distribution generalization. The consistent pattern of gains on the Agent subset across all scales (+31.7 for 4B, +23.4 for 8B, +8.3 for 30B) provides evidence that the training pipeline imparts genuinely transferable agentic skills rather than English-specific surface patterns. However, the regressions on the Special subset (4B: -15.3, 30B: -3.4) and the Normal subset regression for the 8B model (-5.1) indicate that transfer is not uniformly positive — some base model capabilities are degraded by the training, consistent with catastrophic forgetting of pre-existing skills that are not reinforced in the agentic training data.

Critical Assessment

Claim 1: "Systematic environment scaling enables automated construction of 1,000+ fully simulated, verifiable tool-use environments."

What was tested: The paper describes a pipeline that processes over 30,000 APIs and produces over 1,000 domains, but it does not quantitatively validate the quality of these environments beyond two pieces of evidence: (a) a qualitative note that generated environments "exhibit a high degree of consistency" with human-designed τ-bench environments for overlapping domains (Section 2.1), and (b) the downstream agent performance on τ-bench, τ2-Bench, and ACEBench, which is an indirect measure of environment quality mediated by model training.

What was NOT tested: There is no direct measurement of environment fidelity — no comparison of generated tool implementations against reference implementations, no measurement of how often the induced database schemas match ground-truth schemas, no analysis of how many of the 1,000+ domains produce coherent and useful training tasks versus degenerate or nonsensical ones. The claim that environments are "verifiable" is demonstrated for the downstream evaluation benchmarks (where ground truth exists), but the verifiability of environments that are not manually verified is implicit — the pipeline asserts verifiability by construction, but this assertion is not empirically validated.

Assessment: The claim is supported in principle (the pipeline architecture is sound and the results on known benchmarks are strong) but under-validated for the 1,000+ domain scale. The paper demonstrates environment construction for the small number of domains that happen to align with standard benchmarks, but provides no evidence about the quality of the remaining ~990 domains. A more convincing validation would include: (a) human evaluation of a random sample of generated environments for coherence and correctness, (b) training an agent exclusively on generated environments and evaluating on a held-out reference environment to measure domain transfer, or (c) reporting the fraction of generated environments that pass some automated sanity check (e.g., all tools in the domain compile and execute without errors, tool sequences sampled from the dependency graph produce valid database states at least X% of the time). The absence of such validation makes the "1,000+ domains" claim more of an existence proof than a demonstrated capability.

Claim 2: "AgentScaler models achieve state-of-the-art performance among open-source models under 1T parameters."

What was tested: Table 1 provides comprehensive benchmark results across three major agentic benchmarks, comparing AgentScaler-30B-A3B against a broad set of open-source models spanning 36B to 1T+ parameters.

What was NOT tested: The paper does not provide error bars, confidence intervals, or statistical significance tests. Many of the comparisons involve small differences (1–3 percentage points) that could easily fall within sampling variance given the test sets' likely sizes (τ-bench retail has around 115 test tasks, τ2-Bench has unknown but likely similar or smaller test set sizes). For example, on τ2-Bench Retail, AgentScaler-30B-A3B achieves 70.2, Kimi-K2 achieves 70.6, and GPT-o4-mini achieves 70.2 — these are functionally indistinguishable without variance estimates. The claim of "state-of-the-art" is further complicated by the absence of several competitive open-source agentic models that existed at the time of the paper's likely preparation (the paper does not compare against, for instance, fine-tuned versions of Llama-3 or Mistral models specialized for function calling, though the xLAM-2 series partially covers this space).

Assessment: The claim is supported with qualifications. AgentScaler-30B-A3B is clearly in the top tier of open-source models under 1T parameters, and it dominates the xLAM-2 series at comparable parameter counts by large margins. However, the "state-of-the-art" designation is fragile — it depends on the specific set of benchmarks, the absence of statistical characterization, and the specific model release dates (the field is moving fast, and what is SOTA at paper submission may not be at publication). The more defensible claim — which the paper also makes — is that AgentScaler models are competitive with much larger models while using substantially fewer parameters, which is well-supported by the consistent pattern of AgentScaler-30B-A3B matching or approaching models with 10×–100× more parameters.

Claim 3: "AgentScaler-30B-A3B delivers results on par with trillion-parameter and closed-source systems."

What was tested: Direct score comparisons in Table 1.

What was NOT tested: There is no matched-budget comparison. A trillion-parameter model presumably costs far more to run at inference time than a 30B model. If "on par" means "similar accuracy at massively lower inference cost," this is a genuine achievement. If "on par" means "similar accuracy, ignoring inference cost," this is a less meaningful comparison — the trillion-parameter model might achieve its score with dramatically higher latency and cost, or it might reach that score with greedy decoding while AgentScaler requires extensive multi-turn tool interactions. The paper does not report latency, cost, or inference-time compute for any model, making the practical significance of the accuracy-parity claim unclear.

Additionally, the closed-source comparisons are single-point comparisons against specific model versions available at the time of writing. Closed-source models are updated frequently, and the specific versions tested (GPT-5-think, GPT-o3, etc.) may have been superseded or may exhibit different performance characteristics under different prompting regimes.

Assessment: The claim is partially supported. The accuracy numbers in Table 1 genuinely show AgentScaler-30B-A3B within striking distance of the best closed-source models on several benchmarks (e.g., within 2.5 points of GPT-o3 on ACEBench-en Overall, exceeding GPT-5-think on ACEBench-en Overall by 3.5 points). However, the comparison is asymmetric: closed-source models are evaluated "off-the-shelf" with their default prompting strategies, while AgentScaler models benefit from extensive domain-specific training on the same types of tasks they are evaluated on. A more symmetric comparison would fine-tune the closed-source models on the same training data (which is not possible for API-only models) or evaluate AgentScaler in a zero-shot setting on held-out tasks more distant from the training distribution. The paper partially addresses this through the ACEBench-zh evaluation (which is cross-lingual OOD), but the domain overlap (retail, airline, etc.) between training and evaluation is substantial.

Claim 4: "Two-stage training (general foundation → domain specialization) is more effective than single-stage training."

What was tested: The ablation in Figure 3 compares base model → Stage 1 only → Stage 2 (full pipeline), showing monotonic improvement.

What was NOT tested: This ablation does not compare against a single-stage model trained on the combined data from Stage 1 and Stage 2 in a single phase. The proper test of "two-stage is better than single-stage" would train a model on the same total data (general + domain-specific) in a single run, with hyperparameters tuned for that setup, and compare against the two-stage model. The current ablation only shows that adding more training data (Stage 2 after Stage 1) helps — not that the two-stage curriculum is necessary or more efficient than a single-stage approach. Additionally, the paper does not report a "Stage 2 only" ablation (domain-specific training without general foundation), which would isolate the contribution of Stage 1 and test whether the general foundation is truly necessary or whether domain-specific data alone would suffice.

Assessment: The claim is weakly supported. The ablation demonstrates that Stage 2 training improves over Stage 1-only training, which is expected if the Stage 2 data contains useful signal. It does not demonstrate that the sequencing (general before specific) matters, or that the two-stage approach outperforms alternatives. The paper's claim about the two-stage design is more of a plausible hypothesis supported by a positive result than a rigorously tested finding. A stronger experimental design would include the single-stage baseline and the Stage 2-only baseline.

Claim 5: "Long-horizon tool calling remains a fundamental challenge" and "handling extended tool-use chains is still an open problem."

What was tested: The scatter plot in Figure 5 shows a negative correlation between tool call count and accuracy on τ-bench for AgentScaler models.

What was NOT tested: The paper does not compare this correlation against baseline models to determine whether AgentScaler improves long-horizon performance (even if it remains imperfect) or whether the slope of degradation is the same across all models. It does not analyze why long-horizon performance degrades — is it due to accumulated tool-call errors, context window degradation, loss of task coherence, or something else? It does not measure whether the degradation is smooth (each additional tool call adds a constant error probability) or exhibits threshold effects (performance collapses after N tool calls). It does not test interventions that might improve long-horizon performance (e.g., explicit state tracking, hierarchical planning, verification checkpoints).

Assessment: The claim is supported as a qualitative observation but under-analyzed for a paper that presents it as a key finding. The negative correlation is clear from Figure 5, establishing that the problem exists. But without comparison to baselines, without mechanistic analysis, and without exploration of countermeasures, the finding is more of a motivation for future work than a contribution of the current paper. The paper's own recommendation to address this in future work (Section 5) is appropriate but also highlights the preliminary nature of this analysis.

Genuine Weaknesses in the Experimental Design

1. No statistical characterization. All results in Table 1, Table 2, and Figure 3 are point estimates without confidence intervals, standard deviations, or significance tests. Given that many comparisons involve differences of 1–3 percentage points on test sets of unknown but likely moderate size (τ-bench retail has ~115 test tasks based on prior work, τ2-Bench likely has similar per-domain sizes), the ranking of closely-performing models is unreliable. This is a standard limitation of benchmark evaluation papers and does not invalidate the broad patterns (AgentScaler is clearly much better than the base models and competitive with much larger models), but it undermines the specific "state-of-the-art" claims at fine granularity.

2. Single base model family. All AgentScaler models are trained on Qwen-3 variants. The paper provides no evidence that the training pipeline would improve agentic capabilities for models from other families (Llama, Mistral, DeepSeek, etc.). While the Qwen-3 base models show substantial gains from the training, it's unknown whether these gains reflect a property of the training pipeline or a synergy between the pipeline and Qwen-3's specific pretraining characteristics (e.g., instruction-following ability, code generation competence, multilingual capabilities).

3. The "1,000+ domain" claim is under-validated. Only a handful of domains (those overlapping with τ-bench, τ2-Bench, and ACEBench) are empirically validated through downstream task performance. The quality of the remaining ~990+ domains is unknown. The paper does not report: how many of the 1,000+ domains compile and execute without errors, what fraction of sampled tool sequences produce valid database state transitions, whether the induced schemas are internally consistent, or whether human evaluators judge the generated tasks as realistic. This is a significant gap for a paper whose central contribution is automated environment scaling.

4. No comparison against alternative data generation methods. The paper trains exclusively on its own pipeline's data and compares against models trained on other (unspecified) data. It does not conduct a controlled experiment where, for example, the same Qwen-3 base model is fine-tuned on: (a) AgentScaler pipeline data, (b) reverse-paradigm data (Yin et al., 2025), (c) manually written agentic data, or (d) a mix. Such comparisons would isolate the contribution of the data generation pipeline specifically, versus the contribution of simply having more agentic training data of any kind. The current design conflates "our data pipeline helps" with "more agentic training data helps."

5. The base model for the 8B variant is inconsistent with the 4B and 30B variants. AgentScaler-4B and AgentScaler-30B-A3B use thinking-model base checkpoints (Qwen3-Thinking-4B-2507 and Qwen3-Thinking-30B-A3B-2507), while AgentScaler-8B uses the non-thinking Qwen3-8B. This confound makes scale comparisons unreliable — the non-monotonic scaling behavior (8B sometimes worse than 4B) could be due to the thinking vs. non-thinking difference rather than genuine scale effects.

6. Missing key ablations. Beyond the issues already noted:

• No ablation on the amount of Stage 1 data — how much general-domain diversity is needed before returns diminish?
• No ablation on the Louvain community detection threshold — how sensitive are the domain partitions to the similarity threshold τ?
• No ablation on the filtering stages — what is the contribution of each stage to final model quality? Does Stage 3 (exact match) actually improve over Stage 1+2 only?
• No ablation on retaining error-recovery trajectories — does this design choice actually improve robustness, and by how much?
• No ablation on the simulated user model — does the quality of the user simulator matter for downstream agent performance?

7. The long-horizon analysis (Figure 5) lacks quantitative rigor. The scatter plot provides a visual negative correlation but no correlation coefficient, regression slope, or statistical test. The sample sizes at high tool call counts appear small based on bubble sizes in the figure, making the trend at the long-horizon extreme unreliable. Without comparing against baseline models, it's unclear whether AgentScaler shifts the accuracy-vs-length curve upward (same slope, better intercept — meaning it helps at all lengths equally) or changes the slope (meaning it specifically helps or hurts at longer horizons).

Experiments That Would Have Strengthened the Paper

• Human evaluation of generated environment quality on a random sample of domains, measuring whether the tools, schemas, and tasks are coherent and realistic.
• Single-stage vs. two-stage controlled comparison on the same total data, to test whether the curriculum itself matters.
• Cross-model-family replication to test whether the training pipeline generalizes beyond Qwen-3.
• Data scaling curves — training models on 10%, 25%, 50%, 75%, and 100% of the generated data to characterize whether performance is saturating or still improving with more environments.
• Ablation of the number of domains — does training on 100 domains vs. 500 domains vs. 1,000+ domains produce monotonic improvement, or do returns diminish?
• Filtering ablation — train separate models with each filtering stage omitted to measure the marginal value of each filtering criterion.
• Inference cost measurement — report latency, token counts, and estimated FLOPs for AgentScaler models vs. baseline models to contextualize the "efficiency" claims.
• Dedicated long-horizon evaluation with controlled task lengths to precisely characterize the tool-count-vs-accuracy relationship, including comparison against baseline models and analysis of failure modes at different lengths.

6. Limitations and Trade-offs

Limitation 1: Environment Quality Is Validated Only for a Tiny Fraction of the Claimed 1,000+ Domains

The assumption or constraint. The paper's central claim is that its automated pipeline constructs over 1,000 fully simulated, verifiable tool-use environments from 30,000+ APIs. However, the only domains for which environment quality is empirically validated are those that happen to overlap with the evaluation benchmarks: the retail, airline, and telecom domains from τ-bench and τ2-Bench, plus whatever domains ACEBench covers. The paper reports a qualitative observation about consistency with τ-bench (Section 2.1), but this covers perhaps 3–5 domains out of 1,000+. For the remaining ~995 domains, there is no measurement of whether the induced database schemas are internally consistent, whether the generated tool implementations execute without errors, whether sampled tool sequences produce valid state transitions, or whether human evaluators would judge the resulting tasks as coherent and realistic.

The consequence. The entire training pipeline depends on the quality of these environments — if a substantial fraction of the 1,000+ domains produce degenerate, inconsistent, or nonsensical tasks, then the training data is contaminated with noise that could harm rather than help agentic capability development. The paper's claim of "systematic environment scaling" is unsupported at the claimed scale. A practitioner adopting this approach for a new domain cannot know whether the automatically constructed environment will be usable without investing in manual verification — which undermines the very scalability argument the paper makes. Furthermore, the paper provides no diagnostic for detecting bad environments automatically, so there is no mechanism for filtering out low-quality domains before they enter the training data.

What evidence exists in the paper. The qualitative note in Section 2.1: "when generating database structures and formalizing code within specific domains of τ-bench, we observe through manual inspection that our outputs exhibit a high degree of consistency with the official implementations provided by τ-bench." This is the only direct validation of environment quality. The strong downstream benchmark results (Table 1) provide indirect evidence that the pipeline works for the domains it was evaluated on, but these benchmarks cover a tiny subset of the constructed environments. The paper reports no metrics on environment quality for the broader domain set — no compilation success rate, no state-transition validity rate, no human evaluation, no automated sanity checks on the 1,000+ domains.

Mitigation status. The paper does not address this limitation. It does not propose or implement any automated environment quality assessment, does not report the fraction of domains that pass basic sanity checks, and does not analyze whether downstream model performance correlates with manually-verified vs. unverified environment domains. The limitation is not acknowledged in the Limitations section of the paper, which focuses instead on the absence of reinforcement learning and model scale constraints.

Limitation 2: The Two-Stage Curriculum Is Not Tested Against Single-Stage or Stage-2-Only Baselines

The assumption or constraint. The paper proposes a two-stage training framework where Stage 1 builds general tool-use competence across broad domains and Stage 2 specializes the agent for specific vertical contexts. The ablation in Figure 3 compares the base model against Stage 1 only and Stage 2 (the full pipeline), showing monotonic improvement. However, this ablation does not compare the two-stage approach against the most natural alternatives: (a) a single-stage model trained on the combined Stage 1 + Stage 2 data in one phase, which would test whether the curriculum itself matters or merely the total data volume; and (b) a Stage 2-only model trained exclusively on domain-specific data, which would test whether the general foundation in Stage 1 is genuinely necessary or whether domain-specific data alone suffices.

The consequence. The paper claims that "general foundation learning is critical for establishing tool-usage competence, and subsequent domain-specialization further consolidates and contextualizes these capabilities" (Section 3.2). This claim is not supported by the experimental design. It is equally consistent with the evidence that: (a) training on any additional agentic data (whether general or domain-specific, in any order) improves performance; (b) the domain-specific data in Stage 2 is all that matters, and Stage 1 merely provides a modest initialization benefit; or (c) the two stages provide complementary benefits that could be achieved equally well by mixing the data in a single phase. A practitioner implementing this approach cannot know whether the two-stage architecture is worth the additional engineering complexity (maintaining separate training phases, tuning two sets of hyperparameters, deciding when to transition between stages) versus simply training on all available data in one pass.

What evidence exists in the paper. Figure 3 (the only ablation of the training framework) shows base → Stage 1 → Stage 2 for a single model (Qwen3-Thinking-30B-A3B) on ACEBench-en. The paper does not report Stage 2-only performance, single-stage combined performance, or any variant where the ordering or mixing of general and domain-specific data is varied. There is no data scaling analysis showing whether Stage 1 data continues to provide benefits as more domain-specific data is added, or whether Stage 1 benefits are eventually subsumed by sufficient domain-specific training.

Mitigation status. The paper does not acknowledge this as a limitation. The ablation design is presented as sufficient evidence for the two-stage framework, but it tests the wrong comparison. The paper does not suggest future work on optimizing the training curriculum or comparing alternative data mixing strategies. This is a methodological weakness that affects the strength of one of the paper's claimed innovations (see Section 4, Innovation 3 in the prior sections).

Limitation 3: No Inference Cost or Latency Accounting Despite Deployment-Oriented Claims

The assumption or constraint. The paper positions AgentScaler as particularly well-suited for "practical deployment in resource-constrained or latency-sensitive scenarios" (Section 4.2) and endorses the view that "small language models are the future of agentic AI" (Limitations section). These claims rest on the assumption that smaller parameter count translates directly to deployment efficiency. However, the paper provides no measurement of inference cost, latency, or computational budget for any model — neither for AgentScaler variants nor for the baseline models they are compared against. Agentic tasks require multi-turn interactions with multiple tool calls per turn. A 4B model that requires 15 tool calls to complete a task may consume more total inference compute and wall-clock time than a 30B model that completes the same task in 4 tool calls, even though the per-token cost of the 4B model is lower.

The consequence. The paper's efficiency claims are based on a single dimension (parameter count) while ignoring the other dimension that determines real-world cost: number of tokens generated per task. If AgentScaler models achieve their strong benchmark performance by generating more tool calls, more conversational turns, or longer reasoning traces than larger baseline models, then the practical deployment advantage may be smaller than parameter-count comparisons suggest — or may even reverse. Additionally, the simulated human–agent interplay used for training involves multiple rounds of interaction, and the filtering pipeline may preferentially retain longer trajectories (since longer interactions provide more opportunities for the agent to eventually reach the correct final state). This could bias the training data toward interaction patterns that are more expensive at inference time.

What evidence exists in the paper. The paper reports no token counts, no latency measurements, no FLOPs estimates, and no analysis of average trajectory length for AgentScaler models versus baselines. The passˆ1 metric (used for τ-bench and τ2-Bench) measures whether the model eventually succeeds within a single interaction, but does not distinguish between an agent that succeeds in 2 turns versus one that succeeds in 20 turns. The long-horizon analysis in Figure 5 shows that accuracy decreases with tool call count, but does not report the distribution of tool call counts for AgentScaler versus baseline models — it is possible that AgentScaler achieves higher accuracy by being more thorough (making more tool calls) rather than more efficient (succeeding with fewer calls). The paper's claim about deployment suitability is therefore an assertion based on parameter count alone, with no empirical grounding in actual inference costs.

Mitigation status. The paper does not address this limitation. The Limitations section focuses on the absence of reinforcement learning and model scale constraints, but does not mention the absence of inference cost measurement. A deployment-oriented paper making efficiency claims should minimally report average tokens per task and average number of tool calls for its models versus baselines, and ideally report wall-clock latency on comparable hardware.

Limitation 4: The Long-Horizon Tool-Calling Analysis Is Descriptive Rather Than Diagnostic or Comparative

The assumption or constraint. The paper identifies long-horizon tool calling as "a fundamental challenge" and "an open problem" (Section 5), supported by the scatter plot in Figure 5 showing a negative correlation between tool call count and task accuracy. The paper implicitly assumes that this finding is informative for understanding AgentScaler's capabilities and limitations. However, the analysis is purely descriptive: it shows that accuracy drops as tool call count increases for AgentScaler models, but does not compare this degradation against baseline models or analyze the mechanisms underlying the degradation.

The consequence. Several critical questions are left unanswered: Is AgentScaler better at long-horizon tool calling than baseline models (i.e., is the accuracy-vs-length curve shifted upward, indicating improvement at all lengths), or does AgentScaler merely achieve higher aggregate accuracy by performing better on short-horizon tasks while degrading at the same rate on long-horizon ones? Is the degradation caused by cumulative tool-call errors (each call has an independent error probability), context-window saturation (the model loses track of earlier information as the conversation grows), task coherence drift (the model forgets the original user intent), or something else? Without answering these questions, the finding that "long-horizon tool calling remains challenging" is not actionable — it does not tell a practitioner what to fix or what kinds of tasks to avoid deploying AgentScaler on.

What evidence exists in the paper. Figure 5 provides a scatter plot of tool call count versus accuracy for AgentScaler models on τ-bench, broken out by Retail and Airline domains. The trend lines show negative slopes. However, there is no equivalent analysis for baseline models (the base Qwen-3 models, xLAM-2, or closed-source systems), making it impossible to determine whether the degradation slope is specific to AgentScaler or universal across all agentic models. The paper does not report a correlation coefficient, regression statistics, or any quantitative characterization of the relationship. Bubble sizes in the scatter plot indicate sample sizes at each tool call count, but the paper does not report how many tasks exist at each length or discuss the reliability of the trend at high tool call counts where samples are sparse.

Mitigation status. The paper acknowledges this as an open problem and states an intention to address it in future work (Section 5). However, the current analysis is too preliminary to serve as a meaningful contribution — it identifies a problem that is intuitively expected (longer tasks are harder) without characterizing how the proposed method affects it relative to alternatives. A minimal strengthening would include baseline comparisons, quantitative correlation measures, and an analysis of failure modes at different task lengths (e.g., categorize errors as wrong-tool, wrong-argument, premature-termination, or user-misunderstanding, and show how the distribution of error types changes with task length).

Limitation 5: The Difficulty Estimation for the Filtering Pipeline Relies on Oracle Access to Ground-Truth States and Tool Sequences

The assumption or constraint. The three-stage funnel-based filtering framework (Section 3.1) is the mechanism that ensures training data quality. Stage 2 (Environment State Alignment) requires the gold final database state $D_{final}^{gold}$ to compare against the agent's final state. Stage 3 (Function Calling Exact Match) requires the gold tool sequence and arguments. Both of these are established during agentic task construction (Section 2.2) by sampling from the tool dependency graph, generating arguments, and executing tools to compute the ground truth before the human–agent interplay begins. This means filtering requires oracle access to correct tool sequences and final states, which are available in the simulated environments because the system constructs the tasks from known-correct specifications.

The consequence. This filtering strategy works for synthetic environments where the system controls task construction, but it does not provide a mechanism for verifying trajectories generated from real user interactions or from environments where the correct tool sequence is not known in advance. This limits the applicability of the filtering pipeline to simulated training data generation only — it cannot be used for filtering trajectories collected from production deployments, human annotators, or environments with ambiguous or multiple valid solution paths. More subtly, the filtering pipeline validates trajectories against a specific gold tool sequence, which means it may discard trajectories that achieve the correct outcome through different but equally valid tool sequences. The paper claims to handle this by using state alignment (which is lenient to alternative sequences) for write-type tasks, but for read-only tasks, exact sequence matching is enforced — meaning any alternative approach that correctly answers a read-only query without using the exact gold sequence is discarded. This systematically biases the training data toward a single canonical solution path for read-only tasks, potentially reducing the agent's flexibility.

What evidence exists in the paper. The paper describes the filtering framework in Section 3.1 and explicitly notes that Stage 3 (exact match) is needed because "a tool sequence consisting entirely of read operations without any write operations would cause state-based filtering to fail." The paper does not measure how many trajectories are discarded by Stage 3 versus Stage 2, what fraction of discarded Stage 3 trajectories would have produced correct answers through alternative tool sequences, or whether the exact-match requirement affects the diversity of learned behaviors for read-only tasks. There is no ablation comparing models trained with and without Stage 3 filtering.

Mitigation status. The paper does not acknowledge this as a limitation. The filtering approach is presented as a strength (and it is, for ensuring data quality in simulated environments), but the implicit assumption that the gold sequence represents the only valid sequence — or that exact-match filtering is necessary for read-only tasks — is not examined. The paper does not discuss how the approach would extend to settings without oracle access to ground-truth tool sequences, nor does it suggest methods for verifying trajectories when multiple valid solution paths exist. Future work on integrating reinforcement learning (mentioned in the Limitations section) might partially address this by using learned reward signals rather than exact-match verification, but this is not explored.

Limitation 6: Single Base Model Family with No Cross-Architecture Validation

The assumption or constraint. All AgentScaler models are fine-tuned from the Qwen-3 model family (Team, 2025b). The paper implicitly assumes that the training pipeline's effectiveness generalizes across model architectures and pretraining distributions — that the observed gains are attributable to the environment scaling and two-stage training pipeline rather than to a specific synergy between this pipeline and Qwen-3's particular pretraining characteristics (instruction-following ability, multilingual competence, code generation quality, or tool-use-related pretraining data).

The consequence. A practitioner using a different base model (e.g., Llama-4, DeepSeek-V3, Mistral, Gemma) cannot know whether the environment scaling pipeline will produce comparable gains. The paper's training pipeline might depend on Qwen-3-specific properties: its tokenizer's handling of function call syntax, its pretraining exposure to code and structured data, its instruction-tuning quality, or its reasoning capabilities (the 4B and 30B variants use "thinking" models). If these properties are necessary for the pipeline to work effectively, then the approach does not "advance general agentic intelligence" in the model-agnostic sense the paper's title and framing imply. Conversely, if the pipeline works equally well across model families, the paper provides no evidence for this — and the single-family evaluation means the contribution is de facto a method for improving Qwen-3's function-calling capabilities specifically, not a general framework for agentic intelligence.

What evidence exists in the paper. All results in Tables 1 and 2, Figures 3–5, and all analysis use Qwen-3 base models exclusively. The paper does not include any experiment with a non-Qwen base model, nor does it discuss the choice of Qwen-3 in terms of what properties make it suitable for the pipeline. The baselines include models from other families (DeepSeek, Kimi, GPT-OSS, Seed-OSS, xLAM-2), but all of these are evaluated as-is — none are trained with the paper's pipeline — so they serve as performance comparisons for the resulting AgentScaler models, not as tests of whether the pipeline transfers across architectures.

Mitigation status. The paper does not acknowledge this limitation. The title and framing ("Towards General Agentic Intelligence via Environment Scaling") imply a model-agnostic contribution, but the experimental design provides no evidence of cross-architecture generalization. The paper's broader claims about the effectiveness of environment scaling for advancing agentic intelligence are therefore qualified by the possibility that the results are specific to Qwen-3. A minimal cross-validation would involve fine-tuning at least one non-Qwen model (e.g., a Llama variant at similar parameter count) with the same pipeline and comparing the relative improvement over that model's base performance. The paper does not suggest this as future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture, a novel training objective, or a fundamentally different learning algorithm. Instead, it introduces a manufacturing process — a principled pipeline for converting raw API specifications into executable, verifiable training environments at a scale that makes the manual environment construction of prior work (τ-bench, τ2-Bench, xLAM) look like artisanal handcrafting. The magnitude of this contribution is best understood not as a paradigm shift in what agents learn, but as a shift in how the training data for agentic capabilities gets produced — from a human-labor-bound bottleneck to an automated, compute-bound pipeline.

The most concrete change this work causes is de-risking investment in agentic training at scale. Prior to this work, an organization that wanted to build a function-calling agent faced a clear production bottleneck: either (a) collect human demonstrations of tool use (expensive, slow, limits domain coverage), (b) use reverse-paradigm synthesis (sacrifices interaction naturalness), or (c) manually construct simulated environments domain-by-domain (the τ-bench approach — high quality but scales linearly with expert effort). This paper demonstrates that option (c) can be automated: if you have API documentation, you can generate the environment. This changes the calculus from "can we afford to build environments for the domains we care about?" to "how many domains' worth of API documentation can we collect?" — an entirely different resource constraint.

The broader reframing is subtler but arguably more significant: environments stop being prerequisites that humans build and become outputs that a pipeline induces from existing artifacts. This is not merely an engineering convenience. It means that the diversity of agentic training data can, in principle, scale with the diversity of available API documentation — which is already vast and growing independently of agent research. Every new SaaS platform, every new REST API, every new enterprise software suite becomes potential training material without requiring the platform's developers to build agent-specific environments. The bootstrap problem that has constrained agentic AI — you need environments to train agents, but building environments requires domain experts who are scarce — is partially resolved by the observation that the API specifications themselves contain enough structural information to induce the environment.

The paper also provides a unifying diagnostic that reconciles tensions in the prior literature. The reverse paradigm (Yin et al., 2025) produced trajectories with guaranteed correctness but limited naturalness; the forward paradigm (Prabhakar et al., 2025a; Barres et al., 2025) produced naturalistic interactions but struggled with verifiability at scale. This paper shows that both properties are achievable simultaneously — naturalness via simulated user interplay, verifiability via database-state grounding — if you commit to the read–write database abstraction and accept the upfront cost of environment materialization. The apparent tradeoff in prior work was not fundamental; it was a consequence of not having a principled way to ground tool responses in a verifiable state.

The paper's diagnostic contribution on long-horizon tool calling (Figure 5) also shifts the research agenda in a specific direction. Prior work had reported that agent performance degrades on longer tasks, but this was often conflated with dataset difficulty (harder tasks happen to require more steps). The paper's per-trajectory scatter plot makes the relationship explicit and quantitative: longer chains are harder controlling for task content, and this pattern holds even for models trained with extensive environment scaling. This redirects attention from "we need more training data" to "we need architectures or algorithms that can maintain coherence over extended tool-use chains" — a distinct research challenge that environment scaling alone does not address.

Finally, the paper's results on cross-lingual transfer (Table 2: AgentScaler-4B improves ACEBench-zh Agent subset from 6.7 to 38.4) provide early evidence that the skills acquired through environment scaling are not purely surface-level pattern matching on English tool descriptions. The fact that a 4B model trained primarily on English API documentation nearly 6×s its performance on Chinese-language complex tool use suggests that the pipeline imparts genuinely transferable procedural knowledge — knowing how to decompose a user intent into tool calls, not just which tools to call in familiar English contexts. This has implications for multilingual agent deployment that the paper does not fully explore but that the data support.

Follow-Up Research This Work Enables

Direct measurement of environment quality in the 1,000+ unvalidated domains. The paper's central claim — that the pipeline produces over 1,000 coherent, verifiable environments — rests on validation for perhaps 3–5 domains that overlap with standard benchmarks. The most immediate follow-up is a systematic audit: take a random sample of 50–100 domains from the full set of 1,000+, generate 10–20 agentic tasks per domain, and have human raters (or, more scalably, a strong LLM-as-judge with carefully designed rubrics) evaluate whether the tasks are coherent, whether the tool implementations produce sensible results, and whether the induced database schemas are internally consistent. Report the fraction of domains that pass a minimum quality bar, characterize common failure modes (degenerate schemas, tools that always error, incoherent task intents), and correlate domain-level quality metrics with downstream model performance. This would transform the "1,000+ domains" claim from an existence proof to a characterized capability.

Single-stage vs. two-stage training with matched total data. The paper's ablation (Figure 3) shows that Stage 1 + Stage 2 outperforms Stage 1 alone, but does not test whether the two-stage curriculum matters versus simply training on all available data in one phase. A controlled experiment would: (a) train a model on the combined Stage 1 + Stage 2 data in a single phase, with hyperparameters tuned independently for that setting; (b) train a model on Stage 2 data only (domain-specific without general foundation); (c) compare both against the two-stage model on τ-bench, τ2-Bench, and ACEBench. If the single-stage model matches the two-stage model, the curriculum is unnecessary and the gain is purely from data volume. If the Stage 2-only model matches the two-stage model, the general foundation is unnecessary and domain-specific data is sufficient. The paper's claim that two-stage learning is a design principle (Innovation 3) hinges on this comparison, and its absence is the most significant methodological gap in the experimental design.

Data scaling laws for environment diversity. The paper trains on over 1,000 domains but does not measure how many domains are needed. A scaling experiment would train multiple models with identical architectures and training recipes but varying numbers of Stage 1 domains — 10, 50, 200, 500, 1,000 — and measure downstream performance on held-out evaluation domains. This would characterize whether returns to domain diversity are logarithmic (most gains come from the first few hundred domains), linear (each new domain provides roughly constant benefit), or super-linear (diversity compounds). The result would guide practitioners on the ROI of expanding their API corpus versus investing in other improvements. It would also test whether the 1,000+ number in the paper is near the point of diminishing returns or still on the steep part of the scaling curve.

Long-horizon tool-calling: comparative slope analysis and failure-mode taxonomy. Figure 5 shows that AgentScaler accuracy degrades with tool call count, but without baseline comparison or mechanistic analysis, the finding is descriptive rather than diagnostic. A follow-up study would: (a) run the same tool-call-count vs. accuracy analysis for baseline models (Qwen3-Thinking base, xLAM-2, GPT-o3 if API access allows per-trajectory analysis) to determine whether AgentScaler shifts the intercept, changes the slope, or both; (b) categorize each failed trajectory by error type — wrong tool selected, correct tool but wrong arguments, premature task termination, failure to recover from a tool error, user intent misunderstanding — and plot how the distribution of error types changes with trajectory length; (c) test lightweight interventions at inference time (e.g., periodically re-stating the original user intent in the context, inserting explicit state-summarization checkpoints, or using the PRM-style verifier to validate intermediate database states) and measure their effect on the accuracy-vs-length curve. This would transform the open problem from "long-horizon tool calling is hard" to "long-horizon tool calling fails primarily because of X, and intervention Y reduces the degradation slope by Z%."

Cross-model-family replication to establish pipeline generality. The paper trains exclusively on Qwen-3 variants. A replication study would apply the identical pipeline — same API corpus, same domain partitioning, same environment materialization, same two-stage training — to at least two other model families at comparable parameter counts (e.g., Llama-4-8B, Mistral-7B, DeepSeek-V2-Lite). If the pipeline produces comparable relative improvements (e.g., +X% on ACEBench overall for all families), the contribution is genuinely model-agnostic and the paper's framing as "general agentic intelligence" is supported. If improvements are Qwen-3-specific, the contribution is narrower (a method for improving Qwen-3's function calling), and the paper's claims need to be qualified accordingly. This replication is straightforward — the environments are model-agnostic by construction — and its absence is a notable gap given the paper's broad claims.

Integrating reinforcement learning on top of the fully simulated environments. The paper's Limitation section explicitly calls out the absence of RL as a constraint, and the environments are designed with properties (deterministic state transitions, verifiable outcomes, low-latency feedback) that make them well-suited for RL training. A natural follow-up would use the same environments but replace or augment the SFT stage with an RL fine-tuning phase where the agent explores tool-use strategies and receives reward based on final state alignment (for write tasks) or exact-match verification (for read tasks). This would test whether RL can push agentic performance beyond the SFT ceiling, particularly on long-horizon tasks where exploration of alternative tool sequences might discover strategies that the SFT trajectories (which are filtered to match gold sequences) do not contain. The paper's finding that error-recovery trajectories are retained in SFT training suggests that RL could further improve robustness by explicitly rewarding recovery from intermediate failures.

Practical Applications and Downstream Use Cases

Automated agentic testing for enterprise API ecosystems. An organization with a large internal API surface (hundreds of microservices, each with documented endpoints) could run the paper's pipeline to automatically construct a suite of verifiable agentic tasks covering their API landscape. The resulting environments could serve dual purpose: training an internal agent that handles employee or customer requests across the API surface, and regression-testing that agent against known-correct tool sequences whenever APIs are updated. The pipeline's automated materialization means that adding a new internal API automatically generates new training and testing tasks without manual test-case writing. The paper's finding that environment quality for benchmark domains matches human-designed implementations (Section 2.1) provides initial evidence that automatically constructed environments for enterprise APIs would be usable, though domain-specific validation would be needed.

Data generation for fine-tuning small on-device agents. The paper's 4B model achieving 65.9 on ACEBench-en Overall and 38.4 on the Agent subset in Chinese (Table 2) demonstrates that compact models can acquire meaningful agentic capabilities through this training pipeline. A device manufacturer building an on-device assistant that handles local function calling (calendar management, email composition, settings control) could use the environment-scaling approach to generate training data from the device's API surface. The benefit is specifically grounded in the 4B model's results: a model small enough to run on-device achieving performance that, prior to this work, was associated with 30B+ models. The cross-lingual transfer results further suggest that training on English API documentation can partially transfer to non-English deployment contexts, reducing the need for per-language data generation.

Cost-efficient agentic data synthesis for domain-specific enterprise agents. A company building a customer-support agent for a specific vertical (insurance claims processing, telecom troubleshooting, banking transactions) could use the pipeline to generate training data from that vertical's API documentation without hiring domain experts to manually construct environments. The key efficiency claim from the paper is that the environment construction is automated once API specifications are collected. If collecting and cleaning API documentation for a new vertical takes a few person-weeks of engineering effort, and the pipeline then generates thousands of verifiable training trajectories without further human involvement, this represents a dramatic reduction in the cost of building domain-specific agents compared to the status quo of manual environment construction (which the τ-bench authors themselves invested substantial effort in for just two domains). The paper does not quantify this cost reduction directly, but the 1,000+ domain figure implies it is large.

Benchmark saturation monitoring through environment expansion. The paper's approach suggests a strategy for the evaluation community: as agentic models approach ceiling performance on existing benchmarks (AgentScaler-30B-A3B already achieves 75.7 on ACEBench Overall, with some sub-scores in the 80s), the environment-scaling pipeline can generate new evaluation domains from APIs that do not overlap with training data. A benchmark maintainer could periodically expand the evaluation set by running the pipeline on a held-out corpus of APIs, constructing verifiable tasks, and adding them to the benchmark. The verifiability-by-construction property means new tasks come with known-correct answers, avoiding the need for human annotation of evaluation data. This would help the field stay ahead of benchmark saturation — a concern given that Table 1 shows multiple models clustering in the 70–80 range on several domains.

When to Prefer This Method

The paper positions its approach primarily against two alternatives: (a) manual environment construction (the τ-bench paradigm — high fidelity, not scalable) and (b) LLM-simulated tool responses (cheap but hallucination-prone, no ground-truth verifiability). The choice conditions follow directly from the pipeline's design properties:

• Prefer AgentScaler-style automated environment scaling when you have a large corpus of API specifications (hundreds to thousands of documented functions), need training data across many domains, and can tolerate the upfront cost of environment materialization in exchange for downstream verifiability and scalability. The paper's results suggest this is particularly valuable when training smaller models (4B–30B) where data quality matters more than sheer volume. The approach is also preferable when the target deployment involves multi-turn interactions where tool-call errors compound — the filtering pipeline's emphasis on state-alignment verification provides stronger correctness guarantees than LLM-as-judge approaches.

• Stick with manual environment construction when you need only a small number of environments (e.g., 1–3 domains), have domain experts available, and the cost of building the pipeline's infrastructure (API collection, graph construction, code generation, filtering) exceeds the cost of manually implementing the needed tools. The paper acknowledges that its automated environments matched human-designed ones for overlapping domains (Section 2.1), implying that manual construction remains viable when scale is not required. Manual construction may also be preferable when the APIs have complex, non-database-like state (e.g., physical-world effects, external service dependencies) that the read–write database abstraction cannot easily capture.

• Prefer LLM-simulated tool responses only when you need rapid prototyping with minimal infrastructure investment and can tolerate hallucinated or inconsistent tool outputs. The paper's approach requires engineering effort to set up the environment materialization pipeline; LLM simulation requires only a prompt. However, the paper's filtering results demonstrate that LLM-simulated outputs create correctness problems that the database-grounded approach avoids, so simulation is best treated as a temporary substitute during early development rather than a foundation for production training data.