ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration

🎯 Pitch

An 8B model can beat GPT-5 on Humanity’s Last Exam while costing 2.5x less, simply by learning when and which larger models and tools to call. The orchestrator overcomes the 'self-enhancement' and 'other-enhancement' biases that cripple prompted systems, allocating work strategically to mix expert and generalist resources.

1. Executive Summary

This paper introduces ToolOrchestra, a method for training small language models to act as orchestrators that coordinate a diverse toolkit — including basic tools, specialized LLMs, and generalist LLMs — through reinforcement learning with outcome, efficiency, and user-preference-aware rewards. Using ToolOrchestra, the authors produce Orchestrator, an 8B model evaluated on Humanity's Last Exam (HLE), FRAMES, and τ2-Bench, where it achieves 37.1% on HLE, surpassing GPT-5 (35.1%) while being 2.5× more cost-efficient, and outperforms GPT-5 by wide margins on FRAMES and τ2-Bench at roughly 30% of the cost. The method's gains stem from learning a balanced tool-calling policy — Orchestrator allocates calls strategically across GPT-5, GPT-5-mini, Qwen3-32B, and other tools rather than exhibiting the self-enhancement bias (GPT-5 disproportionately delegating to GPT-5-mini) or other-enhancement bias (Qwen3-8B defaulting to GPT-5 in 73% of cases) that plague prompted off-the-shelf orchestrators. The paper establishes that a lightweight orchestration model trained with multi-objective RL can both push the upper bound of intelligence and improve efficiency, demonstrating robust generalization to unseen tools and pricing configurations only when the orchestrator learns adaptive, cost-aware tool selection rather than relying on static heuristics or monolithic model scaling.

2. Context and Motivation

The Core Problem: Monolithic Models Hit a Capability Ceiling, But Tool Use Remains Underexploited

The fundamental tension this paper addresses is deceptively simple: large language models are powerful generalists, yet on the hardest reasoning tasks — those requiring deep multi-step inference, domain expertise, or real-time information retrieval — even frontier models like GPT-5 achieve only 35.1% on HLE. The paper's starting observation is that while tool use has been widely adopted as a way to extend LLM capabilities (adding web search, code interpreters, calculators), the prevailing approach — equipping a single powerful model with utility tools — fundamentally underutilizes what tools could be.

The authors reframe the problem through an orchestration lens: humans, when confronting problems beyond their immediate expertise, do not simply consult reference materials. They strategically delegate sub-problems to domain experts, specialized software, and more knowledgeable colleagues. A mathematician tackling a novel proof might ask a colleague about a specific lemma, run numerical experiments in a CAS, and consult the literature — dynamically choosing which resource to engage based on the sub-problem at hand. In the LLM context, this translates to a system where models themselves become tools: a small orchestrator model should be able to invoke stronger generalist models (GPT-5, Claude), specialized expert models (math-focused LLMs, coding-focused LLMs), and traditional utilities (web search, code execution) — choosing which to use when and in what order based on the reasoning demands of each step.

The paper's central claim is that this orchestration paradigm can achieve both higher intelligence and lower cost than any monolithic model, including those far larger than the orchestrator itself. The 8B Orchestrator outperforms GPT-5 while being 2.5× more efficient — a result that would be impossible if intelligence were simply a function of parameter count.

Why This Problem Matters: Practical and Conceptual Stakes

Practical deployment economics. The cost implications are stark and immediate. Frontier models like GPT-5 and Claude Opus 4.1 are expensive to query, with per-token costs that make high-volume or long-context reasoning economically prohibitive. As Table 1 shows, GPT-5 with basic tools costs 30.2 cents and 19.8 minutes of latency per HLE/FRAMES task on average, while Orchestrator-8B achieves higher accuracy at 9.2 cents and 8.2 minutes. For organizations running thousands or millions of inference queries, a 3× cost reduction with improved accuracy represents a fundamental shift in what is economically viable. The paper's paradigm suggests that the optimal deployment strategy is not "use the largest model you can afford for every task" but rather "use a small, trained orchestrator that invokes expensive models only when necessary."

Capability amplification beyond training data. The paper's framing connects to a deeper conceptual question: can a system composed of models with known capabilities exhibit intelligence exceeding any individual component? The HLE results suggest yes — Orchestrator-8B (37.1%) outperforms both GPT-5 alone (35.1%) and any single tool it has access to, because it learns to decompose problems, route sub-tasks to appropriate experts, and synthesize results. This is the "composite intelligence" thesis: that coordination itself is a form of intelligence that can be learned and optimized, and that the ceiling of a composite system may be higher than the ceiling of its strongest component.

Controllability and alignment. A third motivation, less emphasized but structurally important, is that existing tool-use systems lack fine-grained controllability along two dimensions: cost-efficiency and user preferences. A user might want to minimize API costs for privacy reasons (preferring local models when possible), minimize latency for interactive use, or avoid specific models due to organizational policy. Prompted LLMs largely ignore such preferences — they default to biased invocation patterns (Figure 3) regardless of instructions. ToolOrchestra's multi-objective RL framework bakes efficiency and preference alignment directly into the training signal, producing a model that can be steered at inference time through preference vectors without retraining. This is a step toward deployable, controllable agentic systems rather than research prototypes.

Where Existing Approaches Fall Short

The paper identifies four categories of prior work, each with specific limitations that motivate the orchestration paradigm:

1. Single-model tool-use agents are biased toward themselves or the strongest available tool. Prior research on equipping LLMs with tools (Toolformer, ToolLLM, WebGPT, Search-R1, ToRL, StepTool, SWiRL) focused on integrating utilities like search and calculators into a single model's reasoning loop. The paper's pilot study (Section A) reveals a critical failure mode when this approach is extended to include other models as tools: prompted LLMs exhibit systematic biases in their delegation patterns. When GPT-5 serves as orchestrator with access to GPT-5, GPT-5-mini, Qwen3-32B, and Qwen2.5-Coder-32B, it calls GPT-5 or GPT-5-mini in 98% of cases — what the authors term other-enhancement bias, favoring developmentally related variants regardless of cost considerations or task suitability. Conversely, Qwen3-8B delegates to GPT-5 in 73% of cases — self-enhancement bias, where a weaker model defaults to the strongest available tool rather than learning when weaker (cheaper) tools suffice. These biases are not superficial; they produce demonstrably worse outcomes. GPT-5's performance drops from 35.1 (with basic tools only) to 21.2 on HLE when given access to a broader toolset including specialized and generalist LLMs (Table 1) — precisely because its biased invocation pattern routes tasks to suboptimal tools. The paper argues these biases are "idiosyncratic in the context of model tool calling and warrant their own approach to training" (Section 1) — you cannot fix them through prompting alone; you need RL to learn a balanced policy.

2. Efficiency and preference alignment are afterthoughts, not training objectives. Prior work on efficient tool use (OTC, SMART, Efficient Agents, Self-DC) reduces costs through heuristics — penalty terms added post-hoc, prompt engineering to discourage unnecessary calls, or curated datasets that bake in cost-awareness. But these approaches are fundamentally reactive: they constrain a model that was not trained to optimize for efficiency. ToolOrchestra integrates efficiency directly into the RL reward, making it a first-class training objective alongside accuracy. The model learns to generate efficient trajectories, not just have efficient ones selected from its outputs. Similarly, user preference alignment in tool use is "underexplored" (Section 7.2); existing systems offer no mechanism for a user to say "prefer local models for privacy" and have the agent adapt its strategy accordingly. The preference vector formulation in ToolOrchestra's reward (Equation 2) provides a principled way to encode and optimize for these desiderata.

3. Monolithic scaling ignores the composability of intelligence. The field's dominant paradigm — train ever-larger models (GPT-5, Claude Opus, Qwen3-235B) and prompt them to do everything — treats intelligence as a property of a single model. This paper argues that intelligence can emerge from composition: an 8B model coordinating specialized tools can outperform a ~100× larger model. This challenges the scaling-laws narrative not by denying that larger models are more capable, but by showing that access to larger models as tools can yield more capability per FLOP than being a larger model. The finding that Orchestrator-8B outperforms Qwen3-235B-A22B (32.8 vs. 37.1 on HLE) despite the latter being ~30× larger is the clearest empirical evidence for this claim.

4. Training data for multi-turn tool orchestration is scarce. A practical bottleneck the paper identifies is the lack of verifiable training data for agentic tool-use tasks. Existing datasets cover simple tool calls or single-turn interactions; multi-turn reasoning with heterogeneous tools (models, APIs, search) is underrepresented. The ToolScale synthetic data pipeline (Section 3.3) addresses this gap by generating thousands of tasks with ground-truth action sequences and verifiable execution criteria across 10 domains (finance, sports, e-commerce, medicine, entertainment, railway, restaurant, education, travel, weather). This is not merely a dataset contribution — it is an enabling step for the RL training approach, since GRPO requires reward signals that can be automatically computed at scale.

How This Paper Positions Itself

The paper positions ToolOrchestra at the intersection of three research threads — tool learning, efficient/controllable agents, and composite AI systems — while arguing that none of these alone addresses the full problem.

From tool learning (Toolformer, ToolLLM, ToRL, Search-R1), it inherits the framing of tool use as sequential decision-making and the use of RL for policy optimization. But it diverges in two crucial ways: (a) it expands the tool taxonomy to include other LLMs as first-class tools with varying capabilities and costs, and (b) it trains a dedicated small orchestrator rather than fine-tuning the tool-using model itself to also do the reasoning.

From efficiency and controllability work (OTC, SMART, L1), it adopts the goal of optimizing for cost and latency, but argues that prompting-based or penalty-based approaches are insufficient — what is needed is end-to-end RL where efficiency is part of the reward, not a constraint applied post-hoc. The preference vector mechanism (Equation 2) provides a general interface for user steerability that prior work lacks.

From compound AI systems (Zaharia et al., 2024; Chaudhry et al., 2025) and weak-to-strong generalization (Burns et al., 2024), it draws the architectural insight that intelligence can be composed from components of varying capabilities, but it operationalizes this insight through a specific training methodology (GRPO with multi-objective rewards) and demonstrates it at scale on state-of-the-art benchmarks.

Perhaps most importantly, the paper positions ToolOrchestra as an alternative to the monolithic scaling paradigm, not an augmentation of it. The core empirical claim — that an 8B orchestrator + tools outperforms GPT-5 at lower cost — is an existence proof that orchestration is a viable path to intelligence that scales differently than model size. This is not a claim that model scaling is obsolete; it is a claim that for a given total compute budget, investing in orchestration training may yield higher returns than investing in larger pretraining, particularly when the task distribution requires diverse capabilities that no single model possesses uniformly.

3. Technical Approach

3.1 Reader Orientation

ToolOrchestra is a reinforcement learning training method that teaches a small language model (8B parameters) to act as a central dispatcher — the Orchestrator — that solves hard reasoning tasks by strategically delegating sub-problems to a diverse set of tools, including other language models of varying sizes and costs, rather than attempting to solve everything itself. The system solves the problem of how to maximize both accuracy and cost-efficiency on complex reasoning tasks by learning, through trial and error, when and in what order to invoke expensive frontier models, cheaper specialized models, and traditional utilities like web search or code execution, producing a policy that outperforms any single tool it has access to while spending far less than a monolithic frontier model would.

3.2 Big-Picture Architecture (Diagram in Words)

The ToolOrchestra system has six major components:

ToolScale Synthetic Data Pipeline — generates thousands of verifiable multi-turn agentic tasks with ground-truth action sequences across 10 domains (finance, sports, e-commerce, medicine, entertainment, railway, restaurant, education, travel, weather), simulating realistic databases, tool APIs, and diverse user intents.
Orchestrator Policy Model (Qwen3-8B backbone) — the core 8B-parameter language model being trained. It takes a user query and optional preference instructions as input, then iteratively generates reasoning text followed by structured tool calls, receiving tool outputs as observations, and continuing until it produces a final answer or hits the 50-turn limit.
Heterogeneous Tool Suite — a catalog of tools exposed through a single unified JSON interface, including basic utilities (web search via Tavily API, code interpreter with Python sandbox, local search via FAISS index with Qwen3-Embedding-8B, domain-specific functions), specialized LLMs (math models, coding models), and generalist LLMs (GPT-5, GPT-5-mini, Llama-3.3-70B, Qwen3-32B). Each tool has associated monetary cost and latency metadata.
GRPO Reinforcement Learning Loop — the training algorithm that updates the policy model. For each task in a batch, the current policy generates a group of 8 trajectories; each trajectory is scored by a multi-objective reward function; advantages are computed within the group; and the policy is updated via a clipped surrogate objective.
Multi-Objective Reward Function — computes a scalar reward for each trajectory by combining three signals: binary outcome correctness (solved or not, judged by GPT-5), efficiency penalties (monetary cost in dollars and wall-clock latency), and preference alignment (how well the trajectory's tool usage patterns match a user-specified preference vector over tools, accuracy, cost, and latency). These are combined via a dot product with the preference vector, then normalized within each batch.
Preference Vector Interface — a numeric vector $P = [p_{t_1}, ..., p_{t_n}, p_{\text{outcome}}, p_{\text{compute}}, p_{\text{latency}}]$ where each $p \in [0, 1]$ specifies the user's desired optimization weight for each dimension. This allows zero-shot steering of the Orchestrator's behavior at inference time without retraining.

Information flows as follows: a user query enters the system → the query is concatenated with a system prompt describing available tools and (optionally) a preference instruction → the Orchestrator generates a trajectory of reasoning–action–observation turns → each tool call is executed by the environment (API call, model inference, code execution) and its output is appended to the context → after the trajectory terminates or hits the turn limit, the reward function scores the trajectory on outcome, cost, latency, and preference alignment → during training, the GRPO algorithm uses these rewards to update the policy; during evaluation, only the final answer and efficiency metrics are recorded.

3.3 Roadmap for the Deep Dive

First, the Markov Decision Process formulation (Section 2.1), which defines the problem structure — states, actions, rewards, and the optimization objective — and provides the mathematical scaffolding for everything that follows.
Second, the multi-turn rollout procedure (Section 2.2), which describes the concrete interaction loop: how the Orchestrator interleaves reasoning with tool calls and how the environment responds.
Third, the unified tool-calling interface (Section 3.1), since this is the mechanism by which all tools — including other language models — are made accessible to the Orchestrator, and understanding the interface is prerequisite to understanding the action space the policy learns over.
Fourth, the end-to-end RL training with multi-objective rewards (Section 3.2), which is the core technical contribution: the reward design, the three reward signals (outcome, efficiency, preference), how they are normalized and combined, and the GRPO algorithm that optimizes the policy.
Fifth, the ToolScale data synthesis pipeline (Section 3.3), which generates the training data that makes RL possible — how environments, tools, and tasks are simulated and verified.
Sixth, the user preference construction and general tool configuration randomization (Sections 3.3 and 3.2), which together enable the model to learn preference-following behavior and generalize to unseen tool sets and pricing structures.
Seventh, the training configuration and hyperparameters (Section 4.4), which specify the concrete implementation details needed to reproduce or understand the training dynamic.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a small language model, trained with reinforcement learning to coordinate diverse tools under multi-objective rewards (accuracy + efficiency + user preferences), can serve as a more capable and more cost-effective reasoning agent than any single monolithic model, including frontier models far larger than the orchestrator itself.

Agentic Problem Formulation as a Markov Decision Process

The paper formalizes the multi-turn tool-use problem as a Markov Decision Process (MDP) to make it amenable to reinforcement learning optimization. An MDP is a mathematical framework for sequential decision-making under uncertainty: at each step, an agent observes a state, chooses an action, transitions to a new state based on environment dynamics, and receives a reward. Formulating tool use this way allows the paper to apply policy gradient methods (specifically GRPO) that have been successful in other sequential reasoning domains.

The MDP is defined as $\mathcal{M} = (\mathcal{U}, \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \mathcal{Z}, r, \rho, \gamma)$ following conventions from prior work on agent training (AgentGym, Archer, AgentGym-RL).

The components are:

$\mathcal{U}$ is the space of user instructions (task descriptions like "Cancel user 8612's ticket A03 and refund fully").
$\mathcal{S}$ is the environment state space — the complete internal state of the simulated world, including database contents, tool states, and any persistent information.
$\mathcal{A}$ is the action space — the set of all possible tool calls the Orchestrator can make. Each action $a_k \in \mathcal{A}$ is a structured tool invocation (function name plus typed parameters) selected from the available tool catalog.
$\mathcal{O}$ is the observation space — the outputs returned by tools after execution and fed back to the Orchestrator. An initial observation $o_0 \in \mathcal{O}$ is drawn from the initial state distribution $\rho(\cdot | u)$ .
$\mathcal{T}(s_{k+1} | s_k, a_k)$ is the state transition function, which deterministically updates the environment state after action $a_k$ is executed in state $s_k$ .
$\mathcal{Z}(\cdot | s_{k+1}, a_k)$ is the observation function, which generates the textual or structured output returned to the Orchestrator after the transition.
$r(\tau) \in [0, 1]$ is the reward function defined over a complete trajectory $\tau = h_N$ (the interaction history after $N$ steps), representing whether the task was solved correctly. This is a terminal reward — it is received only at the end of the trajectory, not at intermediate steps.
$\rho$ is the initial state distribution.
$\gamma$ is the discount factor (standard in MDPs, though the paper does not emphasize it since rewards are terminal).

The interaction loop. At each step $k$ , the Orchestrator selects an action $a_k$ according to its policy $\pi_\theta(a_k | h_k)$ , where $h_k = (u, o_0, a_0, o_1, ..., a_{k-1}, o_k)$ is the complete interaction history up to that point. The policy is a conditional distribution over actions given history, parameterized by the model weights $\theta$ . The environment then transitions to a new state $s_{k+1} \sim \mathcal{T}(\cdot | s_k, a_k)$ and emits an observation $o_{k+1} \sim \mathcal{Z}(\cdot | s_{k+1}, a_k)$ , which is appended to the history.

The optimization objective. Each action $a_i$ has associated metadata: a monetary cost $c_i$ , an operational latency $l_i$ , and a user preference alignment score $p_{a_i}$ (drawn from the preference vector's entry for that tool). After $N$ interaction steps yielding trajectory $\tau = h_N$ , the goal is to optimize for three objectives simultaneously:

Maximize the correctness reward $r(\tau) \in [0, 1]$ .
Maximize the total preference alignment $\sum p_{a_i}$ across all actions in the trajectory.
Minimize the total monetary cost $\sum c_i$ .
Minimize the aggregate latency $\sum l_i$ .

These four objectives are in tension — for instance, calling GPT-5 might maximize correctness but also maximize cost — and the preference vector provides a mechanism for trading them off per-user or per-deployment.

Why this formulation matters. By casting tool orchestration as an MDP with terminal outcome rewards and per-step cost/latency metadata, the paper inherits the entire machinery of policy gradient reinforcement learning. The key design choice is that cost and latency are properties of actions, not part of the environment state — they are observable metadata attached to tool definitions, which means the policy can learn to consider them when selecting actions without needing to infer them from environment feedback. This is important because cost is not something the environment "returns" as an observation; it is a known prior that the model should internalize during training.

Multi-Turn Rollout Procedure

The rollout procedure describes the concrete mechanics of how the Orchestrator interacts with tools during a single trajectory (both at training and inference time). It follows a three-phase loop repeated until termination.

Initialization. The rollout begins with a predefined system prompt and the user's task instruction (the question to be answered). The system prompt includes the list of available tools in JSON format (name, description, parameter schema), and optionally a user preference instruction (e.g., "I want to be cost efficient if possible"). The model, acting in the assistant role, generates its first output, which ends with an end-of-sequence (EOS) token.

The reasoning–action–observation loop. Each turn follows a fixed three-step pattern:

Chain-of-thought reasoning. The Orchestrator analyzes the current state — the original question plus all previous tool calls and their responses — and plans the next action. This reasoning text is generated as natural language and may include analysis of what information is still needed, assessment of which tool is most appropriate given the cost/preference constraints, or synthesis of results obtained so far.
Tool call (action). Based on its reasoning, the Orchestrator selects a specific tool from the available set and specifies the parameters to pass. This is output in a structured format (a tool-call block) that the environment parses to extract the function name and arguments. The available tools span three categories: basic utilities (web search via Tavily API, code execution via Python sandbox, local semantic search via FAISS with Qwen3-Embedding-8B, domain-specific functions like get_flight_status), specialized LLMs (math models like Qwen2.5-Math-72B and Qwen2.5-Math-7B, code writers like Qwen2.5-Coder-32B-Instruct and GPT-5-mini acting as code writer), and generalist LLMs (GPT-5, GPT-5-mini, Llama-3.3-70B-Instruct, Qwen3-32B).
Tool response (observation). The tool-call block is extracted from the model's output and executed by the environment. For API calls (web search, domain functions), this means sending an HTTP request and receiving structured or textual results. For LLM-as-tool calls, this means running inference on the specified model with the query extracted from the Orchestrator's parameters and returning the model's generated text. For code execution, the Python code is run in a sandbox and the output (stdout, stderr, return values) is captured. The resulting output is appended to the context under the user role and fed back to the Orchestrator for the next turn.

Termination. The loop repeats until one of two conditions is met: (1) the environment signals that the task is complete (the Orchestrator produces a final answer without a tool call), or (2) the rollout reaches the maximum of 50 turns. The 50-turn limit is a practical constraint to prevent infinite loops in cases where the model fails to converge on an answer.

Why this structure. The reasoning–action–observation loop is modeled on how humans solve complex problems with external resources: think about what you need, consult a resource, integrate the result, and repeat. Critically, the chain-of-thought reasoning is generated before the tool call, which means the model's planning is explicit in the context and can be learned during RL training (the policy gradient flows through the entire sequence, including the reasoning tokens that precede the tool call). This contrasts with approaches that separate planning from execution — here, the reasoning and action are generated autoregressively as a single sequence, making the entire decision process differentiable with respect to the final reward.

Unified Tool-Calling Interface

A critical architectural decision in ToolOrchestra is that all tools — whether they are traditional APIs, specialized models, or generalist LLMs — are exposed through a single unified JSON interface. This uniformity is what makes it possible for an 8B model to coordinate tools of vastly different natures without needing separate handling logic for each category.

JSON tool specification. Each tool is defined as a JSON object with three fields: the tool name (a unique identifier), a natural language description of what the tool does and when it should be used, and a typed parameter schema specifying the names, types, and descriptions of arguments the tool expects. This is the standard function-calling format used by many LLM APIs, extended here to include model-as-tool entries.

LLM-as-tool description generation. When other language models are used as tools, the Orchestrator needs to know their capabilities and limitations to decide when to invoke them. The paper generates these descriptions through a three-step process (Section 3.1):

Randomly sample 10 training tasks from the dataset.
Run each LLM-as-tool on these tasks and record their trajectories — what they attempted, what they succeeded at, what they failed at.
Ask another LLM (GPT-5 is implied but not explicitly named for this purpose) to write a capability description for the tool LLM based on the task instructions, the tool LLM's trajectories, and whether it succeeded or failed on each task.

An example description for Qwen3-32B is provided in Appendix C and reveals the granularity of the resulting description: "The model shows advanced mathematical and quantitative reasoning, often solving complex problems and only faltering on highly specialized or computationally heavy items. Scientific domain knowledge is strong — especially in biology — with solid performance in physics and engineering; chemistry is mixed, with notable weaknesses in exact nomenclature and InChI outputs." This level of detail — specifying not just that the model is good at math, but which subdomains of chemistry it struggles with — enables the Orchestrator to make fine-grained delegation decisions (e.g., routing a chemistry nomenclature question to a different model while using Qwen3-32B for biology).

Why a unified interface matters. Without it, the Orchestrator would need separate logic for "calling an API" versus "prompting another LLM" versus "executing code," which would fragment the action space and make the RL optimization problem harder. By representing all tools identically, the policy gradient sees all actions as instances of the same structured prediction problem — choose a tool name, fill in its parameters — and can learn transferable heuristics about when to delegate regardless of the delegate's nature. This also enables the generalization experiments (Section 6.3): because unseen tools are described in the same JSON format, the Orchestrator can reason about their capabilities from the description alone, without having been trained on their specific behavior.

End-to-End Agentic Reinforcement Learning with Multi-Objective Rewards

This is the core technical contribution of the paper. The Orchestrator is trained via a policy gradient RL algorithm (GRPO) using a reward function that combines outcome correctness, efficiency (cost + latency), and user preference alignment into a single scalar signal. The model generates complete multi-turn trajectories, receives a reward at the end, and the policy gradient updates the model to increase the probability of high-reward action sequences.

Reward Design: The Three Reward Components

The reward for a trajectory $\tau$ is built from three components, each addressing a different objective.

Outcome reward (correctness). This is a binary signal indicating whether the trajectory solved the task:

$r_{\text{outcome}}(\tau) = \begin{cases} 1 & \text{if solved}(\tau), \\ 0 & \text{otherwise}. \end{cases}$

where $\text{solved}(\tau)$ is a predicate that evaluates whether the trajectory's final answer matches the ground truth. For ToolScale-synthesized data, solved is determined by three criteria: execution correctness (database state after running the trajectory matches the state after running the gold action sequence), process fidelity (required information was communicated during the trajectory), and operation completeness (all database entries operated on in the gold sequence were also operated on in the trajectory). For benchmark tasks, GPT-5 is used as a judge to compare the predicted answer to the reference answer.

What it computes: a binary 0 or 1 based on whether the task was completed correctly. This is a sparse, terminal reward — the model receives it only after the entire trajectory, not after each intermediate step. This is the standard setup for outcome-based RL on reasoning tasks (cf. DeepSeek-R1, which also uses binary outcome rewards for math and code problems).

Why this form: binary outcome rewards provide a clean, unambiguous training signal that is cheap to compute (the verification criteria are deterministic for synthetic data) and do not require a learned reward model that could introduce its own biases. The use of GPT-5 as judge for benchmark tasks where deterministic verification is impossible (e.g., open-ended answers) is a pragmatic compromise — it provides flexibility for diverse answer formats but introduces a dependency on a frontier model's judgment quality.

Efficiency rewards (cost and latency). These penalize trajectories that consume excessive resources:

$r_{\text{compute}}(\tau) = -\$(\tau)$

where $\$(\tau)$ is the total monetary cost of trajectory $\tau$ , computed by summing the cost of each tool invocation. For LLM-as-tool calls, cost is calculated by converting input and output tokens to monetary amounts using third-party API pricing systems (TogetherAI, Venice AI, Chutes, NEBIUS, Lambda, Hyperbolic, Cloudflare, Novita, AIML, Fireworks AI — the full list in Appendix E). For basic tools, cost is the API call price (e.g., Tavily search API pricing).

$r_{\text{latency}}(\tau) = -\text{Clock}(\tau)$

where $\text{Clock}(\tau)$ is the total wall-clock time consumed by trajectory $\tau$ , measured as the sum of execution times for each tool call plus model inference time. Latency is measured in minutes (see Table 1 reporting latency values).

What they compute: negative scalars that become more negative as cost and time increase. These are summed into the reward, making them penalties rather than bonuses. The negative sign means that all else being equal, the model is incentivized to produce shorter, cheaper trajectories.

Why this form: the negative linear penalty is the simplest efficiency signal that preserves the property that doubling cost doubles the penalty. More sophisticated forms (e.g., step penalties, budget constraints) could be used, but linear penalties are straightforward to implement and have well-behaved gradients — the policy learns proportionally to how much cost it incurs, not to whether it crosses some arbitrary threshold. The use of monetary cost as a unified metric for both proprietary and open-source models (converting token counts to dollars via API pricing) is a practical design choice that puts all tools on a comparable scale, regardless of whether they are accessed via paid API or self-hosted.

Preference reward. This is the most novel component. Unlike the outcome and efficiency rewards which are objective properties of the trajectory, the preference reward measures how well the trajectory's tool-usage pattern aligns with a user-specified preference vector.

Given a tool set $\{t_1, t_2, ..., t_n\}$ and a trajectory $\tau$ , the paper constructs a feature vector:

$M_\tau = [m_\tau^{t_1}, m_\tau^{t_2}, \ldots, m_\tau^{t_n}, r_{\text{outcome}}(\tau), r_{\text{compute}}(\tau), r_{\text{latency}}(\tau)]$

where $m_\tau^{t_i}$ is the count of how many times tool $t_i$ was invoked in trajectory $\tau$ , $r_{\text{outcome}}(\tau)$ is the binary outcome reward, $r_{\text{compute}}(\tau) = -\$(\tau)$ is the cost penalty, and $r_{\text{latency}}(\tau) = -\text{Clock}(\tau)$ is the latency penalty.

What it represents: a vector of length $n + 3$ that summarizes the trajectory along all dimensions that a user might care about — which tools were used, whether the task was solved, and how much was spent.

During RL training, each element of $M_\tau$ is normalized within the rollout batch to the $[0, 1]$ range:

$M_\tau^{\text{normalized}}[k] = \frac{M_\tau[k] - M_{\min}^{\mathcal{T}}[k]}{M_{\max}^{\mathcal{T}}[k] - M_{\min}^{\mathcal{T}}[k]}$

where $M_{\min}^{\mathcal{T}}[k]$ and $M_{\max}^{\mathcal{T}}[k]$ are the minimum and maximum values of the $k$ -th feature across all trajectories in the current batch $\mathcal{T}$ . If $M_{\max}^{\mathcal{T}}[k] = M_{\min}^{\mathcal{T}}[k]$ (all trajectories in the batch have the same value for this feature), the feature is disregarded by setting it to zero, since it provides no discriminative signal within that batch.

What normalization accomplishes: it converts counts and costs — which can vary by orders of magnitude across different tasks and batches — into a consistent scale, ensuring that no single feature dominates the preference-weighted sum simply because its raw values are larger. Batch-level normalization also adapts the reward scale to the current difficulty distribution: on a batch of easy tasks where everyone uses few tools, small differences in tool counts get amplified; on a batch of hard tasks where everyone uses many tools, the same differences would be attenuated.

The final reward for a trajectory $\tau$ is computed as:

$R(\tau) = \begin{cases} M_\tau^{\text{normalized}} \cdot P & \text{if } r_{\text{outcome}}(\tau) = 1, \\ 0 & \text{otherwise}. \end{cases}$

where $P = [p_{t_1}, p_{t_2}, ..., p_{t_n}, p_{\text{outcome}}, p_{\text{compute}}, p_{\text{latency}}]$ is the user preference vector, with each $p \in [0, 1]$ indicating the desired optimization weight for that dimension. $p_{t_i}$ is how much the user wants to use tool $t_i$ (high values encourage its use, low values discourage it), $p_{\text{outcome}}$ is the weight on correctness, $p_{\text{compute}}$ is the weight on minimizing cost, and $p_{\text{latency}}$ is the weight on minimizing latency.

What it computes: a weighted sum of the normalized feature vector, where the weights are user-specified preferences. Critically, the reward is zeroed out if the trajectory did not solve the task ( $r_{\text{outcome}}(\tau) = 0$ ), regardless of how well it matched preferences or how cheap it was. This hard-gating ensures that correctness is a prerequisite — the model cannot learn to optimize for efficiency at the expense of accuracy by simply doing nothing.

Why this form — the user preference vector. The preference vector is a remarkably general interface for controllability. A few examples from the paper illustrate the range of behaviors it can encode:

$P = [0, 1, 1, 1, 0, 0, 0, 0, 0]$ (Appendix F example) means: prefer using local search and open-source models (Qwen3-235B-A22B, Llama-3.3-70B) over web search or external APIs, with no explicit weighting on accuracy, cost, or latency beyond what the tool preferences imply.
$P = [p_{t_1} = 0, ..., p_{\text{outcome}} = 1, p_{\text{compute}} = 0, p_{\text{latency}} = 0]$ means: optimize purely for correctness, ignoring all efficiency considerations.
$P = [p_{t_1} = 0, ..., p_{\text{outcome}} = 1, p_{\text{compute}} = 1, p_{\text{latency}} = 0]$ means: maximize correctness while minimizing cost, treating latency as irrelevant (appropriate for batch processing where wall-clock time is not a constraint).

The dot-product form means the optimization is linear in preferences, which has the practical advantage that the model can interpolate between preference vectors it has seen during training — if it learned to handle "high accuracy, low cost" and "high accuracy, low latency" separately, it should generalize to "high accuracy, medium cost, medium latency" at test time.

The zeroing-out of incorrect trajectories. This is a crucial design choice with significant implications. By setting $R(\tau) = 0$ when $r_{\text{outcome}}(\tau) = 0$ , the reward function enforces a strict hierarchy: correctness first, everything else second. The model never receives positive reward for an efficient but wrong answer. This avoids the pathological strategy of "call nothing and answer immediately," which would achieve perfect efficiency scores but zero accuracy. The downside is that trajectories that are almost correct (close to the right answer but with a minor error) receive the same zero reward as trajectories that are completely wrong, providing no gradient signal toward incremental improvement. This is a standard tension in outcome-based RL for reasoning and is partially mitigated by GRPO's group-based advantage normalization (Equation 3), which compares trajectories within the same task and can amplify small differences.

GRPO Training Algorithm

The Orchestrator policy $\pi_\theta$ is fine-tuned using Group Relative Policy Optimization (GRPO), a policy gradient algorithm that has been successfully applied to reasoning models (DeepSeek-R1, DeepSeekMath). GRPO is a variant of PPO that replaces the learned value function (critic) with group-based reward normalization, making it simpler to implement and more memory-efficient for large models.

Training loop. For each task (or batch of tasks) in the training data:

The current policy $\pi_\theta$ generates a group of $G = 8$ complete trajectories (the rollout batch $\mathcal{T}$ ), each produced by autoregressively sampling actions from the policy, executing them in the environment, and receiving observations.
Each trajectory $\tau \in \mathcal{T}$ is scored by the multi-objective reward function $R(\tau)$ (Equation 2), which incorporates outcome, efficiency, and preference alignment as described above.
GRPO normalizes these rewards within the group to compute an advantage for each trajectory:

$A(\tau) = \frac{R(\tau) - \text{mean}_{\tau \in \mathcal{T}} R(\tau)}{\text{std}_{\tau \in \mathcal{T}} R(\tau)}$

where $\text{mean}_{\tau \in \mathcal{T}} R(\tau)$ is the average reward across the $G$ trajectories in the batch, and $\text{std}_{\tau \in \mathcal{T}} R(\tau)$ is their standard deviation.

What it computes: a standardized score — centered at zero (trajectories better than average get positive advantages, worse than average get negative advantages) and scaled to approximately unit variance — that tells the policy gradient how much better or worse each trajectory was compared to the alternatives the current policy could have generated for the same task.

Why this form — group-based normalization. The key insight behind GRPO is that for a given task, the absolute reward values are less informative than the relative ordering of trajectories. If all 8 trajectories get rewards around 0.8 (because the task is easy), the advantage for each will be near zero, producing no update — which is correct, since the policy already reliably solves this task. If one trajectory gets 0.9 and the rest get 0.1 (because only one found a correct solution), that trajectory gets a large positive advantage, driving a strong update toward its action sequence. This adaptive scaling means the algorithm automatically focuses learning on tasks where the policy has room to improve and on trajectories within each task that are unusually good (or bad). The absence of a learned value function eliminates the need to train and store a separate critic network, which is significant for an 8B-parameter model where memory is already constrained.

The policy is updated by maximizing the clipped surrogate objective:

$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \min\left( \text{ratio}_\theta(\tau) A(\tau), \text{clip}(\text{ratio}_\theta(\tau), 1 - \epsilon, 1 + \epsilon) A(\tau) \right) \right]$

where $\text{ratio}_\theta(\tau) = \frac{\pi_\theta(\tau)}{\pi_{\text{old}}(\tau)}$ is the likelihood ratio between the current policy and the policy that generated the trajectory (the "old" policy, frozen at the start of the update), and $\epsilon$ is the clipping hyperparameter (not explicitly stated in the paper, but standard for PPO/GRPO is $\epsilon = 0.2$ ).

What it computes: the standard PPO clipped objective. For each trajectory, if the advantage is positive (trajectory was good), the objective encourages increasing its probability under the current policy, but clips the ratio at $1 + \epsilon$ to prevent destructively large updates. If the advantage is negative (trajectory was bad), the objective encourages decreasing its probability, but clips at $1 - \epsilon$ . The $\min$ operation ensures the objective is a lower bound on the unclipped objective, making the update conservative.

Why this form — the clipping mechanism. The likelihood ratio $\text{ratio}_\theta(\tau)$ can become very large if the current policy assigns much higher probability to the trajectory than the old policy did. Without clipping, a single high-advantage trajectory could dominate the gradient and cause catastrophic forgetting or policy collapse. The clip bounds $[1 - \epsilon, 1 + \epsilon]$ restrict how much the policy can change in a single update, requiring multiple iterations to shift behavior significantly. This is especially important for tool-use trajectories, which can be long (up to 50 turns) and have correspondingly extreme likelihood ratios.

Training Stabilization Techniques

The paper identifies three failure modes that destabilize RL training for this agentic setting and proposes filters to address each:

1. Homogeneity filtering. When the standard deviation of rewards in a rollout batch is smaller than 0.1, the batch is discarded for that training step. This occurs when all 8 trajectories produce very similar behaviors and receive similar rewards — typically on tasks that are either trivially easy (all trajectories succeed with similar cost) or impossibly hard (all trajectories fail in similar ways). In either case, the advantage calculation in Equation 3 would produce values near zero (since all rewards are similar), providing no meaningful gradient signal. Training on such batches wastes compute and can introduce noise from floating-point imprecision in the advantage calculation.

2. Format consistency filtering. If a trajectory's output does not conform to the expected tool-call format (e.g., malformed JSON, missing required parameters, invalid function names), it is filtered out before reward computation and policy update. This prevents the model from learning from syntactically invalid outputs that happened to lead to good outcomes by chance, or from being penalized for format errors that are orthogonal to task performance. The filter acts as a hard constraint: format errors result in the trajectory being excluded entirely, which implicitly pushes the policy toward generating valid structured outputs.

3. Invalid output filtering. If a trajectory does not produce a valid answer or output (e.g., the model generates garbled text, fails to terminate with an answer, or produces output that cannot be parsed), it is filtered out. This is distinct from format filtering — a trajectory could have perfectly formatted tool calls but still fail to produce a coherent final answer.

Why these filters are necessary. Tool-use trajectories are long (up to 50 turns) and involve structured outputs that are easy to malform. Without these filters, the model could learn degenerate strategies: generating syntactically invalid tool calls that the environment silently ignores (leading to a loop), producing the same tool call repeatedly (exploiting the homogeneity case), or outputting empty final answers. The filters act as guardrails that keep the training distribution focused on meaningful trajectories. The downside is that they reduce the effective batch size and could introduce bias if certain types of tasks systematically trigger the filters, but the paper does not report this being a practical issue.

What the Model Learns (Implicitly)

The paper does not provide a mechanistic analysis of the learned policy, but the reward structure implies several capabilities the model must develop:

Tool suitability assessment: given a sub-problem (e.g., "what is the Alon-Tarsi number?"), the model must map it to the tool whose description suggests relevant capability. This requires reading tool descriptions and matching them to the semantic content of the current reasoning step.
Cost-benefit reasoning: the model must learn that for some sub-problems, a cheaper tool (Qwen3-32B) suffices, while for others, only GPT-5 will work. This requires internalizing the relationship between problem difficulty and tool capability — a form of learned difficulty estimation.
Sequential decomposition: the model must learn to break a complex task into sub-tasks and route each to an appropriate tool, then synthesize results. This is not explicitly rewarded (there is no "decomposition reward"), but emerges because the outcome reward requires solving the full task, which implicitly requires effective decomposition.
Preference following: the model must condition its tool selection on the numeric preference vector, learning to weight tool usage counts, cost, and latency differently depending on the $P$ values provided. This requires the model to internalize the mapping from preference vector components to behavioral changes — a form of instruction following specific to tool selection.

4. Key Insights and Innovations

Innovation 1: Intelligence as Orchestration — Reframing the Capability Ceiling as a Coordination Problem

The paper's most fundamental conceptual move is reframing the challenge of solving hard reasoning tasks from "build a smarter model" to "build a smarter coordinator of models." Prior work on tool use (Toolformer, ToolLLM, WebGPT, Search-R1, ToRL) treated tools as augmentations of a single model's intelligence — the model does the reasoning, and tools provide factual lookup or computation that the model lacks. ToolOrchestra inverts this relationship: the orchestrator model does not need to be capable of solving the task itself; it only needs to know who to ask and in what order. The orchestrator's intelligence is relational rather than intrinsic — it lies in the edges of the delegation graph, not the nodes.

This is a departure from the implicit assumption running through prior tool-use and reasoning work: that the model wielding the tools must itself be capable enough to solve the task with tool assistance. The standard approach is to take a strong model (GPT-5, Claude) and give it tools to make it stronger. ToolOrchestra shows that an 8B model — which on its own scores 3.2% on HLE with no tools (Table 1) — can coordinate tools to achieve 37.1%, surpassing the very frontier models it delegates to. The orchestrator's own capability is almost irrelevant; what matters is its learned policy for routing sub-problems.

This reframing has deep implications for how the field thinks about scaling. The prevailing narrative — larger models are more capable, so build larger models — implicitly assumes that intelligence is a property of individual models. ToolOrchestra provides an existence proof that intelligence can be a property of composite systems where the coordinator is far weaker than the coordinated components. This is not merely an architectural choice; it is an argument that the unit of analysis for intelligence should be the system, not the model. The finding that Orchestrator-8B outperforms Qwen3-235B-A22B (a ~30× larger model) on all three benchmarks (37.1 vs. 32.8 on HLE, 76.3 vs. 74.2 on FRAMES, 80.2 vs. 75.6 on τ2-Bench, Table 1) while using roughly one-third the cost is the empirical anchor: the composition of weaker and stronger components, mediated by a trained coordination policy, can exceed any individual component.

The paper is also making a diagnostic claim about what limits frontier models on hard benchmarks. GPT-5 achieves 35.1% on HLE with basic tools but drops to 21.2% when given access to a broader toolset including specialized and generalist LLMs (Table 1). The failure is not capability — GPT-5 demonstrably has the knowledge and reasoning capacity to score ~35% — but allocation: when given the freedom to delegate, it delegates poorly, routing tasks to GPT-5-mini due to self-enhancement bias rather than strategically matching sub-problems to appropriate tools. The diagnostic insight is that frontier models are bottlenecked not by what they know, but by how they decide what to use. This recasts the HLE challenge from a knowledge-and-reasoning problem to a meta-reasoning-and-delegation problem, which is a fundamentally different research target.

Innovation 2: The Preference Vector as a Controllability Interface — Steerable Agents Without Retraining

ToolOrchestra introduces a mechanism for making tool-use agents controllable along multiple axes — tool selection, cost, latency, accuracy — through a single numeric interface that requires no retraining to adapt to new user preferences at inference time. The preference vector $P = [p_{t_1}, ..., p_{t_n}, p_{\text{outcome}}, p_{\text{compute}}, p_{\text{latency}}]$ encodes desired optimization weights as numbers in $[0, 1]$ , and it is integrated directly into the RL reward (Equation 2) as a dot product with the normalized trajectory feature vector.

This is not a prompting trick — it is a training methodology that bakes preference-following into the policy's objective function, making it a first-class capability rather than a post-hoc constraint. The distinction matters. Prior work on efficient or controllable tool use (OTC, SMART, Efficient Agents) operates at the inference level: apply penalties, add prompts, or post-process outputs to enforce cost constraints on a model that was trained only for accuracy. These approaches are fundamentally reactive — they constrain a policy that does not internally represent preferences as part of its optimization target. ToolOrchestra's GRPO training with preference-vector-conditioned rewards means the model learns to generate different behaviors for different preference vectors, because the reward it received during training varied with $P$ . The preference vector is part of the state on which the policy is conditioned, not an external filter applied after the fact.

The significance is that this creates a continuously steerable agent. A single Orchestrator checkpoint can be deployed in different contexts — a cost-sensitive batch processing pipeline, a latency-sensitive interactive assistant, a privacy-conscious on-premise deployment — and produce appropriately different tool-use strategies simply by changing the preference vector at inference time. The results in Table 3 demonstrate that this steerability preserves performance: Orchestrator-8B achieves 46.7 on the preference-aware benchmark, substantially above GPT-5 (34.6) and Claude Opus 4.1 (40.2). And because the preference vector is a linear interface (dot product with trajectory features), the model can plausibly interpolate between training preferences — if it learned "high accuracy, low cost" and "high accuracy, low latency" during training, it should generalize to "high accuracy, medium cost, medium latency" without having seen that exact combination.

This is a fundamental advance in agent controllability, not an incremental refinement. Prior work treated user preferences as an afterthought; ToolOrchestra makes them a core training objective, enabling a single model to serve diverse deployment scenarios. The preference-aware benchmark itself (constructed in Section 3.3 by generating (PI, P) pairs validated for consistency) is also a methodological contribution — it provides a principled way to evaluate whether agents actually follow user-specified constraints, which has been largely absent from tool-use evaluations that focus exclusively on accuracy.

Innovation 3: The Bias Diagnosis — Prompted Orchestrators Exhibit Systematic Delegation Pathologies

Before proposing a solution, the paper makes a sharp diagnostic contribution: it identifies and names two specific, systematic failure modes of prompted LLMs acting as orchestrators, and demonstrates that these biases actively degrade performance rather than being merely suboptimal.

The pilot study (Appendix A, Figure 3) is methodologically simple but conceptually powerful. When GPT-5 is prompted to orchestrate a tool set including GPT-5, GPT-5-mini, Qwen3-32B, and Qwen2.5-Coder-32B, it calls GPT-5 or GPT-5-mini in 98% of cases — what the authors term other-enhancement bias, a preference for developmentally related variants of oneself. When Qwen3-8B is the orchestrator, it delegates to GPT-5 in 73% of cases — self-enhancement bias, a weaker model reflexively defaulting to the strongest available tool regardless of cost or suitability. These are not minor quirks; GPT-5's HLE performance drops from 35.1% (with basic tools only) to 21.2% (with expanded toolset including specialized and generalist LLMs, Table 1) — access to more tools makes it less effective, because its biased delegation policy routes tasks to suboptimal tools.

This diagnostic insight matters for two reasons. First, it explains a puzzle in prior results: why some papers find that tool use helps while others find it hurts or has mixed effects. The answer is not about the tools themselves but about the allocation policy — if the orchestrator cannot strategically decide which tool to use when, adding more tools can harm performance by increasing the probability of suboptimal delegation. The field had implicitly assumed that giving a capable model more options would monotonically improve outcomes; ToolOrchestra shows this assumption is false under prompted delegation.

Second, it establishes that prompting cannot fix this through instruction alone. The pilot study instructed models to "achieve best results while maintaining lowest cost" — an explicit efficiency directive — yet the bias patterns persisted. This is not a failure of prompt engineering; it is evidence that the models' pretraining distributions and RLHF tuning have embedded preferences (for self-variants, for the strongest available model) that override explicit cost-efficiency instructions. The implication is that training, not prompting, is necessary to produce unbiased orchestration — which motivates the entire RL-based ToolOrchestra approach.

This diagnostic contribution is intellectually significant beyond the paper's own solution. It identifies a new failure mode class for compound AI systems: delegation bias. As the field moves toward architectures where multiple models collaborate, understanding and mitigating systematic biases in how models route tasks to each other becomes critical. The paper provides both the taxonomy (self-enhancement vs. other-enhancement bias) and the empirical demonstration that these biases have performance consequences, establishing a research problem that will outlive any specific training method.

Innovation 4: Verifiable Synthetic Data for Multi-Turn Agentic Tool Use — ToolScale as an Enabling Contribution

The ToolScale data synthesis pipeline (Section 3.3, Figure 4) addresses a bottleneck that is simultaneously mundane and critical: training RL-based orchestration requires thousands of verifiable multi-turn tool-use trajectories, and no such dataset exists. The contribution is not a novel algorithm but an enabling infrastructure that makes the entire RL training approach feasible.

What makes ToolScale distinctive is the verifiability of its generated tasks. Each synthesized task includes a ground-truth action sequence and a three-criterion evaluation rubric: (1) execution correctness (does the database state after running the trajectory match the state after the gold actions?), (2) process fidelity (was required information communicated?), and (3) operation completeness (were all necessary database entries operated on?). This triple-check means the binary outcome reward — the foundation of the entire RL training loop — can be computed automatically and deterministically, without human annotation and without the circularity of using the model under training as its own judge. This is a hard constraint on data synthesis that prior agentic datasets (which often rely on human-written solutions or model-generated "correct" answers that may not be verifiable) do not consistently satisfy.

The two-stage design — first simulate the environment (database schema, tool APIs), then generate diverse tasks grounded in that environment — is pragmatically smart. By separating environment generation from task generation, the pipeline produces tasks that are internally consistent (all tasks in a domain reference the same database and tools) while maintaining diversity of intents. The 10-domain coverage (finance, sports, e-commerce, medicine, entertainment, railway, restaurant, education, travel, weather) provides enough breadth to train generalizable orchestration, while the per-domain tool sets (14–25 tools per domain, Table 5) provide realistic heterogeneity. The explicit difficulty escalation step ("evolving" tasks by adding constraints and complications) ensures the dataset includes multi-step reasoning challenges rather than simple single-tool lookups.

The significance of the ToolScale contribution extends beyond this paper. The paper commits to releasing the dataset publicly, which means the broader research community gains access to a resource for training and evaluating tool-use agents that did not previously exist. The pipeline itself — environment simulation → diverse task generation → difficulty escalation → automatic verification and filtering — is a reusable template for generating similar data in other domains or with different tool sets. In a field where data scarcity for agentic tasks is a recognized bottleneck, a validated, scalable synthesis pipeline is a contribution with multiplicative impact.

That said, this is an incremental infrastructure contribution rather than a fundamental algorithmic advance. The individual components (LLM-based data generation, programmatic verification) are not novel in themselves; the value lies in the specific assembly and the demonstrated ability to support RL training at scale. The paper's central claims do not depend on ToolScale being uniquely innovative — they depend on it being effective, and the RL training results (37.1% on HLE) provide strong evidence that it is.

Innovation 5: Generalization Through Configurational Diversity — Training-Time Randomization as a Robustness Strategy

The paper makes a subtle but practically important methodological contribution in its approach to generalization. Rather than training the Orchestrator on a fixed tool set and hoping it transfers, or using explicit domain randomization as a separate data augmentation step, ToolOrchestra bakes configurational diversity directly into the training loop: for each training instance, a random subset of tools is made available, and pricing schedules are varied to reflect heterogeneous user access patterns (Section 3.3). The model never sees the same tool configuration twice during training, forcing it to learn tool-selection heuristics that depend on tool capabilities (as described in the JSON specification) rather than tool identities.

This is what enables the generalization results in Table 2. When evaluated with a completely unseen tool set — Claude Opus 4.1, o3-mini, GPT-4o as query writers; Claude and Codestral models as code writers; DeepSeek-Math and OpenMath as math models; Gemma-3-27b as a generalist — Orchestrator-8B achieves 22.0 on HLE and 73.8 on FRAMES, substantially outperforming all prompted baselines (GPT-5: 16.4 and 54.8) at lower cost (34.8 vs. 81.3 cents). The model did not merely memorize which tools to use for which tasks; it learned to read tool descriptions and map them to sub-problem requirements, a capability that transfers across tool identities.

The same principle applies to pricing generalization (Appendix H, Table 4): training with pricing configurations from 10 different third-party API providers (TogetherAI, Venice AI, Chutes, NEBIUS, Lambda, Hyperbolic, Cloudflare, Novita, AIML, Fireworks AI — Appendix E) produces a model that adapts to a held-out pricing regime (DeepInfra) without performance degradation. This is not a theoretical guarantee (there is no proof that configurational diversity produces generalization), but it is a strong empirical regularity with practical implications: training with randomized configurations is a simple, scalable recipe for building orchestrators that work across deployment environments.

What distinguishes this from standard domain randomization is the specificity of the randomization. The paper does not randomly perturb all aspects of the environment — it randomizes the dimensions that matter for generalization (tool availability and cost), while keeping the task semantics fixed. This targeted diversity is more efficient than generic randomization and produces interpretable robustness: the model learns that tools with certain capability descriptions are appropriate for certain sub-problems, regardless of their API endpoint or price. The insight is methodological — diversify the dimensions you want the model to be invariant to, not everything — and it is demonstrated through both tool-set generalization (Section 6.3) and pricing generalization (Appendix H).

This is an incremental but practically significant contribution. The principle is not new (domain randomization has a long history in robotics and sim-to-real transfer), but its application to LLM tool orchestration — where the "domain" is the set of available tools and their costs — is novel, and the empirical demonstration that it produces robust generalization on state-of-the-art benchmarks with a small model is convincing evidence of its effectiveness.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three distinct benchmarks: (1) Humanity's Last Exam (HLE) — a large-scale collection of PhD-level questions across mathematics, humanities, and natural sciences, using only the text-only subset (questions that do not require images) designed to be unambiguous and not solvable by simple web search; (2) FRAMES — a dataset of 824 multi-hop factuality and reasoning questions requiring 2–15 Wikipedia articles, covering numerical, tabular, temporal, and multi-constraint reasoning; (3) τ2-Bench — a benchmark for evaluating conversational tool-use agents across three domains (telecom, retail, airline) where models must use functions like cancel, refund, and check_ticket to solve problems in dialogue with users. HLE and FRAMES are used to assess complex reasoning with tool augmentation; τ2-Bench specifically tests function-calling capability in multi-turn interactions.
Base model(s). The Orchestrator is built on Qwen3-8B (an 8-billion-parameter open-source model), chosen because it represents a "small language model" that the paper hypothesizes can suffice for orchestration if taught to coordinate more intelligent tools strategically. For baselines, the paper evaluates a range of models spanning scales: Qwen3-8B, Llama-3.3-Nemotron-Super-49B-v1, Llama-3.3-70B-Instruct, Qwen3-235B-A22B (a mixture-of-experts model), Claude Opus 4.1, and GPT-5. These cover the spectrum from small open-source models to closed-source frontier systems, enabling comparison of the 8B orchestrator against models up to ~30× larger.
Metrics. Three primary metrics are reported: (1) Accuracy — for HLE, the percentage of text-only questions answered correctly (multiple-choice or short-answer, graded by exact match or GPT-5 as judge); for FRAMES, the percentage of multi-hop factuality questions answered correctly; for τ2-Bench, the percentage of tasks where the agent successfully completes the required function calls and satisfies task criteria. (2) Cost — total monetary cost in US cents, computed by converting input and output tokens to dollar amounts using third-party API pricing systems (TogetherAI for evaluation consistency), summed across all tool invocations in a trajectory. For basic tools, the API call price is used. (3) Latency — total wall-clock time in minutes consumed by a trajectory, including model inference time and tool execution time. These three metrics together capture the accuracy-efficiency tradeoff that is central to the paper's claims. Additionally, for the preference-aware evaluation, a scalar preference adherence score is computed using the formula in Appendix L (Equation 6), which normalizes trajectory feature vectors against a baseline model's behavior and computes a weighted sum with the preference vector.
Baselines. The paper compares against three tiers of systems, all evaluated in the same tool-use setting: (1) No-tool baselines — LLMs answering directly with no access to external tools (Qwen3-8B, Llama-Nemotron-49B, Llama-3.3-70B, Qwen3-235B-A22B, Claude Opus 4.1, GPT-5). (2) Basic tools baselines — the same models equipped with web search (Tavily API), code interpreter (Python sandbox), and local search (FAISS index with Qwen3-Embedding-8B), plus domain-specific functions for τ2-Bench. (3) Expanded toolset baselines — the same models given access to basic tools plus specialized LLMs (Qwen2.5-Coder-32B-Instruct as code writer, GPT-5 and GPT-5-mini as code writers, Qwen2.5-Math-72B and Qwen2.5-Math-7B as math models) and generalist LLMs (GPT-5, GPT-5-mini, Llama-3.3-70B-Instruct, Qwen3-32B). This third tier is the direct comparison point for Orchestrator-8B, since it uses the same tool set. The paper also compares against "existing reported SOTA" results from prior publications (GPT-5, o3, GPT-4o) but notes these are on the full HLE set (including image questions) while the paper's experiments are on the text-only subset.
Generation budget / compute accounting. Compute is measured in monetary cost (US cents) and wall-clock latency (minutes), not in FLOPs or token counts (though token counts underpin the cost calculation). For proprietary models (GPT-5, Claude), official API pricing is used; for open-source models, TogetherAI's pricing system is applied for fair comparison. During evaluation, the Orchestrator is allowed a maximum of 50 turns per task, with inference temperature set to 0 (greedy decoding). For the cost-performance tradeoff analysis (Figure 6), the maximum number of turns is varied (10, 20, 50, 100) to produce different points on the cost-accuracy curve. There is no explicit "generation budget" in the sense of "generate N candidate solutions" — the cost is determined endogenously by the orchestrator's tool-calling decisions, which is precisely what the RL training optimizes.
Cross-validation / statistical protocol. The paper does not employ cross-validation for its main results (likely because the benchmarks have fixed test sets — HLE 500 text-only questions, FRAMES 824 questions, τ2-Bench with tasks across three domains — and the Orchestrator is a single trained model, not a hyperparameter sweep that would require held-out selection). For the preference-aware evaluation, the paper constructs separate training and evaluation sets of preference instruction-vector pairs (Pairs_train and Pairs_eval, Section 3.3) to ensure that preference adherence is tested on unseen preferences. The policy is evaluated by running each test instance once with temperature 0, producing deterministic trajectories for a given model checkpoint.

Main Quantitative Results

Overall Performance Comparison Across Benchmarks

The headline result is presented in Table 1: Orchestrator-8B achieves 37.1% on HLE, 76.3% on FRAMES, and 80.2% on τ2-Bench using the expanded toolset (basic tools + specialized LLMs + generalist LLMs), outperforming all baselines on all three benchmarks while using substantially lower cost and latency.

On HLE specifically:

Orchestrator-8B (37.1%) surpasses GPT-5 with basic tools (35.1%) by 2.0 absolute percentage points, while costing only 9.2 cents vs. 30.2 cents — a roughly 3.3× cost reduction — and requiring 8.2 minutes vs. 19.8 minutes latency.
Orchestrator-8B outperforms GPT-5 with the same expanded toolset (21.2%) by a dramatic 15.9 percentage points. GPT-5's performance drops from 35.1% (basic tools) to 21.2% (expanded toolset) — a finding the paper attributes to biased tool delegation (Section 6.1).
Orchestrator-8B outperforms Claude Opus 4.1 with expanded tools (34.6%) by 2.5 points at roughly 5.7× lower cost (9.2 vs. 52.5 cents) and 3.1× lower latency (8.2 vs. 25.6 minutes).
Orchestrator-8B outperforms Qwen3-235B-A22B with expanded tools (32.8%) by 4.3 points at roughly 3.2× lower cost (9.2 vs. 29.7 cents) and 2.6× lower latency (8.2 vs. 21.2 minutes).

On FRAMES:

Orchestrator-8B (76.3%) surpasses GPT-5 with basic tools (74.0%) by 2.3 points, and far exceeds GPT-5 with expanded tools (57.5%) — an 18.8-point gap.
Orchestrator-8B outperforms Claude Opus 4.1 with expanded tools (72.8%) by 3.5 points and Qwen3-235B-A22B (74.2%) by 2.1 points.
The cost advantage is consistent: 9.2 cents for Orchestrator vs. 30.2 (GPT-5 basic tools), 52.5 (Claude Opus 4.1 expanded), and 29.7 (Qwen3-235B-A22B expanded).

On τ2-Bench:

Orchestrator-8B (80.2%) outperforms GPT-5 with basic tools (77.7%) by 2.5 points, demonstrating strong function-calling capability on a benchmark specifically designed for tool-use evaluation. Detailed τ2-Bench cost and latency statistics are provided in Table 16, showing Orchestrator-8B achieves 80.2% at 10.3 cents and 8.6 minutes vs. GPT-5 basic tools at 77.7%, 31.3 cents, and 20.2 minutes.

Key pattern across all three benchmarks: Qwen3-8B with expanded tools (30.6 HLE, 68.9 FRAMES, 72.3 τ2-Bench) already substantially outperforms Qwen3-8B with basic tools (4.7 HLE, 26.5 FRAMES, 40.7 τ2-Bench), indicating that access to stronger models as tools provides large gains even for a prompted small model. However, Qwen3-8B's performance remains below Orchestrator-8B (30.6 vs. 37.1 on HLE), demonstrating that RL training provides additional gains beyond tool access alone. This comparison isolates the effect of ToolOrchestra's training: same base model (Qwen3-8B), same tool set, but RL-trained vs. prompted — and the RL-trained version is 6.5 points better on HLE, 7.4 on FRAMES, and 7.9 on τ2-Bench.

The GPT-5 degradation phenomenon deserves specific attention: GPT-5's HLE score drops from 35.1 (basic tools) to 21.2 (expanded toolset). This is not a small fluctuation — it is a 39.6% relative decline. The paper attributes this to biased delegation patterns where GPT-5 disproportionately routes tasks to GPT-5-mini (Figure 3, left), but the magnitude of the drop suggests a deeper issue: giving a frontier model more options for delegation can actively harm its performance if it cannot strategically decide which tool to use when. This is the strongest empirical evidence for the paper's core claim that orchestration is a learned skill distinct from raw capability.

Tool-Use Pattern Analysis

Figure 5 and Table 15 provide a detailed breakdown of which tools each orchestrator calls, averaged across HLE, FRAMES, and τ2-Bench. The data reveals stark differences in delegation strategies:

Orchestrator-8B distributes its calls across tools more evenly than any baseline: GPT-5 (1.6 calls per instance), GPT-5-mini (1.7), Qwen3-32B (1.3), local search (1.8), web search (0.7), code interpreter (0.8), with minimal use of Coder-32B (0.2), Math-72B (0.0), Math-7B (0.1), and Llama-70B (0.0). The total calls are spread across 7+ distinct tools.
GPT-5 (as orchestrator) concentrates 71.2% of its tool calls on GPT-5-mini (5.6 calls) and GPT-5 (2.7 calls), with negligible use of other models. Combined, GPT-5 variants account for ~79% of all tool calls. This is the self-enhancement bias quantified: GPT-5 overwhelmingly prefers its own developmentally-related variant.
Claude Opus 4.1 concentrates 67.2% of its calls on GPT-5 (6.2 calls), with minimal distribution to other tools. This is the "default to strongest" pattern — Claude reflexively delegates to the frontier model rather than matching sub-problems to specialized tools.
Qwen3-235B-A22B shows a more balanced pattern than Claude or GPT-5 but still leans heavily on GPT-5 (6.2 calls, roughly 42% of all tool calls), supplemented by code interpreter (2.2) and local search (1.4).
Qwen3-8B (prompted, not RL-trained) delegates to GPT-5 heavily (6.0 calls) with some web search (1.2) and code interpreter (1.6) — the self-enhancement bias pattern from the pilot study replicated at scale.

The critical insight from Table 15 is that Orchestrator-8B uses GPT-5 less than any baseline — 1.6 calls per instance vs. 6.0 (Qwen3-8B), 5.1 (Nemotron-49B), 6.2 (Qwen3-235B), 6.2 (Claude Opus 4.1), and 2.7 (GPT-5 as orchestrator) — yet achieves the highest accuracy across all three benchmarks. This directly validates the paper's efficiency claim: the orchestrator does not outperform by calling GPT-5 more often; it outperforms by calling GPT-5 more strategically, using it only when the sub-problem warrants the cost, and routing other sub-problems to cheaper or more specialized tools. The orchestrator's intelligence lies in the pattern of delegation, not in the volume of delegation to strong models.

Cost-Performance Tradeoff Analysis

Figure 6 plots HLE accuracy as a function of total cost for Orchestrator-8B, GPT-5, Claude Opus 4.1, and Qwen3-235B-A22B, with different points on each curve corresponding to different maximum-turn limits (10, 20, 50, 100 turns). The figure demonstrates that Orchestrator-8B achieves a strictly better accuracy-cost frontier than any baseline.

At any given cost level, Orchestrator-8B achieves higher HLE accuracy. Conversely, to reach a given accuracy target, Orchestrator-8B requires less cost. For example, to achieve ~35% HLE accuracy, Orchestrator-8B spends roughly $0.2–0.3, while GPT-5 requires approximately$ 0.6–0.7 — a 2–3× cost advantage at iso-accuracy. The Orchestrator curve sits to the upper-left of all baseline curves, indicating Pareto-dominant cost-effectiveness.

Notable details from the curve shapes: all models show improving accuracy as the turn limit (and thus cost) increases, but the slopes differ. Orchestrator-8B achieves a large fraction of its ultimate accuracy at very low cost (the curve rises steeply at low costs), then plateaus more gradually. This indicates that the orchestrator learns to solve many problems with relatively few, cheap tool calls — consistent with the balanced tool-use pattern in Figure 5 where expensive GPT-5 calls are reserved for specific needs.

The figure also reveals the diminishing returns of additional turns: the curves flatten at higher costs, suggesting that beyond some budget, additional tool calls do not translate into higher accuracy. This is expected — once the orchestrator has gathered sufficient information and synthesized an answer, further tool invocations are either redundant or indicative of failure to converge. The paper's 50-turn limit appears to be generous enough to capture the plateau for all models.

Generalization to Unseen Tools

Table 2 presents results when Orchestrator-8B is evaluated with a completely unseen tool set — models that were not present in any training configuration: Claude Opus 4.1, o3-mini, and GPT-4o as query writers; Claude Opus 4.1, Claude Sonnet 4.1, and Codestral-22B as code writers; OpenMath-Llama-2-70b and DeepSeek-Math-7b-Instruct as math models; and Claude Opus 4.1, Claude Sonnet 4.1, and Gemma-3-27b-it as generalist models. Basic tools (web search, local search, code interpreter) remain the same as in the main evaluation.

The headline result: Orchestrator-8B achieves 22.0% on HLE, 73.8% on FRAMES, and 48.8% on τ2-Bench with these unseen tools, substantially outperforming all prompted baselines at lower cost (34.8 cents vs. 37.9 for Qwen3-8B, 87.2 for Qwen3-235B, 102.4 for Claude Opus 4.1, 81.3 for GPT-5).

On HLE, the gap is stark: Orchestrator-8B at 22.0% vs. GPT-5 at 16.4%, Claude Opus 4.1 at 17.8%, and Qwen3-235B at 14.7%. Even though the tools available to all models are the same (the unseen set), only Orchestrator-8B effectively leverages them. This is direct evidence that the RL-trained orchestrator learned capability-based tool selection — it reads the generated descriptions for each unseen tool and maps sub-problems to tools based on described strengths — rather than memorizing which specific tool names to call for which tasks.

On FRAMES, the generalization is even more impressive: Orchestrator-8B at 73.8% nearly matches its main-evaluation performance (76.3%) despite using a completely different set of models. This suggests that for factuality and retrieval tasks, the orchestrator's strategy is robust to which strong models are available, as long as the capability descriptions are accurate.

The τ2-Bench generalization result (48.8%) is lower than the main evaluation (80.2%), which is expected given that τ2-Bench involves domain-specific function calling that may require familiarity with specific tool interfaces. Still, Orchestrator-8B outperforms GPT-5 (44.8%) and Claude Opus 4.1 (43.4%), indicating that even with unfamiliar tools, the orchestration policy transfers better than prompted baselines.

A notable pattern in Table 2: the prompted small models (Qwen3-8B, Llama-Nemotron-49B) perform relatively better in this generalization setting than in the main evaluation (e.g., Qwen3-8B scores 12.6 on HLE generalization vs. 30.6 in the main evaluation — a smaller drop proportionally than GPT-5's 21.2 → 16.4). This is likely because prompted models rely heavily on defaulting to the strongest available tool (GPT-5 in the main evaluation), and when that tool is replaced with unfamiliar alternatives, their simple delegation heuristic breaks down. Orchestrator-8B's learned, capability-aware strategy degrades more gracefully.

Pricing Generalization

Appendix H (Table 4) evaluates Orchestrator-8B under a pricing configuration not encountered during training — specifically, the DeepInfra pricing scheme, whereas training used TogetherAI, Venice AI, Chutes, NEBIUS, Lambda, Hyperbolic, Cloudflare, Novita, AIML, and Fireworks AI pricing (Appendix E). The results show robust generalization: Orchestrator-8B achieves 36.9 on HLE, 76.6 on FRAMES, and 80.4 on τ2-Bench at 7.5 cents and 7.8 minutes latency, nearly identical to its main-evaluation performance (37.1, 76.3, 80.2 at 9.2 cents, 8.2 minutes).

This is a critical robustness check because it addresses the concern that the efficiency gains in the main evaluation might be specific to the particular pricing scheme used during training (TogetherAI). The fact that Orchestrator-8B maintains its performance and cost-efficiency under a held-out pricing scheme indicates that it learned a general cost-sensitivity — it adapts its tool selection to relative costs across tools — rather than memorizing absolute cost thresholds. All baselines also maintain similar relative performance under the new pricing, suggesting that the efficiency ordering is consistent across pricing schemes, but the stability of Orchestrator-8B's cost advantage (7.5 vs. ~17–52 for other models) confirms that the efficiency gain is not an artifact of the training pricing.

User Preference Adherence

Table 3 reports results on the preference-aware benchmark, where each question is augmented with a preference instruction (e.g., "prefer local models for privacy" or "optimize for cost over latency") and the model's adherence is scored using the preference-aware reward defined in Appendix L (Equation 6). The preference vector encodes weights on tool usage, outcome accuracy, cost, and latency, and the score reflects how well the trajectory's tool-usage pattern aligns with the preferences while still solving the task.

Orchestrator-8B achieves 46.7 on the preference-aware benchmark, substantially exceeding GPT-5 (34.6), Claude Opus 4.1 (40.2), and Qwen3-235B-A22B (37.9). This is a notable result because GPT-5 and Claude Opus 4.1 are explicitly prompted with the same preference instructions — the gap reflects their inability to faithfully adapt tool-use strategy to user-specified constraints, consistent with the pilot study's finding that prompted models exhibit systematic delegation biases that override explicit efficiency instructions.

The preference adherence metric aggregates across all three benchmarks (HLE, FRAMES, τ2-Bench), indicating that the orchestrator's preference-following capability is not task-specific. The RL training with preference-vector-conditioned rewards (Equation 2) produces a model that can be steered at inference time simply by changing the numeric values in the preference vector, without any retraining or prompt engineering. This is the operationalization of the paper's controllability claim: a single Orchestrator checkpoint can serve different deployment scenarios (cost-sensitive batch processing, latency-sensitive interactive use, privacy-conscious on-premise deployment) by adjusting the preference vector.

A subtlety in interpreting Table 3: all models, including Orchestrator-8B, score lower on the preference-aware version of the benchmarks than on the standard version (compare Orchestrator-8B's 46.7 here with 37.1 on HLE alone — but note the 46.7 is an aggregate score combining all three benchmarks under a different metric, not directly comparable to the HLE accuracy in Table 1). The absolute values are less meaningful than the relative ordering, since the preference-aware metric is a custom scalar computed from normalized trajectory features (Appendix L). What matters is that Orchestrator-8B substantially outperforms baselines on this metric, indicating that its tool-use patterns are more aligned with user-specified preferences while maintaining correctness.

Ablation Studies and Robustness Checks

The paper's ablation studies are primarily implicit in its design choices and comparative analyses, rather than being presented in dedicated ablation sections. The following ablated dimensions can be inferred from the experimental structure:

RL training vs. prompting (implicit ablation): The comparison between Qwen3-8B with expanded tools (prompted, 30.6 HLE, 68.9 FRAMES, 72.3 τ2-Bench, Table 1) and Orchestrator-8B with expanded tools (RL-trained, 37.1 HLE, 76.3 FRAMES, 80.2 τ2-Bench) isolates the effect of ToolOrchestra's RL training. Same base model, same tool set, but the RL-trained version improves by 6.5, 7.4, and 7.9 points respectively. This demonstrates that prompting alone — even with a well-designed system prompt and explicit efficiency instructions — cannot recover the balanced delegation strategy that RL learns.

Expanded toolset vs. basic tools (implicit ablation on tool diversity): Comparing models' performance with basic tools only vs. with basic tools + specialized LLMs + generalist LLMs (Table 1, middle vs. bottom sections) reveals that having access to stronger models as tools helps only if the orchestrator knows how to use them. Qwen3-8B improves dramatically (4.7 → 30.6 on HLE, 26.5 → 68.9 on FRAMES), while GPT-5 degrades (35.1 → 21.2 on HLE, 74.0 → 57.5 on FRAMES). This is an implicit ablation on the orchestrator's allocation capability: the same expanded toolset can be a massive benefit or a massive detriment depending on the orchestration policy. The paper does not run an explicit "RL-trained Orchestrator with basic tools only" comparison, which would have further isolated the contribution of RL training from the contribution of tool access — Orchestrator-8B with basic tools might have revealed whether RL training improves even simple tool use or primarily excels at model-as-tool orchestration.

Preference vector presence/absence (implicit ablation): The comparison between Table 1 (standard evaluation, no explicit preference instructions) and Table 3 (preference-aware evaluation, explicit preference instructions) tests whether the preference-conditioning mechanism works. OCRhestrator-8B's large margin over baselines in Table 3 (46.7 vs. 34.6–40.2) indicates that training with preference vectors produces a genuinely preference-following policy, not one that ignores preferences and coincidentally aligns with some evaluation criterion. However, the paper does not report an explicit ablation where the preference vector is set to zero or randomized at test time, which would have tested whether the model's behavior actually varies with the preference vector or simply learned a fixed efficient strategy that scores well on average.

Training data composition (implicit ablation): The paper mentions training on "the GeneralThought-430K dataset in conjunction with synthetic data" (Section 4.4) but does not ablate the contribution of each data source. It is unclear whether ToolScale synthetic data alone would suffice, or whether the general reasoning data (GeneralThought-430K) is necessary for maintaining base reasoning capabilities during RL training. This is a notable missing ablation, as it would inform practitioners whether they need to construct domain-specific synthetic data or can rely on general reasoning corpora.

Rollout batch size (implicit): The training configuration uses a rollout batch size of 8 (Section 4.4), but the paper does not ablate this choice. Larger batch sizes would provide more trajectories for GRPO's advantage normalization (Equation 3) but increase computational cost. Smaller batch sizes might make the advantage estimates noisier. The choice of 8 is standard for GRPO (cf. DeepSeek-R1) but not empirically justified for this specific setting.

GRPO vs. alternative RL algorithms (not ablated): The paper uses GRPO without comparing to alternatives like PPO with a learned value function, REINFORCE with baseline, or DPO-style preference optimization. The choice is motivated by prior work (DeepSeek-R1, DeepSeekMath) but not empirically validated for tool orchestration. This is defensible — the paper's contribution is the overall training methodology, not the specific RL algorithm — but it leaves open the question of whether GRPO's group-based advantage normalization is particularly well-suited to tool-use trajectories or whether any policy gradient method would work similarly.

Format and homogeneity filters (not ablated as on/off): The training stabilization techniques — homogeneity filtering (discard batch if reward std < 0.1), format consistency filtering, and invalid output filtering (Section 3.2) — are described as necessary for stable training but are not ablated. The paper does not show training dynamics with and without these filters, making it difficult to assess whether they are genuinely critical or merely conservative engineering choices. In principle, filtering out batches with low reward variance could bias training away from tasks that are consistently easy or consistently hard — the paper does not discuss whether this creates a curriculum effect.

Maximum turn limit (ablated implicitly via Figure 6): Figure 6 varies the maximum number of allowed turns (10, 20, 50, 100) and shows the resulting accuracy-cost tradeoff. This is functionally an ablation on the turn budget, revealing that Orchestrator-8B's accuracy improves with more turns but plateaus, and that the cost advantage over baselines holds across all turn limits. The 50-turn limit used in the main evaluation appears to be near the saturation point.

Unseen tool generalization (Section 6.3) and unseen pricing generalization (Appendix H): These are explicit robustness checks testing whether the trained policy transfers to held-out tool identities and held-out pricing schemes. Both show strong generalization, supporting the claim that configurational diversity during training produces a policy that adapts to tool capabilities and relative costs rather than memorizing specific tools or absolute prices.

Critical Assessment

Claim 1: A small orchestrator can outperform frontier models on hard reasoning benchmarks while being substantially more cost-efficient.

What was tested: Orchestrator-8B vs. GPT-5, Claude Opus 4.1, Qwen3-235B-A22B on HLE, FRAMES, and τ2-Bench (Table 1), with cost and latency reported for each.

Assessment: The experiments genuinely demonstrate this claim with important qualifications. Orchestrator-8B's 37.1% on HLE vs. GPT-5's 35.1% is a real improvement (2.0 points), but the absolute accuracy is modest — both systems fail on ~63% of HLE questions. The efficiency claim is strongly supported: Orchestrator-8B's 9.2 cents vs. GPT-5's 30.2 cents is a 3.3× cost reduction with higher accuracy, which is a Pareto improvement (better on both metrics). This cannot be dismissed as a cost-accuracy tradeoff where Orchestrator is cheaper but worse — it is both cheaper and better.

However, the claim "outperforms frontier models" requires careful scoping. Orchestrator-8B outperforms GPT-5 on HLE when GPT-5 uses basic tools (35.1 vs. 37.1). When GPT-5 uses the same expanded toolset as Orchestrator (bottom section of Table 1), GPT-5's performance drops to 21.2, making Orchestrator's margin 15.9 points. This massive gap is partly an indictment of GPT-5's orchestration capability (it delegates poorly) rather than a pure measure of Orchestrator's absolute capability. A fairer comparison would be: Orchestrator-8B (RL-trained, 8B params) vs. the best possible prompted configuration of GPT-5 with any toolset. The paper shows that GPT-5's best prompted configuration is with basic tools only (35.1), still below Orchestrator. So even under a charitable baseline choice, the claim holds.

A more concerning qualification: all baselines use the same prompted orchestration approach. The paper does not compare against an alternative trained orchestrator — e.g., a fine-tuned version of GPT-5-mini or Qwen3-32B using the same ToolOrchestra methodology. This leaves open the possibility that the gains come from the RL training recipe, not from the orchestration paradigm per se, and that applying ToolOrchestra to a larger base model would produce an even better orchestrator. The paper's framing emphasizes the small size of Orchestrator-8B, but it does not establish that small size is necessary for the result — only that it is sufficient.

Claim 2: Prompted LLMs exhibit systematic delegation biases (self-enhancement and other-enhancement) that degrade performance.

What was tested: Pilot study on 300 HLE problems (Appendix A, Figure 3) with GPT-5 and Qwen3-8B as orchestrators, plus the full benchmark results showing GPT-5's performance drop with expanded tools (Table 1).

Assessment: Strongly supported. The pilot study is clean and well-controlled: same tasks, same tool set, same prompt template, different orchestrator models. GPT-5 calls GPT-5 variants 98% of the time; Qwen3-8B delegates to GPT-5 73% of the time. The full-benchmark replication (Table 1) shows these biases have performance consequences: GPT-5 drops from 35.1 to 21.2 on HLE when given an expanded toolset, and the tool-use breakdown (Table 15) confirms the biased calling patterns persist at scale. The diagnostic is crisp and convincing.

A limitation: the pilot study tests only two orchestrator models (GPT-5 and Qwen3-8B). It is possible that Claude Opus 4.1 or Qwen3-235B would exhibit different bias patterns — Table 15 shows Claude delegates heavily to GPT-5 (6.2 calls), consistent with a "default to strongest" bias, but this is not explicitly characterized as self-enhancement or other-enhancement. The bias taxonomy might not be exhaustive. Additionally, the paper does not test whether better prompting could mitigate these biases — while the pilot study included efficiency instructions, it did not explore whether more explicit instructions ("Do NOT default to GPT-5-mini; evaluate each sub-problem independently") could reduce the bias. This is a minor concern given the consistency of the finding, but it means the claim "prompting cannot fix this" is empirically supported for the specific prompts tested, not proven for all possible prompts.

Claim 3: RL training with multi-objective rewards (outcome + efficiency + preferences) produces a balanced, adaptive tool-use policy.

What was tested: The comparison of Orchestrator-8B's tool-use patterns (Figure 5, Table 15) against baselines, the cost-performance tradeoff curves (Figure 6), the preference adherence results (Table 3), and the generalization experiments (Table 2, Appendix H).

Assessment: Supported with strong circumstantial evidence, but the causal role of individual reward components is not isolated. The balanced tool-use pattern in Figure 5 is the most direct evidence: Orchestrator-8B uses a diverse set of tools, uses GPT-5 less than any baseline, yet achieves the highest accuracy. This implies — but does not prove — that the multi-objective reward produced this balance. An ablation where the efficiency penalty is removed (training with outcome reward only) would directly test whether the cost-awareness in the tool-use pattern is caused by the efficiency reward or would emerge from outcome optimization alone (e.g., because shorter trajectories have fewer opportunities for mistakes). Similarly, an ablation where the preference vector is set to uniform (all dimensions equal) vs. varied during training would test whether the preference-following capability is caused by the preference reward or is simply a byproduct of training on diverse configurations.

The paper's design — multiple reward components combined into a single scalar via dot product with the preference vector — makes it impossible to attribute specific behavioral properties to specific reward components from the reported experiments. This is a standard challenge in multi-objective RL (the components are entangled in the scalar reward), but it means the paper demonstrates that the trained policy is efficient and preference-aware, not why or which reward components were necessary.

Claim 4: The orchestrator generalizes to unseen tools and pricing configurations.

What was tested: Evaluation with a completely unseen tool set (Table 2) and a held-out pricing scheme (Appendix H, Table 4).

Assessment: Supported. The generalization to unseen tools is particularly strong on FRAMES (73.8 with unseen tools vs. 76.3 in main evaluation — only a 2.5-point drop) and reasonably strong on HLE (22.0 vs. 37.1 — a 15.1-point drop, but still far above baselines). The drop on HLE is substantial and deserves discussion: it suggests that tool familiarity matters more for PhD-level reasoning (where nuanced capability assessment is critical) than for factuality tasks (where tool descriptions like "strong at biology" suffice for routing). The τ2-Bench generalization drop (80.2 → 48.8) is large, but this is expected given that τ2-Bench involves domain-specific function calling with tools that may have different interfaces than those seen in training. The paper does not analyze whether the τ2-Bench drop is due to interface unfamiliarity or capability mismatch, but the fact that Orchestrator-8B still outperforms GPT-5 (48.8 vs. 44.8) on the unseen-tool τ2-Bench is a meaningful robustness signal.

The pricing generalization is clean but limited in scope: only one held-out pricing scheme is tested. The paper demonstrates that Orchestrator-8B adapts to a new absolute price scale, but it does not test whether the model adapts appropriately when the relative costs of tools are reversed (e.g., if GPT-5 becomes cheaper than GPT-5-mini, does the orchestrator shift its delegation accordingly?). This would be a stronger test of cost-sensitivity.

Missing Experiments That Would Strengthen the Paper

1. Ablation of reward components. The paper would be significantly stronger with experiments training Orchestrator variants with (a) outcome reward only, (b) outcome + efficiency rewards (no preference), and (c) outcome + preference rewards (no efficiency). This would isolate which behaviors are caused by which reward signals and quantify the marginal benefit of each component. Without these ablations, the multi-objective reward design — while intuitively appealing — is not empirically validated as necessary for the results.

2. Training a larger orchestrator with the same method. The paper emphasizes that an 8B orchestrator suffices, but does not test whether a 32B or 70B orchestrator trained with ToolOrchestra would perform better. This matters for the paper's narrative about "small models are sufficient": if a 32B orchestrator achieves 42% on HLE, then the 8B result is impressive but suboptimal, and the "small model" framing is a specific engineering choice rather than a fundamental insight. The absence of this comparison makes it impossible to assess whether the orchestrator's performance is limited by its parameter count or by the RL training methodology.

3. Evaluation on non-text HLE questions. The paper restricts to HLE's text-only subset for fair comparison with baselines. But the full HLE includes image-based questions — evaluating Orchestrator with vision-capable tools (multimodal models as tools) would test whether the orchestration paradigm extends to multimodal reasoning. The paper's framing emphasizes generality ("solving deep and complex problems"), and the restriction to text-only is a limitation on that generality claim.

4. Comparison against an ensemble or mixture-of-agents baseline. The paper compares against monolithic models with tool access, but not against alternative compound architectures — e.g., a fixed mixture-of-agents where multiple models vote on answers, or a round-robin delegation scheme. This would distinguish the value of learned, adaptive orchestration from the value of simply having access to multiple models. The fact that GPT-5's performance drops with access to more models (Table 1) partially addresses this (naive multi-model access can hurt), but a structured non-learned baseline (e.g., "always ask GPT-5 first, then Qwen3-32B if uncertain") would sharpen the comparison.

5. Analysis of failure modes. The paper reports accuracy numbers but provides no qualitative analysis of when Orchestrator-8B fails. Does it fail because it delegates to the wrong tool? Because it cannot synthesize results from multiple tools? Because it hits the turn limit? Because no tool in its arsenal can solve the sub-problem? Understanding failure modes would clarify the limits of the orchestration paradigm and guide future improvements. The pilot study's bias analysis is a good diagnostic for baseline failures; a parallel diagnostic for Orchestrator-8B's failures is missing.

6. Statistical significance and variance. All reported numbers are point estimates from single evaluation runs (temperature 0, so deterministic). The paper does not report confidence intervals, standard deviations, or test-retest reliability. For HLE with 500 text-only questions, a 2-point difference (37.1 vs. 35.1) represents approximately 10 questions — small enough that variance in question difficulty or grading could affect the ordering. Without statistical tests, it is unclear whether Orchestrator-8B's margin over GPT-5 is reliably above noise. This is a common weakness in LLM benchmarking papers but is worth noting given the paper's strong claims of superiority.

6. Limitations and Trade-offs

The Difficulty Estimation and Orchestration Capability Bound: Hard Problems Remain Unsolved

The assumption or constraint. ToolOrchestra implicitly assumes that for any given sub-problem, at least one tool in the available arsenal can solve it — the orchestrator's role is to identify and route to that tool. When no tool possesses the required capability, orchestration cannot create it. The paper does not explicitly state this as an assumption, but it follows directly from the architecture: the orchestrator generates no novel knowledge or reasoning; it only coordinates existing resources. The paper acknowledges this boundary obliquely in Section 8: "Looking ahead, we envision more sophisticated recursive orchestrator systems to push the upper bound of intelligence."

The consequence. On tasks where all available tools — including GPT-5 — lack the necessary knowledge or reasoning capacity, the orchestrator will fail regardless of how expertly it delegates. The HLE result of 37.1% means that on approximately 63% of HLE questions, Orchestrator-8B does not produce a correct answer despite having access to frontier models. The paper provides no analysis of whether these failures occur because the orchestrator routed to the wrong tool, because even the best tool failed, or because the orchestrator's synthesis of multiple tool outputs was erroneous. This distinction matters enormously for deployment: if failures are primarily due to tool capability limits (GPT-5 itself cannot solve the problem), then improving the orchestrator will yield diminishing returns, and further progress requires better tools. If failures are due to routing errors, then improving the orchestration policy — through more training, larger orchestrator models, or better reward design — could close a substantial portion of the remaining gap. Without this diagnostic, a practitioner cannot determine whether investing in better orchestration or better tools will yield higher marginal returns.

What evidence exists in the paper. The paper provides no systematic failure-mode analysis for Orchestrator-8B. The pilot study (Appendix A, Figure 3) diagnoses baseline failure modes (delegation biases), but no equivalent analysis exists for the trained orchestrator. The tool-use breakdown (Figure 5, Table 15) shows aggregate calling patterns but not which calls succeeded or failed. The cost-performance curve (Figure 6) shows diminishing returns with more turns, suggesting that some problems are unsolvable regardless of budget — the curve plateaus rather than approaching 100% — but this is consistent with either tool-capability limits or orchestrator-coordination limits. The generalization experiment (Table 2) offers indirect evidence: the large drop from 37.1 to 22.0 on HLE when using unseen tools suggests that tool capability matters substantially, since the orchestrator's coordination policy transfers but the tools are different. However, this does not disentangle whether the drop is due to weaker unseen tools or the orchestrator's reduced ability to assess unfamiliar tool capabilities.

Mitigation status. The paper does not address this limitation directly. Section 8 gestures toward "more sophisticated recursive orchestrator systems" as future work, which could in principle decompose problems beyond the reach of any single tool by orchestrating multiple rounds of tool-assisted reasoning, but this is speculative. The authors do not propose methods for detecting when a problem is beyond available tool capabilities (which would enable graceful failure or escalation to human operators), nor do they analyze the relationship between tool-capability ceilings and orchestrator performance. A practitioner deploying Orchestrator-8B would need to independently characterize which problem classes are beyond the system's reach and implement fallback strategies.

The Cost of Difficulty Estimation and Initial Exploration Is Not Accounted For

The assumption or constraint. The Orchestrator requires no explicit difficulty estimation step — it learns to assess problem difficulty and appropriate tool routing implicitly through the RL training process. However, the training procedure itself requires generating 8 complete trajectories per task per GRPO update (rollout batch size of 8, Section 4.4), each of which may involve multiple calls to expensive frontier models. The paper states training used 16 NVIDIA H100 GPUs (Section 4.4) but does not report the total training cost in dollars, the number of training steps, the total tokens consumed, or the number of API calls made to proprietary models like GPT-5 during training.

The consequence. The headline efficiency numbers — Orchestrator-8B costs 9.2 cents per HLE task vs. GPT-5's 30.2 cents (Table 1) — measure inference-time cost only. They exclude the substantial upfront cost of RL training, which likely involved tens or hundreds of thousands of calls to GPT-5 and other expensive tools as the policy learned to route effectively. This is the standard "amortize training cost over inference queries" argument: if Orchestrator-8B serves millions of queries, the training cost per query becomes negligible. But the paper provides no data to determine the break-even point. How many inference queries are needed before the per-query savings (30.2 − 9.2 = 21.0 cents saved per HLE task) offset the training cost? If training cost $10,000 in API calls, break-even is ~47,600 queries. If it cost$ 100,000, break-even is ~476,000 queries. Without these numbers, a practitioner cannot assess whether deploying ToolOrchestra is economically rational for their query volume.

Furthermore, the training cost is not a one-time expense if the tool ecosystem changes. New models are released regularly; tool APIs change pricing and capabilities. The orchestration policy may need periodic retraining to adapt, incurring the training cost repeatedly. The generalization results (Table 2, Appendix H) suggest some robustness to tool and pricing changes, but it is unclear how far this extends — a significantly more capable new model with a novel capability profile might require retraining for the orchestrator to leverage it effectively, since the tool's description (generated by the process in Section 3.1) may not capture its new capabilities relative to training-time tools.

What evidence exists in the paper. The paper reports no training cost data whatsoever. The training configuration section (4.4) specifies hardware (16 NVIDIA H100 GPUs), learning rate, sequence lengths, batch sizes, and the GeneralThought-430K dataset, but omits: total training steps, total wall-clock training time, total number of tool calls made during training, or estimated API costs. The paper also does not report how many training trajectories were generated and filtered by the stabilization techniques (homogeneity filtering, format filtering, invalid output filtering), which would affect the effective sample efficiency of training. This is a significant transparency gap for a paper whose central claim is about cost-efficiency.

Mitigation status. Not addressed. The paper does not acknowledge the omission of training cost from the efficiency analysis. The related work section (7.2) discusses efficiency in tool-use agents but focuses exclusively on inference-time efficiency. The authors do not suggest training-cost reporting as a future work item or standard. For a paper arguing that orchestration is more cost-effective than monolithic models, the absence of training cost accounting weakens the economic argument — it demonstrates inference-time efficiency but not total-lifecycle efficiency.

Single Benchmark Domain and Single Model Family Limit Generality Claims

The assumption or constraint. All main experiments use three benchmarks — HLE, FRAMES, and τ2-Bench — which, while diverse in format (PhD questions, multi-hop factuality, conversational function calling), share a common structure: they involve tasks with well-defined correctness criteria that can be evaluated automatically (exact match, GPT-5-as-judge, or deterministic verification of function call sequences). The paper does not evaluate on open-ended generation tasks (creative writing, long-form reasoning without ground-truth answers), interactive tasks with human users in the loop, or tasks where success is multi-dimensional (e.g., helpfulness + harmlessness + accuracy). Additionally, the Orchestrator is built on a single base model family (Qwen3-8B); no experiments test ToolOrchestra with a different base architecture (e.g., Llama, Gemma, or a larger Qwen variant as the orchestrator backbone).

The consequence. The paper's claims about orchestration as a general paradigm — "composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods" (Abstract) — may not transfer to task types where the orchestrator's capabilities are tested differently. Specifically:

Open-ended tasks without verifiable outcomes. The entire RL training pipeline depends on a binary outcome reward r_outcome(τ) ∈ {0, 1} computed automatically via execution verification (for ToolScale synthetic data) or GPT-5-as-judge (for benchmark data). On tasks where correctness is subjective, multi-dimensional, or requires human evaluation (e.g., "write a compelling marketing email," "summarize this legal document accurately and concisely"), the binary outcome signal is unavailable or unreliable. ToolOrchestra cannot be directly applied to such tasks without a learned reward model, which introduces its own biases and training complexity.
Tasks requiring the orchestrator's own deep reasoning. The paradigm assumes the orchestrator's primary role is delegation and synthesis, not original reasoning. On HLE, the orchestrator can route math sub-problems to Qwen2.5-Math-72B and scientific questions to GPT-5. But on tasks where the reasoning itself must be done by the orchestrator — because no tool possesses the capability — the 8B parameter count becomes a hard bottleneck. The paper does not characterize what fraction of HLE success comes from the orchestrator's own reasoning vs. effective delegation, making it unclear whether the approach would work on tasks where delegation is less useful (e.g., generating novel scientific hypotheses that no existing model can produce).
Base model dependence. The choice of Qwen3-8B as the backbone may be consequential. Qwen3-8B might have particular strengths in instruction following or structured output generation that make it well-suited to tool orchestration; a different 8B model (e.g., Llama-3.1-8B) might perform substantially worse under the same training procedure. Without experiments varying the base model, the paper cannot claim that "small language models suffice for orchestration" as a general property — it can only claim that Qwen3-8B suffices.

What evidence exists in the paper. The paper evaluates on only three benchmarks, all with verifiable outcomes. It uses only Qwen3-8B as the backbone. The authors do not discuss these as limitations or scope their generality claims accordingly. The pilot study (Appendix A) tests GPT-5 and Qwen3-8B as prompted orchestrators, but does not test whether other base models exhibit different bias patterns when prompted or would respond differently to ToolOrchestra training. The tool descriptions used for evaluation are generated by the process in Section 3.1, which relies on GPT-5 to characterize tool capabilities — this process itself may have biases that affect orchestrator performance in ways that vary across base models.

Mitigation status. Not addressed. The paper does not acknowledge the restriction to verifiable-outcome benchmarks, the single-base-model limitation, or the potential non-transferability to open-ended tasks. Section 8 suggests future work on "more sophisticated recursive orchestrator systems" but does not mention testing on broader task types or with different base models. This is a significant scope limitation for a paper that frames its contribution in general terms ("paving the way for practical and scalable tool-augmented reasoning systems").

The 50-Turn Limit and Latency Implications Are Underexplored for Interactive Deployment

The assumption or constraint. The evaluation allows a maximum of 50 turns per task, and the Orchestrator's average latency per HLE/FRAMES task is 8.2 minutes (Table 1). Each turn involves at least one tool call — to GPT-5, a specialized model, or a search API — which incurs network and inference latency. The paper measures and reports latency as an efficiency metric, but does not analyze the distribution of latencies, the tail behavior, or the implications for interactive (sub-minute response time) applications.

The consequence. A system that takes 8.2 minutes on average to answer a question is unsuitable for interactive use cases — chat assistants, real-time coding help, or any deployment where users expect responses in seconds. The paper's latency numbers (8.2 minutes for Orchestrator-8B, 19.8 for GPT-5 with basic tools) position the system in the "batch processing" or "deep research" regime, where users submit queries and return later for results. This is a legitimate deployment model (cf. OpenAI Deep Research, Perplexity Deep Research, both cited in Section 7.1), but it is not the general-purpose agent deployment that the paper's framing sometimes implies.

More importantly, the sequential nature of the orchestration paradigm imposes a fundamental latency floor that parallel architectures could avoid. Each turn depends on the output of the previous tool call — the orchestrator cannot dispatch multiple tools in parallel and synthesize results, because the MDP formulation (Section 2.1) is strictly sequential: action a_k depends on history h_k, which includes the observation from action a_{k-1}. This means that even with perfect tool response times, the orchestrator's latency scales at least linearly with the number of turns. In contrast, a system that dispatches multiple independent queries to different tools in parallel and then aggregates could achieve lower latency at the same total FLOP cost, though it would require a different architecture than the sequential MDP used here.

The paper's cost-performance tradeoff (Figure 6) varies the maximum turn limit, showing that accuracy improves with more turns but plateaus — suggesting that the 50-turn limit is generous. However, the paper does not report the actual distribution of turns used by successful vs. unsuccessful trajectories. If most successes require 15–20 turns while most failures hit the 50-turn limit, that reveals different bottlenecks (orchestration efficiency vs. task unsolvability) than if successes and failures have similar turn-count distributions.

What evidence exists in the paper. Table 1 reports average latency per task for each system. The cost-performance curves (Figure 6) implicitly show how accuracy varies with latency (since cost and latency are correlated), but do not display latency on the x-axis. The paper does not report latency distributions, percentiles, or best-case/worst-case latency. The MDP formulation (Section 2.1) explicitly includes latency l_i per action and aggregate latency ∑ l_i in the optimization objective, and the preference reward (Equation 2) can weight latency, but the analysis of how the policy trades off latency against accuracy is limited to the aggregate preference-aware score (Table 3).

Mitigation status. Partially addressed through the preference vector mechanism. If a user values low latency, they can set p_latency high in the preference vector, which during training would penalize long-latency trajectories and presumably produce a policy that uses fewer, cheaper, faster tool calls. However, the paper does not evaluate whether this actually produces low-latency policies (e.g., sub-30-second response times) or merely slightly reduces latency. The minimum possible latency for Orchestrator-8B is not characterized — it depends on how many turns are strictly necessary to solve tasks and how fast the tools respond, neither of which is analyzed. The paper does not propose architectural changes to enable parallel tool dispatch, which would be the most direct way to reduce latency.

The Orchestrator's Adversarial Robustness and Safety Under Tool Misdescription Are Untested

The assumption or constraint. The Orchestrator relies on JSON tool descriptions (name, description, parameter schema) to assess tool capabilities and decide when to invoke each tool. These descriptions are generated by GPT-5 (Section 3.1) and assumed to be accurate representations of tool strengths and weaknesses. The paper tests generalization to unseen tools (Section 6.3) but does not test generalization to inaccurate tool descriptions — scenarios where a tool's description overstates its capabilities, omits critical failure modes, or is adversarially manipulated.

The consequence. In real deployments, tool descriptions may be written by third parties with incentives to exaggerate capabilities, may become stale as models are updated, or may fail to capture nuanced failure modes (e.g., "strong at biology except for taxonomy questions involving recently reclassified species"). The paper's example description for Qwen3-32B (Appendix C) is notably granular — it specifies that the model has "notable weaknesses in exact nomenclature and InChI outputs" in chemistry. If tool descriptions omit such failure-mode details, the orchestrator may route inappropriate sub-problems to tools, degrading accuracy.

More critically, an adversarial tool provider could craft a description that makes their tool appear universally capable, causing the orchestrator to over-delegate to it. Since the orchestrator's training reward depends on outcome correctness (Equation 1), the policy would learn to avoid tools that consistently fail — but only after incurring failed trajectories during training and potentially during early deployment. The paper provides no analysis of how many failed interactions are needed to "unlearn" trust in a misleadingly described tool, or whether the orchestrator can detect description-accuracy mismatches from tool outputs.

This limitation connects to a broader trustworthiness concern: the orchestration paradigm multiplies the surface area for failures. A monolithic model can fail only if its own reasoning is flawed; an orchestrated system can fail if any tool description is inaccurate, if any tool returns misleading output, if the orchestrator misjudges tool capabilities, or if the synthesis of multiple tool outputs introduces errors. The paper's evaluation does not characterize the relative frequency of these failure modes, making it difficult for practitioners to assess the system's reliability in safety-critical deployments.

What evidence exists in the paper. The generalization experiment (Table 2) tests unseen tools whose descriptions were generated by the same process as training tools (GPT-5 writing capability descriptions based on 10 sample tasks). This tests whether the orchestrator can understand new descriptions, but all descriptions are assumed accurate. There is no experiment where tool descriptions are systematically perturbed (e.g., removing failure-mode details, inflating capability claims) to test robustness to description quality. The ToolScale data synthesis pipeline (Section 3.3) verifies that generated tasks are solvable with the provided tools, but does not verify that tool descriptions are complete or free of bias.

Mitigation status. Not addressed. The paper does not discuss description accuracy, adversarial robustness, or safe deployment of orchestrated systems. The configurational diversity during training (varying tool subsets and pricing per instance, Section 3.3) might incidentally train some robustness — the orchestrator sees tools with varying capability profiles and learns to assess tool outputs rather than blindly trusting descriptions. But this is speculative; the paper provides no analysis of whether the orchestrator's decisions are driven by tool descriptions, tool outputs, or both. The preference vector mechanism could in principle encode "trust" levels for different tools, but this is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around tool-augmented LLMs from "what tools can a model use?" to "how should a model decide which tools to use, and when?" The distinction matters because it reframes tool use as a meta-reasoning and resource-allocation problem rather than a capabilities problem. Prior work implicitly assumed that giving a capable model access to more tools would monotonically improve performance — the challenge was integrating tools into the reasoning loop, not deciding among them. ToolOrchestra demonstrates that this assumption is false: GPT-5's HLE accuracy drops from 35.1% to 21.2% when given access to an expanded toolset (Table 1), because its prompting-based delegation policy routes tasks to suboptimal tools. The bottleneck is not tool quality or model capability — it is allocation strategy, and allocation strategy can be learned through RL with appropriately designed rewards.

This is a reframing, not a paradigm shift. The paper does not introduce fundamentally new algorithms (GRPO is from DeepSeek-R1; tool-use MDPs are from AgentGym and Archer; synthetic data generation is a known technique). Rather, it demonstrates that applying these known methods to the specific problem of training a small model to coordinate heterogeneous tools under multi-objective rewards produces a system that outperforms monolithic frontier models at lower cost. The conceptual contribution is the recognition that orchestration is a distinct capability from reasoning, and that it can be trained into a small model without requiring that model to itself possess the capabilities it coordinates. The 8B orchestrator scores 3.2% on HLE with no tools but 37.1% with tools (Table 1) — its value lies entirely in its learned delegation policy, not in its own knowledge.

The paper resolves a contradiction that has been brewing in the tool-use literature. On one side, work like Toolformer, ToolLLM, and Search-R1 showed that tool integration improves LLM performance; on the other side, practitioners observed that strong models sometimes perform worse when given access to weaker tools, or that models exhibit stubborn preferences for particular tools regardless of task suitability. The pilot study (Appendix A, Figure 3) provides a clean diagnostic for why: off-the-shelf models exhibit systematic delegation biases — self-enhancement (GPT-5 defaults to GPT-5-mini) and other-enhancement (Qwen3-8B defaults to GPT-5) — that override explicit instructions and degrade performance. This reconciles the contradictory findings: tool use helps when the allocation policy is effective and hurts when delegation biases route tasks to inappropriate tools. The implication is that tool-use research must evaluate not just whether models can use tools, but whether they use the right tools in the right order — a shift from capability evaluation to strategy evaluation.

The paper also redirects research investment in compound AI systems. Before this work, a natural approach to building better tool-using agents was to start with the strongest available model and add tools. The finding that a small RL-trained orchestrator outperforms GPT-5 with the same tools (37.1 vs. 21.2 on HLE, Table 1) suggests that training a dedicated orchestration policy may yield higher returns than scaling the base model. For organizations building agentic systems, this implies that investment should shift from "use a larger model as the agent" toward "train a small orchestrator and give it access to the largest models as tools." The paper does not prove that a larger orchestrator wouldn't be even better (this ablation is missing), but it establishes that the orchestrator's size is not the binding constraint — the policy's quality is, and policy quality can be improved through RL with well-designed rewards.

Conversely, this work makes pure monolithic scaling less attractive as a research direction for hard reasoning benchmarks. If an 8B orchestrator + tools can match or exceed GPT-5 at 3× lower cost, then the marginal return on scaling the core reasoning model — at least for tasks within the capability range of available tools — is diminished. The caveat is that tool capability ceilings still bind: on problems no available tool can solve, orchestration offers no path forward, and larger pretrained models may be necessary. The paper's results neither prove nor disprove that frontier models are needed as tools in the orchestrated system; GPT-5 remains in the tool set and is called 1.6 times per task on average (Table 15). The landscape change is not that large models become obsolete, but that their role shifts from "monolithic reasoner" to "specialized resource in a composite system."

Follow-Up Research This Work Enables

Training orchestrators with on-policy tool feedback to detect description-accuracy mismatches. The paper shows that the Orchestrator generalizes to unseen tools using GPT-5-generated capability descriptions (Section 6.3, Table 2), but it never tests whether the orchestrator can detect when a tool's description overstates or misrepresents its capabilities. A strong follow-up would train the orchestrator to maintain a learned "trust score" per tool, updated based on whether the tool's outputs are internally consistent, align with other tools' outputs on the same sub-problem, or lead to correct final answers. Concretely: add a reward bonus for correctly identifying (via a special "tool unreliable" action) when a tool has been adversarially misdescribed in 20% of training instances, and evaluate whether the learned distrust transfers to held-out misdescribed tools. The tool-call logs from the paper's existing training runs provide a starting dataset — trajectories where the orchestrator called a tool, received output, and later either succeeded or failed at the task contain implicit signals about tool reliability that could be extracted through hindsight relabeling.

Parallel tool dispatch to break the sequential latency bottleneck. The MDP formulation (Section 2.1) constrains the orchestrator to strictly sequential tool calls: each action depends on the full history including the previous tool's output. This means latency scales at least linearly with the number of tool calls, making the system unsuitable for interactive applications (8.2 minutes average latency, Table 1). A natural extension is to augment the action space with a "parallel dispatch" primitive that allows the orchestrator to specify multiple independent tool calls in a single turn, then receive all outputs before the next reasoning step. This would require modifying the rollout procedure (Section 2.2) to support batched tool execution and the GRPO training to handle variable-length parallel action sequences. The key empirical question is whether the orchestrator can learn when parallel calls are safe (sub-problems are independent) vs. when sequential calls are necessary (later sub-problems depend on earlier results). The paper's preference vector could encode a latency weight that incentivizes the policy to discover parallelizable sub-problems. A strong evaluation would measure the speedup on FRAMES (where multi-hop questions often require sequential retrieval) vs. HLE (where sub-problems in different domains like math and biology might be parallelizable), and compare against the current sequential-only latency numbers in Table 1.

Difficulty-conditioned budget allocation: learning when to escalate to stronger tools. The Orchestrator's tool-use pattern (Figure 5, Table 15) shows it uses GPT-5 sparingly (1.6 calls per task on average) and relies on cheaper tools for most sub-problems. But the paper provides no analysis of whether this allocation is optimal — whether some tasks that failed with cheap tools would have succeeded with earlier escalation to GPT-5, or conversely whether some tasks that succeeded with cheap tools wasted budget on unnecessary GPT-5 calls. A follow-up study could instrument the training process to log, for each trajectory, which tool calls were made and whether the task was solved, then train a budget-conditional policy that receives an explicit compute budget as part of the state and learns to allocate it optimally. Concretely: vary the preference vector's p_compute component across a wide range during training (not just binary high/low as in the paper's preference pairs), and evaluate whether the resulting policy can smoothly interpolate to achieve the accuracy-cost frontier in Figure 6 without the need to vary the turn limit (which is a crude budget-control mechanism). This would also address the missing "when to give up" behavior: the current policy has no mechanism to recognize that a task is beyond all available tools and terminate early, potentially wasting up to 50 turns on unsolvable problems.

Stress-testing orchestrator robustness to tool ecosystem drift. The paper demonstrates generalization to a single held-out tool set (Table 2) and a single held-out pricing scheme (Appendix H, Table 4), but real tool ecosystems change continuously — new models are released, APIs are deprecated, pricing changes, and capability profiles shift. A systematic stress-test would evaluate the orchestrator under progressive tool-set mutation: start with the training tool set, then incrementally replace tools with alternatives of varying capability similarity (e.g., replace GPT-5 with a hypothetical "GPT-5.5" that is 10% more capable, then with one that is 10% less capable; replace a specialized math model with a generalist that happens to be strong at math). Measure the performance degradation curve as a function of mutation distance. The key insight this would reveal is whether the orchestrator's generalization (which the paper attributes to capability-based tool selection learned from descriptions) is robust to incremental drift or only to complete substitution with a new description. If performance degrades smoothly with capability mismatch, the system can be deployed with periodic retraining intervals; if it degrades catastrophically at some threshold, continuous monitoring and rapid retraining become necessary.

Ablating the minimal orchestrator size needed for effective coordination. The paper trains only an 8B orchestrator (Qwen3-8B), but does not test whether even smaller models (1B, 3B) could learn effective orchestration, or whether larger orchestrators (32B, 70B) would perform substantially better. This is a practically urgent question because inference cost for the orchestrator itself is part of the total system cost, and an even smaller orchestrator would further improve the cost-efficiency argument. A scaling study should train Orchestrator variants at 0.5B, 1.5B, 3B, 8B, and 32B parameters (using Qwen3 or Llama-3.2 families for consistent architecture) with identical ToolOrchestra training, then plot HLE/FRAMES/τ2-Bench accuracy against orchestrator parameter count. The paper's hypothesis — that orchestration is a "small model" capability — predicts a plateau: beyond some minimal size needed to understand tool descriptions and plan multi-step delegations, additional parameters should yield diminishing returns because the orchestrator's own knowledge is not the bottleneck. If instead accuracy continues to improve with orchestrator size, the paper's central narrative ("small language models suffice") requires qualification — the 8B result would be an existence proof that orchestration can work at small scale, not evidence that small scale is optimal.

ToolOrchestra for open-ended tasks with learned outcome rewards. The paper's RL training depends on a binary outcome reward r_outcome(τ) ∈ {0, 1} computed via deterministic verification (execution correctness, process fidelity, operation completeness for ToolScale data) or GPT-5-as-judge (for benchmark data). This restricts applicability to tasks with well-defined correctness criteria, excluding open-ended generation, creative writing, summarization, and dialogue. A critical extension would replace the binary outcome reward with a learned reward model trained on human preference judgments of trajectory quality, then apply the same GRPO training with multi-objective rewards (outcome quality from the reward model + efficiency + user preferences). The key empirical question is whether the orchestrator's delegation strategy — which currently optimizes for a clear correctness signal — can be learned from a noisier, more subjective reward model, or whether the binary outcome signal is necessary for the policy to discover effective tool-use patterns. The evaluation would compare orchestrators trained with binary vs. learned rewards on summarization tasks (where tool calls might retrieve source documents, fact-check claims, or refine drafts) and measure both automatic metrics (ROUGE, factuality) and human preference win rates. If learned rewards suffice, ToolOrchestra becomes applicable to the vast majority of real-world LLM use cases that lack ground-truth correctness signals.

Practical Applications and Downstream Use Cases

Cost-efficient batch evaluation and data generation pipelines. The most immediate application is in settings where organizations run large-scale inference over benchmarks or generate training data. On HLE, Orchestrator-8B achieves 37.1% accuracy at 9.2 cents per task vs. GPT-5's 35.1% at 30.2 cents (Table 1). For a pipeline processing 100,000 tasks, this saves approximately $20,900 while producing higher accuracy. The benefit compounds when the orchestrator's output is used as training data for downstream models (distillation, rejection sampling fine-tuning), since higher accuracy at lower cost directly improves data quality per dollar. The preference vector enables further cost control: deploying with p_compute = 1 might sacrifice 1–2 accuracy points for additional cost savings, while p_outcome = 1 would maximize accuracy regardless of cost. The paper does not provide the full accuracy-cost tradeoff curve parameterized by preference vectors (only by turn limits in Figure 6), but the preference mechanism in principle enables fine-grained budget control without changing system architecture.

Privacy-sensitive on-premise deployments with selective cloud escalation. The preference vector's ability to encode tool-specific preferences (Appendix F: a user can specify p_local_search = 1, p_web_search = 0, p_GPT5 = 0 to prefer local models and avoid external API calls) makes Orchestrator-8B directly applicable to enterprise settings where data cannot leave on-premise infrastructure. A deployment could run the 8B orchestrator and open-source tool models (Qwen3-32B, Qwen2.5-Coder-32B, Llama-3.3-70B) on local GPUs, with the preference vector configured to avoid external API calls entirely. When the orchestrator determines that a sub-problem genuinely requires frontier-model capabilities (detected implicitly through repeated failures of local tools), it could escalate to a human operator rather than silently calling a cloud API. The generalization results (Table 2) suggest this works with arbitrary open-source tool sets — the orchestrator adapts to whichever models are locally available based on their capability descriptions. The FRAMES result of 73.8% with a held-out tool set (including models like Gemma-3-27b and DeepSeek-Math-7b that can run on commodity hardware) suggests that a fully local deployment with open-weight models could achieve substantial accuracy on factuality and reasoning tasks without any data leaving the premises.

Self-improving agentic systems through iterative training-data generation. The paper's vision of "iterative self-improvement loops" (Section 8) becomes concretely actionable with ToolOrchestra. A system can use Orchestrator-8B to generate high-quality trajectories on a training corpus (solving tasks by delegating to available tools), filter trajectories by outcome reward, and fine-tune a new orchestrator on the successful trajectories — then repeat. The preference vector provides a knob for controlling the quality-cost tradeoff of generated trajectories: early iterations might use p_outcome = 1, p_compute = 0 to maximize the quantity of correct solutions (accepting high cost), while later iterations shift toward p_compute = 1 to distill efficient strategies. The ToolScale pipeline (Section 3.3) provides a template for generating verifiable training data in new domains, and the generalization results (Section 6.3) suggest that an orchestrator trained on ToolScale domains can be fine-tuned on domain-specific trajectories without catastrophic forgetting. The key resource question the paper does not answer — how many training trajectories are needed for convergence — determines the economic viability of this approach, but the reported training configuration (16 H100 GPUs, Section 4.4) is within reach of many research labs and all industrial teams.

When to Prefer This Method

The paper positions ToolOrchestra against monolithic model scaling (GPT-5, Claude Opus 4.1) and prompted tool-use baselines. The decision conditions emerge from the experimental results rather than being stated as an explicit tradeoff framework, but they are clear enough to operationalize:

Prefer ToolOrchestra-trained orchestrator when:

The task distribution includes problems that are partially within the capability range of available tools — some sub-problems can be solved by specialized or frontier models, even if no single model solves the entire task (the HLE/FRAMES/τ2-Bench regime).
Inference cost is a binding constraint and the query volume is high enough to amortize RL training cost (the 3.3× cost reduction vs. GPT-5 on HLE — 9.2 vs. 30.2 cents — provides a clear economic incentive above some break-even query volume).
Multiple heterogeneous tools are available with varying cost-capability profiles, and a human cannot manually specify routing rules (the Orchestrator's balanced tool-use pattern in Figure 5 emerges from RL, not from hand-crafted heuristics).
User preferences over tools, cost, latency, or privacy vary across deployments or over time, and retraining per deployment is infeasible (the preference vector enables zero-shot steering without model updates).
The base orchestrator model is small enough to run cheaply and the tool set includes models substantially more capable than the orchestrator (the 8B orchestrator outperforms the 235B Qwen3-235B-A22B, validating the "small coordinator + strong tools" architecture).

Prefer monolithic frontier model (GPT-5, Claude) with basic tools when:

The task distribution is narrow and well-matched to the frontier model's specific strengths, and delegation offers no capability advantage (the paper does not test this regime, but it follows from the finding that tool access only helps when tools possess capabilities the orchestrator lacks).
Latency is the absolute priority and the sequential turn structure of orchestration (average 8.2 minutes, Table 1) is unacceptable — a monolithic model can answer in a single inference pass.
The tool ecosystem is unstable, and frequent retraining of the orchestration policy is impractical — prompted baselines degrade under tool-set changes (GPT-5 drops ~40% on HLE when given expanded tools) but require no retraining.

Prefer prompted orchestrator (off-the-shelf LLM with tool access) when:

The available tool set is small and uniform enough that delegation biases (Figure 3) do not cause catastrophic misrouting — e.g., a single code execution tool and a single search tool, where "default to web search" is often correct.
Rapid prototyping is more important than optimal cost-efficiency, and the RL training pipeline (ToolScale data synthesis, GRPO training, multi-objective reward engineering) represents unacceptable upfront investment.

Open question requiring practitioner judgment: The paper does not report training cost, so the break-even query volume for preferring ToolOrchestra over prompted GPT-5 is unknown. A practitioner with low query volume (< 10,000 queries) cannot determine from the paper's data whether the inference-time savings justify the training investment. The onus is on the practitioner to estimate training cost from the reported configuration (16 H100 GPUs, Qwen3-8B backbone, GeneralThought-430K + ToolScale data, rollout batch size 8) and compare against their projected inference volume.