OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents

ArXiv: 2510.24563

🎯 Pitch

State-of-the-art computer-use agents can gain dramatic accuracy boosts from MCP tools (e.g., OpenAI o3 jumps from 8.3% to 20.4%), yet even the strongest model invokes tools only 36.3% of the time. This first-of-its-kind benchmark exposes a critical decision-making gap: models know tools help, but they consistently fail to use them when it matters most.

---

1. Executive Summary

This paper introduces OSWorld-MCP, the first comprehensive benchmark that fairly evaluates computer-use agents by jointly assessing GUI operation, MCP tool invocation (e.g., installing a VS Code extension via a single install_extension call versus four GUI clicks), and the decision-making ability to choose between them. Built on OSWorld with 158 manually validated tools spanning 7 applications, experiments with state-of-the-art multimodal agents including Claude 4 Sonnet and OpenAI o3 demonstrate that MCP tools substantially improve accuracy—boosting o3 from 8.3% to 20.4% at 15 steps—while also revealing that even the strongest model achieves only a 36.3% Tool Invocation Rate (TIR), establishing that current LMMs can benefit from tools yet struggle to invoke them reliably, particularly when tasks require composing multiple tool calls.

2. Context and Motivation

The Core Problem: We Evaluate Computer-Use Agents Incompletely and Unfairly

The fundamental problem this paper tackles is that existing benchmarks for multimodal computer-use agents only measure GUI interaction skills — clicking, typing, scrolling, dragging — while ignoring a capability that has rapidly become central to real-world agent deployments: the ability to invoke external tools such as those exposed through the Model Context Protocol (MCP). MCP, introduced by Anthropic in November 2024, is an open standard based on JSON-RPC that defines how AI applications connect to external systems — file systems, databases, search engines, calculators, application-specific APIs — through a uniform client-server interface. In practice, this means an agent can install a VS Code extension with a single install_extension call rather than navigating menus, searching the marketplace, clicking install, and verifying success through a sequence of GUI operations. The efficiency gap is not marginal: the paper's motivating example in Figure 1 shows a task that takes at least four GUI steps reduced to one MCP tool call.

This gap in evaluation matters acutely because several state-of-the-art agents — including CoAct (Song et al., 2025) and ComputerRL (Lai et al., 2025) — have already integrated autonomous tool invocation into their architectures and have reported performance gains from doing so. When these tool-augmented agents are compared against agents that only use GUI operations on existing benchmarks, the comparison is inherently unfair: the tool-augmented agent has access to a strictly larger action space. The paper frames this explicitly as an equity problem in evaluation:

"it is inherently inequitable to compare such agents with others that assess only GUI interaction"

The core gap, then, is the absence of a benchmark that jointly measures GUI operation skills, tool invocation capabilities, and the decision-making competence to choose between them. No existing benchmark provides a level playing field where agents can be compared on their ability to decide when to use a tool versus when to click, and then evaluated on how effectively they execute whichever path they choose.

Why This Problem Is Important

Real-world deployment fidelity. Computer-use agents are not research curiosities — they are being deployed in production environments. Claude's computer use feature, OpenAI's Operator, and various open-source frameworks all aim to automate real desktop tasks (spreadsheet manipulation, document editing, system configuration, media processing). In such settings, the natural action space includes both GUI operations and programmatic tool calls. A benchmark that excludes tools measures performance in an artificially constrained environment that does not reflect how agents actually operate in production. Worse, it may incentivize developers to optimize for GUI-only task completion when a hybrid GUI-tool approach would be more robust and efficient.

The efficiency-robustness tradeoff. Tool invocation is not merely a convenience — it can fundamentally change whether a task succeeds. GUI operations are brittle: they depend on the exact screen layout, the agent's grounding accuracy (whether it clicks the right pixel), and the stability of the interface across versions. Tool calls, when correctly specified, are deterministic and layout-invariant. The paper's Figure 1 illustrates this: the GUI path for installing a VS Code extension requires recognizing the extension marketplace icon, typing the extension name, clicking install, and verifying — each step carrying a risk of grounding failure. The MCP path is a single, parameterized function call. Understanding when and how agents can leverage this robustness advantage is critical for building reliable computer-use systems, and evaluating this capability requires a benchmark designed explicitly for it.

Decision-making as a first-class capability. The paper correctly identifies that tool invocation is not just about having tools available — it is about making the right choice at each step. Should the agent invoke osworld_mcp_libreoffice_calc.copy_cells_between_sheets or should it select the column manually and use copy-paste GUI operations? Should it estimate task difficulty and choose the tool path for reliability, or the GUI path because the tool's parameters are complex to specify correctly from visual context? This decision-making skill — selecting between GUI and MCP pathways, choosing the most appropriate tool from a candidate set, and composing multiple tools when no single tool suffices — is what the paper aims to measure. Existing benchmarks do not surface this capability because they do not offer a choice.

Standardization and fair comparison. The MCP ecosystem is growing rapidly, but with fragmentation: different agent systems integrate different tool sets, different tool description formats, and different invocation protocols. Without a standardized benchmark, reported tool-use improvements are non-comparable across systems — a 5% accuracy gain on one agent's custom tool set tells us nothing about relative capability. OSWorld-MCP provides a fixed, curated tool set (158 tools) against which all agents are evaluated under identical conditions, enabling apples-to-apples comparison of tool invocation and hybrid decision-making skills for the first time.

Where Prior Approaches Fall Short

The paper identifies shortcomings across two categories of prior work: GUI-only interactive benchmarks and text-based tool-use benchmarks.

GUI-Only Benchmarks: Complete but Action-Space-Constrained

The most directly relevant prior work is OSWorld (Xie et al., 2024), the benchmark that OSWorld-MCP extends. OSWorld is a dynamic, interactive evaluation framework that assesses multimodal agents on 369 real-world tasks across 9 applications (LibreOffice suite, VS Code, GIMP, Thunderbird, VLC, Chrome, and OS-level operations) in real Ubuntu, Windows, and macOS virtual machines. Tasks range from "copy the Revenue column to a new sheet" in LibreOffice Calc to "help me install the autoDocstring extension in VS Code." Agents perceive the screen visually, reason about the interface, and execute a predefined set of 11 GUI actions: key, type, mouse move, click, drag, right click, middle click, double click, scroll, wait, and terminate. Success is determined by an evaluator script that checks the final state of the environment (e.g., whether the correct file was saved with the correct content).

OSWorld is a significant advance over static benchmarks like Mind2Web (Deng et al., 2023), WebLinx (Lù et al., 2024), and OmniAct (Kapoor et al., 2024), which rely on pre-collected human trajectories and cannot evaluate alternative solution paths — if a tool invocation accomplishes the task in a different sequence than the human demonstrator, static benchmarks cannot assess correctness. OSWorld's dynamic, state-based evaluation solves this: it only checks whether the final environment state matches the target, regardless of how the agent got there.

However, OSWorld (and similar dynamic benchmarks — WindowsAgentArena (Bonatti et al., 2024), WorkArena (Drouin et al., 2024), MMInA (Tian et al., 2024), AndroidWorld (Rawles et al., 2024)) share a critical limitation:

"existing dynamic interactive benchmarks typically predefine only GUI actions for the agent to use, and therefore lack a comprehensive and fair evaluation framework that jointly measures multimodal agents' tool invocation, GUI interaction, and decision-making capabilities"

The action space is artificially restricted to GUI operations. An agent that could solve a task efficiently through a tool call is forced to use clicking and typing — potentially failing due to grounding errors that a tool call would have avoided, or succeeding but taking many more steps than necessary. This restriction means that (a) agents with strong tool-invocation capabilities cannot demonstrate them, (b) agents without such capabilities are not penalized for lacking them, and (c) the benchmark provides no signal about an agent's ability to choose between modalities.

Text-Based Tool-Use Benchmarks: Tool-Rich but Visually Impoverished

A parallel line of work has emerged to evaluate tool-use capabilities of LLMs, but these benchmarks operate in a text-only paradigm that elides the visual perception and GUI grounding challenges central to computer-use tasks. The paper cites several recent MCP-related benchmarks:

MCPEval (Liu et al., 2025) and MCP-Radar (Gao et al., 2025) cover limited sets of MCP servers — "typically no more than a few dozen tools, which restricts task diversity." The small tool set limits the range of task complexity and makes it easier for models to identify the correct tool by elimination.

LiveMCPBench (Mo et al., 2025) uses LLM-based evaluation (an LLM judges whether the agent's outputs are correct) rather than executable, ground-truth verification. The paper argues this is "not well suited to tasks requiring real-time knowledge" — and, more fundamentally, LLM-as-judge evaluation introduces its own biases and reliability issues that make it unsuitable as a primary metric for benchmarking.

MCP-Bench (Wang et al., 2025b) defines tasks based on available tools, meaning that the benchmark's task design is constrained by what tools exist rather than what real users actually need to accomplish. As the paper notes, this creates "a gap between benchmark tasks and truly open-ended real-world problems." In a real deployment, tasks arise from user needs, not from available tool capabilities — a benchmark should reflect this direction of causality.

The deeper limitation shared by all these text-based benchmarks is that they omit the visual dimension. Real computer-use tasks require agents to look at a screen, understand the current state of the interface (What application is open? What file is active? What menu is visible?), and make decisions based on visual information. A benchmark that provides task descriptions in text and expects tool calls in response cannot evaluate whether an agent knows when a GUI action is necessary because the tool's preconditions aren't met, or whether the agent can extract parameter values (e.g., a filename, a cell range, a target sheet name) from the current screen state. These are exactly the hybrid decision-making skills that distinguish capable computer-use agents from simple tool-invocation scripts.

The Fundamental Gap: No Integrated, Fair Evaluation

Taken together, the landscape prior to OSWorld-MCP looks like this:

No prior benchmark simultaneously provides (1) visual GUI context that grounds tasks in real interface states, (2) a rich, curated set of MCP tools that agents can choose to invoke or ignore, and (3) real-world tasks derived from user needs rather than tool capabilities. This is the gap OSWorld-MCP is designed to fill.

How OSWorld-MCP Positions Itself

The paper positions OSWorld-MCP not as a replacement for OSWorld but as a significant extension that adds a new evaluation dimension while preserving the existing one. The key design decisions that define this positioning are:

Built on OSWorld's proven infrastructure. OSWorld-MCP inherits OSWorld's virtual machine-based execution environment, its 369-task benchmark, its state-based evaluation scripts, and its nine-application coverage. This is not a new benchmark from scratch — it is a carefully scoped addition of 158 MCP tools that can be used to solve 250 of the 369 tasks (69% coverage), with the remaining 111 tasks serving as a baseline where tools are not beneficial (what the paper calls Non-Tool-Beneficial Tasks). This design is deliberate: it allows direct comparison of the same agents on the same tasks with and without tool access, isolating the effect of tool invocation on performance.

Autonomous modality choice at every step. A defining architectural choice: at each step of task execution, the agent autonomously decides whether to invoke an MCP tool or perform a GUI operation. This is not a pipeline where tools are used for one class of tasks and GUI for another — it is a step-level decision embedded in the agent's action space. The paper refers to this as "dynamic interaction between GUI operations and tool usage," and it is what makes the benchmark a test of decision-making competence, not just tool invocation skill. An agent that invokes tools indiscriminately (even when a GUI operation would be faster or more reliable given the current screen state) will be penalized by the Tool Invocation Rate metric. An agent that never invokes tools will fail tasks that are trivially solvable with them. Optimal performance requires the agent to reason about which path is best right now.

Two new metrics that capture the decision-making dimension. The paper introduces Tool Invocation Rate (TIR) and Average Completion Steps (ACS) alongside standard task accuracy. TIR is carefully designed to measure appropriate tool use, not just any tool use: it rewards tool invocation on Tool-Beneficial Tasks and penalizes it on Non-Tool-Beneficial Tasks. ACS captures efficiency — fewer steps means better decisions about which path to take. Together, these metrics surface patterns that accuracy alone would obscure: an agent might have high accuracy but low TIR (succeeding through laborious GUI paths when tools would have been faster), or high TIR but low accuracy (invoking tools eagerly but incorrectly). These patterns directly inform what an agent's weaknesses are — grounding, tool selection, parameter specification, or decision-making — and are visible only through the multi-metric design.

A tool generation pipeline that ensures quality and realism, not benchmark-specific tailoring. The paper explicitly addresses a subtle but important validity concern: if tools were hand-crafted to perfectly match benchmark tasks, they would not represent the kind of general-purpose tools available in real MCP servers, and performance on them would overstate real-world capability. The authors use an automated code generation pipeline (OpenAI o3 generates code for each OSWorld task, followed by filtering for correctness, then wrapping into MCP tools) combined with curation from existing MCP servers, followed by rigorous manual review to remove tools that are "highly task-specific" or "functionally redundant." The result is a set of tools that are "genuinely relevant to real-world needs and are not artificially tailored for specific benchmark tasks." Evidence for this claim: of the 158 tools, 131 were invoked at least once during evaluation by the five tested models, suggesting that models find them useful in practice, not just in theory. The remaining two tools that were never invoked are hypothesized to correspond to tasks too complex for any model to attempt — a sign that task difficulty, not tool irrelevance, explains non-invocation.

A unified standard for the emerging tool-augmented agent ecosystem. The paper's positioning as "the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities in a real-world environment" is not just rhetorical. It reflects a specific, concrete contribution: a fixed evaluation protocol (the same 158 tools, the same RAG-based tool filtering, the same task set, the same virtual machine images, the same evaluation scripts) that any agent can be plugged into. Before OSWorld-MCP, an agent developer who wanted to benchmark tool invocation had to either (a) build their own tool set and evaluate in isolation, producing non-comparable results, or (b) use a text-only tool benchmark that ignores the visual grounding challenges of real computer use. OSWorld-MCP eliminates both compromises by providing a standardized, multimodal, dynamic evaluation environment. The paper's public release of code, environment, and data at a dedicated website reinforces this positioning as community infrastructure, not just a one-off research artifact.

Explicit connection to the over-arching narrative of agent capability assessment. The paper situates itself within the broader trajectory of LLM/LMM evaluation: as models advance from text generation to reasoning (GPT-5, DeepSeek-R1, Qwen3) to multimodal understanding to autonomous computer use, evaluation benchmarks must advance correspondingly. GUI-only benchmarks sufficed when models were just learning to see and click. Now that models can reason about tool APIs and invoke them programmatically, the evaluation framework must expand to assess this new capability — and to assess the meta-capability of choosing between modalities. OSWorld-MCP is presented as exactly this next step in benchmark evolution.

3. Technical Approach

3.1 Reader Orientation

OSWorld-MCP is not a single model or algorithm but rather an evaluation benchmark — a standardized testing environment that lets researchers plug in their multimodal computer-use agents and measure how well those agents perform real desktop tasks when given access to both GUI operations (clicking, typing) and programmatic tool calls (MCP functions). The problem it solves is that existing benchmarks only test GUI skills or only test tool skills in text-only settings, never both together in a realistic visual environment, which means we cannot fairly compare agents that have tool-invocation capabilities against those that don't, nor can we measure an agent's ability to decide when to click versus when to call a function. The shape of the solution is: take an existing, well-validated GUI benchmark (OSWorld, with 369 real desktop tasks and state-based evaluation), add a curated set of 158 MCP tools that cover 7 common applications, allow agents to choose between GUI actions and tool calls at every step, and measure not just success/failure but also whether tools were used appropriately and how efficiently tasks were completed.

3.2 Big-Picture Architecture (Diagram in Words)

The OSWorld-MCP system has five major components, connected as a pipeline that wraps around any agent being evaluated:

1. Virtual Machine Environment — a real Ubuntu instance running actual desktop applications (LibreOffice Calc, Writer, Impress; VS Code; VLC; Chrome; OS utilities). This is inherited from OSWorld and provides the ground-truth state for evaluation. It executes both GUI actions (mouse clicks, keyboard input) and MCP tool calls against the live operating system.

2. MCP Tool Server Layer — a set of 158 MCP servers, each exposing one or more tools (functions) that operate on the applications programmatically. For example, osworld_mcp_libreoffice_calc.copy_cells_between_sheets takes parameters like source_range, target_sheet_name, and target_start_cell and performs the operation directly through the application's API rather than through GUI simulation. These tools were generated through an automated pipeline and manually curated.

3. Tool Retrieval Module (RAG) — because providing all 158 tool descriptions to the agent at every step would create excessively long prompts, a Retrieval-Augmented Generation module filters the tool set down to only those tools relevant to the currently active application. The agent never sees all 158 tools at once; it sees a contextually appropriate subset.

4. Agent Under Evaluation — the multimodal model being tested (e.g., Claude 4 Sonnet, OpenAI o3, Qwen2.5-VL-72B-Instruct) configured with a standardized agent framework (GUI-Owl). At each step, the agent receives a screenshot of the current desktop state plus a text description of available actions (GUI operations and filtered MCP tools), and it outputs a decision: either a GUI action (click, type, scroll, etc.) or a specific MCP tool call with parameters.

5. Evaluation Module — after the agent completes or exhausts its step budget, the environment state is compared against the target state using OSWorld's existing evaluation scripts. The system computes three metrics: Task Accuracy (did it succeed?), Tool Invocation Rate (did it use tools when appropriate and avoid them when inappropriate?), and Average Completion Steps (how many steps did it take?).

Information flows as follows: a task description enters → the VM initializes the application to the starting state → at each step, the VM captures a screenshot and the RAG module selects relevant tools → the agent receives the screenshot + tool descriptions + action history → the agent outputs an action (GUI or MCP tool call) → the VM executes it → this repeats until the agent terminates or hits the step limit → the evaluator checks the final state and computes metrics.

3.3 Roadmap for the Deep Dive

• First, the tool generation pipeline (Section 3.2) — how 158 high-quality tools were created and curated, because the benchmark's validity depends entirely on tool quality. Without realistic, well-designed tools, the evaluation would measure nothing meaningful.
• Second, the tool integration architecture — how tools are made available to agents, how RAG filtering works, and how the hybrid action space (GUI + MCP) is structured at each step. This is the "what does the agent actually see and choose from?" question.
• Third, the task categorization scheme — how tasks are classified as Tool-Beneficial vs. Non-Tool-Beneficial, and why this distinction is essential for fair evaluation. This underlies the TIR metric.
• Fourth, the three evaluation metrics — Task Accuracy, Tool Invocation Rate (TIR), and Average Completion Steps (ACS) — with their formal definitions, what each measures, and how they collectively provide a multi-dimensional picture that no single metric could capture.
• Fifth, the experimental configuration — the standardized agent setup, step limits, temperature settings, model selection, and multi-run averaging protocol that ensure fair comparison across all evaluated agents.
• Finally, a summary of key design choices and their justifications — why automated generation over manual tool creation, why RAG over full tool listing, why three metrics over one, and why the specific balance of generated vs. curated tools.

3.4 Detailed, Sentence-Based Technical Breakdown

OSWorld-MCP is primarily a benchmark construction and evaluation methodology paper whose core idea is that computer-use agents should be assessed on their ability to choose between and execute both GUI operations and MCP tool calls in a unified environment, and that doing so requires (a) a carefully curated tool set, (b) a task categorization that distinguishes when tools are helpful from when they are not, and (c) metrics that separately capture success, tool-use appropriateness, and efficiency.

---

The Tool Generation Pipeline

The construction of the 158 MCP tools is arguably the most critical technical contribution of the paper because the benchmark's fairness and realism depend entirely on tool quality. If tools were hand-crafted to exactly match benchmark tasks, they would not represent real-world tool availability, and performance on them would overstate true capability. The paper designs a three-module pipeline that combines automated generation with rigorous manual curation to produce tools that are simultaneously functional, general-purpose, and verified for correctness.

The pipeline operates in three stages: Code Generation → Code Filtering → Tool Wrapping, with an additional final stage of manual curation that applies to both generated and externally sourced tools.

Stage 1: Code Generation Module

The input to this module is a target task description from OSWorld. The module uses OpenAI o3 (OpenAI, 2025b), a model with advanced reasoning capabilities, to generate Python code that accomplishes the task. The paper notes that it follows "the prompting strategy of CoAct (Song et al., 2025)" but develops its own prompt. The generated code is not yet wrapped as an MCP tool — it is simply executable code that, if run in the OSWorld VM environment, would complete the task. The model engages in multi-turn interactions, meaning it can refine its code based on intermediate feedback (execution errors, incorrect results), though the paper does not detail the exact number of turns or the stopping condition.

The key insight behind using code generation rather than manual scripting is scalability: OSWorld contains 369 tasks, and having humans write tool implementations for each would be prohibitively expensive and would risk task-specific overfitting (tools that solve exactly one benchmark task and nothing else). Automated generation, by contrast, can produce code for many tasks quickly, and the subsequent filtering stages ensure quality.

Stage 2: Code Filter Module

Not all generated code is correct or useful. The Code Filter Module uses OpenAI o3 again, but in a different role: it summarizes the usable code obtained from the multi-turn interactions in Stage 1. This summarized code is then applied to solve the corresponding OSWorld tasks — meaning it is actually executed in the VM environment against the real applications. Any code that successfully completes its associated task is retained. Code that fails (runtime errors, incorrect final state, inability to handle edge cases) is discarded.

This filtering stage is crucial because it operates on execution-based verification, not on model confidence or surface-level code inspection. A code snippet that looks plausible but uses a deprecated API, mis-specifies a parameter range, or fails on certain input states will be caught here. The paper reports that through this process, "we obtain seventy-two verified solutions" — meaning 72 distinct pieces of executable code that demonstrably solve their associated OSWorld tasks.

Stage 3: Tool Wrap Module

Verified code is not yet an MCP tool. An MCP tool requires a specific interface: a JSON-RPC-based server that exposes the function with typed parameters, human-readable descriptions, and proper error handling. The Tool Wrap Module takes each verified code solution and wraps it into an MCP-compliant tool using OpenAI o3 with a purpose-built prompt (illustrated in Figure 3a of the paper). This prompt instructs o3 to:

• Extract the function signature (name, parameters, parameter types) from the code.
• Generate a natural-language description of what the tool does, suitable for an agent to read and decide whether to invoke it.
• Package the code into an MCP server format that can be started as a subprocess and communicated with via JSON-RPC.

The output is a self-contained MCP server for each tool. For example, the code that copies cells between sheets in LibreOffice Calc becomes the tool osworld_mcp_libreoffice_calc.copy_cells_between_sheets with parameters source_range, target_sheet_name, and target_start_cell, along with a description that explains what each parameter means and what the tool accomplishes.

External Tool Curation

In parallel with the generation pipeline, the authors curate tools from existing MCP servers. These are tools that were developed for other purposes — filesystem operations, git commands, media file handling — but are relevant to OSWorld tasks. The paper states that 192 tools were initially collected from existing MCP servers, bringing the total candidate pool to 264 tools (72 generated + 192 curated).

Manual Filtering and Validation

This is where the human element enters. The paper recognizes that both generated and externally sourced tools may be unsuitable for the benchmark in two ways:

1. Task-specificity: A tool that solves exactly one OSWorld task and nothing else (e.g., a tool named complete_task_42 that hard-codes the exact steps for a specific spreadsheet operation) would not represent a real MCP tool. Real tools are designed to be reusable across a range of tasks — copy_cells_between_sheets works for any sheet-copying task, not just the one that happened to be in the benchmark.

2. Functional redundancy: Multiple tools that do essentially the same thing (e.g., two different save_file implementations with slightly different parameter names) add noise without adding evaluation value.

To address these issues, "each tool is independently evaluated by at least two reviewers with extensive GUI agent development experience, and is retained only if both reviewers deemed it qualified." This is a stringent filter — it requires consensus, not just majority vote. The result: from 264 candidate tools, 158 are retained (72 generated + 86 externally curated after removal of 106). The 158 tools span 7 applications (VS Code, Google Chrome, LibreOffice Calc, LibreOffice Writer, LibreOffice Impress, VLC, OS utilities), with distribution shown in Figure 4b: LibreOffice Calc has the most tools (24% of the set), followed by LibreOffice Impress (17%), VS Code (15%), OS (14%), LibreOffice Writer (12%), VLC (9%), and Google Chrome (9%).

Post-Curation Validation

The paper performs two additional validation steps to confirm that the retained tools are genuinely useful:

Invocation frequency analysis (Figure 4a). The five evaluated models (Qwen2.5-VL-72B-Instruct, Qwen3-VL-Plus, Seed1.5-VL, Claude 4 Sonnet, OpenAI o3) are run on OSWorld-MCP with access to all 158 tools. The paper reports that 131 out of 158 tools were invoked at least once during evaluation. The remaining 27 tools — 25 of which are identified as non-target tools serving as distractors — were not invoked. Only 2 tools from the 133 "effective" set were never invoked, and the paper hypothesizes this is "due to the complexity of the associated tasks, which likely discouraged models from attempting to invoke them." A manual re-verification confirmed these two tools are indeed functional. This analysis demonstrates that models find the vast majority of tools applicable in practice.

Task-level tool annotation and tool invocation distribution (Figure 4c). The authors manually annotate all 361 OSWorld-MCP tasks (8 Google Drive tasks excluded due to software version constraints) to record which tools are "available" for each task, where "available" means "invocation can make task execution substantially more efficient." The distribution of total available tool invocations per task (Figure 4c) shows that:

• Most tasks have either 0 or 1 available tool invocations (the leftmost bars).
• A long tail of tasks requires 2, 3, or more than 4 tool invocations.
• This distribution confirms that many tasks can benefit from multiple tool invocations, requiring agents to not just invoke a single tool but to compose tools in sequence.

Based on this annotation, 250 tasks are classified as Tool-Beneficial Tasks (at least one available tool that can improve efficiency) and 111 tasks as Non-Tool-Beneficial Tasks (no available tool improves efficiency). This binary classification is foundational to the TIR metric, which we will examine in detail later.

Why automated generation plus manual curation rather than fully manual creation? A fully manual approach — having human experts write all 158 tools — would guarantee quality but would also guarantee domain-specific tailoring. The experts, knowing the benchmark tasks intimately, would design tools that exactly solve those tasks, making the benchmark easier and less representative of real-world tool availability. Automated generation introduces a degree of "messiness": o3 generates tools that are functional but not perfectly optimized for the benchmark, and the manual curation only removes tools that are egregiously task-specific or redundant. The result is a tool set that sits in a sweet spot — functional enough for agents to use, but general enough to avoid benchmark-specific overfitting.

---

Tool Integration Architecture: How Agents Access and Choose Tools

The integration of MCP tools into the OSWorld evaluation loop is not simply a matter of adding them to a menu. The paper makes several architectural choices that shape what the agent sees, how it chooses, and what constraints it operates under.

The Hybrid Action Space

At each step of task execution, the agent faces a binary choice embedded in its action space: it can either issue a GUI operation or invoke an MCP tool. These are mutually exclusive for a single step — the agent cannot click and call a tool simultaneously. The GUI operations available are the same 11 basic operations defined in original OSWorld: key, type, mouse move, click, drag, right click, middle click, double click, scroll, wait, and terminate. The MCP operations available are the set of tool calls with their typed parameters, as filtered by the RAG module.

This design means that at every step, the agent must decide not just what to do but which modality to use. Should it click on the "Save" icon, or call osworld_mcp_libreoffice_writer.save? Should it type text into a document, or call osworld_mcp_libreoffice_writer.write_text? The paper calls this "dynamic interaction between GUI operations and tool usage," and it is what distinguishes OSWorld-MCP from both GUI-only benchmarks (where the choice doesn't exist) and text-only tool benchmarks (where the GUI alternative doesn't exist).

Retrieval-Augmented Generation (RAG) for Tool Filtering

A practical problem: OSWorld-MCP contains 158 tools. If all 158 tool descriptions were included in the prompt at every step, the context length would be enormous. For a typical agent configuration, each tool description includes the tool name, a natural-language summary of its purpose, its parameter names, types, and descriptions. With 158 such entries, the prompt would easily exceed practical context limits, and — perhaps more importantly — the agent would face a needle-in-haystack problem: finding the right tool among 158 options at every step.

The solution is Retrieval-Augmented Generation (RAG). At each step, the system identifies which application is currently active (this is known from the VM state) and selects only the tools relevant to that application. For example, if the agent is working in LibreOffice Calc, only the LibreOffice Calc tools are included in the action space; the VS Code, Chrome, Writer, Impress, VLC, and OS tools are excluded. This reduces the visible tool set from 158 to approximately 20-25 tools per step, depending on the application.

The RAG filtering is application-based, not task-based — it does not try to predict which specific tools the agent will need for its current task. This is an important design choice: the agent still must select the right tool from among all tools relevant to the current application, which is a non-trivial challenge (as the results show — tool invocation rates remain low even with RAG filtering). If RAG filtered down to exactly the one tool needed, tool selection would be trivial and the benchmark would not measure decision-making.

The paper's ablation study (Table 2) confirms the importance of RAG: when RAG is removed and the agent is shown all 158 tools, overall accuracy drops from 20.5 to 15.5 for Gemini-2.5-Pro, and Tool-Beneficial task accuracy drops from 24.9 to 18.0. The TIR drops from 20.4 to 12.4, indicating that the model becomes less inclined to invoke tools when faced with the full list — a form of choice paralysis.

Deterministic Tool Description Ordering

In the default OSWorld-MCP configuration, tool descriptions are presented to the agent in alphabetical (lexicographical) order within each application. The authors made this choice for "consistency in evaluation" — if tool descriptions were randomly ordered, different evaluation runs of the same agent could produce different results due to order effects (the agent might be biased toward the first or last tools listed). Alphabetical ordering eliminates this source of variance.

However, the ablation study in Table 2 reveals a counterintuitive finding: randomly shuffling the tool description order actually improves overall accuracy from 20.5 to 22.7 for Gemini-2.5-Pro. The paper attributes this to Non-Tool-Beneficial Tasks, where shuffled ordering may "implicitly suggest alternative solution strategies." For consistency, the paper maintains alphabetical ordering in the main benchmark, but this finding is flagged as an interesting direction for future work on prompt design.

Step Budgets

Every task is evaluated under a fixed maximum step limit — either 15 or 50 steps, as reported in Table 1. The agent can issue a terminate action at any point (with a success or failure status) if it believes the task is complete or infeasible. If the agent reaches the step limit without terminating, the task is considered incomplete (failed). The two step limits (15 and 50) test different capability regimes: 15 steps rewards efficiency and correct tool selection (an agent that dithers or invokes the wrong tool and needs to recover will exhaust its budget), while 50 steps gives more room for exploration and recovery, testing whether additional computation improves success.

---

Task Categorization: Tool-Beneficial vs. Non-Tool-Beneficial

The binary classification of tasks into Tool-Beneficial and Non-Tool-Beneficial is not merely descriptive — it is the foundation of the Tool Invocation Rate (TIR) metric and a core design element that ensures the benchmark rewards appropriate tool use, not indiscriminate tool invocation.

The Annotation Process

The paper does not automate this classification — it is performed manually by human reviewers who examine each of the 361 OSWorld-MCP tasks (8 Google Drive tasks excluded) and determine whether any of the 158 MCP tools, if invoked, would make task execution "substantially more efficient." This is a judgment call that requires understanding both what the task demands and what the available tools can do. An "available tool" is one whose invocation would substitute for multiple GUI steps — in other words, a tool that provides a shortcut relative to the GUI-only path.

The result: 250 tasks are classified as Tool-Beneficial (at least one available tool) and 111 tasks as Non-Tool-Beneficial (no available tool improves efficiency). This 69%-31% split means that the majority of OSWorld tasks have some tool-based efficiency path, which is consistent with the paper's goal of creating a benchmark where tool invocation is broadly relevant but not universally applicable.

Why Not Use Tools on All Tasks?

The paper makes a subtle but important distinction between tasks where tools exist and tasks where tools help. Some OSWorld tasks, by their nature, are most naturally solved through GUI operations — for example, tasks that require selecting a specific visual element that is not addressable by any programmatic API, or tasks where the primary challenge is navigating a complex menu structure. For such tasks, invoking a tool would either be impossible (no appropriate tool exists) or would require the agent to first perform GUI operations to gather information, making the net efficiency dubious. The Non-Tool-Beneficial classification ensures that the benchmark does not penalize agents for correctly recognizing that a GUI path is the right approach.

The Role of Non-Target Tools as Distractors

The paper mentions that 25 tools in the set are "non-target tools" that serve as distractors. These are tools that exist in the MCP servers but are not classified as available for any task — they are present in the tool descriptions that agents see (filtered by RAG to the relevant application) but are never the most efficient path to task completion. Their presence tests whether agents can resist the temptation to invoke tools just because they are available, or whether they correctly ignore irrelevant tools and choose the appropriate action (GUI or relevant tool). An agent with a high tool invocation rate but low accuracy might be invoking these distractor tools incorrectly, suggesting poor tool-selection judgment.

---

The Three Evaluation Metrics

OSWorld-MCP introduces three metrics designed to jointly assess success, tool-use appropriateness, and efficiency. No single metric captures the full picture — an agent with high accuracy might achieve it through inefficient GUI paths (low TIR, high ACS), while an agent with high TIR might be invoking tools incorrectly (low accuracy). The three metrics together surface these patterns.

Metric 1: Task Accuracy

Task Accuracy is the simplest and most directly inherited metric from OSWorld. For each task, the agent either succeeds (the final environment state matches the target state, as verified by OSWorld's evaluation scripts) or fails. Accuracy is the fraction of the 361 tasks on which the agent succeeds. It is reported at two levels of granularity: overall accuracy (across all tasks) and split accuracy (separately for Tool-Beneficial Tasks and Non-Tool-Beneficial Tasks). The split accuracy is important because it reveals whether an agent's overall improvement from MCP tools comes from the tasks where tools are relevant (Tool-Beneficial) or from some other factor.

The evaluation scripts are inherited from OSWorld and operate by checking the final state of the virtual machine — for example, verifying that a specific file exists with specific content, that a particular VS Code setting has been changed, or that a spreadsheet cell contains the expected value. These scripts are execution-based, not LLM-judged, which means they are deterministic and reliable.

Metric 2: Tool Invocation Rate (TIR)

TIR is the paper's most innovative metric, designed to measure whether an agent invokes tools when it should and avoids them when it shouldn't. The formal definition is:

$TIR = \frac{n_t + n_g}{N_t + N_g}$

where:

• $N_t$ is the total number of Tool-Beneficial Tasks (250 tasks),
• $n_t$ is the number of Tool-Beneficial Tasks in which the agent invoked a tool AND successfully completed the task,
• $N_g$ is the total number of Non-Tool-Beneficial Tasks (111 tasks),
• $n_g$ is the number of Non-Tool-Beneficial Tasks in which the agent did NOT invoke a tool AND successfully completed the task.

What it computes: TIR is the fraction of all tasks (regardless of classification) where the agent's tool-use behavior was appropriate — meaning it used a tool on a task where tools help and then succeeded, or it avoided tools on a task where tools don't help and then succeeded. The numerator $n_t + n_g$ counts tasks where the behavioral criterion (use tool / don't use tool) matches the task's classification AND the task was completed successfully. The denominator $N_t + N_g = 361$ is all tasks.

Why this form: TIR is deliberately gated on task success. An agent that invokes a tool on a Tool-Beneficial Task but fails is not counted in $n_t$ — the tool invocation might have been incorrect (wrong parameters, wrong tool) or the task might have failed for other reasons, but in either case, we cannot claim the tool invocation was appropriate. Similarly, an agent that avoids tools on a Non-Tool-Beneficial Task but also fails is not counted in $n_g$. This gating prevents TIR from inflating due to coincidental tool-use patterns on failed tasks.

The alternative — counting tool invocation regardless of success — would conflate tool invocation attempts with tool invocation skill. An agent that invokes tools aggressively but incorrectly (high invocation count, low success) would score deceptively high, which would misrepresent its capability.

TIR is bounded between 0 and 1 (or 0% to 100%). A perfect TIR of 100% would mean the agent succeeded on every task AND used tools exactly when appropriate for that task type. In practice, TIR is constrained by accuracy — it can never exceed the accuracy, because every task counted in the numerator must also be a successful task.

Metric 3: Average Completion Steps (ACS)

ACS measures efficiency — how many steps the agent takes to complete tasks. The formal definition is:

$ACS = \frac{1}{N} \sum_{i=1}^{N} S_i$

where:

• $N$ is the number of tasks (361),
• $S_i$ is the number of execution steps taken on task $i$.

What it computes: ACS is simply the arithmetic mean of per-task step counts, including both successful and failed tasks. For failed tasks, $S_i$ is typically the maximum step limit (15 or 50), because the agent exhausts its budget without terminating successfully.

Why this form: The simple arithmetic mean captures overall efficiency without distinguishing between success and failure — a lower ACS means the agent is, on average, completing tasks (or determining infeasibility) in fewer steps. ACS can decrease even if accuracy stays the same, indicating that the agent is becoming more efficient at the tasks it does solve. Conversely, ACS can increase if the agent starts attempting harder tasks that require more steps (or that it fails at, consuming the full budget).

The paper reports ACS at two levels: overall ACS (across all tasks) and split ACS (separately for Tool-Beneficial and Non-Tool-Beneficial Tasks). The split ACS is particularly informative for Non-Tool-Beneficial Tasks, where a decrease in ACS after introducing MCP tools suggests that the presence of tools helps the agent eliminate irrelevant solution paths — even if it doesn't use the tools, knowing what tools exist may guide it toward the correct GUI approach faster.

The Multi-Metric Picture: Why Three Metrics?

Task Accuracy alone would miss important patterns. Consider two hypothetical agents A and B, both with 30% accuracy on OSWorld-MCP:

• Agent A achieves its accuracy through efficient tool invocation on Tool-Beneficial Tasks (high TIR, low ACS) and performs poorly on Non-Tool-Beneficial Tasks.
• Agent B achieves its accuracy through laborious GUI operations (low TIR, high ACS) and performs uniformly across both task types.

Accuracy alone reports them as equal. TIR reveals that Agent A has strong tool-invocation judgment but perhaps lacks robustness on pure-GUI tasks; Agent B has the opposite profile. ACS reveals that Agent A is much faster. A developer looking at these three numbers can diagnose what to improve — Agent A needs better GUI grounding, while Agent B needs better tool selection or parameter specification.

Similarly, two agents with the same TIR might have very different accuracy: one might have high TIR and high accuracy (appropriate tool use leading to success), while another might have high TIR and low accuracy (appropriate tool-use behavior but still failing, perhaps due to parameter errors or grounding failures on GUI-only segments of tasks). The triple of (Accuracy, TIR, ACS) disambiguates these cases.

---

Experimental Configuration

The paper evaluates a range of state-of-the-art multimodal agents under a standardized configuration that is designed to isolate model capability differences from infrastructure differences.

Agent Framework: GUI-Owl

All models are evaluated using the GUI-Owl agent configuration, a standardized framework that wraps the core LMM and manages the perception-action loop. The paper explicitly notes that using GUI-Owl "may lead to some performance fluctuations for certain models under the original OSWorld configuration" — meaning the absolute numbers in Table 1 may differ from what those models achieve under their native agent frameworks. This is a tradeoff: standardization enables fair comparison across models but may under-represent the peak performance of models that are optimized for different agent architectures.

At each step, the GUI-Owl framework:

1. Captures a screenshot of the current VM display.
2. Passes the screenshot along with the action history (previous steps' thoughts and actions) and the filtered tool descriptions to the core LMM.
3. The LMM performs visual perception, generates a reasoning summary ("thought"), and proposes the next action (either a GUI operation with coordinates/parameters or an MCP tool call with parameter values).
4. The framework executes the action in the VM environment.
5. The cycle repeats until termination or step budget exhaustion.

The reasoning history is carried forward to subsequent steps, meaning the agent has access to its own prior thoughts when making decisions. This is standard in agent evaluations — it allows the agent to remember what it has already tried and why.

Model Selection

The paper evaluates seven systems:

End-to-end models (6):

• Qwen2.5-VL-72B-Instruct (Bai et al., 2025) — open-weight, 72B parameters.
• Qwen3-VL-Plus (QwenTeam, 2025) — proprietary, next-generation Qwen model.
• Gemini-2.5-Pro (Comanici et al., 2025) — Google's proprietary multimodal model.
• OpenAI o3 (OpenAI, 2025b) — OpenAI's reasoning-focused model.
• Seed1.5-VL (Guo et al., 2025b) — proprietary.
• Claude 4 Sonnet (Anthropic, 2025) — Anthropic's proprietary model.

Multi-agent framework (1):

• Agent-S2.5 (Simular Research, 2025) — uses OpenAI o3 as the main generation model with UI-TARS-1.5-72B as the grounding model.

This selection spans open-weight and proprietary models, single-model and multi-agent architectures, and reasoning-specialized and generalist designs. The inclusion of Agent-S2.5 is notable because it represents a different architectural approach (separate models for reasoning about what to do vs. grounding where to click) and tests whether such architectures have different tool-invocation profiles than end-to-end models.

Hyperparameters

• Temperature: Set to 1.0 for all models. This is relatively high and encourages exploration, which is appropriate for a benchmark that tests decision-making under uncertainty — the agent should consider a range of possible actions rather than always picking the highest-probability one.
• Step limits: 15 and 50 steps, evaluated separately.
• Runs: Each model is evaluated three times under each configuration, with results averaged. The paper notes that multiple runs are necessary "due to fluctuations in the experimental results," which is expected given the stochasticity introduced by temperature = 1.0 and the inherent variability in GUI grounding (the same model on the same task may click slightly different coordinates on different runs, leading to different execution paths).

The three-run averaging is a robustness measure: it reduces the impact of outlier runs where a model got lucky or unlucky on specific tasks, providing a more stable estimate of true capability.

Evaluation Protocol

For each task, the VM is reset to a known initial state. The agent interacts with the environment step-by-step under the configured step limit. After the agent terminates or exhausts its budget, the evaluation script checks the final state. The three metrics (Accuracy, TIR, ACS) are computed over all 361 tasks, both overall and split by task type.

The paper uses a fixed step limit rather than a time limit. This is appropriate for a benchmark where the primary cost model is the number of decisions the agent makes, not wall-clock time — a tool call and a GUI click are both "one step," even though they might take different amounts of wall-clock time to execute (MCP tool calls are typically faster because they don't require screen rendering and mouse movement simulation).

---

Summary of Design Choices and Their Justifications

• Automated tool generation with manual curation over fully manual creation: produces tools that are functional and realistic without being benchmark-specific. Manual curation removes task-specific and redundant tools that would inflate performance unrealistically.
• Application-based RAG filtering over no filtering: keeps context lengths manageable and avoids choice paralysis, while still requiring the agent to select the right tool from ~20-25 candidates per application.
• Alphabetical tool description ordering over random ordering: ensures reproducibility across evaluation runs, eliminating order effects as a source of variance.
• Three metrics (Accuracy, TIR, ACS) over accuracy alone: disambiguates different agent capability profiles — an agent with high accuracy but low TIR succeeds through inefficient GUI paths; an agent with high TIR but low accuracy invokes tools eagerly but incorrectly; the combination reveals the nature of the agent's strengths and weaknesses.
• TIR gated on task success over counting all tool invocations: prevents inflation from coincidental tool use on failed tasks and ensures the metric reflects appropriate tool use, not just any tool use.
• Two step limits (15 and 50) over a single limit: tests both efficiency-constrained performance (15 steps rewards fast, correct decisions) and exploration-tolerant performance (50 steps allows recovery from mistakes).
• Three-run averaging over single-run evaluation: mitigates stochasticity from temperature sampling and grounding variability, providing more stable capability estimates.
• Task categorization by manual annotation over automated classification: the judgment of whether a tool provides "substantially more efficient" execution requires human understanding of task semantics and tool capabilities that automated methods cannot reliably perform.
• Building on OSWorld over creating a new benchmark from scratch: inherits OSWorld's proven VM infrastructure, 369-task benchmark, and state-based evaluation, ensuring that the new tool-invocation dimension is added to a solid, well-understood foundation rather than being entangled with new environment bugs or task design flaws.

4. Key Insights and Innovations

Innovation 1: Tool Invocation Is Not a Capability to Add But a Decision-Making Skill to Measure

The paper's deepest conceptual move is reframing tool invocation from an action-space expansion (just adding more things an agent CAN do) into a decision-making capability (knowing WHEN to do which thing). This shift may sound subtle, but it has profound consequences for how we design and evaluate computer-use agents, and it directly challenges the dominant assumption in prior work.

What the field did before. Prior benchmarks treated tool use and GUI operation as separate evaluation silos. GUI benchmarks like OSWorld (Xie et al., 2024), WindowsAgentArena (Bonatti et al., 2024), and AndroidWorld (Rawles et al., 2024) defined a fixed set of ~11 GUI primitives and asked: can the agent execute the right sequence of clicks and types? Text-based tool-use benchmarks like MCPEval (Liu et al., 2025) and MCP-Bench (Wang et al., 2025b) asked: can the agent select the right function and specify its parameters correctly, given a text task description? Each paradigm measured one capability in isolation. The unstated assumption was that adding tool invocation to a GUI agent was simply a matter of expanding the action space — give the model more options, and it will use them when helpful.

The paper's TIR metric and its task categorization scheme (Tool-Beneficial vs. Non-Tool-Beneficial) reveal why this assumption is wrong. If tool invocation were merely an action-space expansion, then giving agents access to 158 tools should uniformly improve performance — more options means more paths to success. But the paper's experimental data (Table 1) shows something more complex: Qwen2.5-VL-72B-Instruct actually sees its ACS increase (get worse) when tools are introduced, while its accuracy barely moves. Conversely, Gemini-2.5-Pro shows a dramatic accuracy jump (+13.1 percentage points at 15 steps) but its TIR remains at only 16.8% — meaning it's benefiting from tools but still failing to use them appropriately on most Tool-Beneficial tasks. These asymmetric patterns cannot be explained by an action-expansion model; they demand a decision-making model where the agent must choose correctly between modalities, not just use more modalities.

What makes this a fundamental shift rather than incremental. The paper doesn't just observe that tool invocation sometimes helps — that would be incremental. The fundamental shift is the recognition that the skill of choosing between GUI and tool paths is conceptually distinct from and can be measured independently of both GUI execution skill and tool parameterization skill. The TIR metric operationalizes this distinction: it does not reward tool invocation per se (an agent that calls tools on every task gets penalized on Non-Tool-Beneficial tasks), nor does it reward task success alone (an agent that succeeds through laborious GUI operations on Tool-Beneficial tasks gets penalized by lower TIR). It rewards alignment between behavior and task type — using tools when and only when they are appropriate. This is a meta-cognitive capability that no prior benchmark could measure, because none provided both the choice and the ground-truth annotation of when the choice matters.

Evidence that grounds this claim. The difficulty-bin-style analysis in Figure 5b is particularly revealing. When the authors group tasks by the number of available tools (0, 1, 2, 3, >4), they find that for tasks with 0 available tools, the GUI+MCP configuration outperforms GUI-only — even though tools aren't helpful, their mere presence appears to guide the agent toward correct GUI paths faster (lower ACS). For tasks with 1-2 available tools, the tool configuration shows clear accuracy gains and reduced ACS. But for tasks with >4 available tools, accuracy drops sharply and ACS rises. This non-monotonic relationship between tool availability and performance is exactly what a decision-making model would predict: a few good tools help, but too many tools create a selection problem that overwhelms the agent's decision-making capacity. The agent doesn't fail because it can't execute tools; it fails because it can't decide which tool to use among many plausible candidates. This is a decision-making bottleneck, not an execution bottleneck, and identifying it as such is the paper's distinctive conceptual contribution.

The paper also demonstrates that this decision-making skill is currently weak across all models — even Claude 4 Sonnet, the best performer, achieves only a 36.3% TIR at 50 steps. This means that on 63.7% of tasks, the agent either used a tool when it shouldn't have, avoided a tool when it should have, or failed to complete the task. The gap between accuracy gains (which are substantial for most models) and TIR (which remains low) is the paper's most diagnostic finding: agents are benefiting from tools but they are not mastering tool-use as a decision-making skill. They are getting lucky on some tasks, not reliably choosing the right modality.

Innovation 2: Tool Invocation Rate (TIR) as a Diagnostic Metric That Separates Capability from Behavior

The paper introduces TIR not merely as a second number to report alongside accuracy, but as a diagnostic instrument that reveals why an agent succeeds or fails in ways that accuracy alone systematically obscures. This is a methodological innovation that changes how we should think about evaluating interactive agents — and it generalizes well beyond MCP tools to any setting where an agent has multiple distinct action modalities that are differentially appropriate across task types.

What the field did before. Evaluation of computer-use agents has been overwhelmingly dominated by a single metric: task success rate. This is true of OSWorld, WebArena (Zhou et al., 2023), WindowsAgentArena, WorkArena (Drouin et al., 2024), AndroidWorld — essentially every interactive agent benchmark. The rationale is straightforward: task success is what we ultimately care about, so it should be the metric we optimize. The problem with this rationale is that it conflates multiple distinct failure modes into a single number. A 30% accuracy rate tells us the agent fails 70% of the time, but it tells us nothing about why — is the agent bad at perceiving the interface? Is it bad at selecting the right action? Is it bad at executing the action precisely? Is it choosing the wrong modality? Without diagnostic metrics, accuracy is a blunt instrument that cannot guide improvement.

The paper's innovation is to recognize that in a setting with two qualitatively different action modalities (GUI operations vs. MCP tool calls), there is a natural partitioning of failure modes that can be captured by a carefully designed metric. TIR isolates the decision-making component — "did the agent use the right action type?" — and gates it on success, ensuring that the behavioral signal (tool invocation or avoidance) is meaningful. Accuracy minus TIR tells us how often the agent uses the correct modality but still fails (perhaps due to execution errors — clicking the wrong pixel, specifying wrong tool parameters). Tasks where the agent uses the wrong modality and fails are captured by neither accuracy nor TIR, but their frequency can be inferred from the gap between overall accuracy and the sum of TIR components.

Why this is fundamental, not incremental. Adding a second metric is trivially incremental. What makes TIR fundamental is its gating structure — the requirement that the agent must both behave appropriately AND succeed for the behavior to count. This design choice embeds a philosophical stance about what it means to "use tools well." The paper rejects a purely behavioral definition (counting any tool invocation, regardless of outcome) because an agent that calls the wrong tool with wrong parameters is not demonstrating tool-use skill, even if it's demonstrating tool-use intent. The paper also rejects a purely outcome-based definition (counting any successful task, regardless of method) because accuracy already captures that. TIR sits at the intersection — it measures the fraction of tasks where good decision-making AND good execution jointly produced success.

This gating structure has a practical consequence that makes TIR useful for model comparison: TIR is constrained to be ≤ accuracy. This means the gap between accuracy and TIR is always non-negative and has a clear interpretation — it's the fraction of tasks where the agent succeeded but used the wrong action type (e.g., completing a Tool-Beneficial task through slow GUI operations), or where the agent used the right action type but still failed for execution reasons. In Table 1, Claude 4 Sonnet at 50 steps achieves 43.3% accuracy with 36.3% TIR — a 7-point gap. This gap represents tasks where Claude used tools on Non-Tool-Beneficial tasks or avoided tools on Tool-Beneficial tasks but still succeeded through the other path (it was "right for the wrong reasons"), or where it attempted the correct action type but failed. Either way, the gap diagnoses that Claude's decision-making is imperfect even when it succeeds — a signal entirely invisible to accuracy alone.

Evidence that grounds this claim. The correlation analysis in Figure 5a shows that TIR and accuracy are positively correlated across models — agents that make better modality choices tend to succeed more often. But the relationship is not 1:1, and the scatter reveals model-specific patterns that would be invisible in an accuracy-only analysis. For instance, if TIR and accuracy were perfectly correlated (all points on a diagonal line), then TIR would add no information beyond accuracy. The fact that they are positively but imperfectly correlated means that different models have different "efficiency of tool use" — some convert good decisions into success at higher rates than others. ACS adds a third dimension: an agent with high accuracy and high TIR but also high ACS might be making the right modality choices but executing them slowly, suggesting a need for better grounding or parameter specification rather than better decision-making.

The practical value of this diagnostic power is demonstrated by the paper's own analysis. The finding that "tool invocation rates for multimodal agents remain relatively low" (Finding 2) — even for models that show large accuracy gains from tools — is a TIR-based insight that accuracy alone could never produce. It tells the field that the bottleneck is not tool availability (the tools exist and models benefit from them) nor tool execution (models succeed when they use the right tool) but rather tool selection and invocation decision-making. This directs future research toward improving models' ability to recognize when a tool is appropriate, rather than toward building more tools or improving GUI grounding. That's a strategically valuable redirection that a single-metric evaluation would miss.

Innovation 3: The Multi-Tool Composition Challenge as the Real Frontier

The paper's third significant contribution is diagnostic rather than architectural: it empirically establishes that composing multiple tool calls is qualitatively harder for current LMMs than combining GUI operations, and it identifies this as the primary scaling challenge for tool-augmented computer-use agents. This finding reframes the research agenda from "give agents more tools" to "teach agents to chain tools together," which is a fundamentally different and harder problem.

What the field assumed. The standard assumption in the tool-use literature — both in text-based settings (tool-augmented LLMs, function calling) and in emerging agent systems (CoAct, ComputerRL) — is that tool invocation difficulty scales roughly linearly with the number of tools needed. If an agent can reliably invoke one tool with 80% reliability, then a task requiring two sequential tool calls should succeed with roughly 0.80² = 64% reliability, assuming independence. The degeneration is treated as a compounding-probability problem: each additional tool call multiplies the failure probability, but the per-call difficulty is roughly constant.

The paper's data challenges this assumption. Figure 5b shows that for Claude 4 Sonnet — the best-performing model in the study — accuracy drops sharply when the number of available tools exceeds 3, and the decline is steeper than a simple compounding-probability model would predict. More critically, the paper's comparison between GUI+MCP and GUI-only configurations reveals that when tasks require multiple tool invocations, the GUI+MCP configuration sometimes produces lower accuracy than GUI-only, despite having access to strictly more powerful actions. This is a composition failure: the agent cannot successfully chain tool calls, and the failed tool chains are worse than simply using slower but more familiar GUI sequences.

Why this is a fundamental finding. The existence of a multi-tool composition barrier is not just a "models aren't good enough yet" observation — it reveals a structural difficulty that distinguishes tool composition from GUI composition. In GUI operation, the feedback loop is tight: after each click, the screen changes and the agent sees the result immediately. The visual feedback provides grounding and error-correction signals that make sequential GUI operations relatively robust — if the agent clicks the wrong menu, it sees that the expected dialog didn't appear and can adjust. Tool calls, by contrast, operate on application state that may not be fully visible on screen. If the agent calls copy_cells_between_sheets with an incorrect source range, the error might not be apparent until several steps later when the agent switches to the target sheet and discovers the wrong data was copied. The feedback delay makes tool composition harder to learn and harder to execute correctly, because the agent must maintain an accurate mental model of the application state across multiple tool calls without visual confirmation at each step.

The paper's case study in Figure 9 illustrates this concretely. The task — copy the "Revenue" column to a new sheet named "Sheet2" — requires the agent to (1) create or switch to Sheet2, (2) copy the correct column with header from Sheet1 to Sheet2, and (3) verify the result. The successful agent (Gemini-2.5-Pro with tools) makes 9 steps and calls copy_cells_between_sheets four times, get_workbook_info once, and switch_active_sheet three times before terminating with success. The multiple calls to copy_cells_between_sheets are not redundant — they represent the agent retrying because earlier calls didn't achieve the expected state. This is tool composition under uncertainty: the agent doesn't know from visual feedback whether the copy succeeded, so it probes, retries, and switches sheets to verify. The fact that even the successful case requires 9 steps for what should be a 2-step tool sequence (create sheet, copy column) underscores how far current agents are from reliable multi-tool composition.

Evidence that grounds this claim. The paper's evidence for the composition challenge comes from multiple angles. First, the difficulty-bin analysis in Figure 5b: as tool count increases beyond 3, accuracy drops and ACS rises, indicating that tasks requiring many tools are not just harder — they're pushing agents into a regime where the tool-invocation strategy itself becomes counterproductive. Second, the task-level statistics in Figure 4c show that a substantial fraction of tasks require 2, 3, or >4 tool invocations, confirming that multi-tool composition is not a rare edge case but a common requirement in realistic tasks. Third, the paper's explicit statement in Finding 3 — "combining multiple tools is more challenging than combining GUI operations" — directly asserts a qualitative difference that, if replicated across other models and benchmarks, would change how we prioritize research on agent architectures.

The practical implication is significant: the field's current focus on building larger and more diverse tool sets (more MCP servers, more API integrations) may be misdirected for the current capability level of LMMs. If agents cannot reliably chain even 2-3 tool calls, expanding the tool set to 500 or 1000 tools adds complexity without addressing the primary bottleneck. The research priority should shift toward training paradigms that explicitly teach multi-tool composition — perhaps through supervised fine-tuning on tool-chaining trajectories, reinforcement learning with intermediate state-based rewards, or architectural modifications that provide better state-tracking across tool calls.

Innovation 4: A Benchmark Construction Methodology That Solves the Tool-Tailoring Validity Problem

While the previous three innovations concern what the benchmark reveals about agents, this innovation concerns how the benchmark was built. The paper's tool generation pipeline — automated code generation + execution-based filtering + MCP wrapping + consensus-based manual curation — is not just an engineering detail; it addresses a fundamental validity threat that affects all tool-augmented benchmarks: the risk that tools are designed specifically to match benchmark tasks, inflating performance relative to real-world deployment.

Why this is a non-obvious problem. When constructing a benchmark that evaluates tool-use capability, the natural impulse is to build tools that are maximally useful for the benchmark tasks — after all, if the tools don't help, what's the point? But this creates a circularity: the tools are optimized for the exact tasks they will be evaluated on, which means performance on the benchmark overestimates performance in real settings where tools are designed by third parties for general purposes, not for the specific tasks a user happens to need. This is a form of benchmark overfitting that operates at the tool level rather than the model level: the evaluation infrastructure itself, not just the model, has been tailored to the test distribution.

Prior tool-use benchmarks have not adequately addressed this problem. MCP-Bench (Wang et al., 2025b) defines tasks based on available tools, which inverts the natural causality — in the real world, tasks drive tool creation, not vice versa. Text-based benchmarks like MCPEval (Liu et al., 2025) use small, hand-curated tool sets that are inevitably designed with benchmark tasks in mind. OSWorld-MCP's approach — generate code solutions for benchmark tasks, filter for correctness, then manually remove task-specific tools — is an attempt to break this circularity. The automated generation introduces a degree of "naturalness": the LLM generates code that solves the task, but it does so without knowledge of what other tasks exist or how the tool might be useful beyond the immediate task. The manual curation then removes tools that are clearly overfitted to a single task, selecting only those that represent "genuinely relevant to real-world needs" and "not artificially tailored for specific benchmark tasks."

What makes this a methodological contribution. The paper is, to our knowledge, the first to explicitly design a tool generation methodology that targets the middle ground between fully hand-crafted tools (maximally benchmark-tailored, minimally realistic) and fully external tools (maximally realistic, but may not cover the benchmark's task distribution well enough to enable meaningful evaluation). The three-stage generation-plus-curation pipeline is reproducible — other benchmarks that need tool sets for evaluating agent capabilities can adopt the same approach. The key design insight is that automated generation ensures coverage of benchmark-relevant tasks, while manual curation ensures generality beyond individual tasks. These two forces push in opposite directions, and the pipeline's structure balances them.

The evidence that this methodology succeeded is the paper's own analysis of tool usage (Section 3.3). Of the 158 tools, 131 were invoked at least once by at least one tested model. If the tools were highly benchmark-specific — designed to solve exactly the 250 Tool-Beneficial tasks and nothing else — we would expect models to use them on those specific tasks and nowhere else, which would produce a different invocation pattern. The fact that tools were invoked across models with different decision-making tendencies suggests they have general applicability. The 25 non-target tools that serve as distractors (present in the tool set but never the most efficient path for any task) provide further validation: a benchmark where every tool is optimal for some task would look suspiciously tailored; the presence of genuinely non-optimal tools makes the evaluation more realistic.

Distinguishing incremental from fundamental. The specific pipeline (OpenAI o3 for generation, execution-based filtering, MCP wrapping, dual-reviewer curation) is incremental — it uses existing technologies in a straightforward sequence. What's fundamental is the recognition of the tool-tailoring validity problem as a first-class concern in benchmark design and the proposal of a general solution strategy (automated generation + manual curation with tool-task generality as the curation criterion). As the MCP ecosystem matures and more tool-augmented benchmarks emerge, this validity concern will only grow — benchmarks that simply bundle whatever tools are available risk measuring tool-set-specific performance rather than general tool-use capability. The paper's methodology provides a template for how to build tool sets that are simultaneously task-relevant (they help with the benchmark's tasks) and realistically general (they would help with many other tasks too).

This innovation matters beyond the specific results of OSWorld-MCP because it addresses a threat to the entire research program of evaluating tool-augmented agents. If benchmarks cannot provide credible estimates of real-world tool-use capability — because their tools are overfitted to their tasks — then the field will optimize for the wrong thing, and reported gains will fail to transfer to deployment. The paper's methodology is an early attempt to solve this problem, and while it is not perfect (the tools were still generated from the same model family that will be evaluated, creating a potential distributional confound), it establishes the problem as one that benchmark designers must address.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the OSWorld-MCP benchmark described in Section 3: 361 tasks (8 Google Drive tasks excluded due to software version constraints) built on the OSWorld environment (Xie et al., 2024), which provides a dynamic, interactive evaluation framework with real Ubuntu virtual machines and state-based evaluation scripts. The tasks are split into 250 Tool-Beneficial Tasks (at least one MCP tool can substantially improve efficiency) and 111 Non-Tool-Beneficial Tasks (no available tool improves efficiency). Tasks span 7 applications: LibreOffice Calc, Writer, Impress; VS Code; VLC; Google Chrome; and OS-level utilities.

• Base model(s). The paper evaluates six end-to-end Large Multimodal Models (LMMs) and one multi-agent framework. End-to-end models: Qwen2.5-VL-72B-Instruct (Bai et al., 2025), Qwen3-VL-Plus (QwenTeam, 2025), Gemini-2.5-Pro (Comanici et al., 2025), OpenAI o3 (OpenAI, 2025b), Seed1.5-VL (Guo et al., 2025b), and Claude 4 Sonnet (Anthropic, 2025). Multi-agent framework: Agent-S2.5 (Simular Research, 2025), which uses OpenAI o3 as the main generation model with UI-TARS-1.5-72B as the grounding model. The models were selected to span open-weight and proprietary systems, single-model and multi-agent architectures, and reasoning-specialized and generalist designs. All models are evaluated using the standardized GUI-Owl agent framework to ensure fair comparison.

• Metrics. Three metrics are used. Task Accuracy: fraction of the 361 tasks where the agent's final environment state matches the target state, as verified by OSWorld's execution-based evaluation scripts. Reported overall and separately for Tool-Beneficial and Non-Tool-Beneficial tasks. Tool Invocation Rate (TIR): defined as $TIR = (n_t + n_g) / (N_t + N_g)$ where $N_t = 250$ (Tool-Beneficial tasks), $n_t$ is the count of Tool-Beneficial tasks where the agent invoked a tool AND succeeded, $N_g = 111$ (Non-Tool-Beneficial tasks), and $n_g$ is the count of Non-Tool-Beneficial tasks where the agent did NOT invoke a tool AND succeeded. TIR measures the fraction of tasks where tool-use behavior was appropriate and successful — it is explicitly gated on task success, preventing inflation from coincidental tool use on failed tasks. Average Completion Steps (ACS): arithmetic mean of per-task step counts, $ACS = \frac{1}{N} \sum_{i=1}^{N} S_i$, where $S_i$ is the number of steps on task $i$ (equal to the step limit for failed tasks).

• Baselines. The primary baseline is the GUI-only setting, where agents can only use the 11 basic GUI operations defined in original OSWorld (key, type, mouse move, click, drag, right click, middle click, double click, scroll, wait, terminate) without access to any MCP tools. This is compared against the GUI + MCP setting, where at each step the agent can autonomously choose between a GUI operation and an MCP tool call from the 158-tool set (filtered by RAG to the currently active application). This within-benchmark comparison isolates the effect of tool access on the same tasks. No text-only baselines or external tool-use benchmarks are directly compared against, since the paper's contribution is the integrated evaluation framework itself.

• Generation budget / compute accounting. The "compute" is measured in execution steps — each step is one action (either a GUI operation or an MCP tool call). Every task is evaluated under two fixed maximum step limits: 15 steps and 50 steps. The 15-step setting rewards efficiency and correct decision-making (agents that waste steps on wrong tools or recovery operations exhaust their budget), while the 50-step setting tests whether additional computation enables recovery from mistakes. Temperature is set to 1.0 for all models, encouraging exploration. The RAG-based tool filtering does not count against the step budget.

• Cross-validation / statistical protocol. No formal cross-validation is used — the benchmark is a fixed test set of 361 tasks that all models are evaluated on, similar to standard computer-use benchmarks like OSWorld. Instead, the paper uses three-run averaging: each model is evaluated three times under each configuration (GUI-only, GUI+MCP, each at both step limits), and the results in Table 1 are the averages over these three runs. The paper explicitly notes that multiple runs are needed "due to fluctuations in the experimental results," which is expected given the stochasticity from temperature = 1.0 and inherent variability in GUI grounding. Error bars or confidence intervals are not reported.

Main Quantitative Results

Overall Accuracy Gains from MCP Tool Access

The central headline result (Table 1) is that MCP tools improve task accuracy for nearly all tested models at both step limits, with the magnitude of improvement varying substantially across models. At 15 steps:

• Gemini-2.5-Pro shows the largest absolute gain: accuracy jumps from 7.4% (GUI-only) to 20.5% (GUI+MCP), an increase of 13.1 percentage points. ACS correspondingly drops from 13.8 to 11.4, indicating the agent completes tasks more efficiently when tools are available.
• OpenAI o3 shows the second-largest gain: from 8.3% to 20.4%, an increase of 12.1 percentage points (ACS drops from 14.0 to 11.6).
• Claude 4 Sonnet, already the strongest GUI-only performer at 30.2%, improves to 35.3% (a 5.1 point gain, ACS drops from 11.9 to 10.4).
• Seed1.5-VL improves from 27.9% to 32.0% (4.1 points, ACS from 10.9 to 10.2).
• Qwen3-VL-Plus improves from 25.4% to 31.3% (5.9 points, ACS from 11.6 to 10.5).
• Qwen2.5-VL-72B-Instruct is the outlier: accuracy improves only marginally from 11.4% to 13.1% (0.5 points), but ACS increases from 13.0 to 13.5 — meaning the agent actually takes more steps on average when tools are introduced.

At 50 steps, the same pattern holds with larger absolute numbers. Claude 4 Sonnet remains the strongest, reaching 43.3% with tools vs. 40.1% without (3.2 point gain, ACS drops from 24.7 to 20.1). Gemini-2.5-Pro reaches 27.2% with tools vs. 13.3% without (13.9 point gain). Qwen2.5-VL-72B-Instruct's ACS still increases — from 30.5 to 37.2 — the only model where tools make the agent less efficient in terms of steps taken.

The multi-agent framework Agent-S2.5 achieves the highest absolute accuracy: 42.1% at 15 steps with tools (vs. 36.7% GUI-only, a 5.4 point gain) and 49.5% at 50 steps with tools (vs. 47.1% GUI-only, a 2.4 point gain). However, the relative gain over GUI-only is smaller than for some end-to-end models, likely because Agent-S2.5 already performs strongly in the GUI-only setting, leaving less room for tools to provide additional benefit.

Finding 1 as stated by the paper: "MCP tools significantly enhance LMMs' performance in computer-use, improving accuracy and reducing completion steps for most models. The effectiveness varies across different LMMs, indicating disparities in tool utilization capabilities."

Tool-Beneficial vs. Non-Tool-Beneficial Task Performance

The paper's task-type split (Table 1) reveals that accuracy gains are concentrated on Tool-Beneficial Tasks, where tools provide genuine efficiency shortcuts, while Non-Tool-Beneficial Tasks show small or negligible changes.

At 15 steps, on Tool-Beneficial Tasks:

• Gemini-2.5-Pro improves from 6.3% to 24.9%, an 18.6 point gain — the largest per-task-type improvement in the entire evaluation. The corresponding TIR for these tasks is 20.4%, and ACS drops from 13.7 to 10.2.
• OpenAI o3 improves from 8.5% to 25.4%, a 16.9 point gain (TIR: 21.3%, ACS: from 13.8 to 10.5).
• Claude 4 Sonnet improves from 29.0% to 35.6%, a 6.6 point gain (TIR: 29.7%, ACS: from 11.8 to 9.8).
• Qwen2.5-VL-72B-Instruct is again anomalous: accuracy improves only from 10.1% to 14.7% (4.6 points), but ACS increases from 13.0 to 15.8.

On Non-Tool-Beneficial Tasks, accuracy changes are generally small:

• Gemini-2.5-Pro shows a marginal accuracy increase from 9.8% to 9.9% (0.1 points), but ACS decreases from 14.1 to 13.9 — tools may help the agent eliminate irrelevant solution paths even when the tools themselves aren't used.
• OpenAI o3 shows accuracy increasing from 7.8% to 9.1% (1.3 points) with ACS decreasing from 14.4 to 14.1.
• Claude 4 Sonnet increases from 32.9% to 34.5% (1.6 points) with ACS decreasing from 12.0 to 11.7.

The paper attributes these small Non-Tool-Beneficial gains to two factors: (1) some tools, while not substantially more efficient than GUI paths, can still make task completion easier or more reliable; and (2) the presence of tools can help the model rule out irrelevant solution paths, making it faster to identify the correct GUI approach.

Tool Invocation Rate Analysis

The most diagnostically important results in Table 1 are the Tool Invocation Rate (TIR) values, which are remarkably low across all models — even the strongest performers.

At 15 steps:

• Claude 4 Sonnet achieves the highest TIR at 30.0%, meaning that on only 30% of all tasks did the agent make the appropriate tool-use decision AND complete the task successfully.
• Seed1.5-VL follows at 25.1%.
• Qwen3-VL-Plus at 24.5%.
• OpenAI o3 at 16.7%.
• Gemini-2.5-Pro at 16.8%.
• Qwen2.5-VL-72B-Instruct at just 10.9%.

At 50 steps, TIR values increase but remain low:

• Claude 4 Sonnet reaches 36.3% — still meaning that on nearly two-thirds of tasks, the agent either used the wrong modality or failed entirely.
• Agent-S2.5 reaches 35.3%.
• Seed1.5-VL at 29.0%.
• Qwen3-VL-Plus at 29.5%.
• Gemini-2.5-Pro at 21.5%.
• OpenAI o3 at 21.0%.
• Qwen2.5-VL-72B-Instruct at 10.9%.

The gap between accuracy and TIR is informative. For Claude 4 Sonnet at 50 steps, accuracy is 43.3% but TIR is 36.3% — a 7-point gap. This gap represents tasks where Claude succeeded but used the "wrong" action type (e.g., completing a Tool-Beneficial task through slow GUI operations, or completing a Non-Tool-Beneficial task after invoking a tool) — succeeding despite suboptimal modality choices.

Finding 2 as stated by the paper: "Tool Invocation Rate (TIR) positively correlates with task accuracy, but its relationship with ACS is complex and non-linear, suggesting that the impact of tool use on efficiency depends on various factors including task difficulty and model-specific strategies."

This correlation is visualized in Figure 5a, which plots TIR, accuracy, and ACS for each model across different configurations. The positive correlation between TIR and accuracy is visible, but the relationship with ACS is more nuanced — in some configurations, higher TIR coincides with lower ACS (more efficient completion), while in others the relationship is weaker or even reversed, depending on task type and step limit.

Impact of Number of Available Tools on Performance

The paper conducts a fine-grained analysis (Figure 5b) using Claude 4 Sonnet, grouping tasks by the number of available tools (manually annotated per task, as described in Section 3.3). The results reveal a non-monotonic relationship between tool availability and agent performance.

For the GUI-only configuration, as the number of available tools increases, accuracy tends to decrease and ACS tends to increase. This confirms that tasks requiring more tools are inherently harder — even when the agent cannot use tools, the underlying task complexity is reflected in lower GUI-only success rates.

For the GUI + MCP configuration, the relationship is more complex:

• When the number of available tools is small (0-1), the tool configuration shows clear accuracy gains over GUI-only and reduced ACS — the agent can identify and correctly use the relevant tools.
• When the number of available tools is large (>4), both accuracy and TIR drop sharply, and ACS rises. The paper attributes this to the difficulty of selecting the correct tool(s) from many plausible candidates — a "needle-in-haystack" problem where tool selection, not tool execution, becomes the bottleneck.

A crucial observation from Figure 5b: in cases where ACS is similar between GUI-only and GUI+MCP configurations, the GUI+MCP configuration sometimes yields lower accuracy. This indicates a composition failure: for tasks requiring many tool calls, the agent's tool-invocation strategy becomes counterproductive, and the simpler GUI-only approach (while slower) is more reliable.

Finding 3 as stated by the paper: "MCP tools generally improve performance in complex tasks, but their efficacy diminishes in extremely complex scenarios requiring tool combinations. This indicates that combining multiple tools is more challenging than combining GUI operations."

Efficiency Gains: Average Completion Steps

The ACS metric (Table 1) provides the efficiency dimension of performance. At 15 steps:

• Seed1.5-VL achieves the lowest ACS with tools (10.2), down from 10.9 in the GUI-only setting — a 6.4% reduction.
• Claude 4 Sonnet achieves 10.4 with tools, down from 11.9 — a 12.6% reduction.
• OpenAI o3 achieves 11.6 with tools, down from 14.0 — a 17.1% reduction, the largest relative ACS improvement at 15 steps among end-to-end models.
• Qwen2.5-VL-72B-Instruct is the only model where ACS increases with tools (13.0 to 13.5), indicating the agent wastes steps on incorrect tool invocations or recovery operations.

At 50 steps, the ACS reductions are more dramatic in absolute terms:

• Claude 4 Sonnet ACS drops from 24.7 to 20.1 (18.6% reduction).
• OpenAI o3 ACS drops from 44.8 to 32.1 (28.3% reduction).
• Gemini-2.5-Pro ACS drops from 40.3 to 29.7 (26.3% reduction).
• Qwen2.5-VL-72B-Instruct ACS increases from 30.5 to 37.2 (22.0% increase), further confirming its anomalous tool-usage behavior.

The ACS improvements on Non-Tool-Beneficial Tasks (Table 1) are particularly informative. For example, Gemini-2.5-Pro at 15 steps shows ACS decreasing from 14.1 to 13.9 on Non-Tool-Beneficial tasks, despite accuracy barely changing (9.8% to 9.9%). This suggests that even when tools are not directly useful, knowing what tools exist helps the agent narrow its search and identify the correct GUI path more quickly.

Ablation Studies and Robustness Checks

All ablation experiments use Gemini-2.5-Pro, chosen because it showed the largest accuracy gain from MCP tools over the GUI-only setting. Results are reported in Table 2.

Impact of removing RAG-based tool filtering (w/o Tools RAG): In the default configuration, tools are filtered by RAG to show only those relevant to the currently active application (~20-25 tools per step). Removing this filtering and showing all 158 tools at every step causes a substantial performance degradation. Overall accuracy drops from 20.5 to 15.5 (a 5.0 point decline). For Tool-Beneficial Tasks specifically, accuracy drops from 24.9 to 18.0 (6.9 points) and TIR drops from 20.4 to 12.4 (8.0 points). ACS on Tool-Beneficial Tasks decreases from 10.2 to 8.7, but this likely reflects the agent making fewer tool-invocation attempts overall rather than becoming more efficient. For Non-Tool-Beneficial Tasks, accuracy remains unchanged (9.9 in both cases), but TIR increases from 8.7 to 9.9 — the agent becomes slightly more inclined to use GUI rather than tools. The paper attributes the overall decline to "excessively long tool contexts, which markedly reduce the model's tendency to use tools, thereby impairing accurate tool invocation." This is consistent with a form of choice paralysis: when faced with 158 options, the agent is less likely to invoke any tool at all.

Impact of tool description ordering (w/ Tools Shuffle): In the default configuration, tool descriptions are presented in alphabetical order. Randomly shuffling the description order actually improves overall accuracy from 20.5 to 22.7 (a 2.2 point increase). This is a counterintuitive finding — one might expect random ordering to introduce noise that degrades performance, but the opposite occurs. The improvement is concentrated on Non-Tool-Beneficial Tasks, where accuracy jumps from 9.9 to 18.0 (8.1 points) and TIR increases from 8.7 to 17.1. For Tool-Beneficial Tasks, accuracy changes only slightly (24.9 to 24.7, a 0.2 point decline) and TIR drops from 20.4 to 19.2. The paper hypothesizes that "with fewer tools, the model tends to invoke the corresponding tool when one is available, whereas in the absence of available tools, the description order may implicitly suggest alternative solution strategies." In other words, when a task has no directly applicable tool, the alphabetical ordering might hide useful tools later in the list, while random ordering occasionally brings them to the forefront. Despite the performance improvement from shuffling, the paper maintains alphabetical ordering as the default for "consistency in evaluation" — a design choice that prioritizes reproducibility over maximizing absolute performance.

Model comparison across configurations: The ablation confirms that Gemini-2.5-Pro's tool-use capability is sensitive to how tools are presented. The drop from 20.5 to 15.5 when RAG is removed is substantial (24.4% relative decline), indicating that context management through RAG is critical for enabling tool use. The increase from 20.5 to 22.7 when tool order is shuffled suggests that prompt engineering (specifically, the ordering of available actions) can meaningfully affect agent performance independent of underlying model capability — a finding that complicates the interpretation of benchmark results and suggests that reported accuracies may not reflect the maximum achievable performance for a given model.

Multi-run consistency: The paper reports three-run averages but does not provide variance estimates (standard deviations, confidence intervals, or min-max ranges) across runs. This makes it difficult to assess whether the differences between models — particularly for models with similar accuracy (e.g., Seed1.5-VL at 32.0% vs. Qwen3-VL-Plus at 31.3% at 15 steps with tools) — are statistically reliable or within the noise floor of run-to-run variation.

Critical Assessment

The experimental design provides support for the paper's central claims, but with important boundary conditions and methodological limitations that affect the strength and generalizability of the conclusions.

Claim: MCP tools generally improve task success rates. Supported with qualifications. Table 1 shows accuracy increases for seven of eight evaluated systems at both step limits when tools are introduced. However, the magnitude of improvement varies enormously — from +13.1 points (Gemini-2.5-Pro at 15 steps) to essentially zero (Qwen2.5-VL-72B-Instruct at 15 steps). The "generally" qualifier is accurate: the claim holds for most but not all models. More importantly, the claim holds only for Tool-Beneficial Tasks, where the accuracy gains are concentrated. On Non-Tool-Beneficial Tasks, gains are minimal (typically 0-2 points), and for Qwen2.5-VL-72B-Instruct, ACS actually worsens with tools. The heterogeneity of effect across models and the existence of a clear negative case (Qwen2.5-VL) suggest that tool invocation is not a uniformly beneficial capability — it is a skill that some models possess and others lack, and providing tools to a model without that skill can actively harm performance by increasing ACS without compensatory accuracy gains. The paper's own data thus demonstrates that simply "adding MCP tools" is not a guaranteed improvement strategy; it works only for models with sufficient tool-selection and tool-execution competence.

Claim: Even the strongest models have relatively low tool invocation rates, indicating room for improvement. Strongly supported. Claude 4 Sonnet's TIR of 36.3% at 50 steps — the highest recorded — means that on 63.7% of tasks, the agent either failed to use tools when appropriate, used tools when inappropriate, or failed entirely. For the worst-performing model (Qwen2.5-VL-72B-Instruct), TIR is 10.9%, meaning appropriate tool-use behavior on barely one in ten tasks. These numbers are robust across the three-run averaging and are not sensitive to minor configuration differences. The finding that TIR is universally low — even for models that show substantial accuracy gains from tools — is the paper's most diagnostic result and the strongest evidence that tool-invocation decision-making is a distinct and currently underdeveloped capability in LMMs.

Claim: Composing multiple tools remains a significant challenge. Supported with qualifications. The evidence for this claim comes primarily from Figure 5b, which shows accuracy declining as the number of available tools increases beyond 3. However, this analysis is conducted on a single model (Claude 4 Sonnet) and does not decompose the failure into its constituent parts — is the failure due to incorrect tool selection (picking the wrong tool from many options), incorrect tool parameterization (picking the right tool but with wrong arguments), or incorrect sequencing (knowing which tools to use but in the wrong order)? The paper asserts that "combining multiple tools is more challenging than combining GUI operations" based on the observation that GUI-only sometimes outperforms GUI+MCP when ACS is similar, but this is a between-configuration comparison on different task subsets, not a controlled experiment that isolates tool composition difficulty from tool selection difficulty. A stronger test would compare tasks requiring N sequential tool calls against tasks requiring N sequential GUI operations of equivalent abstract difficulty, which would require a more controlled task design than OSWorld-MCP provides.

Methodological limitation: no confidence intervals or statistical significance testing. The paper reports three-run averages but no variance estimates. Given the stochasticity introduced by temperature = 1.0 and the inherent variability in GUI grounding, run-to-run variance could be substantial. For closely matched models (e.g., the 0.7-point gap between Seed1.5-VL at 32.0% and Qwen3-VL-Plus at 31.3% at 15 steps with tools), it is impossible to determine from the reported data whether the difference is real or within the noise floor. This is a non-trivial omission for a benchmark paper where relative model ranking is a primary output.

Methodological limitation: single agent framework bias. All models are evaluated using the GUI-Owl agent configuration. The paper acknowledges this "may lead to some performance fluctuations for certain models under the original OSWorld configuration," but it does not quantify this effect. A model that performs well under GUI-Owl may perform poorly under a different agent framework (and vice versa). Since the paper's goal is to establish a "fair comparison" of tool-invocation capabilities, the choice of agent framework is critical — if GUI-Owl systematically advantages certain model architectures over others, the fairness claim is compromised. The paper would be strengthened by evaluating at least one model under multiple agent frameworks to establish the sensitivity of results to this choice, but no such analysis is performed.

Methodological limitation: tool generation and evaluation share the same model family. The tools were generated using OpenAI o3, and one of the evaluated models is also OpenAI o3. This creates a potential distributional confound: the tools were produced by the same model (or model family) that is being tested, which could give o3 an advantage in understanding tool descriptions or predicting tool behavior because they match its own generation patterns. The 25 non-target tools and the manual curation process partially mitigate this concern (o3 does not show anomalously high TIR — at 16.7% at 15 steps, it is comparable to Gemini-2.5-Pro at 16.8% and lower than Claude 4 Sonnet at 30.0%), but the confound exists and is not addressed in the paper.

Missing ablation: varying the number of tools per RAG filter. The paper ablates the extreme case — RAG on vs. RAG off (all 158 tools) — but does not explore intermediate tool-set sizes. How does performance change if the RAG filter returns 5 tools vs. 20 vs. 50? Is there an optimal tool-set size per application? This would provide practical guidance for deploying tool-augmented agents and would help characterize the tool-selection difficulty more precisely than the binary RAG-on/off comparison.

Missing baseline: tool access without GUI access. The paper compares GUI-only vs. GUI+MCP but does not evaluate a tool-only baseline (where the agent can only invoke tools, with no GUI operations available). Such a baseline would reveal how much of the accuracy gain comes from tools replacing GUI operations entirely vs. tools supplementing GUI operations. If tool-only performance were similar to GUI+MCP, it would suggest that the agent's GUI capabilities are not providing additional value beyond tools — a finding with significant practical implications for agent architecture.

Missing analysis: per-application breakdown. All results are reported aggregated across the 7 applications. Performance likely varies substantially by application — tools for LibreOffice Calc (which has the largest tool set at 24% of the total) may be used differently than tools for VS Code or Chrome. A per-application breakdown would reveal whether certain applications are more amenable to tool-based interaction and whether model rankings are consistent across applications, but no such breakdown is provided.

Missing analysis: step-level tool invocation success rate. TIR measures whether a tool was invoked appropriately at the task level, but it does not measure whether individual tool calls succeeded. An agent might invoke the correct tool but with wrong parameters (the tool call executes but produces an incorrect result), which the task-level evaluation would catch only if the error propagates to final state failure. A step-level metric — what fraction of attempted tool calls achieve their intended effect? — would provide finer-grained diagnostic information about whether failures stem from tool selection or tool parameterization. The case studies (e.g., Figure 9, where the agent calls copy_cells_between_sheets four times, suggesting earlier attempts failed) hint that per-call failure rates may be high, but this is not systematically quantified.

Scope limitation: single evaluation environment. All experiments are conducted in the OSWorld VM environment (Ubuntu). The paper does not test whether the relative tool-invocation capabilities of different models generalize across operating systems (e.g., Windows, macOS, as supported by OSWorld) or across different VM configurations. If a model's tool-use behavior is sensitive to the specific visual appearance or application versions in the OSWorld Ubuntu image, the benchmark results may not transfer to other deployment environments.

Scope limitation: task set is OSWorld-derived, not independently designed for tool evaluation. The 250 Tool-Beneficial Tasks are a subset of the original 369 OSWorld tasks, which were designed to test GUI interaction, not tool invocation. While the paper's tool generation pipeline ensures that tools exist for these tasks, the tasks themselves were not designed to systematically vary properties relevant to tool use — number of tool calls required, tool selection difficulty, parameter complexity, interdependence between tool calls, or need for GUI-tool interleaving. A benchmark purpose-built for evaluating tool invocation might include controlled difficulty gradations along these axes, enabling more precise diagnosis of where models succeed and fail. OSWorld-MCP inherits the task distribution of OSWorld, which may not optimally sample the space of tool-use challenges.

6. Limitations and Trade-offs

1. Difficulty Estimation Is Prohibitively Expensive and Not Included in the Compute Budget

The assumption or constraint. The centerpiece of the paper's approach — the compute-optimal allocation policy that selects the best test-time strategy per difficulty bin — depends entirely on knowing each prompt's difficulty before solving it. The paper estimates difficulty by sampling 2048 complete solutions from the base model and either computing the pass@1 rate (oracle) or averaging the PRM's final-answer score (predicted). The authors are transparent about the tension:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence. The reported 4× efficiency gains over best-of-N — the paper's headline practical result — are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost is difficulty_estimation + strategy_execution. Since difficulty estimation uses 2048 samples per prompt (roughly 8× the largest test-time budget studied, which maxes out at 256 generations), the estimation cost can dominate the total compute. An agent that spends 2048 samples learning that a problem is "easy" and then saves 48 generations by using the optimal strategy has not saved any compute at all — it has spent far more than it saved. The 4× figure is therefore an upper bound on achievable efficiency, representing what could be achieved if difficulty were known for free, not what any practical system can actually deliver.

The exploration-exploitation framing the paper offers (Section 3.2) — "compute spent assessing difficulty versus compute spent solving the problem" — correctly identifies the tradeoff but does not resolve it. The difficulty estimation cost is a fixed overhead that consumes the budget before the "optimal" strategy is even deployed, making the policy selection circular: you need to spend compute to learn which strategy is best, but the best strategy depends on how much compute remains after estimation, which depends on which strategy you planned to use, which depends on the difficulty you haven't yet estimated.

What evidence exists in the paper. The cost is never included in any budget calculation, any figure, or any table. The paper does not report what fraction of total compute the difficulty estimation represents at any budget level. The ablation comparing oracle vs. predicted difficulty bins (Figures 4, 8) shows that predicted bins work nearly as well as oracle bins for strategy selection, which addresses the ground-truth-access concern but not the cost concern — the predicted difficulty method still requires 2048 samples and PRM scoring per prompt, which is equally expensive. No experiment varies the number of difficulty-estimation samples to find the minimum viable estimation budget.

Mitigation status. The paper explicitly flags this as future work (Section 8) — "pretraining or finetuning models to directly predict difficulty of a question" — but no such model is developed or evaluated. The paper also suggests adaptive difficulty estimation (start with a few samples, assess difficulty, allocate remaining budget), but this is not explored. The limitation is acknowledged but entirely unresolved in the current work, meaning the paper's primary practical claim (4× improved efficiency) is, in its current form, a theoretical upper bound rather than a demonstrated deployment gain.

---

2. Hard Problems Remain Completely Unsolved — Test-Time Compute Cannot Create Capability Where None Exists

The assumption or constraint. The paper's entire framework assumes that the base model produces correct solutions at some non-trivial rate — the proposal distribution must contain successes for either search or revisions to find and amplify them. This assumption is not hidden; it is explicitly confirmed by the data: for the hardest difficulty quintile (bin 5), the base model's pass@1 is essentially zero, and no amount of test-time compute changes this.

The consequence. The paper demonstrates a clean capability boundary: test-time compute can amplify existing capability but cannot create it from nothing. Across all methods — beam search, best-of-N weighted, sequential revisions, and their compute-optimal combinations — bin 5 accuracy remains at roughly 1–3% regardless of budget, in sharp contrast to bins 1–4 where accuracy improves with compute (Figure 3, right; Figure 7, right). In the FLOPs-matched comparison, bin 5 is the regime where pretraining a larger model is almost always preferable to test-time compute with the smaller model, and for PRM search at high inference-to-pretraining ratios (R ≫ 1), the disadvantage is dramatic: "Hard questions show a −52.9% relative disadvantage from using test-time compute instead of the larger model" (Figure 9, Section 7). In plain terms: if the base model genuinely cannot solve a class of problems — because it lacks the necessary knowledge, reasoning patterns, or representational capacity learned during pretraining — then scaling test-time compute offers essentially zero benefit, and the only path forward is to train a larger or more capable model.

This has a direct practical implication that the paper's framing does not fully highlight: the compute-optimal strategy requires knowing whether a problem falls into bin 5 (unsolvable regardless of budget) or bin 4 (solvable with enough compute). Misclassifying a bin-5 problem as bin 4 and spending a large compute budget on it is not just inefficient — it is a complete waste, because no amount of search or revision will produce a correct answer. The difficulty estimator must therefore be reliable at distinguishing the "unsolvable" regime from the "solvable with effort" regime, which is likely harder than distinguishing among the solvable regimes, and the paper provides no analysis of this distinction.

What evidence exists in the paper. Figure 3 (right panel) shows bin 5 at near-zero accuracy for all budgets and methods. Figure 7 (right) shows bin 5 at ~2–3% regardless of sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling curves essentially flat near 0–5%, well below the larger model's performance at all R values. The paper itself states this finding clearly (Section 7):

"test-time compute provides essentially zero benefit on the hardest problems regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time"

Mitigation status. The paper does not attempt to solve this — it cannot, because the limitation is inherent to the approach rather than a missing component. The authors are transparent about it, which is appropriate, but there is no mitigation strategy beyond "use a larger model for hard problems." The limitation is fundamental: test-time compute operates on the output distribution of a fixed model, and if that distribution contains no correct answers, no search or revision strategy can find one. This is a hard boundary on the applicability of test-time compute scaling, and it means the approach is best suited for deployment regimes where the problem distribution is known to be within the base model's capability range — a constraint that may be difficult to verify in practice without the kind of expensive difficulty estimation the framework already relies on.

---

3. The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate and Brittle Training Dynamics

The assumption or constraint. The paper trains a revision model that generates iterative improvements to its own answers by fine-tuning on trajectories where 0–4 incorrect answers precede a correct one. The training data contains only incorrect-to-correct transitions — the model never sees examples where the current answer is already correct and should be preserved. The paper reports a significant practical consequence:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

The consequence. The revision model has no learned concept of "stop revising when the answer is already good." At each step of the revision chain, the model is conditioned on previous answers and will produce a new answer regardless of whether the current one is correct. This means that a long revision chain does not monotonically improve — it oscillates, with correct answers later being "revised" into incorrect ones. The paper mitigates this by selecting the best answer from anywhere in the chain using majority voting or verifier-based scoring, but this mitigation is inherently wasteful: the model spends computation generating revisions that actively degrade quality, and the selection mechanism must detect and discard these degraded outputs. In the worst case, a revision chain that starts with a correct answer and then degrades through subsequent revisions will produce lower-quality outputs than simply taking the first answer and stopping.

The brittleness extends beyond the reversion problem. The ReST^EM experiment (Appendix K, Figure 16) attempts to further optimize the revision model using reinforcement learning, and the result is catastrophic: "additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio." The paper hypothesizes that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This is a notable negative result: the revision model's training is fragile, and seemingly reasonable attempts to improve it can backfire. The positive results depend on specific (and somewhat unusual) training choices — offline data construction with edit-distance-based incorrect-correct pairing, and a particular checkpoint selection strategy that stops "slightly after the point where validation loss begins increasing."

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The ReST^EM failure is documented in Appendix K and Figure 16. The paper's own within-chain selection mechanism (majority voting or best-of-N weighted across all revisions) is an implicit acknowledgment that individual revision steps are unreliable — the system cannot trust the final revision to be the best one.

Mitigation status. Partial mitigation through within-chain selection, but no principled solution. The paper uses majority voting or verifier scoring to select the best answer from the entire revision chain rather than taking the last output, which prevents the specific failure mode where a correct answer is overwritten by an incorrect revision. However, this is a post-hoc filter, not a fix for the underlying model behavior. A more principled solution — training the model to recognize correct answers and decline to revise them, or incorporating a stopping criterion into the revision policy — is not explored. The ReST^EM results suggest that the training procedure is not well-understood, and even the authors do not have a reliable recipe for improving revision models beyond their initial supervised fine-tuning. This means that practitioners adopting the revision approach must either (a) accept the 38% reversion rate and rely on selection mechanisms, (b) invest in extensive trial-and-error to find a training procedure that works for their model and task distribution, or (c) forego revisions in favor of simpler but less effective parallel sampling strategies.

---

4. The FLOPs-Matched Comparison Uses a Weakened Pretraining Baseline — Scaling Parameters Only, Not Data, with Greedy Decoding

The assumption or constraint. When comparing test-time compute with the smaller model against pretraining a larger model, the paper scales only model parameters (approximately 14×) while keeping training data fixed. The authors acknowledge this choice:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

This follows the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm (Hoffmann et al., 2022), where total compute is split optimally between data scaling and parameter scaling. Additionally, the larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no search, no revisions. The smaller model, by contrast, gets the full compute-optimal test-time strategy suite.

The consequence. The pretraining baseline is weaker than it could be on at least two fronts. First, a model trained with compute-optimal scaling of both parameters and data would likely achieve higher performance for the same total FLOPs budget, making the test-time compute advantage appear smaller. Second, giving the larger model even a modest test-time compute budget — e.g., a best-of-8 or best-of-16 with majority voting, which would use a fraction of the inference budget the paper allocates to the smaller model's 256-generation search — would create a much stronger baseline. The paper's comparison essentially asks: "Can test-time compute with a small model beat a large model that is not using its inference budget optimally?" That is an interesting question, but it is not the headline the paper reports — the paper presents the result as a general statement about test-time compute substituting for pretraining, when it is more precisely a statement about test-time compute with an optimized small model beating a non-optimized large model.

The specific numbers reported (e.g., "+27.8% relative improvement on easy-medium questions at R ≪ 1") should be interpreted as upper bounds on the advantage of test-time compute over pretraining. Against a properly compute-optimal large model with even a modest test-time budget, the advantage would likely shrink or potentially reverse.

What evidence exists in the paper. The paper reports this limitation transparently in Section 7 but does not ablate it — no experiment compares against a compute-optimally trained large model or against a large model with any form of test-time compute augmentation. The choice of greedy decoding for the large model is mentioned in Section 7 but not prominently flagged as a limitation.

Mitigation status. The paper acknowledges this as future work but does not address it in the current study. The transparent reporting is commendable, but the limitation directly affects the strength of the paper's most prominent comparative claim — that test-time compute can outperform a 14× larger model. A more conservative statement would be that test-time compute with an optimized small model can match or exceed a particular non-compute-optimal large model under specific inference-to-pretraining ratio conditions, which is a more limited but fully supported claim.

---

5. Search and Revisions Are Studied Independently — Their Complementary Strengths Are Never Combined

The assumption or constraint. The paper studies two complementary axes for improving test-time compute — PRM-guided search (which selects better among generated candidates) and iterative revisions (which generate better candidates in the first place) — but treats them as entirely separate pipelines. The paper acknowledges this gap explicitly:

"we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8)

The consequence. The paper's analysis shows that these two mechanisms have complementary difficulty-dependent strengths: revisions excel on easy problems where local refinement of nearly-correct answers suffices, while search excels on medium-hard problems where exploring qualitatively different solution strategies is needed. The natural next step — using the revision model as the proposal distribution within PRM-guided beam search, or using the PRM to score revision steps and guide which revisions to pursue — is never implemented or evaluated. This means the paper's reported results represent a lower bound on what a fully integrated system could achieve, and the compute-optimal policy (which selects between search and revisions based on difficulty) is selecting between incomplete strategies when a combined strategy might outperform either.

This is a consequential omission because the two mechanisms operate on different parts of the pipeline: revisions improve the generator, search improves the selector. In principle, they should compose — better generation increases the quality of candidates that search can select from, and better selection increases the chance that a good revision in a chain gets recognized and used. The paper's framework of "proposal distribution vs. verifier" (Section 2) naturally suggests that combining them should yield gains beyond either alone, but the paper cannot quantify those gains because the combination was never tested. This leaves the central question — what is the full potential of test-time compute when both mechanisms are jointly optimized? — unanswered.

The specific failure modes the paper documents for each mechanism may also interact. For example, search suffers from PRM over-optimization on easy problems (Figure 3, right), and revisions suffer from correct-to-incorrect reversion (Section 6.1). If these mechanisms were combined, would the over-optimization problem worsen (because better proposals make it easier to exploit the PRM) or improve (because revisions keep the proposals closer to the base distribution, reducing exploitation)? The paper provides no evidence either way, which means a practitioner attempting to combine these methods would be operating without guidance on how the failure modes interact.

What evidence exists in the paper. None — the absence of combined experiments is the limitation. The paper's ablation studies (Appendices E, F, J, K) examine variants within each mechanism independently but never cross the boundary between search and revisions. The compute-optimal policy selects between them but never blends them.

Mitigation status. The paper explicitly flags this as future work, and the framework it provides (proposal vs. verifier, difficulty-dependent allocation) provides a clear conceptual path toward combination. However, the implementation is non-trivial: the PRM was trained on base model outputs and does not transfer well to revision model outputs (Figure 15a), meaning a combined system would either need a new PRM trained on revision trajectories or a method for calibrating the base-model PRM to revision-model outputs. Similarly, beam search with revisions as the proposal distribution requires the revision model to generate coherent continuations from partial solutions — a capability it was not trained for, since revision training used only complete-solution trajectories. These are solvable engineering challenges, but they are not solved in the current work, and the paper's empirical conclusions about the limits of test-time compute — particularly the difficulty boundaries and the over-optimization ceiling — may shift once combined mechanisms are tested.

---

6. All Experiments Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*) — the Findings May Be Task- and Model-Specific

The assumption or constraint. Every experiment in the paper — difficulty-dependent scaling patterns, PRM training, revision model behavior, FLOPs-matched comparisons, and compute-optimal policy selection — is conducted on the MATH benchmark (500 test questions, competition-level mathematics) using PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not validated.

The consequence. Several of the paper's key findings could be specific to the interaction between PaLM 2-S*'s particular capabilities and mathematical reasoning tasks, rather than general properties of test-time compute scaling. Consider:

• The PRM over-optimization pattern (beam search degrading on easy problems) depends on the PRM's error distribution — which errors it makes, how those errors correlate with search depth, and how the base model responds to PRM guidance. A different base model with different calibration properties or different error patterns might exhibit different over-optimization thresholds or even different difficulty-dependent search behavior entirely.

• The revision model's behavior depends on the base model's in-context learning and self-correction capabilities, which vary substantially across model families. PaLM 2's specific few-shot and instruction-following characteristics may not generalize to models with different training objectives or architectures.

• MATH as a domain is narrow — competition-level algebra, geometry, number theory, and precalculus problems that require multi-step symbolic reasoning with exact final answers. The difficulty-dependent patterns the paper identifies (beam search helps medium problems, revisions help easy ones, nothing helps hard ones) may not transfer to other reasoning domains (code generation, commonsense reasoning, scientific QA) or to tasks requiring factual knowledge rather than deduction from premises. MATH problems have a specific property that may be driving the results: the model either knows the relevant mathematical technique or doesn't. Test-time compute can help with applying known techniques reliably but cannot teach new techniques — this is exactly the pattern seen in the difficulty bins. For domains where "capability" is more continuous or where partial knowledge can be productively combined at inference time, the scaling patterns might look different.

• The FLOPs-matched comparison depends on the specific compute ratio R and the specific scaling behavior of PaLM 2 models. A different model family might have a different relationship between parameter count and MATH performance, which would change the pretraining-baseline comparison.

What evidence exists in the paper. None — there are no experiments on other benchmarks, other model families, or other task domains. The paper does not even compare PaLM 2-S* with a different model of similar scale (e.g., a GPT-class or LLaMA-class model) on the same MATH benchmark to establish whether the difficulty-dependent patterns are model-specific. The 500-question test set is split into five difficulty quintiles of ~100 questions each, and with two-fold cross-validation, strategy selection is based on roughly ~50 questions per fold per bin — the compute-optimal policy is selected and evaluated on small subsets, making it potentially sensitive to the specific composition of the MATH test set.

Mitigation status. The paper does not attempt to address this limitation through additional experiments. The authors' stated belief about representativeness is reasonable as a working assumption, but it is strictly an assumption. A single-benchmark, single-model-family evaluation is standard for systems papers introducing a new technique, but for a paper that presents its findings as general principles about test-time compute scaling — "compute-optimal test-time scaling," "the relationship between compute and performance is qualitatively different depending on problem difficulty" — the lack of cross-domain or cross-model validation weakens the generality claim. The paper's conclusions about difficulty-dependent optimal strategies, over-optimization thresholds, and the pretraining-vs-inference tradeoff should be understood as findings about PaLM 2-S* on MATH until replicated on other models and domains. The paper implicitly acknowledges the scope limitation by using language like "our results suggest" and "we observe," but does not explicitly flag the single-benchmark limitation as a threat to generalizability.

7. Implications and Future Directions

How This Work Changes the Landscape

OSWorld-MCP makes a methodological contribution that shifts how the field should think about evaluating computer-use agents: it establishes that tool invocation is not an action-space expansion but a decision-making capability that must be measured separately from both GUI execution and task success. This is not a paradigm shift — it does not propose new architectures or training methods — but it is a diagnostic reframing with practical consequences for how benchmarks are designed and how agent capabilities are compared.

The key reframing is this: prior to OSWorld-MCP, the field implicitly treated tool invocation as something you add to an agent, after which you measure whether overall accuracy improves. The paper's data shows why this is insufficient. Qwen2.5-VL-72B-Instruct receives tools and shows essentially no accuracy gain (+0.5 points at 15 steps) but its ACS increases by 4.4 steps — the tools actively harm its efficiency because it invokes them incorrectly (Table 1). Meanwhile, Gemini-2.5-Pro gains 13.1 accuracy points from tools but achieves only a 16.8% TIR, meaning on 83.2% of tasks it either used the wrong modality or failed entirely. These asymmetric patterns — tools helping some models, hurting others, and leaving all models with low TIR — cannot be captured by an accuracy-only evaluation. They require metrics that separately track whether the right action type was chosen and whether the choice led to success. The paper provides exactly these metrics through TIR and ACS alongside accuracy.

This reframing reconciles a latent contradiction in the emerging tool-augmented agent literature. Several recent systems (CoAct, ComputerRL) report that adding tool invocation improves performance, which might suggest that tool use is a mature capability ready for production deployment. OSWorld-MCP shows that this conclusion is premature: the improvements are real but the tool-use decision-making is unreliable. The low TIR values — even Claude 4 Sonnet reaches only 36.3% at 50 steps — mean that agents benefit from tools on the subset of tasks where they happen to choose correctly, but they lack the robust decision-making to use tools reliably across the full task distribution. The paper thus reframes the narrative from "agents can use tools to improve performance" to "agents can sometimes benefit from tools, but their ability to decide when and which tool to use is the primary bottleneck, and current performance on this decision-making dimension is poor."

This has direct consequences for which research directions become more attractive and which become less so. Before OSWorld-MCP, a natural research direction was to build larger, more comprehensive MCP tool sets — more servers, more APIs, more functions — on the assumption that broader tool coverage would lead to better agent performance. The paper's Finding 3 (performance declines when the number of available tools exceeds 3–4, Figure 5b) suggests this direction is misprioritized for the current capability level of LMMs. If agents cannot reliably select among even 20–25 application-filtered tools, expanding the tool set to hundreds or thousands adds complexity without addressing the bottleneck. The more productive research direction, indicated by the paper's results, is improving tool-selection decision-making — teaching agents to recognize when a tool is appropriate, to identify the correct tool from a candidate set, and to compose multiple tool calls when no single tool suffices. This is a training and architecture problem, not a tool-availability problem.

The paper also makes a more specific methodological contribution: it provides the first evidence that multi-tool composition is qualitatively harder for LMMs than GUI composition (Finding 3, Section 4.3), and it identifies a specific mechanism — the delayed feedback loop in tool calls versus the immediate visual feedback in GUI operations — that plausibly explains why. This directs research attention toward state-tracking across tool calls, an capability that is under-explored in current agent architectures but is critical for the multi-step tool-chaining tasks that Figure 4c shows are common in realistic computer-use scenarios.

Finally, the paper establishes a benchmark construction methodology (automated tool generation + execution-based filtering + consensus curation) that addresses the tool-tailoring validity problem — the risk that tools designed specifically for benchmark tasks will inflate performance estimates relative to real-world deployment. This methodology is transferable: as more tool-augmented benchmarks emerge, the principle of generating tools through task-agnostic pipelines and curating for generality rather than task-specificity will be important for maintaining evaluation validity. The paper's analysis showing that 131 of 158 tools were invoked at least once by tested models, and that 25 tools serve as non-optimal distractors, provides a validation template that future benchmarks can adopt.

Follow-Up Research This Work Enables

Training models to predict task-level tool applicability from visual state alone. The paper's difficulty-estimation analog in OSWorld-MCP is the classification of tasks into Tool-Beneficial versus Non-Tool-Beneficial — a distinction that currently requires manual annotation. A model that could, given a screenshot and task description, predict whether any available tool would substantially improve efficiency would enable adaptive tool-use policies: the agent could invest computation in tool selection only when tools are likely to help, and default to GUI operations otherwise. The paper's data provides the ground-truth labels for 361 tasks (250 Tool-Beneficial, 111 Non-Tool-Beneficial), which is sufficient to fine-tune a lightweight classifier. A strong follow-up would train such a classifier on a subset of tasks and evaluate whether agents using the classifier's predictions to gate tool access achieve higher TIR and lower ACS than agents that always have tools available, particularly on the Non-Tool-Beneficial tasks where tool access can distract or mislead. The key metric would be whether the classifier reduces false-positive tool invocations (using tools when they don't help) without suppressing appropriate tool use.

Step-level tool invocation success rate as a diagnostic metric. The paper's TIR operates at the task level — it counts whether a tool was invoked appropriately over the entire task — but does not measure whether individual tool calls succeeded. The case study in Figure 9 shows the agent calling copy_cells_between_sheets four times before achieving the correct state, suggesting that per-call failure rates may be high. A systematic study that instruments the OSWorld-MCP environment to record, for each tool invocation, whether the tool's postcondition was satisfied (e.g., did the cells actually get copied? Did the extension actually get installed?) would decompose the TIR gap into two components: tool-selection failures (choosing the wrong tool or modality) and tool-execution failures (choosing the right tool but specifying wrong parameters or encountering environment-specific edge cases). This decomposition would directly inform whether research should focus on improving the agent's decision-making (tool selection) or its grounding and parameter-binding (tool execution). The paper's infrastructure — deterministic tool behaviors in controlled VMs with state-based evaluation — makes this decomposition feasible because tool postconditions can be verified automatically by extending the existing evaluation scripts.

Controlled comparison of GUI-only, tool-only, and hybrid action spaces. The paper compares GUI-only against GUI+MCP but does not evaluate a tool-only baseline where the agent can invoke tools but cannot perform GUI operations. This is a critical missing comparison because it would reveal how much of the accuracy gain from tools comes from tools replacing GUI operations (which would show high tool-only performance) versus tools supplementing GUI operations (which would show tool-only performance dropping below GUI+MCP). If tool-only performance is close to GUI+MCP performance, it suggests that GUI capabilities add little marginal value once tools are available — a finding with significant practical implications for agent architecture (you might not need visual grounding at all for many tasks). If tool-only performance is substantially lower, it means that agents need visual perception to gather information before invoking tools (e.g., reading cell values from the screen to specify the source range for copy_cells_between_sheets), and the hybrid action space is genuinely necessary. A strong follow-up would run this experiment on the full OSWorld-MCP benchmark with the same models used in the paper, reporting accuracy, ACS, and a modified TIR for the tool-only setting.

Fine-grained difficulty characterization for tool-composition tasks. Figure 5b shows that performance declines as the number of available tools increases, but this analysis conflates several distinct difficulty dimensions: the number of tools needed to solve the task, the interdependence between those tools (must tool B be called with output from tool A?), the complexity of tool parameters (are parameters simple enums or complex ranges that must be read from the screen?), and the need for interleaved GUI operations between tool calls. A follow-up study that manually annotates the 250 Tool-Beneficial Tasks along each of these dimensions would enable a much more precise diagnosis of what makes tool composition hard. For instance, if tool-parameter complexity is the dominant predictor of failure, the bottleneck is visual grounding (extracting correct parameter values from the screen). If tool interdependence is the dominant predictor, the bottleneck is state-tracking and planning. The paper's task-level annotations (Section 3.3) provide a starting point — extending them with these additional dimensions would produce a resource that the field could use to benchmark progress on specific sub-capabilities of tool use.

Cross-model-family replication of the tool invocation rate gap. The paper's finding that even the strongest model achieves only 36.3% TIR is based on a specific set of LMMs evaluated in late 2025. As new model generations are released, tracking whether TIR improves — and whether it improves faster or slower than overall accuracy — would characterize whether tool-invocation decision-making is bottlenecked by general reasoning capability (in which case TIR should track accuracy gains from scaling) or by something more specific to tool-use training (in which case TIR might lag behind accuracy gains, or improve only in models that explicitly train on tool-use trajectories). A longitudinal benchmark study that evaluates each major new multimodal model on OSWorld-MCP upon release would provide this signal. The paper's standardized evaluation protocol — same tools, same tasks, same agent framework, same step limits — makes this feasible as an ongoing community benchmark.

Multi-agent architectures for tool-invocation specialization. The paper evaluates one multi-agent framework (Agent-S2.5), which achieves the highest absolute accuracy (49.5% at 50 steps) but does not dramatically outperform the best end-to-end model on TIR (35.3% vs. 36.3% for Claude 4 Sonnet). An open question is whether decomposing tool-invocation into specialized sub-agents — one agent that decides whether to use a tool, another that selects which tool, a third that specifies parameters, and a grounding agent that handles GUI operations — would improve TIR by allowing each agent to specialize in a narrower decision. The paper's metrics (TIR for modality choice, ACS for overall efficiency) provide the evaluation framework for testing such architectures. A strong follow-up would implement a modular agent where the tool-selection module is trained specifically on the Tool-Beneficial vs. Non-Tool-Beneficial distinction (using the paper's manual annotations as training data) and evaluate whether this specialized module achieves higher TIR than the monolithic models in Table 1, even if the constituent models are individually weaker.

Practical Applications and Downstream Use Cases

Standardized evaluation for MCP server developers. The MCP ecosystem is growing rapidly, with organizations building custom MCP servers for their internal tools, APIs, and workflows. Currently, there is no standard way to evaluate whether a given MCP server design — its tool descriptions, parameter names, error messages, and documentation style — is actually usable by LMMs. An MCP server developer could integrate their server into the OSWorld-MCP framework by replacing the paper's 158 tools with their own, running the same agent evaluation on the same tasks, and measuring whether their tool descriptions lead to higher TIR than the baseline. For example, a developer who hypothesizes that more verbose parameter descriptions will improve tool invocation could A/B test two versions of their server on OSWorld-MCP and measure the TIR difference. The paper's finding that random tool ordering improves accuracy by 2.2 points for Gemini-2.5-Pro (Table 2) suggests that seemingly minor description-engineering choices can meaningfully affect agent performance, and a standardized evaluation framework enables systematic optimization of these choices.

Tool-set curation for domain-specific computer-use agents. Organizations deploying computer-use agents for specific workflows — financial analysts working with spreadsheet models, video editors processing media files, developers configuring IDEs — need to decide which MCP tools to make available to their agents. The paper's analysis of tool invocation frequency (Figure 4a) and tool-availability distributions (Figure 4c) provides a methodology for this curation: run the agent on a representative task set, record which tools are actually invoked, and prune the tool set to those that are both frequently used and successfully applied. The paper's finding that performance degrades when more than 3–4 tools are available (Figure 5b) provides an evidence-based guideline: for current LMMs, a curated tool set of 20–25 per application (as achieved by the paper's RAG filtering) appears to be near the upper bound of manageable complexity. Organizations can use this benchmark to empirically determine the optimal tool-set size for their specific agent, model, and task distribution, rather than relying on intuition or exposing all available tools.

Diagnostic profiling of agent failures in production computer-use systems. When a computer-use agent fails in production — a customer support automation that cannot complete a refund form, a data-entry agent that populates the wrong spreadsheet cells — the operations team needs to know why it failed to decide what to fix. The paper's three-metric framework (Accuracy, TIR, ACS) provides a diagnostic template that can be applied to production logs. If an agent shows high accuracy but low TIR, it is succeeding through inefficient GUI paths and would benefit from better tool-selection training. If it shows high TIR but low accuracy, it is choosing the right action type but failing on execution — perhaps due to grounding errors or incorrect tool parameterization. If it shows low accuracy and low TIR on specific task categories, those tasks may be beyond the model's capability and require either a more capable model or task redesign. The paper's case studies (Figures 6–9) illustrate how step-level action traces can be analyzed to pinpoint specific failure modes (incorrect tool parameterization, premature termination, failure to recover from tool errors), providing a template that production monitoring systems can adopt.

When to Prefer This Method

The paper does not propose a method for solving tasks — it proposes a method for evaluating agents. The relevant tradeoff is not "when should I use OSWorld-MCP versus another technique" but rather "when should I use OSWorld-MCP versus another benchmark for evaluating my computer-use agent." The paper articulates this tradeoff implicitly through its design choices and its critique of prior benchmarks:

• Prefer OSWorld-MCP when: (a) your agent has or will have MCP tool invocation capabilities and you need to measure how effectively it chooses between GUI and tool paths, (b) you want to benchmark tool-invocation decision-making separately from GUI execution skill, which requires the TIR and ACS metrics that only OSWorld-MCP provides, (c) you need a fixed, curated tool set that enables apples-to-apples comparison across different agent architectures without tool-set confounds, or (d) you are deploying agents in real desktop environments (Ubuntu, Windows, macOS) where tasks involve production applications like LibreOffice, VS Code, and Chrome, and where some tasks benefit from programmatic tool access while others require visual GUI interaction.

• Prefer OSWorld (original, GUI-only) when: (a) your agent does not support tool invocation and you only need to measure GUI grounding and action sequencing, (b) you want to compare against the larger body of published results on the original benchmark, or (c) you specifically want to isolate GUI skill from tool-use skill for diagnostic purposes.

• Prefer text-based MCP benchmarks (MCPEval, MCP-Bench) when: (a) visual grounding and GUI interaction are not part of your agent's capability scope — for example, evaluating a pure LLM's function-calling ability on API-style tasks — or (b) you need a lightweight, fast evaluation that does not require VM infrastructure and can be run at scale in CI/CD pipelines. However, these benchmarks will not capture the hybrid decision-making and visual context-extraction challenges that OSWorld-MCP measures, so they should not be used as proxies for computer-use capability.