ArXiv: 2512.19432

🎯 Pitch

While current mobile agents ace existing tests with >90% success, MobileWorld slashes top models to a mere 51.7% by forcing them into messy, multi-app workflows and, crucially, asking for help when instructions are vague. The real kicker? Agents completely fail when they must combine GUI taps with external API calls, scoring near 0% on a novel class of hybrid tool-use tasks that mirror real-world complexity.

1. Executive Summary

This paper introduces MobileWorld, a substantially more challenging online benchmark for evaluating autonomous mobile GUI agents that addresses the saturation of existing benchmarks like AndroidWorld, where state-of-the-art agents now exceed 90% success rates. MobileWorld derives its difficulty from long-horizon, cross-application workflows — averaging 27.8 completion steps versus 14.3 in AndroidWorld, with 62.2% of tasks spanning multiple apps — and introduces two novel task categories: agent-user interaction tasks (where agents must proactively request clarification from a simulated user when instructions are deliberately ambiguous, e.g., asking for a contact's email address when only a name is provided) and MCP-augmented tasks (requiring hybrid orchestration between GUI operations and external tool invocations via the Model Context Protocol, e.g., fetching GitHub commit histories via API then composing a summary email through the native Mail app). Under a planner-executor agentic framework equipped with GPT-5, the top-performing agent achieves only 51.7% success rate on MobileWorld, while the best end-to-end model reaches just 20.9%, with performance collapsing on novel task categories — most baseline models score below 10% on agent-user interaction and near 0% on MCP-augmented tasks — establishing that current architectures face fundamental deficiencies in ambiguity detection, collaborative dialogue, and hybrid execution planning that cannot be bridged by scaling existing GUI-only approaches.

2. Context and Motivation

The Core Problem: Mobile Agent Benchmarks Have Stagnated

The fundamental issue this paper addresses is that the field of autonomous mobile GUI agents lacks a benchmark capable of differentiating genuine progress from incremental improvements. AndroidWorld (Rawles et al., 2024) has emerged as the dominant evaluation platform due to its reproducible emulator environment and deterministic programmatic verification — features that earlier benchmarks like AndroidArena, A3, and SPA-Bench lacked because their reliance on third-party commercial apps (Gmail, YouTube, etc.) forced them into noisy "MLLM-as-a-judge" evaluation. However, AndroidWorld's very success has created a crisis: state-of-the-art agentic frameworks now achieve success rates exceeding 90% on its leaderboard, fundamentally limiting its ability to distinguish between incremental improvements and genuine breakthroughs in mobile agent capabilities.

This saturation is not a minor inconvenience — it's an existential problem for a research community. When a benchmark can no longer rank systems meaningfully, it ceases to drive progress. Researchers cannot tell whether a new method represents a genuine advance or simply overfits to the idiosyncrasies of a saturated test set. The paper explicitly frames this as the primary motivation: AndroidWorld is "approaching saturation," and the community needs a new instrument that restores discriminative power.

But saturation alone isn't the full story. Even if AndroidWorld weren't saturated, the paper argues it has deeper structural limitations that make it an inadequate proxy for real-world mobile agent capabilities.

Why This Problem Matters: The Gap Between Benchmarks and Reality

The paper identifies three fundamental disconnects between existing mobile agent benchmarks and the demands of real-world deployment, each with practical significance:

1. Task complexity doesn't reflect actual mobile usage patterns. Current benchmarks — including AndroidWorld — typically feature short-horizon tasks confined within single applications. In AndroidWorld, only 9.5% of tasks involve cross-application workflows, and the average task requires just 14.3 completion steps. Real mobile usage, however, frequently demands long-horizon planning across multiple applications: a user might ask their agent to "find the three most expensive items in my shopping cart, calculate their total price, and email the result to my spouse" — a task spanning e-commerce, calculation, and communication apps. The paper notes that critical application categories are entirely absent from AndroidWorld, specifically e-commerce and enterprise communication, which are "central to daily mobile workflows." This matters because agents optimized for single-app, short-horizon tasks may fail catastrophically when confronted with the cross-app coordination and memory retention required in practice.

2. The assumption of fully-specified instructions is unrealistic. Existing benchmarks provide agents with complete, unambiguous task descriptions. In reality, users are frequently vague or incomplete: "Send an email to Kevin" when Kevin's email address isn't in the device's contacts, or "Remove a few events from my schedule on October 20th" without specifying which ones. The paper argues this isn't a corner case — it's the norm in human-agent interaction — and that agents must proactively engage in clarification dialogues to acquire missing information rather than hallucinating or failing silently. No prior mobile benchmark evaluates this capability, meaning agents optimized on existing suites may develop exactly the wrong behavior (confident hallucination rather than appropriate uncertainty).

3. External tool invocation is becoming essential but remains unevaluated. The Model Context Protocol (MCP) (Anthropic, 2024) has rapidly emerged as a standardized interface for exposing agents to external tools — APIs for GitHub repository analysis, geospatial navigation, financial data retrieval, and academic paper search. The paper notes that "the community is actively debating the integration of GUI operations alongside MCP tools" (Wang et al., 2025), recognizing that next-generation mobile agents must seamlessly combine interface manipulation with external tool invocations. No existing mobile benchmark evaluates this hybrid execution paradigm. This is significant because MCP tools offer capabilities that are either impossible or extremely cumbersome through GUI operations alone — fetching commit histories from a repository, querying real-time stock data, or performing geocoding — and the strategic choice between GUI operations and API calls is itself a capability that agents must demonstrate.

The practical stakes here are substantial. If benchmarks systematically fail to measure capabilities that real-world deployment requires, organizations building mobile agents — for customer support, productivity automation, accessibility, or enterprise workflows — will make deployment decisions based on misleading signals. An agent that achieves 95% on AndroidWorld might be essentially unusable in practice because it cannot handle vague instructions, cross-app workflows, or external tool integration.

Where Prior Approaches Fall Short

The paper systematically catalogs the limitations of existing mobile GUI agent benchmarks (Table 1), revealing a consistent pattern of tradeoffs that no prior work has resolved.

The determinism-vs-realism tradeoff. Benchmarks using commercial third-party applications (AndroidArena with 16 apps/221 tasks, A3 with 20 apps/201 tasks, Pro-Bench with 34 apps/200 tasks, AndroidDaily with 48 apps/235 tasks, SPA-Bench with 66 apps/340 tasks) achieve realism by testing on the actual Gmail, YouTube, and Google Maps apps that users interact with daily. However, this realism comes at a steep cost: these apps require external backend authentication (mandatory login flows), have opaque internal states that cannot be programmatically inspected, and are subject to UI changes that break reproducibility. As a result, these benchmarks must use "MLLM-as-a-judge" evaluation — asking another language model to visually assess whether a task was completed — which "introduces stochasticity and noise into the assessment process." This is fundamentally incompatible with rigorous scientific benchmarking, where a given agent on a given task should always receive the same score.

The reproducibility-vs-coverage tradeoff. Benchmarks that achieve deterministic evaluation (MobileAgentBench with 10 apps/100 tasks, AndroidLab with 9 apps/138 tasks, AndroidWorld with 20 apps/116 tasks) do so by restricting themselves to system applications or apps where backend state can be controlled. However, this severely limits coverage: as Table 1 shows, none of these deterministic benchmarks include applications requiring external backend authentication. AndroidWorld, despite its 20 apps and programmatic verification, omits e-commerce, enterprise communication, and social media — precisely the application categories where real mobile usage concentrates. The paper's characterization of this as a "balance" that existing work fails to achieve is central to its positioning.

The instruction-completeness assumption. All prior benchmarks — both deterministic and realistic — assume tasks come with fully-specified, unambiguous instructions. The paper argues this is a systematic blind spot: none of the benchmarks in Table 1 include agent-user interaction tasks. This means the entire field has been optimizing agents for a task distribution that doesn't match deployment conditions, potentially rewarding behaviors (like confidently proceeding with incomplete information) that are actively harmful in practice.

The GUI-only paradigm. Similarly, no prior mobile benchmark evaluates external tool invocation. While MCP-augmented benchmarks exist — OSWorld-MCP (Jia et al., 2025) for desktop computer-use agents, MCPWorld (Yan et al., 2025b) for API/GUI/hybrid agents — they operate in domains separate from mobile GUI agents. The paper specifically cites these as related but distinct efforts, noting that MobileWorld is "the first mobile benchmark to integrate both agent-user interaction tasks and MCP-augmented tasks within a unified evaluation framework for practical mobile agent deployment."

The τ-bench lineage and its limitations. The paper draws conceptual inspiration from τ-bench (Yao et al., 2024) and τ2-bench (Barres et al., 2025), which introduced dynamic user-agent dialogue and dual-control environments for evaluating conversational agents. τ-bench showed that even GPT-4o succeeds on fewer than 50% of tasks in customer service scenarios with simulated users, and τ2-bench demonstrated significant performance drops when agents must coordinate with users. However, these benchmarks operate in text-based, domain-specific API environments (customer service, travel booking) — not in mobile GUI contexts with visual interface manipulation. The paper's insight is that the user-interaction challenge is at least as acute in mobile settings (where instructions are often terse and context-dependent) but has never been evaluated there.

How This Paper Positions Itself

MobileWorld positions itself not as an incremental improvement to AndroidWorld but as a qualitative step change in mobile agent evaluation along four dimensions that the paper argues are all necessary and none of which any prior benchmark achieves simultaneously:

1. Deterministic evaluation without sacrificing application coverage. The paper's core architectural innovation is using self-hosted, production-grade open-source alternatives to commercial applications: Mattermost instead of Slack, Mastodon instead of X/Twitter, Mall4Uni's Taodian instead of Taobao, and a custom Mail app instead of Gmail. By modifying source code and gaining direct access to backend PostgreSQL databases, the environment achieves what the paper calls "a fully observable and controlled environment" — enabling deterministic verification through database queries, local storage inspection, application callbacks, and textual answer matching — while still covering the application categories (enterprise communication, social media, e-commerce, email) that commercial benchmarks capture but deterministic ones previously couldn't. This is the "balance" the paper claims to strike, and it's what places the ✓ in both the "Backend-Req. Apps" and "Deterministic Evaluation" columns of Table 1 — uniquely among all benchmarks surveyed.

2. Task complexity that reflects real-world difficulty. The paper's task design philosophy is explicit: instruct annotators to create long-horizon tasks integrating multiple challenging dimensions — subgoal combination across apps, fine-grained visual recognition (e.g., extracting data from complex PDF layouts), memory retention across steps, numerical/logical reasoning, implicit temporal or spatial context, and precise instruction following. The result is a benchmark where tasks average 27.8 completion steps (nearly 2× AndroidWorld's 14.3), 62.2% of tasks span multiple apps (vs. 9.5%), and 12.4% involve three or more apps. The paper doesn't claim these numbers are inherently "correct" — rather, they reflect a deliberate effort to create headroom that can discriminate between current and future systems.

3. Novel task paradigms that expose unmeasured capabilities. The agent-user interaction tasks (22.4% of the benchmark) and MCP-augmented tasks (19.9%) are not marginal additions — collectively they account for 42.3% of MobileWorld. The paper's positioning is that these aren't just "harder tasks" but fundamentally different capabilities that the GUI-only paradigm cannot measure. The construction methodology for interaction tasks is deliberate: annotators start with a clear, achievable goal, then systematically remove critical information (replacing a full email address with just a name, removing the specific events to delete from a calendar) and verify that the omitted information is not recoverable from the device state. This ensures that agents must interact to succeed — there is no clever workaround. For MCP tasks, annotators first design tasks solvable only via MCP tools (non-trivial or error-prone through pure GUI), then extend them into hybrid workflows that consume MCP outputs through GUI actions, forcing agents to demonstrate the orchestration capability.

4. A systematic empirical characterization of the capability frontier. Rather than merely claiming MobileWorld is harder, the paper provides detailed evidence of where and why current systems fail. The performance breakdown by task category (Table 6) reveals that the 51.7% overall success rate of GPT-5 + UI-Ins-7B masks stark capability differences: 54.0% on GUI-Only tasks, 62.2% on agent-user interaction, and 51.6% on MCP-augmented. End-to-end models show even more dramatic patterns — Qwen3-VL-235B scores 12.8% on GUI-Only but 4.4% on interaction and 5.4% on MCP; Doubao-1.5-UI-TARS reaches 26.3% on GUI-Only and 32.4% on interaction but cannot attempt MCP tasks at all. This granularity is the paper's evidentiary strategy: it's not just claiming that existing systems are inadequate but precisely characterizing the nature of their inadequacy, establishing a baseline from which future progress can be measured.

The paper explicitly frames this as a necessary platform for a research roadmap. The five research challenges identified in Section 4.6 — ambiguity detection and user engagement, MCP context management, long-term memory and state tracking, complex logic reasoning, and temporal-spatial context awareness — are derived from manual inspection of failure trajectories across evaluated models. This transforms the benchmark from a simple ranking tool into a diagnostic instrument: future researchers can target specific failure modes and measure progress against the corresponding subsets of MobileWorld tasks.

The paper's relationship to the training-inference scaling narrative in the broader LLM literature is implicit but present: just as the community has recognized that scaling pretraining compute has diminishing returns and that inference-time strategies (verifier-guided search, iterative revision) can substitute for model size on certain problems, MobileWorld implies that current mobile agent architectures have hit a capability ceiling that cannot be breached by simply scaling models or training data within the GUI-only paradigm. The introduction of user interaction and MCP tool use represents not just new evaluation dimensions but new affordances — capabilities that fundamentally change what kinds of problems can be solved — and the paper positions these as necessary for the next generation of mobile intelligence.

3. Technical Approach

3.1 Reader Orientation

MobileWorld is a fully containerized Android evaluation environment with 201 hand-crafted tasks that tests whether a mobile GUI agent can complete long-horizon, cross-application workflows — including asking a simulated user for clarification when instructions are deliberately vague and calling external APIs via the Model Context Protocol when GUI operations alone are insufficient. The core problem it solves is that existing mobile agent benchmarks (particularly AndroidWorld) have saturated above 90% success rates and fail to measure three capabilities essential for real-world deployment: handling ambiguous instructions through dialogue, orchestrating GUI actions with external tool calls, and sustaining memory and reasoning across dozens of sequential steps. The solution's shape is a benchmark that is structurally harder (averaging 27.8 steps per task vs. 14.3 in AndroidWorld, with 62.2% of tasks spanning multiple apps), introduces two novel task paradigms (agent-user interaction and MCP-augmented tasks), and achieves deterministic verification on applications that previously required noisy MLLM-as-a-judge evaluation by substituting proprietary services with self-hosted open-source alternatives whose backend databases can be directly queried.

3.2 Big-Picture Architecture (Diagram in Words)

MobileWorld consists of two major components communicating over HTTP, as illustrated in Figure 3:

The Host Machine (left side of Figure 3) is where the GUI agent under evaluation runs. It receives a natural language task instruction, observes the current Android screen as a screenshot, and produces actions from an extended action space that includes standard GUI operations (click, type, scroll, etc.), an ask_user action for requesting clarification from a simulated user, and an mcp_call action for invoking external tools. When the agent issues ask_user, the query is routed to a separate LLM-simulated user agent (implemented with GPT-4.1) that holds the deliberately omitted information in its system prompt. When the agent issues mcp_call, the request goes to an MCP client that forwards it to the appropriate MCP server (one of five servers spanning geospatial, code repository, document processing, financial, and academic paper domains, collectively offering 64 tools), and the structured response is injected back into the agent's observation history.

The Docker Environment (right side of Figure 3) encapsulates an isolated Android ecosystem inside a Docker-in-Docker container. It contains: (1) a rooted Android Virtual Device (AVD) running the emulator with the full suite of 20 applications pre-installed; (2) self-hosted backend services for applications that require server components (Mattermost's PostgreSQL database, Mastodon's PostgreSQL database and media storage, the Mail app's callback server, and the Taodian e-commerce mock backend); (3) an ADB Controller that executes the GUI actions received from the host machine on the emulator and captures screenshots after each action; and (4) an Eval Controller that determines task success by querying one of four verification mechanisms — backend database inspection, local storage examination via ADB, application callback logs, or textual answer matching — and returns a binary score.

Information flows in a closed loop: the host machine sends a task instruction to the agent → the agent observes the current screenshot → the agent selects an action (GUI, ask_user, or mcp_call) → the action is executed on the Docker environment (directly on the emulator for GUI, forwarded to the user agent or MCP server for extended actions) → a new screenshot and any response text are captured → the loop repeats until the agent signals task completion via the status action or exhausts its step budget → the Eval Controller inspects the final state and returns a score.

3.3 Roadmap for the Deep Dive

This section builds understanding in the following order, chosen because each component depends on the ones before it:

  • First, the formal task definition (Section 3.1), which establishes the mathematical framework — POMDP, state/observation/action spaces, transition function, reward — and then extends it with the two novel task types. This is the vocabulary everything else uses.

  • Second, the environment architecture (Section 3.2), covering the extended action space (all 16 primitive actions), the containerization strategy that enables determinism, and crucially, the four verification mechanisms that replace MLLM-as-a-judge evaluation. Without understanding how verification works, the task construction methodology makes no sense — tasks must be designed so their success can be verified through these mechanisms.

  • Third, the benchmark construction methodology (Section 3.3) , which explains how the 201 tasks were created: the general design principles (long-horizon, cross-app, multi-dimensional complexity), the specific procedures for constructing agent-user interaction tasks (deliberate information omission + verification that omitted information is unrecoverable) and MCP-augmented tasks (standalone MCP-solvable subgoal + GUI-dependent extended workflow), and the human-in-the-loop validation protocol that ensures all tasks are actually solvable.

  • Fourth, the data statistics (Section 3.4) , which provides the quantitative characterization of the benchmark along four dimensions: domain distribution, cross-app complexity, task category breakdown, and verification method usage. These numbers operationalize the qualitative design claims made earlier.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and empirical analysis paper whose core idea is that existing mobile agent evaluations are saturated and structurally misaligned with real-world requirements, and that a new benchmark incorporating long-horizon cross-app workflows, agent-user interaction, and MCP-augmented tasks — with deterministic verification achieved through self-hosted open-source applications — can restore discriminative power and reveal fundamental capability gaps in current architectures.

Formal Task Definition: The POMDP Framework and Its Extensions

The paper grounds its task definition in a Partially Observable Markov Decision Process (POMDP) formalism, which provides a precise vocabulary for describing what agents must do and why certain tasks are hard. A POMDP is the standard framework for sequential decision-making under uncertainty: the agent cannot directly observe the full state of the environment (it sees only a screenshot and instruction, not the internal application states or database contents), so it must maintain an implicit belief about what's happening and plan actions accordingly.

The standard Mobile GUI agent task is formalized as a tuple:

(S,O,A,T,R)(\mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R})

where $\mathcal{S}$ is the state space representing the complete mobile environment (every possible configuration of every application, every database entry, every file on the device, every pixel on screen); $\mathcal{O}$ is the observation space, which includes both a natural language instruction and interface representations (primarily screenshots, but also any text responses from user queries or MCP tool calls, collectively forming the agent's interaction history); $\mathcal{A}$ is the action space of standard mobile UI operations (the full set is specified in Table 2 and detailed below); $\mathcal{T}: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ is the deterministic transition function mapping a state and an action to a successor state (tapping a button at coordinates (x,y) transitions to whatever screen that button leads to; typing text transitions to the state with that text in the focused field); and $\mathcal{R}: \mathcal{S} \times \mathcal{A} \to \{0, 1\}$ is a binary reward function indicating task completion — the agent receives a reward of 1 if the final state satisfies the task's success criteria, and 0 otherwise. Importantly, $\mathcal{R}$ is effectively only evaluated at the terminal step (when the agent invokes the status action), making this a sparse-reward setting where intermediate actions give no signal about progress.

Why this formalism matters: The POMDP structure explains why MobileWorld's tasks are harder than AndroidWorld's. In a POMDP, difficulty arises from three sources: the size of the state space $\mathcal{S}$ (more apps, more possible configurations), the length of the action sequence needed to reach a goal state from the initial state (the planning horizon), and the degree of partial observability (how much the agent must infer from limited observations). MobileWorld deliberately increases all three: the state space is larger because it includes applications with complex internal states (Mattermost channels, Mastodon posts, Taodian shopping carts) and external data sources (MCP server responses); the planning horizon is longer (27.8 average steps vs. 14.3); and the partial observability is higher because agent-user interaction tasks deliberately withhold critical information from the initial observation, forcing agents to recognize their own uncertainty.

The paper then extends this framework with three task categories, each modifying the POMDP in a specific way:

GUI-Only Tasks are the baseline, following AndroidWorld's formulation exactly. They come in two subtypes: Task Completion (executing a sequence of GUI actions to drive the system into a specified target state, e.g., "compose and send an email with subject X to recipient Y") and Information Retrieval (locating and extracting specific information from internal mobile data or external sources, then responding with a textual answer, e.g., "what is the highest temperature in Beijing today?"). These tasks use only the standard action space and test the agent's ability to navigate interfaces, locate information, and execute multi-step plans. They constitute 116 of the 201 tasks (57.7%).

Agent-User Interaction Tasks modify the observation function of the POMDP by deliberately withholding information that is necessary for task completion. The task instruction presented to the agent is incomplete — for example, "Send an email to Kevin with the message 'Hello'" when "Kevin" is not stored in the device's Contacts or Email apps with an associated email address. The agent cannot succeed by simply executing GUI actions; the necessary information (Kevin's email address) is not present in any observable state reachable through GUI navigation. Instead, the agent must recognize this information gap and invoke the ask_user action, which is the only mechanism for acquiring the missing information. The ask_user action temporarily extends the observation space: the agent's query is routed to a simulated user agent (a separate LLM instance) that has been configured with the deliberately omitted information in its system prompt. The user agent's textual response becomes part of the agent's observation history alongside the next screenshot. These tasks constitute 45 of the 201 tasks (22.4%).

The construction of the user agent's knowledge is critical and carefully controlled. The simulated user agent is initialized with a system prompt containing: (1) the full, unambiguous task goal (which the GUI agent never sees), and (2) a relevant_information field containing exactly the information omitted from the GUI agent's instruction. The user agent is explicitly instructed: "If the question is not related to the task or no more task-related information is available, you need to refuse to answer in a polite manner. DO NOT make up any information." This prevents the user agent from becoming an oracle that solves the task for the GUI agent — it will only provide the specific missing pieces when asked appropriately, and will reject off-topic or fishing-expedition queries. The user agent uses GPT-4.1 (though the paper notes this is configurable) with temperature set to 0.0.

MCP-Augmented Tasks extend the action space $\mathcal{A}$ with an additional primitive: mcp_call(tool_name, params). These tasks require the agent to invoke one or more external MCP tools, process their structured responses, and then use the retrieved information to complete GUI-based actions. The key challenge is orchestration — the agent must decide when a GUI operation would be inefficient or impossible, recognize that an MCP tool is applicable, formulate the correct tool call with appropriate parameters, parse the structured response (which may be JSON objects, lists, or long text), extract the relevant information, and then feed that information into subsequent GUI actions. For example, in the task illustrated in Figure 2 (right), the agent must: (1) recognize that fetching a GitHub repository's README via browser navigation would be cumbersome and error-prone, (2) select the gitHub_get_file MCP tool from the available set, (3) correctly populate the owner, repo, and path parameters, (4) receive the file content (potentially thousands of tokens), (5) extract or summarize the relevant information, and (6) use the native Mail app's GUI to compose and send a message containing that summary. These tasks constitute 40 of the 201 tasks (19.9%).

When a task is tagged as MCP-augmented, the specifications of the relevant MCP tools are dynamically injected into the planner's system prompt (for agentic frameworks) or provided in a structured format (for end-to-end models). This means the agent knows which tools are available and what parameters they expect, but must still decide when and how to use them. The paper integrates five MCP servers from the Alibaba Bailian marketplace, collectively offering 64 tools across diverse domains, as detailed in Table 3: Amap Maps (15 tools for geocoding, reverse geocoding, weather queries, multi-modal route planning, distance measurement, and location-based search); GitHub (26 tools for repository querying, commit history retrieval, issue tracking, and code analysis); Jina AI (3 tools for document processing and multi-modal content analysis); Stockstar (16 tools for A-share and Hong Kong stock data including fundamentals, derived metrics, and financial analysis); and arXiv (4 tools for academic paper search and retrieval).

Environment Architecture: Containerization, Action Space, and Deterministic Verification

The environment architecture's central design principle is full observability for the evaluator, partial observability for the agent. The agent sees only screenshots and text responses — the standard mobile GUI agent interface. The evaluator, however, has direct access to every application's internal state through backend databases, local storage, and application callbacks. This asymmetry is what enables deterministic verification without constraining the agent's interface to be unrealistic.

Containerized Architecture. MobileWorld is deployed entirely within Docker containers using a Docker-in-Docker pattern (the right side of Figure 3). This means: a host machine runs a Docker container, and inside that container runs another Docker daemon that manages the Android emulator and application backends. This nesting achieves two goals. First, it completely isolates the evaluation environment from the host machine — different host operating systems, GPU configurations, or system libraries cannot affect the emulator's behavior, ensuring that a task that passes on one machine will pass on another. Second, it enables snapshot-based state management: the entire container state (AVD disk image, backend databases, application files) can be captured as a Docker image and restored to a known clean state before each task execution. The paper uses AVD snapshots specifically for the emulator state: "Each task execution begins from a predetermined snapshot, guaranteeing reproducible starting conditions."

The container includes four main components: (1) a rooted Android Virtual Device (AVD) running a standard Android image with all 20 applications pre-installed and pre-configured (contacts, SMS messages, emails, calendar events, and files are preloaded with synthesized or internet-derived data to create a realistic initial state); (2) self-hosted application backends for Mattermost, Mastodon, Mail, and Taodian, each running as separate services within the container, with their databases and file storage initialized from snapshots; (3) an ADB Controller that serves as the bridge between the host machine's action commands and the emulator — it receives action specifications (click coordinates, text to input, scroll direction) over HTTP, executes them via Android Device Bridge commands on the emulator, waits for the screen to stabilize, captures a screenshot, and returns it; (4) an Eval Controller that exposes an HTTP endpoint for score retrieval — when queried, it inspects the appropriate verification target (database, local storage, callback log, or text answer) for the current task and returns a binary score.

The Extended Action Space (Table 2). The action space is the interface between the agent and the environment. MobileWorld defines 16 primitive actions organized into four categories:

GUI Operations (7 actions): click(x, y) — tap at the specified pixel coordinates; double_tap(x, y) — double-tap at coordinates; long_press(x, y) — long-press at coordinates; drag(start_x, start_y, end_x, end_y) — drag from start to end coordinates (used for actions like moving items, drawing, or navigating maps); input_text(text) — type the specified text string into the currently focused input field (the agent must have previously focused a text field via a click action; this is a composite action that includes clicking the field, typing, and pressing enter — the agent does not separately manage keyboard focus); scroll(direction) — scroll in the specified direction (up, down, left, or right), with the paper explicitly noting in the planner prompt that "scroll direction is INVERSE to swipe (scroll down to see lower content)" to prevent a common confusion; wait — pause execution to allow screen content to update after an action that may trigger loading or animation (the agent must explicitly invoke this; the environment does not automatically wait for stability).

Navigation (3 actions): navigate_home — return to the Android home screen (equivalent to pressing the home button); navigate_back — navigate to the previous screen (equivalent to pressing the back button); keyboard_enter — press the enter key on the virtual keyboard (used when input_text does not automatically submit, or when the agent needs to confirm a dialog).

Task Control (2 actions): answer(text) — provide a textual response to the user, used for Information Retrieval tasks where the expected output is a specific string or value (the paper emphasizes: "Note that answer will terminate the task immediately"); status(goal_status) — mark the task as either "complete" or "infeasible", signaling to the evaluator that the agent believes it has finished (or cannot finish) and that evaluation should proceed. This is the only action that triggers the reward function $\mathcal{R}$.

Extended Actions (2 actions): ask_user(text) — request clarification from the simulated user agent; the text parameter is the natural language question the agent wants to ask (e.g., "What is Kevin's email address?"). When invoked, the query is forwarded to the GPT-4.1-based user agent, which generates a response conditioned on its system prompt containing the omitted information. The response text, along with the current screenshot, is appended to the agent's interaction history as the observation for that step. The agent can invoke ask_user multiple times within a single task if further clarification is needed. mcp_call(tool_name, params) — invoke an MCP tool with the specified name and a JSON object of parameters. When invoked, the MCP client on the host machine forwards the request to the appropriate MCP server, which executes the tool (e.g., querying a database, fetching a file, making an API call) and returns a structured response. This response, along with the current screenshot, is injected into the agent's observation history. The agent can make multiple MCP calls within a task.

The paper takes care to handle a critical ambiguity: the planner (in the agentic framework) does not output raw click actions with coordinates. Instead, it outputs a natural language description of the target element (e.g., "the 'Send' button at the bottom right corner"), and a separate grounding executor (UI-Ins-7B) translates this description into pixel coordinates. This design choice is explained in Section 4.1: it enables any general-purpose VLM to serve as the planner without requiring built-in grounding capabilities, and it separates the reasoning problem (what to click) from the perception problem (where that thing is on screen).

The Four Verification Mechanisms. The evaluation system uses four complementary methods to determine whether a task was successfully completed, selected based on the nature of the task's success criteria. The choice of verification method is per-task and fixed at benchmark construction time.

Textual Answer Verification is used for Information Retrieval tasks where the expected output is a specific string or numerical value (22 tasks, 10.9% of the benchmark). When the agent invokes the answer(text) action, the evaluator compares the text against the ground-truth answer using pattern matching with regular expressions or exact string comparison. For example, in a task requesting "the driving distance in kilometers" with instructions to "response only one integer number," the evaluator extracts integers from the answer text and compares against the ground-truth distance. This method works because the ground-truth answers are either deterministic facts (e.g., today's temperature from a weather service, the distance between two cities) or can be pre-computed from the initial system state.

Backend Database Verification is used for tasks involving self-hosted applications with accessible backend databases — Mattermost, Mastodon, and similar apps (95 tasks, 47.3% of the benchmark). After the agent signals completion via status, the evaluator directly queries the PostgreSQL database backing the application and checks whether the expected state change has occurred. For example, in the task "Reply to the toot of gourmet user about Greek food Moussaka, and the reply content should be 'Nice sharing, i love it'," the evaluator queries Mastodon's database for replies to the specified post and verifies that a reply with the exact content string exists. This method is fully deterministic — the database is the ground truth of application state — and does not depend on screen rendering, timing, or visual recognition. The paper's ability to use this method is a direct consequence of the architectural choice to self-host open-source alternatives rather than use commercial applications: Slack's or Twitter's databases are not accessible to benchmark developers, but Mattermost's and Mastodon's are.

Local Storage Inspection is used for applications that store state in on-device databases or files rather than on remote servers (74 tasks, 36.8% of the benchmark). Leveraging the rooted emulator, the evaluator uses Android Device Bridge (ADB) commands to inspect application-specific local storage. For example, in the task "Set a weekend alarm for 8:25 a.m. with the ringtone 'beebeep' and vibration off," the evaluator uses ADB to query the Alarm app's local database and verify that an alarm with the specified time, ringtone, and vibration setting exists. Similarly, calendar events can be validated by examining the Fossify Calendar database, email drafts by inspecting the Mail application's local storage, and file renaming by checking the Downloads folder's contents. This method is also fully deterministic because it reads the actual persistent state that the application itself uses.

Application Callbacks are used for lightweight customized applications where the developers have instrumented the app with callback hooks (10 tasks, 5.0% of the benchmark). For the Taodian e-commerce app and the custom Mail app, the developers implemented callback APIs that capture intermediate states during execution and persist them to a local file. For example, when a user removes items from the Taodian shopping cart, a callback hook fires and logs the cart contents to a file; when an email is sent, the Mail app's callback records the recipient, subject, body, and attachments. The evaluator reads this callback log and checks whether the expected state changes occurred. This method is slightly more invasive (it requires modifying the application source code) but provides precise, timestamped records of user-visible actions that might be difficult to verify through other means.

The paper emphasizes that this multi-faceted verification system "ensures that task outcomes are evaluated deterministically, eliminating the evaluation noise inherent in MLLM-as-a-judge approaches used by prior work." The contrast is with benchmarks like AndroidArena or SPA-Bench, where the evaluator is another LLM that looks at before-and-after screenshots and judges whether the task was completed — a process that introduces stochasticity because the same screenshots may receive different judgments from the same LLM on different runs, or from different LLM judges.

Benchmark Construction: Design Principles, Task Categories, and Validation

The construction methodology is designed to produce tasks that are simultaneously (a) solvable by a competent agent, (b) verifiable through one of the four deterministic mechanisms, and (c) challenging along multiple dimensions that reflect real-world mobile usage complexity.

General Task Design Principles. The paper instructs annotators to design tasks that integrate multiple challenging dimensions, deliberately chosen to expose specific capability gaps. These dimensions are:

  1. Subgoal combination across apps: A single task instruction requires completing multiple subtasks that span different applications, with information flowing from one app to another. For example, "Find the three most expensive items in the TaoDian app shopping cart, calculate their total price, and email the result to my spouse" combines e-commerce browsing, numerical computation, and email composition.

  2. Fine-grained visual recognition: Tasks require extracting specific data from complex visual layouts, such as reading prices from a cluttered shopping cart screen, parsing dates from a calendar grid, or extracting information from PDF documents rendered in the DocsReader app.

  3. Memory retention across steps: Information acquired early in a task must be retained and used later. For example, a task might require searching for a contact's phone number in the Contacts app, then switching to the Messages app to send that number to a third party — the agent must remember the number across the app switch, since Android's clipboard is not guaranteed to preserve it.

  4. Numerical or logical reasoning: Tasks require computation beyond simple lookup, such as summing prices, comparing dates to determine "tomorrow," or counting items matching a criterion.

  5. Implicit temporal or spatial context: Tasks use relative references like "tomorrow" or "this weekend" that must be resolved by observing the system clock displayed on the device's status bar, or "my hometown" that must be inferred from the device's location settings. The agent cannot simply pattern-match the instruction text — it must ground these references in the actual device state.

  6. Precise instruction following: Tasks include constraints on the output format, such as "Response only one integer number. No other text" or "format each line as 'author: commit message'." Even if the agent retrieves the correct information, violating the format constraint results in task failure.

These dimensions are not merely aspirational — they are design constraints that annotators must satisfy. The paper's task validation protocol (described below) ensures that tasks actually require these capabilities.

Initial System State Preparation. All tasks are constructed based on a consistent initial system state — a preloaded device image containing contacts, SMS messages, emails, calendar events, and files. This preloaded content is either synthesized by LLMs (generating realistic but fictional contact lists, message threads, and calendar entries) or derived from the internet (e.g., sample PDFs, publicly available documents). The purpose is to create a realistic but controlled starting point: the initial state is complex enough to support interesting tasks (there are many contacts, messages, and events to search through) but is identical across all evaluation runs, ensuring reproducibility. For applications with self-hosted backends, the database is initialized with pre-generated content (e.g., Mattermost channels with existing message history, Mastodon instances with pre-created posts and user accounts) and snapshotted so that each task execution begins from an identical database state.

Agent-User Interaction Task Construction. The construction procedure for interaction tasks follows a systematic four-step process designed to guarantee that the omitted information is truly necessary and unrecoverable:

  1. Write the complete task: An annotator first writes a clear, fully-specified, and achievable task goal. For example: "Send an email to kevin_zhang@example.com with the message 'Hello'." This complete version serves as the ground truth — the annotator verifies that this task can be completed by navigating to the Mail app, composing a new email, entering the recipient address, typing the message, and sending.

  2. Remove critical information: The annotator identifies one or more pieces of information without which the task cannot be completed, and removes them from the instruction. In the example, the full email address kevin_zhang@example.com is replaced with just "Kevin." The task instruction becomes: "Send an email to Kevin with the message 'Hello'." The ambiguity must be genuine — Kevin must be a name that is not present in the device's Contacts or Email apps with an associated email address, so that no amount of GUI navigation can resolve it.

  3. Verify non-recoverability: The annotator examines the device environment to confirm that the omitted information is not discoverable through any sequence of GUI actions. This includes checking Contacts (Kevin is not saved), Email (no past emails from or to Kevin exist), Messages (no SMS history with Kevin), and any other app that might contain the information. If the information is discoverable somewhere, the task is revised — the placement must ensure that ask_user is the only path to the missing information.

  4. Configure the user agent: The omitted information is embedded in the simulated user agent's system prompt under the relevant_information field. For the example: relevant_information: "Kevin's email address: kevin_zhang@example.com". The user agent is also given the full task goal: "Send an email to kevin_zhang@example.com with the message 'Hello'." The user agent's system prompt is shown in full in Section 3.3 and includes explicit constraints: "If the question is not related to the task or no more task-related information is available, you need to refuse to answer in a polite manner. DO NOT make up any information. You can ONLY give the answer based on the relevant information and the task goal."

This construction ensures that the interaction is necessary (the information cannot be obtained otherwise), focused (the user agent only answers task-relevant questions), and controlled (the user agent's knowledge is exactly the omitted information, no more and no less).

MCP-Augmented Task Construction. The construction procedure for MCP tasks follows a two-phase process:

  1. Design the MCP-only subgoal: Annotators first read the description and sample output of each MCP tool. They then create standalone tasks that can be completed using only MCP tool calls — tasks that would be non-trivial or error-prone through pure GUI operations. For example, "List all commits in the main branch of the mastodon/mastodon repository from the past week" is straightforward via the GitHub MCP's list_commits tool (which returns structured commit data with author, message, and timestamp) but would be extremely tedious through a browser GUI (navigating to GitHub, finding the repository, scrolling through commit history, extracting dates and messages from rendered HTML). Other examples include: querying real-time stock prices via Stockstar (impossible through GUI on a phone without a trading app), geocoding an address via Amap (the Google Maps app on the device provides navigation but not programmatic geocoding), or searching academic papers via arXiv (the browser can access arxiv.org but structured search with filtering is cumbersome).

  2. Extend into a hybrid workflow: The annotator then appends one or more GUI-based actions that consume the MCP output. For example, after retrieving the commit history via MCP, the agent is instructed to "compose and send an email summarizing the changes using the native Email app" — requiring the agent to format the structured JSON response into human-readable text, switch to the Mail app, compose an email with appropriate subject and body, and send it to a specified recipient. This hybrid structure ensures that agents cannot succeed by only calling MCP tools or only performing GUI actions — they must orchestrate both.

The paper integrates five MCP servers with collectively 64 tools (Table 3). These are sourced from the Alibaba Bailian MCP marketplace and are selected to span diverse domains that extend beyond standard smartphone capabilities: Amap Maps for geospatial queries, GitHub for code repository analysis, Jina AI for document processing, Stockstar for financial data, and arXiv for academic literature retrieval. The tools are pre-loaded into the environment and their specifications (tool names, parameter schemas, descriptions) are dynamically injected into the agent's system prompt when it is assigned an MCP-augmented task.

Human-in-the-Loop Task Validation. Every task in MobileWorld undergoes manual validation to ensure solvability. The protocol works as follows: after a task is constructed, an annotator (who may be the same person who designed the task or a different validator) initializes the task environment to the predetermined snapshot, receives the exact task instruction that agents will see, and manually interacts with the MobileWorld environment — executing actions (clicking, typing, scrolling, asking the user agent, calling MCP tools) until they believe the task is complete. The system then runs the evaluator on the resulting state. If the score is 1.0, the task is validated and added to the benchmark. If the score is 0.0, the validator retries the task from the same initial state, up to a maximum of five attempts. If after five attempts the task remains unsolved, "this indicates a potential issue in the task initialization, instruction clarity, or evaluation criteria. Such tasks are flagged and returned to the design phase for revision and reimplementation."

This validation protocol serves multiple purposes: it confirms that the task is actually solvable (not impossible due to a bug or an overly strict evaluation criterion), it provides a rough upper bound on the difficulty (a human can solve it, setting a ceiling on reasonable agent performance), and it catches errors in the evaluation logic (if the human believes they completed the task but the evaluator disagrees, either the task instructions or the evaluation criteria need revision). The five-attempt limit prevents the validator from overfitting to a marginal task by exploiting environment-specific quirks — if a task cannot be solved in five attempts by a human who knows exactly what the task is supposed to accomplish, it's too ambiguous or brittle for automated agents.

Data Statistics: Quantitative Characterization of the Benchmark

The paper provides comprehensive statistics (Table 5 and Figure 4) that quantify the benchmark's composition along four dimensions, each supporting a specific claim about MobileWorld's design.

Domain Distribution (Figure 4). The 201 tasks are distributed across eight application domains: Communication & Messaging (108 tasks, the largest category, reflecting the centrality of email, SMS, and team chat to mobile workflows); Productivity & Organization (76 tasks, including calendar management, file operations, and document reading); Social Platforms (41 tasks, primarily Mastodon interactions); MCP Services (40 tasks, the MCP-augmented task subset); Browser & Media (26 tasks, web search and media consumption); E-commerce (16 tasks, Taodian shopping operations); System & Settings (10 tasks, device configuration); and Navigation & Maps (10 tasks, location-based queries). The paper notes that "approximately 95% of tasks involve third-party applications, aligning with authentic mobile usage patterns" — only 5% are system apps (Settings, Clock, Camera). This is a deliberate contrast with benchmarks that rely heavily on system apps to achieve determinism.

App Complexity (Table 5). The benchmark emphasizes cross-app coordination: only 76 tasks (37.8%) are confined to a single application; 100 tasks (49.8%) involve exactly two applications; and 25 tasks (12.4%) involve three or more applications. Cumulatively, 62.2% of tasks require coordination between multiple apps. This statistic directly supports the paper's claim that MobileWorld better reflects real-world mobile usage, where "practical mobile assistance frequently demands long-horizon planning and cross-application workflows." It also explains the sharp performance drop from AndroidWorld (which has only 9.5% multi-app tasks) to MobileWorld: agents that achieve high success rates on single-app tasks may lack the planning and memory capabilities needed for cross-app workflows.

Task Category Breakdown (Table 5). The 201 tasks are partitioned into 116 GUI-Only tasks (57.7%), 45 Agent-User Interaction tasks (22.4%), and 40 MCP-Augmented tasks (19.9%). The novel task types collectively account for 42.3% of the benchmark — nearly half. This is a substantial fraction that prevents agents from achieving high overall success rates by excelling only at traditional GUI tasks. An agent that scores 90% on GUI-Only tasks but 0% on both novel categories would achieve only $(0.9 \times 116 + 0 \times 45 + 0 \times 40) / 201 = 51.9\%$ overall — which is close to the best observed performance (51.7%), consistent with the empirical finding that even the strongest models struggle on the novel categories.

Evaluation Mode Distribution (Table 5). The verification methods are distributed as: Backend Database Verification (95 tasks, 47.3%), Local Storage Inspection (74 tasks, 36.8%), Textual Answer Verification (22 tasks, 10.9%), and Application Callbacks (10 tasks, 5.0%). The dominance of database and storage verification (84.1% combined) reflects the architectural investment in self-hosted applications with accessible backends. This is the engineering achievement that enables deterministic evaluation on application categories (communication, social media, e-commerce) that commercial benchmarks handle through noisy MLLM judging. The paper's claim that MobileWorld "strikes a balance between production-grade utility and reproducible evaluation" is operationalized through these numbers: the benchmark covers the application categories users actually interact with, but verifies task completion through ground-truth database state rather than visual judgment.

MCP Infrastructure (Table 5). The benchmark includes 5 MCP servers offering 64 total tools (the paper lists 64 in the statistics table, though Table 3 sums to $15 + 26 + 3 + 16 + 4 = 64$, confirming the count). These tools provide capabilities that cannot be replicated through GUI operations — notably the GitHub tools for programmatic repository access, the Stockstar tools for real-time financial data, and the Amap tools for geocoding and route planning. The diversity of MCP domains (geospatial, code, documents, finance, academic literature) ensures that MCP-augmented tasks test general tool-use capability rather than domain-specific knowledge.

4. Key Insights and Innovations

Innovation 1: Deterministic Evaluation on Realistic Applications Through Self-Hosted Open-Source Alternatives

The paper's most architecturally distinctive contribution is not any single task design or metric, but rather a deployment philosophy that resolves a tradeoff the field has treated as fundamental: you can have deterministic, reproducible evaluation, or you can have tasks on the complex, backend-dependent applications people actually use — but not both. MobileWorld demonstrates that this tradeoff is false, and the method for breaking it is conceptually transferable.

What the field did before. Prior mobile agent benchmarks fell into two camps. Benchmarks aiming for ecological validity — AndroidArena (Chai et al., 2025), A3, SPA-Bench (Chen et al., 2024), ProBench (Yang et al., 2025), AndroidDaily (Yan et al., 2025a) — used real commercial applications (Gmail, YouTube, Google Maps, Slack) with real backend authentication. This gave them face validity: agents manipulated the same apps humans use. But it forced them into "MLLM-as-a-judge" evaluation — asking another language model to look at before-and-after screenshots and judge whether the task was completed — because the benchmark developers had no access to Gmail's or Slack's internal databases to verify state changes programmatically. This introduces stochasticity (the same screenshots may receive different judgments on different runs) and makes benchmark scores non-reproducible in the strict sense required for scientific comparison. The other camp — AndroidWorld (Rawles et al., 2024), AndroidLab (Xu et al., 2025), MobileAgentBench — achieved deterministic, programmatic verification by restricting themselves to applications whose internal state they could control: system apps, open-source apps without backend requirements, or apps where task completion can be verified through UI state alone. But this forced them to omit precisely the application categories — enterprise communication, social media, e-commerce — where much of real mobile usage concentrates. AndroidWorld, despite being the gold standard, has no Slack-equivalent, no Twitter-equivalent, and no shopping app.

The field's implicit assumption was that this tradeoff was structural: you can't have backend access to Gmail, therefore you can't deterministically verify email tasks, therefore if you want email tasks you must accept noisy evaluation. MobileWorld's key move is to substitute the application, not the capability. Rather than using Gmail and accepting noisy evaluation, or omitting email tasks to preserve determinism, it uses a self-hosted, production-grade open-source Mail app whose backend database is fully accessible — and designs email tasks that are functionally equivalent to what a Gmail-based benchmark would test. The same substitution applies across categories: Mattermost replaces Slack, Mastodon replaces Twitter/X, Mall4Uni/Taodian replaces Taobao.

Why this is a conceptual advance, not just an engineering convenience. The insight is that verification fidelity and application realism are orthogonal concerns that can be engineered independently — you don't need the same application for both. Realism of the agent interface (the screens, the interaction patterns, the task types) can be achieved through open-source apps that mimic commercial functionality, while verification fidelity can be achieved through backend access that would be impossible with the commercial originals. This decoupling means benchmark designers can optimize each concern separately rather than accepting a compromise. It also means the approach is extensible: as new application categories become important for mobile agents (health tracking, smart home control, financial trading), benchmark designers can identify open-source alternatives, fork and modify them for backend access, and integrate them into the containerized environment — without needing cooperation from commercial vendors.

Evidence anchor. The consequence is visible in Table 1, where MobileWorld is the only benchmark with checkmarks in both the "Backend-Req. Apps" and "Deterministic Evaluation" columns. It's also visible in Table 5's evaluation mode distribution: 47.3% of tasks use backend database verification and 36.8% use local storage inspection — verification methods that are only possible because the applications are self-hosted and instrumented. This isn't a minor implementation detail; it's the architectural foundation that makes the entire benchmark's claim to "reproducible, deterministic verification" credible for task categories (messaging, social media, e-commerce) where prior work had to accept evaluation noise.

Limits of the innovation. The approach trades off literal application identity for functional equivalence. An agent that succeeds on Mattermost tasks might fail on actual Slack if Slack's UI differs in ways that Mattermost doesn't capture — and Slack's UI does change over time, which a self-hosted fork won't track. The paper acknowledges this implicitly by describing the apps as "open-source alternatives" rather than claiming they're identical. The claim isn't that Mattermost is Slack, but that the capabilities tested — navigating channels, composing messages, searching history, managing group conversations — are the same capabilities enterprise communication agents need. Whether this functional equivalence holds across all task dimensions (UI layout conventions, error handling patterns, notification behavior) is an empirical question the paper doesn't investigate.

Innovation 2: Deliberate Information Withholding as a Capability Probe for Ambiguity Recognition

The agent-user interaction tasks are not simply "harder tasks" or "tasks with an extra step of asking a question." They represent a methodological innovation in capability probing: by systematically constructing tasks where critical information is provably unrecoverable through any sequence of GUI actions, the benchmark creates a controlled setting where the only path to success is recognizing one's own knowledge boundary and initiating clarification. This transforms ambiguity from an accidental property of lazy instruction-writing into a deliberate, calibrated experimental variable.

What the field did before. Every prior mobile agent benchmark — and most agent benchmarks in general — provided fully-specified task instructions. The implicit assumption was that task instructions are complete and unambiguous, and the agent's job is execution, not clarification. This assumption shapes agent design: architectures optimized under complete-instruction regimes learn to proceed confidently with whatever information they have, because in the training/evaluation distribution, proceeding confidently is always correct. The τ-bench (Yao et al., 2024) and τ2-bench (Barres et al., 2025) introduced user-agent dialogue for conversational agents in text-based customer service domains, but these operated in API-tool environments, not mobile GUI contexts, and their ambiguity arose from policy interpretation (what is the company's refund policy for this situation?) rather than from missing factual parameters (what is this person's email address?).

MobileWorld's distinctive contribution is the construction methodology for creating provably-ambiguous tasks. The four-step procedure described in Section 3.3 — write complete task, remove critical information, verify non-recoverability from device state, configure user agent with exactly the omitted information — ensures that the ambiguity is genuine (not solvable through clever GUI navigation), the required interaction is focused (the user agent only answers task-relevant questions), and the information asymmetry is controlled (the user agent knows exactly what the GUI agent needs, no more). The "verify non-recoverability" step is particularly important: the annotator must confirm that the omitted information is not discoverable through any sequence of GUI actions — checking Contacts, Email history, Messages, and any other app that might contain it. This prevents agents from succeeding through "cleverness" (finding the information through exhaustive search) rather than through the capability being tested (recognizing ambiguity and asking).

Why this is a diagnostic instrument, not just a task category. The paper's insight is that ambiguity recognition is a capability that can be measured independently of execution capability — and that current architectures lack it. Table 6 provides the stark evidence: end-to-end GUI models that achieve reasonable performance on fully-specified GUI tasks (e.g., Doubao-1.5-UI-TARS at 26.3% GUI-Only SR) show dramatically different behavior on interaction tasks (32.4%) — and this apparent "improvement" is misleading because, as Table 7 reveals, Doubao-1.5-UI-TARS achieves a UIQ of only 0.13, meaning it frequently issues redundant or ineffective clarification requests. The UIQ metric itself is an innovation in capability measurement, designed specifically to capture this distinction: it rewards agents that succeed with minimal queries (efficient clarification) while penalizing those that either fail to ask when needed or ask unnecessarily. The fact that UIQ scores are "consistently lower than User-Interaction SR across all models" (Section 4.5) means that even when agents succeed on interaction tasks, they're doing so inefficiently — asking too many questions, asking the wrong questions, or asking questions in contexts where no clarification is needed. This granularity transforms the evaluation from pass/fail into a profile of interaction quality.

Evidence anchor. The failure analysis in Section 4.6, Challenge 1 (Figure 6) provides the qualitative complement: the agent, given the instruction "I want to drive to Tianjin, China from my hometown. Please check the driving distance," hallucinates "Shanghai" as the departure city rather than asking the user to specify. This isn't a perception failure or a planning failure — it's a metacognitive failure: the agent doesn't recognize that "my hometown" is an ambiguous reference that it cannot resolve from the device state.

Significance beyond MobileWorld. The construction methodology — deliberate information withholding with verified non-recoverability — is transferable to other agent benchmarks. Any domain where agents receive natural language instructions can adopt this approach: deliberately omit parameters that are necessary for task completion, verify that the omitted parameters cannot be inferred from the environment, and instrument a simulated counterparty (user, customer, collaborator) with the missing information. This creates a controlled testbed for a capability — knowing what you don't know and asking about it — that is central to safe deployment but has been systematically neglected in benchmark design.

Innovation 3: MCP-Augmented Tasks as a Testbed for Strategic Tool Selection, Not Just Tool Execution

The MCP-augmented tasks in MobileWorld test something fundamentally different from standard tool-use benchmarks. Most tool-use evaluations (including OSWorld-MCP, which the paper cites) measure whether an agent can execute a tool correctly when it's provided — given a tool specification, can the agent format the right parameters and parse the response? MobileWorld's MCP tasks measure something more subtle: whether the agent can strategically choose between GUI operations and API calls when both pathways are available, and whether it can orchestrate the two modalities when neither alone suffices.

What the field did before. OSWorld-MCP (Jia et al., 2025) introduced MCP tool invocation into computer-use agent evaluation, but its focus was on tool execution fidelity: given a set of available tools, can the agent invoke the right one with the right parameters? Their key finding was that MCP tools improve task success rates (from 8.3% to 20.4% for OpenAI o3) but tool invocation rates remain low at 36.3% — suggesting that agents fail to recognize when to use tools even when they can execute them correctly. MCPWorld (Yan et al., 2025b) advanced this by evaluating API, GUI, and hybrid agents, but operated in desktop rather than mobile contexts and didn't construct tasks specifically to force the GUI-vs-API decision.

MobileWorld's contribution is the task construction methodology that forces the strategic choice. MCP-augmented tasks are built in two phases: first, design a subgoal that is solvable via MCP but non-trivial or error-prone through pure GUI; second, extend it with GUI-dependent follow-up actions that consume the MCP output. This structure means the agent faces a genuine decision: attempt the subgoal through GUI (tedious, error-prone, but familiar) or through MCP (efficient if done correctly, but requires correct tool selection, parameter formatting, and response parsing). The benchmark doesn't tell the agent which pathway to use — it only provides the tool specifications and the task instruction. The agent must recognize for itself that a GitHub repository's commit history is better fetched via API than by navigating github.com in Chrome, or that geocoding a city name is better done through Amap's MCP tool than through manual map navigation.

Why this is a distinct capability from tool execution. The distinction between tool execution and strategic tool selection maps onto a well-known challenge in agent design: the difference between knowing how and knowing when. An agent that can perfectly format MCP tool calls but never recognizes when a GUI operation would be more appropriate (or vice versa) would score well on execution-focused benchmarks but poorly on MobileWorld. Conversely, an agent with strong strategic reasoning but imperfect tool execution might outperform a better executor that makes worse modality choices. The paper's results in Table 7 provide suggestive evidence for this distinction: Qwen3-VL models make comparable numbers of MCP calls to agentic frameworks (2.32–3.84 vs. 1.91–2.63 average MCP calls per task) but achieve dramatically lower MCP success rates (0.0–5.4% vs. 48.6–51.6%). The paper attributes this to "inaccuracies in generated tool names and arguments" — an execution failure, not a selection failure. Meanwhile, Claude-4.5-Sonnet makes fewer MCP calls (1.91) than Gemini-3-Pro (2.63) but achieves higher MCP SR (50.0% vs. 48.6%), suggesting more selective tool use — calling tools only when they're likely to help, rather than calling them indiscriminately.

Evidence anchor. The example in Figure 12 illustrates the orchestration challenge concretely: the agent must (1) recognize that fetching commit history via gitHub_list_commits is the right approach, (2) correctly format the tool call with owner, repo, and perPage parameters, (3) receive the structured JSON response, (4) extract author and message fields from each commit object, (5) format them as "author: commit message" per the instruction's format constraint, and (6) switch to the Mail app and compose an email with the formatted content. Failure at any of these six steps — strategic (step 1), execution (step 2), parsing (steps 3–4), formatting (step 5), or GUI orchestration (step 6) — causes task failure. The benchmark's design ensures that partial capability (e.g., perfect tool execution but poor GUI orchestration) is measured as failure, not partial credit.

Significance beyond MobileWorld. As MCP adoption accelerates and agents increasingly operate in environments where GUI and API access coexist, the strategic selection problem will become central. MobileWorld provides the first systematic testbed for this capability in mobile contexts, and its construction methodology — design MCP-solvable subgoals, then extend with GUI-dependent follow-ups — provides a template for other domains. The finding that even the best agents achieve only ~51% on these tasks suggests this is an open research problem, not an engineering detail.

Innovation 4: A Granular Failure Taxonomy That Operates as a Research Roadmap

The paper's final contribution is methodological rather than architectural: a diagnostic framework that transforms benchmark evaluation from a ranking exercise into a capability decomposition. Rather than reporting aggregate scores and concluding "agents aren't good enough yet," the paper's error analysis in Section 4.6 identifies five distinct failure modes — each supported by concrete trajectory evidence — and characterizes which model architectures exhibit which failures. This turns the benchmark into an instrument that tells researchers what to work on next, not just how well current systems perform.

What the field did before. Most agent benchmarks report aggregate success rates, perhaps broken down by task category or difficulty. Some include qualitative error analysis, but this is typically illustrative rather than systematic — a few representative failures are shown, but there's no claim that they exhaustively characterize the capability gaps. The result is that benchmark papers tell you that systems fail, but not precisely why or what kind of research would address the failures.

MobileWorld's error analysis (Section 4.6) is more ambitious. The five challenges — ambiguity detection and user engagement, MCP context management, long-term memory and state tracking, complex logic reasoning, and temporal-spatial context awareness — are derived from "manual inspection of failed task trajectories across evaluated models" and are presented as an exhaustive categorization of the types of failure observed, not just examples. Each challenge is illustrated with a specific task and trajectory (Figures 6–8 and the accompanying text for Challenges 3–5), showing exactly where in the execution the agent went wrong and what capability would have been needed to succeed.

The taxonomy is significant because it partitions the problem space into research subproblems. Challenge 1 (ambiguity detection) is fundamentally a metacognitive capability — recognizing knowledge boundaries — that current end-to-end models almost completely lack (Qwen3-VL models score near 0% on interaction tasks). Challenge 2 (MCP context management) is an engineering problem — designing retrieval strategies and context window management for long tool responses — that affects even strong agentic frameworks. Challenge 3 (long-term memory) is an architecture problem — current agents have no mechanism for tracking which subtasks they've completed, leading to the destructive loops illustrated in Figure 8. Challenge 4 (complex logic reasoning) is a foundation model capability problem — numerical computation and multi-step reasoning remain unreliable even in frontier models. Challenge 5 (temporal-spatial awareness) is a grounding problem — agents don't connect their actions to the real-world context implied by device state (system clock, location settings).

Why this is more than qualitative color. Each challenge implies a different research direction and a different evaluation criterion. A researcher working on metacognitive architectures can measure progress against Challenge 1 by tracking agent-user interaction SR and UIQ. A researcher working on context management can measure progress against Challenge 2 by tracking MCP-augmented task SR with tools that return long responses. A researcher working on memory architectures can design experiments specifically targeting Challenge 3 tasks (the file renaming example in Figure 8 is essentially a unit test for state tracking). The paper doesn't just say "agents are bad at these things" — it provides the specific task subsets and metrics that operationalize each challenge, making the benchmark a diagnostic instrument rather than just a leaderboard.

Evidence anchor. The failure examples in Section 4.6 are chosen to illustrate distinct failure modes, and the paper is explicit that they represent "representative cases" identified through systematic inspection. The five-challenge taxonomy is presented as the summary of this inspection, not as an a priori framework that the benchmark was designed to test. This empirical grounding — the challenges emerged from the data rather than being imposed by the benchmark designers — gives them credibility as genuine capability gaps rather than artificial difficulty dimensions.

Limits and relationship to other sections. The challenge taxonomy doesn't propose solutions — it's descriptive, not prescriptive. But by mapping failures to specific capabilities, it provides a structure that the field can use to organize research efforts. The fact that the paper's own Conclusion section uses this taxonomy to outline future directions ("foundation model improvements through reinforcement learning" for Challenge 4, "agentic framework innovations addressing context length limitation" for Challenges 2 and 3) demonstrates its utility as a roadmap. This is a form of contribution — conceptual clarity about what problems need solving — that is distinct from the benchmark itself but enabled by its design.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use the full MobileWorld benchmark of 201 tasks, constructed as described in Section 3.3 and characterized in Table 5. The tasks span 20 applications across eight domains (Figure 4), with 116 GUI-Only tasks, 45 agent-user interaction tasks, and 40 MCP-augmented tasks. No separate train/validation/test split is used; the benchmark is evaluated in its entirety as a test set, with strategy selection (e.g., which agentic framework configuration) performed through direct comparison on the benchmark itself — the paper reports results without cross-validation.

  • Base model(s). The paper evaluates a broad spectrum of state-of-the-art models spanning two architectural paradigms. For the agentic framework (planner-executor architecture described in Section 4.1), the planners tested are: GPT-5 (OpenAI, 2025), Claude-4.5-Sonnet (Anthropic, 2025), and Gemini-3-Pro (Comanici et al., 2025) — all frontier proprietary LLMs accessed via official API. The grounding executor is UI-Ins-7B (Chen et al., 2025), a state-of-the-art GUI grounding model, fixed across all agentic framework runs. For end-to-end evaluation, the paper tests: GUI-Owl-7B, GUI-Owl-32B (Ye et al., 2025), UI-Venus-7B, UI-Venus-72B (Gu et al., 2025), Qwen3-VL-8B, Qwen3-VL-32B, Qwen3-VL-235B-A22B (Qwen Team, 2025), GELab-Zero-4B (Team, 2025), and Doubao-1.5-UI-TARS (Seed, 2025). Open-source models are deployed using vLLM (Kwon et al., 2023) on a server with 8 NVIDIA H20 GPUs; proprietary models are accessed through their official APIs. Temperature is set to 0.0 for all models.

  • Metrics. The paper defines five metrics (Section 4.2). Success Rate (SR) is the proportion of tasks where the agent achieves the task objective, producing a binary score s_i ∈ {0, 1} for each task i, with SR = (1/N) * Σ s_i (Equation 1). Category-specific SRs are reported for GUI-Only, agent-user interaction, and MCP-augmented subsets (Equation 2). Average Completion Steps (Ave. Steps) measures the mean number of action steps taken across all execution trajectories, including both successes and failures (Equation 3); under comparable success rates, lower values indicate greater efficiency. Average User Queries (Ave. Queries) counts the mean number of ask_user invocations on agent-user interaction tasks (Equation 4). User Interaction Quality (UIQ) is a composite metric that, for each interaction task, computes q_i = s_i / c_i if the agent invoked at least one ask_user action (and q_i = 0 otherwise), then normalizes Σ q_i by the sum of true interaction tasks plus any non-interaction tasks where the agent spuriously invoked ask_user (Equation 5–6). This rewards efficient clarification (success with few queries) and penalizes both failure to ask when needed and unnecessary queries on non-interaction tasks. Average MCP Calls (Ave. MCP Calls) measures the mean number of MCP tool invocations on MCP-augmented tasks (Equation 7).

  • Baselines. The primary comparisons are between the agentic framework (planner-executor with frontier LLMs + UI-Ins-7B) and end-to-end GUI-specialized models. Within the agentic framework, the three planner choices (GPT-5, Claude-4.5-Sonnet, Gemini-3-Pro) serve as baselines for each other. Within end-to-end models, comparisons span model scales (7B to 235B parameters) and architectures (GUI-Owl, UI-Venus, Qwen3-VL, GELab-Zero, Doubao-1.5-UI-TARS). Some end-to-end models cannot be evaluated on all task categories: MCP-augmented tasks are excluded for models without tool invocation support (GUI-Owl, UI-Venus, GELab-Zero, Doubao-1.5-UI-TARS), and agent-user interaction tasks are excluded for models without ask_user actions (GUI-Owl, UI-Venus).

  • Generation budget / compute accounting. All agents are evaluated with a maximum of 50 action steps per task (Section 4.3). This step budget is the universal unit of compute across all methods — an agent that exceeds 50 steps without invoking status is considered to have failed. The paper does not perform FLOPs-matched comparisons across model scales; all comparisons are at the same maximum step budget. Task execution is not wall-clock-time-constrained, though the environment does not artificially slow down agents.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. All 201 tasks are evaluated exactly once per agent configuration. The user agent for agent-user interaction tasks is implemented using GPT-4.1 (temperature 0.0), a fixed choice across all evaluated agent configurations — meaning the simulated user's behavior is deterministic and identical for all agents tested. Results are reported as point estimates without statistical significance testing.

Main Quantitative Results

Overall Success Rates and the Saturation Gap

Headline finding. The best agent achieves only 51.7% overall success rate on MobileWorld — compared to the 90%+ success rates reported on AndroidWorld by the same class of agentic frameworks — confirming that MobileWorld restores discriminative power. Table 6 presents the complete success rate breakdown.

The agentic framework with GPT-5 + UI-Ins-7B achieves the highest overall SR at 51.7%, substantially ahead of Gemini-3-Pro + UI-Ins-7B (46.3%) and Claude-4.5-Sonnet + UI-Ins-7B (43.8%). These numbers should be contextualized against the AndroidWorld saturation point: the paper reports that "state-of-the-art agentic frameworks now achieve success rates exceeding 90%" on AndroidWorld (Section 1), and Figure 1 visualizes this as "90+" vs. "51.7" for SOTA Accuracy on AndroidWorld and MobileWorld respectively. The 51.7% represents a dramatic drop — nearly halving the success rate — for the same class of architecture evaluated under comparable conditions (the GPT-5 agentic framework, which represents the current frontier).

The gap between agentic frameworks and end-to-end models is stark and consistent. The best end-to-end model, Doubao-1.5-UI-TARS, achieves only 20.9% overall SR — less than half of GPT-5's performance. Other end-to-end models cluster tightly at the low end: GELab-Zero-4B at 10.9%, UI-Venus-72B at 10.4%, Qwen3-VL-235B-A22B at 9.5%, Qwen3-VL-32B at 9.0%, and GUI-Owl-7B, GUI-Owl-32B, UI-Venus-7B, and Qwen3-VL-8B all hovering around 4.5–5.5%. This clustering near 5–11% for most end-to-end models — including the 235B-parameter Qwen3-VL variant — suggests a fundamental capability ceiling in current end-to-end architectures that does not improve substantially with model scale alone.

Category-Specific Performance Breakdown

GUI-Only tasks (116 tasks). These represent the task type most similar to prior benchmarks, requiring only standard GUI operations with fully-specified instructions. The best performance comes from Gemini-3-Pro + UI-Ins-7B at 55.6% GUI-Only SR, followed by GPT-5 + UI-Ins-7B at 54.0% and Claude-4.5-Sonnet + UI-Ins-7B at 47.8% (Table 6). This is notable because GPT-5 achieves the highest overall SR (51.7%) despite not having the highest GUI-Only SR — its strength in other categories compensates. Among end-to-end models, Doubao-1.5-UI-TARS leads at 26.3%, followed by UI-Venus-72B at 16.4%, GELab-Zero-4B at 16.1%, and Qwen3-VL-235B-A22B at 12.8%. The fact that even the strongest end-to-end model achieves only 26.3% on the "easiest" task category — tasks that are structurally similar to what these models were designed for — indicates that MobileWorld's GUI-Only tasks are substantially harder than those in prior benchmarks, even before considering the novel task categories.

The substantial gap between agentic frameworks (~48–56%) and end-to-end models (~8–26%) on GUI-Only tasks suggests that the long-horizon, cross-app, and multi-dimensional complexity built into these tasks (memory retention, numerical reasoning, precise instruction following) disproportionately challenges end-to-end architectures, which may lack the explicit planning and state-tracking mechanisms that the planner-executor decomposition provides.

Agent-user interaction tasks (45 tasks). These tasks require recognizing ambiguous instructions and proactively requesting clarification. The performance pattern here is the most revealing in the entire evaluation. GPT-5 + UI-Ins-7B achieves 62.2% interaction SR — higher than its GUI-Only SR (54.0%), indicating that GPT-5 is not merely capable of user interaction but actually performs better on tasks requiring interaction than on fully-specified GUI tasks. Claude-4.5-Sonnet + UI-Ins-7B follows at 37.8%, and Gemini-3-Pro + UI-Ins-7B at 24.4% (Table 6). The 37.8 percentage point gap between GPT-5 and Gemini-3-Pro on this task category is the largest single-model discrepancy in the benchmark and reveals that user interaction capability varies dramatically across frontier models — it is not a uniform strength.

Among end-to-end models, Doubao-1.5-UI-TARS achieves 32.4% interaction SR, while Qwen3-VL-235B-A22B manages only 4.4%, Qwen3-VL-32B achieves 6.7%, GELab-Zero-4B achieves 6.7%, and Qwen3-VL-8B scores 0.0% (Table 6). The paper explicitly notes that "most baseline models score below 10% on agent-user interaction" (Section 1), and the data bears this out: excluding Doubao-1.5-UI-TARS, no end-to-end model exceeds 6.7%. The Qwen3-VL family's near-zero performance — including the 235B variant — is particularly informative because these models were augmented with an ask_user action in their action space (Section 4.3), meaning the failure is not in lacking the capability to ask but in failing to recognize when asking is necessary. This is the metacognitive failure the benchmark was designed to detect.

MCP-augmented tasks (40 tasks). These tasks require hybrid orchestration between MCP tool invocations and GUI operations. GPT-5 + UI-Ins-7B achieves 51.6% MCP SR, followed by Claude-4.5-Sonnet + UI-Ins-7B at 50.0% and Gemini-3-Pro + UI-Ins-7B at 48.6% (Table 6). The relative compression among agentic frameworks on this category (only a 3 percentage point spread) contrasts with the wide spread on interaction tasks and suggests that MCP tool orchestration is a more uniform challenge — even the strongest models find it difficult, and model ranking is less differentiated.

End-to-end models show a near-total collapse on MCP tasks. Qwen3-VL-235B-A22B achieves 5.4%, Qwen3-VL-32B achieves 2.7%, and Qwen3-VL-8B achieves 0.0% (Table 6). All other end-to-end models (GUI-Owl, UI-Venus, GELab-Zero, Doubao-1.5-UI-TARS) simply cannot be evaluated on MCP tasks because they lack architectural support for tool invocation entirely. The paper reports these as "—" in Table 6 rather than 0.0%, making the distinction between "cannot attempt the task" and "attempts but always fails." Even the Qwen3-VL models, which were adapted to support MCP tool calls using the OSWorld-MCP format (Section 4.3), achieve results near zero because "a large proportion of tool invocation failed" due to "inaccuracies in generated tool names and arguments" (Section 4.5).

Execution Efficiency and Interaction Quality (Table 7)

Table 7 provides the detailed metrics beyond success rates, revealing behavior patterns that aggregate SR masks.

Average completion steps. Despite having the highest success rate, GPT-5 + UI-Ins-7B requires the most steps on average (27.8), reflecting that it persists longer on difficult tasks rather than terminating early. Gemini-3-Pro + UI-Ins-7B is the most efficient among high-performing agents, with an average of 24.2 steps at 46.3% SR — roughly 3.6 fewer steps per task than GPT-5, though with 5.4 percentage points lower success. The paper explicitly notes that "this higher tool utilization correlates with stronger MCP SR performance" for Gemini-3-Pro, suggesting a speed-accuracy tradeoff where Gemini-3-Pro is more decisive but sacrifices some hard tasks.

The step counts for low-performing end-to-end models must be interpreted carefully. Doubao-1.5-UI-TARS averages only 20.9 steps and GUI-Owl-7B averages 20.6 — but as Section 4.5 notes, "this is largely attributable to their lower success rates — these models often terminate early due to failure rather than efficient task completion." UI-Venus-72B's high step count (34.2) at low SR (10.4%) is flagged as "indicating inefficient behavior" — the model takes many actions but rarely succeeds, suggesting it gets stuck in loops or pursues incorrect strategies at length rather than recognizing failure.

User interaction quality. The UIQ metric (Table 7) reveals that all models' interaction efficiency is substantially below their interaction success rate — meaning that even when agents succeed on interaction tasks, they ask more questions than necessary, and they sometimes ask questions on non-interaction tasks where no clarification is needed. GPT-5 achieves UIQ of 0.40 on an interaction SR of 62.2%, with an average of 1.11 queries per interaction task. Claude-4.5-Sonnet achieves UIQ of 0.25 with 0.76 average queries. Gemini-3-Pro achieves UIQ of only 0.19 with 0.36 average queries — its low query count combined with its low interaction SR (24.4%) suggests it under-utilizes user clarification, failing to ask even when needed. The paper notes this explicitly: "This suggests it under-utilizes user clarification even when beneficial."

Doubao-1.5-UI-TARS presents an intriguing pattern: it achieves 32.4% interaction SR with 1.22 average queries (the highest query rate of any model), but its UIQ is only 0.13. The paper interprets this gap as evidence that "Doubao-1.5-UI-TARS frequently issues redundant or ineffective clarification requests" — it asks frequently but often asks the wrong questions or asks in situations where clarification doesn't help. This demonstrates the UIQ metric's diagnostic value: raw interaction SR can be inflated by models that ask promiscuously, while UIQ penalizes inefficient clarification.

Qwen3-VL models achieve 0.0 UIQ because they essentially never invoke ask_user (average queries of 0.00–0.04), meaning they fail interaction tasks by not recognizing ambiguity at all rather than by asking poorly.

MCP tool integration. Average MCP calls per task (Table 7) show an inverse relationship with success for Qwen3-VL models: Qwen3-VL-32B makes the most MCP calls on average (3.84) but achieves only 2.7% MCP SR, while GPT-5 makes 2.23 calls at 51.6% SR. The paper's explanation is that "due to inaccuracies in generated tool names and arguments, a large proportion of tool invocation failed" for Qwen3-VL — the models attempt to use tools but cannot correctly format the calls, so higher invocation counts don't translate to higher success. Among agentic frameworks, Gemini-3-Pro leads with 2.63 average MCP calls, followed by GPT-5 at 2.23 and Claude-4.5-Sonnet at 1.91, with MCP SR following the same ranking (48.6%, 51.6%, 50.0% respectively) — but the correlation between call count and success is weak, suggesting that call quality (correct tool selection and parameter formatting) matters more than call quantity.

Comparison with AndroidWorld: Step Distribution (Figure 5)

Figure 5 provides the most direct quantitative comparison between MobileWorld and AndroidWorld in terms of task length, measured under the same agentic framework (GPT-5 + UI-Ins-7B) at the same maximum step budget of 50. AndroidWorld tasks show a distribution concentrated in the 0–15 step range, with an average of 14.3 completion steps. MobileWorld tasks show a substantially right-shifted and right-skewed distribution, with an average of 27.8 steps — an increase of +13.5 steps (nearly double). The paper notes that "a large proportion [of MobileWorld tasks] require over 20 steps." This step-count differential is not merely a quantitative difference; it reflects the qualitative shift from single-app, short-horizon tasks to long-horizon, cross-app workflows that require sustained planning and memory. Since both benchmarks are evaluated at the same 50-step maximum, highly complex MobileWorld tasks that would require more than 50 steps will appear as failures — meaning the 51.7% SR likely understates what a larger step budget might achieve for the strongest agents.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of systematically removing components and measuring performance impact. The empirical design instead relies on cross-model comparisons and category-level performance breakdowns to isolate the effects of architectural choices and task characteristics. The following analyses serve the diagnostic function that ablations would in a systems paper.

Agentic framework vs. end-to-end architecture. The comparison between agentic frameworks (GPT-5, Claude-4.5-Sonnet, Gemini-3-Pro + UI-Ins-7B) and end-to-end models (all others) in Table 6 serves as an implicit architecture ablation. The planner-executor decomposition consistently outperforms end-to-end approaches by large margins (best agentic 51.7% vs. best end-to-end 20.9% overall), with the gap largest on MCP-augmented tasks (agentic ~50% vs. end-to-end ~0–5%) and smallest on GUI-Only tasks where Doubao-1.5-UI-TARS reaches 26.3% compared to Gemini-3-Pro's 55.6%. The paper attributes this gap to two factors: "insufficient capability in handling the complex reasoning and cross-app coordination required by MobileWorld" and "lack of support for agent-user interaction and MCP tool invocation" (Section 4.4). However, this is not a controlled ablation — the planner and executor models differ, the action spaces differ (end-to-end models have limited or no support for ask_user and mcp_call), and the prompting strategies differ — so the gap reflects the compound effect of multiple design choices rather than any single architectural decision.

Planner model scaling within agentic framework. The three frontier LLMs tested as planners (GPT-5, Claude-4.5-Sonnet, Gemini-3-Pro) with the same grounding executor (UI-Ins-7B) provide an implicit scaling comparison. GPT-5 leads overall (51.7%), followed by Gemini-3-Pro (46.3%) and Claude-4.5-Sonnet (43.8%). However, the ranking is not uniform across categories: Gemini-3-Pro leads on GUI-Only (55.6%), GPT-5 leads dramatically on interaction (62.2% vs. 37.8% and 24.4%), and all three are compressed on MCP (48.6–51.6%). This suggests that different frontier models have different capability profiles, and that the planner choice interacts with task category in ways that simple "stronger model = better performance" narratives don't capture. The paper does not test weaker planners (e.g., GPT-4o, Claude-3.5-Sonnet) to establish a scaling trend.

Model scale within end-to-end families. The Qwen3-VL family is evaluated at three scales: 8B, 32B, and 235B-A22B (Table 6). Overall SR increases from 5.5% (8B) to 9.0% (32B) to 9.5% (235B) — a surprisingly flat scaling curve, with the 42× parameter increase from 8B to 235B yielding only 4 percentage points of improvement. On GUI-Only tasks, the progression is 9.4% → 11.9% → 12.8%, again near-flat. On interaction tasks, it's 0.0% → 6.7% → 4.4% (non-monotonic!). On MCP tasks, it's 0.0% → 2.7% → 5.4%. This scaling behavior — minimal gains from massive parameter increases, and non-monotonicity on interaction tasks — is a strong negative result suggesting that current end-to-end architectures have a capability ceiling that scaling alone cannot overcome. The GUI-Owl family (7B vs. 32B) shows a similar flat pattern: 4.5% → 5.5% overall. UI-Venus (7B vs. 72B) shows somewhat more scaling benefit: 5.5% → 10.4% overall, but still modest relative to the 10× parameter increase.

MCP tool integration: Qwen3-VL with OSWorld-MCP format. The paper adapted Qwen3-VL models to support MCP tool calls using the OSWorld-MCP format (Jia et al., 2025), as described in Section 4.3. This adaptation constitutes an implicit ablation of MCP support: even with tool invocation capability added to the action space, Qwen3-VL models achieve near-zero MCP SR (0.0–5.4%) despite making comparable numbers of tool calls to agentic frameworks (2.32–3.84 vs. 1.91–2.63 average MCP calls per task, Table 7). The paper attributes this to "inaccuracies in generated tool names and arguments" — the models attempt to use tools but cannot execute them correctly. This reveals that adding MCP capability to an end-to-end model is not sufficient; the model must also be capable of using that capability correctly, which requires precise instruction following and structured output generation that current end-to-end architectures lack.

User agent consistency. The simulated user agent is implemented with GPT-4.1 at temperature 0.0 across all evaluated agent configurations (Section 4.3). This fixed choice means that all agents face the identical simulated user — when GPT-5 achieves 62.2% interaction SR and Gemini-3-Pro achieves 24.4%, the difference is entirely attributable to the GUI agent's behavior, not to variability in the user agent's responses. The paper does not ablate the user agent model (e.g., testing whether a weaker user agent makes interaction tasks easier or harder) or the temperature setting.

Completion step comparison with AndroidWorld (Figure 5). The side-by-side step distribution comparison under identical conditions (GPT-5 + UI-Ins-7B, 50 max steps) is a robustness check that the complexity difference between MobileWorld and AndroidWorld is genuine and not an artifact of different evaluation protocols. The +13.5 step average increase is measured under the same agent, same framework, and same step budget, confirming that MobileWorld tasks are structurally longer.

Task category difficulty ordering. The per-category SR breakdown in Table 6 reveals a consistent difficulty ordering across agentic frameworks: interaction tasks are easiest for GPT-5 (62.2%) but hardest for Gemini-3-Pro (24.4%); MCP tasks are consistently challenging for all agentic frameworks (48.6–51.6%); GUI-Only tasks show the widest spread (47.8–55.6%). The absence of a uniform difficulty ordering (where one category is always hardest) supports the paper's claim that different task categories test different capabilities — if all categories simply reflected "general agent capability" with varying difficulty, we would expect a uniform ranking, but instead we see model-specific strengths and weaknesses.

Critical Assessment

This section evaluates whether MobileWorld's experimental results substantiate the paper's central claims, identifying what is demonstrated, what is not, and what additional experiments would strengthen the evidence.

Claim: MobileWorld effectively addresses benchmark saturation.

The evidence for this claim is strong and direct. The best agent achieves 51.7% overall SR (Table 6) compared to 90%+ on AndroidWorld — a roughly 40 percentage point drop for the same class of frontier agentic framework. This gap is large enough to restore discriminative power: the spread between the best (GPT-5, 51.7%) and worst (GUI-Owl-7B, 4.5%) is 47.2 percentage points, providing ample headroom for future improvements. The category-level breakdown reveals that even within the "easiest" category (GUI-Only), the best agent achieves only 55.6%, confirming that difficulty is not merely driven by the novel task types but is distributed across the benchmark. However, AndroidWorld's 90%+ figure is cited from the AndroidWorld leaderboard and not replicated under the paper's exact evaluation conditions — the paper reports that GPT-5 + UI-Ins-7B averages 14.3 steps on AndroidWorld (Figure 5), implying that it achieves high success rates there, but no AndroidWorld SR is explicitly reported for this specific framework. A direct SR comparison with identical agent configurations on both benchmarks would make this claim more precise.

Claim: Novel task categories (agent-user interaction, MCP-augmented) expose fundamental capability gaps.

This claim is well-supported by the quantitative results, though with important nuance. On interaction tasks, the evidence is stark: Qwen3-VL models score 0.0–6.7% despite having ask_user in their action space; GELab-Zero-4B scores 6.7%; and even the best end-to-end model (Doubao-1.5-UI-TARS) reaches only 32.4% with inefficient clarification (UIQ of 0.13). The metacognitive failure — not recognizing when clarification is needed — is clearly demonstrated. However, the claim that these are "fundamental" gaps (as opposed to gaps that could be closed by fine-tuning on interaction data) is not directly tested: end-to-end models were not fine-tuned on interaction tasks, so their poor performance could reflect zero-shot generalization failure rather than an inherent architectural limitation. An experiment where end-to-end models are fine-tuned on interaction-style data and re-evaluated would distinguish between these interpretations.

On MCP tasks, the evidence for "fundamental" gaps is stronger for end-to-end models (most cannot even attempt the tasks, and those that can score near zero) but weaker for agentic frameworks, where performance is a non-trivial ~50%. The paper claims "current models struggle to effectively orchestrate between GUI actions and external tool invocations" (Section 4.4), but ~50% is more accurately characterized as "partial capability" rather than "struggle" — these models succeed on roughly half of MCP-augmented tasks. The failure analysis (Challenge 2, Figure 7) identifies context overflow from long MCP responses as a specific failure mode, but this is presented as a single illustrative example rather than a systematic characterization of what causes the other ~50% of MCP task failures.

A missing experiment that would clarify the nature of MCP task difficulty: testing agentic frameworks on a version of MCP-augmented tasks where the MCP output is pre-summarized or truncated, to determine whether the primary bottleneck is tool execution (formatting the right call), context management (handling long responses), or orchestration (integrating tool output with GUI actions). The current results conflate these three failure modes.

Claim: Performance collapse on novel categories reveals "fundamental deficiencies in current architectures regarding active user interaction and tool-use."

This claim requires careful qualification. For end-to-end architectures, the evidence is strong — models that were not designed for tool invocation cannot invoke tools, and models that lack explicit uncertainty modeling do not ask for clarification. But the paper presents these as architectural deficiencies when they may be training data deficiencies. An end-to-end model trained on trajectories that include ask_user actions and MCP tool calls might perform substantially better — the paper does not test this because it evaluates off-the-shelf models rather than fine-tuned variants. The claim that deficiencies are "fundamental" to the architecture is an overstatement given that no architectural modification experiments were run (e.g., adding an explicit uncertainty estimation module to an end-to-end model, or comparing with and without tool-use training).

For agentic frameworks, the interaction SR of 62.2% (GPT-5) and MCP SR of 51.6% (GPT-5) suggest these architectures are partially capable of user interaction and tool use — the deficiency is one of degree, not absence. The paper's framing of these as "fundamental deficiencies" overstates the case for agentic frameworks while accurately characterizing the end-to-end situation.

Claim: Long-horizon, cross-application workflows create substantially greater difficulty.

The step count comparison (Figure 5) provides strong quantitative support: 27.8 vs. 14.3 average steps, with a right-shifted distribution. The cross-app statistics (62.2% multi-app tasks vs. 9.5% in AndroidWorld) support the structural claim. However, difficulty is inferred from success rates — the paper doesn't run an ablation where task length is varied while holding other dimensions constant, or where cross-app requirements are added to otherwise identical tasks. The contributions of individual difficulty dimensions (memory retention, numerical reasoning, fine-grained visual recognition, etc.) to the performance gap are not isolated. A researcher reading these results cannot determine whether the 51.7% overall SR is primarily bottlenecked by, say, memory retention or by numerical reasoning — the failure analysis (Section 4.6) provides qualitative examples of each failure mode but no quantitative decomposition of their relative frequencies.

Assessment of specific experimental design choices.

The single-step-budget design (all agents evaluated at max 50 steps) is a reasonable practical choice but creates a confound: agents that are slower but more methodical may hit the step limit on complex tasks, artificially depressing their SR relative to faster but comparably capable agents. The absence of any step-budget sensitivity analysis (e.g., evaluating GPT-5 at 30, 50, 75, and 100 steps) means we cannot tell whether the 51.7% SR represents a capability ceiling or a budget ceiling.

The absence of cross-validation or repeated trials is a limitation for a benchmark claiming to enable "rigorous community benchmarking." With 201 tasks and a single evaluation run per agent configuration, the reported SRs are point estimates without confidence intervals. A 1.0 percentage point difference between two agents on a 201-task benchmark has a standard error of roughly 0.7 percentage points under a binomial model — meaning small differences are statistically indistinguishable. The gap between Gemini-3-Pro (46.3%) and Claude-4.5-Sonnet (43.8%) is 2.5 percentage points, which is likely significant, but gaps of 1–2 percentage points at the high end of the table may not be reliable. The paper's leaderboard-style reporting without uncertainty quantification is standard for the field but limits the benchmark's utility for fine-grained comparisons.

The fixed user agent (GPT-4.1, temperature 0.0) is a strength for reproducibility — all GUI agents face the identical simulated user. However, it also means that agent-user interaction SR depends on the match between the GUI agent's communication style and GPT-4.1's interpretation patterns. A GUI agent that phrases clarification questions in a way that GPT-4.1 reliably understands would have an advantage unrelated to its general interaction capability. The paper does not test with alternative user agent models (e.g., Claude, Gemini) to assess robustness to this choice.

Experiments that would strengthen the paper.

Several experiments are conspicuous by their absence and would substantially increase confidence in the paper's claims:

  1. Step budget sensitivity analysis. Evaluating GPT-5 + UI-Ins-7B at multiple step budgets (e.g., 25, 50, 75, 100) would reveal whether the 51.7% SR is near the asymptotic performance of this architecture on MobileWorld or whether additional steps would yield substantial gains. If performance plateaus at 50 steps, the step budget is not a confound; if it continues rising, the reported SRs understate agent capability.

  2. Fine-tuned end-to-end baselines. Evaluating an end-to-end model fine-tuned on trajectories that include ask_user and mcp_call actions would distinguish between "current off-the-shelf models can't do this" and "end-to-end architectures fundamentally can't do this." Without this, the claim of "fundamental deficiencies" is unproven for end-to-end architectures.

  3. MCP difficulty decomposition. A controlled experiment where MCP-augmented tasks are modified to pre-truncate tool responses (removing context overflow as a failure mode) or to pre-select the correct tool (removing tool selection as a failure mode) would identify the primary bottleneck in MCP task performance. This would guide future research toward context management, tool selection, or response parsing.

  4. AndroidWorld replication under identical conditions. Reporting AndroidWorld SR for GPT-5 + UI-Ins-7B under the same 50-step budget would make the "saturation" claim precise: "GPT-5 achieves X% on AndroidWorld vs. 51.7% on MobileWorld" is a stronger statement than "agents achieve 90%+ on AndroidWorld" (a claim about unspecified agents).

  5. Human performance baseline. The paper's human-in-the-loop validation protocol (Section 3.3) confirms that all tasks are solvable by humans, but no systematic human evaluation is reported. A human performance baseline on a subset of tasks would contextualize the 51.7% SOTA — is this 51.7% of the way to human performance, or is human performance near 100% on these tasks, making 51.7% genuinely low?

  6. Failure mode frequency analysis. The five research challenges in Section 4.6 are illustrated with representative examples, but the paper does not report how frequently each failure mode occurs across the evaluated models. Knowing that "long-term memory failures account for 30% of GPT-5's errors" vs. "MCP context management accounts for 45%" would substantially sharpen the research roadmap.

Overall assessment. The experimental results provide strong support for MobileWorld's primary claim — that it is substantially more challenging than AndroidWorld and that current models (especially end-to-end architectures) have significant room for improvement. The category-level performance breakdown convincingly demonstrates that novel task types (interaction, MCP) expose capability gaps not measured by prior benchmarks. However, the paper's stronger claims about "fundamental deficiencies" in current architectures, and the precise characterization of why agents fail, are supported more by illustrative examples than by systematic decomposition. The benchmark's diagnostic potential — its ability to tell researchers which capability to improve — is established in principle but not yet realized quantitatively, because the relative contribution of different failure modes to the performance gap is not measured. These limitations are largely typical of benchmark papers (which prioritize breadth of evaluation over depth of analysis for any single model) and do not undermine the benchmark's value as a more challenging and more realistic evaluation instrument, but they do mean that the paper's research roadmap claims should be treated as hypotheses to be validated rather than firm conclusions from the data.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Claims

The assumption or constraint. The agent-user interaction and MCP-augmented task paradigms in MobileWorld require the benchmark infrastructure to know in advance which tasks need user interaction and which MCP tools to provide — this is pre-configured per task at benchmark construction time. However, the paper's research roadmap (Challenges 1 and 2 in Section 4.6) envisions agents that autonomously detect ambiguity and strategically select tools. In any real deployment of an agent inspired by these challenges, the agent would need to decide for itself when clarification is needed or when MCP tools should be invoked — a decision that may require additional inference compute (e.g., an explicit uncertainty estimation step, or multiple LLM calls to evaluate whether the current instruction is ambiguous). The paper does not account for this decision-making overhead in any step-count or compute budget. The 50-step maximum budget treats an ask_user action as costing one step (the same as a click), which is reasonable for the simulated environment where the user agent responds instantly. But the metacognitive process of recognizing that one should ask — which for GPT-5 involves analyzing the instruction, the device state, and the interaction history to detect ambiguity — is implicitly amortized into the planner's standard reasoning per step. In practice, robust ambiguity detection might require explicit verification steps, separate uncertainty-classifier models, or additional LLM calls, none of which are modeled or costed.

The consequence. The 51.7% overall SR achieved by GPT-5 + UI-Ins-7B is measured in a setting where the task category (GUI-Only, interaction, MCP) determines what action types are needed, and the agent does not incur additional cost to discover this. A real-world agent that must autonomously determine "is this instruction ambiguous?" and "should I use an MCP tool or navigate via GUI?" would either (a) make these decisions as part of its standard per-step reasoning (adding latency and potentially reducing the effective step budget for task execution), (b) use a separate classifier or verification step (adding compute not captured in the step count), or (c) risk making incorrect modality decisions that lead to task failure. The paper's step-count metric and SR comparisons would not capture the cost of (a) or (b), and would conflate modality-selection errors with execution errors in case (c). A practitioner building a deployable agent based on MobileWorld's findings would discover that the benchmark's task structure (where the need for interaction or MCP is designed into the task instruction and tool availability is pre-configured) provides a cleaner signal than real-world instructions, where ambiguity is often subtle and tools may or may not be relevant. The performance achievable in practice may be lower than the benchmark numbers suggest because the agent must solve a harder meta-decision problem.

What evidence exists in the paper. The paper does not measure or discuss the computational cost of the metacognitive processes required for the novel task categories. Section 4.5 analyzes interaction quality through UIQ and average queries, but this measures the outcome of the decision (did the agent ask? how many times?) rather than the cost of making the decision. Section 4.6, Challenge 1 (Figure 6) shows a failure case where the agent doesn't recognize ambiguity, but doesn't quantify what additional compute would be needed to reliably detect such ambiguity. The step-count comparison with AndroidWorld (Figure 5) shows a +13.5 step increase, but this reflects longer task execution, not additional metacognitive overhead, because the pre-configured task structure means agents don't need to spend steps deciding whether to enter an interaction or MCP mode.

Mitigation status. The paper does not address this limitation. The benchmark is designed as an evaluation instrument, not a deployment recipe, so the absence of metacognitive cost modeling is understandable for a benchmark paper. However, the paper's framing of the five research challenges (Section 4.6) as a "roadmap toward more autonomous and user-aware mobile intelligence" (Section 1) implies deployment relevance that would require solving this cost problem. Future work could address it by: (a) measuring the inference FLOPs or wall-clock time consumed by the planner's reasoning per step, not just step count, to capture the cost of metacognitive deliberation; (b) designing tasks where the need for interaction or MCP is not signaled by the task category, forcing agents to discover it and measuring the discovery cost; or (c) reporting a separate "modality selection accuracy" metric that isolates the metacognitive decision from execution capability.

Limited to a Single Domain (Math-like Mobile Tasks) with No Evidence of Cross-Domain Generalization

The assumption or constraint. All 201 tasks in MobileWorld are Android mobile GUI tasks — the benchmark does not include iOS tasks, desktop GUI tasks, web-based tasks, or non-GUI mobile interactions (voice, gesture, sensor-based). Within the Android mobile domain, the application coverage (Table 8, Appendix B) spans eight categories, but all tasks require structured, goal-directed workflows with verifiable completion states — there are no open-ended creative tasks, no tasks requiring subjective quality judgments, no tasks where "success" is ambiguous, and no tasks involving real-time multi-agent coordination. The paper explicitly positions MobileWorld as addressing the saturation of AndroidWorld and evaluates only on the Android platform using a specific set of 20 applications. The five research challenges identified in Section 4.6 are derived from failure analysis on this specific task distribution.

The consequence. The capability gaps identified — particularly around ambiguity detection, MCP context management, and long-term memory — are measured on tasks that share structural properties: they are goal-directed, have deterministic success criteria, involve navigating structured app interfaces, and use synthetic or pre-loaded data rather than live real-world data. It is unknown whether the same capability gaps appear with similar severity on: (1) tasks requiring subjective judgment (e.g., "find a thoughtful birthday gift for my sister within my budget" — where success depends on user satisfaction, not a verifiable state change); (2) tasks on different platforms with different UI paradigms (iOS's navigation patterns differ from Android's; desktop applications have different interaction affordances); (3) tasks with live, changing data (real email inboxes, real social media feeds, real e-commerce listings) where the agent must handle temporal dynamics absent from MobileWorld's snapshot-based initialization; or (4) tasks where the appropriate tool choice is ambiguous or involves tradeoffs (e.g., should the agent use an MCP financial data tool that costs API credits, or scrape a free but slower web source?). The paper's claim to reflect "real-world mobile usage" (Section 1) is qualified by this domain specificity — it reflects a particular slice of real-world usage (structured, verifiable, goal-directed Android tasks) and may not generalize to other slices.

A practitioner evaluating whether MobileWorld results predict their agent's performance in a different domain — iOS deployment, desktop automation, customer service with subjective success criteria, enterprise workflows with live data — would find no evidence in the paper to support such generalization. The benchmark's difficulty relative to AndroidWorld is established, but its difficulty relative to, say, OSWorld (Xie et al., 2024) or WindowsAgentArena (Bonatti et al., 2024) is unknown. The relative ranking of models (GPT-5 > Gemini-3-Pro > Claude-4.5-Sonnet for overall SR) might not hold on a different task distribution — Gemini-3-Pro leads on GUI-Only tasks (55.6% vs. GPT-5's 54.0%), suggesting that model ranking is category-dependent within MobileWorld itself, and might shift further on tasks with different characteristics.

What evidence exists in the paper. The paper provides comprehensive statistics on MobileWorld's task distribution (Table 5, Figure 4, Table 8) but does not compare results to any non-mobile benchmark. The related work section (Section 2) surveys desktop and web benchmarks (OSWorld, WebArena, WindowsAgentArena) as context, but no cross-benchmark evaluation is performed — none of the evaluated agents are tested on both MobileWorld and a non-mobile benchmark to assess whether MobileWorld performance correlates with general agent capability. The paper's comparison to AndroidWorld is limited to the step-count distribution (Figure 5) and the headline saturation claim (90%+ vs. 51.7%), without a systematic correlation analysis (e.g., per-task success correlation between the two benchmarks for the same agent). The application coverage (Table 8) reveals that some categories have very few tasks — Camera (3 tasks), Clock (7 tasks), Settings (7 tasks) — making per-category SR estimates noisy and limiting conclusions about domain-specific agent strengths.

Mitigation status. The paper does not address domain generalization. The title and abstract position MobileWorld as a benchmark for "autonomous mobile agents" without qualifying the domain scope, though the Introduction (Section 1) describes it as evaluating "mobile GUI agents" and the experiments are Android-specific. The paper's "future research directions" (Section 5) suggest "foundation model improvements through reinforcement learning" and "agentic framework innovations" but do not mention cross-platform or cross-domain evaluation. This is a standard limitation of single-benchmark papers and does not diminish MobileWorld's contribution as an Android benchmark, but it means that the paper's broader claims about "next-generation mobile intelligence" and "real-world automation complexity" should be interpreted as applying to the Android mobile domain specifically, pending replication on other platforms and task types.

The Benchmark's Fixed Step Budget (50 Steps) Creates a Confound Between Agent Capability and Timeout-Induced Failure

The assumption or constraint. All evaluated agents operate under a uniform maximum of 50 action steps per task (Section 4.3). An agent that has not invoked the status action within 50 steps is treated as having failed the task — the evaluator does not assess partial progress or award partial credit. This step budget was chosen to match the AndroidWorld evaluation protocol and to keep total evaluation time tractable across 201 tasks and ~15 model configurations. However, the step-count analysis (Figure 5) reveals that MobileWorld tasks have an average completion length of 27.8 steps under GPT-5 + UI-Ins-7B, with a right-skewed distribution where a substantial fraction of tasks require more than 40 steps. This means that some fraction of tasks — particularly the most complex multi-app, long-horizon workflows — are bumping against the 50-step ceiling. The paper does not report what fraction of failures are due to step-budget exhaustion vs. incorrect action sequences.

The consequence. The reported success rates are a lower bound on agent capability — an agent that would succeed at step 55 is counted as a failure, indistinguishable from an agent that takes an incorrect action at step 5 and never recovers. This creates a confound between execution speed (how many steps an agent needs to complete a task) and execution accuracy (whether the agent chooses correct actions). An agent that is more methodical, verifies its work at intermediate steps, or engages in thorough clarification dialogue (multiple ask_user calls to refine ambiguous instructions) may consume more steps and hit the budget ceiling on complex tasks, artificially depressing its SR relative to a faster but less careful agent. Conversely, an agent that rushes through tasks with fewer verification steps might complete them within 50 steps but make more errors — the budget constraint rewards speed over thoroughness in a way that may not reflect deployment priorities, where accuracy is typically valued over per-task latency (within reason).

The interaction between step budget and task complexity is particularly acute for MCP-augmented tasks. These tasks require multiple modalities (MCP tool call → parse response → GUI navigation → compose output), and each MCP call with a large response may require additional steps to scroll through, extract relevant information from, or re-query the tool. An agent that makes two MCP calls with 20k-token responses (as in the Challenge 2 example, Figure 7) may spend many steps just managing context, leaving fewer steps for the GUI portion of the task. The 50-step budget may systematically penalize MCP-augmented tasks more than GUI-Only tasks, making the MCP SR gap partially attributable to budget constraints rather than purely to tool-use capability.

What evidence exists in the paper. Figure 5 shows that MobileWorld tasks have a substantially right-shifted step distribution compared to AndroidWorld, with many tasks requiring 40+ steps. The paper does not report the distribution of failed task trajectories by step count, so it's impossible to determine what fraction of the 48.3% of tasks that GPT-5 fails are near-misses (failed at step 45–50) vs. early failures (failed at step 5–15). The paper notes that UI-Venus-72B requires the highest average steps (34.2) at low SR (10.4%), flagging this as "inefficient behavior" (Section 4.5), but does not analyze whether extending the step budget would improve its SR (suggesting capability masked by the budget) or not (confirming genuinely inefficient behavior). The completion step comparison in Table 7 reports means across all trajectories (successes and failures), obscuring the distribution tail.

Mitigation status. The paper acknowledges this limitation implicitly by choosing 50 steps as a practical evaluation constraint rather than a claim about sufficient budget. Section 4.3 states "All models are evaluated with a maximum of 50 steps" without arguing that this budget is adequate or analyzing sensitivity to the choice. The paper does not propose varying the step budget in future work. A simple and informative experiment — evaluating the best agent (GPT-5 + UI-Ins-7B) at multiple step budgets (25, 50, 75, 100) and reporting the SR scaling curve — would reveal whether performance is near-asymptotic at 50 steps or would continue improving substantially with additional budget. This would also provide a more nuanced performance metric than a single-point SR: the step-efficiency curve (SR vs. step budget) characterizes both an agent's ultimate capability ceiling and its speed of convergence, which is more informative for deployment planning than a single timeout-limited SR.

Verifier Quality and Reliability: The Four Verification Mechanisms Are Not Validated for Coverage or Correctness

The assumption or constraint. MobileWorld's deterministic evaluation rests on four verification mechanisms — textual answer matching, backend database verification, local storage inspection, and application callbacks — that are claimed to "ensure that task outcomes are evaluated deterministically, eliminating the evaluation noise inherent in MLLM-as-a-judge approaches" (Section 3.2). Each task is assigned exactly one verification method at construction time (Table 5: 47.3% database, 36.8% storage, 10.9% textual, 5.0% callbacks). The paper assumes that these verification methods correctly determine task success with 100% accuracy — no false positives (marking a failed task as successful) and no false negatives (marking a successful task as failed). However, the paper does not report any systematic validation of the verifiers themselves. The human-in-the-loop task validation protocol (Section 3.3) ensures that tasks are solvable by humans (a human can achieve a score of 1.0), but this validates the task design and evaluator jointly — it does not independently validate that the evaluator correctly classifies all possible final states.

The consequence. Several failure modes could compromise evaluation accuracy, and their likelihood varies by verification method. For textual answer matching (22 tasks), the evaluator uses regular expressions or exact string comparison. This is vulnerable to format mismatches: if the ground truth is "11" (the temperature in Celsius) and the agent answers "11°C" or "11 degrees" or "The temperature is 11°C," the regex must be flexible enough to extract the correct answer without also extracting false positives from incorrect answers that coincidentally contain "11." The paper's task instructions explicitly constrain output format (e.g., "Response only one integer number. No other text") to mitigate this, but agents may still violate these constraints. A task where the ground truth temperature is 11 but the agent says "11" in a different context (e.g., mentioning a different temperature of 11 by coincidence) could produce a false positive if the regex is too permissive, or a false negative if the regex is too strict and rejects a valid answer.

For backend database verification (95 tasks) and local storage inspection (74 tasks), the evaluator queries the application's database or file system after the agent signals completion. These methods assume that the database/storage state is a complete and accurate representation of task completion. However, state changes can occur for reasons unrelated to the agent's actions: a background process in the self-hosted application could modify the database, the snapshot restoration could be imperfect (leaving residual state from a previous evaluation run), or the agent could achieve the database state change through an unintended path (e.g., sending an email by exploiting a bug rather than through the intended GUI flow). The paper does not report whether the Docker-based snapshot restoration is tested for perfect fidelity (are there any database fields that fail to reset? any race conditions in the backend services?), and does not test whether the evaluator correctly handles edge cases (e.g., multiple emails with the same subject line — does the evaluator check the correct one?).

For application callbacks (10 tasks), the callback mechanism is custom-instrumented into the Mail and Taodian apps. This means the evaluation depends on code that is not part of the original application and could contain bugs — missing callback firings for certain interaction paths, incorrectly serialized callback data, or callbacks that fire in a different order than expected. The paper does not report testing the callback instrumentation against ground-truth human execution across all interaction paths.

The systematic consequence is that MobileWorld's headline SR numbers (51.7% for GPT-5, etc.) may contain evaluation errors whose magnitude and direction are unknown. If verifier false negatives dominate, the reported SRs understate true agent capability. If false positives dominate, they overstate it. Even a 2–3% error rate across 201 tasks (4–6 misclassified tasks) would shift SR by 1–1.5 percentage points — comparable to the gap between some model pairs in Table 6 (e.g., Gemini-3-Pro at 46.3% vs. Claude-4.5-Sonnet at 43.8%, a 2.5 percentage point difference). This means fine-grained model comparisons at the top of the leaderboard may not be reliable without verifier validation.

What evidence exists in the paper. The paper's human-in-the-loop validation protocol (Section 3.3) provides partial evidence: human annotators execute each task and must achieve a score of 1.0 from the evaluator for the task to be included in the benchmark. This confirms that the evaluator does not produce false negatives for human execution paths — the evaluator correctly identifies when a human has completed the task. However, this does not validate against false positives (could an agent achieve a 1.0 without actually completing the task?) or false negatives on non-human execution paths (could an agent complete the task in a way the human didn't try, which the evaluator incorrectly rejects?). The five-retry limit in the validation protocol also means that tasks requiring more than 5 human attempts were redesigned (Section 3.3), which may filter out tasks with particularly subtle evaluation criteria but does not provide a quantitative measure of verifier reliability.

Mitigation status. The paper does not report verifier validation experiments and does not discuss evaluation accuracy as a potential limitation. This is a gap shared by most benchmark papers — verifier validation is rarely performed systematically — but it is particularly consequential for MobileWorld because the benchmark's key innovation over prior work is precisely its claim to deterministic, noise-free evaluation. The paper's strong contrast with "MLLM-as-a-judge" approaches (Section 3.2, Table 1) sets a high bar for evaluation reliability that the paper does not empirically validate. Future work could address this by: (a) running human annotators on a sample of agent-generated task completions (both successes and failures as judged by the verifier) to measure verifier precision and recall; (b) testing snapshot restoration fidelity by running the same task twice from the "same" snapshot and verifying identical initial states; (c) adversarial testing — deliberately designing agent trajectories that should fail but might trick the verifier (e.g., sending an email with the right content but to the wrong recipient, where the database query might accidentally match).

End-to-End Model Baselines Are Tested Out-of-Distribution Without Fine-Tuning, Limiting Conclusions About Architecture vs. Training

The assumption or constraint. The paper evaluates end-to-end GUI models (GUI-Owl, UI-Venus, Qwen3-VL, GELab-Zero, Doubao-1.5-UI-TARS) in a zero-shot setting on MobileWorld — these models were not fine-tuned on MobileWorld tasks, were not exposed to MobileWorld's specific applications during training, and (for most) were not designed to support the ask_user or mcp_call actions. The Qwen3-VL models were adapted to support these actions by augmenting their action space (Section 4.3), but this is a prompting-level adaptation, not fine-tuning. The paper acknowledges these constraints: MCP-augmented tasks are excluded for models without tool invocation support; agent-user interaction tasks are excluded for models without user query actions. The paper interprets the resulting low performance (4.5–20.9% overall SR for end-to-end models, Table 6) as evidence of "fundamental deficiencies in current architectures regarding active user interaction and tool-use" (Section 4.4) and as revealing "a fundamental deficiency in current architectures" (Section 1).

The consequence. The inference that poor end-to-end performance reflects architectural deficiencies rather than training distribution mismatch is not supported by the experimental design. An alternative explanation is equally consistent with the data: end-to-end models are capable of the required behaviors (recognizing ambiguity, invoking tools, following long-horizon plans) but were not trained on data that includes these behaviors, and would perform substantially better if fine-tuned on MobileWorld-style tasks. The paper's own agentic framework results provide indirect evidence for this alternative: the planner-executor architecture succeeds not because it has a fundamentally different neural architecture, but because the planner (GPT-5) is a general-purpose LLM with broad training that includes instruction-following, tool-use, and dialogue abilities, while the grounding executor handles the perception subtask. An end-to-end model fine-tuned on trajectories that include ask_user and MCP tool calls might close much of the gap without architectural changes.

This distinction matters for the paper's research roadmap claims. If the gap is primarily a training data problem, the prescription is "collect training data that includes user interaction and tool-use trajectories, then fine-tune end-to-end models" — a data engineering solution. If the gap is primarily an architectural problem, the prescription is "design new architectures with explicit uncertainty modeling, memory mechanisms, and modular tool-use components" — an architecture research solution. The paper leans toward the latter interpretation (Section 4.6 challenges describe "fundamental" gaps requiring "agentic framework innovations") but the experiments do not distinguish between these explanations. A practitioner choosing between investing in better training data vs. a new architecture for their mobile agent product would find no experimental evidence in the paper to guide this decision.

The comparison between end-to-end models and agentic frameworks is further confounded by model scale and training compute. GPT-5 is a massive frontier model with unknown (but certainly enormous) training compute; Qwen3-VL-235B-A22B is the largest end-to-end model tested but is a Mixture-of-Experts architecture with different training objectives. The paper does not control for total pretraining compute, making it impossible to determine whether the agentic framework's advantage comes from the planner-executor decomposition or simply from using a much more capable base model as the planner.

What evidence exists in the paper. The paper reports end-to-end model performance in Table 6 (overall and per-category SR) and Table 7 (efficiency metrics). The results clearly show large gaps between agentic frameworks and end-to-end models, and between end-to-end models on GUI-Only vs. interaction/MCP tasks. However, the paper does not include any fine-tuned baselines — no model is fine-tuned on MobileWorld training data (the benchmark has no training split) or on synthetic trajectories that include interaction and MCP actions. The scaling analysis within model families (Qwen3-VL 8B → 32B → 235B; GUI-Owl 7B → 32B; UI-Venus 7B → 72B) shows minimal or non-monotonic improvements from scale (Section 5 ablation discussion), but this is evidence about the limits of zero-shot scaling, not about the limits of fine-tuned scaling. The paper's Discussion/Conclusion section (Section 5) proposes "foundation model improvements through reinforcement learning" as a future direction but does not frame this as an alternative explanation for the current results.

Mitigation status. The paper is transparent about which models can and cannot be evaluated on which task categories (Section 4.3, Table 6 notations), which is a strength — it does not claim that end-to-end models were evaluated on tasks they cannot attempt. However, it does not acknowledge the training distribution confound as a limitation on the interpretation of the end-to-end vs. agentic gap. The paper's characterization of the gap as revealing "fundamental deficiencies" goes beyond what the experiments can support, because the experiments confound architecture, training data, model scale, and prompting strategy. A more circumspect interpretation — that off-the-shelf end-to-end GUI models perform poorly on MobileWorld's novel task categories, and that closing this gap (whether through fine-tuning, architecture changes, or both) is an open research challenge — would be equally informative without overclaiming. Future work could address this by: (a) fine-tuning an end-to-end model (e.g., Qwen3-VL-32B) on a held-out set of MobileWorld-style tasks with interaction and MCP actions, then evaluating on the remaining tasks to measure the effect of training distribution; (b) testing whether a smaller agentic framework (e.g., using Qwen3-VL-32B as the planner instead of GPT-5) maintains the advantage over same-scale end-to-end models, controlling for model capability.

No Systematic Human Performance Baseline Contextualizes the 51.7% State-of-the-Art

The assumption or constraint. The paper's human-in-the-loop task validation protocol (Section 3.3) confirms that all 201 tasks are solvable by humans — annotators can execute each task and achieve a score of 1.0 from the evaluator. However, this validation serves as a binary existence proof (is the task solvable?) rather than a performance measurement (how well do humans perform on these tasks, and with what efficiency?). The paper does not report any systematic human evaluation: no human success rate, no human average completion steps, no human interaction quality metrics, no human MCP tool-calling accuracy. The human validation protocol is designed for task quality assurance, not for establishing a performance ceiling — annotators may retry tasks up to five times (Section 3.3), may have designed the tasks themselves (introducing familiarity bias), and operate without the step-count or time pressure that agents face (though agents are not wall-clock-constrained either).

The consequence. The headline 51.7% SR achieved by GPT-5 + UI-Ins-7B is difficult to interpret without a human baseline. Is 51.7% roughly halfway to human performance, or is human performance near 100% on these tasks (making 51.7% genuinely low), or is human performance itself substantially below 100% due to task complexity (making 51.7% more impressive)? Each interpretation has different implications for the benchmark's saturation timeline and the research community's priorities. If humans achieve ~95% SR on MobileWorld, the 51.7% SOTA represents substantial headroom and the benchmark will remain discriminative for years. If humans achieve ~70% SR (because some tasks are genuinely difficult even for humans — requiring careful reading of long MCP outputs, precise numerical calculations, or sustained attention across 40+ steps), then 51.7% is much closer to the human ceiling and the benchmark may saturate more quickly than the paper anticipates.

Human performance data would also calibrate the difficulty of specific task categories. The paper's category-level analysis shows GPT-5 achieving 62.2% on interaction tasks and 51.6% on MCP tasks — but if humans also find MCP tasks substantially harder than interaction tasks (perhaps because parsing long tool responses is cognitively demanding), then the MCP SR gap between humans and agents may be similar to or smaller than the interaction SR gap, changing the relative priority of research on these categories. Without human baselines, the paper's implicit assumption that 100% is the meaningful ceiling (and that all categories are equally "solvable" in principle) goes unexamined.

The human validation protocol's five-retry allowance also creates an ambiguity about task difficulty: a task that required five human attempts to solve successfully (and barely passed validation) is included in the benchmark on equal footing with a task solved on the first attempt. The paper does not report the distribution of human attempts needed for task validation, which would provide a coarse human difficulty rating. Tasks that were near the five-retry threshold may be disproportionately hard for agents as well, contributing to the SR gap in ways that reflect inherent task difficulty rather than agent-specific deficiencies.

What evidence exists in the paper. The paper states that human annotators validated all tasks (Section 3.3) and that tasks requiring more than five attempts were returned for redesign — implying that all included tasks were solvable within five human attempts. No quantitative human performance data is reported. The step-count comparison with AndroidWorld (Figure 5) reports only agent step counts, not human step counts. The paper's comparison to AndroidWorld uses the 90%+ SOTA SR on AndroidWorld as a reference point, but this is an agent-agent comparison, not an agent-human comparison.

Mitigation status. The paper does not discuss the absence of human baselines as a limitation. This is a common omission in benchmark papers — human evaluation is expensive and time-consuming, and the primary goal is to rank agents, not to measure the human-agent gap. However, for a benchmark that positions itself as driving progress toward "real-world automation complexity" and "next-generation mobile intelligence" (Section 1), knowing whether the 51.7% SOTA represents the beginning, middle, or end of the path to human-level performance would substantially strengthen the paper's claims about headroom and the research roadmap's urgency. Future work could address this by: (a) conducting a systematic human evaluation on a stratified sample of MobileWorld tasks (e.g., 50 tasks spanning all categories and difficulty levels) with multiple annotators per task, reporting human SR, average steps, and per-category performance; (b) calibrating a "human difficulty rating" for each task based on the number of validation attempts needed; (c) using the human baseline to project how many years of progress (at current improvement rates) are needed to reach human-level performance, informing the community's expectations about benchmark saturation.

7. Implications and Future Directions

How This Work Changes the Landscape

MobileWorld is not a new method, but a new instrument — and like any well-designed instrument, its primary effect is to change what the field can see. Prior to this work, the mobile agent community was navigating with a saturated compass: AndroidWorld's 90%+ success rates told researchers that their systems were working, but couldn't tell them where the remaining problems lay or how to prioritize investments. MobileWorld replaces that saturated signal with a rich, structured picture of capability and failure.

The shift is from aggregate ranking to capability decomposition. Current mobile agent evaluation — across all benchmarks surveyed in Table 1 — produces a single number: success rate. This number conflates planning ability with perception accuracy, long-horizon memory with short-horizon execution, and (in AndroidWorld's case) cannot distinguish between an agent that genuinely understands task structure and one that has overfit to the benchmark's specific applications. MobileWorld's category-level breakdown (GUI-Only, agent-user interaction, MCP-augmented) and its five-metric evaluation suite (SR, Ave. Steps, Ave. Queries, UIQ, Ave. MCP Calls) enable what the paper's error analysis (Section 4.6) demonstrates in qualitative form: a decomposition of agent capability into distinct, measurable dimensions that can be tracked independently over time.

This is a methodological shift, not merely a harder benchmark. The key insight — reflected in the paper's construction of interaction and MCP tasks — is that a benchmark should function as a diagnostic instrument, not just a leaderboard. The difference is operational: a leaderboard tells you who won; a diagnostic instrument tells you what capability to improve next. The five research challenges in Section 4.6 demonstrate this function — they emerged from systematic inspection of failure trajectories rather than being imposed a priori, and they map each failure mode to a specific task subset and metric that future researchers can target. A team working on memory architectures can measure their progress against Challenge 3 by evaluating on the subset of tasks requiring multi-step state tracking (like the file renaming example in Figure 8). A team working on metacognitive uncertainty estimation can target Challenge 1 using the interaction SR and UIQ metrics. This is the paper's most durable contribution: it transforms mobile agent evaluation from a single-axis ranking problem into a multi-axis profiling problem.

Reconciling the determinism-realism tradeoff. Prior benchmarks forced a choice: deterministic evaluation (AndroidWorld, AndroidLab) or realistic application coverage (AndroidArena, SPA-Bench). MobileWorld demonstrates that this tradeoff is false — and more importantly, it demonstrates how it is false. The approach is not to somehow gain backend access to commercial applications (which is infeasible) but to substitute the application identity while preserving the capability being tested. Mattermost tests enterprise communication capabilities, not Slack specifically; Mastodon tests social media interaction capabilities, not Twitter specifically. This decoupling of application identity from functional capability is a conceptual move that transfers to any domain where proprietary services create evaluation bottlenecks — desktop benchmarks (OSWorld faces similar issues with Microsoft Office), web benchmarks, and IoT agent benchmarks could all adopt this substitution strategy. The paper's detailed description of backend construction (Appendix B.2 — modifying Mattermost's Docker deployment, snapshotting Mastodon's PostgreSQL database, instrumenting the Mail app with callbacks) serves as a reproducible engineering template, not just a one-off solution.

The consequence is that future benchmark designers no longer need to accept MLLM-as-a-judge evaluation as the price of application realism. The paper provides both the conceptual justification (verification fidelity and application realism are orthogonal concerns) and the practical recipe (self-hosted open-source forks with backend access). This opens up benchmark design space: if you can find or build an open-source application that exercises the same agent capabilities as a commercial one, you can achieve deterministic evaluation without sacrificing coverage.

Revealing the metacognitive gap. Perhaps the most surprising empirical finding is the non-monotonicity of capability across task categories. GPT-5 achieves 62.2% on interaction tasks — higher than its 54.0% on GUI-Only tasks. Gemini-3-Pro shows the opposite pattern: 55.6% on GUI-Only but only 24.4% on interaction. This means that "stronger model" is not a uniform property — it decomposes into sub-capabilities (GUI navigation, ambiguity detection, tool orchestration) that different frontier models possess to different degrees. This finding reframes the research problem: rather than building "better" agents in an undifferentiated sense, the field needs to build agents that are strong in specific sub-capabilities, and MobileWorld provides the measurement infrastructure to track progress on each one.

The metacognitive gap is particularly significant because it has been systematically invisible in prior benchmarks. When all tasks provide complete instructions, there is no signal about whether an agent can recognize its own knowledge boundaries. MobileWorld's interaction tasks — where critical information is deliberately withheld and verified to be unrecoverable through GUI navigation — create a controlled probe for this capability. The near-zero performance of Qwen3-VL models (0.0–6.7% interaction SR) despite having ask_user in their action space demonstrates that the failure is not in lacking the action but in failing to recognize when to use it. This is a metacognitive failure, not an execution failure, and it requires fundamentally different solutions (uncertainty estimation, explicit belief state modeling, out-of-distribution detection) than execution-oriented improvements (better grounding, more training data).

Shifting research priorities. The paper's results make certain research directions more attractive and others less so:

  • More attractive: (1) Metacognitive architectures that explicitly represent uncertainty about task parameters — the 38 percentage-point gap between GPT-5 and Gemini-3-Pro on interaction tasks shows this is not an incremental frontier-model improvement but a capability with high variance across architectures; (2) context management and retrieval strategies for long tool responses — Challenge 2 (Figure 7) shows that even when agents correctly invoke MCP tools, the resulting 20k-token responses can cause downstream failures, and the paper's finding that MCP SR is relatively uniform across agentic frameworks (~49–52%) suggests this is a shared bottleneck, not a model-specific weakness; (3) memory and state-tracking mechanisms — Challenge 3 (Figure 8) demonstrates destructive loops from forgotten subtask completion, a failure mode that no amount of model scaling alone will fix; (4) hybrid training data that includes interaction and tool-use trajectories — the paper's end-to-end models were evaluated zero-shot, and the gap between them and agentic frameworks may be partially (or largely) closeable through fine-tuning.

  • Less attractive: (1) Purely GUI-based agent architectures that assume complete instructions — the 42.3% of MobileWorld tasks requiring interaction or MCP tool use means that GUI-only agents have a hard ceiling well below 100% on this benchmark, and as real-world deployment increasingly demands these capabilities, GUI-only approaches will be systematically disadvantaged; (2) scaling end-to-end model parameters without addressing the metacognitive and memory gaps — the Qwen3-VL family's near-flat scaling curve (5.5% at 8B, 9.5% at 235B) suggests that throwing more parameters at the problem yields diminishing returns when the architecture lacks mechanisms for the required behaviors; (3) developing ever-more-sophisticated search or planning algorithms for single-app, short-horizon tasks — the step-count comparison with AndroidWorld (27.8 vs. 14.3 average steps) shows that MobileWorld's difficulty comes from structural task properties (cross-app workflows, memory requirements, ambiguity), not from needing better search within a single app.

Follow-Up Research This Work Enables

Fine-tuned end-to-end baselines to distinguish architecture from training distribution. The paper's end-to-end models were evaluated zero-shot on MobileWorld — they were not trained on data involving ask_user actions, MCP tool calls, or long-horizon cross-app trajectories. Their poor performance (4.5–20.9% overall SR, near zero on interaction and MCP tasks) could reflect either architectural limitations or training distribution mismatch. A critical follow-up would fine-tune a representative end-to-end model (e.g., Qwen3-VL-32B or UI-Venus-72B) on a held-out set of MobileWorld-style tasks that include interaction and MCP tool-use trajectories, then evaluate on the remaining benchmark. If fine-tuned end-to-end SR approaches the agentic framework's 51.7%, the capability gap is primarily a training data problem — solvable through data engineering. If it remains substantially below, the gap is architectural — requiring new mechanisms (explicit uncertainty modeling, modular tool-use components, memory architectures). The paper's own observation that Qwen3-VL makes comparable numbers of MCP calls to agentic frameworks (2.32–3.84 vs. 1.91–2.63, Table 7) but achieves dramatically lower MCP SR (0.0–5.4% vs. 48.6–51.6%) suggests that execution accuracy (correctly formatting tool names and arguments) is part of the problem, and fine-tuning might improve this. This experiment would transform the paper's "fundamental deficiencies" claim from hypothesis to demonstrated fact — or refute it, pointing toward data-centric solutions.

MCP difficulty decomposition: context overflow vs. tool selection vs. orchestration. The paper identifies MCP-augmented tasks as a major challenge (best agentic SR: 51.6%) and provides a qualitative example of context overflow from long tool responses (Challenge 2, Figure 7). However, the relative contribution of different MCP failure modes — incorrect tool selection, parameter formatting errors, context overflow from long responses, inability to extract relevant information from structured output, and failures in the GUI follow-up actions — is unknown. A controlled experiment would ablate each difficulty source: (a) run agents on MCP-augmented tasks where tool responses are pre-truncated to relevant excerpts, removing context overflow as a failure mode; (b) run agents where the correct tool is pre-selected and only parameter formatting is required; (c) run agents where the MCP output is pre-parsed into the exact information needed for the GUI follow-up; (d) compare SR under each condition to the baseline MCP SR from Table 6. The difference between (a) and baseline quantifies the context overflow bottleneck; between (b) and (a) quantifies tool selection difficulty; between (c) and (b) quantifies output parsing difficulty; between baseline and (c) quantifies GUI orchestration difficulty. This decomposition would directly inform whether research priority should go to context management strategies (Jones & Kelly, 2024), tool selection training, or better structured output parsing — each of which implies different architectural interventions.

Cheap difficulty estimation and adaptive strategy selection for deployment. The paper's compute-optimal allocation framework (implicit in the category-level performance breakdown) demonstrates that different task categories require different agent capabilities, but MobileWorld currently pre-configures which tasks need interaction and which MCP tools are available. In deployment, an agent must autonomously determine whether clarification is needed and whether MCP tools should be invoked — and must make this determination without consuming excessive compute. A natural follow-up would train a lightweight classifier that takes only the task instruction text (and optionally the first screenshot) as input and predicts: (a) does this task require user interaction? (b) which, if any, MCP tools are likely needed? This classifier could be trained on MobileWorld's task metadata (interaction vs. non-interaction labels, MCP tool requirements) and evaluated on held-out tasks. A strong result — say, 90%+ accuracy in predicting interaction necessity and 80%+ in predicting relevant MCP tools — would enable adaptive agent architectures that deploy different strategies (thorough clarification protocol, tool-use prompting) based on the classifier's output, reducing wasted compute on unnecessary metacognitive deliberation. The paper's own difficulty estimation is oracle-based (the task category is known), so this follow-up directly addresses a deployment gap that the paper identifies but does not solve.

Cross-benchmark correlation analysis to assess generalizability. MobileWorld claims to reflect "real-world mobile usage," but all results are on a single benchmark with a specific task distribution. A systematic cross-benchmark evaluation would test whether MobileWorld performance predicts performance on other mobile benchmarks (AndroidWorld, A3, SPA-Bench) or non-mobile benchmarks (OSWorld for desktop, WebArena for web). Specifically: (a) evaluate the same set of agents (GPT-5 + UI-Ins-7B, Qwen3-VL-235B, Doubao-1.5-UI-TARS) on both MobileWorld and AndroidWorld, computing per-task or per-category correlation; (b) evaluate on OSWorld-MCP to see whether MCP tool-use capability transfers across platforms; (c) evaluate on τ-bench to see whether user interaction capability transfers from mobile GUI to text-based customer service domains. A high cross-benchmark correlation would validate MobileWorld as a general mobile agent capability measure; a low correlation would indicate that MobileWorld's difficulty is domain-specific, and that mobile agent evaluation requires a suite of complementary benchmarks rather than a single "gold standard." The paper's AndroidWorld step-count comparison (Figure 5) provides a starting point — extending this to full SR comparison would be the natural next step.

Memory architecture ablations for long-horizon state tracking. Challenge 3 (Figure 8) demonstrates a specific failure: agents forget which subtasks they've completed, leading to destructive loops. This is a clean testbed for memory architectures. A follow-up experiment would implement and compare several memory mechanisms on the subset of MobileWorld tasks requiring multi-step state tracking (the file renaming task, tasks requiring counting or summing across multiple screens, tasks where information from one app must be retained across a switch to another): (a) no explicit memory (the current baseline); (b) a sliding window of recent actions and observations (the implicit memory in most current architectures); (c) an explicit structured memory that tracks completed subtasks, extracted information, and pending goals, updated by the planner at each step; (d) a retrieval-augmented memory that stores all past observations and retrieves relevant ones via embedding similarity. The primary comparison metric would be SR on state-tracking tasks, with secondary metrics on step efficiency (does memory reduce redundant actions?) and UIQ (does memory reduce redundant user queries?). This experiment is directly enabled by MobileWorld because the benchmark provides tasks where state-tracking is the bottleneck — in benchmarks with short, single-app tasks, memory architecture differences would be invisible because the tasks don't require multi-step state maintenance.

Human performance baseline for capability calibration. The paper validates that all 201 tasks are solvable by humans (Section 3.3, five-retry protocol) but does not report systematic human performance. A follow-up study would recruit multiple annotators (unfamiliar with the specific tasks, to avoid familiarity bias from the task designers) to complete a stratified sample of MobileWorld tasks (e.g., 50 tasks spanning all categories and difficulty levels) under conditions comparable to the agent evaluation (50-step budget, same observation interface). The results would provide: (a) a human SR ceiling — is it ~95% (agents have far to go), ~75% (agents are closer than they appear), or something else? (b) per-category human SR to calibrate relative difficulty — if humans also find MCP tasks substantially harder than GUI-Only tasks, the 51.6% MCP SR is more impressive than it appears; (c) human step counts to contextualize the 27.8 average agent steps — do humans complete these tasks in 15 steps, 25 steps, or 40 steps? This baseline would transform MobileWorld from a pure agent-agent ranking tool into an instrument that can measure progress toward human-level performance, giving the research community a clearer sense of how much headroom remains and which categories are closest to saturation.

Practical Applications and Downstream Use Cases

On-device mobile assistants with proactive clarification for vague commands. Current mobile assistants (Siri, Google Assistant) handle structured commands well ("set a timer for 10 minutes") but fail silently or hallucinate on vague requests ("send an email to Kevin"). MobileWorld's interaction task design — where critical parameters are deliberately withheld — provides a testbed for developing and evaluating the proactive clarification capability that would make assistants robust to real-world instruction ambiguity. A product team building a next-generation mobile assistant could use MobileWorld's interaction task subset (45 tasks) and the UIQ metric to measure progress: an assistant that achieves UIQ of 0.40 (GPT-5 level) would be substantially more usable than one at 0.13 (Doubao-1.5-UI-TARS level) because it would ask the right questions at the right time rather than issuing redundant or irrelevant clarification requests. The paper's finding that even GPT-5 achieves only 62.2% interaction SR means there is substantial headroom for improvement before this capability is solved, making MobileWorld directly relevant as a development target.

Enterprise automation workflows with hybrid GUI and API tool use. Many enterprise workflows — processing invoices received via email, updating CRM records based on Slack conversations, generating reports from data in multiple SaaS tools — require exactly the hybrid GUI-API orchestration that MobileWorld's MCP-augmented tasks test. A company building an enterprise automation agent could use the MCP-augmented task subset (40 tasks) to evaluate whether their agent can strategically choose between GUI navigation and API calls, correctly format tool invocations, and handle the context management challenges of long API responses. The paper's finding that MCP tool invocation accuracy is a major bottleneck for end-to-end models (Qwen3-VL makes comparable numbers of calls but achieves near-zero success due to formatting errors) provides a concrete diagnostic: if an enterprise agent's MCP calls fail at high rates, the problem is likely in the model's ability to correctly specify tool names and parameters, not in its strategic decision to use tools. This points toward solutions like constrained decoding for tool calls or fine-tuning on tool-use trajectories, rather than higher-level planning improvements.

Benchmark-driven training data generation for mobile agent fine-tuning. MobileWorld's task construction methodology — designing provably-ambiguous tasks by removing critical information and verifying non-recoverability, building hybrid MCP-GUI workflows — is a template for generating training data, not just evaluation tasks. An organization building a mobile agent could use this methodology to create thousands of synthetic training tasks that exercise specific capabilities: ambiguity detection (by systematically removing parameters from otherwise-complete tasks), tool orchestration (by designing MCP-solvable subgoals with GUI-dependent completions), and long-horizon memory (by creating tasks requiring information retention across app switches). The paper's detailed descriptions of task construction (Section 3.3) and environment setup (Appendix B.2) provide the recipe. The key advantage over generic training data is that tasks constructed this way have verifiable ground truth — success can be evaluated deterministically through the same verification mechanisms the benchmark uses, enabling automated reward signal generation for reinforcement learning or rejection sampling fine-tuning. This directly addresses the training distribution gap that the paper identifies as a likely contributor to end-to-end model failures.

When to Prefer MobileWorld Over Alternative Benchmarks

The paper positions MobileWorld against a landscape of mobile agent benchmarks (Table 1), and the experimental results (Section 4) demonstrate specific conditions under which MobileWorld provides information that other benchmarks cannot. The decision rule is:

  • Prefer MobileWorld when evaluating agents intended for real-world deployment where users provide vague or incomplete instructions and where external tool integration (MCP or similar APIs) is part of the agent's operational environment. AndroidWorld cannot test these capabilities at all (no interaction tasks, no MCP tasks), and commercial-app benchmarks cannot do so deterministically. The 51.7% SOTA SR means MobileWorld will remain discriminative for the foreseeable future, whereas AndroidWorld's 90%+ SR means it can no longer distinguish top-tier agents.

  • Prefer MobileWorld when diagnosing specific capability gaps — particularly metacognitive ambiguity detection, tool orchestration, long-horizon memory, and temporal-spatial grounding. The category-level SR and five-metric evaluation suite (Table 7) provide granular performance profiles that AndroidWorld's aggregate SR cannot. A team unsure why their agent is failing can run it on MobileWorld and identify whether the bottleneck is GUI navigation (GUI-Only SR), interaction quality (UIQ), MCP execution (MCP SR and Ave. MCP Calls), or step efficiency (Ave. Steps) — enabling targeted improvement rather than undifferentiated model scaling.

  • Prefer MobileWorld when deterministic evaluation is non-negotiable (e.g., for scientific benchmarking, leaderboard maintenance, or regulatory compliance) but the application domains of interest include communication, social media, or e-commerce. Prior deterministic benchmarks (AndroidWorld, AndroidLab, MobileAgentBench) omit these categories entirely; MobileWorld covers them through self-hosted open-source alternatives with backend database access. The cost is that task verification requires maintaining the containerized backend infrastructure (Docker-in-Docker, snapshot management), which is more operationally complex than AndroidWorld's purely on-device verification — but the coverage benefit justifies the infrastructure cost for teams that need these application categories.

  • Prefer AndroidWorld (or a smaller deterministic benchmark) when (a) evaluating agents that lack any support for user interaction or MCP tool calls (most current end-to-end models), since these agents would score near zero on 42.3% of MobileWorld tasks regardless of their GUI capability — a comparison would be uninformative; (b) the research question is specifically about single-app, short-horizon GUI navigation in isolation from other capabilities; or (c) operational simplicity is paramount and the team cannot maintain the Docker-in-Docker infrastructure MobileWorld requires.