ArXiv: 2602.16855

🎯 Pitch

An 8-billion-parameter GUI agent outperforms a model 9× its size (72B) on OSWorld by synthesizing step-by-step reasoning traces during training and using a novel multi-platform RL algorithm to prevent the policy from collapsing on long-horizon tasks. The same model family achieves a 80.3 on ScreenSpotPro via a crop-based refinement strategy, surpassing Google’s Gemini-3-Pro without relying on a massive proprietary API.

1. Executive Summary

This paper introduces GUI-Owl-1.5, a family of native GUI agent models spanning instruct/thinking variants at sizes from 2B to 235B parameters, built on Qwen3-VL and trained to perform automated GUI operations across desktop, mobile, browser, and other platforms. The system achieves state-of-the-art results on 20+ benchmarks—including 56.5 on OSWorld, 71.6 on AndroidWorld, and 48.4 on WebArena—through three named mechanisms: a Hybird Data Flywheel (combining virtual environments with cloud-based sandboxes for scalable trajectory and grounding data generation, including DAG-based task synthesis and web-rendering-based simulators for CAPTCHA-prone scenarios), Unified Enhancement of Agent Capabilities (injecting GUI knowledge via QA data and world modeling, synthesizing chain-of-thought reasoning with step-wise observation/memory/reflection, and enabling multi-agent role specialization via a Manager-Worker-Reflector-Notetaker framework), and Multi-platform Environment RL Scaling (an MRPO algorithm that unifies device-conditioned policy learning across mobile/desktop/web environments with an online rollout buffer to mitigate GRPO outcome collapse and alternating optimization to reduce cross-platform gradient interference). The 32B-Instruct variant achieves 80.3 on ScreenSpotPro with crop-based refinement, surpassing even the large-scale Gemini-3-Pro, while the 8B-Thinking variant outperforms the prior 72B UI-TARS-2 on OSWorld (52.9 vs. 53.1), establishing that strong parameter efficiency is achievable in GUI agents but only when virtual-environment-based data generation and multi-stage RL are combined to address long-horizon task execution across heterogeneous platforms.

2. Context and Motivation

The Core Problem: We Can Build GUI Agents, But We Can't Build Them to Work Well Everywhere

The fundamental gap this paper addresses is not that GUI agents don't exist—they do, and some work remarkably well under narrow conditions. The gap is that no existing approach simultaneously satisfies three requirements for practically deployable GUI automation: (1) scalable, high-quality data generation across diverse platforms and applications, (2) comprehensive agentic capabilities beyond simple clicking and typing, and (3) stable multi-platform policy learning that doesn't degrade when moving between device types.

This triad of problems is deeply interconnected in ways the paper makes explicit. You cannot train a unified multi-platform agent without data from all platforms, but collecting high-quality trajectory data from each platform requires solving platform-specific challenges—mobile apps have CAPTCHAs and anti-bot mechanisms that terminate automated exploration, desktop environments demand precise drag-and-drop and spreadsheet manipulation that exploration-based data collection rarely captures cleanly, and browser environments involve heterogeneous page structures that resist template-based automation. Meanwhile, even if you solve the data problem, the agent needs capabilities that go beyond single-step perception and action: it must remember prices mentioned three turns ago, reflect on whether its last click actually navigated to the right page, decide when to invoke an external API versus manipulating the GUI directly, and coordinate with other specialized agents in multi-agent setups. And if you try to train a single policy on all this data simultaneously using reinforcement learning, the gradients from mobile tasks (where success often means navigating a multi-screen food delivery workflow) conflict with gradients from desktop tasks (where success often means precise spreadsheet cell manipulation), causing optimization to oscillate rather than converge.

The paper positions this triad as the bottleneck preventing GUI agents from graduating from impressive demos to reliable deployment. Section 1 enumerates these challenges directly:

"the development of robust and practically usable GUI agents still faces several challenges. (1) The efficiency of real-world data collection... (2) The adaptation to multiple platforms... (3) The comprehensive agentic capabilities: The General GUI Agent should be capable of completing tasks efficiently, not limited to GUI-only operations."

Why This Matters: The Gap Between Demos and Deployment

The practical importance of solving this triad is hard to overstate because GUI automation touches nearly every software interaction humans perform. When a user books a flight, they interact with a web browser GUI (searching, comparing prices, filling forms). When they order food delivery, they interact with a mobile GUI (browsing restaurants, customizing orders, confirming payment). When they analyze data, they interact with a desktop GUI (spreadsheets, visualization tools). A truly capable GUI agent would automate all of these—not by accessing backend APIs (which are often unavailable, inconsistent across services, or require authentication) but by using the same visual interfaces humans use.

This matters for several concrete deployment scenarios the paper highlights (Section 1):

  • Edge-cloud collaboration: Small instruct models (2B–8B) running on edge devices handle high-frequency, latency-sensitive interactions while larger thinking models in the cloud handle complex planning and multi-step reasoning. The paper explicitly positions its model family sizes as enabling this spectrum: "Smaller instruct models, which do not produce thoughts, enable faster inference and can be deployed on edge devices to support high-frequency, real-time interactions while addressing security and privacy concerns."

  • Multi-platform coordination: A single agent that works across desktop, mobile, and browser eliminates the need to build, maintain, and integrate separate automation systems per platform—a significant practical engineering cost. The case study in Figure 9, where the agent searches for follower counts across two different social media apps on mobile, illustrates a task that inherently spans multiple application interfaces within a single platform. The extension to cross-platform tasks (e.g., verifying a desktop spreadsheet against data from a mobile app) is the natural next step that a unified policy enables.

  • Tool and MCP integration: The paper emphasizes that real-world tasks frequently require mixing GUI operations with API calls—reading a file via a tool call to understand what code needs modification, then using GUI operations in a terminal to execute it. The case study in Figure 11 demonstrates exactly this: the agent reads source code via filesystem_read_text_file, fixes an insertion sort implementation via filesystem_edit_file, opens a terminal via GUI operations to execute the script, and verifies output by reading a log file. This interleaving of tool use and GUI manipulation is not supported by many prior GUI agent frameworks.

The theoretical significance is equally important: this paper provides empirical evidence for what scaling multi-platform GUI agent training actually requires. Prior work either trained single-platform agents (which leaves the cross-platform generalization question unanswered) or trained multi-platform agents with limited data and RL (which left the scaling behavior unknown). GUI-Owl-1.5's training recipe—particularly the MRPO algorithm with its online rollout buffer and alternating optimization—represents one of the first systematic attempts to characterize and solve the optimization challenges that emerge when scaling GUI RL across heterogeneous environments.

Where Prior Approaches Fall Short

The paper's critique of prior work operates along three axes corresponding to the three problems it identifies, though the critiques are distributed across the introduction and technical sections rather than gathered in a single related-work section.

Axis 1: Data Collection Efficiency and Quality

The prior approach to trajectory data for GUI agents falls into roughly three categories, each with documented failures:

Agent exploration on real devices (e.g., the approach used in Mobile-Agent-v3, GUI-Owl's predecessor). An existing agent model (or a prompted general-purpose VLM) interacts with real apps on real devices, and successful trajectories are collected for training. This approach has two failure modes the paper documents in Section 2.2.2:

First, real-world apps actively resist automated exploration: "Real-world applications and software often incorporate CAPTCHA verification, anti-bot mechanisms, and other protective measures that can interrupt or terminate the agent's exploration process." This is an engineering reality that makes scaling trajectory collection across many apps expensive—each app may require custom handling to avoid triggering bot detection, and the handling is fragile to app updates.

Second, exploration without ground-truth feedback produces noisy trajectories: "Real-world environments cannot provide accurate feedback, which results in low efficiency of trajectory generation via agent exploration, and often yields trajectories that contain erroneous or redundant steps." The agent may think it succeeded when it actually failed, and these false-positive trajectories poison the training data. The paper's use of checkpoint predicates ϕk(ot)\phi_k(o_t) in the trajectory collection pipeline (Section 2.2.2) is a direct response to this problem: by defining subtask-completion conditions that can be automatically verified, the system can truncate trajectories at the last verified correct step rather than including noise beyond that point.

Human annotation solves the quality problem but fails on scalability and cost. The paper acknowledges this obliquely: "For difficult tasks that remain unsolved after repeated automated attempts, we collect expert demonstrations via a cloud annotation platform." The implication is clear—human annotation is used only as a fallback for the hardest cases where automation consistently fails, not as the primary data source.

Template-based or LLM-generated synthetic data (used in prior work but not extensively detailed in this paper) produces high volume but low diversity and fails to capture the visual complexity of real GUIs. The paper's response is the Hybird Data Flywheel, which combines virtual environments (web-rendering-based simulators) for scalable, feedback-rich trajectory generation with DAG-based task synthesis for coverage of high-frequency workflows and a small amount of human annotation for challenging edge cases. The key insight is that the virtual environments provide two things real devices cannot: precise subtask-level feedback (the simulator can programmatically check whether a drag operation actually moved the target element to the correct position) and resistance to bot detection (there is no anti-automation mechanism to trigger). However, the paper does not extensively benchmark prior data collection approaches to quantify the advantage of the flywheel—the ablation in Table 11 only compares with and without virtual environment data within GUI-Owl-1.5's own pipeline, not against alternative data collection methods.

Axis 2: Comprehensive Agentic Capabilities

Prior GUI agent models (including the paper's own predecessor GUI-Owl) treated GUI automation primarily as a perception-action problem: look at the screen, decide what to click, click it, repeat. This is sufficient for simple navigation tasks but fails on tasks requiring memory, reflection, or tool use. The paper identifies specific capability gaps:

Memory over long horizons. In tasks like "Check the weather in Paris and London for next Monday and record it in the memo" (Section 2.3.2), the agent must extract temperature values from one screen, retain them across several navigation steps to a different app, and then recall them when filling in the memo fields. Prior native agent models like GUI-Owl-7B achieve only 14.6 on MemGUI-Bench (Table 9), indicating that basic perception-action training does not instill this retention capability. The paper's insight is that memory must be explicitly trained, not just hoped-for: the unified CoT synthesis pipeline injects "memory" as an explicit reasoning step in the training data, teaching the model to identify information worth remembering, record it in the thought stream, and reference it later.

Reflection and error recovery. When an agent clicks the wrong button, it needs to recognize the mistake (the screen didn't change as expected), understand what went wrong, and try an alternative action. Prior agents often continue blindly after errors. The paper's unified CoT synthesis addresses this by generating reflection annotations after each action step, comparing the expected screen state transition against the actual transition. These annotations become training supervision that teaches the model to evaluate its own actions. The Reflector role in the multi-agent framework (Section 2.3.3) formalizes this as a separate verification step, though the paper is clear that the training data for this verification is synthetic—generated by prompting a proprietary VLM to compare before/after screenshots, not from human verification.

Tool and MCP invocation. Most prior GUI agents operate in a closed GUI-only action space (click, type, scroll, swipe). Real tasks frequently require external computation: reading a file, querying a database, calling a search API. The paper significantly expands the action space (Section 2.1) to include structured tool calls and MCP interactions. This is not merely an implementation detail—it requires the model to learn when GUI manipulation is appropriate (clicking through menus to open a file) versus when a tool call is more efficient (directly reading the file via a function call), and to interleave them within a single trajectory. The OSWorld-MCP and MobileWorld benchmarks (Table 1) specifically evaluate this mixed capability, and GUI-Owl-1.5-32B-Instruct's scores of 47.6 and 46.8 respectively far exceed prior open-source models (MAI-UI-235B-A22B at 41.7 on MobileWorld), suggesting this training is effective.

World modeling (predicting GUI state transitions). The paper introduces an interesting capability that goes beyond standard agent training: teaching the model to predict how the screen will change before taking an action. Section 2.3.1 describes this: "given a screenshot and the action executed at that step, we prompt a proprietary model (e.g., Claude-4.5) to produce a fine-grained description of the subsequent screenshot, explicitly highlighting the state transitions." These action-conditioned state-transition descriptions become training data. The intuition is that a model that can anticipate consequences will make better decisions—it can mentally simulate "if I click this button, a dialog will appear" and use that prediction to plan. This is a form of model-based RL thinking baked into supervised pre-training, but the paper does not provide an explicit ablation isolating the effect of world modeling data on downstream task performance. Its contribution is bundled into the overall pre-training corpus.

Axis 3: Multi-Platform RL Training Stability

The most technically specific gap the paper addresses is the instability of multi-platform reinforcement learning for GUI agents. Prior work that attempted to train GUI agents with RL either focused on a single platform (avoiding the cross-device optimization interference) or used supervised fine-tuning alone (avoiding RL entirely but leaving performance on the table). The paper characterizes the specific failure modes:

GRPO outcome collapse. When using GRPO (Group Relative Policy Optimization) for GUI tasks, it is common that all nn trajectories sampled for a given task produce identical outcomes—either all succeed or all fail. This makes the group useless for learning because there's no contrast between successful and unsuccessful trajectories to compute advantage estimates. The paper formalizes this as the Collapse event in Section 2.4.3:

Collapse(Gn)≜(∑τ∈GnZ(τ)∈{0,n})\text{Collapse}(\mathcal{G}_n) \triangleq \left(\sum_{\tau \in \mathcal{G}_n} Z(\tau) \in \{0, n\}\right)

where Z(τ)∈{0,1}Z(\tau) \in \{0, 1\} is the binary success/failure outcome. When all rollouts succeed, there's no negative signal; when all fail, there's no positive signal. The online rollout buffer (oversample knkn trajectories, then subsample nn with guaranteed outcome diversity) is a practical response to this statistical problem. The mathematical property the paper asserts—that uniform subsampling preserves the on-policy marginal distribution—is straightforward to prove (it's a consequence of exchangeability under i.i.d. sampling) but the engineering insight of using oversampling strategically to avoid outcome collapse while maintaining on-policy guarantees is novel in the GUI RL context.

Training-inference tokenization mismatch. This is a subtle but critical engineering problem that the paper explicitly addresses (Section 2.4.3). In RL training for language-conditioned policies, the environment-side inference service generates action text and returns it as a string. The training process then re-tokenizes that string to compute log-probabilities for the policy gradient. But if the inference-side and training-side tokenizers produce different token-ID sequences for the same text (which can happen with non-canonical tokenization, especially for structured outputs like tool calls with special characters), then:

log⁡πθ(y∣x)∣train-tokenize(y)≠log⁡πθ(y∣x)∣infer-tokenize(y)\log \pi_\theta(y \mid x)\big|_{\text{train-tokenize}(y)} \neq \log \pi_\theta(y \mid x)\big|_{\text{infer-tokenize}(y)}

This breaks the assumption that the policy gradient is computed with respect to the same distribution that generated the action, corrupting the gradient estimate. The fix—transporting the original inference token IDs alongside the text payload—is simple in retrospect but addresses a real failure mode that would be easy to miss in single-machine development setups where inference and training share the same tokenizer.

Cross-device gradient interference. The paper's observation that mixing trajectories from different device families in a single RL batch causes optimization instability is intuitive in retrospect but has not been formalized in prior GUI agent literature. The alternating optimization schedule—train on one device family at a time, cycling through mobile/desktop/web—is presented as a practical solution, and Figure 8(b) provides empirical support: mix-platform training oscillates while interleaved training achieves stable improvement. The mechanism is gradient conflict: ⟨gmobile,gdesktop⟩<0\langle g_{\text{mobile}}, g_{\text{desktop}} \rangle < 0, meaning progress on mobile tasks requires parameter updates that hurt performance on desktop tasks and vice versa. The alternating schedule isolates these conflicts temporally, allowing each device family to make progress during its own training stage before the policy switches to the next family.

What Prior Models Specifically Lacked

The paper's extensive benchmark tables make clear what prior models could not do. Reading across Tables 1–9, several patterns emerge that contextualize GUI-Owl-1.5's contributions:

  • Single-platform GUI models (MAI-UI, OpenCUA, UI-Venus, EvoCUA) achieve competitive or superior performance on their target platform but don't demonstrate cross-platform generalization. For instance, MAI-UI-235B-A22B achieves 76.7 on AndroidWorld but its desktop/browser performance isn't reported in Table 1 because it's not designed for those platforms.

  • Multi-platform GUI models from prior work (GUI-Owl-7B/32B, UI-TARS-72B, OS-Atlas) show multi-platform capability but substantially lower task completion rates. GUI-Owl-32B achieves only 58.0 on ScreenSpot-Pro (Table 4) compared to GUI-Owl-1.5-32B-Instruct's 72.9, despite both being 32B models—suggesting that the gap isn't just about scale but about training methodology.

  • General-purpose VLMs prompted for GUI tasks (Qwen3-VL variants, GPT-4o, Claude, Gemini) show surprisingly strong grounding performance but much weaker end-to-end task completion. Qwen3-VL-32B-Think achieves 63.7 on AndroidWorld (Table 1) with no GUI-specific training—respectable, but clearly behind GUI-Owl-1.5-8B-Thinking's 71.6, which uses one-quarter the parameters. This gap between grounding capability (knowing where to click) and task completion (actually finishing multi-step tasks) is exactly the space that the paper's training pipeline targets.

  • Proprietary models (Claude-4-Sonnet, Gemini-2.5-Pro, OpenAI Operator) represent the upper bound of what's possible without open-source replication. GUI-Owl-1.5 matches or exceeds several of these on specific benchmarks (e.g., GUI-Owl-1.5-32B-Instruct at 75.45 on GUI Knowledge Bench vs. o3 at 73.30) but the paper doesn't claim universal superiority—the goal is state-of-the-art among open-source models, which it achieves.

How This Paper Positions Itself

GUI-Owl-1.5 is explicitly positioned as an evolutionary improvement over GUI-Owl (Ye et al., 2025, the predecessor from the same team), not a revolutionary departure. Section 2 states: "GUI-Owl-1.5 is a multimodal model for GUI operations, building on the previous GUI-Owl. Compared to its predecessor, it offers three main improvements: (1) a broader action space; (2) improved context retention; (3) enhanced design in synthetic data generation, cross-platform adaptation, and agent capabilities."

This is honest framing: the core architecture (end-to-end VLM trained on trajectory data with CoT reasoning) is inherited from GUI-Owl. What's new is the scale and systematization of the training pipeline—the Hybird Data Flywheel, the unified CoT synthesis, the multi-platform RL framework—that enables the model to handle more platforms, more complex tasks, and more nuanced agent capabilities than its predecessor.

The paper's implicit argument is that we have reached a point in GUI agent development where the bottleneck is no longer model architecture or base model capability but rather training data quality, training data scale, and training procedure robustness. GUI-Owl-1.5 doesn't introduce a new transformer variant or a novel attention mechanism; it introduces better ways to generate training data (virtual environments, DAG synthesis, CoT annotation), better RL training strategies (online rollout buffer, alternating optimization), and better coverage of agent capabilities (memory, reflection, tool use). The fact that a 2B model trained with this pipeline can outperform a 72B model trained with a prior pipeline (GUI-Owl-1.5-2B-Instruct at 43.5 on OSWorld vs. UI-TARS-72B-DPO at 27.1, per Table 1) is the paper's strongest empirical argument for this position.

The paper also positions itself within the native agent model paradigm rather than the agent framework paradigm. This distinction is important: an agent framework (like Mobile-Agent-v2 or Agent S2) wraps a general-purpose VLM in a scaffold of planning, verification, and memory modules, using the VLM as a component. A native agent model bakes these capabilities into the model weights through end-to-end training. The advantage of the native approach (when it works) is reduced latency (no multiple VLM calls per step), reduced cost (one model handles everything), and the potential for capabilities to emerge from joint training that wouldn't emerge from separately engineered modules. The advantage of the framework approach is flexibility (swap out the VLM for a better one) and interpretability (each module's behavior can be debugged independently). GUI-Owl-1.5 doesn't argue that the native approach is universally superior—in fact, Section 2.3.3 describes a multi-agent framework that can use GUI-Owl-1.5 as a component—but it demonstrates that native training can achieve results that framework-based approaches haven't matched.

What Makes This Paper's Approach Distinctive

Three design choices set GUI-Owl-1.5 apart from contemporaneous work, and understanding these choices is essential to contextualizing its results:

First, virtual environments as a data scaling strategy. Most prior work (UI-TARS, MAI-UI, even GUI-Owl) relies primarily on agent exploration on real devices or human annotation for trajectory data. GUI-Owl-1.5's use of web-rendering-based virtual environments—essentially, simulated versions of real apps built for the purpose of generating training data—is a significant engineering investment that pays off in data quality and quantity. The virtual environments provide exact feedback on whether each subtask was completed correctly, eliminating the noisy trajectory problem, and they can generate infinite variations of complex scenarios (different spreadsheet layouts, different CAPTCHA patterns) without hitting anti-bot protections. The ablation in Table 11 quantifies this: removing virtual environment data drops PC-Eval from 75.4% to 42.0% and Mobile-Eval from 86.7% to 50.0%. These drops are dramatic enough to suggest that virtual environment data is not merely helpful but essential for the tasks those benchmarks test (precise atomic operations like drag-and-drop, document editing, and CAPTCHA-heavy app scenarios).

Second, explicit capability injection through CoT synthesis. Rather than hoping that a model trained on action sequences will incidentally learn to plan, reflect, and remember, the paper artificially constructs training examples that demonstrate these capabilities. The unified CoT synthesis pipeline (Section 2.3.2) takes raw trajectory data and enriches it with observation descriptions, memory annotations, reflection on action outcomes, and task progress tracking—all generated by prompting proprietary VLMs and LLMs. This is a form of knowledge distillation from stronger models into the training data, not into the model directly. The ablation in Table 10 confirms this matters: removing CoT synthesis drops OSWorld from 52.9% to 47.4% and AndroidWorld from 71.6% to 65.0%. The gap is smaller than for virtual environments, suggesting CoT synthesis provides important but incremental improvements rather than being the dominant factor.

Third, the MRPO RL framework's specific solutions to multi-platform optimization. Many recent GUI agent papers incorporate some form of RL (UI-TARS-2, EvoCUA, MAI-UI), but the paper's explicit treatment of outcome collapse, tokenization consistency, and cross-device gradient interference as separate, solvable problems is more systematic than typical. The online rollout buffer with Swap1 (Section 2.4.3) in particular is a clean contribution: it maintains on-policy guarantees while dramatically increasing the probability of informative training groups. The probability that a naive group of size nn is informative (not collapsed) is 1−pn−(1−p)n1 - p^n - (1-p)^n where pp is the success probability. For a task where the model succeeds 90% of the time with n=4n=4, this probability is only about 34%. With the oversample-and-select approach using k=4k=4 (sample 16, select 4), the probability of an informative pool jumps to 1−p16−(1−p)16≈81%1 - p^{16} - (1-p)^{16} \approx 81\%, and the Swap1 mechanism further guarantees informativeness when the pool itself is diverse. This converts what was a statistical barrier to effective RL training into a solved (or at least substantially mitigated) problem.

Unresolved Tensions in the Paper's Framing

A careful reader should note several tensions that the paper doesn't fully resolve:

The relationship between model size and capability is non-monotonic. The 8B-Thinking variant outperforms the 32B-Thinking variant on AndroidWorld (71.6 vs. 69.8, Table 1), which is unexpected if larger models are strictly more capable. The paper doesn't explain this reversal; it could reflect differences in training convergence, overfitting, or interaction between model size and the specific RL training dynamics for mobile tasks. This is worth flagging because it complicates the paper's narrative of a clean scaling story.

The difficulty estimation for virtual environments is not discussed. The paper claims virtual environments provide accurate feedback, but building a simulator that accurately reflects real app behavior is itself a significant engineering challenge. How much effort went into building each virtual environment? How many apps were virtualized? What was the coverage of edge cases (e.g., network errors, unexpected popups, OS-level interruptions)? The paper doesn't provide these details, making it difficult to assess whether the virtual environment approach scales to the long tail of real-world applications or is practical only for a curated set of high-frequency scenarios.

The COT synthesis pipeline uses proprietary models. The observation descriptions, memory annotations, and reflections that make the CoT data valuable are generated by prompting proprietary VLMs and LLMs. This means the training data quality depends on the quality of those proprietary models at the time of data generation. If those models have systematic weaknesses (e.g., missing certain types of screen changes, hallucinating memory content), those weaknesses propagate into the training data and potentially into the trained model. The paper doesn't analyze failure modes of the CoT synthesis pipeline, which would be valuable for understanding the ceiling of this approach.

The multi-agent framework is described but not systematically evaluated. Section 2.3.3 outlines the Manager-Worker-Reflector-Notetaker framework in detail, and the paper states that this framework was used during trajectory collection, but there is no ablation comparing the multi-agent data collection approach against a single-agent approach. This makes it impossible to determine how much of the performance gain comes from the multi-agent data structure versus other factors.

Despite these tensions, the paper's core contributions—the virtual environment data pipeline, the unified CoT synthesis, and the MRPO RL framework—represent concrete, well-motivated responses to clearly articulated problems in GUI agent training. The benchmark results provide strong empirical evidence that these contributions collectively produce state-of-the-art open-source GUI agents, even if the precise attribution of credit among the components remains somewhat uncertain.

3. Technical Approach

3.1 Reader Orientation

This paper describes an engineering system for building a family of vision-language models (collectively called GUI-Owl-1.5) that can look at screenshots of desktop, mobile, and browser interfaces, reason about what actions to take, and execute those actions to complete complex multi-step tasks like "search for the ModelScope Community account on Xiaohongshu and Douyin and calculate their total follower count." The core problem is that training such an agent requires high-quality demonstration data on all platforms, robust reasoning capabilities beyond single-step perception, and stable reinforcement learning across wildly different device environments—and the solution takes the form of a three-stage training pipeline (pre-training, supervised fine-tuning, RL) fed by a hybrid data engine that combines virtual simulated environments with real-device exploration, augmented with synthesized chain-of-thought reasoning and specialized multi-platform optimization techniques.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components, organized as a data flywheel feeding a training pipeline:

  1. Hybird Data Flywheel (Data Engine): A multi-source data generation pipeline that produces two categories of training data—(a) grounding data for teaching the model to locate specific UI elements given natural language queries, built through synthetic GUI generation, multi-window composition, trajectory mining, tutorial extraction, and negative sample construction; and (b) trajectory data for teaching the model to execute multi-step tasks, built through DAG-based task synthesis, automated agent rollouts on real devices with checkpoint-based validation, human annotation for hard cases, and virtual-environment-based trajectory production with exact subtask feedback.

  2. Unified CoT Synthesis: A post-processing pipeline that takes raw trajectory data and enriches each step with synthesized reasoning: screen observation descriptions, memory annotations (what to remember for later), reflection on action outcomes (did the expected screen change occur?), task progress tracking, and tool invocation reasoning. This is generated by prompting proprietary VLMs and LLMs on the raw trajectories.

  3. GUI Knowledge Injection: A parallel data stream that crawls software documentation, forums, and QA platforms to construct GUI-related QA pairs and world-modeling data (action-conditioned predictions of how the screen will change), augmenting the agent's understanding of application semantics beyond what trajectories alone provide.

  4. Base Model (Qwen3-VL): The pre-trained vision-language model that serves as the initialization point, providing fundamental visual understanding and language generation capabilities before any GUI-specific training.

  5. Three-Stage Training Pipeline: (a) Pre-training on the full corpus of grounding data, trajectory data, QA/VQA knowledge data, world-modeling data, and tool invocation data; (b) Supervised Fine-Tuning (SFT) specifically on multi-device trajectory data with CoT annotations, augmented grounding data, tool invocation supervision, and browser interaction data; (c) Reinforcement Learning (MRPO) that optimizes the SFT model against online environment rewards using a unified device-conditioned policy with specialized techniques to handle outcome collapse, tokenization consistency, and cross-platform gradient interference.

  6. Multi-Agent Framework (Mobile-Agent-v3.5): An optional deployment mode where the trained model can serve as specialized roles (Manager for planning, Worker for execution, Reflector for verification, Notetaker for memory) within a structured multi-agent loop, used both during data collection and as a runtime option for complex tasks.

Information flows as follows: raw data sources (real devices, virtual environments, web crawls) → Data Flywheel produces grounded/trajectory/QA data → Unified CoT Synthesis enriches trajectories with reasoning → Pre-training on all data types → SFT on agent-specific data → MRPO RL with online environment rollouts on multiple platforms → Deployable GUI-Owl-1.5 models (instruct/thinking variants at 2B/4B/8B/32B/235B).

3.3 Roadmap for the Deep Dive

  • First, the formal problem formulation (Section 2.1): the input/output spaces, the action space expansion, and the context management mechanism—since everything downstream depends on what the model actually predicts and how history is represented.
  • Second, the grounding data pipeline (Section 2.2.1): how the system generates high-quality localization training data, including hard grounding data synthesis and scalable extension strategies—because grounding is a prerequisite capability that the trajectory and agent systems build upon.
  • Third, the trajectory data collection pipeline (Section 2.2.2): the DAG-based task synthesis, checkpoint-based trajectory validation, human annotation fallback, and virtual environment trajectory production—because this is the core data engine that determines what the model learns to do.
  • Fourth, the agent capability enhancement pipeline (Section 2.3): GUI knowledge injection, world modeling, unified CoT synthesis, and the multi-agent collaboration framework—because these transform raw trajectory data into reasoning-rich training examples.
  • Fifth, the three-stage training paradigm (Section 2.4): pre-training corpus composition, SFT alignment, and the MRPO RL algorithm in detail—because this is where data becomes model capability through carefully staged optimization.
  • Sixth, the MRPO algorithm's specific technical mechanisms: the online rollout buffer for GRPO under outcome collapse, training-inference log-prob alignment via token-ID transport, and alternating multi-device optimization—because these are the paper's most original technical contributions to RL for GUI agents.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and engineering paper whose core idea is that state-of-the-art multi-platform GUI agents require not a novel architecture but rather a systematic integration of (1) scalable, high-fidelity data generation spanning both simulated and real environments, (2) explicit reasoning capability injection through synthetic chain-of-thought annotation, and (3) multi-platform RL with specialized stabilization mechanisms. The technical depth lies in the specific design of the data pipelines and the RL algorithm, not in model architecture innovations.

Problem Formulation: What the Model Actually Predicts

The paper formulates GUI automation as a multi-turn interactive decision-making problem where the agent perceives, acts, and receives feedback in a closed loop (Section 2.1). At each step $t$, the agent receives two inputs:

  • Visual observation $\mathcal{I}_t \in \mathbb{R}^{H \times W \times 3}$: a screenshot of the current GUI state, treated as a standard RGB image.
  • User instruction $\mathcal{L}_t$: a natural language command describing the user's intent.

Given these inputs, the agent produces two outputs:

  • Action conclusion $\mathcal{C}_t$: a natural language explanation summarizing what action is being taken and why—this is the model's internal reasoning made explicit.
  • Tool call $\mathcal{A}_t$: a structured function call that actually executes the action in the environment—this is the machine-readable command that the device executes (e.g., click(x=342, y=891), type(text="Paris"), or an MCP tool invocation like filesystem_read_text_file(path="/src/sort.py")).

The crucial design choice here is the separation of reasoning from execution: $\mathcal{C}_t$ is human-readable justification, $\mathcal{A}_t$ is machine-executable command. This dual-output structure means the model can be trained to produce coherent reasoning while maintaining precise action formats, and the reasoning tokens can be used or discarded depending on whether the instruct or thinking variant is deployed.

After executing $\mathcal{A}_t$, the environment transitions to a new state, providing updated visual feedback $\mathcal{I}_{t+1}$ for the next turn. This iterative process continues until a termination condition is met (task completion, timeout, or safety stop).

Action space expansion. Compared to the predecessor GUI-Owl, the paper significantly expands $\mathcal{A}_t$ beyond primitive GUI operations (click, type, scroll, swipe, drag) to include:

  • External tool calls: structured invocations of computational tools (e.g., file system operations, database queries, search API calls).
  • MCP (Model Context Protocol) invocations: a standardized protocol for tool interaction that the model must learn to use correctly.
  • Structured function calls with parameters: e.g., filesystem_edit_file(path, old_str, new_str) with multi-line string arguments.

This expansion means the model must learn when to interact with the GUI directly (navigating menus, clicking buttons) versus when to bypass the GUI and call an API, and must learn the syntax and semantics of the tool-calling interface. The paper states this explicitly:

"This extension enables the agent to orchestrate complex workflows across heterogeneous systems, such as querying databases through APIs, invoking specialized computational tools, and integrating with third-party services."

Context management for long-horizon tasks. GUI tasks can span dozens of steps, and naively including all previous screenshots in the model's context window would rapidly exceed memory and computational limits. The paper adopts a sliding window with hierarchical compression (Section 2.1):

The context at step $t$ is divided into two parts:

  • Recent context (full retention): The most recent $N$ complete dialogue turns are retained with all modalities—screenshots, instructions, conclusions, and actions. This preserves fine-grained visual and textual information for immediate decision-making.

  • Historical context (compressed summary): Earlier interactions beyond the $N$-turn window are condensed into a purely textual summary $\mathcal{S}_{1:t-N-1}$, formed by concatenating the action conclusions from those earlier steps:

S1:t−N−1=concat(C1,C2,…,Ct−N−1)\mathcal{S}_{1:t-N-1} = \text{concat}(\mathcal{C}_1, \mathcal{C}_2, \ldots, \mathcal{C}_{t-N-1})

where $\mathcal{C}_i$ is the natural language action conclusion for step $i$, and $\text{concat}$ is simple string concatenation.

What this computes: a compressed representation of the distant history that discards screenshots (the most memory-intensive modality) while preserving the semantic content of what happened (through the natural language conclusions). The recent window $N$ retains full multimodal information for contextual reasoning about the current situation.

Why this form: the design trades off visual fidelity against memory efficiency. Screenshots far in the past are rarely needed for precise spatial reasoning—the model primarily needs to know what happened (e.g., "I opened the weather app and recorded Paris temperature as 18°C"), not exactly what the screen looked like 15 steps ago. Concatenating natural language conclusions provides this semantic summary at minimal token cost. Alternative approaches like learned compression or keyframe selection would be more complex and potentially lossy in unpredictable ways; simple concatenation of human-readable text is transparent and leverages the fact that the model already generates these conclusions as part of its output.

The paper does not specify the exact value of $N$ (the recent window size) in this section, which is a notable omission for reproducibility. The interaction flow in Figure 3 visually depicts "Compressed Histories" as a separate input component alongside the current screenshot and task instruction, confirming the architecture but not the hyperparameter.

Grounding Data Construction Pipeline

Grounding—the ability to locate a specific UI element given a natural language description like "the search button in the top-right corner"—is a prerequisite for GUI automation. Every click, type, and scroll action requires the model to produce precise coordinates for the target element. The paper argues that existing grounding datasets suffer from "limited complexity and diversity," motivating a comprehensive data augmentation framework with two complementary strategies (Section 2.2.1, Figure 4).

Hard Grounding Data Generation: Synthetic Complex Scenarios

This strategy targets scenarios that are difficult, expensive, or impossible to collect through standard means—professional software with complex interfaces, high-resolution multi-window setups, and domain-specific applications where manual annotation would require expert knowledge.

Challenging App GUI Grounding Data Synthesis. The paper generates synthetic screenshots of professional applications using MLLMs (Multi-modal Large Language Models). The process works as follows:

  1. Source material: annotated UI elements and reference interfaces from existing datasets provide templates for what realistic professional software looks like—button placements, menu structures, dialog box layouts.

  2. Synthesis: an MLLM generates new professional application screenshots by composing these elements in novel configurations, creating diverse and realistic-looking interfaces that the model will need to ground against.

  3. Quality assurance: the generated interfaces undergo "iterative quality assessment and refinement mechanisms"—validation checks identify unrealistic or domain-inaccurate interfaces, and corrective regeneration fixes them. The paper does not specify what these validation checks consist of (e.g., rule-based verification of layout constraints, VLM-based realism scoring), which is a notable omission for assessing the pipeline's robustness.

The key insight: by controlling the generation process, the system can create large volumes of challenging grounding examples (complex layouts, unusual element placements, domain-specific terminology) that would be rare in naturally occurring screenshots but are essential for robust grounding in professional software contexts.

Multi-window High-resolution Grounding Data Synthesis. Real desktop environments frequently involve multiple overlapping windows at high resolutions, where the target UI element might be partially occluded, in the background, or on a secondary monitor. Standard grounding datasets typically use single-window, moderate-resolution screenshots, creating a distribution shift at deployment.

The synthesis procedure:

  1. Candidate organization pools: the system maintains pools of window configurations varying in count (2, 3, 4+ windows), layout patterns (side-by-side, cascaded, grid), and resolution options (standard HD through 4K+).

  2. Composition: given an existing single-window grounding dataset, the system composites these windows into multi-window scenarios by sampling from the organization pools.

  3. Spatial constraint validation: the system ensures target UI elements remain unoccluded—if a grounding query asks "click the Save button," that button must actually be visible in the final composite image, not hidden behind another window. This constraint is enforced algorithmically using bounding box information.

Why synthesize rather than collect real multi-window data: collecting genuine multi-window screenshots with annotated grounding labels at scale would require instrumenting real user sessions or hiring annotators to create specific window configurations—both prohibitively expensive. Synthesis provides controlled complexity (you can dial up the number of windows or the resolution systematically) and guaranteed label quality (you know exactly where each element is because you placed it there).

High-Quality Grounding Data Extension: Scalable Augmentation

This strategy aims for cost-effective, high-volume data augmentation from existing resources, targeting scale rather than complexity.

Trajectory-based grounding extraction. The paper mines grounding annotations from existing trajectory data collected on real devices and in simulated environments. The key observation: every click action in a trajectory implicitly contains a grounding annotation—the model knew where to click when it executed that step. The procedure:

  1. Extraction: for each action in existing trajectories, extract the natural language description of what the agent was trying to click (from the action conclusion or task context) and the actual click coordinates.

  2. Quality filtering: a "critic model" (likely a VLM prompted to judge grounding quality) filters these extracted pairs, retaining only high-fidelity grounding annotations. The paper does not provide details on the critic model's architecture or prompting strategy.

  3. Validation criterion: the critic ensures the natural language description uniquely and accurately identifies the clicked element—filtering out ambiguous descriptions like "click the button" (which button?) in favor of specific descriptions like "click the blue Submit button in the lower-right corner of the form."

This approach leverages the fact that successful trajectories already encode correct grounding information—the agent clicked the right thing, so the click location + context provides a positive grounding example. The filtering step is critical because not all clicks in a trajectory are precisely targeted (e.g., scrolling doesn't target a specific element).

Tutorial-based knowledge mining. Application tutorials (video walkthroughs, written guides, forum posts) contain rich grounding-related information embedded in natural language: "click the File menu, then select Save As from the dropdown." The paper parses these tutorials to extract grounding-oriented question-answer pairs:

  1. Parsing: for video tutorials, the system analyzes embedded subtitles to identify action descriptions; for written guides, it parses instructional text.

  2. Spatial-semantic relationship identification: the system links action descriptions to likely on-screen locations. For example, "click the Format menu" implies a grounding query targeting the Format menu item, which can be paired with a screenshot where that menu is visible.

  3. QA pair generation: the system produces comprehensive grounding-oriented QA pairs that capture real-world usage patterns—e.g., Q: "Where is the Format menu?" A: coordinates (150, 45).

The advantage of tutorial mining: tutorials demonstrate how actual users interact with applications, capturing workflows and element names that might not appear in synthetic data. The disadvantage: the spatial alignment between tutorial text and screenshots is noisy (the tutorial might describe an older version of the software), and the paper does not discuss how this alignment noise is handled.

Infeasible query generation. This addresses a critical gap in grounding datasets: they typically contain only positive examples where the queried element actually exists on screen. At deployment, the model will encounter infeasible queries (e.g., "click the Print button" when no Print button exists), and it must learn to recognize and refuse these rather than hallucinating coordinates on a random location. The generation procedure:

  1. Strategic random pairing: the system randomly pairs queries from one interface with elements from a different interface—e.g., a query about a "Save button" paired with a screenshot that contains only navigation elements.

  2. Multi-model consensus filtering: multiple models (likely VLMs with different architectures or prompts) independently judge whether each random pairing is genuinely infeasible. A pairing is retained only if the models agree that the queried element does not exist in the screenshot.

  3. Consensus criterion: this filtering step prevents false negatives where a model incorrectly claims an element doesn't exist (e.g., because the model itself has poor grounding capability). By requiring consensus, the system reduces the noise in the negative examples.

Why negative examples matter: a grounding model trained only on positive examples will always output coordinates somewhere—it has no concept of "this element doesn't exist." At deployment, this leads to the agent clicking random locations when it misunderstands a task or when the expected UI element has moved/changed. Training with explicit negative examples teaches the model to output a refusal token or an "element not found" signal, enabling the agent to recognize and recover from grounding failures.

The paper does not report the scale of the grounding dataset (number of examples per category, distribution across device types), nor does it provide ablations isolating the contribution of each grounding data source. The grounding performance results in Tables 3–7 show substantial improvements over the predecessor GUI-Owl, but the relative importance of synthetic hard grounding vs. trajectory-based extraction vs. tutorial mining vs. negative examples is not disentangled.

Trajectory Data Collection Pipeline

Trajectory data—sequences of (screenshot, action) pairs demonstrating complete task execution—is the primary supervision for teaching the model what to do. The paper builds a "hybrid trajectory corpus" that combines four complementary sources, each addressing different failure modes of the others (Section 2.2.2, Figure 5).

Task Production via Human-Authored DAGs

The core mechanism for generating diverse, realistic task instructions is a Directed Acyclic Graph (DAG) authored by human annotators for each application domain. This is the paper's solution to the problem of LLM hallucination in task generation—if you ask an LLM to generate task instructions, it may produce nonsensical task sequences that don't correspond to valid UI flows. The DAG constrains generation to known-valid sequences.

Formally, for each application domain, annotators construct:

G=(V,E),V={vi}i=1∣V∣,E⊆V×VG = (V, E), \quad V = \{v_i\}_{i=1}^{|V|}, \quad E \subseteq V \times V

where each node $v_i$ represents an atomic subtask (e.g., "open the search bar," "enter a search query," "tap the first search result"), and each edge $(v_i, v_j) \in E$ represents a feasible transition—meaning that after completing subtask $v_i$, the UI state naturally allows proceeding to subtask $v_j$.

Let $S \subseteq V$ be the set of valid start nodes (subtasks that can begin a task) and $T \subseteq V$ be the set of valid terminal nodes (subtasks that can conclude a task). A task is synthesized by sampling a path from $S$ to $T$:

p=(v1,…,vK),v1∈S,  vK∈T,  (vk,vk+1)∈Ep = (v_1, \ldots, v_K), \quad v_1 \in S, \; v_K \in T, \; (v_k, v_{k+1}) \in E

What this computes: a valid multi-step task specification as a sequence of $K$ atomic subtasks that are guaranteed (by the DAG structure) to represent a realistic, executable workflow in the target application. Each node $v_k$ is associated with a sub-instruction template $d(v_k)$ that optionally contains slots for diverse entities (e.g., "search for [PRODUCT_NAME]" where [PRODUCT_NAME] can be instantiated with different products).

The final task instruction $\mathcal{I}(p)$ is composed by concatenating and rewriting the ordered sub-instructions:

I(p)=Compose⁡(d(v1),d(v2),…,d(vK))\mathcal{I}(p) = \operatorname{Compose}(d(v_1), d(v_2), \ldots, d(v_K))

Why this form: the DAG provides controllable coverage of high-frequency operation patterns while preventing LLM hallucination. By sampling diverse paths through the graph and instantiating templates with different entities, the system generates large volumes of realistic task instructions without requiring an LLM to invent task structure from scratch. The human annotation cost is in constructing the DAG (one-time per application domain), after which task generation is automated.

The \operatorname{Compose} function is not formally specified—the paper says it concatenates and rewrites sub-instructions, but the rewriting mechanism (template-based? LLM-based?) is not detailed. This is a gap for reproducibility, as the quality of composed instructions affects the naturalness of the training data.

Automated Trajectory Generation with Checkpointing, Truncation, and Task Repair

Given a synthesized task instruction $\mathcal{I}(p)$, the system attempts to execute it automatically using an agent interacting with a real device environment $\mathcal{E}$. This produces a raw trajectory:

τ={(ot,at)}t=1T\tau = \{(o_t, a_t)\}_{t=1}^T

where $o_t$ is the observation at step $t$ (screenshot, UI structure if available, device metadata) and $a_t$ is the executed action (touch, keyboard input, or tool call).

The critical challenge: automated agents are imperfect. They often partially complete a task (e.g., open the right app and search correctly, but fail to extract the correct information) and then continue producing erroneous steps. Including these erroneous steps in training data would teach the model to make mistakes. The paper's solution is checkpoint-based validation with truncation and repair.

Checkpoint predicates. For each subtask node $v_k$ in the DAG path $p$, the system defines a binary predicate:

ϕk:O→{0,1},ϕk(ot)=1 iff subtask vk is satisfied at ot\phi_k: \mathcal{O} \rightarrow \{0, 1\}, \quad \phi_k(o_t) = 1 \text{ iff subtask } v_k \text{ is satisfied at } o_t

What this computes: a boolean check of whether the current observation $o_t$ satisfies the completion condition for subtask $v_k$. For example, if $v_k$ is "open the search bar," $\phi_k(o_t) = 1$ when the screenshot shows an active search bar.

The paper does not specify how $\phi_k$ is implemented—possibilities include VLM-based verification (prompting a VLM to check if the subtask is complete), UI-tree-based heuristics (checking if specific UI elements are present/active), or OCR-based text matching. The implementation choice affects the reliability of the validation; VLM-based verification could introduce its own errors, while heuristic-based verification may miss subtle failure modes.

Prefix completion score. To assess how much of the task was completed before the agent went off-track:

ck(τ)=max⁡t∈[1,T]ϕk(ot)c_k(\tau) = \max_{t \in [1, T]} \phi_k(o_t)

What this computes: for subtask $k$, whether it was achieved at any point during the trajectory. The $\max$ over time means the system is generous—if the agent correctly completed subtask $k$ at step 5 but then later navigated away (undoing that completion), the subtask is still counted as completed.

Longest completed prefix length:

m(τ)=max⁡{m∈{0,…,K}:∀k≤m,  ck(τ)=1}m(\tau) = \max \{ m \in \{0, \ldots, K\} : \forall k \leq m, \; c_k(\tau) = 1 \}

What this computes: the longest contiguous prefix of the subtask path where all subtasks up to that point were completed at some time during the trajectory.

Why this form (prefix rather than subset): GUI tasks are sequential by nature—you must complete subtask 1 before subtask 2, subtask 2 before subtask 3, etc. Checking for a prefix ensures the trajectory demonstrates correct ordering of operations. A subset-based criterion (did the agent complete subtasks 1, 3, and 5 but not 2 and 4?) would accept trajectories where the agent took an incorrect path but coincidentally hit some target states.

Decision and repair. If $m(\tau) = K$ (all subtasks completed in order), the trajectory is accepted as correct. If $m(\tau) < K$, the system:

  1. Truncates the trajectory to the last verified checkpoint:

t⋆=max⁡{t:ϕm(τ)(ot)=1},τ′={(ot,at)}t=1t⋆t^\star = \max\{t : \phi_{m(\tau)}(o_t) = 1\}, \quad \tau' = \{(o_t, a_t)\}_{t=1}^{t^\star}

What this computes: the last time step where the most-recently-completed subtask $m(\tau)$ was satisfied, and cuts off everything after that point.

  1. Repairs the original task by removing the completed subtasks:

prem=(vm(τ)+1,…,vK),Irem=I(prem)p_{\text{rem}} = (v_{m(\tau)+1}, \ldots, v_K), \quad \mathcal{I}_{\text{rem}} = \mathcal{I}(p_{\text{rem}})

This produces a new, shorter task containing only the subtasks the agent failed to complete.

  1. Stores $(\mathcal{I}_{\text{rem}}, \tau')$ as a partially-correct instance: the trajectory $\tau'$ provides clean supervision for the successfully executed segment, while the remaining instruction $\mathcal{I}_{\text{rem}}$ is queued for another collection attempt.

Why this repair mechanism matters: without it, a trajectory that fails at step 7 of 10 is either discarded entirely (wasting the correctly executed first 6 steps) or included with noisy labels (teaching the model incorrect behavior for steps 7–10). The truncation-and-repair approach salvages the useful prefix while discarding the unreliable suffix, and the repaired instruction provides a natural next task for continued data collection. Over time, the system should converge to collecting full trajectories for all tasks.

Human Annotation on Real Devices

For tasks that remain unsolved after repeated automated attempts (the paper does not specify a retry limit or threshold), the system falls back to human annotation:

"For difficult tasks that remain unsolved after repeated automated attempts, we collect expert demonstrations via a cloud annotation platform. Annotators directly operate the same real device environments and record gold trajectories $\tau^{\text{human}}$ aligned with the task instruction, ensuring high-quality supervision for hard cases."

The paper does not specify what fraction of trajectories come from human annotation versus automated collection, the qualifications of the annotators, the annotation interface, or quality control procedures. These are important for assessing data quality but are presumably considered operational details rather than research contributions.

Virtual Environment-Based Trajectory Production

This is arguably the paper's most significant data innovation. The motivation is explicit (Section 2.2.2): relying solely on real-world environments for trajectory generation has two fundamental limitations that make pure automation unreliable:

  1. Anti-automation mechanisms: "Real-world applications and software often incorporate CAPTCHA verification, anti-bot mechanisms, and other protective measures that can interrupt or terminate the agent's exploration process." This is a hard constraint—if a food delivery app detects automated behavior and blocks the account, no amount of retry logic will produce a trajectory.

  2. Absence of accurate feedback: "Real-world environments cannot provide accurate feedback, which results in low efficiency of trajectory generation via agent exploration, and often yields trajectories that contain erroneous or redundant steps." The checkpoint predicates $\phi_k$ on real devices rely on VLM-based or heuristic verification, which is imperfect and can produce false positives (thinking the agent succeeded when it didn't).

Virtual environments as a solution. The paper develops "a suite of web-rendering-based virtual environments targeting fine-grained primitive actions (e.g., scroll, drag) and high-frequency difficult scenarios (e.g., document and spreadsheet editing, popular applications)." These are essentially web-based simulations of real application interfaces, built specifically for trajectory generation.

These virtual environments serve two primary purposes that real environments cannot:

  • Precise subtask-level feedback: the simulator exposes programmatic subtask-completion predicates $\tilde{\phi}_k(\tilde{s}_t) \in \{0, 1\}$ that can be checked exactly—there's no VLM error or heuristic ambiguity, because the simulator knows the ground-truth state. If the task is "drag the file to the recycle bin," the simulator knows the file's position and the recycle bin's bounds, and can deterministically check whether the drag operation succeeded.

  • Resistance to anti-automation: there are no CAPTCHAs or bot detection in a simulated environment, so the agent can explore freely without being blocked.

Agent rollout + critic in virtual environments. The procedure mirrors the real-device pipeline but with exact feedback:

  1. Sample a scenario $\omega$ and DAG path $p = (v_1, \ldots, v_K)$.

  2. The agent produces a simulated trajectory:

τ~={(o~t,a~t)}t=1T~\tilde{\tau} = \{(\tilde{o}_t, \tilde{a}_t)\}_{t=1}^{\tilde{T}}

  1. Compute the prefix-progress score using the simulator's exact predicates:

c~k(τ~)=max⁡tϕ~k(s~t),m~(τ~)=max⁡{m:∀k≤m,  c~k(τ~)=1}\tilde{c}_k(\tilde{\tau}) = \max_t \tilde{\phi}_k(\tilde{s}_t), \quad \tilde{m}(\tilde{\tau}) = \max\{m : \forall k \leq m, \; \tilde{c}_k(\tilde{\tau}) = 1\}

  1. Accept if $\tilde{m}(\tilde{\tau}) = K$ (complete success); otherwise truncate to the last verified checkpoint:

t~⋆=max⁡{t:ϕ~m~(τ~)(s~t)=1},τ~′=τ~1:t~⋆\tilde{t}^\star = \max\{t : \tilde{\phi}_{\tilde{m}(\tilde{\tau})}(\tilde{s}_t) = 1\}, \quad \tilde{\tau}' = \tilde{\tau}_{1:\tilde{t}^\star}

Scalable Automated Trajectory Generation via scripted execution. For scenarios where the canonical correct operation is known and can be standardized, the paper takes automation a step further: instead of having an agent explore and potentially make mistakes, the system directly executes a script or RPA (Robotic Process Automation) policy $\rho$:

τ~rpa=Rollout⁡(E~,ρ,ω,p),m~(τ~rpa)=K\tilde{\tau}^{\text{rpa}} = \operatorname{Rollout}(\tilde{\mathcal{E}}, \rho, \omega, p), \quad \tilde{m}(\tilde{\tau}^{\text{rpa}}) = K

What this computes: a guaranteed-correct trajectory produced by executing a pre-defined policy in the virtual environment. The $\operatorname{Rollout}$ function simply executes the script and records the resulting observations and actions; the $\tilde{m} = K$ confirms that the script indeed completed all subtasks.

The paper gives the example of a virtual word document editor: for a task like "center-align the title and change its font to 16pt bold," an LLM decomposes the instruction into atomic operations that the virtual editor can execute (select text → click center-align button → open font dialog → change size to 16 → click bold → confirm), and these operations are fed into the simulator to produce a clean trajectory.

Why scripted virtual-environment generation is powerful: it produces perfect trajectories at essentially zero per-trajectory cost beyond the initial environment construction. There's no exploration noise, no retries, no checkpoint validation failures. The limitation is that the scripted approach only works for tasks where the correct sequence of actions is known in advance and can be expressed algorithmically—it doesn't help with open-ended tasks where multiple valid strategies exist or where the correct action depends on dynamic content.

What the virtual environments actually simulate. The paper mentions "document and spreadsheet editing" and "popular applications" but does not enumerate the specific virtual environments built. The ablation in Table 11 uses PC-Eval (atomic desktop operations like drag/scroll, office document and spreadsheet editing) and Mobile-Eval (Chinese mobile apps for food delivery, ride-hailing, ticket booking). From these evaluation domains, we can infer that virtual environments were built for at least: (1) desktop document editing (word processor), (2) desktop spreadsheet manipulation, (3) desktop drag-and-scroll interactions, (4) several Chinese mobile application scenarios. The full scope of virtual environments is not disclosed.

Cost of virtual environment construction. A major practical question that the paper does not address: how much engineering effort was required to build each virtual environment? A virtual environment that accurately simulates a food delivery app with realistic search, ordering, and payment flows is a non-trivial software project. The paper's "Vibe Coding" reference (Section 1) suggests that some environments were generated with AI assistance, but the specific process and effort are not detailed. This is a critical omission for assessing the scalability of the approach—can a small team build virtual environments for the long tail of applications, or is this approach practical only for high-frequency, high-value application domains?

Agent Capability Enhancement Pipeline

Raw trajectory data teaches the model what to do (click here, type that), but a capable GUI agent needs higher-order skills: understanding application semantics, anticipating screen changes, reasoning about task progress, remembering transient information, and invoking tools appropriately. The paper introduces three complementary strategies to inject these capabilities (Section 2.3, Figure 6).

GUI Knowledge Injection

The goal is to enrich the model's understanding of GUI applications beyond what trajectory data alone conveys. The paper crawls "a substantial volume of data from diverse sources on the Internet" to construct a knowledge base, then converts this into training data in two formats.

Data sources (three categories):

  1. Official documentation and tutorials: software manuals, help pages, and tutorial content for applications like Microsoft Office and LibreOffice. These contain authoritative information about feature locations, keyboard shortcuts, and workflow descriptions.

  2. Software forums and Q&A platforms: user-generated content from platforms like WPS Academy and Baidu Jingyan (a Chinese how-to platform). These capture real user questions and answers, reflecting the knowledge gap between application functionality and user understanding.

  3. Web navigation information: extracted from existing open-source web datasets, covering how to navigate websites and web applications.

QA and VQA data construction. After cleaning the crawled data (the paper does not specify cleaning criteria), LLMs rewrite the information into two formats:

  • Task-level QA: e.g., Q: "How do you insert a table of contents in Microsoft Word?" A: "Go to the References tab, click Table of Contents, and select a style from the dropdown."

  • Step-level VQA (Visual Question Answering): pairs a screenshot of an application interface with a question about a specific element or function visible in that screenshot.

This data is incorporated into the pre-training corpus, not as a separate training stage, so the model absorbs GUI knowledge alongside other capabilities.

World Modeling. This is a more sophisticated capability injection: teaching the model to predict how the screen will change before executing an action. The intuition is that a model with an internalized understanding of GUI dynamics will make better decisions—it can mentally simulate "if I click this dropdown, a menu will appear with these options" and plan accordingly.

The construction procedure (paraphrasing Section 2.3.1):

  1. From existing trajectory recordings, take a pair of consecutive screenshots and the action that was executed between them (e.g., screenshot_before, action="click Settings button", screenshot_after).

  2. Prompt a proprietary VLM (the paper names Claude-4.5 as an example) with the first screenshot and the action, asking it to "produce a fine-grained description of the subsequent screenshot, explicitly highlighting the state transitions"—newly appeared dialogs, changed text fields, shifted focus, updated visual elements.

  3. The VLM's description (which is conditioned on the action but not on the actual subsequent screenshot—it's a prediction, not a description) becomes a training target.

  4. The model is trained to generate these state-transition descriptions given a screenshot and an action, learning to anticipate consequences.

Why this matters for agent performance: this is a form of model-based reasoning injected into what is fundamentally a model-free policy (the agent doesn't actually simulate during inference; it just predicts actions). By training the model to predict consequences during pre-training, the model's internal representations may become more causally structured—the model learns that "clicking Save" typically causes "a file dialog appears" or "a confirmation message appears," and this knowledge can inform action selection even if the explicit prediction isn't generated at inference time.

The paper does not provide an ablation isolating the effect of world modeling data, so its contribution to downstream task performance is unknown. It is bundled into the pre-training corpus alongside grounding data, trajectory data, and QA data, making its individual impact impossible to assess from the reported results.

Unified CoT Synthesis

This is the mechanism that transforms raw action sequences into reasoning-rich training examples. The core idea: after collecting trajectory data (through agent exploration, human annotation, or virtual environments), the system post-processes each step to generate synthetic reasoning content—observation descriptions, memory annotations, reflections on action outcomes, and task progress assessments. These are generated by prompting proprietary VLMs and LLMs, not by humans.

The pipeline for step $i$ of a trajectory (Section 2.3.2):

Step 1: Screen observation and query-relevant extraction. A VLM is prompted to:

  • Describe the screen content at step $i$
  • Extract information from the screenshot that is relevant to the user query

Step 2: Memory management. For queries that require memorizing on-screen information (the paper gives the example "Check the weather in Paris and London for next Monday and record it in the memo"):

  • Extract query-relevant content that may be needed in subsequent steps (e.g., the temperature values for Paris and London)
  • Incorporate this information into a memory annotation that will be referenced in later steps

Step 3: Progress reflection. The system feeds three pieces of information to a VLM:

  • The action parameters executed at step $i$
  • The screenshot before execution (the state at step $i$)
  • The screenshot after execution (the state at step $i+1$)
  • The user query

The VLM determines whether the execution outcome matches expectations:

  • If the screen state changed as expected (e.g., clicking the search button caused search results to appear), the task progress is updated.
  • If the change is inconsistent with expectations (e.g., clicking a button produced no visible change, or an error dialog appeared), the system generates corresponding reflections and error corrections to inform the next action decision.

Step 4: Thought and conclusion synthesis. The observation, memory, reflection, and progress information from steps 1–3 are fed into an LLM to synthesize:

  • Thought: a simulation of the agent's reasoning process integrating all the above information for action decision-making—this is what the model will be trained to generate as $\mathcal{C}_t$ (the action conclusion)
  • Conclusion: a concise action decision

If the trajectory involves tool invocation, the tool definitions from the tool set are also provided as input to the LLM, so the synthesized thought incorporates reasoning about tool selection and invocation (e.g., "I could navigate through the file menu, but using filesystem_read_text_file is faster").

What capabilities this CoT synthesis enables (stated explicitly in the paper):

  • Long-horizon decision-making: by including reflection on execution outcomes and analysis of task progress, the synthesized CoT teaches the model to track where it is in the overall task and adjust its strategy accordingly, rather than treating each step independently.

  • Memory capability: by explicitly annotating information worth remembering and referencing it in later steps' thoughts, the CoT teaches the model to extract and retain key on-screen information across multiple steps.

Key caveat: the CoT data is synthetic and dependent on proprietary model quality. The observation descriptions, memory annotations, and reflections are generated by prompting proprietary VLMs and LLMs. If those models make systematic errors (missing screen changes, hallucinating remembered information, incorrectly judging progress), those errors propagate into the training data. The paper does not analyze the quality or error rate of the synthesized CoT, which is a significant omission for assessing this approach's ceiling.

Complementarity of virtual environments and CoT synthesis. The paper is clear that these are complementary: virtual environments improve trajectory coverage and quality (ensuring the underlying action sequences are correct), while CoT synthesis enhances reasoning and decision-making supervision (teaching the model why those actions are correct and what to think about while executing them). The ablations in Tables 10 and 11 support this: removing either causes substantial but not catastrophic drops, suggesting both contribute meaningfully but neither is solely responsible for the model's performance.

Multi-Agent Collaboration

The paper introduces the Mobile-Agent-v3.5 framework as a structured multi-agent system that can use GUI-Owl-1.5 as a component (Section 2.3.3). The framework is used during data collection (generating trajectories in a structured multi-agent format) and serves as an optional deployment mode. It instantiates four role-specialized modules that execute in a closed loop.

Problem setup. Given a user instruction $I$ and the current device state $S_t$ (screenshot, UI tree, device metadata), the goal is to produce an action $a_t \in \mathcal{A}$ that drives the environment to $S_{t+1} \sim P(\cdot \mid S_t, a_t)$ until termination.

Roles and state variables. The system maintains four agent roles plus a shared state:

The system state at step $t$ is:

Xt≜(I,St,SSt,Ft−1,Nt)X_t \triangleq (I, S_t, SS_t, F_{t-1}, N_t)

where $I$ is the user instruction, $S_t$ is the current device state, $SS_t$ is the ordered subgoal list, $F_{t-1}$ is the latest feedback from the previous step, and $N_t$ is persistent notes (memory across steps).

  • Manager (Planner): decomposes the instruction into subgoals and updates them dynamically:

SS0=fM(I,KRAG),SSt+1=uM(SSt,Ft,St+1)SS_0 = f_M(I, K_{\text{RAG}}), \quad SS_{t+1} = u_M(SS_t, F_t, S_{t+1})

where $f_M$ is the initial subgoal generation function, $K_{\text{RAG}}$ denotes optionally retrieved external knowledge (the paper mentions this but does not detail the retrieval mechanism), and $u_M$ is the subgoal update function that modifies the remaining subgoals based on feedback and new state.

  • Worker (Executor): given the current context, selects a subgoal and produces the next action:

at∼πW(⋅∣I,St,SSt,Ft−1,Nt)a_t \sim \pi_W(\cdot \mid I, S_t, SS_t, F_{t-1}, N_t)

The Worker can optionally produce a structured tuple with rationale and a normalized action schema. The paper notes this but doesn't specify the schema format.

  • Reflector (Verifier): after executing $a_t$ on the device, judges the transition and provides diagnostic feedback:

(jt,ϕt)=fR(St,at,St+1),jt∈{SUCCESS,FAILURE}(j_t, \phi_t) = f_R(S_t, a_t, S_{t+1}), \quad j_t \in \{\text{SUCCESS}, \text{FAILURE}\}

and the feedback is set to $F_t \triangleq (j_t, \phi_t)$, where $\phi_t$ is the diagnostic feedback (presumably natural language explaining what went wrong or confirming success).

  • Notetaker (Memory): upon successful progress, extracts and stores salient transient information:

Nt+1={uC(Nt,St+1)if jt=SUCCESS,Ntotherwise.N_{t+1} = \begin{cases} u_C(N_t, S_{t+1}) & \text{if } j_t = \text{SUCCESS}, \\ N_t & \text{otherwise}. \end{cases}

What this computes: the Notetaker only updates persistent notes when the previous action was successful—if the action failed, the old notes are retained because the failed action presumably didn't produce reliable new information.

Execution loop. The framework iterates $(SS_t, a_t, F_t, N_t)$ updates until all subgoals are completed or a termination condition is met (success, timeout, safety stop). The paper emphasizes that this design "isolates planning, execution, verification, and memory" while remaining compatible with the multi-platform interfaces used throughout training and evaluation.

How GUI-Owl-1.5 fits in. The paper states that GUI-Owl-1.5 can function not only as a standalone end-to-end agent but also as the specialized roles within this framework. During trajectory collection, the framework is used for agent exploration, and the resulting multi-agent trajectories (with explicit Manager plans, Worker actions, Reflector feedback, and Notetaker memory updates) become part of the training data. This means GUI-Owl-1.5 learns both the end-to-end mode and the decomposed multi-agent mode during training, enabling flexible deployment.

Important ambiguity. The paper does not specify whether the four roles share the same model weights or are separate instantiations of GUI-Owl-1.5 with different prompting. It also does not specify whether the multi-agent data collection format significantly differs from the single-agent format after CoT synthesis—the CoT synthesis already produces observation, memory, reflection, and progress annotations, which map naturally onto the Reflector and Notetaker roles. The multi-agent framework may be more of a runtime decomposition pattern than a fundamentally different training data format.

Three-Stage Training Paradigm

GUI-Owl-1.5 is initialized from Qwen3-VL and trained through three sequential stages, each with expanding task coverage and increasing optimization sophistication (Section 2.4).

Pre-training

The pre-training stage constructs a large-scale corpus that extends beyond basic GUI understanding. The corpus includes (Section 2.4.1):

  1. UI recognition and trajectory data: inherited from GUI-Owl's pre-training, covering basic GUI element perception and action sequences.

  2. QA and VQA knowledge data: the GUI knowledge data from Section 2.3.1 (software documentation, forum Q&A, web navigation), strengthening general visual reasoning and knowledge comprehension about applications.

  3. World-modeling data: the action-conditioned state-transition descriptions from Section 2.3.1, training the model to predict how GUI states change in response to actions.

  4. Tool invocation data: training examples that familiarize the model with tool-calling semantics and MCP interaction patterns from the earliest stage, ensuring the expanded action space is represented throughout training.

The paper does not provide hyperparameters for the pre-training stage (learning rate, batch size, number of steps, data mixture ratios), which limits reproducibility. The rationale for including all data types in pre-training (rather than deferring some to SFT) is that early exposure to the full range of modalities and task types allows the model to develop general representations that support all downstream capabilities, rather than trying to bolt on tool-use or world-modeling understanding during fine-tuning.

Supervised Fine-Tuning (SFT)

SFT aligns the pre-trained model with diverse agentic tasks across multiple devices. The SFT data specifically covers (Section 2.4.2):

  • Multi-device trajectory data with CoT annotations: the complete trajectory collection with synthesized thoughts, conclusions, memory annotations, and reflections from the unified CoT synthesis pipeline.

  • Augmented grounding data: the full grounding corpus from Section 2.2.1, including hard grounding synthesis, trajectory-based extraction, tutorial mining, and infeasible queries.

  • Structured tool invocation supervision: training examples for both conventional tool calls and MCP-based interactions, teaching the model when and how to use tools alongside GUI operations.

  • Dedicated browser interaction data: data capturing the unique characteristics of web-based GUIs (heterogeneous page structures, dynamic content loading, URL-based navigation).

The paper states that this stage "transforms the pre-trained model into a capable multi-device agent supporting GUI manipulation, tool invocation, and browser automation with explicit reasoning." No SFT hyperparameters are provided.

Reinforcement Learning: MRPO

The final stage is MRPO (Multi-platform Reinforcement Policy Optimization), a large-scale RL framework that addresses four specific challenges in GUI agent training (Section 2.4.3, Figure 7). This is the most technically novel component of the training pipeline.

Challenge 1: Multi-device RL with a unified policy. The system must optimize a single policy across three device families: $\mathcal{D} = \{\text{mobile}, \text{desktop}, \text{web}\}$. Each device defines its own environment $\mathcal{E}_d$, action space $\mathcal{A}_d$, and observation stream. The solution is a device-conditioned policy:

πθ(a∣o,d),a∈Ad\pi_\theta(a \mid o, d), \quad a \in \mathcal{A}_d

What this computes: the probability of action $a$ given observation $o$ and device type $d$. The device type $d$ is provided as a conditioning signal (likely as a text token in the prompt), allowing the policy to specialize its behavior based on which platform it's operating on.

Why this form: a unified policy without device conditioning would need to learn to infer the device type from the observation alone (which is possible but adds complexity) and to produce device-appropriate actions without explicit context. Device conditioning simplifies the learning problem by making the device identity an explicit input, allowing the shared backbone to learn cross-device representations while the device-specific behavior is controlled by the conditioning signal.

Challenge 2: Online rollout buffer for GRPO under outcome collapse. The paper uses GRPO-style grouped rollouts. For a task $x$, the system samples a group of $n$ trajectories $\{\tau_i\}_{i=1}^n$ from the current policy. GRPO computes advantage estimates within each group by comparing trajectory outcomes—but this only works if the group contains both successful and unsuccessful trajectories.

The problem: for many GUI tasks, it is common that all $n$ rollouts produce identical outcomes (all success or all failure). Let $Z(\tau) \in \{0, 1\}$ denote the binary terminal outcome. The collapse event is:

Collapse(Gn)≜(∑τ∈GnZ(τ)∈{0,n})\text{Collapse}(\mathcal{G}_n) \triangleq \left(\sum_{\tau \in \mathcal{G}_n} Z(\tau) \in \{0, n\}\right)

What this computes: a boolean indicating whether the group is uninformative—either every trajectory succeeded ($\sum = n$) or every trajectory failed ($\sum = 0$). In either case, there's no contrast between good and bad trajectories to compute meaningful advantage estimates.

A traditional solution would be to use a replay buffer (mixing trajectories from previous policies to increase diversity), but this introduces off-policy bias. The paper's solution is an online rollout buffer that increases within-group diversity while remaining on-policy.

The procedure (Section 2.4.3):

  1. Oversample: for each task $x$, temporarily sample $kn$ rollouts on-policy:

Gkn(x)={τi}i=1kn,τi∼πθ(⋅∣x)\mathcal{G}_{kn}(x) = \{\tau_i\}_{i=1}^{kn}, \quad \tau_i \sim \pi_\theta(\cdot \mid x)

What this computes: a pool of $kn$ trajectories, all from the current policy. The oversampling factor $k$ is a hyperparameter; the paper does not specify its value.

  1. Subsample with diversity guarantee: the training group $\widehat{\mathcal{G}}_n(x)$ is constructed from the pool through a procedure that explicitly avoids collapse.

Define the pool-diversity event:

A≜(0<∑τ∈GknZ(τ)<kn)\mathcal{A} \triangleq \left(0 < \sum_{\tau \in \mathcal{G}_{kn}} Z(\tau) < kn\right)

What this computes: a boolean indicating whether the pool itself contains both successes and failures. If the pool is all-success or all-failure, no subsampling strategy can produce a diverse group—the model is either perfect or completely incapable on this task, and no amount of clever sampling helps.

The probability of pool diversity is:

P(A)=1−pkn−(1−p)kn,p≜Pτ∼πθ(⋅∣x)[Z(τ)=1]\mathbb{P}(\mathcal{A}) = 1 - p^{kn} - (1-p)^{kn}, \quad p \triangleq \mathbb{P}_{\tau \sim \pi_\theta(\cdot \mid x)}[Z(\tau) = 1]

What this computes: the probability that a pool of $kn$ i.i.d. Bernoulli trials with success probability $p$ contains at least one success and at least one failure. This probability grows rapidly with $k$—for $p = 0.9$ and $n = 4$, a standard group ($k = 1$) has only about 34% chance of diversity, but an oversampled group with $k = 4$ has about 81% chance.

The construction of the training group:

G^n(x)={Subsample⁡n(Gkn(x)),¬Collapse⁡(Subsample⁡n(Gkn(x))),Swap1⁡(Subsample⁡n(Gkn(x)),Gkn(x)),Collapse⁡(Subsample⁡n(Gkn(x)))∧A,∅,¬A,\widehat{\mathcal{G}}_n(x) = \begin{cases} \operatorname{Subsample}_n(\mathcal{G}_{kn}(x)), & \neg\operatorname{Collapse}(\operatorname{Subsample}_n(\mathcal{G}_{kn}(x))), \\ \operatorname{Swap1}(\operatorname{Subsample}_n(\mathcal{G}_{kn}(x)), \mathcal{G}_{kn}(x)), & \operatorname{Collapse}(\operatorname{Subsample}_n(\mathcal{G}_{kn}(x))) \wedge \mathcal{A}, \\ \varnothing, & \neg\mathcal{A}, \end{cases}

What this computes (case by case):

  • Case 1: if a random subsample of $n$ trajectories from the pool happens to already be diverse (not collapsed), use it directly—this is the ideal outcome.

  • Case 2: if the random subsample collapsed (all success or all failure) but the overall pool $\mathcal{A}$ is diverse (it contains both outcomes), apply $\operatorname{Swap1}$: replace one random element in the subsample with a random element of the opposite outcome from the pool. This guarantees $0 < \sum_{\tau \in \widehat{\mathcal{G}}_n} Z(\tau) < n$.

  • Case 3: if the pool itself is not diverse (all trajectories have the same outcome), return $\varnothing$ (empty group)—this task is skipped for this training step because no informative group can be formed.

Why this works (on-policy guarantee): the paper asserts that uniform subsampling from an on-policy pool preserves the marginal distribution:

E[1n∑τ∈Gn(x)f(τ)]=Eτ∼πθ(⋅∣x)[f(τ)]\mathbb{E}\left[\frac{1}{n} \sum_{\tau \in \mathcal{G}_n(x)} f(\tau)\right] = \mathbb{E}_{\tau \sim \pi_\theta(\cdot \mid x)}[f(\tau)]

What this means: for any statistic $f(\tau)$, the expected value over the subsampled group equals the expected value under the current policy. This holds because $\mathcal{G}_{kn}(x)$ is i.i.d. on-policy and $\mathcal{G}_n(x)$ is an exchangeable uniform subset—the subsampling doesn't introduce bias because all trajectories were generated by the same policy. The $\operatorname{Swap1}$ operation in Case 2 technically breaks strict uniformity, but the paper argues the estimator remains "approximately unbiased" (the formal analysis of the bias introduced by $\operatorname{Swap1}$ is not provided).

Challenge 3: Training-inference log-prob alignment via token-ID transport. This is a subtle but important engineering problem. In the RL training loop, the inference service (running on the environment side) generates action text $y$ from the current policy and executes it, producing an outcome. The training process then needs to compute the log-probability $\log \pi_\theta(y \mid x)$ to estimate the policy gradient.

The problem: if the inference-side tokenizer maps $y$ to token IDs differently from the training-side tokenizer (which can happen due to non-unique tokenization of special characters, whitespace, or structured outputs like tool calls), then:

log⁡πθ(y∣x)∣train-tokenize(y)≠log⁡πθ(y∣x)∣infer-tokenize(y)\log \pi_\theta(y \mid x)\big|_{\text{train-tokenize}(y)} \neq \log \pi_\theta(y \mid x)\big|_{\text{infer-tokenize}(y)}

What this means: the log-probability computed at training time doesn't match the log-probability that was actually used to sample the action at inference time. This breaks the assumptions of KL regularization (which compares the current policy to a reference policy using log-probabilities) and policy gradient estimators (which rely on the gradient of the log-probability of the actually-executed action).

The fix: transport the original inference token IDs alongside the textual payload. For each generated sequence $y$, the environment returns both the text $y$ and the token-ID sequence $\mathbf{t}^{\text{infer}} = (t_1, \ldots, t_L)$ that was used to sample $y$. The training process then computes:

log⁡πθ(y∣x):=∑i=1Llog⁡πθ(ti∣x,t<i)\log \pi_\theta(y \mid x) := \sum_{i=1}^{L} \log \pi_\theta(t_i \mid x, t_{<i})

What this computes: the log-probability of the exact same discrete event that was executed in the environment, guaranteed by using the inference-side token IDs. The sum is over the $L$ tokens in the generated sequence, with each term being the log-probability of token $t_i$ given the prefix context and all preceding tokens.

Why this matters: without this fix, the gradient estimates would be corrupted by tokenization mismatches, potentially causing the policy to optimize in directions that don't correspond to the actions it actually takes. This is a concrete example of how distribution shift between training and inference infrastructure (different machines, different tokenizer versions, different tokenization configurations) can silently break RL training.

Challenge 4: Alternating multi-device optimization to reduce gradient interference. Mixing trajectories from different devices in a single RL batch can induce strong gradient conflicts because the action spaces, UI conventions, and domain priors differ substantially across device families.

Let $g_d = \mathbb{E}_{\tau \sim \pi_\theta, \mathcal{E}_d}[\nabla_\theta \mathcal{L}(\tau)]$ be the device-specific policy gradient for device family $d$. A naive mixture update would use a weighted sum:

g=∑dλdgdg = \sum_{d} \lambda_d g_d

What this computes: the combined gradient as a linear combination of device-specific gradients with weights $\lambda_d$.

Why this fails: when $\langle g_{d_1}, g_{d_2} \rangle < 0$ frequently—meaning the directions that improve mobile performance are opposed to the directions that improve desktop performance—the optimization becomes a "tug-of-war." Parameters oscillate as mobile updates push in one direction and desktop updates push in the opposite, preventing stable convergence on either objective.

The solution: alternating optimization. The paper adopts an alternating schedule across training stages:

θ(s+1)←θ(s)−ηgds,ds∈D\theta^{(s+1)} \leftarrow \theta^{(s)} - \eta g_{d_s}, \quad d_s \in \mathcal{D}

What this computes: at each training stage $s$, the policy is updated using gradients from only one device family $d_s$, with learning rate $\eta$. Device families are visited cyclically or via a curriculum.

Why this works: each stage trains on a single device family (potentially with multiple environments within that family), allowing the optimizer to make focused progress on that device's objective without interference from other devices' gradients. When the stage switches to a different device family, the policy adapts to that family's requirements. Over multiple cycles, the policy should converge to a solution that performs well on all devices, because the shared backbone forces some degree of parameter sharing even though the optimization is temporally separated.

The empirical support comes from Figure 8(b): mix-platform training exhibits performance oscillation (the zigzag pattern in the validation curve), while interleaved training achieves stable improvement and avoids the oscillation. The paper does not provide a formal convergence analysis or theoretical justification for why alternating optimization outperforms mixed optimization beyond the gradient conflict argument.

Unstable-task-focused RL training. Figure 8(a) shows an additional ablation: comparing full dataset training against training focused on "unstable tasks" (derived from multi-round rollouts where the policy's success rate varies significantly). Unstable-task-focused training achieves faster convergence and higher final accuracy. The paper's interpretation: prioritizing challenging, high-variance tasks for RL optimization provides more informative learning signal than uniform training across all tasks, because tasks where the policy already performs consistently (always succeeding or always failing) contribute little to policy improvement. This is a form of curriculum learning based on policy uncertainty rather than task difficulty.

The paper does not specify the criteria for identifying "unstable tasks" or the mechanism for deriving them from multi-round rollouts beyond the brief mention. The ablation is presented in Figure 8 but not discussed in detail in the main text beyond the figure caption.

RL training hyperparameters. The paper does not provide specific hyperparameters for the MRPO stage—no learning rate, batch size, $k$ (oversampling factor), $n$ (group size), number of training stages, stage duration, or device visitation schedule. This is a significant gap for reproducibility, especially given that MRPO is one of the paper's named contributions. Presumably these details are in the open-source release or will be provided in a future technical report, but their absence from the paper limits independent validation.

Summary of Design Choices and Their Justifications

Synthetic ground-truth feedback via virtual environments over pure real-device exploration: real devices provide no reliable feedback signal for whether an action was correct, requiring noisy VLM-based or heuristic verification. Virtual environments provide deterministic, programmatic feedback ($\tilde{\phi}_k$ predicates), enabling clean trajectory truncation and scalable, high-quality data generation. This choice is validated by the Table 11 ablation showing dramatic drops when virtual environment data is removed.

DAG-based task synthesis over LLM-generated tasks: LLMs can hallucinate impossible task sequences (e.g., "click the Save button" in a read-only viewer). Human-authored DAGs constrain task generation to valid UI flows while still enabling diverse path sampling and template instantiation. The human cost is in DAG construction (one-time per application), after which task generation is automated and hallucination-free.

Checkpoint-based trajectory truncation over full-trajectory inclusion: including erroneous steps after the agent goes off-track would teach the model to make mistakes. Truncating at the last verified correct step ($t^\star$) salvages the useful prefix while discarding unreliable data. The repair mechanism (creating $\mathcal{I}_{\text{rem}}$ for the incomplete portion) enables iterative data collection until full trajectories are obtained.

Synthetic CoT annotation over human reasoning annotation: generating step-by-step reasoning with human annotators would be prohibitively expensive at scale. Prompting proprietary VLMs and LLMs to synthesize observation/memory/reflection/progress content enables reasoning-rich training data at automation scale. The tradeoff is dependence on proprietary model quality and potential propagation of systematic errors.

Device-conditioned unified policy over separate per-device policies: a single shared backbone enables cross-device transfer learning and parameter efficiency (one model for all platforms), while the device token allows platform-specific behavior. Separate per-device policies would eliminate gradient interference but sacrifice parameter sharing and require maintaining multiple models.

Online rollout buffer with Swap1 over replay buffer for GRPO diversity: replay buffers introduce off-policy bias that can destabilize policy gradient estimates. The oversample-and-select approach with guaranteed outcome diversity via Swap1 maintains on-policy guarantees while reducing the probability of uninformative (collapsed) training groups by approximately 2–3x in typical success-rate regimes.

Alternating multi-device optimization over mixed-device training: mixing gradients from heterogeneous platforms causes optimization oscillation ($\langle g_{\text{mobile}}, g_{\text{desktop}} \rangle < 0$). Alternating stages isolate device-specific adaptation, enabling focused progress on each platform while keeping a shared backbone for cross-device generalization.

Token-ID transport over text re-tokenization for log-prob computation: re-tokenizing inference-generated text on the training side can produce different token sequences than those used to sample the action, silently corrupting policy gradient estimates. Transporting the original inference token IDs guarantees exact log-probability matching between inference and training.

4. Key Insights and Innovations

Innovation 1: Virtual Environments Are Not Just a Data Shortcut—They Are a Debugging Tool That Reveals How Much GUI Agent Training Depends on Clean Feedback

The dominant assumption in GUI agent data collection—visible across the lineage from Mobile-Agent through UI-TARS to MAI-UI—has been that scaling up real-device exploration with clever filtering will eventually solve the data quality problem. Give the agent more attempts, use verifiers to discard bad trajectories, add human annotation for the hardest cases, and the noise will wash out. The paper's introduction of web-rendering-based virtual environments (Section 2.2.2) is superficially just another data source, but what makes it intellectually distinctive is the diagnostic function it serves: by providing an environment where feedback is deterministic and programmatically exact, the virtual environments reveal how much of prior GUI agent performance was bottlenecked not by model capacity or training scale but by the fundamental noisiness of real-device feedback.

This is not obvious from the mechanism description alone. The key move is the contrast between how checkpoint predicates work in real environments versus virtual ones. On real devices, $\phi_k(o_t)$ must be implemented through VLM prompting or heuristic UI-tree inspection—both noisy, both with non-trivial false-positive rates. When the system truncates a real-device trajectory at $t^\star$, it might be truncating at a point where the agent has already started making mistakes but the VLM verifier hasn't caught them yet. In the virtual environment, $\tilde{\phi}_k(\tilde{s}_t)$ is deterministic: the simulator knows whether the file was dragged to the correct folder because it tracks object positions programmatically. The truncation point is exact, and the resulting trajectory is clean in a way that no amount of filtering of real-device data could achieve.

The ablation in Table 11 quantifies what this diagnostic reveals: removing virtual environment data drops PC-Eval from 75.4% to 42.0% and Mobile-Eval from 86.7% to 50.0%. These are not marginal degradations; they suggest that for certain task categories—precise atomic operations like drag-and-drop, office document editing, and CAPTCHA-heavy app scenarios—the real-device trajectory data is so noisy that the model essentially fails to learn these skills without synthetic ground-truth feedback. Prior work like GUI-Owl and UI-TARS reported non-trivial performance on similar benchmarks without virtual environments, but the ablation suggests those results were achieved despite the data quality problem, likely by overfitting to the subset of tasks where real-device exploration happened to produce clean trajectories.

This finding reframes the data collection problem for GUI agents: it's not that we need more data or more diverse data in aggregate—we need clean feedback data for specific skill categories, and virtual environments provide it. Prior approaches treated all trajectory data as fungible, scaling up volume and hoping quality followed. GUI-Owl-1.5's insight is that data quality is skill-dependent: for simple navigation tasks (open app, search, tap result), real-device exploration with VLM-based verification works adequately; for precise manipulation tasks (drag, spreadsheet editing, CAPTCHA handling), only deterministic feedback produces learnable trajectories.

The implication for the field is that virtual environment construction should not be seen as an optional augmentation but as a necessary investment for specific capability categories. The paper doesn't fully explore how to identify which capabilities require virtual environments versus which can be learned from real-device data, but the ablation provides a template: if removing virtual environment data causes a cliff in performance on a specific benchmark, that benchmark's capabilities are fundamentally feedback-limited on real devices.

A limitation worth noting: the paper does not disclose the engineering cost of building these virtual environments. If building a virtual spreadsheet editor that accurately simulates all formatting operations, formula evaluation, and dialog box interactions requires months of engineering effort, then the insight is more diagnostic than practical—it tells us what's missing from current approaches without providing a scalable path to fill the gap. The paper's use of "Vibe Coding" (Section 1) to generate virtual environments hints at a lower-cost construction pipeline, but without details, the scalability of this approach remains uncertain.

Innovation 2: Explicit Capability Injection Through Synthetic Reasoning Annotation Recasts GUI Agent Training from Implicit Skill Acquisition to Curriculum Design

The default approach to training GUI agents—used by GUI-Owl, UI-TARS, MAI-UI, and essentially all prior native agent models—has been to train on trajectory data and hope that higher-order capabilities (planning, reflection, memory, tool selection) emerge implicitly from the action sequences. If the model sees enough examples of multi-step tasks, the reasoning goes, it will learn to track progress across steps, remember relevant information, and reason about tool use without explicit supervision for those skills.

GUI-Owl-1.5 rejects this hope-based approach and replaces it with something more structured: explicit capability injection through synthetic reasoning annotation (Section 2.3.2). The unified CoT synthesis pipeline takes raw trajectory data—sequences of (screenshot, action) pairs—and post-processes each step to add observation descriptions, memory annotations (what information to retain for later steps), reflection on action outcomes (did the expected screen change occur?), and progress tracking. These annotations are generated by prompting proprietary VLMs and LLMs, not by humans, making the approach scalable.

What makes this intellectually distinctive is not the mechanism—prompting stronger models to annotate training data is a well-established technique (see distillation, rejection sampling, STaR). It's the reframing of what GUI agent training is fundamentally about. Prior work treated GUI agent training as an imitation learning problem: given expert demonstrations, learn to reproduce the actions. Under this framing, the CoT annotations are just a richer form of demonstration that includes not just actions but the reasoning behind them. But the paper's ablation (Table 10) tells a different story: removing CoT synthesis causes OSWorld to drop from 52.9% to 47.4% and AndroidWorld from 71.6% to 65.0%. These are non-trivial but not catastrophic drops—nowhere near the cliff caused by removing virtual environment data. This suggests that the CoT annotations are not primarily teaching the model new skills that were absent from the action sequences; they're helping the model organize and deploy skills it already has more effectively.

If this interpretation is correct, the CoT synthesis is performing a function closer to curriculum design than to demonstration augmentation. By providing explicit memory annotations, the training data teaches the model when to remember information, not just that information exists. By providing reflection annotations, it teaches the model how to evaluate its own actions, not just what actions to take. By providing progress tracking, it teaches the model where it is in the task hierarchy, not just what the next step is. These are metacognitive skills that may be implicit in the action sequences (a model that consistently succeeds at multi-step tasks must be doing something like progress tracking internally), but making them explicit in the training data changes the learning problem from "infer the metacognitive structure from action outcomes" to "learn to produce the metacognitive structure as an intermediate output."

This reframing has implications beyond GUI agents. It suggests that for complex sequential decision-making tasks, the bottleneck in learning from demonstrations may not be the quality of the demonstrations but the explicitness of the auxiliary reasoning that accompanies them. Two demonstrations with identical action sequences could have radically different training value depending on whether the reasoning behind the actions is made explicit. The paper doesn't systematically test this hypothesis—it doesn't compare CoT synthesis against alternative methods for making reasoning explicit (e.g., having humans annotate reasoning, using different CoT synthesis prompts, varying the amount of reasoning detail)—but the ablation establishes the phenomenon and opens the door for more principled investigation.

A critical tension the paper does not resolve: if the CoT annotations are generated by proprietary models (Claude-4.5, etc.), and those models have their own systematic weaknesses (missing certain types of screen changes, hallucinating memory content, misjudging task progress), then GUI-Owl-1.5's reasoning capabilities are bounded by the quality of those proprietary models at the time of data generation. The paper doesn't analyze whether GUI-Owl-1.5's failures correlate with errors in the synthesized CoT annotations, which would be valuable for understanding the ceiling of this approach. If the CoT annotations are sometimes wrong, the model is being trained to produce reasoning that doesn't match reality—potentially creating a form of reasoning hallucination that degrades rather than improves decision-making.

Innovation 3: The Online Rollout Buffer Identifies and Solves a Statistical Problem in GUI RL That Was Previously Treated as an Engineering Inconvenience

When prior work applied RL to GUI agents (UI-TARS-2, EvoCUA), the problem of GRPO outcome collapse—all $n$ sampled trajectories for a task producing identical success or failure outcomes, making the group uninformative for advantage estimation—was treated as an engineering inconvenience. The typical solution was to discard collapsed groups and sample more, accepting the wasted computation as a cost of doing RL on sparse-reward tasks. Some work used replay buffers to increase diversity, accepting the off-policy bias as a necessary tradeoff.

GUI-Owl-1.5's contribution here is not the online rollout buffer mechanism itself—oversampling and subsampling with diversity guarantees is a standard technique in Monte Carlo estimation. What's distinctive is the formal characterization of the problem as a statistical event with a computable probability, followed by a solution that maintains on-policy guarantees while provably reducing the probability of uninformative groups. The paper defines the collapse event precisely:

Collapse(Gn)≜(∑τ∈GnZ(τ)∈{0,n})\text{Collapse}(\mathcal{G}_n) \triangleq \left(\sum_{\tau \in \mathcal{G}_n} Z(\tau) \in \{0, n\}\right)

and computes the probability of avoiding collapse with the oversample-and-select approach:

P(A)=1−pkn−(1−p)kn\mathbb{P}(\mathcal{A}) = 1 - p^{kn} - (1-p)^{kn}

This formalization matters because it transforms the problem from "sometimes RL training doesn't work well" to "for a task with success probability $p$ and group size $n$, the fraction of training steps that are wasted on uninformative groups is $p^n + (1-p)^n$." For typical values (e.g., $p = 0.9$, $n = 4$), this is $0.9^4 + 0.1^4 \approx 0.66$—meaning two-thirds of training steps produce no useful learning signal in the naive approach. The oversampling factor $k$ reduces this waste dramatically: at $k = 4$, the wasted fraction drops to $0.9^{16} + 0.1^{16} \approx 0.19$—a 3.5× reduction in wasted computation.

The Swap1 operation in the buffer construction adds a further guarantee: even when the random subsample is collapsed but the pool is diverse (the second case in the $\widehat{\mathcal{G}}_n(x)$ construction), the system forcibly creates a diverse training group by swapping one element for an opposite-outcome element from the pool. The paper asserts this produces an "approximately unbiased" estimator, which is a claim worth scrutinizing—the Swap1 operation introduces a selection bias because it conditions on the pool being diverse and the subsample being collapsed, which means the swapped-in trajectory is not a uniformly random sample from the policy. The paper doesn't provide a formal bias analysis, which is a limitation, but the practical effect is likely small relative to the benefit of having informative training groups.

What makes this more than just an engineering optimization is that it changes the scaling properties of RL for GUI agents. In the naive approach, as the policy improves (increasing $p$), the fraction of wasted training steps increases—RL becomes less data-efficient as the model gets better, which is perverse. The online rollout buffer decouples policy quality from training efficiency: by increasing $k$, the system can maintain a fixed level of group diversity regardless of how good the policy becomes. This means RL training can continue to make progress even as success rates climb, rather than stalling out because every training group looks identical. The paper doesn't explicitly make this argument about scaling properties, but it's implicit in the mathematical structure of the collapse probability.

The practical significance is validated by Figure 8(a), which shows that focusing RL training on "unstable tasks" (those where the policy's success rate varies across rollouts, making diverse groups naturally more common) achieves faster convergence and higher final accuracy than uniform training across all tasks. This is a direct consequence of the collapse problem: tasks where the policy always succeeds or always fails are precisely the ones that produce collapsed groups, so excluding them from RL (or downweighting them) focuses computation on the tasks that actually generate learning signal. The paper's unstable-task selection is a coarse version of this insight; the online rollout buffer provides a more principled approach that could potentially extract signal even from high-confidence tasks by forcing diversity through oversampling.

Innovation 4: Alternating Multi-Platform RL Reveals That Cross-Device Gradient Interference Is a First-Order Bottleneck in Unified Agent Training

Most multi-platform GUI agent work (UI-TARS, OS-Atlas) has treated cross-platform training as a data aggregation problem: collect trajectories from all platforms, mix them together, and train a single model. The hope is that shared visual understanding (what a button looks like, how scrolling works) will transfer across platforms, while platform-specific behaviors will be learned from the platform-specific data in the mixture. This is the standard approach in multi-task learning more broadly.

GUI-Owl-1.5's MRPO framework identifies and addresses a failure mode that prior work either didn't observe or didn't report: cross-device gradient interference where progress on one platform directly impedes progress on another. The paper formalizes this as $\langle g_{\text{mobile}}, g_{\text{desktop}} \rangle < 0$—the inner product of the mobile and desktop policy gradients is negative, meaning the parameter updates that improve mobile performance degrade desktop performance and vice versa. When these conflicting gradients are mixed in a single update, the optimization oscillates without converging.

What makes this insight distinctive is that it recasts multi-platform training from a data diversity problem to an optimization stability problem. Prior work's focus on collecting more diverse, higher-quality data from all platforms implicitly assumes that the training procedure can handle the diversity once the data exists. The MRPO results suggest this assumption is false: even with high-quality data from all platforms, naive mixed training fails because the optimization dynamics cannot reconcile the conflicting gradient signals. The solution—alternating optimization stages where each stage trains exclusively on one device family—is simple in retrospect but represents a conceptual shift: cross-platform generalization is achieved not by jointly optimizing a shared objective but by temporally separating the optimization of device-specific objectives, relying on the shared model backbone to transfer what's been learned across stages.

The empirical evidence in Figure 8(b) is striking: mix-platform training exhibits clear oscillation (the zigzag validation curve), while interleaved training shows smooth improvement. This is not a subtle difference—it's the difference between converging and not converging. The paper doesn't provide a theoretical analysis of why the gradients conflict (e.g., whether it's due to incompatible action spaces, different reward structures, or genuinely contradictory visual features), which limits the generalizability of the insight. Are there platform pairs where gradients don't conflict, and mixed training would work fine? Is the conflict inherent to mobile vs. desktop, or does it depend on the specific tasks used during RL? Without answers to these questions, the finding remains empirical rather than theoretical.

The practical implication is significant for anyone building multi-platform agents: training on all platforms simultaneously may be counterproductive, and an alternating schedule may be necessary regardless of data quality. This is a design principle that challenges the default approach in the field and suggests that future multi-platform agent work should systematically characterize gradient interference before committing to a mixed-training approach.

The paper's alternating schedule is a simple cycle through device families, but the framework opens the door to more sophisticated curriculum strategies: training on easier platform-task combinations first, measuring gradient similarity to decide when to switch platforms, or using multi-objective optimization techniques (e.g., gradient projection methods that prevent conflicting updates to shared parameters). The paper doesn't explore these, but the identification of gradient interference as a first-order bottleneck creates the motivation for such exploration.

A limitation of this innovation is that it's not clearly separable from the other MRPO components in the experimental results. The paper doesn't provide an ablation comparing alternating optimization against mixed optimization with the online rollout buffer held constant, or vice versa. So while Figure 8(b) shows that alternating beats mixing, we can't determine whether this advantage persists at different degrees of outcome collapse mitigation or whether the two techniques interact. The paper's claim that MRPO as a whole improves performance is well-supported; the attribution of credit among its four components is less clear.

Innovation 5: Expanding the GUI Agent's Role from Interface Manipulator to System Orchestrator

Prior GUI agents—including the paper's own predecessor GUI-Owl—operated within a restricted action space: click, type, scroll, swipe, drag. Their job was to manipulate graphical interfaces. This made them powerful within their domain but fundamentally limited: when a task required reading a file's contents, querying a database, or calling a computational API, the agent had to route through the GUI (opening a file browser, navigating to the file, opening it in a text editor, scrolling to the relevant section)—a slow and error-prone process when a direct tool call would be instantaneous and reliable.

GUI-Owl-1.5's expansion of the action space to include tool calls and MCP invocations (Section 2.1) is technically straightforward—add new function-calling tokens to the output vocabulary. But the intellectual move is more significant: it reframes the GUI agent's role from interface manipulator to system orchestrator. The agent is no longer just a user of graphical interfaces; it's a coordinator that decides, at each step, whether the most efficient path to the goal involves clicking through menus, calling an API, executing a shell command, or some interleaved combination. This decision-making capability—knowing when to use which modality—is a qualitatively different skill than either GUI manipulation or tool calling in isolation.

The case study in Figure 11 demonstrates this concretely: the agent reads source code via filesystem_read_text_file, identifies and fixes a bug via filesystem_edit_file, opens a terminal through GUI operations to execute the script, and verifies the output by reading a log file. This isn't just GUI operation + tool calling as separate capabilities; it's the interleaving of both within a single coherent task execution, with the agent making modality-switching decisions in context based on what each step requires.

What makes this more than an incremental feature addition is that it changes the success criteria for GUI agent training. Prior work could evaluate GUI agents purely on whether they successfully navigated interfaces and completed tasks. GUI-Owl-1.5 must also be evaluated on whether it chooses the right modality for each subtask—and the OSWorld-MCP and MobileWorld benchmarks (Table 1) are specifically designed to test this. The 32B-Instruct's scores of 47.6 and 46.8 on these benchmarks substantially exceed prior open-source models (MAI-UI-235B-A22B at 41.7 on MobileWorld) and approach proprietary models (Claude-4-Sonnet at 43.3 on OSWorld-MCP). These results suggest that the interleaved tool-use capability is not just bolted on but genuinely integrated into the agent's decision-making.

The significance beyond raw performance is that this expansion changes the landscape of what GUI agents can be used for. A GUI-only agent is limited to automating tasks that can be completed entirely through graphical interfaces—which excludes an enormous class of real-world workflows that involve reading configuration files, running scripts, checking API responses, or querying databases. A system orchestrator that can seamlessly switch between GUI and tool modalities can automate these mixed-modality workflows end-to-end, dramatically expanding the space of automatable tasks. The paper doesn't quantify this expansion or survey the new task categories that become feasible, but the case study provides a concrete existence proof.

A tension the paper doesn't address: the model must learn to decide when tool use is appropriate versus when it's an unnecessary complexity. In the case study, reading a file via filesystem_read_text_file is clearly superior to navigating through a file browser GUI. But in a task where the file is already open on screen, a GUI-based approach might be faster. The model needs to develop this judgment, and errors in modality selection could lead to inefficient or failed task execution. The paper doesn't analyze modality-selection accuracy as a separate metric, which would be valuable for understanding how well the orchestrator capability works in practice.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on more than 20 benchmarks spanning four capability dimensions. The key end-to-end online benchmarks include OSWorld-Verified (computer use, 369 tasks per the original paper), AndroidWorld (mobile use, dynamic benchmark for autonomous agents), MobileWorld (mobile use with agent-user interaction and MCP-augmented environments), OSWorld-MCP (computer use with tool invocation), WindowsAgentArena (Windows desktop agent evaluation), WebArena (browser use, realistic web environment), VisualWebArena (visual web tasks, multimodal evaluation), WebVoyager (end-to-end web agent benchmark), and Online-Mind2Web (online browser evaluation). Grounding benchmarks include ScreenSpot-Pro (high-resolution professional software grounding), MMBench-GUI-L2 (broad coverage across mobile/desktop/web), OSWorld-G and OSWorld-G-Refine (fine-grained grounding annotations), and ScreenSpot-V2 (mobile/desktop/web grounding). GUI understanding benchmarks include the GUI Knowledge Benchmark (interface perception, interaction prediction, instruction understanding) and MemGUI-Bench (memory of mobile GUI agents in dynamic environments). The test sets are the standard released splits for each benchmark.

  • Base model(s). All GUI-Owl-1.5 variants are initialized from Qwen3-VL (Bai et al., 2025a), a vision-language model family spanning multiple scales. The paper evaluates 6 representative versions: GUI-Owl-1.5-2B-Instruct, GUI-Owl-1.5-4B-Instruct, GUI-Owl-1.5-8B-Instruct, GUI-Owl-1.5-8B-Thinking, GUI-Owl-1.5-32B-Instruct, and GUI-Owl-1.5-32B-Thinking. The Qwen3-VL base provides fundamental visual understanding and language generation before any GUI-specific training. The choice of Qwen3-VL establishes a strong initialization point, and the multi-size evaluation enables assessment of how GUI-specific training benefits scale with parameter count.

  • Metrics. The primary metric across all end-to-end benchmarks is task success rate (%), defined as the fraction of test tasks where the final goal state is achieved regardless of the specific path taken. This is the standard metric for OSWorld, AndroidWorld, WebArena, and related benchmarks—the environment deterministically checks whether the goal condition is satisfied at task termination. For grounding benchmarks, the metric is accuracy (%): the fraction of grounding queries where the model's predicted bounding box or point correctly localizes the target element, with correctness thresholds varying by benchmark (e.g., ScreenSpot-Pro uses element-level matching across text and icon categories). For the GUI Knowledge Benchmark, the metric is accuracy across eight subcategories: interface state, widget function, layout semantics, action effect, action type, action parameter, goal interpretation, and task planning. For MemGUI-Bench, the metric is task success rate. All benchmarks use the standard evaluation protocols defined by their respective authors.

  • Baselines. The paper compares against a comprehensive set of models organized in the benchmark tables. General-purpose models include GPT-4o, Claude-3.7/4/4-Sonnet/4-5-Sonnet, Gemini-2.5-Pro/3-Pro, Qwen3-VL variants (8B/32B/235B-A22B, instruct and thinking), Qwen2.5-VL, Seed1.5-VL, Seed1.8, Kimi K2.5, OpenCUA o3, InternVL3-72B, and Doubao-V-Pro. GUI models (single-platform) include OpenCUA (7B/32B/72B), EvoCUA (8B/32B), MAI-UI (8B/32B/235B-A22B), JEDI (3B/7B), GUI-G2-7B, GTA1 (7B/32B/72B, including New variants), InfiGUI-R1-3B, UI-Venus (7B/72B), UGround-V1 (7B/72B), Step-GUI-8B, Aguvis-7B, and OS-Atlas-Base (4B/7B). GUI models (multi-platform) include UI-TARS-72B-DPO, UI-TARS-1.5 (7B and unspecified size), UI-TARS-2, GELab-Zero (4B/8B), GUI-Owl (7B/32B, the direct predecessor), and CogAgent. Workflow/proprietary agents (evaluated on MemGUI-Bench) include Agent-S2, M3A, T3A, Mobile-Agent-E, Mobile-Agent-V2, SeeAct, and AppAgent, all using Gemini-2.5-Pro as the backbone. For browser benchmarks, additional baselines include Browser-Use, Claude-CUA-3.7, Operator, Gemini-CUA, Navigator, Magnitude + Claude-4-Sonnet, VisualWebArena + GPT-4o, Tree Search + GPT-4o, WALT + GPT-5, SGV + Gemini-2.5-Flash, DeepSky Agent + Claude-4-Sonnet, OAgent + Gemini-3-Pro, WebStar (7B/32B), DynaWeb-8B, ViGoRL-7B, Llama-3-70B-Instruct + Tree Search, and AgentSymbiotic-8B. The diversity of baselines covers the spectrum from general-purpose VLMs to specialized GUI agents, from small open-source models to large proprietary systems.

  • Generation budget / compute accounting. For end-to-end online benchmarks, each benchmark defines its own interaction budget implicitly through environment constraints (e.g., maximum steps, timeout), and the paper follows each benchmark's standard protocol. The paper does not introduce a unified "generation budget" metric across benchmarks; instead, each benchmark comparison is apples-to-apples within the benchmark's own constraints. For grounding benchmarks, the model makes a single prediction per query (no multi-sample aggregation). The Zoom-In refinement variant on ScreenSpot-Pro uses a two-stage process: first localize a coarse region, then crop and zoom in for refined grounding—this uses 2 model calls per query rather than 1, which is noted in the tables with a ° superscript.

  • Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests for any benchmark results. All benchmark numbers are presented as point estimates (single accuracy/success rate per benchmark), which is standard practice in GUI agent evaluation where full test-set evaluations are often deterministic given fixed environment seeds and model sampling parameters. The ablation studies (Tables 10, 11; Figure 8) similarly report single-run results without error bars or multiple seeds. This is a limitation of the current evaluation—without variance estimates, it's impossible to determine whether differences of 1–2 percentage points between models are statistically meaningful or within noise. The field convention of reporting point estimates on fixed test sets partially mitigates this (the same test examples are used for all models), but does not account for training variance or environment stochasticity.

Main Quantitative Results

The paper reports results across four capability dimensions: end-to-end task completion on online environments (Tables 1, 2, 9), grounding (Tables 3–7), comprehensive GUI understanding (Tables 8, 9), and multi-agent/memory capabilities (Table 9). I organize this section by these dimensions, highlighting head-to-head comparisons against the strongest prior models.

End-to-End Task Completion on Computer and Mobile Use

Headline results (Table 1). On OSWorld-Verified, the most widely adopted computer use benchmark, GUI-Owl-1.5-8B-Thinking achieves 52.9, surpassing all general-purpose models at any scale including Qwen3-VL-235B-A22B-Think (38.1) and approaching the single-platform specialist EvoCUA-8B (56.7). The 32B-Instruct variant reaches 56.5, outperforming UI-TARS-2 (53.1) and approaching Claude-4.5-Sonnet (62.9). On AndroidWorld, GUI-Owl-1.5-8B-Thinking attains 71.6, on par with UI-TARS-2 (73.3) and competitive with MAI-UI-235B-A22B (76.7). On OSWorld-MCP, which evaluates mixed GUI-and-tool invocation, GUI-Owl-1.5-32B-Instruct scores 47.6, substantially exceeding Claude-4-Sonnet (43.3) and Qwen3-VL-235B-A22B-Think (39.1). On MobileWorld, the 32B-Instruct scores 46.8, outperforming MAI-UI-235B-A22B (41.7). On WindowsAA, the 32B-Instruct achieves 44.76, far ahead of Qwen3-VL-32B-Think (42.9) and Qwen3-VL-8B-Instruct (28.8).

Parameter efficiency. A striking pattern across Table 1 is the performance of small GUI-Owl-1.5 variants against much larger models. GUI-Owl-1.5-2B-Instruct achieves 43.5 on OSWorld, exceeding UI-TARS-72B-DPO (27.1) by 16.4 points with ~36× fewer parameters. On AndroidWorld, the same 2B model scores 67.9, outperforming Qwen3-VL-32B-Instruct (57.3). The 4B-Instruct (48.2 on OSWorld) also substantially outperforms the 72B UI-TARS-72B-DPO. These comparisons demonstrate that the training pipeline—not parameter count—is the dominant factor in these performance differences, since the base Qwen3-VL models without GUI-specific training achieve much lower scores (e.g., Qwen3-VL-8B-Instruct at 33.9 on OSWorld vs. GUI-Owl-1.5-8B-Instruct at 52.3).

Thinking vs. Instruct. The Thinking variants consistently outperform their Instruct counterparts on benchmarks requiring long-horizon planning: on AndroidWorld, 8B-Thinking (71.6) beats 8B-Instruct (69.0); on OSWorld, 8B-Thinking (52.9) edges out 8B-Instruct (52.3). However, the pattern is not universal—on WindowsAA, 8B-Thinking (35.07) substantially outperforms 8B-Instruct (31.66), while on OSWorld-MCP, 8B-Instruct (41.8) outperforms 8B-Thinking (38.8), suggesting that for tool-heavy tasks, explicit reasoning may not provide the same benefit. More notably, the 32B-Thinking variant sometimes underperforms 32B-Instruct: on OSWorld (56.0 vs. 56.5), on OSWorld-MCP (43.8 vs. 47.6), and on MobileWorld (42.8 vs. 46.8). The paper does not explain these reversals, which complicate the narrative that thinking-mode training universally improves performance.

Comparison to single-platform specialists. Several single-platform models achieve numerically higher scores than GUI-Owl-1.5 on specific benchmarks: MAI-UI-235B-A22B reaches 76.7 on AndroidWorld (vs. GUI-Owl-1.5-32B-Instruct's 69.8), and EvoCUA-32B reaches 56.7 on OSWorld (vs. 56.5). However, these single-platform models do not report results on the other platforms—MAI-UI doesn't report OSWorld scores, and EvoCUA doesn't report AndroidWorld scores. GUI-Owl-1.5's distinctive claim is that it achieves competitive or superior performance while being a single unified model across all platforms, which the single-platform specialists cannot demonstrate. The cross-platform capability is implicit in the fact that the same model weights achieve strong results across all Table 1 columns simultaneously.

End-to-End Task Completion on Browser Use

Headline results (Table 2). On WebArena, GUI-Owl-1.5-8B-Thinking achieves 46.7 and 32B-Thinking achieves 48.4, surpassing all open-source models by wide margins (the next best open-source model, AgentSymbiotic-8B, scores 43.2). On VisualWebArena, 32B-Thinking reaches 46.6, approximately 2× higher than the best open-source alternative (WebStar-32B at 48.6? No—WebStar-32B gets 48.6 on WebVoyager, not VisualWebArena; the VisualWebArena open-source baseline is ViGoRL-7B at 11.2). On WebVoyager, 32B-Thinking achieves 82.1, competitive with proprietary systems like Magnitude + Claude-4-Sonnet (93.9) and Operator (87.0), while dramatically outperforming WebStar-7B (44.8) and WebStar-32B (48.6). On Online-Mind2Web, 8B-Thinking achieves 48.6 and 32B-Thinking achieves an unreported value (shown as "—" in Table 2, suggesting the experiment was not run or not reported), while the best proprietary model (Navigator) reaches 78.7.

Thinking vs. Instruct on browser tasks. The advantage of Thinking variants is more consistent and pronounced on browser tasks than on computer/mobile tasks: WebArena (46.7 vs. 45.7), VisualWebArena (40.8 vs. 39.4), WebVoyager (78.1 vs. 69.9), and Online-Mind2Web (48.6 vs. 41.7)—the Thinking variant outperforms Instruct on every browser benchmark where both are reported. The largest gap is on WebVoyager (+8.2 points), consistent with the paper's claim that Thinking variants provide "pronounced gains on tasks requiring long-horizon planning."

Gap to proprietary systems. On browser benchmarks, the gap between GUI-Owl-1.5 and the strongest proprietary systems remains substantial. On WebArena, DeepSky Agent + Claude-4-Sonnet reaches 66.9, 18.5 points above GUI-Owl-1.5-32B-Thinking (48.4). On VisualWebArena, WALT + GPT-5 reaches 52.9, 6.3 points above GUI-Owl-1.5-32B-Thinking (46.6). On WebVoyager, Magnitude + Claude-4-Sonnet reaches 93.9, 11.8 points above GUI-Owl-1.5-32B-Thinking (82.1). These gaps suggest that while GUI-Owl-1.5 achieves state-of-the-art open-source performance, the proprietary frontier remains substantially ahead for browser-based tasks specifically. This could reflect differences in base model capability (Claude, GPT-5, and Gemini have stronger general web understanding), differences in training data scale or quality for web tasks, or architectural advantages of the proprietary systems' agent frameworks.

Grounding Capability

Headline results (Tables 3–7). On ScreenSpot-Pro (Table 4), GUI-Owl-1.5-32B-Instruct achieves 72.9 without crop tool, surpassing all existing GUI agents (including single-platform specialists, multi-platform models, and grounding-specialized models) as well as the large-scale general model Gemini-3-Pro. With crop-based refinement (Zoom-In), it attains 80.3, the highest reported score on this benchmark at the time of writing, exceeding MAI-UI-32B + Zoom-in (73.5°) by 6.8 points and GTA1-New-32B (63.6) by 16.7 points. On MMBench-GUI-L2 (Table 3), GUI-Owl-1.5-32B-Instruct achieves 86.84 overall, slightly behind MAI-UI-32B (91.3) but substantially ahead of the predecessor GUI-Owl-32B (82.97) and UI-TARS-72B-DPO (74.25). On OSWorld-G (Table 5), GUI-Owl-1.5-32B-Instruct achieves 66.8, competitive with MAI-UI-32B (67.6) and ahead of EvoCUA-32B (63.9). On ScreenSpot-V2 (Table 7), GUI-Owl-1.5-32B-Instruct scores 95.3, matching UI-Venus-72B (95.3) and exceeding the predecessor GUI-Owl-32B (93.2).

Category-level performance on ScreenSpot-Pro. The ScreenSpot-Pro results (Table 4) are broken down by application category (Development, Creative, CAD, Scientific, Office, OS) and element type (Text vs. Icon). Several patterns emerge: (1) GUI-Owl-1.5 performs substantially better on Text elements than Icon elements across all variants—for example, 32B-Instruct achieves 86.9 on OS Text vs. 44.9 on OS Icon. This gap is consistent with prior work and reflects the difficulty of grounding small iconic elements at high resolution. (2) The Zoom-In refinement provides larger gains on Icon elements than on Text elements—for 32B-Instruct, Zoom-In improves Office Text from 91.5 to 93.2 (+1.7) but Office Icon from 56.6 to 75.5 (+18.9), confirming that the crop-and-zoom strategy primarily helps with small-target localization. (3) Performance varies substantially across application categories, with Office apps being the strongest (91.5/56.6 for 32B-Instruct Text/Icon) and CAD being the weakest (80.2/48.4). This variation likely reflects the distribution of training data across application types.

Comparison to grounding-specialized models. Several models are designed specifically for grounding (UGround, JEDI, GUI-G2, GTA1) rather than end-to-end task execution. GUI-Owl-1.5-32B-Instruct generally matches or exceeds these specialists: on ScreenSpot-V2 (Table 7), it achieves 95.3 vs. UI-Venus-72B (95.3) and GTA1-32B (93.2); on ScreenSpot-Pro (Table 4), it achieves 72.9 vs. MAI-UI-32B (67.9) and GTA1-72B (58.4). The fact that a single unified model matches specialized grounding models suggests that the grounding training data and pipeline are effective, and that joint training on grounding and trajectory tasks does not degrade grounding performance.

Instruct vs. Thinking on grounding. The Thinking variants consistently underperform their Instruct counterparts on grounding benchmarks. On ScreenSpot-Pro (Table 4), 32B-Thinking scores 57.0 vs. 32B-Instruct's 72.9—a 15.9-point gap. On OSWorld-G (Table 5), 32B-Thinking scores 57.6 vs. 66.8—a 9.2-point gap. On ScreenSpot-V2 (Table 7), 32B-Thinking scores 93.2 vs. 95.3—a 2.1-point gap. This pattern is consistent with the design of the thinking variants: they allocate tokens to explicit reasoning before producing grounded coordinates, which may help for complex multi-step tasks but apparently adds noise or distracts from the single-step localization objective of grounding benchmarks. The paper does not discuss this tradeoff, but it's important for deployment decisions—if grounding accuracy is the primary requirement, the Instruct variants are clearly preferable.

Comprehensive GUI Understanding

GUI Knowledge Benchmark (Table 8). GUI-Owl-1.5-32B-Instruct achieves an overall accuracy of 75.45, establishing the highest performance among all evaluated models including proprietary ones such as o3 (73.30), Gemini-2.5-Pro (71.69), and GPT-5-Chat (70.97). The eight subcategory breakdown reveals specific strengths: the model achieves particularly high scores on widget function understanding (92.65, far exceeding o3's 84.12 and Gemini-2.5-Pro's 84.36) and goal interpretation (88.67, competitive with o3's 95.47 and Claude-Sonnet-4's 94.82). The weakest subcategory for the Instruct variant is state information understanding (77.06, below o3's 83.03) and effect prediction (70.69, roughly matching Claude-Sonnet-4-5's 71.55). Interestingly, the Thinking variant (73.36 overall) underperforms the Instruct variant (75.45) by 2.09 points, continuing the pattern from grounding benchmarks.

MemGUI-Bench (Table 9). On the easy split of MemGUI-Bench, GUI-Owl-1.5-32B achieves 27.1, substantially outperforming all prior native agent models: Qwen3-VL-8B-Instruct (18.8), GUI-Owl-7B (14.6), UI-Venus-7B (14.6), UI-TARS-1.5-7B (8.3), and CogAgent (0.0). Even the 8B variant (22.9) surpasses all existing native baselines. However, several workflow-based systems using Gemini-2.5-Pro as a backbone achieve higher scores: Agent-S2 (41.7), M3A (39.6), and T3A (31.2). The gap between native agent models and workflow-based systems on this memory-intensive benchmark suggests that while GUI-Owl-1.5's explicit memory training (through CoT synthesis annotations) substantially improves memory capability over prior native agents, there is still a gap to the external-memory architectures used in workflow systems. The paper does not evaluate on MemGUI-Bench's hard split.

What these benchmarks measure vs. what the agent needs. The GUI Knowledge Benchmark evaluates whether the model knows about GUI concepts (what does this widget do? what will happen if I click this?), while MemGUI-Bench evaluates whether the model can retain and use information across steps in actual task execution. The strong performance on the knowledge benchmark (75.45, SOTA) combined with the still-modest performance on the memory benchmark (27.1, behind workflow systems) highlights an important capability gap: GUI-Owl-1.5 has excellent declarative GUI knowledge but still struggles with the procedural challenge of maintaining and referencing memories during long-horizon task execution. This gap is consistent with the Thinking variants' advantage on long-horizon browser benchmarks (where memory matters more) and suggests that further improvements in memory management—either through better CoT synthesis or through architectural changes—could yield substantial gains on memory-intensive tasks.

Ablation Studies and Robustness Checks

The paper includes four ablation experiments: two validating data pipeline components (virtual environments and unified CoT synthesis), and two validating RL training strategies (unstable-task selection and interleaved multi-platform training).

Virtual environment-based trajectory production (Table 11): Removing trajectory data produced by virtual environments causes dramatic drops on both PC-Eval (75.4% → 42.0%, a 33.4-point drop) and Mobile-Eval (86.7% → 50.0%, a 36.7-point drop). These drops are among the largest single-component ablations reported in the paper, confirming that virtual environment data is essential—not merely helpful—for the specific task categories these benchmarks evaluate (atomic desktop operations like drag/scroll, office document editing, and popular Chinese mobile app scenarios). The experiment uses GUI-Owl-1.5-8B-Thinking for both conditions.

Unified CoT synthesis (Table 10): Removing the unified CoT synthesis pipeline causes drops on OSWorld (52.9% → 47.4%, a 5.5-point drop) and AndroidWorld (71.6% → 65.0%, a 6.6-point drop). These drops are substantial but an order of magnitude smaller than the virtual environment ablation, suggesting that CoT synthesis provides important but incremental improvements to reasoning and decision-making, while virtual environment data is existential for certain skill categories. The experiment uses GUI-Owl-1.5-8B-Thinking.

Unstable-task-focused RL training (Figure 8a): Training RL exclusively on "unstable tasks" (derived from multi-round rollouts where the policy's success rate varies) achieves faster convergence and higher final PC validation accuracy than training on the full dataset. The paper interprets this as evidence that prioritizing challenging, high-variance tasks provides more informative learning signal—tasks where the policy already performs consistently (always succeeding or always failing) contribute little to policy improvement because they produce collapsed GRPO groups. The paper does not provide specific numerical values from Figure 8a in the text, making it difficult to quantify the exact advantage; the claim is based on visual inspection of the convergence curves. The criteria for identifying unstable tasks and the mechanism for deriving them are not specified beyond "derived from multi-round rollouts."

Interleaved vs. mix-platform RL training (Figure 8b): Mix-platform training (simultaneous multi-platform data optimization) exhibits performance oscillation in the validation curve, while interleaved training (switching from Mobile to PC at step 10) achieves stable improvement. The oscillation in mixed training is clearly visible in Figure 8b, and the interleaved curve shows smoother monotonic improvement, validating the paper's claim that cross-platform gradient interference is a real optimization problem. However, the paper does not report final numerical scores for the two conditions, relying solely on the visualization. Additionally, the specific interleaving schedule (switch from Mobile to PC at step 10) is mentioned in the caption but the rationale for this specific schedule is not provided—it's unclear whether different schedules would produce different results.

Missing ablations. Several important ablations are absent: (1) No ablation isolating the effect of world modeling data on downstream task performance—it's bundled into the pre-training corpus. (2) No ablation comparing the multi-agent data collection framework against single-agent collection—the contribution of the Manager-Worker-Reflector-Notetaker structure during data generation is unknown. (3) No ablation on the grounding data pipeline components individually—the contribution of challenging app synthesis vs. multi-window synthesis vs. trajectory-based extraction vs. tutorial mining vs. infeasible query generation is not disentangled. (4) No ablation comparing the online rollout buffer against alternative solutions for GRPO outcome collapse (e.g., simply discarding collapsed groups, using a replay buffer, using larger group sizes). (5) No ablation on the token-ID transport mechanism—the magnitude of the log-probability discrepancy without this fix is not quantified. (6) No sensitivity analysis on the oversampling factor k in the online rollout buffer—it's unclear whether performance is sensitive to this hyperparameter.

Critical Assessment

Claim 1: GUI-Owl-1.5 achieves state-of-the-art results on 20+ GUI benchmarks among open-source models

What was tested: The paper evaluates on an extensive set of benchmarks (Tables 1–9) and compares against a comprehensive list of open-source baselines. The claim is well-supported for the benchmarks where GUI-Owl-1.5 appears and baselines are reported. On OSWorld-Verified (Table 1), GUI-Owl-1.5-32B-Instruct (56.5) exceeds the best open-source multi-platform model (UI-TARS-2 at 53.1) and all general-purpose open-source models. On ScreenSpot-Pro (Table 4), GUI-Owl-1.5-32B-Instruct (72.9) exceeds all open-source models. On the GUI Knowledge Benchmark (Table 8), it even exceeds proprietary models (75.45 vs. o3's 73.30). On AndroidWorld (Table 1), it's competitive but not definitively SOTA—MAI-UI-235B-A22B reports 76.7 (single-platform specialist) and UI-TARS-2 reports 73.3 (multi-platform), while GUI-Owl-1.5-8B-Thinking gets 71.6.

Weaknesses in this claim: (1) "20+ benchmarks" includes many benchmarks that are correlated—ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G, OSWorld-G-Refine, and MMBench-GUI-L2 all measure grounding capability, and strong performance on one tends to imply strong performance on others. The effective number of independent capability dimensions tested is smaller than 20. (2) Several benchmarks are evaluated with a single GUI-Owl-1.5 variant only—for example, the 32B variants are not evaluated on most browser benchmarks (Table 2 shows "—" for 32B-Instruct on WebArena, VisualWebArena, and Online-Mind2Web), making the "SOTA across all benchmarks" claim incomplete for the largest models. (3) The comparison tables are populated from published results, which means some baselines were evaluated under different conditions (different number of allowed steps, different screen resolutions, different timeout thresholds). The paper does not discuss whether all comparisons are strictly apples-to-apples in terms of evaluation protocol.

Claim 2: The Hybird Data Flywheel improves data collection efficiency and quality

What was tested: The Table 11 ablation shows that removing virtual environment data causes dramatic performance drops on PC-Eval (75.4% → 42.0%) and Mobile-Eval (86.7% → 50.0%). The Table 10 ablation shows that removing CoT synthesis causes drops on OSWorld (52.9 → 47.4) and AndroidWorld (71.6 → 65.0).

Weaknesses in this claim: (1) The claim is about "efficiency and quality," but the ablations measure only quality (final performance), not efficiency (data collection cost, time, or human effort). The paper does not report how much faster, cheaper, or more scalable the virtual environment pipeline is compared to the alternative (pure real-device exploration with human annotation). Without efficiency metrics, the claim about improved efficiency is unsupported. (2) The virtual environment ablation conflates two effects: removing the virtual environment data itself and removing the precise subtask-level feedback that virtual environments provide. The paper cannot distinguish whether the performance drop is due to losing the volume of data or the quality (clean feedback) of the data. (3) The PC-Eval and Mobile-Eval benchmarks are in-house—their construction, difficulty, and representativeness are not documented. The dramatic drops on these benchmarks may overstate the importance of virtual environments if these benchmarks are specifically designed to test the skills that virtual environments target. (4) The paper does not report what fraction of the total training data comes from virtual environments vs. real devices vs. human annotation, making it impossible to assess the claim about "efficiency of data collection" in absolute terms.

Claim 3: Unified Enhancement of Agent Capabilities improves reasoning, memory, and tool use

What was tested: The CoT synthesis ablation (Table 10) shows performance drops on two benchmarks. The multi-agent framework is described but not ablated. The tool-use capability is evaluated on OSWorld-MCP and MobileWorld (Table 1) where GUI-Owl-1.5 performs strongly. Memory capability is evaluated on MemGUI-Bench (Table 9) where GUI-Owl-1.5 substantially outperforms prior native agents. GUI knowledge is evaluated on the GUI Knowledge Benchmark (Table 8) where it achieves SOTA.

Weaknesses in this claim: (1) The multi-agent contribution is completely unevaluated—there is no ablation comparing performance with vs. without multi-agent data collection or multi-agent deployment. The paper describes the Manager-Worker-Reflector-Notetaker framework in detail (Section 2.3.3) but provides no experimental evidence that it helps. (2) The world modeling contribution is unevaluated—it's bundled into pre-training with no ablation. (3) The tool-use evaluation on OSWorld-MCP and MobileWorld shows strong performance, but there's no ablation showing that the explicit tool invocation training (as opposed to the expanded action space alone) is responsible. (4) The memory evaluation on MemGUI-Bench shows substantial improvement over prior native agents (14.6 → 27.1) but still lags behind workflow-based systems (41.7 for Agent-S2 with Gemini-2.5-Pro), suggesting the memory capability is improved but still not at the level that external memory architectures achieve. (5) The CoT synthesis ablation (Table 10) shows drops of 5.5–6.6 percentage points—meaningful but not transformative—and it's unclear which aspect of CoT synthesis (observation, memory, reflection, progress tracking) contributes most.

Claim 4: MRPO addresses challenges of multi-platform conflicts and low training efficiency

What was tested: Figure 8a shows that unstable-task-focused training converges faster than full-dataset training. Figure 8b shows that interleaved training avoids the oscillation observed in mix-platform training.

Weaknesses in this claim: (1) The online rollout buffer—the most technically novel component of MRPO—is not directly ablated. There is no comparison showing performance with vs. without the online rollout buffer, or with the buffer vs. alternative approaches (replay buffer, discarding collapsed groups). The paper describes the mechanism in detail (Section 2.4.3) but provides no experimental evidence that it improves training outcomes. (2) The token-ID transport mechanism is similarly unevaluated—the paper asserts that tokenization mismatch "breaks" KL regularization and policy gradient estimators but doesn't quantify the magnitude of the problem or demonstrate that the fix matters. (3) The Figure 8 ablations are presented as validation curves without final numerical scores, making it difficult to quantify the advantage of the proposed techniques. The curves show qualitative differences (faster convergence, less oscillation) but without numbers, the practical significance is unclear. (4) "Low training efficiency" is claimed as a problem that MRPO addresses, but the paper doesn't report training efficiency metrics (wall-clock time, number of environment interactions, GPU hours). The claim about efficiency remains unquantified. (5) The four MRPO components (unified device-conditioned policy, online rollout buffer, token-ID transport, alternating optimization) are never evaluated independently, making it impossible to determine which components matter and by how much.

Overall Assessment

The experimental section provides strong evidence that the complete GUI-Owl-1.5 training pipeline produces models that achieve state-of-the-art open-source performance across a wide range of GUI benchmarks. The breadth of evaluation—covering computer use, mobile use, browser use, grounding, knowledge, memory, and tool invocation—is a genuine strength that establishes the multi-platform capability claim.

However, the paper's ablations are insufficient to validate the specific technical contributions it names. The Hybird Data Flywheel, Unified Enhancement of Agent Capabilities, and MRPO are each complex multi-component systems, but the ablations test only one or two components of each, and never in isolation from other changes. Several of the paper's most distinctive technical ideas—the world modeling data, the multi-agent framework's contribution to training, the online rollout buffer, the token-ID transport—are not ablated at all. The virtual environment ablation (Table 11) is the strongest single-component ablation, but even it conflates data volume and data quality effects.

The paper would be strengthened by: (1) A systematic component ablation that adds/removes each major component (virtual environments, CoT synthesis, multi-agent data, world modeling data, online rollout buffer, token-ID transport, alternating optimization) and measures the marginal contribution of each, ideally on multiple benchmarks. (2) Reporting training efficiency metrics—data collection cost, training time, environment interactions—to validate the efficiency claims. (3) Including variance estimates or multi-seed results to establish whether small performance differences are statistically significant, particularly for the thinking vs. instruct comparisons where the direction of advantage varies across benchmarks. (4) Evaluating all model variants on all benchmarks to avoid the "—" entries in Tables 1 and 2, which create the impression of selective reporting. (5) A direct comparison between the hybrid data flywheel approach and a pure real-device exploration baseline matched for total annotation budget (including human annotation time and virtual environment construction effort), to validate the efficiency claim.

The central empirical contribution—that a systematically trained multi-platform GUI agent can match or exceed single-platform specialists and far larger general-purpose models—is well-supported by the extensive benchmark comparisons. The paper's more specific claims about why the approach works (which components contribute, how much each matters, whether the mechanisms function as described) are supported unevenly, with several key claims backed only by qualitative argument rather than quantitative evidence.

6. Limitations and Trade-offs

Virtual Environment Construction Cost Is Unaccounted for and Likely Dominates the Data Pipeline Investment

The assumption or constraint. The Hybird Data Flywheel depends on web-rendering-based virtual environments that simulate real applications—word processors, spreadsheets, food delivery apps, ride-hailing apps, ticket booking systems—and provide deterministic, programmatic feedback on whether each subtask was completed correctly. The paper treats these environments as available infrastructure but does not disclose what they cost to build. Section 2.2.2 describes their function in detail but says nothing about the engineering effort, the number of environments constructed, or the process for creating new ones beyond a passing reference to "Vibe Coding" in Section 1:

"we synthesize virtual environments via Vibe Coding to create high-frequency, complex atomic operations and apps featuring challenging cases such as pop-ups and CAPTCHA-style verifications."

This is the only mention of the construction methodology, and it is neither defined nor quantified.

The consequence. If building a single high-fidelity virtual spreadsheet editor (one that accurately simulates all formatting operations, formula evaluation, dialog box interactions, and edge cases) requires weeks or months of engineering effort, the virtual environment pipeline does not scale to the long tail of real-world applications. The approach would be practical only for a curated set of high-frequency, high-value application domains—exactly the domains where the paper's benchmarks (PC-Eval, Mobile-Eval) already show strong performance. A practitioner deciding whether to adopt this approach cannot assess the total investment required, because the paper reports only the benefit (dramatic performance improvements in Table 11: +33.4 points on PC-Eval, +36.7 points on Mobile-Eval) without the cost.

Furthermore, the paper does not establish that the virtual environment approach works at all for novel applications not represented in the training environments. If performance on a new app depends on having a virtual environment for that app during training, then GUI-Owl-1.5's strong benchmark results may reflect overfitting to the specific application domains where virtual environments were built, rather than generalizable GUI automation capability. The paper cannot address this concern because it does not disclose which applications were virtualized.

What evidence exists in the paper. The ablation in Table 11 quantifies the benefit of virtual environment data by removing it and measuring the performance drop on PC-Eval (75.4% → 42.0%) and Mobile-Eval (86.7% → 50.0%). These drops are the largest single-component effects reported in the paper, confirming that virtual environment data is essential for the skills these benchmarks test. However, the benchmarks are in-house and their relationship to real-world application diversity is undocumented. The paper provides no evidence about construction cost, number of virtual environments, coverage of application categories, fidelity to real application behavior, or generalization to applications without virtual environments.

Mitigation status. The paper does not address this limitation. It does not report construction costs, propose methods for reducing them, or discuss the scalability of virtual environment creation to the long tail of applications. The "Vibe Coding" reference hints at AI-assisted construction but is not elaborated, leaving the most significant practical barrier to adoption completely unquantified.

Difficulty Estimation for Hard Problems Is Essentially Unsolved—the Agent Cannot Self-Improve on Tasks It Fundamentally Cannot Do

The assumption or constraint. The entire training pipeline—from trajectory collection through RL optimization—implicitly assumes that the base model (Qwen3-VL) and the exploration agents used during data collection have some non-trivial probability of producing correct trajectories for the target tasks. For tasks where no automated agent can produce a correct trajectory, the system falls back to human annotation (Section 2.2.2: "For difficult tasks that remain unsolved after repeated automated attempts, we collect expert demonstrations"). This fallback is mentioned but not quantified.

The consequence. This creates a fundamental capability bound that no amount of subsequent training or RL can overcome: GUI-Owl-1.5 can only learn to perform tasks that either (a) the exploration agent could complete at least occasionally, providing successful trajectories for training, or (b) humans demonstrated manually. For genuinely novel or out-of-distribution tasks—applications with unfamiliar interfaces, workflows that differ substantially from the training distribution, or tasks requiring reasoning capabilities beyond the base model's competence—the pipeline produces no training signal. The model may still attempt these tasks at deployment but will likely fail, and the paper provides no mechanism for the model to recognize its own incompetence and gracefully refuse or escalate.

This matters for deployment because real-world GUI automation will encounter novel applications and edge cases constantly. A practitioner needs to know: what is the failure mode when the agent encounters something it wasn't trained for? Does it refuse? Does it attempt and fail silently? Does it loop? The paper's benchmarks are all drawn from known distributions (standard datasets or in-house benchmarks built around the training applications), so they cannot answer this question. The consistent near-zero performance on the hardest MemGUI-Bench tasks (Table 9 shows prior native agents at 0.0–14.6%, and while GUI-Owl-1.5 improves this, the paper only reports the easy split) and the absence of any out-of-distribution evaluation means the capability boundary is uncharacterized.

What evidence exists in the paper. The paper does not directly evaluate out-of-distribution generalization or characterize the failure modes on tasks outside the training distribution. The benchmark results (Tables 1–9) are all on standard datasets whose task distributions overlap substantially with likely training data—OSWorld tasks involve standard desktop applications, AndroidWorld tasks involve standard mobile workflows, WebArena tasks involve standard web interactions. The human annotation fallback is mentioned in Section 2.2.2 but its scope (what fraction of tasks required human annotation, for which application domains, at what cost) is not reported. The MemGUI-Bench easy-split results (Table 9) show substantial improvement over prior native agents, but the hard split is not evaluated, leaving open the question of whether performance degrades gracefully or catastrophically on harder memory tasks.

Mitigation status. The paper does not address this limitation. There is no discussion of out-of-distribution detection, confidence estimation, refusal mechanisms, or uncertainty quantification. The multi-agent framework (Section 2.3.3) includes a Reflector that judges action outcomes, but this is used for in-distribution verification (did the action produce the expected screen change?), not for recognizing when a task is fundamentally beyond the agent's capability. The paper does not propose or evaluate any mechanism for the model to signal when it cannot complete a task.

The Multi-Agent Framework Contribution Is Described but Completely Unevaluated

The assumption or constraint. Section 2.3.3 describes the Mobile-Agent-v3.5 framework in substantial detail—a Manager-Worker-Reflector-Notetaker architecture with formalized role definitions, state variables, and update functions. The paper positions this as a key component of the "Unified Enhancement of Agent Capabilities":

"we incorporate multi-agent collaboration data collected via the Mobile-Agent-v3.5 framework, allowing the model to function not only as a standalone end-to-end agent but also as specialized roles (e.g., planner, executor, verifier) within structured multi-agent systems."

This framing implies that the multi-agent framework contributes to the model's capabilities, either as a data collection strategy (producing better training trajectories) or as a deployment architecture (enabling better task execution through role specialization).

The consequence. There is no experimental evidence that the multi-agent framework provides any benefit whatsoever. The paper never compares: (a) trajectories collected with the multi-agent framework vs. without it, (b) model performance when deployed in multi-agent mode vs. end-to-end mode, or (c) the contribution of multi-agent training data vs. single-agent training data. A practitioner cannot determine whether implementing the Manager-Worker-Reflector-Notetaker framework is worth the engineering effort, or whether the same performance could be achieved with end-to-end training alone.

The detailed formalization in Section 2.3.3 (equations for Manager subgoal planning, Worker action generation, Reflector verification, Notetaker memory updates) occupies substantial space in the paper but is severed from any empirical validation. This is not just an omission—it means the paper's "Unified Enhancement of Agent Capabilities" claim rests on the CoT synthesis ablation (Table 10) and the benchmark results, neither of which isolates the multi-agent contribution.

What evidence exists in the paper. None. There is no ablation comparing multi-agent vs. single-agent data collection, no comparison of multi-agent vs. end-to-end deployment, and no discussion of how much of the model's performance can be attributed to the multi-agent framework vs. the other capability enhancement strategies. The MemGUI-Bench results (Table 9) compare GUI-Owl-1.5 against both native agent models and workflow-based systems (which use external multi-agent orchestration), but this comparison involves different model backbones (GUI-Owl-1.5 vs. Gemini-2.5-Pro with agent frameworks), making it impossible to attribute performance differences to the multi-agent architecture vs. the base model.

Mitigation status. The paper does not acknowledge this as a limitation or call for future evaluation of the multi-agent framework. The framework is presented as a contribution but its empirical status is completely unresolved.

Model Scale and Performance Are Non-Monotonic, and the Paper Provides No Explanation

The assumption or constraint. The paper presents a family of models scaling from 2B to 32B parameters (with a 235B-A22B variant mentioned but not evaluated in the main benchmark tables), implying that larger models within the family should generally perform better on GUI tasks. This is the standard scaling assumption: more parameters → more capacity → better performance, all else being equal.

The consequence. The benchmark results contradict this assumption in multiple places, with no explanation offered. The most notable reversals (Table 1):

  • 8B-Thinking (71.6) outperforms 32B-Instruct (69.8) and ties 32B-Thinking (69.8) on AndroidWorld
  • 32B-Thinking (56.0) underperforms 32B-Instruct (56.5) on OSWorld
  • 32B-Thinking (43.8) substantially underperforms 32B-Instruct (47.6) on OSWorld-MCP
  • 32B-Thinking (42.8) substantially underperforms 32B-Instruct (46.8) on MobileWorld
  • 32B-Instruct (69.8) underperforms 8B-Thinking (71.6) on AndroidWorld

These reversals are not marginal—the OSWorld-MCP gap is 3.8 points and the MobileWorld gap is 4.0 points, which are substantial relative to the differences between competing models in the same tables. The pattern suggests that the Thinking training procedure interacts negatively with increased model scale, or that the 32B models were not trained to convergence, or that the RL optimization dynamics differ across model sizes in ways that advantage smaller models on certain task types.

For a practitioner selecting which model variant to deploy, this non-monotonicity makes the choice non-obvious. If 8B-Thinking outperforms 32B-Instruct on mobile tasks but 32B-Instruct substantially outperforms 8B-Thinking on tool-use tasks, which model should be deployed for a mixed workload? The paper provides no guidance.

What evidence exists in the paper. The evidence is visible directly in Table 1, where the four thinking/instruct variants at two scales are reported on multiple benchmarks. The non-monotonicities are not subtle—they are plainly visible in the numbers. The paper does not comment on them, does not provide hypotheses, and does not discuss implications for model selection or training.

Mitigation status. Not addressed. The paper presents the benchmark results without discussing the non-monotonic scaling behavior, leaving practitioners to draw their own conclusions from an inconsistent pattern.

The Paper's RL Contributions Are Described but Their Individual Effects Are Not Isolated

The assumption or constraint. MRPO (Multi-platform Reinforcement Policy Optimization) is presented as a unified RL framework that addresses four specific challenges: (1) multi-device policy unification, (2) GRPO outcome collapse via online rollout buffer, (3) training-inference log-prob alignment via token-ID transport, and (4) cross-device gradient interference via alternating optimization. The paper describes each component in mathematical detail (Section 2.4.3) and presents the framework as one of three named contributions.

The consequence. Only two of the four MRPO components receive any experimental evaluation: unstable-task-focused training (related to the outcome collapse problem) in Figure 8a, and alternating vs. mixed multi-platform training (addressing gradient interference) in Figure 8b. The online rollout buffer—arguably the most technically novel component with its formal analysis of collapse probability and Swap1 mechanism—is never ablated. The token-ID transport mechanism is never ablated nor is the magnitude of the log-probability discrepancy without it quantified. The device-conditioned policy formulation is described but its contribution relative to an unconditioned policy is never measured.

This means a practitioner cannot determine which MRPO components are necessary and which are incidental. If the performance gains come primarily from alternating optimization and unstable-task selection, the online rollout buffer and token-ID transport may be unnecessary complexity. Conversely, if the online rollout buffer is essential for stable training but its effect is masked by other improvements, practitioners who omit it may experience training instability that the paper's results don't anticipate.

What evidence exists in the paper. Figure 8 provides evidence for unstable-task selection (faster convergence) and alternating optimization (avoiding oscillation). These are the only MRPO ablations. The online rollout buffer, token-ID transport, and device-conditioned policy receive no experimental validation whatsoever. The paper's mathematical analysis of the collapse probability (Section 2.4.3) establishes the existence of the problem but does not demonstrate that the proposed solution improves training outcomes.

Mitigation status. The paper does not acknowledge this as a limitation. The MRPO components are presented as a package, and their individual contributions are left unevaluated. The ablation in Figure 8 addresses related but distinct training strategy choices (task selection and platform scheduling) rather than the specific mechanisms the paper introduces.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture or a novel learning algorithm in the traditional sense—it introduces a training methodology that systematically addresses the three bottlenecks that have kept GUI agents from crossing the gap between impressive demos and reliable deployment: data quality (the virtual environment pipeline), reasoning explicitness (the unified CoT synthesis), and multi-platform optimization stability (the MRPO framework). The methodological shift is from treating GUI agent training as an imitation learning problem on collected trajectories to treating it as a curriculum design and optimization stability problem where what matters most is not the volume of training data but the fidelity of feedback during data generation and the absence of gradient conflict during policy optimization.

The most important reframing concerns data quality. Prior work implicitly treated GUI trajectory data as fungible—more trajectories, more diverse trajectories, more platforms, and the model will improve. GUI-Owl-1.5's ablation in Table 11 shows this assumption is false for specific skill categories: removing virtual environment data causes PC-Eval to drop from 75.4% to 42.0% and Mobile-Eval from 86.7% to 50.0%. These are not marginal degradations—they suggest that for precise manipulation tasks (drag-and-drop, spreadsheet editing, CAPTCHA handling), no amount of noisy real-device trajectory data substitutes for the deterministic feedback that virtual environments provide. This finding reframes the data collection problem: the bottleneck is not data volume but feedback fidelity for specific skill categories. Future work should identify which capabilities are feedback-limited (requiring deterministic verification) versus volume-limited (improving with more diverse examples), rather than treating all training data as equivalent.

The second reframing concerns multi-platform training. The MRPO results in Figure 8b—where mixed-platform RL oscillates while alternating optimization converges stably—establish that cross-device gradient interference is a first-order optimization bottleneck in unified agent training. This challenges the default approach in multi-task learning, where mixing data from all tasks is assumed to be harmless or beneficial. For GUI agents specifically, the action spaces, reward structures, and visual conventions differ so substantially across platforms that joint optimization becomes a tug-of-war. The alternating schedule is a simple solution, but the underlying insight—that gradient conflict measurement should precede mixed training—generalizes beyond this paper. Researchers building multi-platform or multi-domain agents should now measure ⟨g_platform_A, g_platform_B⟩ before deciding on a training strategy, rather than assuming that data diversity alone will produce a unified policy.

The paper also reconciles a latent contradiction in prior GUI agent work. Several recent papers (UI-TARS, MAI-UI, GUI-Owl) achieved strong results on specific platforms through specialized training, but no single open-source model was competitive across desktop, mobile, and browser simultaneously. The explicit assumption—visible in the single-platform focus of MAI-UI and OpenCUA—was that platform specialization was necessary for strong performance. GUI-Owl-1.5 demonstrates that a unified model can match or exceed single-platform specialists (e.g., 56.5 on OSWorld vs. EvoCUA-8B at 56.7; 71.6 on AndroidWorld vs. UI-TARS-2 at 73.3) while simultaneously performing well across browser benchmarks (48.4 on WebArena, 46.6 on VisualWebArena). This suggests that platform specialization was an artifact of insufficient training methodology, not an inherent tradeoff. The practical implication is that organizations building GUI agents can consolidate around a single model rather than maintaining separate per-platform models, reducing engineering complexity.

However, the paper also sharpens the boundary between what test-time training improvements can and cannot achieve. The MemGUI-Bench results (Table 9) show that native agent models still lag substantially behind workflow-based systems for memory-intensive tasks (GUI-Owl-1.5-32B at 27.1 vs. Agent-S2 with Gemini-2.5-Pro at 41.7). The GUI Knowledge Benchmark (Table 8) shows strong declarative knowledge (75.45, SOTA) but the memory benchmark reveals that using that knowledge effectively over long horizons remains a challenge that explicit CoT training only partially addresses. This establishes a clear capability frontier: the current approach excels at perception, grounding, and medium-horizon planning, but long-horizon memory management may require architectural changes (external memory stores, structured state tracking) rather than just better training data.

The research directions that become more attractive after this work:

  • Feedback-quality-aware data collection: the virtual environment ablation establishes that feedback fidelity matters enormously; future work should measure and improve feedback quality for every data source, not just scale volume.
  • Gradient-aware multi-task optimization: the alternating optimization results motivate systematic measurement of gradient interference in multi-platform, multi-domain agent training.
  • Architectural memory support: the MemGUI-Bench gap between native agents and workflow systems suggests that explicit memory architectures (rather than purely CoT-based memory) are the next frontier.
  • Lightweight difficulty and capability estimation: the paper shows that different training strategies work for different task types (virtual environments for precise manipulation, CoT synthesis for planning-heavy tasks), but provides no mechanism for identifying which strategy a novel task needs—this is a natural next step.

The research directions that become less attractive:

  • Pure data scaling without feedback improvement: the Table 11 ablation shows that removing specific high-quality data sources causes catastrophic drops, suggesting that throwing more noisy data at the problem has diminishing returns.
  • Single-platform GUI agent development: the paper demonstrates that unified multi-platform training is not only feasible but achieves competitive performance, reducing the rationale for building separate per-platform models.
  • Agent framework approaches that rely on proprietary model APIs: the strong performance of small native models (2B-Instruct at 43.5 on OSWorld exceeding 72B UI-TARS-DPO at 27.1) suggests the native model paradigm is closing the gap with framework-based approaches, making the cost and latency advantages of native models increasingly compelling.

Follow-Up Research This Work Enables

Characterizing which capabilities are feedback-limited versus volume-limited in GUI agent training. The virtual environment ablation (Table 11) establishes that some skills (precise drag-and-drop, spreadsheet editing, CAPTCHA-heavy scenarios) require deterministic feedback to be learned, but the paper does not provide a systematic method for identifying which capabilities fall into this category. A follow-up study would train GUI agents with progressively degraded feedback quality (from deterministic virtual-environment feedback through VLM-based verification to no explicit feedback beyond task completion) on a diverse set of atomic skills, measuring the performance cliff for each skill as feedback degrades. The result would be a capability taxonomy—identifying which skills need clean feedback (and thus require investment in virtual environments or human annotation) and which skills can be learned from noisy real-world exploration. This would directly inform resource allocation for building the next generation of GUI agents.

Measuring and mitigating cross-platform gradient interference in detail. The MRPO results (Figure 8b) establish that mixing mobile and desktop gradients causes optimization oscillation, but the paper does not characterize the interference: which specific parameters conflict? Is the conflict concentrated in early vision layers (where mobile and desktop interfaces look different) or in later reasoning layers (where action selection strategies diverge)? A follow-up would compute per-layer gradient cosine similarity matrices across device families during RL training, identifying which layers benefit from shared training and which suffer from conflict. The finding would inform architectural decisions—potentially motivating shared vision encoders with platform-specific policy heads, or gradient projection methods that allow beneficial parameter sharing while preventing destructive interference, without the wall-clock cost of alternating optimization.

Combining native agent training with external memory architectures. The MemGUI-Bench results (Table 9) reveal a persistent gap between native agent models (GUI-Owl-1.5-32B at 27.1) and workflow-based systems using external memory (Agent-S2 at 41.7). The paper's CoT synthesis teaches the model to record memories in its thought stream, but this approach is bounded by the context window and the model's ability to attend to distant tokens. A follow-up would augment GUI-Owl-1.5 with a structured external memory store (e.g., a key-value store for task-relevant information, or a scratchpad that persists across the context window boundary) and measure whether the CoT training data—which already demonstrates what information should be remembered—can be leveraged to train the model to write to and read from external memory. The key experiment: does adding external memory to a model trained with CoT-based memory annotations close the gap with workflow systems, or does effective memory use require fundamentally different training?

Stress-testing generalization to applications without virtual environment coverage. The paper does not disclose which applications were virtualized or evaluate on held-out application categories. A critical stress-test would evaluate GUI-Owl-1.5 on a benchmark composed entirely of applications that were not represented in the virtual environment training data, measuring whether the virtual environment benefit transfers to novel interfaces or whether performance collapses to the non-virtual-environment baseline (the 42.0% and 50.0% levels from Table 11). If transfer is strong, virtual environments are a general capability amplifier; if transfer is weak, they are essentially an expensive form of domain-specific fine-tuning. The experiment requires constructing a benchmark of applications from categories disjoint from the training virtual environments—for example, if training virtualized office and food-delivery apps, testing on healthcare, finance, and education apps.

Ablating the MRPO online rollout buffer and Swap1 mechanism in isolation. The online rollout buffer (Section 2.4.3) is the most technically novel MRPO component, with formal analysis of collapse probability and an unbiasedness argument, but it is never experimentally validated. A focused RL training study would compare four conditions for handling GRPO outcome collapse: (a) naive GRPO with discarded collapsed groups, (b) the online rollout buffer with Swap1 as described, (c) a simple oversample-and-filter approach (sample kn trajectories, discard groups where all outcomes are identical, use the remainder), and (d) an off-policy replay buffer. The key metrics would be sample efficiency (environment interactions to reach a given success rate) and final policy performance, measured on a single platform to isolate the RL dynamics from the cross-platform issues. This would determine whether the formal machinery of Swap1 and unbiasedness arguments actually matter in practice, or whether simpler diversity-promoting strategies work equally well.

Can CoT synthesis quality be self-improved through iterative training? The paper's CoT synthesis pipeline uses proprietary VLMs (Claude-4.5) to generate reasoning annotations, creating a dependency on external model quality and a ceiling on reasoning capability. A follow-up would explore whether GUI-Owl-1.5 can be used to re-annotate its own training trajectories after initial training—generating CoT for new trajectories using the trained model, filtering for quality using task success as a proxy signal, and retraining on the self-generated annotations. The experiment would measure whether CoT quality (and downstream task performance) improves, plateaus, or degrades across iterations. A positive result would establish a self-improvement loop that reduces dependence on proprietary models; a negative result would confirm that CoT quality is fundamentally bounded by the annotator model and cannot be bootstrapped.

Practical Applications and Downstream Use Cases

Edge-cloud collaborative mobile automation. The paper demonstrates that small Instruct variants achieve strong performance without the latency cost of generating reasoning tokens: GUI-Owl-1.5-2B-Instruct attains 67.9 on AndroidWorld and 43.5 on OSWorld, outperforming models with 10-36× more parameters (UI-TARS-72B-DPO at 27.1 on OSWorld). This enables a deployment architecture where a 2B or 4B model runs entirely on-device for high-frequency, latency-sensitive interactions (opening apps, navigating menus, filling forms), while a cloud-based 32B-Thinking model handles complex multi-step tasks requiring planning and reflection. The on-device model addresses privacy concerns (screenshots never leave the device for routine tasks) and eliminates network latency for simple interactions, while the cloud model provides capability depth when needed. The paper's demonstration that the same model weights work across platforms means a single model family serves phone, tablet, and desktop use cases, reducing the operational complexity of maintaining separate per-device models.

Cost-efficient batch data extraction and process automation. For organizations that need to extract information or perform operations across many applications at scale—insurance claims processing across legacy systems, financial data aggregation from multiple web portals, regulatory compliance checking across document management systems—GUI-Owl-1.5's multi-platform capability means a single deployed model handles desktop, web, and mobile interfaces. The parameter efficiency demonstrated in Table 1 (8B-Thinking at 52.9 on OSWorld, competitive with much larger models) means this can be done with moderate GPU requirements rather than requiring datacenter-scale deployments. The agent's tool-use capability (47.6 on OSWorld-MCP) further enables mixed workflows: extract data via GUI navigation, process it via API calls, and enter results into a different application—all within a single automated pipeline rather than requiring separate automation scripts per application.

Training data generation for self-improving GUI agents. The virtual environment pipeline (Section 2.2.2) demonstrates a method for generating clean, validated GUI trajectories at scale without human annotation. This infrastructure can be repurposed for iterative self-improvement: deploy GUI-Owl-1.5 in virtual environments, collect trajectories on tasks where it succeeds, use those trajectories as additional training data for the next model iteration, and repeat. The deterministic feedback in virtual environments ensures that only genuinely successful trajectories enter the training set, avoiding the noise amplification problem that plagues self-training on real-device rollouts. The paper's finding that unstable-task-focused RL training converges faster (Figure 8a) suggests a natural curriculum: focus data collection and retraining on tasks where the model's success rate is in the informative middle range (20-80%), rather than wasting compute on tasks the model has already mastered or fundamentally cannot do.

Accessibility and assistive technology. A 2B-parameter model that can reliably navigate mobile and desktop interfaces opens the door to on-device GUI agents for users who cannot interact with standard graphical interfaces—people with motor impairments, visual impairments, or cognitive disabilities that make precise pointing and clicking difficult. The model's grounding capability (72.9 on ScreenSpot-Pro without crop tool) means it can locate and interact with UI elements from natural language descriptions, enabling voice-driven interface control. The small model size (2B parameters) is critical here because assistive technology must run with low latency on consumer hardware, not datacenter GPUs. The instruct variant's faster inference (no reasoning token generation) is well-suited for the real-time interaction requirements of assistive use.

When to Prefer This Method

The paper does not explicitly position GUI-Owl-1.5 against a named alternative methodology with clear tradeoff criteria. It compares against a broad set of models (general-purpose VLMs, single-platform GUI specialists, multi-platform GUI agents, proprietary systems) but does not articulate decision rules for when a practitioner should adopt the GUI-Owl-1.5 training pipeline versus the training pipelines of UI-TARS, MAI-UI, or other contemporaneous approaches. The paper's contribution is a specific training recipe—hybrid data flywheel, unified CoT synthesis, MRPO—and the implicit recommendation is that this recipe produces stronger multi-platform agents than prior recipes, but the paper does not analyze why a practitioner would choose a different approach or under what conditions the advantages diminish. A forced decision matrix would therefore be generic rather than grounded in the paper's own analysis.