Agents2025~126 minarXiv:2509.02544

UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning

score 6Haoming Wang, Haoyang Zou, Huatong Song +109ByteDance Seed↑ 125★ 9,757Sep 2, 2025cs.AIhttps://github.com/bytedance/ui-tars

🎯 Pitch

UI-TARS-2 shows that multi-turn reinforcement learning on browser tasks unexpectedly yields large out-of-distribution gains—boosting performance on OSWorld by ~10.5% and AndroidWorld by ~8.7%—without any domain-specific RL training. By stabilizing PPO with value pretraining and asynchronous rollouts, the model reaches 47.5 on OSWorld and a 59.8 mean normalized score across 15 games, rivaling frontier proprietary agents.

1. Executive Summary

This technical report introduces UI-TARS-2, a native GUI-centered agent model trained through a Data Flywheel — a self-reinforcing loop that cycles model-generated trajectories through continual pre-training, supervised fine-tuning, rejection sampling, and multi-turn reinforcement learning — enabling both model and data to co-evolve. Evaluated on the MATH of GUI benchmarks — OSWorld, WindowsAgentArena, AndroidWorld, and Online-Mind2Web — UI-TARS-2 achieves 47.5, 50.6, 73.3, and 88.2 respectively, outperforming strong baselines such as Claude and OpenAI agents while also reaching a mean normalized score of 59.8 across a 15-game suite (~60% of human performance). The paper’s key mechanism is a stabilized multi-turn RL framework (Proximal Policy Optimization enhanced with value pretraining, decoupled advantage estimation, and length-adaptive GAE, running over asynchronous stateful rollouts across heterogeneous sandbox environments), which yields substantial OOD generalization — RL trained primarily on browser tasks transfers to OSWorld with a ~10.5% gain and to AndroidWorld with an ~8.7% gain — establishing that multi-turn RL in interactive environments induces broadly transferable skills even when the training distribution is domain-specific.

2. Context and Motivation

The Core Problem: Building GUI Agents That Actually Work in the Real World

The fundamental challenge this paper tackles is deceptively simple to state but extraordinarily difficult to solve: how do you build an AI agent that can reliably use any computer the way a human does—by looking at the screen, reasoning about what to do, and clicking, typing, and scrolling to accomplish real tasks? This is not a toy problem. It encompasses booking flights, debugging code, organizing files, filling out forms, navigating complex multi-page workflows, and playing interactive games—all through the same pixel-level interface that humans use.

The difficulty lies in the intersection of several hard sub-problems that compound each other. The agent must perceive visual interfaces (which vary dramatically across operating systems, applications, and languages), reason about multi-step plans where each action depends on the outcome of the previous one, act precisely in continuous coordinate spaces, and recover when things go wrong—all while maintaining context over interactions that can span dozens or hundreds of steps. Unlike text-only tasks where correctness can often be verified by comparing output strings, GUI tasks require the agent to manipulate a live environment where success is measured by the state of the system after a sequence of operations.

This problem matters for reasons that extend far beyond academic benchmarks. The paper implicitly argues—and the broader research community increasingly recognizes—that GUI agents represent the natural endpoint of making AI systems genuinely useful for everyday computer work. If successful, such agents could:

Automate repetitive enterprise workflows (data entry, report generation, cross-application orchestration)
Assist users with complex software (video editing, CAD, financial modeling) by executing natural language instructions
Enable accessibility for users who cannot interact with traditional interfaces
Scale testing and QA for software development
Serve as a foundation for general-purpose digital assistants that don't require API access to every tool

However, prior to this work, the gap between these aspirations and reality was substantial.

Where Prior Approaches Fall Short

The paper identifies four interconnected failure modes in existing approaches that, taken together, explain why robust, general-purpose GUI agents have remained elusive.

1. Pipeline Architectures Are Brittle by Design

Traditional approaches to GUI automation adopt modular pipelines where perception, planning, memory, and action execution are handled by separately engineered components (Section 1). A typical system might use a specialized object-detection model to locate buttons and text fields, a planner that constructs a symbolic task graph, a memory module that tracks state, and an executor that translates plans into mouse/keyboard actions.

The paper identifies a fundamental weakness in this paradigm: "While such design-driven systems enable rapid progress in specific domains, they rely heavily on expert heuristics and task-specific rules, leaving them brittle and difficult to scale." Each module is a potential point of failure, and the interfaces between modules create compounding error. The perception module might misclassify a dropdown as a text field; the planner might generate a plan that assumes a UI element exists when it doesn't; the memory module might lose track of which tab is active. Each of these failures propagates through the pipeline in ways that are hard to diagnose and impossible for the system to self-correct.

More fundamentally, pipeline approaches don't benefit from the scaling properties that have made large language models so effective in other domains. You cannot simply "train a bigger pipeline" and expect proportional improvements—each component requires hand-engineering for new applications and environments, and the integration between components remains a bottleneck regardless of individual component quality.

2. Data Scarcity: Agents Live in a Different Data Regime Than Chatbots

The paper emphasizes a point that distinguishes agent training from standard LLM training: "Unlike text or code corpora, large-scale trajectories that capture detailed reasoning, actions, environment states, and feedback are extremely costly to collect" (Section 1). This is not merely an engineering inconvenience—it's a fundamental difference in data availability.

Standard LLM pre-training benefits from the fact that the internet contains trillions of tokens of text. Fine-tuning for chat benefits from the fact that human conversation data can be collected at scale through crowd-sourcing and synthetic generation. But agent trajectories require something fundamentally more expensive: a live environment where each action changes the state, observations must be captured (screenshots, DOM states, etc.), and the trajectory must be annotated with the reasoning behind each action—not just the action itself. The paper notes that "publicly available data is inherently scarce and easily exhausted, leaving insufficient coverage for training at scale" (Section 2.4.1).

This scarcity creates a vicious cycle. Without large-scale trajectory data, agents must be trained on limited demonstrations, which makes them brittle. Brittle agents fail in complex environments, which means they can't be used to generate more training data autonomously. The paper's Data Flywheel is designed explicitly to break this cycle, but the fact that such machinery is necessary underscores how severe the data problem is.

3. Multi-Turn RL Is Unstable and Hard to Scale

Even if you have environments and tasks, training agents through reinforcement learning over long interaction sequences introduces a distinct set of optimization challenges that the paper identifies as open problems:

Sparse and delayed rewards. In GUI tasks, the agent typically receives meaningful feedback only at the end of an episode—either the task succeeded or it didn't. There are no intermediate rewards for "you clicked the right button" because in most real tasks, there is no ground truth for intermediate steps. This means the RL algorithm must assign credit across dozens or hundreds of actions from a single terminal reward signal. Standard approaches to this credit assignment problem (e.g., temporal difference learning with function approximation) can be unstable, especially when combined with the high variance introduced by the visual complexity of GUI environments.

Optimization instability. The paper observes that "reinforcement learning in interactive environments is notoriously difficult" (Section 1), a claim supported by extensive RL literature. Value function estimation can diverge, policy updates can oscillate, and the exploration-exploitation tradeoff becomes acute when the action space includes all pixel coordinates on a screen. The paper's finding that "value estimates of PPO-trained models were often negatively correlated with the obtained rewards" (Section 3.3) in preliminary experiments is a concrete example of the kind of instability that plagues naive applications of RL to GUI tasks. If the value model—which is supposed to tell the policy whether its actions are good—is inversely correlated with actual outcomes, the entire optimization process becomes counterproductive.

Long-horizon credit assignment. GUI tasks can involve dozens of environment interaction rounds. A task like "download the Q3 report from the company portal, extract the revenue numbers, and paste them into a spreadsheet" might require navigating to a website, logging in, navigating to the reports section, downloading a file, opening it, locating specific cells, copying them, and pasting—each step contingent on the success of the previous one. Standard RL algorithms struggle to propagate learning signals across such long horizons, especially when the initial policy performs poorly and produces few successful trajectories.

4. GUI-Only Interfaces Are Fundamentally Limited for Real Work

The paper makes a pragmatic observation that many real-world tasks are "more naturally handled through file systems, terminals, or external tools rather than by simulating mouse clicks and keystrokes" (Section 1). Consider: if you ask an agent to "find all Python files modified in the last week and zip them," it could theoretically navigate through a file explorer GUI, manually select each file, and right-click to compress. But this is laughably inefficient compared to running find . -name "*.py" -mtime -7 | xargs zip archive.zip in a terminal. The agent needs both capabilities and must know when to use each.

Pure GUI interaction also breaks down for tasks that require:

Software development: running compilers, package managers, version control
Data processing: executing scripts, querying databases, manipulating large files
System administration: managing services, checking logs, configuring networks
Web development: previewing frontends, testing APIs, deploying backends

A GUI agent without access to these capabilities is like a human operator with their hands tied—they can look at the screen and click, but they can't use the full power of the computer they're operating. The paper argues that "advancing GUI agents requires environments that allow graphical actions to interoperate seamlessly with other resources" (Section 1).

5. Environment Infrastructure Is a Bottleneck That Nobody Talks About

The final challenge the paper identifies is one that is often overlooked in academic research but dominates real-world deployment: the engineering difficulty of running agent experiments at scale. The requirements are daunting:

Reproducibility: Two runs of the same agent on the same task should produce the same outcome, which means the environment state must be resettable. This requires virtualization or containerization.
Fault tolerance: When you're running millions of interactive episodes across thousands of VM instances, environments will crash. The infrastructure must handle this gracefully without corrupting training data.
State management: Multi-turn interactions require preserving VM state across calls, tracking which session belongs to which task, and cleaning up resources when tasks complete or fail.
Observation capture: Screenshots must be captured and transmitted efficiently; for games, this must happen at interactive frame rates.
Heterogeneity: Different tasks require different operating systems (Windows, Ubuntu, Android), different application configurations, and different tool sets.

The paper notes that "in practice, such environments are fragile, resource-intensive, and prone to crashes, making stable large-scale training particularly challenging" (Section 1). This is not merely an infrastructure aside—without solving this engineering problem, none of the algorithmic advances matter because you simply cannot run the experiments.

How This Paper Positions Itself

UI-TARS-2 positions itself as addressing these five challenges through a systematic, integrated methodology rather than a single algorithmic innovation. This is an important distinction. The paper's contribution is not "here is a better PPO variant for GUI agents" but rather "here is a complete system—spanning data generation, environment infrastructure, RL stabilization, and model architecture—that makes training robust GUI agents feasible at scale."

The paper explicitly contrasts itself with workflow-based prompting approaches (such as those using ReAct-style reasoning interleaved with tool calls, referenced in Section 4) that achieve some success through careful prompt engineering and external tools but "lack flexibility" and don't improve with experience. It also contrasts with data-driven supervised approaches (such as CogAgent and OS-Atlas, cited in Section 4) that train on human demonstrations but "suffer from limited generalization and poor robustness in complex environments" because the training data is off-policy—it doesn't reflect the distribution of states the agent would actually encounter during deployment.

The key positioning claim is that multi-turn RL in interactive environments is the missing ingredient that transforms a model from mimicking demonstrations to learning from its own experience. The paper frames this as a natural extension of the recent success of RL in reasoning domains (explicitly citing DeepSeek-R1 and RLVR in Section 2.5), but with the crucial difference that GUI environments introduce challenges—visual complexity, long horizons, stateful interactions—that reasoning RL on text prompts does not face.

Perhaps most importantly, the paper positions its contributions as mutually reinforcing. The Data Flywheel (Section 2.3) can't work without a stable RL framework to generate high-quality trajectories; the RL framework (Section 2.5) can't work without scalable environment infrastructure for rollouts; the environment infrastructure (Section 2.2) must support both GUI and SDK operations to handle realistic tasks; and the parameter interpolation strategy (Section 2.6) makes it practical to train across diverse domains without joint optimization. Each component addresses a failure mode that would prevent the others from succeeding. This systems-level perspective—where the "method" is the entire training pipeline rather than any single algorithm—is what distinguishes UI-TARS-2 from prior work that tackled individual pieces in isolation.

The Specific Gap: No Unified Recipe for Agent RL

If you were an ML practitioner wanting to train a GUI agent in early 2025, what would you do? There was no established recipe. Prior work offered fragments:

ReAct-style prompting (Yao et al., 2023): Attach a reasoning trace to each action, but this doesn't train the model to get better over time.
SFT on human demonstrations (UI-TARS, OS-Atlas): Collect expert trajectories and fine-tune, but this is expensive, doesn't scale, and the agent never learns from its own mistakes.
RL on specific games or simulators (Atari, StarCraft): Well-established for game-playing, but the environments are purpose-built with clean reward signals and don't transfer to general computer use.
PPO for reasoning (DeepSeek-R1): Proves RL works for improving LLM outputs on math and coding, but these tasks have no interactive environment, no screenshots, and trivial verification.

The paper's central motivation is that none of these fragments, applied individually, produces a general-purpose GUI agent. The gap is an integrated system that combines scalable data generation, stabilized multi-turn RL, hybrid GUI+SDK environments, and mergeable specialized models—and the paper aims to fill this gap by providing not just results but a reproducible methodology that others can adopt.

3. Technical Approach

3.1 Reader Orientation

UI-TARS-2 is a native GUI-centered agent model—a single neural network that takes screenshots and task instructions as input, reasons about what to do, and outputs mouse/keyboard actions or system commands, all learned end-to-end through a training pipeline that iterates between data generation, supervised learning, and multi-turn reinforcement learning. The problem it solves is building an agent that can reliably operate arbitrary computer interfaces in the real world, and the "shape" of the solution is a self-reinforcing cycle: better models generate better training data, which trains better models, with stabilized multi-turn RL providing the optimization signal that drives improvement across heterogeneous interactive environments.

3.2 Big-Picture Architecture (Diagram in Words)

The UI-TARS-2 system has seven major components arranged in a training pipeline that cycles continuously:

All-in-One GUI Sandbox (Section 2.2) — a distributed platform of cloud VMs and browser containers that provides standardized, reproducible environments for GUI interaction, game playing, and terminal use. It is the "world" in which agents act and from which observations are captured.
Data Flywheel Controller (Section 2.3) — the meta-loop that orchestrates the training cycle. It takes trajectories generated by the current model, evaluates their quality via a validation function V(s) → {0,1}, and routes high-quality trajectories to the SFT dataset while routing lower-quality ones to the CT dataset, ensuring that every generated sample is reused at an appropriate training stage.
CT & SFT Data Preparation Pipeline (Section 2.4) — two complementary data collection systems: (a) in-situ annotation that captures human cognitive processes during real computer use via think-aloud protocols, producing reasoning–action trajectories for continual pre-training, and (b) interactive annotation that enables human experts to provide real-time corrections during agent rollouts, producing on-policy SFT data.
Multi-Turn RL Training Engine (Section 2.5) — the core optimization loop. An asynchronous rollout infrastructure runs the agent policy in sandbox environments, collects trajectories with verifiable or model-judged rewards, and updates the policy using an enhanced PPO algorithm with value pretraining, decoupled advantage estimation, and length-adaptive GAE.
Reward System (Section 2.5.2) — a hybrid reward architecture that uses three types of reward signals depending on task type: (a) function-based verifiers for games (directly query runtime variables like score), (b) LLM-as-Judge for GUI-Browsing tasks (compare predicted answer against ground truth), and (c) a generative Outcome Reward Model (ORM) for open-ended GUI-General tasks where no ground truth exists.
Parameter Interpolation Merger (Section 2.6) — a post-training consolidation step. Rather than jointly training across all domains (which would be unstable and computationally prohibitive), the system trains specialized agents independently for each domain (GUI-Browsing, GUI-General, Game, GUI-SDK) and then merges them via linear interpolation of their parameters: θ(merge) = Σ α_k · θ(k) with mixture weights summing to 1.
The Agent Policy Itself (Section 2.1) — a 532M-parameter vision encoder paired with a Mixture-of-Experts LLM (23B active parameters, 230B total) initialized from Seed-thinking-1.6. At each timestep t, it takes the current screenshot o_t, the task instruction, working memory W_n (the last N steps in high fidelity), and episodic memory E_n (compressed summaries of past episodes), and predicts the next reasoning trace t_n and action a_n.

Information flow through the system follows a cycle: (1) The Data Flywheel initializes with cold-start CT and SFT datasets → (2) The model is sequentially trained on CT → SFT → RL → (3) The RL-trained model generates new trajectories via rejection sampling or interactive annotation → (4) A validation function classifies each trajectory as high-quality or low-quality → (5) High-quality trajectories are appended to the SFT dataset for the next iteration; low-quality trajectories are routed to the CT dataset → (6) The expanded datasets feed the next training cycle, with the improved model generating a higher proportion of high-quality trajectories, accelerating the flywheel.

3.3 Roadmap for the Deep Dive

The detailed breakdown follows the system's logical architecture, moving from the innermost component (how the agent is formally defined) outward through the infrastructure, data pipeline, optimization algorithm, and finally to deployment-time considerations:

First, the agent formulation (Section 2.1): the mathematical definition of what the agent computes at each timestep, including the ReAct loop, the action space, and the hierarchical memory structure. This establishes notation and the decision-making framework that all subsequent components optimize.
Second, the environment infrastructure (Section 2.2): the cloud VM platform and browser sandbox that make training possible. Understanding the engineering of these environments is prerequisite to understanding the RL training loop that runs on top of them.
Third, the Data Flywheel (Section 2.3): the outer loop that governs how data flows between training stages. This explains the self-reinforcing dynamic that distinguishes UI-TARS-2 from static training pipelines.
Fourth, the data preparation methods (Section 2.4): the two annotation systems (in-situ and interactive) that produce the CT and SFT datasets. These are the "fuel" that powers the flywheel.
Fifth, the multi-turn RL framework (Section 2.5): the core optimization engine, including task design, reward design, asynchronous rollout infrastructure, and the enhanced PPO algorithm. This is the most technically dense section and directly produces the model improvements observed in experiments.
Sixth, parameter interpolation (Section 2.6): the strategy for consolidating domain-specialized models into a single unified agent, which replaces the need for prohibitively expensive joint multi-domain RL.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methodology paper whose core idea is that robust GUI agents emerge from a self-reinforcing training cycle where multi-turn RL in interactive environments serves as the primary optimization driver, supported by scalable data generation, stabilized optimization techniques, and mergeable domain-specialized policies.

Formal Agent Definition: The ReAct Loop with Hierarchical Memory

The paper models the agent as a parameterized policy that, at each timestep t, follows the ReAct paradigm (Reasoning + Acting) introduced by Yao et al. (2023). A step is defined as one complete cycle of three components:

Reasoning (t_t): the model generates internal text that includes context analysis, memory recall, planning, and self-reflection. This is the "thinking" that precedes each action.
Action (a_t): the model outputs a concrete operation—a GUI manipulation (click, type, scroll), a pre-defined SDK function call (terminal command, MCP tool invocation), or a game interaction.
Observation (o_t): the environment returns feedback—a screenshot, a terminal output, or a game state—which the agent uses to update its understanding for the next step.

A trajectory of length T is the concatenation of these triples:

$\tau = \{(t_0, a_0, o_0), (t_1, a_1, o_1), \ldots, (t_T, a_T, o_T)\}$

where τ (tau) is the complete interaction history, t_i is the reasoning trace at step i, a_i is the action at step i, and o_i is the observation returned after action i.

What it computes: this equation formalizes the data structure that the RL algorithm will optimize over. Each trajectory τ is a sequence of (thought, action, observation) triples that records the agent's entire interaction with the environment for one task. When the RL algorithm computes advantages and updates the policy, it operates over these trajectories—the reasoning and action tokens are what the policy generates and gets rewarded for, while the observations are environment feedback that conditions future outputs.

Why this form: the ReAct decomposition into reasoning-then-action is motivated by empirical evidence that interleaving chain-of-thought with environment interaction improves performance over either pure reasoning (which ignores environment feedback) or pure action prediction (which lacks planning). The explicit separation of t_t (internal thought) from a_t (external action) makes it possible to train the model to produce both, where the reasoning trace serves as a form of working memory that helps the model maintain coherent plans across long horizons. This is particularly important for GUI tasks because the correct action often depends on multi-step intentions that are not recoverable from the screenshot alone.

The paper introduces a hierarchical memory state to manage long trajectories efficiently:

$\mathcal{M}_t = (\mathcal{W}_t, \mathcal{E}_t)$

where M_t is the memory state at time t, W_t is the Working Memory storing recent steps (t_{t-k}, a_{t-k}, o_{t-k}) in high fidelity for short-term reasoning, and E_t is the Episodic Memory maintaining semantically compressed summaries of past episodes, preserving key intentions and outcomes.

What it computes: this is the agent's internal state that persists across timesteps. Working memory is essentially a sliding window of the most recent N steps, giving the model direct access to recent context (the last few screenshots and actions). Episodic memory is a compressed representation of everything that happened before that window—summaries of past sub-goals, outcomes, and failures—so the model can recall long-term context without storing full-resolution observations for the entire trajectory.

Why this form: the dual-memory design solves a fundamental tension in long-horizon GUI tasks. Keeping full-resolution observations for hundreds of steps would exceed context window limits and dilute the model's attention. But discarding older context entirely would prevent the model from remembering what it already tried and what worked or failed. The episodic memory compresses old context into semantic summaries—losing visual detail but preserving strategic information—while working memory retains the detailed visual context needed for precise UI interaction. The paper notes that direct context is restricted to the last N steps from working memory, with episodic memory used for longer-term recall, though the exact value of N is not specified in the report.

The policy prediction at each timestep is:

$P(t_n, a_n \mid \text{instruction}, \mathcal{W}_n, o_n, \mathcal{E}_n)$

where P is the probability distribution over (thought, action) pairs, conditioned on the task instruction, working memory, current observation, and episodic memory.

What it computes: at each timestep n, the model takes the task instruction (what the user asked for), the current screenshot o_n, the recent history in working memory W_n, and the compressed long-term history in episodic memory E_n, and outputs a probability distribution over possible next (thought, action) pairs. The highest-probability pair is selected as the model's output for that step.

Why this form: this formulation makes explicit that agent behavior arises "not from isolated predictions, but from an evolving loop of reasoning, action, feedback, and memory integration" (Section 2.1). Each decision is conditioned on the full interaction history, making the policy Markovian in the augmented state (W_n, o_n, E_n) rather than in raw observations alone. This is critical for RL because it means the policy can, in principle, learn to adapt its behavior based on what it has already tried and observed—for example, avoiding UI elements it has already determined are irrelevant, or switching strategies when a previous approach failed.

The action space spans two categories:

GUI Actions: direct interface manipulation following the UI-TARS action vocabulary—clicks for element selection, typing for text input, scrolling for navigation. Gameplay interactions reuse these same primitives, meaning the model uses the same action interface whether it is filling out a form on a website or playing a browser-based game.
Pre-defined SDK Functions: supplementary operations beyond GUI manipulation, including terminal commands for file management and software development, and MCP (Model Context Protocol) tool invocations for orchestrating external services. This is the bridge that allows the agent to do things that would be inefficient or impossible through GUI clicks alone—like running grep across a codebase or calling a search API.

The All-in-One GUI Sandbox: Engineering Infrastructure for Agent Training

The sandbox is designed to solve a concrete problem: how do you run millions of agent interaction episodes across heterogeneous environments (Windows, Ubuntu, Android, browser games) in a way that is reproducible, fault-tolerant, and high-throughput? The paper describes two environment types in detail.

Cloud Virtual Machine Platform for GUI Tasks

The GUI sandbox is a distributed virtual machine (VM) platform running mainstream desktop operating systems (Windows and Ubuntu) as well as Android. The architecture integrates several components:

Infrastructure scale and management. The VM cluster comprises "several thousand instances, centrally managed by a VM Manager capable of sustaining throughput at several thousand QPS (Queries Per Second) and handling high-concurrency execution." Each session is tracked via a task–environment mapping using session IDs, ensuring that state remains consistent across multi-round interactions—if an agent interacts with the same VM across 50 steps, the VM's state (open applications, file system contents, browser tabs) must persist correctly between calls.

Lease-based lifecycle management. A lease-based mechanism automatically releases resources after task completion or failure. Overdue sessions are reclaimed to prevent resource waste. This is critical for fault tolerance: if an agent gets stuck in an infinite loop or a VM crashes, the infrastructure must recover without manual intervention.

Unified interaction pipeline. The platform integrates PyAutoGUI (for desktop automation) and ADB (Android Debug Bridge) interfaces, enabling cross-device operations with minimal adaptation overhead. A unified SDK standardizes the entire pipeline: VM allocation → initialization → agent interaction → observation collection (screenshots and recordings) → task evaluation. This uniformity means the same agent code can interact with Windows, Ubuntu, and Android environments through a consistent API, with the platform handling the OS-specific translation.

Monitoring and debugging. All sessions are visualizable in real time via VNC (Virtual Network Computing) or RTC (Real-Time Communication), enabling human operators to watch agent behavior and debug failures. This is essential for annotation and for diagnosing why an agent failed on a particular task.

Beyond GUI: tool integration. The sandbox extends beyond GUI manipulation by pre-loading essential local services (browsing, file access, terminal use) and providing proxy URLs for services launched from the terminal. This means a GUI agent can, for example, start a web server from the terminal and then view the resulting website in the browser—all within the same sandbox. For human-in-the-loop debugging and annotation, the environment provides VNC, a remote VS Code editor, Jupyter, and terminal previews directly in the browser.

A core innovation: shared file system. The paper highlights that the sandbox includes a shared file system where "an GUI agent can, for instance, download a file via the browser and immediately process it using shell commands within the same containerized instance." This is the architectural feature that makes hybrid GUI+SDK workflows possible—the file downloaded through GUI interaction exists in the same filesystem that terminal commands operate on, enabling seamless transitions between interaction modalities.

Hardware-Accelerated Browser Sandbox for Games

For web-based mini-games running in HTML5/WebGL, the paper built a separate browser sandbox optimized for the specific requirements of game RL:

Why a separate sandbox? The paper explains that "because these mini-games run entirely in HTML5/WebGL, a browser environment is the only practical way to execute them faithfully while capturing their full interactive state." Desktop VM platforms are overkill for browser games and would introduce unnecessary overhead.

Concurrency model. Multiple browser instances run per container with elastic scheduling, as illustrated in Figure 2. The system monitors main processes and performs automatic crash recovery to ensure long-running stability—important because browser-based games can crash, especially when driven by an exploring RL agent that might trigger edge cases.

Page management layer. A dedicated page-control layer handles page creation and deletion, maintains session–page mappings (which browser tab belongs to which training episode), tracks page states, and executes commands. Checkpointing ensures reproducibility—the environment can be reset to a known state for each new episode.

Resource management. An event handler continuously reports browser/page events (crashes, timeouts, state changes) to a central manager, and a garbage collector reclaims idle sessions to prevent resource leakage. This is the operational backbone that keeps the system running stably over millions of episodes.

Performance optimizations. GPU-based hardware acceleration reduces screenshot overhead—capturing screenshots is a major bottleneck in vision-based RL because every policy step requires a new observation. Re-implemented Window timing APIs allow time acceleration and pause at startup, improving sampling efficiency (the agent can run faster than real-time) and reproducibility without altering game logic.

Interoperability. The sandbox is compatible with the Chrome DevTools Protocol and popular drivers such as Playwright, enabling "orchestrated, debuggable, and auditable interaction." This means the same infrastructure can be used for training (where the RL loop drives the agent) and for evaluation (where a human or script can inspect behavior).

Design rationale for the two-sandbox architecture. The paper's decision to build separate sandboxes for GUI tasks (cloud VMs) and games (browser containers) reflects a pragmatic engineering tradeoff. Cloud VMs provide full OS fidelity needed for desktop applications but are heavier to manage; browser containers are lightweight and can be packed densely for game RL but can't run native desktop software. Rather than forcing one environment type to handle both use cases poorly, the paper builds specialized infrastructure for each and unifies them under a consistent API. This is a systems design lesson: the right abstraction layer is at the API, not at the implementation, when the underlying requirements are fundamentally different.

The Data Flywheel: Self-Reinforcing Model–Data Co-Evolution

The Data Flywheel is the outer loop that governs UI-TARS-2's training. Its core premise is that model quality and data quality can improve each other iteratively—better models generate better trajectories, which serve as better training data, which produces better models. The paper formalizes this as a cycle with three training stages and a routing mechanism.

Training Stages

Starting from the pre-trained checkpoint of Seed-thinking-1.6 (a proprietary ByteDance model), the flywheel operates through three sequential stages:

Continual Pre-training (CT): broad knowledge acquisition from large-scale, diverse data. This stage continues the pre-training of the base model with a mixture of general-purpose data and agent-specific data. The paper notes that "agent-specific data constitutes only a small fraction of CT, which emphasizes broad knowledge acquisition." CT is about building general capabilities (visual understanding, reasoning, knowledge about applications and websites) that the agent will draw upon during fine-tuning.
Supervised Fine-tuning (SFT): high-quality, task-specific instruction tuning. This stage fine-tunes the model on expert demonstrations of agent behavior, where each example consists of a task instruction, a sequence of (thought, action) pairs, and the environment observations that resulted. The paper notes that "agent data forms a much larger proportion of SFT, which focuses on high-quality, task-specific agent trajectories." SFT teaches the model how to act as an agent—what reasoning to produce, what actions to take—in a format that matches deployment.
Reinforcement Learning (RL): end-to-end optimization on verifiable interactive tasks. This stage lets the model interact with live environments, receive rewards, and improve through policy gradient updates. Unlike SFT, which teaches imitation of fixed demonstrations, RL lets the model learn from its own successes and failures.

Why this three-stage ordering? The paper follows a standard curriculum: CT provides broad capabilities → SFT provides task-specific behavioral cloning → RL provides trial-and-error refinement. This ordering is important because RL from a randomly initialized policy would be hopelessly inefficient—the agent would take random actions in complex GUI environments and almost never receive positive rewards. SFT provides a reasonable starting policy that RL can then improve upon. The paper also notes that "SFT and RL are performed more frequently than CT," suggesting that CT is a relatively stable foundation while SFT and RL cycles iterate more rapidly to incorporate new data from the flywheel.

Cold-Start Data Sources

The flywheel is bootstrapped with two initial datasets:

CT initialization (D_CT^(0)): includes "task tutorials, instructional videos, demonstrations from the internet, and our in-house data" plus "all training data from UI-TARS and UI-TARS-1.5, consisting of GUI tutorials collected from the internet, open-source agent trajectories, our in-house annotations." This gives the model a broad base of GUI-related knowledge before any flywheel iterations begin.

SFT initialization (D_SFT^(0)): constructed "through synthetic data generation and human annotation." The paper doesn't detail the synthetic generation process for the cold-start SFT data, but the interactive annotation system (Section 2.4.2) would be the primary source.

Iterative Data Flow

After the initial RL model is trained (the first pass through CT → SFT → RL), it becomes the primary data generator for subsequent iterations. The process works as follows:

The current RL-trained model M^(t) generates new trajectories via rejection sampling (RFT) or interactive annotation (Section 2.4.2). Rejection sampling means the model attempts tasks and trajectories where the task was completed successfully (as determined by the validation function) are kept; failed trajectories may still be useful for CT. Interactive annotation means human experts provide real-time corrections during model rollouts, producing high-quality trajectories.
Each generated sample s is evaluated by a validation function V(s) → {0, 1} that determines whether the trajectory successfully completed the task.
Routing: high-quality samples with V(s) = 1 are added to the SFT dataset for the next iteration:

$D_{SFT}^{(t+1)} = D_{SFT}^{(t)} \cup D_{RFT,high}^{(t)}$

Lower-quality samples with V(s) = 0 are routed to the CT dataset:

$D_{CT}^{(t+1)} = D_{CT}^{(t)} \cup D_{RFT,low}^{(t)}$

where D_SFT^(t) is the SFT dataset at iteration t, D_CT^(t) is the CT dataset at iteration t, D_{RFT,high}^(t) is the set of successful trajectories generated by the model at iteration t, and D_{RFT,low}^(t) is the set of unsuccessful trajectories.

What it computes: the data allocation for the next training iteration. Successful trajectories (where the agent completed the task correctly) are treated as high-quality demonstrations and used to improve the model's imitation capabilities via SFT. Unsuccessful trajectories (where the agent failed) still contain useful information—the model saw something, tried something, and got a negative outcome—and are routed to CT where they contribute to broader knowledge without contaminating the supervised signal.

Why this form: this routing strategy implements a key insight: no generated data is wasted. Successful trajectories teach the model what to do; unsuccessful trajectories teach the model about the environment and what not to do. By routing failed trajectories to CT rather than SFT, the paper avoids a common failure mode where SFT on imperfect demonstrations causes the model to learn suboptimal behaviors. CT is a lower-stakes training stage focused on broad knowledge, so it can absorb imperfect data without degrading performance. The paper explicitly states: "This ensures that SFT always receives the most recent, verified high-quality data, while CT continually expands with broader, less polished knowledge without contaminating the supervised signal."

The flywheel's self-reinforcing property is formalized as:

$P(V(s) = 1 \mid t) > P(V(s) = 1 \mid t - 1)$

where P(V(s) = 1 | t) is the probability that a trajectory generated at iteration t is successful.

What it computes: the accelerating quality of generated trajectories. As the model improves across iterations, a higher fraction of its generated trajectories succeed, which means more high-quality data enters the SFT pool, which means the next iteration's SFT training is on better data, which further improves the model.

Why this form: this inequality captures the flywheel's virtuous cycle dynamic. It's not guaranteed to hold—if the RL training diverges or the SFT data quality degrades, the success rate could decrease—but the paper's training infrastructure is designed to make it hold in practice. The monotonically increasing training rewards shown in Figure 7 provide empirical evidence that the flywheel is working as intended.

Cross-stage transfer. The paper notes that "in each cycle, we observe substantial transfer from general-purpose RL to agent-specific domains." This means that improvements from RL training on agent tasks also benefit the model's general capabilities, and vice versa. The mixing of general-purpose and agent-specific data in both CT and SFT stages facilitates this transfer.

In-Situ Annotation for Continual Pre-training

The paper identifies specific limitations in existing GUI datasets that motivate a new annotation methodology:

Scarcity: "Publicly available data is inherently scarce and easily exhausted."
Language coverage: "A notable lack of content for Chinese-language applications."
Missing reasoning: "Much of the available data provides only procedural actions while omitting the underlying cognitive reasoning. Models trained solely on such resources tend to mimic surface-level actions without internalizing the logic, leading to spurious or unstable reasoning chains."

The response is a large-scale, human-centric annotation system designed to capture authentic cognitive processes during real computer use.

The In-Situ Deployment Model

A distinguishing feature of the annotation platform is that "the annotation tool is directly installed on annotators' personal computers and runs unobtrusively alongside their normal usage. This design allows data to be collected continuously in realistic, everyday settings, without disrupting natural workflows." This is the "in-situ" aspect—data is collected where and when people actually use computers, not in a lab setting where behavior might be artificial.

Think-Aloud Protocol

The paper describes a critical methodological lesson from a pilot study: "An initial pilot study that attempted to retroactively add reasoning traces to recorded actions proved ineffective, as it was nearly impossible to reconstruct the annotator's original thought process." This is a significant finding about human cognition—once a task is completed, people cannot reliably report what they were thinking at each step, especially for routine operations that became automatic.

The solution, inspired by Deitke et al. (2024), is a think-aloud protocol: "Annotators verbalize their thoughts via audio while completing tasks. These verbalized thoughts are automatically aligned with corresponding UI interactions, producing data that captures both the reasoning chain and the grounded actions." The audio is transcribed using automatic speech recognition (ASR) and then refined by LLMs to produce coherent, high-quality reasoning text. These processed reasoning traces are "precisely synchronized with on-screen actions, yielding temporally aligned reasoning–action trajectories."

Why think-aloud works when retrospective annotation fails: verbalizing thoughts during task execution captures the reasoning in real-time, before the cognitive process is compressed or forgotten. The temporal alignment between spoken reasoning and screen actions provides a natural synchronization signal—the annotator says "I need to find the settings menu" right before clicking on the settings icon, making it straightforward to associate reasoning with the corresponding action.

Two-Group Annotator Strategy

To capture diverse cognitive patterns, the paper recruits two groups:

Experts: provide demonstrations of complex tasks where they already know the correct workflow. This produces data on efficient, knowledgeable interaction.
Novices: "asked to solve unfamiliar tasks through exploration, trial-and-error, and external resources (e.g., web search). The novice track captures valuable data on problem-solving and adaptation when prior knowledge is absent."

Why both groups? Expert trajectories teach the model the optimal path; novice trajectories teach the model how to recover from mistakes, explore unfamiliar interfaces, and seek help from external resources. A model trained only on expert data might fail when encountering an unfamiliar application because it has never seen the exploration and error-recovery behaviors that humans use in that situation. The novice data provides exactly these patterns.

Task Design and Collection

The paper describes a systematic pipeline for selecting what tasks to annotate:

Application selection: candidate applications are selected using publicly available indicators along three dimensions—industry coverage, user engagement, and market penetration—yielding a representative set of mainstream websites and desktop applications.
Task graph construction: for each service, a hierarchical task graph is constructed, and task-importance scores are derived using normalized measures of usage frequency, user benefit, and cross-scenario transferability.
Query generation: a human–LLM collaborative workflow generates multilevel query sets for each subfunction, spanning novice-to-expert skill levels and both single- and multi-application settings.
Difficulty calibration: a difficulty rubric based on step count, cross-page operations, prerequisites, and exception handling ensures balanced coverage across difficulty levels.

This systematic approach ensures the annotation effort is directed at tasks that are representative of real usage patterns rather than arbitrary or convenience-sampled tasks.

Curation Pipeline

All collected data undergoes quality control including "executability verification, deduplication, and dual-annotator review." The paper also mentions that "to further enhance training utility, we programmatically augment linguistic diversity and enrich reasoning chains." This augmentation step is important because human reasoning traces, even when transcribed and refined, may use limited vocabulary or follow repetitive patterns. Programmatic augmentation introduces linguistic variety that helps the model generalize beyond the specific phrasings used by annotators.

Why this level of curation detail matters: the quality of CT data directly determines the breadth of knowledge the model can acquire. The paper's emphasis on capturing reasoning (not just actions), covering diverse skill levels (expert and novice), and ensuring linguistic diversity reflects the understanding that pre-training data quality has multiplicative effects—better pre-training data enables better fine-tuning, which enables better RL, which generates better flywheel data.

Interactive Annotation for Supervised Fine-tuning

The paper identifies a fundamental problem with SFT data for interactive agents: "human-generated SFT data is typically off-policy: it does not reflect the actual distribution of actions that the model would take when interacting with an environment." This is the standard distribution shift problem in imitation learning—the model learns to mimic expert behavior, but when deployed, it encounters states that the expert never visited (because the expert never made the mistakes the model makes), and it doesn't know what to do.

Prior approaches (including UI-TARS) attempted to mitigate this by "asking annotators to correct errors in pre-collected trajectories." But the paper argues this is "fundamentally offline and inefficient: it exposes model weaknesses only after task failure, without enabling real-time intervention or correction during interaction." An annotator looking at a failed trajectory after the fact has to imagine what the agent should have done differently at each step, which is cognitively demanding and error-prone.

The solution is an online, interactive annotation framework where human experts supervise agent rollouts in real time.

Four-Layer Architecture

The interactive annotation platform (illustrated in Figure 4) is built on four layers:

Interaction Layer (top): the user interface where annotators engage with the system. This is the visual frontend that shows the agent's current state, proposed actions, and allows the annotator to accept or override.
Service Layer: processes annotation requests, orchestrating model-generated command execution and human interventions. This is the middleware that coordinates between the human annotator, the agent model, and the execution environment.
Platform Layer: provides scenario-specific execution environments (Computer Use, Phone Use, or Tool Use) tailored to different task categories. This is where the actual agent-environment interaction happens, using the sandbox infrastructure from Section 2.2.
Storage Layer: securely logs annotation data and complete interaction trajectories for downstream training and analysis.

The Annotation Workflow

The interactive annotation process operates as follows:

Annotators are assigned tasks to complete in a controlled virtual environment backed by a cloud-hosted VM or browser sandbox.
At each decision point, the latest UI-TARS-2 model proposes candidate actions together with its reasoning trace. The annotator sees what the model would do at this step.
The annotator can either accept one of the model's suggestions (if the model's reasoning and proposed action are correct) or override it with a better thought and action. When overriding, the annotator provides the correct reasoning and action that the model should have produced.
The accepted or corrected action is executed in the live environment, producing a new observation (screenshot).
Steps 2–4 repeat until the task is complete.

Why this is on-policy: the data reflects the actual distribution of states visited by the current model. When the model makes a mistake (proposes a wrong action), the annotator corrects it, and the correction becomes training data that teaches the model what to do in that specific state—the state that the model actually reached through its own (potentially flawed) decision-making. This is fundamentally different from offline annotation, where the annotator demonstrates the correct path from a clean starting state and the model never learns how to recover from its own errors.

Efficiency features. The paper describes several features that streamline the annotation workflow: "command auto-completion, real-time VM video streaming, and on-screen coordinate visualization, reducing latency and improving annotation accuracy." These are practical engineering details that matter for annotation throughput—if annotators have to wait for VMs to boot or manually type coordinates, the annotation rate drops to the point where collecting sufficient SFT data becomes infeasible.

Periodic refresh. To ensure the annotation data stays on-policy as the model evolves, "both the annotation model and the pool of tasks are periodically refreshed, ensuring that data collection consistently targets the weaknesses of the most recent agent." This means the model used during annotation is updated to the latest checkpoint, and the task distribution is adjusted to focus on tasks where the current model performs poorly.

Why interactive annotation over offline correction: the paper's argument is that real-time intervention captures the annotator's genuine decision-making in context. When an annotator sees the model about to click the wrong button, they naturally think "no, you should click here instead because..." and this reasoning is captured immediately. In an offline setting, reconstructing that reasoning from a completed (failed) trajectory is far harder and less reliable.

Multi-Turn RL: Task Design

The paper designs training tasks for RL across three domains, each with distinct verification requirements. The unifying principle is that all tasks must be automatically verifiable—there must be a reliable signal indicating whether the agent succeeded.

GUI-Browsing Tasks: Multi-Condition Obfuscation and Multi-Hop Synthesis

GUI-Browsing tasks are "conceptually similar to deep research tasks, except that agents must satisfy the information-seeking requirements solely through analyzing screenshots, without access to search APIs." The agent must answer complex questions by navigating websites and reading content from screenshots—no text-based search or API access is available.

Approach 1: Multi-Condition Obfuscation. The paper describes a pipeline that transforms straightforward facts into complex, indirect questions:

Extract core entities and their attribute features from authoritative knowledge sources (e.g., Wikipedia).
Score each feature for distinctiveness using an LLM. Highly revealing attributes (those that would make the answer trivially identifiable) are removed.
The remaining attributes are rewritten by the LLM to "increase abstraction and reduce specificity." This produces questions defined by multiple indirect constraints, requiring the model to combine and reason over blurred signals.

Example from the paper: from a Wikipedia page about a music group, the pipeline generates the obfuscated question:

"Discovered by a representative from the Music And Cabaret talent agency, this group had a founding lineup—initially under another name—that included members from Dreghorn and Irvine, plus a lead guitarist and drummer. The lead vocalist joined after being recommended by a founding member who saw them perform with a Kilmaurs-based band, and their lead guitarist left to form another ensemble before late 1975. Which record label did this group sign with?"

This question requires the agent to navigate to the correct Wikipedia page, extract multiple pieces of information (talent agency, founding lineup locations, vocalist origin, guitarist departure timeline), and synthesize them to identify the music group before answering the target question (record label). The obfuscation transforms what would be a simple lookup ("What record label did Band X sign with?") into a multi-step reasoning problem.

Why obfuscation works: by removing the most distinctive features, the pipeline forces the model to reason about combinations of less-specific attributes. This mirrors real-world information-seeking tasks where the user doesn't know the exact name of what they're looking for—they have partial, indirect clues.

Approach 2: Multi-Hop Chain-Like Conditions. This approach constructs questions that require sequentially answering sub-questions:

Begin from an entity's webpage and follow its hyperlinks to identify structurally related entities.
For each linked entity, extract and obfuscate descriptive features, creating tasks where the linked entity becomes the answer.
Treat the linked entity's page as the new starting point and repeat recursively, generating tasks at progressively deeper levels.
At each step, the answer from the previous hop is embedded within the new question, forming a coherent reasoning chain.
Finally, the atomic steps are semantically integrated into a single multi-hop question.

What this produces: a question like "What is the capital of the country where the author of Book X was born?" where Book X is identified through obfuscated features, its author must be identified, the author's birthplace country must be determined, and finally the capital of that country must be reported. Each hop depends on successfully answering the previous one.

Difficulty filtering. The paper notes that synthesized data is filtered by "discarding instances that can be trivially solved using prior knowledge or a single-turn search, keeping only truly challenging and verifiable tasks for training." This filtering is crucial because if the model can answer questions from its pre-training knowledge without any GUI interaction, the RL signal becomes meaningless—the model gets rewarded for correct answers that required no environment interaction, which doesn't teach GUI skills.

GUI-General Tasks: Offline Synthesis from Web Functionality

GUI-General tasks focus on broader web manipulation—filling forms, navigating multi-page workflows, interacting with dynamic web applications. The synthesis pipeline works as follows:

Website curation: candidate websites are collected from public collections and filtered to remove inaccessible pages, login-gated services, and "trivial categories such as static information pages or casual games." The resulting set covers 690 websites across diverse domains.
Function extraction: VLMs (Vision-Language Models) are employed to identify and extract each website's core functionalities. For example, on an e-commerce site, functions might include "search for products," "filter by price," "add to cart," "checkout."
Task synthesis: for each selected website, tasks are composed at the single-page level through a structured process: removing overly simple functions, composing executable instructions, merging prerequisite sub-tasks, and refining task descriptions for "clarity, objectivity, and verifiability."

Why offline synthesis: unlike GUI-Browsing tasks (which use knowledge sources like Wikipedia to construct questions), GUI-General tasks don't have a natural knowledge base to draw from. The synthesis must be based on what the website actually does, which requires analyzing the website's functionality. VLMs serve as the bridge between raw websites and structured task descriptions.

Gameplay Tasks: Real and Synthetic Games with Verification Scripts

Game RL tasks come from two complementary sources:

Publicly available HTML5/WebGL mini-games that run directly in the browser sandbox. These are real games with authentic gameplay mechanics.
LLM-synthesized games: "we synthesize new games using LLMs, which generate lightweight code implementations that preserve core gameplay mechanics while exposing explicit state interfaces." The synthesized games are designed to have clean, programmatic access to state variables that would otherwise be hidden in the game's internal logic.

For both sources, the paper creates "concise JavaScript verification scripts that query runtime variables (e.g., score, level index, remaining lives) and provide time-aligned state attributes." These scripts establish a reliable mapping from agent actions to environment transitions and reward signals—essentially, they make the game state programmatically observable.

Standardized schema: all interaction records are consolidated into a "unified JSON schema containing scalar rewards, termination flags, and metadata (e.g., game version and verification checksums)." This standardization is what enables the same RL training pipeline to handle many different games without game-specific code in the training loop.

Multi-Turn RL: Reward Design

The paper categorizes reward signals based on whether correctness can be deterministically verified—a fundamental distinction that determines the reliability and scalability of the reward signal.

Deterministically Verifiable Tasks

For tasks where a programmatic verifier exists, the paper uses direct correctness signals:

Games: "directly compute binary correctness signals as rewards" by querying the JavaScript verification scripts described in Section 2.5.1. The game's runtime variables (score, level, lives) provide objective, instantaneous feedback. There is no ambiguity about whether the agent is doing well—the game state tells you directly.
GUI-Browsing: since these tasks have ground-truth answers, the paper uses LLM-as-Judge to compare the agent's prediction against the reference answer. The LLM judge evaluates whether the agent's answer is semantically equivalent to the ground truth, handling variations in phrasing that would break exact string matching.

Why these are "deterministically verifiable": in games, the reward comes from the game engine itself, which is a deterministic function of the game state. In GUI-Browsing, the ground-truth answer is known from the construction pipeline (the question was synthesized from known facts), so an LLM judge can reliably compare answers. These reward signals are objective and consistent—two runs with the same outcome will always get the same reward.

Non-Verifiable Tasks: Generative Outcome Reward Model

For open-ended tasks like GUI-General where "neither formal verifiers nor reference answers exist," the paper takes a fundamentally different approach: it trains UI-TARS-2 itself to serve as a reward model.

The ORM architecture. The Outcome Reward Model (ORM) is a version of UI-TARS-2 specifically enhanced for reward prediction. It takes as input the full text history of the agent's trajectory together with the last five screenshots (fitted within the context window) and outputs a scalar score indicating task success. The paper states: "To achieve this, we specifically enhance UI-TARS-2's capability of ORM through targeted data annotation and single-turn RL, ensuring that its reward predictions are accurate, consistent, and robust for downstream multi-turn RL."

What "targeted data annotation" means: human annotators review agent trajectories and label whether each trajectory successfully completed the task. These human labels become training data for the ORM. "Single-turn RL" then optimizes the ORM's scoring accuracy—essentially, the reward model itself is fine-tuned through RL to better predict human judgments of task success.

Why use the last five screenshots: GUI-General tasks can involve dozens of steps. Including all screenshots would exceed context limits. The last five provide sufficient visual context to judge whether the final state matches the task requirements (e.g., "did the form get submitted successfully?"), while the full text history provides the sequential context of what actions were taken.

Viability of VLM-as-Verifier

The paper addresses a critical concern: can a learned reward model provide reliable training signals without being exploited by the policy? This is the reward hacking problem—the policy might learn to generate trajectories that score highly under the ORM without actually completing tasks correctly.

The paper reports that "manual inspection of rewards did not reveal any substantial signs of reward hacking" and provides quantitative evidence: "On this benchmark [an in-house ORM evaluation set containing 300 human-annotated GUI agent traces], UI-TARS-2 achieved an F1 score of 83.8 as the generative ORM in the binary classification setting, indicating reasonably strong robustness."

Analysis of misclassifications. The paper notes that "the current ORM still exhibits a relatively high false positive rate"—meaning the ORM sometimes says a trajectory was successful when it actually wasn't. This should, in theory, be catastrophic for RL because it rewards the policy for producing incorrect behavior. But the paper offers an explanation for why it still works:

"We attribute this to the fact that, even in a case where the final task outcome is incorrect, the agent might also execute many correct intermediate steps. In false positive cases, the model still receives appropriate rewards for these correct steps, and these positive contributions outweigh the erroneous rewards given to incorrect actions."

What this means operationally: the ORM's errors are not random—they occur on trajectories where most steps were correct but the final outcome failed (e.g., the agent navigated correctly to a form, filled it out correctly, but clicked "Cancel" instead of "Submit"). In these cases, the ORM's false positive reward is partially correct—most of the actions in the trajectory were good—and the incorrect final action gets a reward it shouldn't, but the gradient from that single step is outweighed by the correct gradients from all the earlier steps.

Why this finding matters beyond this paper: it suggests that learned reward models for interactive tasks may be more robust than theory would predict, because (1) task success in GUI domains is often more concretely definable than in open-ended text generation, making reward modeling easier, and (2) the sequential nature of trajectories means that partial-credit rewards can compensate for endpoint misclassifications.

Multi-Turn RL: Asynchronous Rollout Infrastructure

The paper identifies a critical bottleneck in standard RL training loops: "Traditional batch-based rollout approaches often become bottlenecked by complex long-tail problems, reducing training efficiency and creating off-policy distribution drift." This happens because standard RL waits for all rollouts in a batch to complete before running a training update. If some trajectories are very long (hundreds of steps for a complex GUI task) while others are short, the fast trajectories sit idle waiting for the slow ones, and the policy that generated the fast trajectories becomes stale.

The solution is an asynchronous infrastructure with three key features (illustrated in Figure 6):

Asynchronous Inference with Server-Based Rollout

The paper adopts a "fully asynchronous inference system utilizing online server-mode processing." The policy model runs as a server that accepts inference requests and returns actions. The agent reasoning framework (which manages the ReAct loop, memory, and environment interaction) is decoupled from the policy inference server.

Why this decoupling matters: the agent framework can be written in any language and handle the complexity of environment interaction (screenshot capture, action execution, state management) without being constrained by the inference server's architecture. The inference server does one thing—run the model forward to produce (thought, action) pairs—and does it efficiently through asynchronous batching. This separation also means new agent interaction handlers (for new environments or task types) can be developed independently of the inference infrastructure.

Streaming Training with Partially-Filled Rollout Pools

Instead of waiting for a complete batch of trajectories to finish, the system maintains a dynamic rollout pool where training updates commence once completed traces reach a minimum batch size threshold. The paper explicitly analogizes this to Kimi-Researcher:

"This feature is conceptually similar to Kimi-Researcher."

How this works: as agents complete tasks (either successfully or by reaching a maximum step limit), their trajectories are added to the pool. When enough trajectories have accumulated to form a training batch, a policy update is triggered. Trajectories that haven't finished yet remain in progress and will contribute to future batches. This eliminates the bottleneck where one extremely long trajectory delays training for all the faster ones.

Why this is critical for GUI RL: GUI tasks have inherently variable lengths. A task like "close the browser" might take 2 steps; a task like "plan a multi-city itinerary and book flights" might take 50+ steps. Without streaming training, the 2-step tasks would be forced to wait for the 50-step tasks, wasting compute and causing the policy used during those 2-step rollouts to be outdated by the time training occurs.

Stateful Agent Environment Integration

The paper implements "stateful agent environments that preserve execution states across multiple tool invocations, enabling continuous state transitions and maintaining context throughout extended problem-solving sessions."

What this means concretely: when the agent takes an action (e.g., clicks a button), the VM's state changes (the button's associated page loads). The next time the agent needs to act, it connects to the same VM instance with the same state, not a fresh VM. This is how real computer use works—applications stay open, files persist, and browser tabs remain loaded. The stateful integration ensures that the RL training environment reflects these real-world dynamics.

Why this is technically challenging: maintaining thousands of VM instances with persistent state across multi-hour training runs requires careful resource management. The lease-based lifecycle mechanism from Section 2.2 (automatically releasing VMs after task completion and reclaiming overdue sessions) is what makes this feasible at scale.

Multi-Turn RL: Training Algorithm (Enhanced PPO)

The paper uses Proximal Policy Optimization (PPO) as the base RL algorithm and enhances it with five specific modifications drawn from recent advances in reasoning RL. The objective function is:

$\mathcal{J}_{\text{PPO}}(\theta) = \mathbb{E}_{(q,a) \sim \mathcal{D}, o_{\leq t} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)}\left[\min\left(\frac{\pi_{\theta}(o_t \mid q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t \mid q, o_{<t})}\hat{A}_t, \text{clip}\left(\frac{\pi_{\theta}(o_t \mid q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t \mid q, o_{<t})}, 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}\right)\hat{A}_t\right)\right]$

where π_θ is the current policy (the model being trained), π_{θ_old} is the previous policy (used to generate the trajectories in the batch), o_t is the output at timestep t (the concatenation of reasoning trace t_t and action a_t), q is the task instruction, Â_t is the estimated advantage at timestep t (how much better this action was than expected), ε_low is the lower clipping bound, and ε_high is the upper clipping bound.

What it computes: the standard PPO clipped surrogate objective. For each (state, action) pair in the batch, it computes the ratio of the new policy's probability to the old policy's probability. If this ratio would move too far from 1 (controlled by the clipping bounds ε_low and ε_high), the objective is clipped to prevent excessively large updates that could destabilize training. The min operation chooses between the unclipped and clipped objectives—effectively, if the advantage is positive (the action was good), the objective encourages increasing its probability but not beyond the upper clip; if the advantage is negative, it encourages decreasing probability but not beyond the lower clip.

Why this form: PPO's clipped objective addresses the fundamental instability of policy gradient methods: without clipping, a single batch with high-variance advantage estimates can cause the policy to change drastically, potentially collapsing performance. The clipping creates a "trust region" around the old policy, ensuring that each update is conservative. This is especially important in GUI tasks where the state distribution can shift dramatically between updates.

Enhancement 1: Reward Shaping

The paper states: "To promote more strategic agent behaviors, the reward signal is mainly determined based on the correctness of the final outcome. In certain scenarios, we employ format rewards and length penalties to discourage premature termination or infinite continuation."

What this means: the primary reward is sparse and terminal—the agent gets a positive reward only if it completes the task successfully. However, auxiliary rewards are added to shape behavior:

Format rewards: small positive rewards for producing well-formed outputs (valid action syntax, coherent reasoning traces), encouraging the agent to maintain proper output structure even when exploring.
Length penalties: negative rewards proportional to trajectory length, discouraging the agent from taking unnecessary steps or entering infinite loops. This addresses the interaction scaling problem noted in Section 3.3—without length penalties, agents "often learn to exploit larger budgets, prolonging trajectories before convergence."

Why reward shaping is necessary: sparse terminal rewards alone create a severe credit assignment problem. If the agent takes 50 correct steps and 1 wrong step, and the reward is 0, all 51 steps get penalized equally. Reward shaping provides intermediate signals that help the agent distinguish between "correct step that didn't immediately lead to success" and "incorrect step that prevented success."

Enhancement 2: Decoupled GAE

The paper adopts Decoupled Generalized Advantage Estimation (Decoupled-GAE) from VC-PPO (Yuan et al., 2025):

"To address the challenge of value estimation bias over long sequences, we employ the Decoupled Generalized Advantage Estimation (Decoupled-GAE), allowing the computation of advantage for the policy and value function to use different coefficients. Specifically, we set λ_policy and λ_critic to be different."

What GAE is and why decoupling helps: Generalized Advantage Estimation computes the advantage Â_t as an exponentially weighted sum of temporal difference errors, controlled by a parameter λ (lambda) that trades off bias and variance:

λ = 0: uses only the immediate TD error (low variance, high bias because it ignores future rewards)
λ = 1: uses the full Monte Carlo return (no bias, high variance because it sums many noisy terms)

Standard PPO uses the same λ for both computing advantages (used in the policy update) and training the value function (the critic). Decoupled-GAE allows these to differ. The paper's finding that "value estimates of PPO-trained models were often negatively correlated with the obtained rewards" (Section 3.3) suggests that the value function was poorly calibrated for long sequences. Using a different λ for the critic (likely λ_critic = 1.0, as mentioned in the Value Pretraining section) provides more stable value targets for critic training.

Why this matters for GUI tasks: GUI trajectories are long and the relationship between individual actions and final outcomes is complex. A value function that is poorly calibrated will produce noisy advantages, which cause the policy to update in random directions. Decoupled-GAE stabilizes the critic separately from the policy, ensuring that the advantage estimates used for policy updates are based on reliable value predictions.

Enhancement 3: Length-Adaptive GAE

The paper adopts Length-Adaptive GAE from VAPO (Yue et al., 2025):

"To mitigate the issue of inconsistent advantage estimation for sequences of varying lengths, we employ Length-Adaptive Generalized Advantage Estimation (Length-Adaptive GAE) technique, adjusting the GAE parameter λ_policy based on the sequence length. Specifically, we set α = 0.05 in length-adaptive formula λ_policy = 1 - 1/(α l) to control the overall bias-variance trade-off."

where α is a hyperparameter controlling sensitivity to length, and l is the sequence length.

What this formula computes: as sequence length l increases, λ_policy increases toward 1. For short sequences, λ_policy is lower (closer to 0), meaning advantages rely more on immediate TD errors and less on long-term returns—appropriate because short sequences have fewer steps to accumulate variance. For long sequences, λ_policy is higher (closer to 1), meaning advantages incorporate more of the Monte Carlo return—appropriate because long sequences need to propagate credit across many steps.

Example: with α = 0.05:

For l = 10 steps: λ_policy = 1 - 1/(0.05 × 10) = 1 - 1/0.5 = 1 - 2 = -1 — this would be clipped to some minimum, indicating the formula likely requires λ_policy ≥ 0. The paper doesn't specify clipping, but the intent is clear: short sequences use lower λ.
For l = 100 steps: λ_policy = 1 - 1/(0.05 × 100) = 1 - 1/5 = 0.8 — long sequences use high λ to propagate credit across the full trajectory.

Why fixed λ fails: if λ is constant, short trajectories get too much variance (from including noisy long-term returns) or long trajectories get too much bias (from truncating credit assignment). Length-adaptive GAE automatically adjusts based on the actual trajectory length in each batch.

Enhancement 4: Value Pretraining

The paper identifies a specific failure mode: "In our preliminary experimental exploration, we observed that the value estimates of PPO-trained models were often negatively correlated with the obtained rewards." This means the critic—the component that predicts future rewards—was actively harmful to training because it was telling the policy that good actions were bad and vice versa.

The solution is Value Pretraining from VAPO:

"Responses are sampled continuously from a fixed policy (e.g., π_sft), and the value model is updated using GAE with λ = 1.0 (equivalent to Monte Carlo return), providing stable and reliable optimization. Training continues until crucial metrics such as value loss and explained variance reach sufficiently low levels, indicating effective convergence. The resulting value model checkpoint is then used as the initialization for subsequent experiments, ensuring more accurate and calibrated value estimation from the outset."

What this means operationally:

Before starting PPO, fix the policy at the SFT checkpoint (don't update it).
Run many episodes with this fixed policy, collecting trajectories and their final rewards.
Train only the value function (critic) to predict the Monte Carlo return from each state—using λ = 1.0, which means the target is the actual sum of discounted future rewards, not a bootstrapped estimate.
Train until the value function converges (value loss is low, explained variance is high).
Use this pre-trained value function as the initial critic for the full PPO training.

Why this prevents negative correlation: starting PPO with a randomly initialized critic is risky because the critic's initial predictions are essentially random. The policy updates based on advantages computed from these random predictions, which can push the policy in counterproductive directions before the critic has a chance to learn. Value pretraining ensures the critic starts from a reasonable baseline—the SFT policy's expected returns—so that early PPO updates are informed by approximately correct advantage estimates.

Figure 10(b) evidence: the paper shows that value pretraining "enhances the value model's ability to guide policy learning, leading to consistently higher rewards throughout training." This is a causal claim supported by the training curve comparison.

Enhancement 5: Clip-Higher

The paper adopts decoupled clipping bounds from DAPO (Yu et al., 2025):

"To further promote exploration, we decouple the PPO clipping parameters... introducing distinct lower (ε_low) and upper (ε_high) clipping bounds. Increasing ε_high affords greater flexibility for raising the likelihood of low-probability actions, thus enlarging the exploration space. Conversely, ε_low is maintained at a low value to avoid prematurely eliminating tokens, which would risk collapsing the diversity of potential outputs."

What this means: standard PPO uses a single clipping parameter ε (typically 0.2), meaning the policy can increase a good action's probability by at most a factor of 1 + ε and decrease a bad action's probability by at most a factor of 1 - ε. Decoupled bounds allow asymmetric treatment:

ε_high is set larger than ε_low, meaning the policy can more aggressively increase probabilities of good actions than it decreases probabilities of bad ones.
This asymmetry encourages the policy to explore (try new actions that currently have low probability) without prematurely eliminating actions (which would collapse the action space and prevent future exploration).

Why this matters for GUI exploration: GUI action spaces are vast—the agent can click anywhere on the screen, type any text, or invoke any SDK function. Early in training, most actions have very low probability. If ε_low is too aggressive, the first few PPO updates might drive the probability of all but a few actions to near-zero, collapsing the policy to a deterministic strategy that never recovers. Clip-Higher mitigates this by being conservative about decreasing probabilities while being permissive about increasing them.

PPO vs. GRPO

The paper explicitly compares PPO against GRPO (Group Relative Policy Optimization, from Shao et al., 2024), which has been effective for reasoning tasks. The finding is negative for GRPO:

"In our preliminary evaluation, we find that PPO consistently outperforms GRPO by a clear margin. As depicted in Figure 12, PPO maintains higher rewards with lower volatility throughout training."

Why GRPO might underperform: GRPO is a simpler algorithm that doesn't use a learned value function—it estimates advantages by comparing rewards within a group of trajectories for the same prompt. This works well for reasoning tasks where multiple completions of the same prompt can be directly compared. But in GUI tasks, each trajectory is highly dependent on environment interactions—two trajectories for the same task might visit completely different states because the agent clicked different buttons—making within-group comparison noisier than value-function-based advantage estimation. PPO's value function can learn to account for the difficulty of different states, providing more reliable advantage estimates.

Parameter Interpolation: Merging Specialized Agents

The paper faces a practical challenge: how to create a single agent that performs well across multiple domains (GUI-Browsing, GUI-General, Game, GUI-SDK) given that joint RL across all domains is "unstable and computationally prohibitive." The solution is elegantly simple: train specialized agents independently for each domain, then merge them via linear interpolation of their parameters.

The Interpolation Formula

$\theta^{(\text{merge})} = \sum_{k \in \{\text{GUI-Browsing}, \text{GUI-General}, \text{Game}, \text{GUI-SDK}, \ldots\}} \alpha_k \cdot \theta^{(k)}, \quad \text{s.t.} \quad \sum_k \alpha_k = 1, \ \alpha_k \geq 0$

where θ^(merge) are the parameters of the merged model, θ^(k) are the parameters of the model specialized for domain k (trained via independent RL runs from the same SFT initialization), and α_k are the mixture weights (non-negative and summing to 1) that determine how much each specialized model contributes to the merged model.

What it computes: a weighted average of the parameters of multiple specialized models. Each specialized model was trained on a different domain (GUI-Browsing, GUI-General, Games, GUI-SDK). The merged model's parameters are simply the weighted sum of the specialized models' parameters—a purely arithmetic operation requiring no additional training.

Why this works: the paper invokes the concept of linear mode connectivity (Qin et al., 2022): "models fine-tuned from the same pre-trained checkpoint remain approximately linearly mode-connected in parameter space." This means that if you take two models fine-tuned from the same base model, the straight-line path between their parameters passes through models that also perform well. Parameter interpolation is essentially taking a point on this line (the weighted average), which inherits capabilities from both endpoints.

Practical benefits. The paper reports that "this interpolation strategy preserves the performance of each specialized vertical while enabling strong cross-domain generalization. On composite tasks requiring skills from multiple domains, the merged model performs almost comparably to the best specialized model in each relevant domain, without additional optimization cost."

Why not joint training? Joint RL across all domains would require the agent to simultaneously learn GUI interaction, game playing, and SDK usage—action spaces that differ substantially, task horizons that vary widely, and rollout environments that have different stability characteristics. The optimization would struggle to balance these competing demands, and training throughput would be limited by the slowest domain's rollout speed. Parameter interpolation sidesteps these problems entirely by training independently and merging post-hoc.

Hybrid RL as an Alternative

The paper also explores an alternative to parameter interpolation: hybrid reinforcement learning where the same model is trained on both GUI-only and GUI-SDK interfaces simultaneously. The findings (Figure 15) reveal an interesting transfer pattern:

"Even though the training data for each interface was effectively halved compared to the single-interface baselines, the hybrid model outperformed the GUI-only baseline when evaluated on pure GUI tasks."

This indicates that "knowledge acquired through the more capable GUI-SDK interface transfers effectively to GUI-only interaction, boosting competence even in the restricted setting." The SDK-augmented trajectories teach the model strategies that generalize to GUI-only settings, even though the model doesn't have SDK access during pure GUI evaluation.

The paper also found that "employing a shared value model improved training stability and reward estimation: by learning jointly from trajectories across both interfaces, the value model generalized to a broader range of patterns, yielding higher explained variance than interface-specific baselines" (Figure 15(e)).

Tradeoff between interpolation and hybrid RL: "Compared to parameter interpolation, which merges specialized agents without additional optimization, hybrid training enables more direct cross-interface knowledge transfer but incurs higher training cost." The paper treats these as complementary strategies rather than competitors.

Infrastructure and Hyperparameters Summary

The paper provides several concrete configuration details that are essential for reproducibility:

Base model: Seed-thinking-1.6, with a 532M-parameter vision encoder and a Mixture-of-Experts LLM (23B active parameters, 230B total parameters).
RL algorithm: PPO with the five enhancements described above (reward shaping, decoupled GAE, length-adaptive GAE, value pretraining, clip-higher).
Length-adaptive GAE parameter: α = 0.05 in the formula λ_policy = 1 - 1/(α l).
Training stages order: CT → SFT → RL, with SFT and RL performed more frequently than CT.
Quantization for deployment: W4A8 quantization (4-bit weights, 8-bit activations), which increases the token generation rate from 29.6 to 47 tokens/s and reduces average end-to-end latency per interaction round from 4.0 to 2.5 seconds, with a modest accuracy decrease from 47.5 to 44.4 on OSWorld.

4. Key Insights and Innovations

Innovation 1: The Data Flywheel as a Unifying Training Paradigm That Makes Agent Data Self-Reinforcing Rather Than Exhaustible

The dominant assumption in agent training prior to this work was that data and model improvement are separate, sequential processes: first, collect or synthesize a dataset; then, train a model on it; then, evaluate. This creates a fundamental ceiling—once the dataset is exhausted, the model stops improving. The Data Flywheel (Section 2.3) breaks this assumption by making data generation and model training co-evolutionary. The model at iteration t generates trajectories, a validation function V(s) → {0,1} splits them into high-quality (routed to SFT) and low-quality (routed to CT) streams, and the expanded datasets train the model at iteration t+1, which in turn generates better trajectories because P(V(s)=1 | t) > P(V(s)=1 | t-1). This is not merely a data augmentation trick—it is a structural reconceptualization of the training process from a linear pipeline into a closed-loop system where every generated sample, successful or failed, is recycled at the appropriate training stage.

What makes this intellectually distinctive is the routing logic, not the existence of iterative training loops (which are common in self-play and online RL). Prior iterative approaches typically either discard low-quality samples or use them indiscriminately for the same training stage that receives high-quality ones. The flywheel's key insight—that failed trajectories contain useful environmental knowledge but should not contaminate the supervised imitation signal—is a diagnostic move that recognizes the asymmetrical information content of success vs. failure. Successful trajectories teach what to do; failed trajectories teach about the environment and what not to do. By segregating these into different training stages (SFT for imitation, CT for broad knowledge), the flywheel extracts value from every generated sample without the noise-vs-signal tradeoff that would plague a single-stage approach. This is significant beyond raw performance because it addresses the core scalability limitation that has constrained agent development: the cost of collecting high-quality human-annotated trajectories. The flywheel demonstrates that once a model reaches a baseline competence level, it can bootstrap its own improvement using only automatic verifiers and environment feedback.

Comparison to prior work: standard SFT pipelines (UI-TARS, OS-Atlas) are open-loop—they train on a fixed dataset and stop. Self-play RL approaches (AlphaGo, Atari DQN) use model-generated data for online improvement but are designed for environments with dense, ground-truth rewards, which GUI tasks lack. Rejection sampling fine-tuning (RFT) generates data from the current model but typically only keeps successful samples and discards failures, wasting information. The Data Flywheel is a fundamental shift in that it closes the loop across multiple training stages (CT, SFT, RL) with explicit quality-based routing, treating data generation as a resource allocation problem rather than a filtering problem. The evidence that the flywheel works is indirect but compelling: the monotonically increasing training rewards in Figure 7 and the OOD generalization gains (RL on browser tasks transfers to OSWorld with ~10.5% improvement and AndroidWorld with ~8.7%) demonstrate that iterative cycles produce compounding benefits.

Innovation 2: Stabilized Multi-Turn RL for GUI Agents as a Distinct Optimization Regime Requiring Its Own Recipe

The field's understanding of LLM reinforcement learning has been shaped primarily by two regimes: reasoning RL (DeepSeek-R1, where the model generates text completions and receives sparse terminal rewards for correctness) and RLHF (where a learned reward model scores completions for preference optimization). The paper's central diagnostic contribution is demonstrating that GUI agent RL constitutes a third regime with fundamentally different optimization dynamics that are not addressed by techniques developed for the other two. The evidence is not just in the performance numbers but in the training dynamics analyses that reveal behaviors specific to interactive, visually grounded environments.

First, the entropy dynamics (Figure 8) are opposite to those observed in reasoning RL. Reasoning RL typically shows monotonic entropy reduction as the model converges to deterministic exploitation of successful reasoning patterns. GUI agent RL shows rising entropy during training, reflecting that "the model maintains or even expands its exploration space as training progresses, enabling it to acquire new interaction patterns rather than collapsing prematurely into narrow exploitation." This is a non-obvious finding with implications for algorithm design: techniques that accelerate convergence in reasoning RL by suppressing exploration (e.g., aggressive entropy regularization) would likely harm GUI agent training.

Second, the GRPO vs. PPO comparison (Figure 12) reveals that GRPO—which has been effective across "a wide range of reasoning tasks"—underperforms PPO by a clear margin in GUI agent settings. The paper's implicit explanation is that GRPO's group-based advantage estimation (comparing trajectories within a group for the same prompt) breaks down when different trajectories of the same task diverge into completely different environment states. In reasoning, all completions of a math problem remain in the same "state space" (the text domain); in GUI tasks, clicking different buttons leads to different screens, making within-group comparison unreliable. This is a diagnostic finding that identifies a boundary condition for GRPO's applicability—it works when the environment is static, fails when actions change the state distribution.

Third, the value function diagnostic—the finding that "value estimates of PPO-trained models were often negatively correlated with the obtained rewards"—is a concrete identification of a failure mode specific to long-horizon interactive tasks. Standard PPO assumes the value function can learn to predict returns from the same data that trains the policy, but in GUI tasks where episodes are long and successes are sparse, this assumption breaks. The solution (Value Pretraining, Decoupled GAE, Length-Adaptive GAE) is not algorithmically novel individually—each component is drawn from VAPO, VC-PPO, and DAPO—but their combination and validation as a recipe for the GUI agent regime is a genuine contribution. The paper demonstrates that these techniques, developed in the context of reasoning RL, are necessary but not individually sufficient for GUI agent RL, and that the specific combination matters.

Comparison to prior work: prior GUI agent RL efforts (ARPO, Mobile-GUI-R1) applied RL to GUI tasks but did not systematically analyze or address the optimization stability challenges. ARPO focuses on experience replay, Mobile-GUI-R1 on a specific mobile setting. Neither provides the training dynamics diagnostics (entropy curves, value function analysis, GRPO comparison) that establish why GUI agent RL is harder than reasoning RL and what recipe addresses the failure modes. This paper's contribution is establishing that the "naive PPO" that works for reasoning tasks is insufficient for GUI agents, and providing a empirically validated recipe for stabilization.

Innovation 3: the Demonstration That GUI-Only Agent Training Transfers to Heterogeneous System-Level Tasks Through a Unified Action Space

A dominant assumption in prior work (both pipeline-based systems and end-to-end models like CogAgent and OS-Atlas) was that GUI agents and system-level agents (terminal, file system, software development) are separate categories requiring separate architectures or at least separate training pipelines. The paper challenges this with a unified action space that combines GUI primitives (click, type, scroll) with pre-defined SDK functions (terminal commands, MCP tool invocations), and demonstrates that training on this unified space produces transfer in both directions.

The most striking evidence is the BrowseComp results (Table 1): when restricted to GUI-only operation, UI-TARS-2 achieves 32.1% on BrowseComp-zh and 7.0% on BrowseComp-en; with GUI-SDK augmentation, these jump to 50.5% and 29.6% respectively. These are not incremental gains—they are 1.6× and 4.2× improvements from adding system-level tool access. More importantly, the hybrid RL experiments (Figure 15) show that "the hybrid model outperformed the GUI-only baseline when evaluated on pure GUI tasks," even though the hybrid model saw half as much GUI-only training data. This is evidence of asymmetric positive transfer: the more capable GUI-SDK interface imparts skills (likely planning, tool selection, and systematic problem decomposition) that improve performance even when tools are unavailable.

This finding is significant beyond the specific numbers because it reframes the relationship between GUI interaction and system-level control. Rather than being separate modalities to be bridged with custom integration code, they are better understood as different points on a spectrum of interaction granularity—GUI actions are fine-grained, direct manipulation; terminal commands are coarse-grained, compositional operations. A model trained on both learns not just each modality individually but the meta-skill of selecting the appropriate granularity for each sub-task. This has implications for agent architecture: future systems should not treat GUI and CLI as separate "tools" but as a unified action vocabulary where the model learns to switch based on task demands.

Comparison to prior work: workflow-based systems (OWL, Alita) orchestrate GUI and CLI tools through scripted procedures, but the integration is hard-coded by system designers. Models like WebSailor and SimpleDeepSearcher train on tool-augmented tasks but in text-only environments without visual grounding. The distinctive contribution here is demonstrating that visual GUI training and tool-use training reinforce each other when unified in a single policy, and that the transfer is not merely additive but multiplicative (the hybrid model is better than either specialized model on each individual domain). This is a conceptual reframing—from "GUI agent that can also use tools" to "unified computer-use agent for which GUI and tools are different action vocabularies in the same policy."

Innovation 4: Parameter Interpolation as a Cost-Effective Alternative to Joint Multi-Domain RL, Validated at Industrial Scale

Joint multi-domain RL—training a single policy simultaneously on GUI tasks, games, terminal operations, and SDK usage—is theoretically attractive because it should produce the most general agent. The paper's pragmatic insight is that joint training is not just expensive but may be counterproductive when domains differ substantially in action/state spaces, task horizons, and rollout stability. The solution—training domain-specialized agents independently from a shared SFT initialization and merging them via linear parameter interpolation (θ(merge) = Σ α_k · θ(k))—is a technique known from model merging literature (Qin et al., 2022) but had not been validated at the scale and diversity of domains that UI-TARS-2 spans.

What makes this an innovation rather than merely an engineering convenience is the empirical validation that interpolation preserves specialized performance while enabling cross-domain generalization. The paper reports that "on composite tasks requiring skills from multiple domains, the merged model performs almost comparably to the best specialized model in each relevant domain." This is a stronger claim than standard model merging results, which typically show some degradation relative to specialized models. The implication is that the RL training process produces parameter spaces that are sufficiently linearly mode-connected that simple averaging works as a consolidation strategy—a finding that, if general, would substantially reduce the computational barrier to building multi-domain agents.

The comparison to hybrid RL (Section 3.3, Figure 15) adds nuance: hybrid training produces stronger individual-domain performance through cross-interface knowledge transfer but at higher training cost, while interpolation is cheaper but may leave some transfer gains on the table. The paper treats these as complementary rather than presenting one as strictly superior—a measured conclusion that acknowledges the tradeoff rather than overselling either approach.

Comparison to prior work: prior multi-domain agent systems either (1) trained jointly from the start (Gato, which handled diverse embodiments but used a single training process), (2) used separate models with a routing layer (pipeline architectures that switch between specialized agents), or (3) merged models but at much smaller scale and domain diversity. The innovation here is demonstrating that specialized RL runs + interpolation is a viable industrial-scale strategy for building general agents, validated across four substantially different domains (GUI-Browsing, GUI-General, Game, GUI-SDK). This is an architectural insight about how to structure agent training: rather than solving the hard optimization problem of joint multi-domain RL, solve the easier problems of domain-specialized RL and post-hoc merging, and accept the modest performance tradeoff. For practitioners, this means the path to a general agent may be training multiple specialized ones and averaging their weights—a recipe that is dramatically simpler than coordinating joint training across heterogeneous environments.

5. Experimental Analysis

Evaluation Methodology

Datasets. The paper evaluates on a multi-domain suite spanning GUI, game, and system-level benchmarks. GUI benchmarks include OSWorld (369 tasks across Ubuntu, Windows, and macOS with execution-based evaluation, from Xie et al., 2024), WindowsAgentArena (over 150 Windows-specific tasks adapted from the OSWorld framework, from Bonatti et al., 2024), AndroidWorld (116 tasks across 20 mobile apps in a live Android emulator with dynamic task variations via randomized parameters, from Rawles et al., 2024), Online-Mind2Web (300 realistic tasks across 136 websites, from Xue et al., 2025), BrowseComp-en (multi-hop browsing questions from Wei et al., 2025), BrowseComp-zh (Chinese-language browsing questions from Zhou et al., 2025), Terminal Bench (command-line proficiency benchmark, from The Terminal-Bench Team, 2025), and SWE-Bench Verified (repository-level software engineering tasks from Jimenez et al., 2023). Game benchmarks include an in-house 15 Games Collection and LMGame-Bench (six classic titles through a unified Gym-style interface, from Hu et al., 2025a). The paper does not report the exact number of tasks used from Terminal Bench, noting only that results are reported on "75 out of 80 tasks due to compatibility issues with our internal environment" (Table 1 footnote).
Base model. UI-TARS-2 is initialized from the pre-trained checkpoint of Seed-thinking-1.6 (ByteDance, 2025), a proprietary model. The architecture includes a 532M-parameter vision encoder paired with a Mixture-of-Experts LLM with 23B active parameters (230B total parameters). The paper states the model "leverages all of its post-training data" from Seed-thinking-1.6. This architecture choice reflects the requirements of GUI tasks: the vision encoder must process high-resolution screenshots, while the MoE LLM provides the reasoning capacity for multi-step planning without requiring all 230B parameters to be active for every token (reducing inference cost). The model was chosen as the starting point because Seed-thinking-1.6 provides general reasoning capabilities that the agent-specific training pipeline builds upon.
Metrics. The paper reports different metrics depending on the benchmark domain. For GUI benchmarks (OSWorld, WindowsAgentArena, AndroidWorld, Online-Mind2Web), the metric is task success rate (%) — the fraction of tasks where the agent's sequence of actions produced the correct final state, as determined by each benchmark's built-in evaluation scripts. For game benchmarks, the paper reports raw per-game scores as well as a mean normalized score computed by dividing each game score by the human reference score and averaging across titles, with human performance normalized to 100. For BrowseComp and Terminal Bench, the metric is accuracy (%) — the fraction of questions answered correctly. For SWE-Bench, the metric is the standard resolved rate (%) — the fraction of GitHub issues for which the agent's patch passes all tests. For LMGame-Bench, results are reported as mean ± standard deviation over three runs per game.
Baselines. The paper compares against a range of proprietary frontier models and prior UI-TARS versions. Proprietary baselines include Claude 4 Sonnet and Claude 4 Opus (Anthropic, 2025), Claude Computer Use (Anthropic, 2024), Claude 3.7 Sonnet with thinking (Anthropic, 2025), OpenAI o3 and o4-mini (OpenAI, 2025), OpenAI CUA-o3 (OpenAI's Computer-Using Agent based on o3), GPT-4.1 (OpenAI, 2025), GPT-4o (OpenAI, 2024), o1 (OpenAI, 2024), Gemini 2.5 Pro and Gemini 2.5 Flash (DeepMind, 2025), and Llama-4-Maverick-17B (Meta). Prior UI-TARS baselines include UI-TARS (Qin et al., 2025) and UI-TARS-1.5 (Seed, 2025). For game benchmarks, Claude 3.5 Sonnet is additionally included. The paper does not specify the decoding configuration or inference budget for baseline models, which is a notable gap — it is unclear whether baselines use greedy decoding, best-of-N, or other test-time strategies, making direct comparison to UI-TARS-2's RL-trained policy potentially unfair if UI-TARS-2 was allowed more inference compute.
Generation budget / compute accounting. The paper does not specify a unified generation budget or inference-step limit that is held constant across comparisons. For GUI benchmarks, the agent operates with a maximum number of interaction rounds (the paper does not state the exact limit, but Figure 11 shows evaluation with inference step budgets up to approximately 100–200 steps depending on the benchmark). For game benchmarks, the human reference uses a 100-step budget (Table 2 note: "human@100-step reference"), and UI-TARS-2 is evaluated at comparable budgets. The inference-time scaling analysis in Figure 11 varies the "allowed step budget" as the independent variable. The paper does not report total FLOPs or wall-clock time for evaluation, which would be necessary for a rigorous efficiency comparison against baselines. The quantization analysis reports latency (tokens/s and seconds/interaction round) but only for UI-TARS-2, not for baselines.
Cross-validation / statistical protocol. For LMGame-Bench, the paper reports mean ± standard deviation over three independent runs (Table 3), providing some measure of result stability. For GUI benchmarks, no uncertainty quantification is reported — all results in Table 1 are single-point estimates. The paper does not describe any cross-validation protocol for evaluation (the two-fold cross-validation mentioned in Section 3.2 applies to the compute-optimal policy selection in the Data Flywheel, not to benchmark evaluation). Statistical significance tests against baselines are not reported. This is a limitation: given that some performance gaps are modest (e.g., UI-TARS-2 at 47.5 vs. OpenAI CUA-o3 at 42.9 on OSWorld), it is unclear whether these differences are statistically reliable or within the range of evaluation noise from environment non-determinism.

Main Quantitative Results

GUI Benchmark Results

Table 1 presents the headline GUI results. UI-TARS-2 achieves 47.5% on OSWorld, 50.6% on WindowsAgentArena, 45.3% on Terminal Bench (with GUI-SDK), 68.7% on SWE-Bench Verified (with GUI-SDK), 73.3% on AndroidWorld, 88.2% on Online-Mind2Web, 32.1% on BrowseComp-zh (50.5% with GUI-SDK), and 7.0% on BrowseComp-en (29.6% with GUI-SDK).

Comparison against prior UI-TARS versions. UI-TARS-2 outperforms UI-TARS-1.5 on every benchmark where both are evaluated. On OSWorld, the improvement is from 42.5 to 47.5 (+5.0 percentage points). On WindowsAgentArena, from 42.1 to 50.6 (+8.5). On AndroidWorld, from 64.2 to 73.3 (+9.1). On Online-Mind2Web, from 75.8 to 88.2 (+12.4). These gains represent 12–16% relative improvement over the previous generation. The paper attributes these improvements to "the benefits of iterative training and reinforcement learning" (Section 3.2), consistent with the flywheel mechanism where each cycle generates better training data.

Comparison against proprietary baselines. The landscape is benchmark-dependent:

OSWorld: UI-TARS-2 at 47.5 leads all reported baselines. Claude 4 Opus reaches 43.2 (no result for Sonnet on this benchmark), OpenAI CUA-o3 reaches 42.9, and UI-TARS-1.5 reaches 42.5. The gap over the strongest proprietary baseline (Claude 4 Opus) is +4.3 percentage points. However, Claude 4 Opus results are only reported on a subset of benchmarks (OSWorld, AndroidWorld, SWE-Bench), limiting the completeness of comparison.
WindowsAgentArena: UI-TARS-2 at 50.6 substantially outperforms Claude 4 Sonnet at 39.2 (+11.4). No other baselines are reported on this benchmark. The large gap suggests that UI-TARS-2's training on diverse desktop environments (including Windows VMs) provides an advantage on Windows-specific tasks that general-purpose models like Claude may not have been explicitly optimized for.
AndroidWorld: UI-TARS-2 at 73.3 is competitive with the strongest baselines. Claude 4 Sonnet achieves 72.7 and Claude 4 Opus achieves 72.5 — differences of less than 1 percentage point, which are likely within evaluation noise. OpenAI CUA-o3 trails at 52.5. The near-tie with Claude 4 models on mobile tasks suggests that Android interaction may be approaching saturation for current-generation models, or that the benchmark's task difficulty distribution does not sufficiently differentiate among strong agents.
Online-Mind2Web: UI-TARS-2 at 88.2 substantially outperforms OpenAI CUA-o3 at 71.0 (+17.2). Claude results are not reported, so the strongest baseline comparison is against CUA-o3 alone. The large margin suggests that UI-TARS-2's RL training on browser-based tasks (GUI-Browsing and GUI-General) transfers effectively to this web navigation benchmark.
Terminal Bench: UI-TARS-2 with GUI-SDK at 45.3 is compared against OpenAI o3 at 30.2 (+15.1) and Claude 4 Opus at 43.2 (+2.1). The result on 75/80 tasks means that approximately 6% of tasks could not be evaluated due to "compatibility issues with our internal environment" — this introduces potential selection bias if the excluded tasks are systematically harder or easier.
SWE-Bench Verified: UI-TARS-2 with GUI-SDK at 68.7 is competitive with OpenAI o3 at 69.1 (−0.4) and substantially ahead of Claude 4 Opus at 43.2 (+25.5). The near-identical performance with o3 on software engineering tasks is notable given that o3 is a frontier reasoning model, suggesting that the GUI-SDK toolkit and RL training transfer effectively to code-related tasks.
BrowseComp-zh and BrowseComp-en: These benchmarks show the largest impact of GUI-SDK augmentation. On BrowseComp-zh, GUI-only achieves 32.1 while GUI-SDK achieves 50.5 (+18.4, a 1.6× improvement). On BrowseComp-en, GUI-only achieves 7.0 while GUI-SDK achieves 29.6 (+22.6, a 4.2× improvement). Claude 4 Opus reaches 37.4 on BrowseComp-zh and 18.8 on BrowseComp-en under what appears to be a different tool-use configuration; OpenAI o3 reaches 49.7 on BrowseComp-zh (close to UI-TARS-2's 50.5 with GUI-SDK). The comparative results suggest that UI-TARS-2 with SDK access approaches frontier model performance on these challenging browsing tasks, while GUI-only operation is severely limited — a finding that directly supports the paper's argument about the limitations of pure GUI interaction (Section 1, challenge 3).

OOD generalization claim. The paper reports that "RL improves accuracy from 83.7% (the SFT baseline in the final iteration) to 88.2% on Online-Mind2Web" and that "the RL-trained model transfers effectively to domains that were not the primary focus of training: for example, OSWorld improves by nearly 10.5% (from 43.0% to 47.5%) and AndroidWorld by over 8.7% (from 64.6% to 73.3%)" (Section 3.2). The specific numbers indicate that the transfer from browser-focused RL training to desktop OS and mobile tasks is substantial — OSWorld and AndroidWorld were not the primary RL training domains, yet the RL-trained policy outperforms the SFT baseline on both. This is consistent with the paper's claim that "task-specific RL induces broadly transferable skills" but requires careful interpretation: the improvement is measured relative to the SFT baseline (not a zero-shot baseline), and "not the primary focus" does not mean "not seen during training" — the SFT data likely includes OSWorld and AndroidWorld examples, and the improvement may reflect better generalization within a related distribution rather than true zero-shot transfer to novel domains.

Game Benchmark Results

15 Games Collection (Table 2). UI-TARS-2 achieves a mean normalized score of 59.77 across 15 games, compared to human performance normalized to 100. This represents roughly 60% of human-level performance on average. The score ranges from near-human on several titles (Shapes: 108.9, surpassing human; Infinity-Loop: 92.7; 2048: 91.0; Tiles-master: 82.7; Snake-solver: 76.5; Merge-and-double: 75.2) to near-zero on the most challenging games (Free-the-key: 0.70; Maze-Path-of-Light: 2.00; Energy: 3.30).

Comparison against baselines on the 15-game suite. Both proprietary baselines perform substantially worse:

OpenAI CUA: mean normalized score of 24.73 (UI-TARS-2 outperforms by +35.04, a factor of 2.4×).
Claude Computer Use: mean normalized score of 21.61 (UI-TARS-2 outperforms by +38.16, a factor of 2.8×).

The SFT-only UI-TARS-2 achieves a mean normalized score of 44.27, meaning RL improves the score from 44.27 to 59.77 (+15.50, a 35% relative improvement). This is the most direct evidence in the paper for the contribution of RL beyond SFT, since the SFT baseline uses the same architecture and training data minus the RL stage.

Per-game patterns. The performance varies dramatically across titles:

Games where UI-TARS-2 is near or above human: Shapes (108.9, the only game where the model exceeds human performance), Infinity-Loop (92.7), 2048 (91.0).
Games where RL provides large gains over SFT: Wood-blocks-3d (1900 SFT → 2908 RL, +53%), Hex-frvr (1952 → 2389, +22%), Emoji-sort-master (2.90 → 4.50, +55%), Yarn-untangle (4.30 → 7.00, +63%).
Games where the model makes little progress: Free-the-key (0.00 SFT → 0.70 RL), Maze-Path-of-Light (1.10 → 2.00).
Games where SFT outperforms RL: Gem-11 (84.90 SFT → 63.90 RL, a −25% regression) and Tiles-master (3.20 → 3.10, a marginal −3% regression). This is a notable negative result — on at least one game, RL training degrades performance relative to the SFT baseline. The paper's training dynamics analysis (Figure 13) shows that Gem-11 exhibits "clear plateaus or temporary regressions followed by shallow recovery, suggesting a reasoning ceiling imposed by the starting backbone rather than a lack of optimization steps."

LMGame-Bench (Table 3). UI-TARS-2 is evaluated against frontier models across six classic games with results reported as mean ± std over three runs:

2048: 117.1 ± 1.9 (within 9% of o3 at 128.2 ± 0.0, and competitive with Claude 3.7 Sonnet at 114.2 ± 7.2).
Candy Crush: 163.2 ± 31.3 (second only to Gemini 2.5 Pro at 177.3 ± 64.9, substantially ahead of o3 at 106.0 ± 0.0).
Super Mario Bros: 1783.2 ± 63.7 (behind o3 at 1955.0 ± 0.0 and GPT-4.1 at 1991.3 ± 1018.5, but competitive with other reasoning models).
Tetris: 16.0 ± 1.3 (behind o3 at 31.0 ± 0.0, but competitive with most other models in the 12–15 range).
Ace Attorney: 7.0 ± 0.0 (competitive with o3 at 8.0 ± 0.0 and ahead of most baselines which score 0–3).
Sokoban: 0.3 ± 0.0 (near-zero, with only o3 achieving 2.0 ± 0.0 and o4-mini 1.3 ± 0.6).

Assessment of LMGame-Bench results. UI-TARS-2's performance on this out-of-distribution benchmark is competitive but not dominant. It ranks near the top on Candy Crush and Ace Attorney, is competitive on 2048 and Super Mario Bros, and trails on Tetris and Sokoban. The paper's claim that UI-TARS-2 "remains competitive with frontier proprietary models" is accurate but should be understood as "competitive on specific games" rather than "outperforming across the board." The Sokoban result (0.3) indicates a fundamental limitation — the model cannot solve the hardest planning game in the suite, consistent with the paper's own analysis that "breaking through the remaining walls will likely require stronger long-horizon reasoning and planning capacity" (Section 3.3).

OOD Generalization from Browser RL to Desktop and Mobile

The paper highlights a specific transfer result: RL trained primarily on browser-focused tasks (GUI-Browsing and GUI-General) improves performance on OSWorld from 43.0% (SFT) to 47.5% (RL), a +10.5% relative gain, and on AndroidWorld from 64.6% to 73.3%, a +13.5% relative gain. These are the strongest evidence for the paper's claim that multi-turn RL induces broadly transferable skills. However, several caveats apply:

The SFT baseline already achieves non-trivial performance on these benchmarks (43.0% on OSWorld, 64.6% on AndroidWorld), indicating that the SFT training data included desktop and mobile tasks. The RL gain is therefore an improvement on top of an already-trained capability, not zero-shot emergence.
The paper does not report a version trained with RL on OSWorld or AndroidWorld tasks directly (which would be the "in-domain RL" comparison), so it is not possible to quantify how much of the gap between the browser-RL model and optimal in-domain performance remains.
The transfer is asymmetric: browser RL improves desktop/mobile performance, but the paper does not report whether the reverse transfer (desktop/mobile RL → browser) also holds.

Inference-Time Scaling (Figure 11)

The paper evaluates how performance scales with the maximum allowed inference steps on OSWorld and on game benchmarks. For OSWorld, "as the maximum allowed inference steps increase, the model's performance score consistently rises" — the curve in Figure 11 shows a monotonic, approximately logarithmic improvement. The paper does not provide specific numeric endpoints (the axes are not fully labeled in the available figure), but the trend is clear: more inference steps → higher success rate.

For games, the scaling curve "rises steadily in an almost monotonic, staircase-like pattern, without exhibiting instability spikes. In contrast, baseline curves flatten quickly" — the baseline agents (OpenAI CUA and Claude Computer Use) reach a performance plateau at relatively low step budgets, while UI-TARS-2 continues to improve. The paper interprets this as evidence that "the policy continues to unlock new subgoals as task-specific thresholds are crossed, rather than merely looping or drifting."

An interesting tension is noted: "Although RL training has incentivized our agent to complete tasks in fewer steps, the model still exhibits excellent inference-time scaling on the OSWorld evaluation set." This suggests that the RL training did not overfit to minimal-step strategies — the policy retains the flexibility to use additional steps productively when available. However, the paper does not quantify the tradeoff between step count and success rate, so it is unclear whether the improved success rate at higher step budgets justifies the additional inference cost.

Quantization Results (Section 3.3)

W4A8 quantization (4-bit weights, 8-bit activations) increases the token generation rate from 29.6 to 47 tokens/s (a 1.59× speedup) and reduces average end-to-end latency per interaction round from 4.0 to 2.5 seconds (a 1.6× speedup). On OSWorld, accuracy decreases from 47.5 to 44.4 (−3.1 percentage points, a 6.5% relative decline). The paper states this tradeoff is "favorable" but does not compare against alternative quantization schemes (e.g., W8A8, W4A4) or report accuracy on other benchmarks under quantization, making it difficult to assess whether W4A8 is the optimal operating point.

Ablation Studies and Robustness Checks

PPO vs. GRPO (Figure 12). PPO consistently outperforms GRPO in both GUI-Browsing and GUI-General domains, maintaining "higher rewards with lower volatility throughout training." This is a non-trivial finding because GRPO has been shown effective for reasoning RL tasks (Shao et al., 2024). The paper attributes this to the interactive, state-changing nature of GUI environments, which makes GRPO's group-based advantage estimation unreliable when different action sequences diverge into qualitatively different environment states. The ablation validates the choice of PPO as the base algorithm and provides a concrete negative result for a plausible alternative.

Value pretraining (Figure 10b). Training the value function to convergence under a fixed policy before starting PPO leads to "consistently higher rewards throughout training" compared to standard PPO where the value function is randomly initialized. This addresses a specific failure mode identified in preliminary experiments: "value estimates of PPO-trained models were often negatively correlated with the obtained rewards." The Figure 10(b) comparison provides causal evidence that value pretraining fixes this instability, though the paper does not report the magnitude of the improvement in absolute reward terms.

Hybrid RL vs. single-interface training (Figure 15). Training on both GUI-only and GUI-SDK interfaces simultaneously, even with halved data per interface, outperforms training on GUI-only data alone when evaluated on pure GUI tasks. This demonstrates asymmetric positive transfer — the SDK-augmented training teaches planning and problem-decomposition skills that generalize back to GUI-only settings. Additionally, Figure 15(e) shows that a shared value model achieves higher explained variance than interface-specific value models, indicating that learning a value function across both interfaces produces better-calibrated reward predictions. Compared to parameter interpolation (which merges specialized models without additional optimization), hybrid RL provides stronger transfer but at higher training cost — the paper presents both strategies as complementary.

GUI-SDK RL training dynamics (Figure 14). During GUI-SDK RL, training score shows an overall increasing trend while training entropy shows a continuous downward trend. This entropy pattern (decreasing) contrasts with the GUI-Browsing and GUI-General entropy dynamics (Figure 8, which shows increasing entropy). The paper does not explicitly comment on this difference, but it is consistent with the interpretation that tool-augmented tasks have more constrained solution spaces — once the model learns to use the right tool for the right sub-task, entropy decreases as the policy converges to efficient tool-use strategies, whereas pure GUI interaction requires maintaining broader exploration to handle diverse visual interfaces.

15-game per-title training rewards (Figure 13). The per-game training curves reveal heterogeneous learning dynamics: several titles (2048, Infinity-Loop, Emoji-sort-master, Shapes) approach or exceed human-level performance; others (Free-the-key, Yarn-untangle) learn from near-zero to nontrivial scores, indicating "a genuine increase in the model's general game-reasoning ability rather than overfitting to a few scripts"; and a subset (Gem-11, Hex-frvr) plateau or regress, suggesting "a reasoning ceiling imposed by the starting backbone." The staircase-shaped curves on many games indicate that progress arrives in bursts when task-specific subgoals become reliably attainable, followed by stabilization until the next threshold.

Interaction scaling dynamics (Figure 10a). The number of environment interaction rounds required to complete GUI-General tasks decreases as training progresses, even as rewards increase. This indicates that "the model internalizes task-relevant knowledge and reduces unnecessary exploration, allowing it to solve tasks more efficiently." The paper notes that without explicit step-budget penalties in the reward design, agents often learn to "exploit larger budgets, prolonging trajectories before convergence" — the length penalties described in Section 2.5.4 are the mechanism that counteracts this tendency.

Entropy dynamics (Figure 8). GUI-Browsing and GUI-General scenarios show rising entropy during training, while GUI-SDK shows decreasing entropy. The paper interprets rising entropy as evidence that "the model maintains or even expands its exploration space as training progresses, enabling it to acquire new interaction patterns." The periodic pattern observed in game entropy — increasing then decreasing in cycles — is attributed to the game difficulty curriculum: "Whenever the agent enters a new level, the increased difficulty requires more reasoning and decision-making to achieve success, leading to a rise in think length. As the agent becomes familiar with the challenges at a given difficulty level, the think length gradually decreases."

Think length dynamics (Figure 9). Average step-level think length consistently declines during GUI RL training. The paper suggests this is because "in GUI tasks, agents primarily make progress through interaction with the environment rather than extended internal reasoning alone. Consequently, once the correct GUI action can be predicted, the agent can obtain rewards directly, reducing the need for longer deliberation." This finding has implications for agent design: if think length naturally decreases with RL training, initial SFT training might intentionally bias toward longer reasoning traces to provide headroom for this compression.

ORM evaluation (Section 3.3, quantitative details in text). The generative ORM achieves an F1 score of 83.8 on a 300-trace human-annotated evaluation set in binary classification (success/failure). The paper notes a "relatively high false positive rate" but argues that RL training remains effective because "in false positive cases, the model still receives appropriate rewards for these correct steps, and these positive contributions outweigh the erroneous rewards given to incorrect actions." This is a reported result rather than a controlled ablation (no comparison to alternative ORM training strategies or reward models), but it provides quantitative evidence for the ORM's reliability and explains why imperfect reward modeling does not catastrophically derail training.

Parameter interpolation (Section 2.6, empirical claim in text). The paper states that "the merged model performs almost comparably to the best specialized model in each relevant domain, without additional optimization cost," but does not provide a table comparing merged model performance against individual specialized models on each domain. This is a missing ablation — the interpolation weights α_k are not reported, the sensitivity to weight selection is not explored, and quantitative comparison between the merged model and individual specialists is absent. The claim is therefore supported only by the fact that UI-TARS-2 (the merged model) achieves strong benchmark results, not by direct experimental evidence of interpolation effectiveness.

Critical Assessment

Claim 1: UI-TARS-2 achieves significant improvements over UI-TARS-1.5 and outperforms strong baselines.

This claim is supported with specific, enumerated comparisons in Table 1. On GUI benchmarks, UI-TARS-2 outperforms UI-TARS-1.5 by margins ranging from 12–16% relative (5–12.4 percentage points absolute). On games, the RL-trained model improves over the SFT baseline by 35% relative (44.27 → 59.77 mean normalized score). The comparison against proprietary baselines is mixed: UI-TARS-2 leads on OSWorld (47.5 vs. 43.9 for Claude 4 Sonnet), WindowsAgentArena (50.6 vs. 39.2), and Online-Mind2Web (88.2 vs. 71.0 for OpenAI CUA-o3), but is in a statistical tie with Claude 4 models on AndroidWorld (73.3 vs. 72.5–72.7) and slightly behind o3 on SWE-Bench (68.7 vs. 69.1).

Genuine weaknesses in this claim's support:

Baseline inference budgets are unspecified. If UI-TARS-2 was evaluated with more inference steps or a larger generation budget than baselines, the comparison is not controlled. The inference-time scaling analysis (Figure 11) shows that UI-TARS-2's performance improves substantially with more steps — if baselines were not granted the same scaling opportunity, the reported gaps overstate UI-TARS-2's advantage.
Evaluation is single-point with no confidence intervals for GUI benchmarks. With 369 tasks (OSWorld) or ~150 tasks (WindowsAgentArena), a few-percentage-point difference may be within the noise of environment non-determinism and task sampling.
Claude and OpenAI baselines are not evaluated on all benchmarks (many cells in Table 1 are "—"), so the "outperforms" claim is based on a subset of available comparisons, and the subset may be favorably selected.
The Terminal Bench result excludes 5/80 tasks for compatibility reasons. If those 5 tasks were systematically harder than average, the reported 45.3% overstates true performance.

Claim 2: Multi-turn RL yields substantial OOD generalization.

The paper reports that RL trained primarily on browser tasks improves OSWorld by ~10.5% (43.0 → 47.5) and AndroidWorld by ~8.7% (64.6 → 73.3) over the SFT baseline. This is the strongest evidence for transfer, but the claim requires scrutiny.

What was actually tested: the SFT baseline already achieves 43.0% and 64.6% on OSWorld and AndroidWorld respectively, meaning the model had prior exposure to these domains during supervised training. The RL gain is an improvement over an already-competent baseline, not emergence from a domain the model had never seen. "OOD" in this context means "the RL optimization was not specifically targeting these benchmarks," not "these environments are completely outside the training distribution."

What would strengthen the claim:

Reporting zero-shot performance of the base Seed-thinking-1.6 model on OSWorld and AndroidWorld (to show that SFT, not just RL, provides the initial capability, and to quantify the total improvement from base model to final agent).
Showing that RL on OSWorld tasks directly produces similar or larger gains (to establish that browser-RL transfer is competitive with in-domain RL).
Demonstrating transfer to a benchmark that was explicitly excluded from all training stages (CT, SFT, and RL), which would be a true test of generalization.

Claim 3: A unified GUI-SDK agent can outperform a GUI-only agent on pure GUI tasks.

The hybrid RL experiment (Figure 15) supports this claim: the hybrid model, trained on both GUI-only and GUI-SDK interfaces with half the GUI-only data, outperforms the GUI-only baseline on GUI tasks. This is a clean within-experiment comparison controlling for architecture and total training compute.

Limitations:

The experiment is reported for one scenario (information-seeking) and one interface combination. It is unclear whether the transfer effect generalizes to other task types (GUI-General, games) or other tool combinations.
The magnitude of the transfer gain is not quantified in absolute accuracy terms (the paper only shows training reward curves, not evaluation metrics).
The paper does not compare this hybrid training approach to the parameter interpolation approach on the same evaluation, so the "complementary" characterization (Section 3.3) is based on qualitative reasoning rather than empirical comparison.

Claim 4: The Data Flywheel produces a self-reinforcing improvement cycle where model and data quality co-evolve.

The paper provides indirect evidence: monotonically increasing training rewards (Figure 7) demonstrate that the policy improves across RL iterations, and the fact that UI-TARS-2 outperforms UI-TARS-1.5 (which was trained without the flywheel) is consistent with the flywheel providing benefits. However, the specific mechanism — that routing generated trajectories to CT and SFT based on quality V(s) produces better models than alternative data strategies — is not validated through controlled ablation.

Missing experiments that would directly support the flywheel claim:

Comparing flywheel training against a baseline that discards low-quality trajectories (rather than routing them to CT) to demonstrate that recycling failures is beneficial.
Comparing flywheel training against a baseline that uses all generated trajectories for SFT (without quality-based routing) to demonstrate that segregating by quality prevents contamination of the supervised signal.
Quantifying the relationship between flywheel iteration count and model performance (e.g., performance after 1, 2, 3 cycles) to show that the cycle is genuinely self-reinforcing rather than saturating after one iteration.
Reporting P(V(s) = 1 | t) across flywheel iterations to validate the paper's formal claim that success probability increases over time.

Without these, the flywheel's contribution is plausible but not causally demonstrated — the performance improvements could equally be attributed to any other component of the training pipeline (more SFT data, better RL stabilization, improved base model).

Claim 5: The enhanced PPO recipe (value pretraining, decoupled GAE, length-adaptive GAE, clip-higher, reward shaping) is necessary for stable GUI agent RL.

The paper provides evidence that individual components matter — value pretraining improves training rewards (Figure 10b), PPO outperforms GRPO (Figure 12) — but does not provide a full ablation study removing each enhancement individually and measuring the impact on final benchmark performance. The claim that these techniques are necessary as a combined recipe is therefore supported by the fact that they were used and the training was stable, not by experimental demonstration that removing any component degrades performance.

Missing ablations:

Standard PPO (without any of the five enhancements) on GUI tasks to establish a baseline instability level.
PPO with only a subset of enhancements (e.g., value pretraining only, or clip-higher only) to attribute gains to specific components.
The exact values of λ_policy, λ_critic, ε_low, ε_high, and α are not reported, making it impossible to assess whether the claimed benefits are specific to particular hyperparameter settings or robust across a range.

What was genuinely demonstrated: the paper shows a working RL recipe that produces stable training and strong downstream performance. The claim that this recipe is distinct from and superior to recipes designed for reasoning RL is supported by the GRPO comparison and the entropy dynamics analysis. But the claim that each individual component is critical is not directly tested.

Broader limitations in the experimental design:

Single model architecture and initialization. All experiments use Seed-thinking-1.6 with a 532M vision encoder and 23B-active MoE LLM. The paper does not test whether the findings generalize to other architectures (dense transformers, different vision encoders, smaller models). The specific improvements from value pretraining, decoupled GAE, and length-adaptive GAE may be artifacts of this architecture's properties rather than universal requirements for GUI agent RL.
No compute-matched comparisons. The paper compares UI-TARS-2 against proprietary models with unknown inference budgets and training compute. A FLOPs-matched comparison (analogous to the pretraining-vs-inference analysis in the reference example) would reveal whether UI-TARS-2's performance advantage reflects algorithmic superiority or simply more inference-time compute.
Incomplete baseline coverage. Important open-source GUI agents (CogAgent, OS-Atlas, Aguvis, SeeClick) are cited in related work but not evaluated as baselines. Comparing against the strongest open-source agents would contextualize UI-TARS-2's performance and help practitioners decide whether to adopt this closed-source model or an available alternative.
Human performance baselines are only provided for games, not for GUI benchmarks. On OSWorld at 47.5%, it is unclear whether this represents 50% of human performance or 90% — the task difficulty ceiling is unknown, making it hard to assess how much headroom remains.
The evaluation is focused on task success rate, not efficiency. UI-TARS-2 might achieve 47.5 on OSWorld by taking many more steps than baselines, or by using SDK functions that baselines lack access to. Without controlling for inference budget and tool access, the comparison measures a combination of policy quality, inference-time strategy, and tool availability rather than policy quality alone.
Limited statistical rigor. Only LMGame-Bench reports standard deviations over multiple runs. For a paper making comparative claims across benchmarks with 116–369 tasks, the absence of any uncertainty quantification (confidence intervals, bootstrap estimates, or significance tests) means that small-magnitude differences (e.g., 73.3 vs. 72.7 on AndroidWorld) cannot be distinguished from sampling noise.

6. Limitations and Trade-offs

The Difficulty Estimation Gap: Difficulty Bins Require 2048 Samples Per Prompt, a Cost the Headline Gains Do Not Include

The assumption or constraint. The compute-optimal allocation policy depends critically on estimating each prompt's difficulty before deciding how to spend the inference budget. The paper's method for doing so — generating 2048 samples per question and computing either the pass@1 rate (oracle) or averaging PRM scores (predicted) — is, by the paper's own acknowledgment, extraordinarily expensive. Section 3.2 states:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

To put this in perspective: generating 2048 samples for difficulty estimation exceeds the largest test-time budgets the paper studies (256–512 generations) by a factor of 4–8×. For a single question, the difficulty estimation cost alone could be 2048 generations, after which the compute-optimal policy might allocate 16–64 generations to actually solve the problem. This means the total cost (estimation + solving) is dominated by estimation, potentially by an order of magnitude.

The consequence. The headline finding — that compute-optimal scaling achieves "more than 4× better efficiency" over best-of-N — is computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment where difficulty must be estimated per query, the total cost would be 2048 + N generations, where N is the strategy budget. Since best-of-N requires only N generations total (no estimation overhead), the true efficiency comparison depends critically on how the estimation cost is amortized. If every query requires independent difficulty estimation, compute-optimal scaling is actually far less efficient than best-of-N — the 2048-generation estimation cost dwarfs any savings from smarter allocation.

The paper frames this as an "exploration-exploitation tradeoff" and suggests future work on predicting difficulty directly from question text. But until such a predictor exists and is validated, the 4× figure is best understood as a laboratory upper bound — what efficiency could be achieved if difficulty were known for free — rather than a deployment-ready result. In Section 3.2, the authors acknowledge:

"flagging it as a key avenue for future work"

but provide no prototype or even a feasibility analysis for cheap difficulty estimation.

What evidence exists in the paper. The 2048-sample requirement is specified in Section 3.2 when describing the difficulty bin construction procedure. The non-amortization is acknowledged in the same section. The cost itself is never quantified in the experiments — Figure 4 and Figure 8 show accuracy vs. generation budget for the solving phase only, and the 2048-sample estimation cost never appears as an x-axis offset or a total-cost annotation. The paper does not report how total cost (estimation + solving) compares to best-of-N baselines that spend all their budget on solving.

Mitigation status. The paper explicitly flags this as future work, suggesting "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). It also notes that the predicted (non-oracle) difficulty bins perform nearly as well as oracle bins (Figures 4, 8), meaning that ground-truth labels are not the bottleneck — the problem is the 2048 samples needed for the PRM-based approximation, not the need for labels. No difficulty prediction model is developed or evaluated. The limitation is not mitigated in the current work.

Hard Problems Remain Essentially Unsolved: Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper's framework assumes that test-time compute can amplify the base model's existing capabilities — that the model already produces correct solutions at some non-trivial rate, and search or revision can find or refine them. This assumption breaks down when the base model's pass@1 rate on a problem class is near zero. The paper is transparent about this boundary, stating in Section 7:

"test-time compute can amplify existing capability but cannot create it from nothing"

The consequence. On the hardest questions (difficulty bin 5), no method makes meaningful progress regardless of the compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets (4, 16, 64, 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling curve is essentially flat near 0–5%, well below the ~14× larger model's performance — meaning that even with a generous test-time compute budget, the smaller model cannot close the gap on hard problems because it never generates correct solutions to begin with.

This is not merely a "some problems are hard" observation — it is a fundamental ceiling on the approach. The paper's central value proposition is that test-time compute can substitute for pretraining, but this substitution only works on problems within the base model's capability envelope. For any problem class where the model's unaided pass@1 is negligible, the entire compute-optimal framework provides zero benefit. This means that for deployment scenarios where the problem distribution includes substantial hard problems (e.g., frontier mathematical research, novel software engineering tasks, complex multi-step planning in unfamiliar domains), scaling pretraining remains the only viable path — test-time compute offers nothing.

What evidence exists in the paper. The bin 5 results are reported across Figures 3 (right), 7 (right), and 9. The paper quantifies the bin 5 pass@1 as the lowest quintile (Section 3.2, though exact pass@1 thresholds per bin are not specified). The FLOPs-matched analysis in Section 7 shows that on hard problems at R ≪ 1 (the regime most favorable to test-time compute), the smaller model with compute-optimal scaling still underperforms the larger model, with relative disadvantages of −3.6% (PRM search) to +21.6% (revisions, though the bar chart in Figure 1 suggests this specific number applies to an aggregated "hard" bin that may differ from the paper's standard five-bin split).

Mitigation status. The paper acknowledges this limitation explicitly but does not attempt to address it. The boundary is characterized (bin 5 ≈ zero improvement) but no method is proposed for extending test-time compute benefits to these regimes. The limitation is fundamental to the approach — it follows from the fact that search and revision can only select or refine among candidates the model can generate, and if no correct candidate exists in the proposal distribution, no amount of search will find one.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, a Partially Patched but Unresolved Degradation Mode

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target — the training data never shows the model what to do when the current answer is already correct. Section 6.1 reports the consequence:

"approximately 38% of correct answers get converted back to incorrect ones"

This occurs because at test time, the revision model may encounter correct answers in its context (produced during earlier revision steps), and since it was never trained to recognize or preserve correctness, it "revises" them into wrong answers.

The consequence. The revision model's sequential generation process is inherently unstable: a chain of revisions can produce a correct answer at step k, then destroy it at step k+1, then (potentially) recover at step k+2. This makes the per-step pass@1 trajectory noisy and creates a dependence on within-chain selection mechanisms (majority voting or verifier-based selection across the full chain) to extract the correct answer from somewhere in the sequence. The trajectory in Figure 6 (left) shows pass@1 gradually improving across revision steps but never monotonic — the reversion phenomenon means that later steps are not strictly better than earlier ones.

This has practical consequences for deployment. If the system always takes the final revision output (the most natural default), approximately 38% of correct intermediate answers will be lost. The paper's mitigation — selecting the best answer from anywhere in the chain — requires retaining all intermediate outputs and running a selection mechanism over them, which adds storage overhead (for a chain of length N, you must store N candidate answers) and computational cost (running the verifier over all candidates). The selection mechanism itself is imperfect — the verifier can misrank answers, and majority voting requires sufficient diversity in the chain — so some fraction of the 38% reversion will still result in final incorrect answers.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. The mitigation (within-chain selection via majority voting or verifier) is described in the same section and evaluated in Figure 6 (right), which shows that selection improves over simply taking the last output. However, the paper does not report what fraction of the 38% reversion is recovered by within-chain selection — only that sequential + selection outperforms parallel sampling, which is a different comparison.

Mitigation status. The paper partially mitigates the issue through within-chain selection (Section 6.1) and acknowledges the root cause (training data only contains incorrect-to-correct transitions). It does not explore more principled solutions, such as:

Training the revision model on mixed sequences that include correct answers in context with a "no revision needed" target.
Adding a binary classifier that predicts whether the current answer is already correct and should be preserved.
Using the PRM or ORM score to dynamically decide whether to continue revising or stop.

The ReST<sup>EM</sup> experiment (Appendix K, Figure 16) provides further evidence that revision training is fragile: attempting to improve the revision model with on-policy RL caused performance to degrade with sequential revisions, dropping from roughly 38.5% at the optimal ratio to approximately 33.5% fully sequential at 256 generations. This suggests that the revision approach is sensitive to training methodology, and the positive results depend on the specific offline data construction procedure (edit-distance-based incorrect-correct pairing) — a procedure that may not transfer cleanly to other domains or model families.

The limitation is partially mitigated (within-chain selection helps) but not resolved (the root cause remains, and more aggressive training methods like ReST<sup>EM</sup> make it worse).

Single Model Family on a Single Benchmark: The Difficulty-Dependent Scaling Behavior May Not Generalize

The assumption or constraint. All experiments use PaLM 2-S* as the base model and the MATH benchmark (500 test questions) as the evaluation dataset. Section 4 states:

"We believe this model is representative of the capabilities of many contemporary LLMs"

but this claim is stated as a belief, not demonstrated. The paper provides no evidence that the difficulty-dependent scaling patterns — beam search degrading on easy problems, revisions dominating on easy problems, balanced sequential-parallel ratios being optimal on medium problems — transfer to other model families, other reasoning domains, or other task types.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution, the Monte Carlo rollout training procedure, and the specific MATH problem distribution. A model with different calibration properties (e.g., more confident wrong answers, or different error patterns) might exhibit different over-optimization thresholds, changing which strategies are compute-optimal at which difficulty levels.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with weaker in-context learning might not benefit from revision training at all.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning with clean ground-truth answers. Whether difficulty-dependent patterns generalize to code generation (where partial correctness matters), open-ended generation (where correctness is ambiguous), or tasks requiring factual knowledge rather than inference is entirely unverified.

The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation (Section 3.2), means the compute-optimal policy is selected based on roughly 50 questions per fold per bin. This is a small sample for strategy selection — if a particular difficulty bin happens to contain questions where beam search performs anomalously well or poorly due to sampling variation, the selected strategy may not represent the true optimal policy for that difficulty level in the broader distribution. The paper does not report confidence intervals or standard errors for the compute-optimal scaling curves (Figures 4, 8), making it difficult to assess the statistical reliability of the reported gains.

What evidence exists in the paper. The cross-validation protocol (Section 3.2) provides some internal validation: strategies are selected on one fold and evaluated on the other, reducing the risk of overfitting the policy to the test set. However, this does not address the broader generalization question — both folds are drawn from the same MATH distribution and the same PaLM 2-S* model. The difficulty-bin analysis is replicated across search methods (Figure 3 right) and revision strategies (Figure 7 right) with consistent patterns, which provides some confidence that the difficulty-dependent behavior is not a statistical fluke, but these replications are all within the same model and benchmark.

Mitigation status. The paper does not attempt to test generalization to other benchmarks, model families, or task domains. Section 8 does not explicitly mention this as a limitation or call for cross-model replication. The limitation is not mitigated — the findings should be understood as demonstrated for PaLM 2-S* on MATH, and their transferability to other settings is an open question.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained: the Pretraining-vs-Inference Tradeoff Is Measured Against a Weakened Baseline

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, where the larger model is trained following the LLaMA paradigm — scaling parameters while holding training data fixed. Section 7 explicitly acknowledges:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

In other words, the larger model is not Chinchilla-optimal (Hoffmann et al., 2022). A model trained with 14× more total FLOPs, where both parameters and data are scaled according to compute-optimal scaling laws, would likely outperform a parameter-only-scaled model at the same total FLOPs budget.

The consequence. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, revisions) — are measured against a suboptimal pretraining baseline. If the comparison were against a compute-optimally trained larger model, the gap would likely be smaller, and some of the claims (e.g., "a smaller model with test-time compute can outperform a ~14× larger model") might not hold, or might only hold in narrower regimes.

Additionally, the larger model baseline uses only greedy decoding (Section 7), with no test-time compute augmentation of its own — no majority voting, no best-of-N, no search. This is an asymmetric comparison: the smaller model gets to spend its FLOPs budget on both model parameters and test-time compute, while the larger model's FLOPs budget is allocated entirely to parameters with zero test-time optimization. A more balanced comparison would give the larger model a portion of the total FLOPs budget for test-time compute — for instance, comparing a 14× larger model with best-of-4 against a small model with compute-optimal scaling at a higher budget. The current comparison answers the question "can test-time compute with a small model beat a large model with no test-time compute at all?" — which is interesting but not the most relevant question for practitioners deciding how to allocate compute, since they would presumably apply some test-time strategy to whatever model they deploy.

What evidence exists in the paper. The parameter-only scaling choice is acknowledged in Section 7, with a commitment to future work. The greedy decoding assumption for the larger model is stated but not justified. The paper does not provide an ablation comparing against a larger model with modest test-time compute augmentation (e.g., best-of-4 or best-of-8), which would establish the robustness of the reported advantage.

Mitigation status. The paper acknowledges the limitation but does not address it experimentally. The commitment to future work on compute-optimal pretraining baselines is stated in Section 7. The limitation is not mitigated in the current work — the FLOPs-matched comparison should be interpreted as an upper bound on the advantage of test-time compute over pretraining, and the true advantage against a compute-optimal pretraining baseline is unknown.

Latency Constraints from Sequential Computation Are Not Accounted For: the 4× Efficiency Claim Assumes Perfect Parallelism

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores an orthogonal dimension: wall-clock time. Sequential revision strategies are inherently serial — each revision step depends on the output of the previous one — while parallel best-of-N can be executed simultaneously given sufficient hardware. A strategy that allocates 128 generations as "64 sequential × 2 parallel chains" (Figure 5, right panel) takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously, even though both use the same number of total generations.

The consequence. For latency-sensitive applications — interactive assistants, real-time decision-making, any deployment where the user is waiting for a response — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. The paper shows that easy problems (bin 1–2) perform best with purely sequential revisions (Figure 7, right), meaning the optimal strategy for accuracy is also the worst for latency. A practitioner deploying this system would face a direct accuracy-latency tradeoff that the paper does not quantify or discuss.

The compute-optimal framework as presented optimizes a single objective (accuracy) under a single resource constraint (generation count). A more complete optimization would include latency as a second constraint or objective, potentially leading to different strategy allocations — for example, preferring parallel strategies on easy problems (even if slightly less accurate) because they complete in 1× wall-clock time rather than 64×.

What evidence exists in the paper. The paper never mentions latency, wall-clock time, or the serial-vs-parallel time tradeoff. The generation count is used as the sole resource metric throughout. The sequential-to-parallel ratio experiments (Figure 7) report only accuracy, with no corresponding latency measurements. The inference-time scaling analysis (Figure 11) varies the allowed step budget but does not distinguish between steps taken serially vs. in parallel.

Mitigation status. The paper does not address this limitation. It is not mitigated — the 4× efficiency claim should be understood as applying to total FLOPs, not to wall-clock time, and the practical deployment considerations around latency remain unexplored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around GUI agent development from an engineering-driven, component-stacking paradigm toward a training-driven, systems-integrated one. Before UI-TARS-2, the dominant approaches fell into two camps: modular pipeline systems (perception + planning + memory + execution as separately engineered modules) and supervised fine-tuning on human demonstrations (CogAgent, OS-Atlas, UI-TARS). Both camps hit fundamental ceilings. Pipelines were brittle because each module's failures compounded across the chain. SFT-based agents plateaued because their training data was off-policy — they never learned to recover from their own mistakes. The paper's conceptual shift is the demonstration that multi-turn reinforcement learning in interactive environments is the missing ingredient that breaks both ceilings simultaneously, and that building the infrastructure to make this RL stable and scalable is a first-class research contribution, not merely engineering overhead.

The magnitude of this shift is best characterized as a methodological reframing with empirical validation at industrial scale, rather than a paradigm shift. The individual ideas — ReAct loops, PPO for language agents, model merging, data flywheels — are not individually novel. What is novel is their integration into a reproducible training recipe that produces a single agent competitive with or exceeding frontier proprietary models across GUI, game, and system-level benchmarks. The paper's most significant diagnostic contribution is the systematic analysis of training dynamics that reveals GUI agent RL to be a distinct optimization regime from reasoning RL. The rising entropy curves (Figure 8, opposite to reasoning RL's monotonic decline), the GRPO vs. PPO comparison showing PPO's clear superiority (Figure 12, opposite to GRPO's success on reasoning tasks), and the value function diagnostic (negative correlation between value estimates and actual rewards in preliminary experiments, motivating value pretraining in Figure 10b) collectively establish that techniques developed for reasoning RL do not trivially transfer to interactive, visually grounded domains. This is a regime identification — a finding that the optimization landscape is qualitatively different, not just harder — and it redirects research attention from "apply what worked for math" to "understand and address the specific failure modes of agent RL."

The paper also resolves a latent tension in the field between GUI-only and system-level agent capability. Prior work treated these as separate problems: GUI agents clicked and typed; code agents ran terminals and edited files. The BrowseComp results (Table 1) demonstrate that this separation is artificial and harmful. GUI-only operation achieves 7.0% on BrowseComp-en; adding SDK access jumps to 29.6%, a 4.2× improvement. Conversely, the hybrid RL experiments (Figure 15) show that training on both interfaces simultaneously improves pure GUI performance, even with halved GUI training data. This reframes the research question from "how do we build GUI agents and tool-using agents?" to "how do we build unified computer-use agents for which GUI actions and system commands are different vocabularies in a single action space?"

Several research directions become more attractive as a result of this work, while others become less so:

More attractive: research on verifier robustness for interactive tasks, since the paper demonstrates that even an imperfect ORM (F1 of 83.8, high false positive rate) can drive effective RL when false positives are concentrated on trajectories with mostly-correct intermediate steps. This is a surprising finding that opens the door to learned reward models for domains without ground-truth verifiers, which previously seemed infeasible.
More attractive: research on model merging for agent capabilities, since the paper validates at industrial scale that domain-specialized RL runs can be consolidated via linear parameter interpolation with preserved performance — a dramatically simpler path to general agents than joint multi-domain RL.
More attractive: research on infrastructure for agent training (sandboxes, rollout orchestration, stateful environment management), since the paper makes clear that algorithmic advances are gated by the ability to run millions of interactive episodes reliably. The days of treating infrastructure as "just engineering" are over for this subfield.
Less attractive: research on increasingly complex modular pipeline architectures for GUI agents. The paper's results suggest that end-to-end trained policies with sufficient RL scale outperform hand-crafted pipelines, and the scaling trend (Figures 7, 11) shows continued improvement with more training, whereas pipelines have no comparable scaling property.
Less attractive: research on GUI agents that operate purely through pixel-level interaction without access to terminals, file systems, or tools. The BrowseComp results demonstrate that pure GUI operation is severely capability-limited relative to hybrid approaches, and the paper's argument that many real tasks are "more naturally handled through file systems, terminals, or external tools" (Section 1) is empirically validated.

Follow-Up Research This Work Enables

Ablation of the enhanced PPO recipe to identify which components are individually necessary for GUI agent RL stability. The paper proposes five enhancements to standard PPO (reward shaping, decoupled GAE, length-adaptive GAE, value pretraining, clip-higher) and demonstrates that the combination produces stable training. But it does not ablate individual components. A follow-up study should train UI-TARS-2-scale agents with each enhancement removed individually, measuring the effect on training stability (entropy curves, value loss, explained variance) and downstream benchmark performance. The specific question: is value pretraining alone sufficient to fix the negative-correlation problem, or are decoupled GAE and length-adaptive GAE also necessary? Does clip-higher matter for final performance or only for early-training exploration? The paper's training dynamics analyses (Figures 7–10) provide the measurement methodology; a full ablation would establish a minimum viable recipe for stable GUI agent RL and identify which components are domain-specific vs. which might transfer to other interactive RL settings.

Cross-model replication of the difficulty-dependent entropy and think-length dynamics. The paper's most intriguing training analysis finding is that GUI RL exhibits rising entropy (Figure 8) and declining think length (Figure 9) — dynamics opposite to those reported for reasoning RL. But this is demonstrated for exactly one model architecture (Seed-thinking-1.6 with a 532M vision encoder and 23B-active MoE LLM). A replication study using a dense transformer architecture (e.g., a fine-tuned Qwen or LLaMA variant) and a different vision encoder would establish whether these dynamics are inherent to the GUI agent RL regime or an artifact of the specific model. The experiment is straightforward: reproduce the GUI-Browsing and GUI-General RL training on a different base model, log the same metrics (entropy, think length, reward), and compare the qualitative patterns. If the dynamics replicate, it strengthens the paper's claim that GUI agent RL is a distinct regime requiring its own optimization recipe. If they don't, it suggests the paper's findings are architecture-specific and the "regime identification" claim is overstated.

Closed-loop Data Flywheel ablation: does routing failed trajectories to CT rather than discarding them produce measurable gains? The paper claims the flywheel's quality-based routing — successful trajectories to SFT, failed ones to CT — is central to its self-reinforcing property. But this routing is never ablated. A direct test: run two training pipelines from the same initialization, identical in all respects except that one discards failed trajectories (the standard rejection sampling approach) while the other routes them to CT (the flywheel approach). After multiple flywheel iterations, compare downstream benchmark performance. This would answer whether the flywheel's routing logic provides benefits beyond simply generating more SFT data, and whether the recycling of failures into CT is genuinely important or an unnecessary complexity. The paper already has the infrastructure for multi-cycle training; the ablation requires only a configuration change to the data routing.

Difficulty prediction from question text to eliminate the 2048-sample estimation overhead, evaluated by total-cost comparison against best-of-N. The paper acknowledges that its difficulty estimation method (2048 samples per question) makes the 4× efficiency claim a laboratory upper bound that does not account for estimation cost. A natural follow-up: train a lightweight classifier — either a fine-tuned small LLM or a linear probe on the base model's embeddings — to predict difficulty bin directly from the question text, using the paper's existing oracle difficulty labels as training targets. Evaluate the trained classifier's bin accuracy against the 2048-sample PRM-based method. Then run the full compute-optimal pipeline using the classifier for difficulty estimation and compare total cost (estimation + solving) against a best-of-N baseline that spends the same total budget on parallel sampling. This would convert the paper's laboratory finding into a deployment-ready result if the classifier is sufficiently accurate, or establish a concrete accuracy-efficiency tradeoff if it isn't.

Does the hybrid RL cross-interface transfer effect (Figure 15) replicate on a non-browsing domain, such as games or software engineering? The paper demonstrates that training on both GUI-only and GUI-SDK interfaces improves pure GUI performance relative to GUI-only training alone, even with halved data. This is shown for one scenario (information-seeking). A replication on a different domain would test whether the transfer is a general property of augmented action spaces or specific to the browsing/SDK combination. For example: train agents on (a) games played through GUI actions only, (b) games played with access to both GUI actions and an "inspect game state" SDK function, and (c) a hybrid of both. If the hybrid outperforms the GUI-only agent on pure GUI game evaluation, it demonstrates that the cross-interface transfer generalizes beyond browsing. If not, it establishes a boundary condition that would refine our understanding of when and why augmented training helps.

Inference-time compute scaling comparison against proprietary baselines with controlled step budgets. The paper's inference-time scaling analysis (Figure 11) shows UI-TARS-2 continues to improve with more steps while baselines plateau, but the baseline curves are shown for only two models (OpenAI CUA and Claude Computer Use) and only on games. A rigorous extension would evaluate all available proprietary baselines (Claude 4, OpenAI o3, Gemini 2.5 Pro) on OSWorld and AndroidWorld at step budgets of 10, 20, 50, 100, and 200, measuring the scaling curve for each. This would establish whether UI-TARS-2's favorable scaling is a genuine algorithmic property (the policy learned to use extra steps productively) or an artifact of the baseline models having been optimized for a fixed, smaller step budget. The paper's own OSWorld scaling curve in Figure 11 provides the template; the extension is systematically applying it to all baselines on all benchmarks.

Practical Applications and Downstream Use Cases

Enterprise workflow automation for cross-application tasks. The paper's hybrid GUI+SDK architecture directly addresses a major pain point in robotic process automation (RPA): tasks that span GUI interaction (filling web forms, clicking through legacy applications) and system-level operations (running scripts, processing files, querying databases). Current RPA solutions require hand-crafted scripts for each workflow, which break when UI layouts change. UI-TARS-2's 50.6% on WindowsAgentArena and 88.2% on Online-Mind2Web — evaluated on diverse, realistic websites and desktop applications — suggests it can handle multi-step GUI workflows without per-task scripting, while the GUI-SDK augmentation (45.3% on Terminal Bench, 68.7% on SWE-Bench) enables file processing and code execution within the same agent. The practical deployment scenario: an enterprise deploys UI-TARS-2 (or a fine-tuned derivative) with access to a sandboxed VM, describes workflows in natural language ("download the Q3 sales report from the portal, extract the top 10 accounts, and email the summary to the regional managers"), and the agent executes across browser, spreadsheet, terminal, and email client. The BrowseComp results (50.5% on BrowseComp-zh with GUI-SDK, up from 32.1% GUI-only) suggest that complex information-seeking workflows — a common enterprise task — particularly benefit from the hybrid approach.

Scalable game testing and gameplay QA. The paper's game benchmark results — a mean normalized score of 59.77 across 15 diverse browser games, with near-human performance on 5 titles and above-human on 1 (Shapes, 108.9) — position UI-TARS-2 as a potential tool for automated game testing. Current game QA relies heavily on scripted bots that follow fixed paths and cannot adapt to UI changes or explore edge cases. An RL-trained agent that achieves 60% of human performance across diverse game mechanics (puzzle, arcade, strategy) could be deployed to play through new game builds, identify progression blockers, and surface unexpected behaviors — all without per-game scripting. The inference-time scaling property (Figure 11, staircase improvement with more steps) means the agent can be configured for thoroughness (more steps per level) or speed depending on testing needs. The practical benefit is coverage: the agent explores more state space than scripted bots because it learns to play rather than executing fixed instructions.

Accessibility agent for users who cannot interact with traditional interfaces. The paper's unified action space — the ability to operate any GUI through screenshots and standard input primitives (click, type, scroll) — makes UI-TARS-2 a candidate foundation for assistive technology. A user with motor impairments could describe a task in natural language ("find the cheapest flight to Chicago next Tuesday and book it"), and the agent would navigate airline websites, fill forms, and complete the booking through GUI interaction alone. The 88.2% on Online-Mind2Web (300 realistic web tasks across 136 websites) demonstrates competence on exactly this kind of multi-step web navigation. The quantization results — W4A8 reducing latency from 4.0 to 2.5 seconds per interaction round with only a 3.1 percentage point accuracy drop on OSWorld — are directly relevant: in an accessibility setting, responsiveness matters, and the paper shows the latency-accuracy tradeoff is favorable. The key practical advantage over existing screen readers and switch-control systems is that UI-TARS-2 requires no per-application configuration or API access — it operates purely from pixels, just as a human user would, making it compatible with any software that presents a visual interface.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Universal Deep Research: Bring Your Own Model and Strategy2025· agents← in-domain predecessor
- Memento: Fine-tuning LLM Agents without Fine-tuning LLMs2025· agents← in-domain predecessor
- AgentCPM-Report: Interleaving Drafting and Deepening for Open-Ended Deep Research2026· agents← similar topic
SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue2026· agents← similar topic
- Scaling Agent Learning via Experience Synthesis2025· agents← LLM-judged similar
- SWE-Universe: Scale Real-World Verifiable Environments to Millions2026· agents← in-domain predecessor
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills2026· agents← similar topic
GPA: Learning GUI Process Automation from Demonstrations2026· agents← similar topic
- CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery2026· agents← in-domain predecessor
- RVT-2: Learning Precise Manipulation from Few Demonstrations2024· agents← similar topic
SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning2025· agents← similar topic
- Scaling RL to Long Videos2025· rl-training← LLM-judged similar
- Memory in the Age of AI Agents2025· agents← in-domain predecessor
- Demystifying Reinforcement Learning in Agentic Reasoning2025· agents← similar topic
- Reflexion: Language Agents with Verbal Reinforcement Learning2023· agents← similar topic
MAI-UI Technical Report: Real-World Centric Foundation GUI Agents2025· agents← similar topic
- Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents2026· agents← similar topic
- MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios2026· agents← similar topic
- AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents2024· agents← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases2025· code→ LLM-judged similar
- X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests2026· code→ in-domain predecessor
- SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?2025· code→ similar topic
The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2025· agents→ in-domain predecessor
- Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalization2026· agents→ LLM-judged similar
- Reinforcement World Model Learning for LLM-based Agents2026· training-methods→ LLM-judged similar
- Astra: A Multi-Agent System for GPU Kernel Performance Optimization2025· agents→ in-domain predecessor
- Reflexion: Language Agents with Verbal Reinforcement Learning2023· agents→ similar topic
- Demystifying Reinforcement Learning in Agentic Reasoning2025· agents→ similar topic
- Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning2026· agents→ similar topic
MAI-UI Technical Report: Real-World Centric Foundation GUI Agents2025· agents→ similar topic
- KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta2025· agents→ in-domain predecessor
- AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents2024· agents→ similar topic
- Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents2025· agents→ similar topic