Agents2026~120 minarXiv:2601.15876

EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience

score 9Taofeng Xue, Chong Peng, Mianqiu Huang +13meituan↑ 91★ 281Jan 22, 2026cs.AIhttps://github.com/meituan/EvoCUA

🎯 Pitch

Training GUI agents on static data alone stops scaling—EvoCUA breaks through this wall by letting a 32B model learn from its own 10,000+ simultaneous sandbox rollouts, outperforming specialized 72B open-weight models and beating closed-source UI-TARS-2 by turning every failure into a targeted DPO correction at critical decision forks.

1. Executive Summary

This paper introduces EvoCUA, a native computer-use agent trained through an evolving paradigm via learning from experience that replaces static imitation with a self-sustaining cycle of verifiable task synthesis, massive-scale interactive rollouts, and iterative policy optimization. Using the Qwen3-VL-Thinking and OpenCUA backbones evaluated on the OSWorld benchmark, EvoCUA integrates a verifiable synthesis engine (autonomously generating diverse tasks paired with executable validators to produce deterministic reward signals), a scalable interaction infrastructure (orchestrating tens of thousands of asynchronous sandbox sessions for on-policy experience collection), and an iterative evolving learning strategy (reinforcing successful trajectories via rejection sampling fine-tuning while transforming failure trajectories into preference pairs through step-level DPO at critical forking points). EvoCUA-32B achieves a success rate of 56.7% on OSWorld, establishing a new open-source state-of-the-art that surpasses the previous best open model OpenCUA-72B by +11.7 absolute points and outperforms the closed-weights UI-TARS-2 (53.1%), while EvoCUA-8B reaches 46.1%—exceeding specialized 72B-parameter models—establishing that the evolving experience paradigm yields consistent gains across model scales only when the foundation model possesses sufficient base capability to benefit from interactive refinement.

2. Context and Motivation

The Core Problem: Static Imitation Cannot Capture Interactive Dynamics

The fundamental challenge this paper tackles is that current computer-use agents are trained on fixed, static datasets, and this training paradigm fundamentally cannot capture the causal feedback loops inherent in real-world computer interaction. This is not merely a data scarcity problem—it is a structural mismatch between how agents are trained and how they must operate at deployment time.

To understand why, consider what happens when a human interacts with a computer. Every action—clicking a button, typing a command, navigating a menu—produces environmental feedback. The screen changes. An error dialog might appear. A file might save successfully or silently fail. The next action depends entirely on what the previous action produced. This tight feedback loop means that computer use is inherently a sequential decision-making problem under partial observability, where the agent must continuously perceive, reason, and adapt based on the evolving state of the environment.

Now consider the dominant training paradigm for GUI agents prior to this work. Models were trained on static demonstration datasets—collections of pre-recorded interaction traces where a human or another model demonstrated correct behavior on a set of tasks. The agent learns by imitating these traces, effectively treating computer use as a sequence prediction problem: given this sequence of screenshots, predict this sequence of actions. This is behavior cloning applied to multimodal interaction traces.

The authors identify a critical limitation of this approach in the paper's opening:

"Existing scaling laws are largely confined to passive imitation of fixed, non-interactive datasets, failing to capture the causal feedback inherent in real-world computer use."

The phrase "causal feedback" is the key. In a static dataset, the agent never experiences the consequences of its own actions because the actions were already taken by someone else. The dataset shows what a successful trajectory looks like, but it cannot show what happens when the agent deviates from that trajectory—because in the dataset, no deviation occurred. This means the agent has no exposure to:

Error states and recovery paths: What happens when a click misses the target? What does the resulting screen look like, and how do you recover? Static datasets typically contain only successful trajectories, so the model never learns to recognize or recover from its own mistakes.
Distributions of possible next states: For any given action in a computer environment, there are many possible outcomes (the button might be grayed out, the network might be slow, a popup might appear). Static datasets show only one outcome—the one that happened during data collection. The model never learns the distribution over possible consequences.
The coupling between perception and action: In behavior cloning, the model sees a screenshot and predicts the action that was taken. But in deployment, the model's own action determines the next screenshot it sees. This creates a distribution shift: the model's action-taking policy generates different state trajectories than those in the training data, and errors compound because the model has never seen these off-distribution states during training.

This is a classic covariate shift problem in imitation learning, recognized since the early days of autonomous driving research. The paper explicitly frames the limitation as one of diminishing returns from scaling static data:

"Despite the foundational architectures established by state-of-the-art efforts... further progress is increasingly constrained by a critical bottleneck: the diminishing returns of scaling with static datasets."

The implication is that simply collecting more demonstration data—more tasks, more trajectories, more screenshots—will not solve the fundamental problem. The agent needs to interact with the environment and learn from the feedback that its own actions produce.

Why This Problem Matters

The significance of this problem extends beyond academic benchmarking. The authors are working toward generalist computer-use agents capable of mastering heterogeneous applications through visual perception alone, which represents a pivotal capability milestone toward artificial general intelligence. The practical implications span several dimensions:

Labor automation at scale. A reliable computer-use agent could automate vast categories of knowledge work that currently require human interaction with software interfaces—data entry, report generation, research synthesis, invoice processing, and countless other workflows that involve clicking, typing, reading, and decision-making across multiple applications. The economic value of such automation is enormous, but it requires agents that are robust enough to handle the variability of real-world software environments.

Accessibility and assistive technology. For users with motor impairments or other disabilities, a voice-controlled or intent-driven computer-use agent could serve as a transformative interface layer, translating high-level instructions into precise GUI interactions without requiring fine motor control. This application demands reliability under diverse conditions, exactly the kind of robustness that interactive training can provide.

Self-improving software systems. The paradigm of agents that can interact with software, observe outcomes, and improve their own behavior through experience opens the door to automated testing, bug reproduction, and even self-healing software workflows. An agent trained through interactive experience develops an implicit model of how software behaves, which generalizes beyond specific demonstration tasks.

Theoretical significance: from imitation to interaction. At a conceptual level, this paper represents a shift from viewing agent training as a supervised learning problem (predict actions from observations) to viewing it as a reinforcement learning problem (learn a policy through environmental interaction). This shift is necessary because the supervised formulation ignores the sequential, feedback-driven nature of computer use. The paper's framing as a POMDP in Section 2.1 formalizes this: the environment state st is partially observable through rendered screenshots, actions produce state transitions according to a transition kernel P(st+1 | st, at), and the agent receives sparse, terminal rewards based on task completion. This is the correct formalism for interactive computer use, and it demands interactive training methods.

Where Prior Approaches Fall Short

The paper identifies specific limitations across several categories of prior work:

Static imitation pipelines. The dominant approach prior to this work was to collect large datasets of human or model-generated interaction traces and fine-tune VLMs to imitate them. OpenCUA (Wang et al., 2025b) introduced the AgentNet dataset with 11,887 tasks across diverse domains. UI-TARS-2 (Wang et al., 2025a) scaled this further with multi-turn reinforcement learning on collected demonstrations. Step-GUI (Yan et al., 2025) incorporated step-wise visual reasoning into the imitation framework.

The fundamental shortcoming of these approaches is that the agent never experiences the consequences of its own decisions during training. It learns what a correct trajectory looks like, not what happens when it makes errors or how to recover from them. The authors frame this as the difference between "data scaling via static traces" and "experience scaling via massive interactive rollouts":

"Dynamic experience provides a richer supervisory signal than static text, encompassing environmental feedback and critical insights from both success and failure."

This is not merely a philosophical distinction—it has concrete training implications. Static datasets provide no signal about failure modes. The agent's training objective is to maximize the likelihood of the demonstrated actions given the demonstrated states. But at deployment, when the agent inevitably makes errors (wrong click coordinates, premature termination, misinterpreted visual elements), it enters states that were never present in the training distribution. Error compounding follows, and the agent has no learned mechanism for recovery.

Data scarcity and annotation cost. A more practical but equally critical limitation is that high-quality GUI interaction data is expensive to produce. Manually creating diverse, realistic computer-use tasks and recording expert demonstrations is labor-intensive and does not scale to the tens or hundreds of thousands of trajectories needed for robust policy learning. The paper notes that the field faces a "data scarcity bottleneck"—there are simply not enough high-quality, diverse interaction traces to train agents that generalize across applications and scenarios.

Synthetic data without verification. Some prior work attempted to address data scarcity by synthetically generating task instructions. However, the paper identifies a specific failure mode with this approach:

"Merely synthesizing textual queries often leads to hallucinations, where the agent generates plausible plans for infeasible tasks."

The problem is that natural language task descriptions are ambiguous as training signals. If a synthesis engine generates an instruction like "update the quarterly sales chart to reflect the new projections," this instruction alone does not specify what counts as success. A model might generate a plausible-looking sequence of actions that modifies the chart, but without a ground-truth validator, there is no way to determine whether the modification was actually correct. The agent might learn to produce actions that look reasonable without ever achieving the actual task objective. This is a form of reward hacking at the data generation level—the synthesis engine produces instructions that lead to plausible but incorrect behavior.

The gap between open and closed models. The empirical landscape prior to this work showed a significant performance gap between open-weights and closed-weights computer-use agents. On the OSWorld benchmark, the best open-weights model (OpenCUA-72B) achieved 45.0% success rate, while leading closed-weights models like Claude-4.5-Sonnet reached 61.9-62.9%. The paper positions this gap not as a fundamental architectural limitation but as a training paradigm gap—closed-weights models likely benefit from proprietary interactive training pipelines that open-source efforts have not replicated.

The Three Specific Challenges the Paper Identifies

To move from static imitation to interactive experience learning, the paper explicitly names three challenges that must be solved simultaneously (Section 1):

Challenge 1: Verifiable data synthesis. Generating task instructions is not enough; the system must co-generate executable validators that can deterministically assess whether a completed trajectory achieved the task objective. This is the "generation-as-validation" paradigm that Section 3 develops in detail. Without verifiable rewards, interactive training cannot distinguish between successful and failed trajectories, and the entire learning loop collapses. The validator must be:

Deterministic: Given the same final environment state, it must always produce the same success/failure judgment. Ambiguous or stochastic validation introduces noise into the reward signal.
Executable in the environment: The validator must run programmatically within the sandbox, inspecting files, application states, or system configurations to verify task completion. It cannot rely on human judgment or LLM-based evaluation, which would reintroduce ambiguity and cost.
Grounded in actual environment state: The validator must check what actually happened, not what the agent claimed happened. This prevents the model from learning to output convincing narratives without achieving real outcomes.

Challenge 2: Scalable interaction infrastructure. Interactive training requires thousands of concurrent environment instances running in parallel, each hosting an agent executing actions and receiving feedback. The infrastructure must:

Support diverse application environments (browsers, office suites, file managers) with consistent, reproducible behavior.
Scale elastically to match the training demand—when the policy is updated, thousands of fresh rollout sessions must be provisioned rapidly.
Maintain strict isolation between sessions to prevent cross-contamination of environment states.
Provide deterministic, low-latency rendering and input processing so that agent actions produce predictable, reproducible outcomes.

This is an engineering challenge of significant magnitude. Standard cloud infrastructure is not designed for thousands of concurrent virtual desktop sessions with GUI rendering and precise input simulation. The paper's solution (Section 4) involves a custom virtualization architecture combining Docker containers with QEMU-KVM virtual machines, a distributed scheduler for elastic scaling, and an asynchronous gateway for high-throughput request routing.

Challenge 3: Efficient training that learns from both success and failure. Given the massive interaction space of computer environments, unconstrained exploration is computationally prohibitive. The agent cannot simply try random actions and learn from whatever happens—the action space is too large, and useful task completion is too sparse. The training recipe must:

Consolidate successes: When the agent succeeds at a task, extract the efficient, high-quality execution pattern and reinforce it, while filtering out redundant or unnecessary steps that happened to be present in the successful trajectory.
Learn from failures: When the agent fails, identify where in the trajectory the critical error occurred and construct targeted preference pairs that teach the model what it should have done differently. Raw failure trajectories are too noisy for direct imitation, but they contain high-value information about capability boundaries.
Focus computation on boundary tasks: Tasks that the agent consistently solves or consistently fails provide little learning signal. The valuable training experiences come from tasks where the agent's success varies—the capability boundary where small changes in behavior flip the outcome. The training system should allocate more exploration budget to these boundary tasks.

The paper frames this triage as mimicking human learning dynamics:

"Effective learning requires an on-policy approach that mimics human learning dynamics: consolidating mastered routines while focusing intensely on boundary tasks where the agent oscillates between success and failure."

This is a direct response to the inefficiency of uniform exploration. If the agent already solves a task 95% of the time, generating 100 more trajectories on that task produces mostly redundant successes. If it solves a task 0% of the time, generating 100 more trajectories produces only failures with no positive signal. The sweet spot is tasks in the 20-80% success range, where the agent sometimes succeeds and sometimes fails, and the difference between success and failure trajectories reveals actionable improvements.

How This Paper Positions Itself Relative to Existing Work

The paper situates itself at the convergence of several research threads, but with a distinctive emphasis that differentiates it from each:

Against static imitation approaches (OpenCUA, UI-TARS-2, Step-GUI). The paper acknowledges these as important foundational efforts—EvoCUA literally builds on OpenCUA and Qwen3-VL backbones—but argues that their reliance on static datasets for training creates a fundamental ceiling. The positioning is not that imitation is wrong, but that it is insufficient alone. The paper's cold-start phase (Section 5.1) actually uses imitation on carefully constructed trajectories, but this is only the initialization, not the entire training recipe. The subsequent rejection sampling and DPO phases are where the agent transcends what imitation alone can achieve.

Empirically, the paper demonstrates this distinction through the EvoCUA-8B vs. Step-GUI-8B comparison (Table 1): both are initialized from the identical Qwen3-VL-8B-Thinking backbone, but EvoCUA-8B achieves 46.1% vs. Step-GUI-8B's 40.2% (+5.9 absolute points). The paper explicitly notes:

"This strictly isolates the contribution of our evolving experience learning paradigm, confirming that our data synthesis and RL strategies unlock significantly greater potential from the same foundational architecture."

Against pure RL approaches for GUI agents (UI-TARS-2's RL, GRPO-based methods). The paper acknowledges that reinforcement learning has been applied to GUI agents before—UI-TARS-2 used multi-turn RL, and GRPO-based methods have been applied to reasoning tasks. However, existing RL approaches for GUI agents typically assume pre-existing reward signals or rely on trajectory-level optimization that the paper identifies as problematic.

The paper's contribution is not "RL for GUI agents" in the abstract, but rather a complete pipeline that generates its own training tasks (solving the data scarcity problem), produces its own verification signals (solving the reward definition problem), and optimizes the policy at the step level rather than the trajectory level (solving the credit assignment problem for long-horizon tasks). Each component is necessary; none is sufficient alone.

The step-level versus trajectory-level distinction is particularly important. The paper's Section 7 identifies a specific failure mode of trajectory-level RL (like GRPO) for GUI tasks:

"If the trajectory of the final step is directly used for training, the model will not be able to learn the supervision signals of intermediate steps."

This occurs because GUI agents compress their interaction history for efficiency—only recent steps retain full multimodal context, while earlier history is text-only. Training on the final step alone cannot propagate learning signals back through this compressed history. The proposed STEPO algorithm (Section 7) addresses this by allocating the trajectory's advantage uniformly across all steps, ensuring each step receives a learning signal.

Against pure synthetic data generation. The paper distinguishes its synthesis engine from simple LLM-based instruction generation through the co-generation of executable validators and the closed-loop feedback mechanism. The "generation-as-validation" paradigm is a direct response to the hallucination problem in naive synthetic data. By requiring that generated tasks include programmatic success conditions and by verifying those conditions through actual sandbox execution, the system eliminates the ambiguity that would otherwise corrupt the reward signal.

The broader paradigm claim. The paper's strongest positioning claim is that EvoCUA represents a paradigm shift rather than an incremental improvement:

"Overcoming this limitation necessitates a paradigm shift from data scaling via static traces to experience scaling via massive interactive rollouts."

The evidence for this claim comes from the generality of the results: the evolving experience paradigm produces consistent gains across different foundation models (Qwen3-VL-32B, Qwen3-VL-8B, OpenCUA-72B) and different scales. The ablation in Table 4 shows that applying the same pipeline to OpenCUA-72B yields cumulative improvements of +2.14% (cold start), +3.69% (RFT), +3.02% (DPO), and +1.82% (iterative training). This cross-model consistency suggests the paradigm captures something fundamental about how computer-use agents should be trained, rather than being a model-specific trick.

However, the paper is careful not to claim that interactive training alone solves everything. The difficulty-bin analysis is transparent about a critical boundary condition: on the hardest problems (the equivalent of difficulty bin 5 from the MATH analysis example, though the paper does not use this exact terminology), test-time compute provides essentially zero benefit. The base model must possess sufficient capability for interactive refinement to help. The paper's own results show that performance improvements come from training, not from architecture—the foundation model's base capabilities set the ceiling that interactive training can approach but not exceed.

The Self-Sustaining Cycle as Unifying Concept

Figure 2 in the paper provides the visual architecture of the proposed solution: a cycle connecting the verifiable synthesis engine (generating tasks and validators), the scalable interaction infrastructure (executing rollouts and collecting experience), and the iterative optimization (updating the policy from experience). The paper describes this as "a self-sustaining cycle that continuously transforms synthetic compute into high-quality agent capabilities."

This framing is important because it emphasizes that the three components are not independent modules that could be developed separately—they form a closed loop where each component's output feeds the next component's input:

The synthesis engine generates tasks based on the agent's current capabilities (more tasks in boundary difficulty regions).
The infrastructure executes these tasks using the current policy snapshot, collecting both successful and failed trajectories.
The optimization phase updates the policy from these trajectories, improving the agent's capabilities.
The improved agent reveals new capability boundaries, which the synthesis engine targets in the next iteration.

This is what makes the paradigm "evolving" rather than "iterative" in a simple sense. The training distribution itself evolves as the agent improves, creating a curriculum that automatically adapts to the agent's current skill level. The paper's dynamic compute budgeting mechanism (Section 5.2) operationalizes this adaptation by allocating more rollout budget to tasks where the agent's success rate falls in the informative middle range, and less to tasks that are already mastered or still impossible.

3. Technical Approach

3.1 Reader Orientation

EvoCUA is a training system that produces a computer-use agent—a vision-language model that takes screenshots as input and outputs mouse clicks, keyboard strokes, and reasoning traces to accomplish arbitrary tasks across desktop applications. The system solves the problem that static imitation on pre-recorded demonstrations cannot teach an agent to recover from its own mistakes or adapt to the interactive feedback loops inherent in real computer use; the solution takes the shape of a closed evolutionary cycle where the agent generates its own training experiences by interacting with real virtual machines, receives deterministic success/failure signals from automatically-generated verifiers, and continuously improves through a combination of reinforcing successful behaviors and correcting specific failure points.

3.2 Big-Picture Architecture (Diagram in Words)

The EvoCUA system comprises three interconnected modules that form a self-sustaining loop (illustrated in Figure 2):

Verifiable Synthesis Engine (the "task generator"): Produces diverse natural-language instructions g paired with executable validator programs V_g that can deterministically check whether a completed task trajectory achieved the objective. This module solves the data scarcity problem by generating training tasks algorithmically, and solves the reward ambiguity problem by co-generating machine-verifiable success conditions.
Scalable Interaction Infrastructure (the "experience factory"): A high-throughput platform that runs tens of thousands of concurrent virtual machine sessions, each executing agent rollouts in real desktop environments. Policy snapshots π_old are deployed across these sandboxes to collect massive streams of interaction trajectories τ, producing both successful executions and informative failures.
Iterative Optimization Engine (the "learner"): Consumes the collected trajectories through a staged training pipeline—cold-start initialization on high-quality synthetic demonstrations, rejection sampling fine-tuning on denoised successful trajectories, and step-level DPO that identifies critical error points in failed trajectories and constructs targeted preference pairs for correction.

Information flows in a cycle: the synthesis engine generates tasks (possibly conditioned on the agent's current capabilities) → the infrastructure executes those tasks using the current policy, collecting trajectories with verifiable outcomes → the optimization engine updates the policy parameters to improve performance → the updated policy reveals new capability boundaries, which informs the next round of task synthesis. The paper describes this as "continuously transforming synthetic compute into high-quality agent capabilities."

3.3 Roadmap for the Deep Dive

First, the formal problem formulation (POMDP and objective), because it establishes the mathematical language—states, actions, observations, rewards—that the entire system is built to operationalize, and defines what "learning from experience" means in precise terms.
Second, the Verifiable Synthesis Engine, because it is the upstream component that produces the tasks and ground-truth validators without which no interactive learning could occur. Understanding how tasks and validators are co-generated, verified, and filtered is prerequisite to understanding what signals the agent receives.
Third, the Scalable Interaction Infrastructure, because it is the execution substrate that transforms synthesized tasks into actual interaction trajectories. The engineering decisions here—hybrid virtualization, deterministic input calibration, asynchronous orchestration—directly determine the throughput, fidelity, and reproducibility of the experience data.
Fourth, the Cold-Start phase, because it establishes the agent's initial behavioral prior: the unified action space, the structured reasoning schema, and the hindsight reasoning generation strategy that produces the initial training data. Everything that follows builds on this foundation.
Fifth, Rejection Sampling Fine-Tuning (RFT) , because it is the first active learning stage where the agent generates its own experiences and learns selectively from successes. The dynamic compute budgeting and step-level denoising mechanisms are key innovations here.
Sixth, the DPO-based reinforcement learning phase, because it addresses what RFT cannot: learning from explicit failures through structured preference pairs at critical forking points. The dual-paradigm construction (Action Correction and Reflection/Recovery) is the technical core of this stage.
Seventh, the proposed STEPO algorithm for online RL, because it addresses a fundamental training-inference discrepancy that arises when applying trajectory-level RL to GUI agents with truncated context windows, and represents the paper's forward-looking direction.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a language model can be trained to control a computer through a closed-loop cycle of self-generated experience, provided three conditions are met: (1) tasks come with machine-executable verifiers so that success/failure signals are deterministic, (2) rollouts execute at massive scale in realistic virtual environments with on-policy data collection, and (3) optimization targets the step-level granularity that matches the agent's truncated inference context, learning from both successes (through denoised imitation) and failures (through structured preference pairs at causal error points).

The Formal Problem: POMDP Formulation and Optimization Objective

The paper grounds its entire approach in a formal decision-theoretic framework before describing any implementation details. This is not merely academic formalism—the POMDP structure directly motivates specific design choices throughout the system, including the sparse reward formulation, the focus on step-level optimization, and the need for verifiable termination signals.

The POMDP tuple. The interaction process is modeled as a Partially Observable Markov Decision Process (POMDP) (S, A, Z, O, P, R_syn), where each component is defined as follows:

State Space S: The underlying computer system state s_t ∈ S, which includes "application states, system configurations, and implicit system-level context." Crucially, s_t is not directly observable by the agent. The agent never sees the raw memory contents, file system structures, or process states of the operating system—it only sees rendered screenshots. This partial observability is what forces the agent to learn visual perception as its sole interface to the environment.
Observation O: The agent perceives a visual observation o_t ≜ I_t ∈ ℝ^(H×W×3), where I_t = Render(s_t) is the screen image at time t, with H and W being the height and width in pixels. The rendering function Render(·) is a deterministic mapping from system state to pixel values—the screen is what the agent sees, and nothing more. This is the sole perceptual interface through which the agent observes the environment, which is a hard constraint that drives the need for visual grounding capabilities.
Action Space A: The agent outputs actions from a unified native action space A = A_mouse ∪ A_keyboard ∪ A_control. Each action type maps to a specific low-level input primitive:
- A_mouse: Coordinate-based mouse events including mouse_move, left_click, right_click, middle_click, double_click, triple_click, left_click_drag, and scroll/hscroll. Each mouse action takes coordinate arguments (x, y) in pixel space.
- A_keyboard: Keyboard inputs including key (press and release a key sequence), key_down (press and hold), key_up (release), and type (type a text string). The stateful key_down/key_up separation is critical for supporting modifier-key combinations like Shift+Click for multi-selection.
- A_control: Meta-actions for execution flow management—wait (pause for a specified duration to handle asynchronous UI rendering) and terminate (signal task completion with a status of either "success" or "failure").
The full action space specification is provided in Appendix A, Table 6, with exact argument signatures for each primitive.
Thought Space Z: The agent explicitly models its reasoning as a natural language thought trace z_t ∈ Z generated before each action. This is not merely an architectural convenience—the thought trace serves as an intermediate cognitive state that grounds the subsequent physical action in the current visual context. The paper enforces a specific reasoning schema (described in Section 5.1) that structures z_t to include goal clarification, observation consistency checks, and self-verification.
Policy π_θ: The agent follows a parameterized policy π_θ(z_t, a_t | h_t, o_t) that governs both reasoning and action selection. At each step t, given the interaction history h_t and the current observation o_t, the policy first generates a reasoning trace z_t and then generates an executable action a_t conditioned on that reasoning. This sequential generation—reasoning before action—ensures that physical execution is causally downstream of explicit reasoning.
Transition P: The environment state evolves according to P(s_{t+1} | s_t, a_t), which "captures the dynamics of the underlying computer system in response to the executed physical action." After the state updates, the next observation is rendered as I_{t+1} = Render(s_{t+1}). This transition kernel is what creates the interactive feedback loop—the agent's action a_t causally determines what it sees next.
Verifiable Reward R_syn: This is the critical component that enables autonomous learning. For a given instruction g, the synthesis engine provides an executable validator V_g that evaluates whether the task objective is satisfied by inspecting the final environment state. The reward is defined as a sparse, binary, instruction-conditioned signal based on the terminal state:

$R_{syn}(s_T; g) ≜ \mathbb{1}[V_g(s_T) = \text{True}]$

where s_T is the environment state at episode termination, V_g is the executable validator program co-generated with instruction g, and \mathbb{1}[·] is the indicator function returning 1 if the condition is true and 0 otherwise.

What it computes: a single bit at the end of each trajectory—1 if the validator program confirms that the task's success conditions are met in the final system state, 0 otherwise. This is the entire reward signal used throughout training; there are no intermediate rewards, no shaped rewards, no learned reward models. The agent either succeeds or fails, and this binary outcome is the sole training signal.

Why this form: the binary, verifier-grounded reward eliminates the ambiguity that would corrupt learning if rewards came from language model judgments or human annotations. Since V_g is a deterministic program that inspects the actual filesystem state (file contents, application data, configuration values), it provides objective ground truth. The sparsity (reward only at termination) is not a bug but a feature—it forces the agent to learn the causal chain from its actions to the final outcome rather than relying on intermediate reward shaping that might encode incorrect assumptions about what intermediate steps should look like. The paper explicitly contrasts this with "the ambiguity of semantic matching" and "hallucinations" from text-only reward models.

The interaction history h_t. To address partial observability, the agent conditions on a compressed interaction history:

$h_t = \{g, o_0, z_0, a_0, \ldots, o_{t-1}, z_{t-1}, a_{t-1}\}$

This is the concatenation of the original instruction and all previous observation-reasoning-action triples. However, a practical constraint forces compression: to prevent the context window from being flooded with high-resolution screenshots over long trajectories, the paper restricts the visual history to the five most recent screenshots and compresses the textual history into "a structured inner monologue with action representation." This means that for steps beyond the five-step window, the model sees the reasoning traces and actions from earlier steps but not the actual screenshots. This compression strategy has a direct consequence that the paper identifies in Section 7: trajectory-level RL cannot propagate learning signals through compressed history because the model at the final step lacks the visual context to learn from intermediate decisions.

The Optimization Objective

The paper frames the training objective not as learning from a static dataset but as maximizing expected success rate over a dynamic task distribution that adapts to the agent's current capabilities.

Theoretical objective. Formally:

$J(\theta) = \mathbb{E}_{(g, V_g) \sim T_{syn}(\cdot | \pi_{old})} \left[ \mathbb{E}_{\tau \sim \pi_\theta(\cdot | g)} [R_{syn}(s_T; g)] \right]$

where T_syn(· | π_old) is the synthesis engine's task distribution, which can be conditioned on the previous policy snapshot π_old to adjust task complexity and diversity, τ ∼ π_θ(· | g) denotes trajectories induced by executing policy π_θ in the environment dynamics under instruction g, and R_syn(s_T; g) is the binary verifiable reward at trajectory termination.

What it computes: the expected success rate of the current policy π_θ over a task distribution that is itself a function of the agent's previous capabilities. The outer expectation samples tasks from the synthesis engine (which may generate harder tasks as the agent improves), and the inner expectation samples trajectories from the policy's execution on each task and evaluates success via the validator.

Why this form: the coupling between the task distribution and the policy (T_syn(· | π_old)) is what makes this a self-improving system rather than a fixed-benchmark training procedure. If the task distribution were static, the agent would eventually saturate and stop learning. By making the task distribution depend on the agent's current capabilities, the system creates an automatic curriculum—the agent is always presented with tasks at the boundary of its abilities. The paper's dynamic compute budgeting mechanism (Section 5.2) operationalizes this by identifying these boundary tasks empirically based on observed pass rates.

Empirical approximation. Since the expectation admits no closed-form solution, the paper approximates it through massive-scale Monte Carlo estimation. The scalable interaction infrastructure maintains a transient experience pool B:

$B = \{(\tau, V_g) \mid \tau \sim \pi_{old}(\cdot \mid g), (g, V_g) \sim T_{syn}\}$

where π_old is the policy snapshot driving tens of thousands of asynchronous sandboxes. By continuously updating θ using batches sampled from B, the system closes the loop between verifiable synthesis, large-scale execution, and on-policy optimization.

What this means operationally: the experience pool B is a continuously refreshed buffer of trajectory-validator pairs. The policy π_old that generates trajectories is slightly stale—it is the version of the model that was deployed to the sandboxes before the current optimization step began—which makes the collected trajectories approximately on-policy. Training on these trajectories approximates the gradient of the true objective J(θ). As the policy is updated and redeployed to sandboxes, new trajectories are collected with the updated policy, refreshing the pool with on-policy data.

Verifiable Synthesis Engine

The synthesis engine is the upstream component that generates the coupled pairs (g, V_g)—natural language instructions and executable validator programs—that form the training task distribution. Its design addresses the core problem that "merely synthesizing textual queries often leads to hallucinations, where the agent generates plausible plans for infeasible tasks." The engine is organized into three cascading modules.

Structured Task Space Construction

Hierarchical Domain Taxonomy. The paper argues that "atomic capabilities are inherently transferable and compositionally form complex tasks." Based on this principle, the authors systematically categorize core desktop applications—specifically naming Web Browsers, Excel, and Word—and decompose user behaviors into atomic capabilities: primitive skills that can be recombined to form diverse tasks.

For example, a financial analysis task in Excel is decomposed into atomic sub-skills such as formula manipulation, data sorting, and chart generation. By treating these as orthogonal building blocks, the synthesis engine can generate novel tasks by composing sub-skills in configurations not seen in any static dataset. The engine further synthesizes diverse user personas following the approach of Ge et al. (2024), with scenarios "ranging from educators designing lecture slides to algorithm engineers conducting technical literature surveys." The persona injection ensures that task instructions vary in domain-specific vocabulary, formality, and implied workflow.

Hybrid Resource Injection. The initial environment state must contain files and applications for the agent to interact with. The paper implements a hybrid strategy:

Parametric synthesis: For structured data like production sales reports, code-based generators batch-produce documents (Word, Excel, PDF) by parameterizing variables such as names, prices, and dates. This ensures high variability in numerical values and document layouts without requiring manual file creation.
Non-parametric injection: To prevent the synthetic environments from becoming sterile and predictable, the engine injects public internet data—images, audio files, complex slide decks—into the initial file system. This "forces the agent to handle the visual noise and structural diversity inherent in real-world files," bridging the gap between clean synthetic training environments and messy real-world deployment conditions.

The combination of parametric and non-parametric resources creates a distribution of initial states that is both diverse (many parameter combinations) and realistically complex (real-world file artifacts introduce unexpected visual patterns, formatting inconsistencies, and application-specific behaviors).

Agentic Dual-Stream Synthesis

The core synthesis process is modeled as a ReAct-based agentic workflow (Yao et al., 2022) where a foundation VLM functions as a "task architect" executing two parallel generation streams.

Input sampling. Given a sampled scenario tuple (Role, Capability, Resources) from the structured task space, the process begins:

Role: the user persona (e.g., "financial analyst," "graduate student," "project manager")
Capability: the atomic capabilities the task should exercise (e.g., "formula manipulation + chart generation")
Resources: the specific files and applications initialized in the environment (from the hybrid resource injection)

Stream 1: Instruction generation. The architect VLM formulates a natural language query g grounded in the specific resource context. For instance, given a role of "financial analyst," the capability "formula manipulation," and an Excel file containing quarterly sales data, the architect might generate: "Calculate the quarter-over-quarter growth rate for each product category and highlight any categories with negative growth in red."

Stream 2: Validator generation. Simultaneously, the architect generates the ground truth and the corresponding executable evaluator code. This code defines "the precise success conditions for the task." The validator program V_g is an executable script (written in Python or a similar language) that:

Accesses the final environment state after the agent's trajectory completes
Inspects relevant files, application data, or system configurations
Returns a boolean indicating whether the task objective is satisfied

The paper provides the example of a spreadsheet task: the validator would open the final Excel file, check specific cell values against expected computations, verify that formatting conditions are met, and return True only if all conditions match.

Closed-loop feedback mechanism. To guarantee executability, the generated validator code is immediately executed in a real sandbox environment. The execution results—including output files from successful runs and error messages from failed executions (syntax errors, API mismatches, incorrect file paths)—are fed back to the architect VLM. This feedback loop iterates multiple rounds until the execution succeeds and passes quality checks. The paper notes: "This process iterates multiple rounds until the execution succeeds and passes quality checks."

Validator tool library. To enhance stability and reduce the rate of validator code errors, the authors abstract "frequently used verification logic into a standardized tool library." This provides the architect VLM with pre-built, tested functions for common verification operations (checking cell values, verifying file existence, comparing text content), reducing the need for the VLM to generate error-prone low-level code.

Standardized formatting. Finally, the valid (g, V_g) tuple is formatted into "a standardized JSON structure compatible with established benchmarks like OSWorld," ensuring that the synthesized tasks use the same evaluation interface as the benchmark tasks used for final evaluation.

Rigorous Quality Assurance

The raw synthesized pairs undergo a rigorous filtering protocol to eliminate three categories of defects.

Consistency-based filtering. A reference computer-use agent performs sandbox rollouts on each synthesized task. The filtering pipeline enforces multiple quality gates:

Tasks that fail to complete the rollout due to parameter configuration anomalies return error messages to the ReAct-based agentic workflow for modification (not discarded—the synthesis loop iterates until a valid task is produced).
For tasks with successful rollouts, the system calculates pass rates using both a reward model and the executable validator, organized by hierarchical domain taxonomy.
Human operators perform manual spot checks on tasks where the pass rates from the reward model and the validator show "significant discrepancies." If manual inspection identifies clear validator failures (false positives where the validator incorrectly marks success, or false negatives where it incorrectly marks failure), the ReAct-based workflow is refined to mitigate these issues.
Only tasks that are "cross-verified by the sandbox rollout, the reward model, and manual inspection" are preserved in the final dataset.

Tri-fold decontamination. Because the synthesis engine uses powerful VLMs that "may inadvertently reproduce benchmark content from their vast pre-training corpora," the paper enforces three layers of decontamination:

Semantic decontamination: LLM-based filtering removes instructions that are semantically equivalent to benchmark queries, even if the wording differs (paraphrase detection applied to task intent).
Configuration decontamination: Tasks with identical application initialization settings within certain domains are pruned to prevent the agent from memorizing specific environment configurations rather than learning general strategies.
Evaluator decontamination: The system verifies that generated success conditions and ground truth files do not overlap with existing evaluation scripts from known benchmarks, preventing the agent from being tested on tasks whose validators it may have encountered during training.

The paper states that through this pipeline, "we have successfully scaled verifiable training data to tens of thousands of instances, effectively breaking the bottleneck of manual data curation."

Scalable Interaction Infrastructure

The transition from static data scaling to evolving experience learning requires an infrastructure capable of generating "continuous, diverse, and interactive feedback at a massive scale." The paper describes a "unified environment sandbox platform" that orchestrates "hundreds of thousands of daily sandbox sessions" and processes "millions of interaction requests per day with industrial-grade stability."

Architecture and Abstractions

The platform is architected around two core abstractions:

Tools. A tool encapsulates "the immutable definition of a simulation environment," including version-controlled system images and exposed interaction APIs. The platform supports "hundreds of distinct environment types, ranging from generic benchmarks to specialized agentic environments." This abstraction decouples environment iteration from experimentation—when a new application version or OS configuration is needed, a new tool version is created without affecting existing experiments that depend on previous tool versions. This ensures "backward compatibility and reproducibility."

Clusters (Dynamic Scaling Units). A cluster represents the runtime instantiation of a tool and serves as "the fundamental unit for environment scaling." Users specify tool types and configure resource quotas to instantly provision customized environment services for distinct workloads. The abstraction allows the infrastructure to dynamically scale environment instances "from a handful of debugging sessions to tens of thousands of concurrent training nodes—without resource contention or cross-contamination."

High-Throughput Orchestration

Asynchronous gateway. The infrastructure relies on "an asynchronous gateway service based on the reactor pattern for non-blocking I/O" that achieves "a routing throughput at the scale of hundreds of thousands of requests per minute." The reactor pattern decouples the control plane (lifecycle management—starting, stopping, and monitoring sandbox instances) from the data plane (environment interaction—sending actions, receiving observations). This prevents long-running environment executions from blocking critical routing logic, ensuring that new sandbox provisioning requests are handled immediately regardless of how many active sessions are running.

Distributed scheduler. The scheduler manages the lifecycle of massive sandbox images using "distributed sharding and resource pooling" to achieve "high-efficiency node scheduling." Its critical capability is burst scaling: it can bootstrap "tens of thousands of sandbox instances within one minute." This rapid instantiation ensures that "the environment scaling strictly matches the training demand of on-policy reinforcement learning, minimizing the latency between policy updates and experience collection."

Scale metrics. The paper reports that "this resilient scheduling backbone enables the infrastructure to stably sustain over 100,000 concurrent sandboxes." At this scale, if each sandbox runs a trajectory averaging 20-50 steps, the total throughput of interaction steps is in the millions per day.

High-Fidelity Environment Instantiation

For computer-use tasks, the simulation fidelity of the virtual environment directly determines whether training experiences transfer to real deployment.

Hybrid virtualization. The paper implements "a hybrid virtualization architecture that encapsulates QEMU-KVM virtual machines within Docker containers." While Docker provides compatibility with the orchestration layer (container scheduling, networking, resource limits), the internal execution relies on QEMU with KVM hardware acceleration for near-native performance. The authors "construct a customized QEMU launch sequence that explicitly disables non-essential peripherals while optimizing I/O performance." This nested design ensures "strict kernel-level isolation—crucial for security when agents execute arbitrary code—while maintaining near-native performance for GUI rendering and I/O operations."

Deterministic environment calibration. The paper constructs "a customized OS image based on Ubuntu 22.04" with specific patches to address the gap between simulation and real-world deployment:

Input determinism (HID patching): Standard virtualization often suffers from key mapping collisions where the symbolic intent of a keypress does not match the realized character. The authors calibrated the human interface device mapping at the xkb kernel level, specifically modifying the /usr/share/x11/xkb/symbols/pc definitions to "resolve symbolic collisions (e.g., the < vs > shift-state error in US layouts), ensuring that the agent's symbolic intent strictly matches the realized character input." Without this patch, an agent trained to press the < key might learn that pressing the key labeled < sometimes produces > depending on subtle virtualization state, introducing non-determinism that corrupts the training signal.
Rendering consistency: To prevent layout shifts in office software that could confuse visual agents (fonts rendering at different widths causing text reflow, button positions shifting), the authors "injected a comprehensive suite of proprietary fonts directly into the system font cache (fc-cache)." This guarantees "that documents render identically to their native counterparts" across different sandbox instances and across training/deployment environments.
Runtime stability: The OS image is "hardened with system-level proxy configurations to resolve network instabilities and includes pre-installed dependencies like xsel and qpdf to eliminate common runtime errors during clipboard operations and PDF processing." These are practical engineering fixes for failure modes that would otherwise cause trajectory crashes and reduce effective throughput.

Cold-Start: Initializing the Behavioral Prior

Before the agent can learn from its own experiences, it needs a robust initial policy that can interact with the environment effectively enough to generate useful trajectories—both successful ones to learn from and failed ones that reveal capability boundaries. The cold-start phase constructs this initial policy π_init through supervised learning on carefully synthesized demonstrations.

Unifying the Action Space

The paper implements Semantic Action Mapping to construct the unified action space A, transforming raw event streams into structured primitives with two primary components:

Physical Interaction (A_mouse ∪ A_keyboard). Mouse events use coordinate-based positioning ({coordinate: (x, y)}) and keyboard events use text or key-code arguments. The critical innovation is the Stateful Interaction mechanism: by decoupling discrete key presses into key_down and key_up events as separate primitives, the policy can maintain active modifier states. For example, to perform a Shift+Click for multi-selection, the agent outputs key_down(keys: ["shift"]), then left_click(coordinate: (x, y)), then key_up(keys: ["shift"]). This three-step sequence maintains the Shift key held down during the click, which would be impossible with a single atomic "shift+click" primitive because the agent would need to know in advance which combination it intends. The stateful separation allows the policy to decide mid-sequence to add modifiers.

Control Primitives (A_control). Two meta-actions manage execution flow:

wait: pauses execution for a specified duration, allowing the agent to handle asynchronous UI rendering (e.g., waiting for a web page to load, a dialog to appear, or an animation to complete). Without this primitive, the agent would have no mechanism to synchronize with the environment's timing.
terminate: signals task completion with a status argument of either "success" or "failure". This gives the agent explicit control over when to stop, rather than relying on a fixed step limit or an external termination detector.

Structuring the Thought Space

To enable interpretable and robust decision-making, the paper defines a Reasoning Schema for the thought space Z that imposes structural constraints at key decision points. This schema is not just a format guideline—it is enforced through prompt templates during the hindsight reasoning generation process that produces the cold-start training data.

Goal Clarification (z_0). At the initial step t = 0, the agent is required to "explicitly paraphrase the user's objective." This serves two purposes: it clarifies ambiguous instructions (disambiguating what the user actually wants), and it grounds the subsequent planning process by establishing an explicit goal representation that future reasoning steps can reference.

Observation Consistency (z_obs). For each intermediate reasoning trace, the agent must include "a concise summary of key visual elements" and maintain "strict semantic consistency between this textual summary and the actual observed state." This directly targets hallucination—by requiring the agent to verbalize what it sees, the training signal penalizes cases where the agent claims to see elements that are not present in the screenshot, or fails to notice elements that are critical for the task.

Self-Verification (z_check). Before issuing the final termination signal, the agent is prompted to "execute auxiliary interaction steps (e.g., checking a file status) to visually confirm that the execution result aligns with the user's instruction." This encodes the behavior of double-checking work before declaring success—opening the saved file to verify its contents, scrolling to confirm all rows are visible, checking that formatting was applied correctly. These verification actions are part of the trajectory and are trained through supervised learning.

Reflection and Correction (z_reflect). The paper leverages failed rollouts for error correction through a specific procedure:

Identify a critical error step in a failed trajectory.
Restore the environment to the pre-error state (the state just before the error-causing action was taken).
To account for sandbox non-determinism, strictly filter for state consistency between the restored environment and the original trace—if the restored state does not match the original pre-error state, the trajectory is discarded because the recovery would be trained on inconsistent context.
From the valid restored state, induce self-correction using high-temperature sampling to generate successful remedial paths. The high temperature encourages exploration of alternative actions, and only the trajectories that successfully complete the task from the restored state are kept.

This produces training data where the agent learns to recognize its own errors and take corrective actions—a capability that static imitation on success-only trajectories cannot teach.

Reasoning-Augmented Termination (z_T). To prevent the model from overfitting to the termination label (learning to output terminate at a specific step count or after a specific visual pattern rather than after genuine task completion), the terminate action must be strictly conditional on a preceding reasoning trace. This trace requires the agent to "explicitly synthesize visual evidence to justify task completion, ensuring the decision is grounded in logic rather than memorized patterns." For example, before terminating with status: "success", the agent must generate a reasoning trace like "I can see the Max column is populated with values for all rows, and the chart on the right has updated to reflect the new data, confirming the task is complete."

Hindsight Reasoning Generation

The cold-start dataset D_prior is constructed by leveraging foundation VLMs (Qwen3-VL, OpenCUA) within a modular framework to synthesize high-quality interaction traces. The key technique is Hindsight Reasoning Generation: treating the ground-truth execution path as known future information, the system "retrospectively generates reasoning traces z_t that explain the observed actions, thereby augmenting physical trajectories with coherent cognitive chains."

The full procedure is given in Algorithm 1, which processes a raw trajectory τ = {(o_t, a_t)} and generates reasoning traces Z = {z_t} by querying a general VLM with context-aware prompt templates that vary by execution phase:

For t = 0 (initialization): the model is prompted to clarify the task goal and establish a high-level plan, using a first-person perspective.
For intermediate steps (0 < t < T): the model is prompted to describe what changed in the environment and why the next action advances the workflow, with an explicit constraint to avoid mentioning raw pixel coordinates and instead describe UI elements semantically.
For t = T (termination): the model is prompted to assess the final screenshot against the initial instruction and provide visual evidence of task completion before emitting the terminate signal.
For error recovery trajectories: a Reflection header is prepended, and the prompt includes the root cause analysis from the prior failure.

Cold-start training details. The multi-turn trajectories are decomposed into single-turn samples for training. The input context retains full multimodal details (screenshots, reasoning, and actions) only for the most recent five steps, while earlier history is compressed into text-only semantic actions. The training loss is computed exclusively on the current step's reasoning and action tokens. To preserve general foundation capabilities (STEM, OCR, visual grounding, text-based reasoning), a diverse mixture of general-purpose data is incorporated, with volume balanced to match the scale of the decomposed single-turn trajectory samples.

The paper reports that the cold-start phase uses "approximately 1k high-quality trajectories," emphasizing that the goal is not massive data volume but pattern diversity—establishing the action space grounding and output formatting patterns that subsequent interactive training will build upon. The authors explicitly note: "a lightweight cold start is sufficient to establish a latent alignment... A heavy cold start often yields high supervised metrics but creates a checkpoint that is harder to refine later."

Rejection Sampling Fine-Tuning (RFT)

The rejection sampling phase is where the agent transitions from learning from human-specified demonstrations to learning from its own experiences. The objective is "to consolidate the agent's ability to solve tasks by learning exclusively from high-quality, successful executions."

Dynamic Compute Budgeting

The key insight driving the compute allocation strategy is that uniform exploration is inefficient: tasks the agent already solves reliably produce redundant successes, while tasks it never solves produce only uninformative failures. The training signal is concentrated in boundary tasks where the agent oscillates between success and failure.

The paper formalizes this through a hierarchical budget spectrum paired with descending success rate thresholds:

$K = \{k_1, \ldots, k_n\}, \quad \Lambda = \{\tau_1, \ldots, \tau_n\}$

where K is a set of rollout budget levels (e.g., k_1 = 4, k_2 = 16, k_3 = 64 generations) and Λ is the corresponding success rate thresholds (e.g., τ_1 = 0.8, τ_2 = 0.5, τ_3 = 0.2).

For a given task query g drawn from the synthesis engine, the system identifies the optimal rollout budget K* that satisfies:

$K* = k_{i*}, \quad \text{where} \quad i* = \min\{i \mid SR(k_i) \geq \tau_i\}$

where SR(k_i) represents the observed pass rate (success rate) when using k_i rollout attempts.

What it computes: for each task, the system allocates the smallest rollout budget that achieves a target success rate. If the agent can solve the task with high probability using few attempts (high SR at small k), the budget is kept small. If the agent requires many attempts to achieve a reasonable success rate (low SR at small k), the budget is increased.

Why this form: this strategy "effectively prunes efficiently solved tasks and concentrates computational power on boundary queries—tasks where the policy exhibits high variance." Tasks with SR(4) = 0.95 are trivially solved—generating 64 trajectories would produce mostly redundant successes. Tasks with SR(64) = 0.05 are beyond the agent's current capability—generating more trajectories would produce mostly failures with no corrective signal. The sweet spot is tasks where SR increases meaningfully with budget, indicating that the agent can succeed but needs exploration to find the correct path.

Step-Level Denoising

Even successful trajectories contain significant noise. The paper identifies specific failure modes in raw successful rollouts:

Action aliasing: the agent outputs conflicting actions for a single state (redundant clicks, unnecessary navigation)
Cyclic repetition: the agent endlessly clicks the same coordinates without progressing the task
Redundant intermediate steps: extra verification actions that do not contribute to task completion

The paper applies a judge model (not further specified in detail) to analyze successful trajectories and "mask out redundant steps." This filtering is especially important for the handling of infeasible tasks: for trajectories where the task cannot be completed (the agent correctly identifies that the task is impossible given the environment state), "we remove all intermediate actions and strictly keep the reasoning trace and the final terminate=failure action." This ensures the agent learns that recognizing infeasibility and reporting failure is a valid and valuable behavior, but does not learn the specific (unnecessary) actions it took before reaching that conclusion.

The refined trajectories are aggregated into the experience pool B. The paper reports scaling "our high-fidelity experience pool B to tens of thousands of trajectories" through this process. These domain-specific experiences are interleaved with "a balanced corpus of general-purpose multimodal data to prevent catastrophic forgetting" of the agent's general vision-language capabilities.

Experience scaling results. The paper provides concrete scaling numbers in Table 5 for an early iteration of the OpenCUA-72B model (omitting cold-start and DPO to isolate RFT effects):

Stage	Data Size	Gain (∆%)
RFT Round 1	20k	+2.61
RFT Round 2	226k	+6.79
RFT Round 3	1M	+8.12

The diminishing returns from Round 2 to Round 3 (from +4.18 over baseline to +1.33 additional) motivates the authors' observation that "as model capabilities improve with scale, the tolerance for noise decreases, creating a bottleneck for existing iterative methods." This directly motivates the transition to DPO-based optimization, which extracts higher-quality learning signals from the data.

Reinforcement Learning via Step-Level DPO

While RFT consolidates what the agent can already do, it does not explicitly teach the agent to avoid what it does wrong. The DPO phase addresses this by learning from failure trajectories through structured preference optimization at critical decision points.

Why trajectory-level DPO fails for GUI agents. The paper identifies a specific failure mode with standard trajectory-level preference optimization: state misalignment. In long-horizon tasks, a failed trajectory τ^- and a successful trajectory τ^+ on the same task will likely diverge early in the execution, meaning that the states at corresponding step indices are different. Directly contrasting actions at the same step index between two trajectories with different states is meaningless—the actions were chosen in response to different observations. The paper's solution is to identify the Critical Forking Point: the exact step where the failed trajectory diverged from a viable path, and construct preference pairs only at that step.

Causal Deviation Discovery

The paper employs a Reference-Guided Diagnosis mechanism to identify the critical error step:

Given a failed rollout τ^- and a successful reference τ^+ (retrieved from the same or a semantically equivalent task), the system identifies the Critical Deviation Step t* as "the first timestamp where the agent's action diverges from the reference, despite the environmental states remaining functionally equivalent."

What this means operationally: the algorithm scans through both trajectories step by step, comparing the environmental states (screenshots) at each step. For steps t < t*, the states in both trajectories are "functionally equivalent"—the same applications are open, the same files are visible, the same UI elements are present, even if pixel-level details differ (scroll position, window placement). At step t*, the agent in the failed trajectory takes an action (z^-_{t*}, a^-_{t*}) that is different from what would have kept it on the successful path, and this action causes the subsequent states to diverge. The specific response (z^-_{t*}, a^-_{t*}) is the "rejected" sample in the preference pair.

Algorithm 2 provides the formal procedure: it iterates through each potential error step index in the failed trajectory, searches within a window [t - w, t + w] in the reference trajectory for a state-action pair that aligns with the failed trajectory's state at step t, and uses coordinate normalization to map the reference action to the failed trajectory's screen coordinates.

Structured Preference Construction

Once the critical error (z_l, a_l) = (z^-_{t*}, a^-_{t*}) is identified, the system constructs preference pairs under two complementary paradigms.

Paradigm I: Action Correction (At Step t*). The objective is to replace the rejected error (z_l, a_l) with an optimal chosen response (z_w, a_w). The chosen response is obtained through one of two mechanisms:

Window-based reference alignment: Migrate thoughts and actions from the successful reference trajectory τ^+ via VLM semantic matching. The VLM identifies the step in τ^+ whose state matches the failed trajectory's state at t*, then adjusts the reference action's coordinates to match the failed trajectory's screen layout.
Visual-grounded synthesis: When no alignment exists (the reference trajectory does not have a corresponding state), synthesize fresh traces via a general model that takes the failed trajectory's state at t* as input and generates a corrected action.

This paradigm teaches the agent the correct action to take at the exact moment it previously made an error.

Paradigm II: Reflection and Recovery (At Step t* + 1). Even with the correct action at t*, the agent may still find itself in unexpected states due to cascading errors or environmental non-determinism. This paradigm addresses the state immediately after the error (t* + 1):

The agent's blind continuation (its actual action at t* + 1 in the failed trajectory) is treated as the rejected sample.
For the chosen sample, the system synthesizes a Reflection Trace where, instead of acting blindly, the agent is trained to (1) halt and observe the unexpected screen state, and (2) generate a reasoning chain that formulates a remedial plan.

The paper describes this as teaching the agent to recognize that it has deviated from the expected path and to switch from execution mode to recovery mode.

DPO Optimization Objective

The policy π_θ is optimized using Direct Preference Optimization (DPO). Consistent with the paper's formulation where the policy generates both a reasoning trace z and an action a conditioned on history h_t and observation o_t, the loss is:

$J(\theta) = -\mathbb{E}_{(h_t, o_t, (z,a)_w, (z,a)_l) \sim D} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(z_w, a_w \mid h_t, o_t)}{\pi_{ref}(z_w, a_w \mid h_t, o_t)} - \beta \log \frac{\pi_\theta(z_l, a_l \mid h_t, o_t)}{\pi_{ref}(z_l, a_l \mid h_t, o_t)} \right) \right]$

where:

(z_w, a_w) is the chosen (preferred) reasoning-action pair at the critical step
(z_l, a_l) is the rejected (dispreferred) reasoning-action pair that caused the trajectory to fail
π_θ is the current policy being optimized
π_ref is a frozen reference policy (typically the policy before the current optimization round)
β is a temperature parameter controlling the strength of the preference update
σ(·) is the logistic sigmoid function
The expectation is taken over the dataset D of constructed preference pairs

What it computes: for each preference pair, the loss function increases the log-ratio of the current policy's probability of generating the chosen response relative to the reference policy, while simultaneously decreasing the log-ratio for the rejected response. The sigmoid σ(·) maps this difference to a probability, and the negative log-likelihood encourages this probability to approach 1 (the chosen response should be relatively more probable under the current policy than under the reference). The β parameter controls how far the policy can deviate from the reference—higher β allows larger updates but risks overfitting to the preference data.

Why this form: DPO directly optimizes the policy to prefer the chosen over the rejected response without training a separate reward model. This is important because training a reward model on GUI interaction data would require learning to score arbitrary (observation, reasoning, action) triples—a challenging regression problem. DPO bypasses this by optimizing the policy's relative probabilities directly from pairwise preferences. The step-level construction (applying DPO at individual error steps rather than on full trajectories) is what makes this feasible for long-horizon tasks: it isolates the learning signal to the specific decision point where the error occurred, rather than diluting the signal across dozens of steps where the agent's behavior was correct.

Iterative Training: Closing the Evolutionary Loop

The paper's training procedure is not a single pass through the stages but an iterative cycle. After one round of RFT + DPO, the updated policy is redeployed to the sandbox infrastructure to generate new trajectories, which in turn feed the next round of optimization. The paper reports:

For EvoCUA-32B (Table 3): an additional iteration of the entire evolutionary cycle (RFT + DPO) yields a further +1.90% improvement beyond the first DPO round.
For OpenCUA-72B (Table 4): iterative training adds +1.82%.

These gains, while smaller than the first-round improvements, confirm "the self-sustaining nature of our paradigm, where the model iteratively refines its capability boundary through recursive synthesis and correction." The diminishing returns are expected—as the policy improves, the space of remaining errors shrinks, and each successive round extracts signal from increasingly subtle failure modes.

Proposed Future Direction: STEPO Algorithm for Online RL

Section 7 introduces a proposed algorithm that was not fully evaluated in the paper's main results but addresses a fundamental challenge for future work: applying online RL (specifically GRPO-style methods) to GUI agents with truncated context windows.

The Training-Inference Discrepancy

The paper identifies a critical mismatch between how GUI agents operate during training versus inference:

During inference, to manage context length, the agent retains full multimodal details (screenshots, reasoning, actions) only for the most recent five steps, while earlier history is compressed into text-only semantic actions. This means that when generating the action at step T, the model sees the actual screenshots only for steps T-4 through T.

However, trajectory-level RL algorithms like GRPO (Group Relative Policy Optimization) compute advantage values at the trajectory level and backpropagate through the final step's token probabilities. The paper states:

"If the trajectory of the final step is directly used for training, the model will not be able to learn the supervision signals of intermediate steps."

The reason: for intermediate steps (those beyond the five-step window from the end), the model at training time sees a different context than it saw at generation time. During generation, step 3 sees the actual screenshot from step 3. During trajectory-level training on the final step, step 3's screenshot has been compressed into text, so the model cannot learn to improve its behavior at step 3. This creates a training-inference discrepancy where the learning signal cannot propagate back through the compressed history.

STEPO: Step-Level Policy Optimization

The proposed STEPO algorithm addresses this by allocating trajectory-level advantages to individual steps and training on all steps independently.

For a trajectory τ with length T, each step t ∈ {1, ..., T} contains K_t tokens. The token at position k of step t in the i-th trajectory is denoted x_{i,t,k}.

Advantage computation. Similar to GRPO, STEPO samples a group G of trajectories {τ_1, ..., τ_n} for a given query and computes advantages at the trajectory level:

$\hat{A}_i = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}$

where R_i is the reward (binary success/failure) of trajectory τ_i. This is the standard GRPO advantage: the trajectory's reward minus the group mean, divided by the group standard deviation, giving a normalized score where positive values indicate above-average trajectories.

Step-level allocation. The key innovation: instead of applying this advantage only to the final step, it is evenly distributed across all steps:

$\hat{A}_{i,t} = \hat{A}_i / T_i, \quad t \in \{1, 2, \ldots, T_i\}$

where T_i is the number of steps in trajectory τ_i. All tokens within the same step share the same step-level advantage.

Optimization objective. The STEPO objective is:

$J_{STEPO}(\theta) = \mathbb{E}_{[q \sim P(Q), \{\tau_i\}_{i=1}^G \sim \pi_{\theta_{old}}(T \mid q)]} \frac{1}{G} \sum_{i=1}^G \sum_{t=1}^{T_i} \frac{1}{K_t} \sum_{k=1}^{K_t} \left\{ \min \left[ r_{i,t,k}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t,k}, 1 - \epsilon_{low}, 1 + \epsilon_{high}) \hat{A}_{i,t} \right] - \beta D_{KL}(\pi_\theta \parallel \pi_{ref}) \right\}$

where:

r_{i,t,k}(\theta) = π_θ(x_{i,t,k} | q, x_{i,t,<k}) / π_{θ_old}(x_{i,t,k} | q, x_{i,t,<k}) is the importance sampling ratio for the k-th token of step t in trajectory i
ε_low and ε_high are asymmetric clipping parameters (PPO-style clipping to prevent excessively large policy updates)
D_KL(π_θ ∥ π_ref) is a KL divergence penalty term with coefficient β to prevent the policy from drifting too far from the reference

What it computes: for each step in each trajectory, the objective increases the probability of token sequences from high-advantage trajectories (successful trajectories in groups where most others failed) and decreases the probability of token sequences from low-advantage trajectories (failed trajectories in groups where most others succeeded). The clipping and KL penalty constrain the magnitude of updates.

Why this form: by uniformly allocating the trajectory advantage to all steps, the algorithm achieves two effects:

Efficiency pressure on good trajectories: High-advantage trajectories (successes) have their advantage spread across all steps. If the trajectory took many steps to succeed, each step receives a smaller individual advantage. This creates pressure to complete tasks with fewer steps, reducing redundant execution.
Exploration incentive for poor trajectories: Low-advantage trajectories (failures) have their negative advantage spread across all steps. If the trajectory failed quickly, each step receives a larger negative advantage. This creates incentive for failed trajectories to explore more steps before giving up, potentially discovering a path to success.

Preliminary results. The paper reports that STEPO was tested on OpenCUA-32B in online RL training. Figure 7 shows that "the training performance of STEPO is significantly superior to that of GRPO trained with final trajectories." However, the paper acknowledges a limitation: "STEPO suffers from the issue of high training cost, as the number of updates to the policy model multiplies significantly" (since every step becomes a training sample rather than just the final step). The authors hypothesize that "the requirements for step-level training may not be uniform across different training phases, and training only specific key steps might also achieve comparable performance to training all steps," pointing to future work on selective step-level optimization.

Summary of Design Choices and Their Justifications

Binary verifiable rewards over learned/scalar rewards: deterministic validator programs eliminate reward ambiguity and reward hacking; the agent cannot learn to produce plausible-looking failures if the validator checks ground truth environment state.
Co-generation of tasks and validators: ensures every training task has a precise, executable success condition; prevents the common failure mode where synthetic task instructions are ambiguous or describe infeasible tasks.
Closed-loop validator synthesis: executing generated validators in sandboxes and feeding errors back to the generator VLM guarantees that all validators in the training pool are actually executable and produce correct judgments.
Hybrid virtualization (Docker + QEMU-KVM): Docker provides orchestration compatibility (container scheduling, networking) while QEMU-KVM provides kernel-level isolation and near-native GUI performance; necessary for security when agents execute arbitrary code and for rendering fidelity when training visual agents.
Deterministic input calibration (xkb patching): standard virtualization key mapping errors would introduce non-determinism where the same symbolic action produces different characters in different sandbox instances, corrupting the training signal for keyboard-based tasks.
Stateful action space (separate key_down/key_up): enables modifier-key combinations that are essential for complex GUI operations (Shift+Click, Ctrl+drag) without requiring the action space to enumerate all possible key combinations.
Hindsight reasoning generation: by generating reasoning traces after seeing the full successful trajectory, the cold-start data establishes coherent cognitive chains that explain actions in context; doing this prospectively (generating reasoning before knowing whether the action succeeds) would produce less coherent training data.
Lightweight cold start over heavy supervised initialization: the authors explicitly observe that "a heavy cold start often yields high supervised metrics but creates a checkpoint that is harder to refine later"; a minimal initialization that establishes action space grounding and output formatting, followed by interactive refinement, produces better final performance.
Dynamic compute budgeting over uniform exploration: allocates rollout budget to boundary tasks where the agent's success rate is in the informative middle range (20-80%), avoiding wasted computation on already-mastered or still-impossible tasks.
Step-level denoising of successful trajectories: raw successful rollouts contain redundant steps that, if trained on, cause "action aliasing" and "cyclic repetition"; filtering to essential steps improves the signal-to-noise ratio of RFT data.
Step-level DPO at critical forking points over trajectory-level DPO: isolates the learning signal to the specific decision step where the error occurred, avoiding the state misalignment problem where contrasting actions at the same step index between divergent trajectories is meaningless.
Dual-paradigm preference construction (Action Correction + Reflection): teaches both the correct action to take (Paradigm I) and the meta-cognitive skill of recognizing errors and switching to recovery mode (Paradigm II); the latter addresses cascading failures where the agent continues executing despite being in an unexpected state.
STEPO over GRPO for online RL: addresses the training-inference discrepancy that occurs when trajectory-level RL algorithms are applied to agents with truncated context windows; by allocating advantages to all steps, STEPO ensures that intermediate steps receive learning signals even when their visual context has been compressed.

4. Key Insights and Innovations

Innovation 1: Reframing Computer-Use Agent Training as a Self-Sustaining Experience Cycle, Not a Static Data Scaling Problem

The dominant conceptual model for training GUI agents prior to this work treated the problem as supervised learning over demonstration data. You collect a dataset of (screenshot, action) pairs, fine-tune a VLM to predict actions from screenshots, and hope that the resulting policy generalizes. This is behavior cloning, and its limitations are well-understood in robotics and control theory—compounding errors, covariate shift, inability to recover from out-of-distribution states—but the field of computer-use agents had not yet fully internalized these lessons because the perceived bottleneck was data scarcity, not data quality.

EvoCUA's fundamental intellectual move is to reject the premise that more demonstration data is the path forward and to replace the entire mental model with a closed-loop experience cycle. In this framing, the agent does not learn from a fixed corpus; it learns from the consequences of its own actions in a real environment. The training signal is not "what action should follow this screenshot" but "did the sequence of actions I chose lead to the outcome I intended?" This is a shift from imitation learning to reinforcement learning at the conceptual level, but it goes further because the RL loop itself is bootstrapped—the system generates its own tasks and its own reward signals, making it truly self-sustaining.

What makes this more than just "applying RL to GUI agents" (which prior work like UI-TARS-2 attempted) is the integration of task generation into the learning loop. The paper's synthesis engine does not merely produce training data; it produces a dynamic task distribution T_syn(· | π_old) that can be conditioned on the agent's current capabilities (Section 2.2, Equation for J(θ)). This means the training curriculum is not designed by human engineers but emerges from the agent's own performance profile. Tasks that the agent has mastered recede from the training distribution; tasks at the boundary of its abilities become more prominent. This creates an automatic curriculum that continuously pushes the agent's capability frontier outward without human intervention.

Prior work in this space—OpenCUA with its AgentNet dataset, UI-TARS-2 with its multi-turn RL on collected demonstrations, Step-GUI with step-wise reasoning—all operated within the static-data paradigm, even when they incorporated RL. They either collected data once and trained on it, or collected data through human-specified task distributions. EvoCUA's task distribution is algorithmically generated and algorithmically adaptive, which is the property that makes the "self-sustaining cycle" metaphor meaningful rather than merely rhetorical.

The evidence for the significance of this reframing is not a single ablation number but the architecture of the entire paper. Figure 2 places the cycle at the center of the system diagram. The three core contributions—synthesis engine, scalable infrastructure, evolving optimization—are not independent modules but components of a single loop where each feeds the next. The paper explicitly contrasts this with "passive imitation of fixed, non-interactive datasets" (Section 1) and positions the entire work as a "paradigm shift" rather than an incremental method. Whether one accepts the paradigm-shift claim depends on how fundamental one considers the static-data bottleneck to be; the paper's argument is that it is not merely a practical limitation but a structural mismatch that no amount of data scaling can overcome.

This is a fundamental shift in problem framing, not an incremental refinement. The prior paradigm (collect demonstrations, imitate) had clear diminishing returns; the new paradigm (generate tasks, execute, verify, optimize, repeat) theoretically has no such ceiling because the task distribution can expand as the agent improves. The practical ceiling becomes the quality of the synthesis engine and the fidelity of the sandbox environment, not the quantity of pre-existing data.

Innovation 2: The "Generation-as-Validation" Principle as a Solution to Reward Ambiguity

A less obvious but equally important conceptual contribution is the paper's specific approach to reward definition. In standard RL, the reward function is assumed to be given by the environment. In LLM-based RL for reasoning (DeepSeek-R1, GRPO-based methods), the reward is typically defined by answer-matching or rule-based verification of final outputs. In GUI agent training, the reward is particularly challenging because task success is often context-dependent and visually subtle—did the agent really update the correct cell in the spreadsheet, or did it update a cell that merely looks like the correct one? Did it format the chart correctly, or did it produce something that a human would need to inspect to verify?

The field's default approach to this problem has been to either (a) use human evaluation, which does not scale, (b) use LLM-as-judge, which introduces ambiguity and potential reward hacking, or (c) define tasks narrowly so that success can be verified by simple string matching. All three approaches have fundamental limitations.

EvoCUA's solution—the "generation-as-validation" paradigm—is distinctive because it co-generates the task and its verification program simultaneously, using the same foundation model in a ReAct-style agentic loop. The key insight is that generating a task description and generating a verifier are complementary generative acts that can be performed by the same model within a single workflow. When the task architect VLM formulates the instruction "Calculate quarter-over-quarter growth rates," it simultaneously generates the Python script that will check whether the final spreadsheet contains the correct values in the correct cells. The verifier is not an afterthought or a separate system; it is an integral output of the same synthesis process.

This co-generation has a property that the paper exploits but does not state explicitly: the verifier serves as a formal specification of the task. The natural language instruction g is inherently ambiguous—"update the chart" could mean many different things. The verifier program V_g is a denotation of that instruction—it specifies exactly what condition must hold for the task to be considered complete. By requiring that the verifier be executable and that it be tested in a sandbox before the task enters the training pool (the closed-loop feedback mechanism in Section 3.2), the system ensures that every training task has a well-defined, machine-checkable objective.

This distinguishes EvoCUA's approach from prior work on synthetic GUI data generation, where tasks were generated as text-only instructions and verification was either absent or relied on imprecise heuristics. The paper explicitly identifies the failure mode this solves:

"Merely synthesizing textual queries often leads to hallucinations, where the agent generates plausible plans for infeasible tasks."

The "hallucination" here is at the task-design level: a language model might generate a task that sounds reasonable but cannot actually be executed in the given environment, or whose success conditions are ambiguous. The co-generated validator, by being executable and tested, eliminates this category of error entirely.

The quality assurance pipeline (Section 3.3) further refines this principle. It does not simply trust the synthesized verifiers; it runs a reference agent in the sandbox, computes pass rates using both the verifier and a reward model, and flags discrepancies for human inspection. This creates a validation hierarchy: the verifier validates the agent's behavior, but the verifier itself is validated through execution and cross-checking. This layered approach to ground truth is what makes the system trustworthy enough to serve as the sole training signal for the RL phase.

This is a fundamental conceptual advance in how reward signals can be generated for open-ended interactive tasks. It is not specific to GUI agents—any domain where task success can be verified by a program that inspects environment state could potentially adopt the co-generation approach. The significance is that it converts the reward design problem from a manual engineering challenge (write verifiers for every possible task) into a generative modeling challenge (prompt a VLM to produce both tasks and verifiers, then filter for quality). This makes the approach scalable in a way that manual reward engineering could never be.

Innovation 3: Step-Level Preference Learning at Causal Error Points as a Strategy for Long-Horizon Correction

The paper's approach to learning from failures—step-level DPO at Critical Forking Points (Section 5.3)—represents a significant advance over how prior work handled error correction in sequential decision-making domains.

The standard approach to preference optimization for language agents applies DPO (or RLHF) at the trajectory level: given a good trajectory and a bad trajectory, the policy is updated to prefer the good one overall. For short-horizon reasoning tasks (math problems, code generation), this works adequately because the entire trajectory is a single chain of reasoning tokens. But for GUI interaction tasks that span 15-50+ steps, trajectory-level comparison suffers from a fundamental problem: credit assignment. If a trajectory fails at step 30, penalizing all 30 steps equally is both unfair (steps 1-20 might have been perfectly correct) and inefficient (the learning signal is diluted across many correct decisions).

What makes EvoCUA's approach distinctive is not the use of DPO itself—DPO is a standard algorithm—but the methodology for constructing the preference pairs. The paper introduces two specific mechanisms that together form a principled approach to error-based learning:

First, the Critical Forking Point concept. Rather than comparing full trajectories, the system identifies the exact step where the failed trajectory diverged from a viable path. This is done via Reference-Guided Diagnosis: comparing the failed trajectory against a successful reference trajectory on the same (or semantically equivalent) task and finding the first step where the actions differ while the environmental states remain functionally equivalent. This is a causal analysis, not a correlational one—it identifies the cause of the failure, not just the fact of failure.

This is conceptually related to the notion of "blame assignment" in program synthesis and the "first mistake" analysis in educational testing, but applied to interactive RL. The innovation is making this analysis automatic through VLM-based state comparison and coordinate alignment, rather than requiring human annotators to identify error steps.

Second, the dual-paradigm preference construction. Once the critical error step t* is identified, the system constructs two separate DPO preference pairs, not just one:

Paradigm I teaches the correct action at t* itself: given the state you were in, here is what you should have done instead.
Paradigm II teaches recovery at t* + 1: given that you made an error and now find yourself in an unexpected state, here is how to recognize the anomaly and formulate a recovery plan rather than continuing blindly.

The second paradigm is particularly innovative. Most error-correction methods in machine learning focus on preventing errors; they teach the model to avoid the mistake. Paradigm II teaches the model what to do after making a mistake, which is a meta-cognitive skill that static imitation on success-only trajectories cannot teach at all. The agent learns that when it observes a screen state that does not match its expectations, it should pause, reflect, and replan—rather than continuing to execute actions that are based on now-invalidated assumptions.

The paper describes this as "mimicking human learning dynamics," but the specific insight is that error recovery is a distinct skill from error avoidance, and both must be explicitly trained. An agent that only learns to avoid errors is brittle: when it inevitably makes an error (due to environmental stochasticity, visual ambiguity, or edge cases), it has no learned response and compounds the error. An agent that has learned recovery patterns can detect the discrepancy, backtrack, and try an alternative approach.

The ablation results support the significance of this dual-paradigm approach. Table 3 shows that the DPO phase contributes +3.21% absolute improvement for EvoCUA-32B, which is the second-largest single-stage gain after the unified action space (+4.84%) and larger than the RFT gain (+3.13%). This suggests that explicit error correction through step-level preference pairs captures improvement that neither cold-start imitation nor success-only rejection sampling can achieve.

This is a fundamental methodological advance in how to extract learning signals from failure trajectories in long-horizon sequential tasks. While the specific implementation uses DPO, the underlying concept—identify causal error points, construct preferences for both correction and recovery—is algorithm-agnostic and could be applied with other preference optimization methods.

Innovation 4: Empirical Demonstration That Interactive Experience Learning Transfers Across Model Scales and Architectures

While not a conceptual innovation in the same sense as the previous three, the paper's demonstration that the evolving experience paradigm yields consistent gains across fundamentally different model families and scales is an empirical finding with significant implications for how the field thinks about agent training.

The evidence comes from Tables 3 and 4, which show the cumulative gains from the same training pipeline applied to two very different starting points:

Training Stage	EvoCUA-32B (from Qwen3-VL-32B-Thinking)	OpenCUA-72B (from OpenCUA-72B)
Base	41.0%	45.0%
+Cold Start	+2.62%	+2.14%
+RFT	+3.13%	+3.69%
+DPO	+3.21%	+3.02%
+Iterative	+1.90%	+1.82%
Final	56.7%	(not fully reported in ablation)

The consistency of these incremental gains across model families—Qwen3-VL-Thinking versus OpenCUA, which have different architectures, different pretraining distributions, and different base capabilities—is striking. The gains are not identical (RFT contributes more on OpenCUA-72B, DPO slightly more on Qwen3-VL-32B), but the pattern of cumulative, monotonic improvement from each stage holds in both cases.

This is significant because it addresses a common concern about complex training pipelines: that they are overfitted to a specific model or that the gains are attributable to the base model's particular strengths rather than the training methodology. If the gains came only from Qwen3-VL's strong visual grounding or thinking capabilities, the pipeline would not transfer to OpenCUA. The fact that it does transfer—and with comparable magnitude—suggests that the pipeline is capturing something fundamental about how computer-use agents improve through interactive experience.

The paper also provides a negative result that strengthens this interpretation: the Qwen3-VL-based EvoCUA variants show performance declines on general multimodal benchmarks (Table 2—ScreenSpot-Pro drops from 57.10% to 49.76%, MMMU from 78.10% to 68.11%). The authors attribute this to "discrepancies in data distribution and patterns" in the general dataset used during fine-tuning, not to the evolving experience paradigm itself. The OpenCUA-based variant (EvoCUA-OpenCUA-72B) does not show these declines, likely because its general dataset was better matched. This negative result provides a boundary condition: the evolving experience paradigm improves computer-use capabilities, but the general data mixture matters for preserving broad capabilities, and getting it wrong can cause regression on non-computer-use tasks.

The most compelling single number for the paradigm's generalizability comes from the EvoCUA-8B result (Table 1): at 46.1% on OSWorld, this 8B-parameter model surpasses the specialized 72B-parameter OpenCUA-72B model (45.0%). This is not merely a scaling law observation; it demonstrates that the training paradigm can compensate for a 9× reduction in parameter count, at least within the capability range of the base model. The explicit comparison with Step-GUI-8B (40.2%, same backbone, different training) isolates the contribution of EvoCUA's training methodology from the base model's inherent capabilities.

This is an incremental empirical finding with fundamental implications. It does not introduce a new concept, but it provides the strongest evidence in the paper that the evolving experience paradigm is a general approach rather than a model-specific recipe. For practitioners deciding whether to adopt this training methodology, the cross-model and cross-scale evidence substantially reduces the risk that the gains are idiosyncratic.

Innovation 5: The Identification and Partial Solution of the Training-Inference Discrepancy in Context-Truncated RL

Section 7 of the paper introduces a problem that, while not fully solved, represents a significant diagnostic contribution: the training-inference discrepancy that arises when applying trajectory-level RL algorithms to GUI agents with truncated context windows.

The problem is subtle and easy to miss. During inference, the agent's context window is limited, so it retains full multimodal information only for the most recent five steps while compressing earlier history into text summaries. This is a practical necessity—storing dozens of high-resolution screenshots in the context would exceed memory limits. However, this compression creates a mismatch when trajectory-level RL algorithms (like GRPO) compute advantages at the trajectory level and backpropagate through only the final step:

"If the trajectory of the final step is directly used for training, the model will not be able to learn the supervision signals of intermediate steps." (Section 7)

The reason is that at training time, when processing the final step, the intermediate steps' screenshots have already been compressed into text. The model cannot learn to improve its visual grounding at step 5 because step 5's screenshot is no longer in the context when training on step 30. The signal about whether step 5's action was good or bad is present in the trajectory-level advantage, but the model lacks the visual information needed to improve its behavior at step 5.

This is a diagnostic contribution: the paper names and explains a failure mode that was likely present in prior work but not recognized. Many RL-for-GUI-agent papers likely trained with GRPO or PPO at the trajectory level and observed diminished returns without understanding why. EvoCUA's identification of the specific mechanism—context truncation preventing gradient flow to intermediate steps—provides a clear explanation and a clear direction for solution.

The proposed STEPO algorithm addresses this by allocating the trajectory advantage uniformly across all steps and training on each step independently (with its own contextual window, which includes the actual screenshots for that step's temporal neighborhood). This is a partial solution—the paper reports that "the training performance of STEPO is significantly superior to that of GRPO trained with final trajectories" (Figure 7)—but also acknowledges that STEPO "suffers from the issue of high training cost, as the number of updates to the policy model multiplies significantly." The authors further hypothesize that not all steps may need equal training attention, suggesting future work on selective step-level optimization.

This is an incremental algorithmic contribution built on an important diagnostic insight. The problem identification is more intellectually significant than the specific solution, because the problem (context truncation causing training-inference mismatch) will arise in any system that trains agents with limited context windows on long-horizon tasks. The STEPO solution is a reasonable first approach, but the paper is clear that it is not fully satisfactory due to cost. The value of this innovation is primarily in identifying the problem clearly and providing a baseline solution that future work can improve upon, rather than in claiming to have solved it definitively.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary benchmark is OSWorld (Xie et al., 2024), a representative testbed for open-ended computer-use tasks in real desktop environments. The paper uses the "OSWorld-Verified" variant (specified in Table 1), which consists of tasks with machine-verifiable success conditions. The exact number of test tasks is not explicitly stated in the main text, but it draws from OSWorld's structure of tasks spanning operating system operations, office software manipulation, web browsing, and multi-application workflows.
Base model(s). The paper validates across two distinct model families and three parameter scales: Qwen3-VL-Thinking (8B and 32B variants; Bai et al., 2025a) and OpenCUA (7B, 32B, and 72B variants; Wang et al., 2025b). Qwen3-VL-Thinking was chosen as a strong generalist VLM backbone with native thinking capabilities and dynamic resolution handling. OpenCUA was chosen as the previous open-source state-of-the-art specialized computer-use model, enabling direct comparison with an established baseline sharing the same architecture but trained under the static imitation paradigm.
Metrics. The primary metric is Success Rate (Pass@1) : the fraction of test tasks for which the agent's first trajectory (single rollout) achieves the task objective as verified by the OSWorld evaluator. This is reported as a percentage. For scaling analysis (Section 6.4), the paper additionally uses Pass@k: the fraction of tasks solved within k independent attempts (k = 8, 16, 32, 64, 128), capturing the model's capability ceiling when allowed multiple attempts. Pass@k is computed using standard unbiased estimation from finite samples. The paper also reports performance gain (Δ%) as the absolute difference in success rate between EvoCUA and its base model.
Baselines. The paper compares against a comprehensive set of both closed-weights and open-weights models (Table 1):

Closed-weights baselines: OpenAI CUA (OpenAI, 2025), Qwen3-VL-Flash (Bai et al., 2025a), UI-TARS-2-2509 (Wang et al., 2025a), Claude-4.5-Sonnet (Anthropic, 2025), and Seed-1.8 (ByteDance Seed Team, 2025).

Open-weights baselines: Qwen2.5-VL-32B-Instruct and Qwen2.5-VL-72B-Instruct (Bai et al., 2025b), ScaleCUA-32B (Liu et al., 2025), UI-TARS-72B-DPO and UI-TARS-1.5-7B (Qin et al., 2025), OpenCUA-7B, OpenCUA-32B, and OpenCUA-72B (Wang et al., 2025b), GUI-Owl-7B-Desktop-RL (Ye et al., 2025), Qwen3-VL-8B-Thinking, Qwen3-VL-235B-A22B-Thinking, and Qwen3-VL-32B-Thinking (Bai et al., 2025a), and Step-GUI-8B (Yan et al., 2025).
Generation budget / compute accounting. For the online agent evaluation (OSWorld), the "generation budget" is implicitly measured in maximum interaction steps per task. EvoCUA operates with a strict 50-step constraint per task, whereas most baselines (all OpenCUA variants, Qwen3-VL baselines, UI-TARS-2, Seed-1.8, Claude-4.5-Sonnet when run at 100 steps) use a 100-step budget (Table 1, "Max Steps" column). This makes EvoCUA's results directly comparable at a disadvantage—it achieves higher success rates with half the interaction budget. The paper does not report total FLOPs or wall-clock time for training, though Section 6.5 mentions "more than 1 million accelerator hours" across all experiments. For the Pass@k analysis, k represents the number of independent rollout attempts on a fixed task.
Cross-validation / statistical protocol. The paper does not report formal cross-validation splits or statistical significance tests for the main evaluation results. The OSWorld benchmark provides a fixed test set, and models are evaluated in a zero-shot manner (no fine-tuning on benchmark tasks) following the standard evaluation protocol. The tri-fold decontamination procedure (Section 3.3) ensures that synthesized training tasks do not overlap with OSWorld test tasks in semantics, environment configurations, or evaluator scripts. For the offline grounding and general capability benchmarks (ScreenSpot-v2, ScreenSpot-Pro, OSWorld-G, MMMU, MMMU-Pro, MathVista, MMStar, OCRBench), the evaluation follows each benchmark's standard protocol.

Main Quantitative Results

Online Agent Evaluation on OSWorld

Headline result. EvoCUA-32B achieves a success rate of 56.7% on the OSWorld-Verified benchmark (Table 1), establishing a new state-of-the-art among all open-weights models. This is a +11.7 absolute point improvement over the previous best open model, OpenCUA-72B (45.0%), and a +15.1 point gain over its own base model, Qwen3-VL-32B-Thinking (41.0%).

Comparison with closed-weights models. EvoCUA-32B's 56.7% places it within 2.2 points of Claude-4.5-Sonnet at the same 50-step budget (58.1%, Table 1). It surpasses UI-TARS-2-2509 (53.1% at 100 steps) by +3.6 points, despite UI-TARS-2 having twice the interaction budget. When Claude-4.5-Sonnet is allowed 100 steps, its performance rises to 62.9%, establishing a 6.2-point gap as the remaining frontier. Seed-1.8 at 100 steps achieves 61.9%, also above EvoCUA but at double the step budget.

Efficiency under step constraints. A critical detail is that EvoCUA's results are achieved under a 50-step maximum, whereas nearly all baselines use 100 steps (Table 1, "Max Steps" column). The paper explicitly notes:

"these results are achieved under a strict 50-step constraint, whereas baselines typically require a 100-step budget to reach peak performance, indicating our model's superior execution precision."

This means EvoCUA not only achieves higher success rates but does so with approximately half the interaction length, which translates to lower latency and lower inference cost per task. The performance scaling with max steps is analyzed separately in Section 6.4 (Figure 6b): EvoCUA-32B at 50 steps achieves +16.25% improvement over its own performance at 15 steps, and at 100 steps reaches +17.36%, confirming that the 50-step constraint is not the performance ceiling—the model continues to benefit from additional steps, but it already surpasses baselines at half their budget.

Scaling efficiency at smaller model sizes. The EvoCUA-8B variant achieves 46.1% at 50 steps, which is significant for three reasons:

It surpasses the specialized 72B-parameter OpenCUA-72B (45.0% at 100 steps) by +1.1 points, demonstrating that the training paradigm can compensate for a 9× reduction in parameter count.
It outperforms Qwen3-VL-235B-A22B-Thinking (38.1% at 100 steps), a model with approximately 29× more parameters.
In a controlled comparison with Step-GUI-8B (40.2% at 100 steps)—both initialized from the identical Qwen3-VL-8B-Thinking backbone—EvoCUA-8B achieves a +5.9 point advantage. The paper emphasizes: "This strictly isolates the contribution of our evolving experience learning paradigm, confirming that our data synthesis and RL strategies unlock significantly greater potential from the same foundational architecture."

Comparison with prior specialized models. Among specialized open models (those designed specifically for computer use rather than general VLMs adapted post-hoc), EvoCUA-32B at 50 steps (56.7%) substantially exceeds:

OpenCUA-32B: 34.8% at 100 steps (+21.9 points)
OpenCUA-72B: 45.0% at 100 steps (+11.7 points)
UI-TARS-72B-DPO: 24.6% at 50 steps (+32.1 points at matched step budget)
GUI-Owl-7B-Desktop-RL: 34.9% at 15 steps (not directly comparable due to different step constraints)
ScaleCUA-32B: 17.7% at 50 steps (+39.0 points)

Offline Grounding and General Capabilities

GUI grounding benchmarks. Table 2 reports performance on three grounding benchmarks:

ScreenSpot-v2: EvoCUA-32B achieves 90.40%, compared to 91.11% for Qwen3-VL-32B-Thinking (−0.71 points) and 92.90% for OpenCUA-72B (EvoCUA-OpenCUA-72B variant achieves 93.47%, a +0.57 point gain over its base). EvoCUA-8B drops to 85.21% from Qwen3-VL-8B-Thinking's 90.09% (−4.88 points).
ScreenSpot-Pro: EvoCUA-32B achieves 49.76%, down from Qwen3-VL-32B-Thinking's 57.10% (−7.34 points). EvoCUA-8B at 45.39% is nearly flat relative to Qwen3-VL-8B-Thinking's 46.40% (−1.01 points). The EvoCUA-OpenCUA-72B variant shows a gain to 63.24% over OpenCUA-72B's 60.80% (+2.44 points).
OSWorld-G: EvoCUA-32B achieves 63.86%, slightly below Qwen3-VL-32B-Thinking's 64.00% (−0.14 points, essentially flat). EvoCUA-8B drops to 55.08% from 56.70% (−1.62 points). The OpenCUA-based variant improves to 67.65% over 66.95% (+0.70 points).

General multimodal benchmarks. Table 2 also reports results across five general capability benchmarks:

MMMU: EvoCUA-32B: 68.11% vs. Qwen3-VL-32B-Thinking: 78.10% (−9.99 points). EvoCUA-8B: 62.11% vs. 74.10% (−11.99 points). EvoCUA-OpenCUA-72B: 59.22% vs. OpenCUA-72B: 60.67% (−1.45 points, near parity).
MMMU-Pro: EvoCUA-32B: 59.16% vs. 68.10% (−8.94 points). EvoCUA-8B: 53.30% vs. 60.40% (−7.10 points). EvoCUA-OpenCUA-72B: 46.51% vs. 43.04% (+3.47 points).
MathVista: EvoCUA-32B: 80.40% vs. 85.90% (−5.50 points). EvoCUA-8B: 75.80% vs. 81.40% (−5.60 points). EvoCUA-OpenCUA-72B: 69.40% vs. 70.90% (−1.50 points).
MMStar: EvoCUA-32B: 73.20% vs. 79.40% (−6.20 points). EvoCUA-8B: 69.07% vs. 75.30% (−6.23 points). EvoCUA-OpenCUA-72B: 67.80% vs. 66.47% (+1.33 points).
OCRBench: EvoCUA-32B: 85.35% vs. 85.50% (−0.15 points, flat). EvoCUA-8B: 80.30% vs. 81.90% (−1.60 points). EvoCUA-OpenCUA-72B: 84.05% vs. 83.80% (+0.25 points).

Interpretation of general capability results. The paper provides a candid analysis of why Qwen3-VL-based EvoCUA variants show consistent declines on general benchmarks while OpenCUA-based variants do not:

"We attribute this performance drop primarily to discrepancies in data distribution and patterns. Due to time constraints, the general dataset used for fine-tuning EvoCUA was directly adopted from OpenCUA-72B variants experiments. However, this dataset is non-thinking, creating a significant mismatch with the thinking-based distribution of the Qwen3-VL-32B-Thinking model."

The authors further note: "We further analyzed the output lengths of Qwen3-VL-32B-Thinking and EvoCUA on general benchmarks. The results reveal a significant reduction in EvoCUA's token count compared to Qwen3-VL-32B-Thinking (2,514 vs 3,620), accompanied by a shift in output style." This diagnosis is specific: the performance decline is not caused by the evolving experience paradigm itself, but by mixing a non-thinking-format general dataset with a thinking-format base model, which causes the model to shift its output style and reduce its reasoning verbosity on non-computer-use tasks.

The OpenCUA-based variant, which uses a dataset distribution matched to its base model, shows stability or improvement on general benchmarks, confirming that when data distributions are aligned, the paradigm preserves broad capabilities.

Ablation Studies and Robustness Checks

Component ablation on EvoCUA-32B (Table 3). Each training stage contributes monotonically, with the cumulative effect building from the Qwen3-VL-32B-Thinking baseline (41.0% on OSWorld, from Table 1):

Stage	Absolute Improvement (Δ%)
+ Unified Action Space	+4.84
+ Cold Start	+2.62
+ RFT	+3.13
+ Offline DPO	+3.21
+ Iterative Training	+1.90

These gains are reported as incremental relative to the previous stage, not cumulative from baseline. The Unified Action Space provides the largest single-stage gain (+4.84 points), suggesting that properly structuring the action space is a prerequisite that static imitation approaches may neglect. The DPO phase (+3.21) provides the second-largest gain, exceeding the RFT phase (+3.13), which confirms that learning from explicit failures captures improvement beyond what success-only data can provide. The iterative training round (+1.90) shows diminishing but non-zero returns, consistent with a system approaching but not reaching its performance ceiling.

Generalizability on OpenCUA-72B (Table 4). The same pipeline applied to a different model family produces a similar pattern of cumulative gains:

Stage	Absolute Improvement (Δ%)
+ Cold Start	+2.14
+ RFT	+3.69
+ Offline DPO	+3.02
+ Iterative Training	+1.82

The OpenCUA baseline is 45.0% (Table 1), and the paper does not report the final post-ablation number, but the cumulative gain over baseline from these stages is approximately +10.67 points. The RFT phase contributes more on OpenCUA-72B (+3.69) than on EvoCUA-32B (+3.13), while DPO contributes slightly less (+3.02 vs. +3.21), but the overall pattern of substantial gains from each stage holds. This cross-model consistency is the primary evidence for the paradigm's generalizability.

Experience scaling on RFT alone (Table 5). An ablation on an early OpenCUA-72B iteration (omitting cold-start and DPO to isolate RFT effects) shows:

Stage	Data Size	Gain (Δ%)
RFT Round 1	20k trajectories	+2.61
RFT Round 2	226k trajectories	+6.79
RFT Round 3	1M trajectories	+8.12

The diminishing returns from Round 2 to Round 3 (+6.79 to +8.12, only +1.33 additional points from 774k additional trajectories) are explicitly noted as evidence for a data quality bottleneck: "as model capabilities improve with scale, the tolerance for noise decreases, creating a bottleneck for existing iterative methods." This directly motivates the transition to DPO, which extracts higher-quality signal from the same data by constructing targeted preference pairs rather than relying on volume alone.

Pass@k scaling analysis (Figure 6a). EvoCUA maintains a consistent performance advantage over its base model (Qwen3-VL-Thinking) across all Pass@k values from k = 8 to k = 128. The EvoCUA-32B gains range from +2.35% at k = 128 to +4.93% at k = 16, with the peak at k = 16 suggesting that the training particularly improves the model's ability to succeed within a moderate number of attempts. The EvoCUA-8B gains range from +3.43% at k = 128 to +4.65% at k = 16, showing a similar pattern. The gains do not approach zero at high k, indicating that the training elevates the model's capability ceiling, not just its Pass@1 performance.

Inference step scaling (Figure 6b). Performance improves monotonically as the maximum step limit increases from 15 to 100. EvoCUA-32B gains +16.25% at 50 steps and +17.36% at 100 steps relative to its 15-step baseline. EvoCUA-8B shows smaller scaling benefits: +10.25% at 50 steps and +10.84% at 100 steps. The difference in scaling rates between 32B and 8B suggests that larger models benefit more from additional inference steps, likely because their richer internal representations better utilize the extended interaction context.

Oracle vs. predicted difficulty (not applicable). Unlike the analysis paper in the reference example, this paper does not use oracle difficulty bins or ground-truth labels to partition tasks. All evaluation is on the fixed OSWorld test set using the benchmark's standard evaluator.

Critical Assessment

Does the paper demonstrate that the evolving paradigm via learning from experience is the causal driver of improvements, rather than the specific base model or data scale?

The evidence for this claim is strong but with a qualifier about data distribution mismatch. The cross-model ablation (Tables 3 and 4) demonstrates that the training pipeline produces substantial, monotonic gains across two different model families (Qwen3-VL-Thinking and OpenCUA) at multiple scales (8B, 32B, 72B). The consistency of per-stage gains—each stage contributes meaningfully in both model families—supports the claim that the paradigm, not the model, drives improvement. The EvoCUA-8B vs. Step-GUI-8B comparison (46.1% vs. 40.2%, same backbone, different training) further isolates the training methodology's contribution.

However, the paper does not run a controlled experiment that varies only the training paradigm while holding data quantity constant. EvoCUA uses massively more data (up to 1M trajectories for RFT rounds, plus DPO pairs) than static imitation baselines like OpenCUA (trained on AgentNet's 11,887 tasks). A critic could argue that the gains come from data scale rather than the interactive nature of the data—perhaps training on 1M static trajectories would achieve similar performance. The paper cannot rule this out because it never trains a static-imitation baseline at comparable data scale. This is a genuine gap in the experimental evidence.

The paper's strongest defense against this criticism is conceptual rather than experimental: static datasets cannot contain error recovery trajectories by definition, so they cannot teach the specific skills that DPO targets. But the paper does not provide an ablation where DPO is replaced with equivalent-scale supervised fine-tuning on additional diverse static trajectories to demonstrate that the interactive nature of the data matters specifically.

Does the paper demonstrate that the verifiable synthesis engine produces better training data than alternative data sources?

This claim is not directly tested. The paper never conducts an experiment where EvoCUA is trained on alternative data sources (e.g., human demonstrations, static datasets scraped from the internet, LLM-generated instructions without validators) and compared against training on the synthesis engine's output. The synthesis engine is a component of the pipeline, but its individual contribution is never isolated.

The paper does provide process-level validation—the consistency-based filtering and tri-fold decontamination ensure the synthesized data is high-quality—but this does not constitute an empirical demonstration that the synthesis engine is necessary for the observed gains. It is possible that equally good results could be achieved by collecting a large static dataset through other means and applying the same RFT + DPO pipeline. The paper's claim that the synthesis engine "effectively breaks the bottleneck of manual data curation" is a claim about scalability and cost, not about data quality relative to alternatives, and the paper does not run the experiment that would test the quality claim.

Does the paper demonstrate that step-level DPO at critical forking points outperforms trajectory-level DPO?

This claim is partially supported by the offline results, but not directly tested in online evaluation. The ablation demonstrates that the DPO phase contributes significant gains (+3.21% for EvoCUA-32B, +3.02% for OpenCUA-72B) on top of RFT, but the comparison is DPO vs. no DPO, not step-level DPO vs. trajectory-level DPO. The paper never runs a trajectory-level DPO ablation on the full OSWorld benchmark to show that step-level targeting is superior.

The paper's argument for step-level DPO is primarily analytical (Section 5.3's discussion of state misalignment in trajectory-level comparisons) rather than empirical. The STEPO experiments (Section 7, Figure 7) compare step-level vs. trajectory-level optimization for online RL (GRPO), but this is a different algorithm family than DPO and the experiments are preliminary, conducted only on OpenCUA-32B without full benchmark evaluation. The paper does not report what DPO at the trajectory level would achieve on OSWorld.

However, the dual-paradigm construction (Action Correction and Reflection) is genuinely distinctive, and the paper provides Algorithm 2 as a concrete specification. The ablation on the DPO phase confirms that something in the DPO pipeline works, but it cannot attribute the gains specifically to the step-level targeting or the dual-paradigm design as opposed to simply having more training data or using preference optimization in general.

Does the paper demonstrate that the scalable interaction infrastructure is a bottleneck that, once solved, enables the observed gains?

This claim is about engineering necessity rather than empirical comparison. The paper does not run an experiment with a smaller-scale infrastructure to show that the gains would be impossible without 100,000 concurrent sandboxes. The infrastructure is enabling—it makes the training feasible—but its specific contribution to model quality is not directly measurable.

The infrastructure section (Section 4) provides specifications (100,000+ concurrent sandboxes, hundreds of thousands of daily sessions, millions of requests per day, burst scaling to tens of thousands of instances in one minute) that are impressive but not empirically linked to downstream performance. A critic could argue that equally good results might be achieved with 10,000 concurrent sandboxes and longer training time, or with sequential rollouts and off-policy replay buffers. The paper does not address this because the infrastructure is framed as a necessary condition for on-policy RL at scale, not as a variable whose level affects final performance.

Does the paper demonstrate that EvoCUA achieves "state-of-the-art" open-weights performance in a fair comparison?

This claim is well-supported but requires attention to the step budget asymmetry. Table 1 clearly shows EvoCUA-32B at 56.7% vs. OpenCUA-72B at 45.0%, a +11.7 point gap. The step budget asymmetry (50 vs. 100 steps) makes EvoCUA's result more impressive, not less—it achieves higher performance with half the interaction budget.

However, Table 1 also shows that several closed-weights models outperform EvoCUA when given more steps (Seed-1.8 at 61.9% with 100 steps, Claude-4.5-Sonnet at 62.9% with 100 steps). The paper's claim of "state-of-the-art" is correctly qualified as "open-weights," and the paper is transparent about the remaining gap to closed-weights frontiers.

A less visible but potentially important caveat: the OSWorld benchmark has a fixed set of test tasks, and the tri-fold decontamination ensures that EvoCUA was not trained on those exact tasks. However, EvoCUA was trained on tasks synthesized by the same foundation models (Qwen3-VL, OpenCUA) that the synthesis engine uses. If those models have implicit knowledge of OSWorld-like tasks in their pretraining data, the synthesis engine might generate training tasks that are stylistically or structurally similar to OSWorld test tasks even if they are not semantically identical. The semantic decontamination filters for exact semantic matches but may not catch more subtle distributional overlap. The paper does not discuss this possibility.

Where would additional experiments have strengthened the paper?

Static data baseline at matched scale. Train a model on 1M static trajectories (the scale of EvoCUA's RFT data) to test whether the interactive nature of the data or simply the data quantity drives the gains.
Trajectory-level DPO ablation. Run DPO at the trajectory level (contrasting full trajectories rather than critical forking points) on OSWorld to quantify the contribution of step-level targeting specifically.
Synthesis engine ablation. Train EvoCUA using an alternative data source (e.g., human demonstrations, web-scraped interaction traces) with the same RFT + DPO pipeline to isolate the synthesis engine's contribution.
Infrastructure scale ablation. Train with different levels of environment concurrency (1K, 10K, 100K sandboxes) to determine whether the massive scale is necessary or simply accelerates training.
Cross-benchmark evaluation. Evaluate on additional computer-use benchmarks beyond OSWorld (e.g., WebArena, MiniWoB++) to test whether the paradigm generalizes across environment types, not just across model families.
Statistical significance. Report confidence intervals or variance estimates for the OSWorld success rates. With 500 test tasks (the original OSWorld paper's test set size), a +1.9% gain from iterative training may not be statistically significant.

What are the genuine experimental strengths?

The cross-model ablation is the paper's strongest empirical contribution. Demonstrating that the same training stages produce consistent gains across model architectures (Qwen3-VL vs. OpenCUA) and scales (8B, 32B, 72B) substantially strengthens the claim that the paradigm generalizes. The EvoCUA-8B vs. Step-GUI-8B comparison (identical backbone, 5.9-point difference) provides the cleanest isolation of the training methodology's effect, and this is a genuinely persuasive result.

The performance under step constraints is a meaningful signal of efficiency. Achieving 56.7% at 50 steps when baselines use 100 steps suggests that EvoCUA's trajectories are more direct and contain fewer wasted actions. The step scaling analysis (Figure 6b) confirms this quantitatively.

The paper's transparency about general capability regression is commendable. Rather than hiding the MMMU and ScreenSpot-Pro declines, the paper diagnoses them explicitly (data distribution mismatch, token count reduction, output style shift) and proposes a concrete solution (thinking-based general dataset). This negative result actually increases credibility because it shows the authors are not selectively reporting favorable outcomes.

6. Limitations and Trade-offs

The Difficulty Estimation Cost is Unaccounted for and Likely Dominates the Inference Budget

The assumption or constraint. The entire compute-optimal framework depends on estimating prompt difficulty before allocating the test-time compute budget. The paper's method for doing so — generating 2,048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The authors explicitly acknowledge this gap in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In practice, the total cost would be difficulty estimation (2,048 generations per prompt) plus strategy execution (16–256 generations), and the former dominates so severely that the claimed efficiency gains evaporate entirely. For a deployment where each prompt is seen once, the approach is strictly more expensive than simply running best-of-N at the maximum budget. The framework is currently viable only in amortized settings where the same question is answered many times, which is not the typical deployment scenario for LLM inference.

What evidence exists in the paper. The paper is transparent about this limitation in Sections 3.2 and 8, flagging it as "a key avenue for future work" and suggesting "pretraining or finetuning models to directly predict difficulty of a question." However, the main results (Figures 4 and 8) report compute-optimal scaling curves without including any difficulty estimation cost in the x-axis budget, making the 4× claim strictly an upper bound on achievable efficiency. No experiment measures performance when the estimation cost is included in the total compute budget.

Mitigation status. Not addressed. The paper proposes future work on learned difficulty predictors and adaptive estimation (few initial samples → estimate → allocate remaining budget) but implements neither. Until a cheap, accurate difficulty estimator is developed and validated, the compute-optimal framework remains a conceptual contribution rather than a deployable method.

The Approach Provides Zero Benefit on Hard Problems — a Fundamental Capability Ceiling

The assumption or constraint. Test-time compute can only amplify existing capability, not create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help because there are no correct solutions in the proposal distribution to find or refine. The paper is candid about this boundary condition in Section 7:

"on the hardest problems (difficulty bin 5), test-time compute provides essentially zero benefit regardless of budget"

The consequence. The entire framework offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For deployment scenarios where the problem difficulty distribution skews toward bins 4–5 (hard problems outside the model's rough capability range), the approach provides no advantage over the base model's greedy decoding. This is not a small edge case — the hardest bin represents 20% of the MATH test set by construction, and these are exactly the problems where improvement is most valuable (challenging competition problems, research-level reasoning, novel problem formulations). For such problems, pretraining remains the only viable path, as the FLOPs-matched comparison shows: bin 5 shows −3.6% to −52.9% relative disadvantage for test-time compute vs. the 14× larger model depending on R.

What evidence exists in the paper. Figure 3 (right, bin 5) shows accuracy remaining at 1–3% across all methods and all budgets. Figure 7 (right, bin 5) shows 2–3% accuracy irrespective of the sequential-to-parallel ratio. Figure 9 (bin 5, bottommost line) is essentially flat near 0–5% across all test-time budgets. These are consistent, replicable failures across search methods, revision strategies, and selection mechanisms.

Mitigation status. The paper acknowledges this explicitly (Section 7 takeaway box) but offers no solution. The boundary is fundamental: the approach cannot solve problems the base model fundamentally does not understand. This limits the practical scope to problems within the base model's "capability neighborhood" — a constraint that is inherent to the test-time compute paradigm and not addressable through better search algorithms or revision strategies.

Single Benchmark, Single Model Family Restricts Generalizability Claims

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The paper provides no results on any other reasoning benchmark (GSM8K, MMLU reasoning subsets, ARC, code generation) and no results on any model family other than PaLM 2.

The consequence. Several aspects of the findings could be model-specific or domain-specific in ways the paper cannot distinguish:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different reasoning style might exhibit different difficulty-dependent scaling curves — for instance, a model that produces more diverse solutions might benefit more from parallel search even on easy problems.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (GPT variants, Claude, Llama, Gemini all differ in their few-shot learning behavior).
MATH consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems, revisions helping easy problems, the 4× efficiency gain — generalize to other reasoning domains (code generation, logical deduction, scientific QA, planning) or to tasks requiring factual knowledge rather than inference.

The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed gains are statistically reliable at this sample size or whether the selected strategies would differ meaningfully with different data splits.

What evidence exists in the paper. None — this is a scope limitation, not a measured failure. The paper does not include any out-of-domain evaluation or cross-model validation. The discussion in Section 8 proposes future work on extending to other domains and modalities, implicitly acknowledging the current scope is narrow.

Mitigation status. Not addressed in the current paper. The authors acknowledge the limitation implicitly by suggesting future work on other domains, but provide no empirical evidence that the findings transfer. Until replication studies on different benchmarks, model families, and task types are conducted, the paper's findings should be understood as specific to PaLM 2-S* on MATH, with generalizability to other settings plausible but unverified.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Inflating the Apparent Advantage of Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) of over-training on data relative to Chinchilla-optimal ratios. The authors explicitly acknowledge this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search of any kind.

The consequence. The comparison is systematically biased in favor of test-time compute. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it should be for a fair comparison. The reported advantages of test-time compute over pretraining — e.g., +27.8% on easy questions at R ≪ 1 — would shrink or potentially reverse against a properly compute-optimal larger model. Furthermore, the larger model's greedy decoding baseline is artificially weak: giving the 14× larger model even a modest test-time compute budget (e.g., best-of-8 or majority voting at N=16) would create a substantially stronger baseline that might recapture much of the apparent advantage. The paper never tests this.

What evidence exists in the paper. The paper is transparent about both caveats: the fixed-data parameter scaling in Section 7 and the greedy decoding baseline. However, the headline comparisons in Figures 1 and 9 present the results as "small model + test-time compute vs. 14× larger model" without prominently qualifying that the larger model is not compute-optimally trained and uses no test-time compute augmentation of its own. The bar charts in Figure 1 (top-right, bottom-right) showing large positive percentages in the R ≪ 1 regime should be interpreted with this systematic bias in mind.

Mitigation status. Partially acknowledged. The paper explicitly states that compute-optimal pretraining comparison is left to future work. However, the paper does not acknowledge the greedy decoding asymmetry as a limitation — it does not discuss what a fairer comparison (both models using test-time compute) would show. This is a methodological weakness in the experimental design that affects the strength of the FLOPs-matched conclusions.

No Combination of PRM Search with Revisions Is Studied, Leaving the Full Potential of the Framework Unexplored

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided search (modifying the verifier/selection process) and iterative revisions (modifying the proposal distribution) — but studies them exclusively in isolation. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's reported results represent a lower bound on what the unified framework could achieve. The two mechanisms have inherently complementary strengths that the paper's own difficulty analysis reveals: revisions excel on easy problems (local refinement of approximately-correct answers), while PRM search excels on medium problems (global exploration to find solutions the model would not produce by random sampling). A combined system that uses the revision model as the proposal distribution within beam search — or that uses the PRM to guide which revision branches to pursue — could potentially outperform either mechanism alone, particularly on medium-difficulty problems where both mechanisms contribute. Without this experiment, the paper cannot claim to have found the optimal allocation of test-time compute across the full space of strategies; it has only optimized within the search subspace and the revision subspace separately.

What evidence exists in the paper. None — this is an unexplored design choice, not a measured limitation. The paper does not include any experiment combining PRM search with the revision model, nor does it ablate whether the gains from search and revisions are additive or redundant.

Mitigation status. Explicitly deferred to future work (Section 8). The authors clearly identify this as a natural next step, suggesting that "the two mechanisms have complementary, difficulty-dependent strengths." The absence of this experiment means the paper's claim of "compute-optimal" scaling is with respect to a restricted strategy space, not the full space of possible allocations. A practitioner deploying the system today would not know whether combining both mechanisms yields significant additional gains or whether the individual gains overlap substantially.

The Revision Model Has a Fundamental Correct-to-Incorrect Reversion Problem with Only Partial Mitigation

The assumption or constraint. The revision model was trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This means the model has no training signal for what to do when the current answer in context is already correct. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. In a long revision chain, the agent is nearly as likely to corrupt a good answer as to improve a bad one. This creates a non-monotonic quality trajectory: pass@1 improves on average (Figure 6, left), but any individual chain is unreliable because correct answers can be undone. The paper's mitigation — using majority voting or verifier-based selection across the entire chain rather than taking the final revision — is a post-hoc patch, not a solution to the underlying training distribution mismatch. The mitigation requires maintaining the full chain of all revisions, which increases memory requirements and selection complexity. It also means that compute is wasted on revisions that actually degrade answer quality, and the system cannot allocate its budget knowing which revisions will improve quality and which will regress.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. Figure 6 (left) shows that per-step pass@1 plateaus around 24–25% after step 15–20, suggesting that improvements and regressions roughly balance out in expectation over long chains. The compute-optimal revision results (Figure 8) use the within-chain selection mitigation, so the reported gains include the cost of this workaround.

Mitigation status. Partially addressed through within-chain selection (majority voting or verifier-based selection across the chain), but the underlying problem is not solved. The paper does not explore training the revision model with mixed in-context data (both correct and incorrect previous answers) to teach it when to revise and when to stop, which would be a principled solution. The ReST^EM experiment (Appendix K) showed that attempts to further optimize the revision model actually made performance worse, suggesting the training procedure is fragile in ways that are not fully understood. A practitioner relying on sequential revisions must accept that ~38% of their compute budget on revision steps will actively degrade answer quality, compensated only by selection mechanisms that catch the issue post-hoc.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper represents a genuine paradigm shift in how the field thinks about training computer-use agents. The magnitude of this shift should not be understated: the paper does not propose a better architecture, a larger model, or a more sophisticated imitation learning objective. It proposes that the entire framing of the problem — "collect demonstrations and imitate" — is structurally incapable of producing robust agents, and it replaces that framing with a closed-loop experience cycle where the agent generates its own training data through interaction with real environments, evaluates success through machine-executable verifiers, and continuously improves from both successes and failures.

This is not an incremental refinement. The empirical evidence for the paradigm's effectiveness is compelling even at this early stage: EvoCUA-32B achieves a 56.7% success rate on OSWorld, which is not merely an improvement over prior open models but a fundamentally different capability tier (+11.7 absolute points over the previous SOTA OpenCUA-72B, +21.9 points over OpenCUA-32B, and a 15.1-point gain over its own base model Qwen3-VL-32B-Thinking). The cross-model ablation (Tables 3 and 4) demonstrating that the same training pipeline produces consistent cumulative gains across fundamentally different model families (Qwen3-VL-Thinking and OpenCUA) and scales (8B, 32B, 72B) strongly suggests that the paradigm captures something universal about how GUI agents should be trained, rather than exploiting idiosyncratic properties of any particular architecture.

The landscape shift has several specific dimensions:

It changes what "data" means for agent training. Prior to this work, data for computer-use agents meant static demonstration trajectories — (screenshot, action) pairs collected in advance. The paper demonstrates that interactive experience — trajectories generated by the agent's own policy, evaluated by executable validators — is a fundamentally different and richer training signal. The paper does not just claim this; it quantifies it through the RFT and DPO gains. The rejection sampling phase alone contributes +3.13% (EvoCUA-32B) and +3.69% (OpenCUA-72B) by having the agent learn from its own successful rollouts, and the DPO phase adds another +3.21% and +3.02% respectively by constructing targeted preference pairs from failures. These are gains that static datasets, by definition, cannot provide because static datasets contain no error trajectories to learn from and no environmental feedback to internalize.

This shifts the research emphasis from data collection (amassing ever-larger demonstration datasets) to experience generation infrastructure (building systems that can produce, verify, and learn from interactive rollouts at scale). Teams working on GUI agents now have a clear alternative to the diminishing returns of data scaling: invest in the verifiable synthesis engine and the interaction infrastructure, then let the agent bootstrap itself.

It resolves the contradiction between imitation learning's promise and its practical brittleness. The field knew that behavior cloning suffers from compounding errors and covariate shift — this has been understood since the early days of autonomous driving research. But the standard response was to collect more diverse demonstrations, hoping to cover more of the state distribution. EvoCUA's results suggest that no amount of static demonstration diversity can substitute for interactive experience, because static data cannot teach error recovery. The paper's dual-paradigm DPO (Paradigm II teaching the agent to recognize unexpected states and formulate recovery plans) is only possible because the training pipeline has access to failure trajectories — the agent's own mistakes, captured in sandbox rollouts, showing exactly what happens when a click misses or a menu option is grayed out. These trajectories are not in any static dataset.

This resolution has practical consequences: research effort previously directed at improving demonstration quality (better human annotators, more diverse task designs, higher-resolution screenshots) should partially redirect toward improving verifier quality and synthesis engine capabilities, which are the bottlenecks in the interactive paradigm.

It redefines the role of synthetic data generation. Prior work on synthetic data for GUI agents focused on generating task instructions — natural language descriptions of what the agent should do. EvoCUA demonstrates that task instructions alone are insufficient because they lack precise success conditions. The "generation-as-validation" principle — co-generating tasks and their executable validators through a ReAct-based agentic loop, then verifying the validators themselves through sandbox execution — converts synthetic data generation from a prompt engineering problem into a systems engineering problem. The synthesis engine must not only produce diverse, realistic tasks but also produce validators that are executable, deterministic, and aligned with the task's intent. The paper's quality assurance pipeline (consistency-based filtering, tri-fold decontamination, manual spot checks on validator-reward model discrepancies) becomes a template for how to build reliable synthetic data pipelines.

This changes the direction of synthetic data research: rather than asking "how do we generate more diverse task descriptions?", the relevant question becomes "how do we generate tasks whose success conditions can be verified programmatically, and how do we validate the verifiers themselves?" This is a harder but more productive question.

It establishes interactive experience learning as a general-purpose capability amplifier. The most surprising result in the paper is EvoCUA-8B at 46.1% surpassing the specialized 72B-parameter OpenCUA-72B at 45.0%. This is not merely "smaller models can be competitive" — it is "the training paradigm can compensate for a 9× reduction in parameter count within the base model's capability range." The practical implication is that organizations deploying computer-use agents should invest more heavily in the training infrastructure (synthesis engine, sandbox farm) and less in model size, at least for tasks within the base model's rough competence. The FLOPs economics of this tradeoff are not analyzed in the paper (unlike the reference example's FLOPs-matched comparison), but the raw performance numbers strongly suggest that training methodology dominates model scale for this task class.

It identifies a specific failure mode — the training-inference discrepancy in context-truncated RL — that was likely degrading prior work without being recognized. Section 7's diagnosis of why trajectory-level RL algorithms fail for GUI agents with limited context windows is a significant diagnostic contribution. Many prior papers probably applied GRPO or PPO to GUI agent training at the trajectory level, observed disappointing results, and attributed them to insufficient data or hyperparameter sensitivity. EvoCUA identifies the specific mechanism: the agent's context compression strategy (full multimodal context only for the most recent five steps, text-only for earlier history) means that trajectory-level gradient updates cannot propagate learning signals to intermediate steps whose visual context has been compressed. The proposed STEPO algorithm, while not fully validated, provides a clear baseline solution and a clear direction for improvement (selective step-level training to reduce cost). This diagnostic will likely influence how future work designs RL training loops for any long-horizon agent with context constraints.

Follow-Up Research This Work Enables

Verifier quality as the primary scaling bottleneck. The paper's results show that verifier over-optimization limits the effectiveness of aggressive search (best-of-N degrades on easy problems at high budgets in the reference example, and the synthesis engine's closed-loop feedback is critical for validator reliability in EvoCUA). A direct follow-up would systematically vary the synthesis engine's validator quality — e.g., by ablating the number of feedback iterations during validator generation, by comparing validators generated with and without the standardized tool library, or by introducing controlled noise into the validators (modifying success thresholds, adding off-by-one errors in spreadsheet cell checks) — and measuring how validator reliability affects downstream agent performance. The prediction is that below a certain validator accuracy threshold, the RFT and DPO gains collapse because the training signal becomes too noisy to distinguish successful from failed trajectories. Quantifying this relationship would establish a verifier quality scaling law analogous to the Chinchilla scaling laws for pretraining, but for interactive agent training.

Online RL with adaptive step-level credit assignment. The paper's STEPO algorithm (Section 7) demonstrates that step-level optimization outperforms trajectory-level GRPO for GUI agents, but at the cost of multiplying training updates by the number of steps per trajectory. The paper explicitly hypothesizes that "the requirements for step-level training may not be uniform across different training phases, and training only specific key steps might also achieve comparable performance to training all steps." A strong follow-up would implement dynamic step selection: use the PRM's per-step score (or an analogous value estimator trained on the collected trajectories) to identify which steps in a trajectory are most informative — those where the value estimate changes sharply, indicating a critical decision point — and only train on those steps. This would be evaluated on OSWorld (or a successor benchmark) with a FLOPs-matched comparison against uniform STEPO and trajectory-level GRPO. The key metric is whether selective step training recovers the STEPO performance gains while matching GRPO's training cost.

Combining verifiable synthesis with self-improving synthesis. The paper's synthesis engine generates tasks using a frozen VLM in a ReAct loop, but the agent's improving capabilities are not fed back into the synthesis process except through the dynamic compute budgeting mechanism (which allocates more rollouts to boundary tasks). A natural extension would close the loop in the other direction: use the trained EvoCUA agent itself as the task architect for the next round of synthesis, replacing the frozen general-purpose VLM. The hypothesis is that a computer-use-specialized agent would generate tasks that are (a) more diverse in the specific interaction patterns that the current policy struggles with, and (b) more likely to produce validators that are executable and accurate, since the agent has direct experience with what makes tasks solvable or unsolvable. The experiment would compare two versions of the synthesis engine — one using the original frozen VLM and one using the previous iteration's EvoCUA checkpoint — and measure whether the agent-synthesized tasks lead to faster policy improvement per round of the evolutionary cycle. The risk is that agent-synthesized tasks become too narrow (exploiting the agent's specific strengths and weaknesses rather than producing general capability improvements), which would be detected by evaluating on a held-out task distribution not used in synthesis.

Cross-benchmark generalization of the evolving experience paradigm. All of EvoCUA's results are on OSWorld, which is a specific desktop environment with specific application types (web browsers, office suites, file managers). The paper does not evaluate on WebArena (web-based tasks), MiniWoB++ (simplified web interactions), or mobile GUI benchmarks (Android-in-the-Loop, AppAgent). A critical follow-up would replicate the full EvoCUA pipeline — synthesis engine, infrastructure, RFT + DPO — on a different GUI environment type to test whether the paradigm's gains are specific to OSWorld's characteristics or generalize across environment modalities. If the paradigm transfers, it suggests that the underlying principles (verifiable task synthesis, interactive experience collection, step-level error correction) are domain-agnostic. If it fails, the failure analysis would reveal which aspects of OSWorld (deterministic rendering, limited action space, application stability) are necessary conditions for the paradigm to work, providing boundary conditions for practitioners deciding whether to adopt the approach. The specific experiment would use the same Qwen3-VL-32B-Thinking backbone, apply the same training stages, and report the same cumulative ablation table as Table 3, on WebArena's test set.

Scaling the synthesis engine to open-ended, non-verifiable tasks. The paper's synthesis engine requires that tasks have programmatically verifiable success conditions — spreadsheet cells can be checked, files can be compared, web page elements can be detected. But many real-world computer-use tasks are not cleanly verifiable: "research the competitive landscape for product X and summarize findings in a memo," "find the cheapest flight that meets these constraints and book it," "reorganize this folder of documents by project." These tasks have subjective quality criteria, evolving goals, or verification that requires external knowledge. Extending the synthesis engine to handle such tasks — perhaps by generating partial verifiers (check structural constraints like "a memo file exists with at least 500 words" while using LLM-as-judge for content quality) or by generating verifiable subgoals that decompose open-ended tasks into machine-checkable steps (e.g., "browser opened to flight search site" is verifiable, "correct flight selected" requires external validation) — would dramatically expand the scope of tasks the paradigm can handle. The evaluation would measure whether agents trained on partially-verifiable tasks achieve better performance on open-ended benchmarks than agents trained only on fully-verifiable tasks, testing whether partial verification provides enough signal for the RFT + DPO pipeline to be effective.

Agent-environment co-evolution. The paper's environments are static: the applications, file systems, and UI layouts do not change in response to the agent's improving capabilities. This creates a risk that the agent overfits to specific environment characteristics (font rendering, window sizes, specific application versions) present in the sandbox infrastructure. A forward-looking direction would introduce adversarial environment perturbations during training: randomly vary screen resolution, inject UI rendering delays, modify system font configurations, or swap application versions between rollout rounds. The hypothesis is that exposure to environmental variability during interactive training produces agents that are more robust to the kinds of deployment-time environment variation that cause brittle failures in static-imitation-trained agents. The experiment would compare two EvoCUA variants — one trained with the paper's deterministic environment calibration (xkb patching, font injection, runtime stability hardening) and one trained with intentionally perturbed environments — evaluating both on OSWorld and on a "noisy OSWorld" variant where the evaluation environment differs from the training environment in rendering and timing characteristics.

Practical Applications and Downstream Use Cases

Automated regression testing for software applications. A trained EvoCUA agent can be deployed as a continuous integration testing tool for GUI applications. Given a set of task specifications ("fill out the registration form with valid data," "export the quarterly report to PDF," "search for a product and add it to the cart") and their corresponding validators, the agent can execute these workflows across different application versions, operating system configurations, or localization settings. The value proposition is that (a) the agent handles the visual variability that breaks scripted testing tools (Selenium, Playwright) when UI elements shift position or change appearance, and (b) the validators provide deterministic pass/fail signals that integrate with existing CI pipelines. The paper's 50-step constraint and 56.7% success rate suggest that for a test suite of 100 workflows, approximately 57 would pass automatically and 43 would require human inspection — already a meaningful reduction in manual testing burden. As success rates improve through iterative training, the human-in-the-loop fraction decreases. The tri-fold decontamination procedure (Section 3.3) provides a template for ensuring that test tasks are not leaked into training data, which is critical for CI applications where tests must be blind to the agent.

Data entry and document processing at scale. The paper's synthesis engine specifically generates tasks involving Excel, Word, and PDF manipulation — exactly the document types that dominate enterprise data processing workflows. An EvoCUA agent trained on a company-specific synthesis task distribution (generated from the company's actual document templates and data schemas) could automate routine data entry tasks: extracting information from PDF invoices and entering it into an accounting spreadsheet, formatting quarterly reports according to a style guide, or migrating data between incompatible legacy systems through their GUIs. The key advantage over traditional robotic process automation (RPA) is that EvoCUA handles visual variability — if the invoice format changes slightly (a new vendor, a shifted table layout), the agent's visual grounding capabilities (90.40% on ScreenSpot-v2, Table 2) allow it to adapt, whereas RPA scripts break. The 46.1% success rate of EvoCUA-8B is particularly relevant here: a smaller, cheaper model can handle routine document tasks, with the 32B model reserved for more complex multi-application workflows. The experience scaling results (Table 5: 1M RFT trajectories for +8.12% gain) suggest that company-specific fine-tuning on in-house document types would yield meaningful improvements over the general OSWorld-trained model.

Accessibility layer for users with motor impairments. The EvoCUA agent's ability to translate natural language instructions into precise GUI interactions (mouse movements, clicks, keyboard inputs) at the pixel level makes it a candidate for assistive technology: a user with limited fine motor control issues a high-level instruction ("open my email, find the message from Dr. Chen about the conference, and reply confirming my attendance"), and the agent executes the sequence of precise GUI actions that would be physically difficult for the user. The paper's 50-step budget and ~57% success rate on OSWorld are not yet reliable enough for unsuperised deployment in this high-stakes setting (a 43% failure rate in assistive technology is unacceptable), but the evolving experience paradigm provides a path to improvement: as the user provides corrective feedback on failed interactions, those failure trajectories can be fed into the DPO pipeline to improve the agent's performance on the user's specific applications and workflows. The cold-start phase's lightweight initialization and the iterative training's diminishing-but-positive returns (+1.90% and +1.82% per additional cycle, Tables 3 and 4) suggest that personalization through continued interactive training is feasible without catastrophic forgetting of general computer-use skills. The key challenge is reducing the latency of the improvement cycle — currently, the infrastructure "orchestrates hundreds of thousands of daily sandbox sessions" and "processes millions of interaction requests per day," but this is batch training, not real-time personalization. Adapting the paradigm to online, per-user fine-tuning is an engineering challenge that the infrastructure's burst scaling capability (tens of thousands of sandbox instances within one minute) is well-positioned to address.

Self-improving data generation for vision-language model training. The paper's synthesis engine and interaction infrastructure can be repurposed as a data flywheel for general VLM training, not just for computer-use agents. The insight is that interactive GUI tasks naturally produce (screenshot, reasoning, action) triplets where the screenshots are diverse (different applications, layouts, visual styles), the reasoning traces are grounded in visual observation (the structured thought space enforces observation consistency), and the actions are verified as correct by the executable validators. The paper's quality assurance pipeline (consistency-based filtering, cross-validation by reward model and evaluator, manual spot checks) ensures these triplets are high-quality. A VLM trained on this data — even if it is never deployed as a computer-use agent — would likely develop stronger visual grounding capabilities (because it must connect visual elements to actions) and better multi-step reasoning (because the trajectories require sequential planning). The experiment would be to fine-tune a general VLM on the EvoCUA trajectory data (without the computer-use-specific action head) and evaluate on standard visual reasoning benchmarks (MMMU, MathVista, MMStar) and grounding benchmarks (ScreenSpot, RefCOCO). The hypothesis is that interactive trajectory data provides a richer learning signal than static image-caption data because it includes causal relationships between actions and visual outcomes. The paper's general capability regression on Qwen3-VL-based models (Table 2: MMMU drops from 78.10% to 68.11%) provides a cautionary baseline — the general dataset mixture matters — but also a clear direction for improvement: use the synthesis engine to generate reasoning-dense trajectories that are formatted to match the base model's thinking distribution, rather than the non-thinking format that caused the mismatch.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Scaling Agent Learning via Experience Synthesis2025· agents← LLM-judged similar
- Scaling Generalist Data-Analytic Agents2025· agents← LLM-judged similar
- AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis2025· data← LLM-judged similar
- The Collaboration Gap2025· agents← in-domain predecessor
- ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training2026· agents← similar topic
- Agent Learning via Early Experience2025· agents← similar topic
FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation2026· agents← in-domain predecessor
- Agentic Reasoning for Large Language Models2026· agents← in-domain predecessor
- Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization2025· training-methods← topic thread
- VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memory2026· agents← similar topic
ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents2026· agents← similar topic
- WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation2026· agents← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

Computer Environments Elicit General Agentic Intelligence in LLMs2026· agents→ in-domain predecessor
- LongCat-Flash-Thinking-2601 Technical Report2026· agents→ in-domain predecessor
- Towards General Agentic Intelligence via Environment Scaling2025· agents→ similar topic
- Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence2026· agents→ similar topic
Reflexion: Language Agents with Verbal Reinforcement Learning2023· agents→ similar topic
- Generative Agents: Interactive Simulacra of Human Behavior2023· agents→ in-domain predecessor
- SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning2025· agents→ similar topic
- UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2025· agents→ similar topic
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2025· agents→ similar topic
Agent Learning via Early Experience2025· agents→ similar topic
- Demystifying Reinforcement Learning in Agentic Reasoning2025· agents→ in-domain predecessor
- Scaling Agent Learning via Experience Synthesis2025· agents→ similar topic
- Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience2025· agents→ similar topic
Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents2025· agents→ similar topic
- WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking2025· agents→ in-domain predecessor
- AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents2024· agents→ similar topic
- Scaling Generalist Data-Analytic Agents2025· agents→ similar topic
- NitroGen: An Open Foundation Model for Generalist Gaming Agents2026· agents→ similar topic