ArXiv: 2605.12501

🎯 Pitch

Computer-use agents fail most on complex, long-tail interactions like dragging, drawing, or editing tablesβ€”exactly the operations that dominate real-world task failures. A new benchmark, CUActSpot, reveals that simply scaling click-centric GUI data isn't enough; only training on diverse synthetic data spanning text, tables, canvases, and natural images alongside widgets lets a 4B model surpass all open-source competitors under 32B parameters.

1. Executive Summary

This paper studies the long-tail challenge in computer-use agent grounding by introducing CUActSpot, a manually curated benchmark spanning five modalitiesβ€”GUI, text, table, canvas, and natural imageβ€”and a broader set of actions (click, drag, draw) than prior click-centric benchmarks, alongside a renderer-based data synthesis pipeline that programmatically generates screenshots and coordinates, then uses an LLM to produce matching instructions and action traces across ~50M samples. The core insight is that variety scalingβ€”increasing task and modality diversity rather than simply scaling data volume within a single modalityβ€”is the key driver of generalization, as shown through waterfall ablation studies where adding diverse modalities progressively improves performance across all benchmarks, with the trained Phi-Ground-Any-4B outperforming all open-source models below 32B parameters on CUActSpot while achieving an OSWorld grounding score of 42.4 (exceeding models with substantially higher ScreenSpot-Pro scores), establishing that existing widget-and-click-centric benchmarks systematically overestimate real-world grounding capability when complex interactions and non-GUI modalities dominate the failure distribution.

2. Context and Motivation

The Long-Tail Problem in Computer-Use Agent Grounding

The fundamental problem this paper addresses is that computer-use agents (CUAs) fail disproportionately on complex, low-frequency interactions β€” precisely the operations that matter most in real-world workflows. The authors frame this as a long-tail pattern in GUI operations, where "a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures" (Abstract). This is not merely a statistical curiosity; it is the primary bottleneck preventing CUAs from achieving the reliability needed for user trust and practical deployment.

To ground this claim empirically, the authors conduct a user study of GPT-5.4's computer-use capability on the Azure OpenAI platform (Section 1), collecting nearly 200 tasks across three scenarios: work (productivity applications like Word, PowerPoint, Excel, Photoshop), web usage, and gaming. The failure analysis, summarized in the upper portion of Figure 2, reveals that Action Grounding β€” the ability to translate a natural-language instruction into precise screen coordinates for actions β€” is the dominant source of error in the work setting, which is "also the scenario users care about most." The lower portion of Figure 2 provides a concrete end-to-end example: GPT-5.4 tasked with making a PowerPoint presentation introducing itself, where the failure mode involves incorrect coordinate prediction for complex operations.

Critically, the failure rates are not uniform across action types. The paper observes that "the failure rate for complex interactions is far higher than that for simple clicking" (Section 1). When the authors break down the error categories, coordinate errors on operations like dragging β€” which involve multiple key points and sequential dependencies β€” occur substantially more frequently than on simple single-point clicks. This is the core empirical observation motivating the entire work: existing models have a systematic blind spot for complex actions, and existing evaluation benchmarks fail to detect it.

The Pervasive Widget-and-Click-Centric Prior

The paper identifies a deeply entrenched assumption that has shaped the development of GUI grounding research: that action grounding is fundamentally about clicking on GUI widgets. The authors trace how this prior became self-reinforcing through a co-evolution of data pipelines, benchmarks, and model development (Section 2).

Historical origins. Early GUI agents decomposed screens into enumerable widgets via accessibility trees, DOM hierarchies, or Set-of-Marks prompting, then asked models to select discrete widget IDs. This paradigm, described in Section 2's discussion of prior work, "naturally frames action grounding as a widget-centric, click-centric task." The interaction model was simple: identify the target widget, click its center. This approach worked reasonably well for basic web navigation and form-filling, which dominated the early agent benchmarks.

The data-benchmark co-evolution trap. As the field matured toward purely visual grounding β€” where models directly output screen coordinates rather than selecting from enumerated elements β€” the widget-and-click-centric prior persisted through a feedback loop. On the data side, Section 2 notes that "construction pipelines largely inherit the web-crawl and accessibility-tree paradigm, producing widget bounding boxes and click labels over tens of millions of elements." When crawling websites and extracting DOM elements, the natural outputs are button bounding boxes and click targets. On the evaluation side, benchmarks followed suit: they evaluate whether a predicted point falls within a target widget's bounding box. This protocol is shared across SeeClick, ScreenSpot, ScreenSpot-Pro, and UI-Vision β€” the major grounding benchmarks.

ScreenSpot-Pro as an instructive case. The authors use ScreenSpot-Pro as Exhibit A for how benchmarks can push difficulty in the wrong direction. ScreenSpot-Pro makes tasks harder by using professional high-resolution software with tiny targets, but "remains single-click on GUI widgets" (Section 2). A task like "click the dodge tool icon button in Photoshop" tests two things simultaneously: whether the model knows what a dodge tool icon looks like (software-specific knowledge) and whether it can localize that icon on screen (grounding). The paper argues this conflates two distinct capabilities, and that software-specific knowledge often dominates the difficulty. As Section 5.2 puts it: "if solving a test case requires knowledge of a particular application, then the model must have been trained on data from that application, which encourages model development to focus on covering the software included in the benchmark rather than on learning genuinely generalizable grounding capabilities."

What is missing. Across the entire benchmark landscape, the paper identifies specific gaps that have remained "largely untouched" (Section 2): non-widget modalities such as tables, canvases, and natural images, and finer-grained operations like drawing, dragging to select text spans, adjusting cell boundaries, tracing object boundaries, and other multi-point interactions. These are precisely the types of operations that occur in productivity software β€” the work setting that users care about most, per the GPT-5.4 failure analysis.

The Benchmark-Reality Mismatch: Evidence from OSWorld

The paper provides compelling evidence that existing grounding benchmarks do not align with real-world agent performance. Table 3 presents a controlled experiment using OSWorld, an end-to-end agentic benchmark that involves richer interactions but measures task-level outcomes. The experimental design is clean: "we uniformly employed GPT-5.4 to generate single-step natural-language instructions and required each grounder to predict the corresponding action parameters. In this way, planning was controlled across all methods, and grounding was the only variable" (Section 5.2).

The results are striking. The authors select two models β€” GUI-Owl-1.5-8B-Instruct (ScreenSpot-Pro: 71.1) and MAI-UI-8B (ScreenSpot-Pro: 65.8) β€” whose ScreenSpot-Pro scores are substantially higher than GPT-5.4's own grounding (44.5) and Phi-Ground-Any-4B (26.3). Yet on OSWorld:

  • GUI-Owl-1.5-8B-Instruct achieves only 37.7
  • MAI-UI-8B achieves 38.2
  • GPT-5.4 achieves 44.1
  • Phi-Ground-Any-4B achieves 42.4

Despite a ~45-point gap on ScreenSpot-Pro, the end-to-end agentic performance is essentially flat or even reversed. The paper interprets this as evidence of "notable mismatch with real-world scenarios" (Section 5.2). The models that excel at widget-clicking on professional software interfaces do not necessarily perform better when the task requires dragging table cells, selecting text spans, or manipulating canvas objects β€” the interaction types that dominate real productivity workflows.

The ScreenSpot-Pro vs. UI-Vision gap as a diagnostic. Table 2 reveals another symptom of the benchmark-reality mismatch. Many recent models exhibit a large performance gap (Ξ”) between ScreenSpot-Pro and UI-Vision β€” over 20 points for models like GUI-Owl-1.5-8B-Instruct (Ξ” = 33.7), MAI-UI-8B (Ξ” = 25.1), and Phi-Ground-4B-16C (Ξ” = 13.5). By contrast, older models like OS-Atlas-Base-7B (Ξ” = 9.9) and UGround-V1-7B (Ξ” = 18.2), along with GPT-5.4 (Ξ” = 6.6), show much smaller gaps. The paper is careful not to claim this gap is "direct evidence of overfitting" β€” it could reflect legitimate differences in benchmark design, software coverage, or training data composition β€” but the pattern is suggestive. Models developed and optimized after ScreenSpot-Pro became the dominant benchmark show larger discrepancies with other grounding evaluations, implying that benchmark-specific optimization may be occurring.

The authors provide direct evidence for this interpretation through a fine-tuning experiment: "we further fine-tuned our pretrained Phi-Ground-Any model by incorporating the common-software data used in Phi-Ground, which was collected through Bing Search and may overlap with both benchmarks. The fine-tuning process included only click-based tasks. The results show substantial gains on both benchmarks, while performance on CUActSpot instead declined" (Section 5.2). This is a smoking gun: adding click-only app-specific data boosts ScreenSpot-Pro and UI-Vision scores but actually hurts performance on the more interactionally diverse CUActSpot. The benchmarks are measuring something, but that something is not the full breadth of grounding capability needed for real computer use.

The Dual Bottlenecks: Benchmarks and Data

The paper distills its diagnosis into two specific bottlenecks (end of Section 1):

Bottleneck 1: Lack of benchmarks for evaluating complex operations. Existing grounding benchmarks evaluate one thing β€” whether a model can click on the right widget β€” and declare the model "good at grounding" if it scores well. But as the OSWorld experiment shows, high widget-clicking scores do not translate to high agentic performance. What is missing is a benchmark that evaluates the full range of mouse-based actions that CUAs actually need to perform: dragging from one point to another, drawing along paths, selecting text spans, adjusting table boundaries, tracing object boundaries in images, and doing all of this across diverse modalities (not just GUI widgets, but text documents, spreadsheets, presentation canvases, and natural images).

Bottleneck 2: Lack of large-scale training data for complex interactions. The widget-and-click-centric prior has produced enormous datasets β€” tens of millions of web elements with click labels β€” but almost no data for drag operations, multi-point drawing, or interactions with non-widget modalities. The paper argues this is a data scarcity problem at its core: "we hypothesize that this issue largely stems from the scarcity of data for complex interactions" (Abstract). If models have never been trained on examples of dragging a cell corner to extend a formula, they cannot be expected to perform such operations reliably, regardless of the underlying model architecture.

These two bottlenecks are interdependent. Without benchmarks that evaluate complex operations, there is no pressure on the research community to collect or synthesize training data for them. Without such training data, even if benchmarks existed, model performance would be poor. The paper's dual contribution β€” a benchmark (CUActSpot) and a data synthesis pipeline β€” is designed to break this deadlock simultaneously from both sides.

Theoretical Positioning: The "Variety Scaling" Hypothesis

Beyond the practical bottlenecks, the paper advances a theoretical claim about what drives generalization in computer-use grounding. The authors observe that "compared with simply scaling the amount of training data within a single modality, increasing data diversity substantially improves the model's general interactive capability, a phenomenon we term variety scaling" (Section 1).

This is a direct challenge to the prevailing "scale is all you need" ethos that has dominated language model research. The paper is not claiming that raw data volume is irrelevant β€” the 50M samples they generate are substantial β€” but rather that what matters is the diversity of tasks and modalities, not the number of examples within any single one. The waterfall ablation experiments in Figure 5 are designed to test precisely this hypothesis, showing that adding 1M table samples provides benefits not just for table grounding but also for GUI, text, and canvas modalities (cross-modal transfer), and that scaling a single modality from 2M to higher volumes yields diminishing returns that plateau quickly.

This connects to a broader debate in deep learning about whether generalization arises from massive data scale or from carefully curated diversity. For text-based LLMs, the argument has largely been settled in favor of scale: more tokens, from whatever source, tend to improve performance. But the paper suggests that for visuomotor tasks like screen grounding β€” where the model must learn to map natural language instructions to precise spatial coordinates across dramatically different visual contexts β€” diversity may be more fundamental than scale. A model that has seen 10M examples of clicking buttons will still not know how to drag a selection marquee around a cloud shape on a canvas, because that requires learning a different spatial reasoning primitive. Only by explicitly covering the diversity of interaction types can the model acquire these primitives.

The paper provides evidence for this through the cross-task generalization analysis in Table 4: the model trained on 20 detailed task types succeeds on 27 of the 33 detailed tasks in CUActSpot, "suggesting limited compositional generalization across detailed tasks" (Section 5.3). The model can combine primitives learned from separate training tasks β€” for example, merging text interaction skills with visual region manipulation skills to handle text embedded within images, even when such compositions were never explicitly present in the training data. This compositional generalization is precisely what variety scaling aims to maximize.

How This Paper Positions Itself

The paper positions itself as addressing a gap between current research focus and real-world deployment needs. The existing GUI grounding research community has been optimizing for a narrow, widget-click-centric problem formulation, driven by benchmarks that reward models for memorizing software-specific icon locations and widget layouts. Meanwhile, the actual failure modes of deployed CUAs β€” documented through the GPT-5.4 user study β€” reveal that complex, multi-point interactions across diverse modalities are the true bottleneck.

The paper's four contributions β€” benchmark, data synthesis pipeline, trained model, and empirical studies β€” are designed as an integrated intervention. CUActSpot is not proposed as a replacement for existing benchmarks but as a complementary diagnostic that probes capabilities the current benchmarks miss. The data synthesis pipeline is not presented as the definitive solution to data scarcity but as a demonstration that scalable, renderer-based generation can produce effective training data for interaction types that are difficult or impossible to collect through web crawling. The Phi-Ground-Any-4B model serves as a baseline validation that training on such data produces usable grounding capabilities. And the empirical studies β€” particularly the waterfall ablations and the OSWorld comparison β€” provide evidence for the variety scaling hypothesis and the benchmark-reality mismatch.

The paper does not claim to solve the long-tail problem entirely. The Limitations section (Section 6) acknowledges that CUActSpot "does not exhaustively cover real-world workflows, especially long-horizon and stateful scenarios" and that "improving alignment with real-world distributions remains an important direction for future work." Rather, the paper aims to redirect the field's attention toward the complex interactions that currently dominate failure distributions, and to provide the tools (benchmark, data pipeline, empirical insights) needed for the community to make progress on this under-addressed problem.

3. Technical Approach

3.1 Reader Orientation

This paper develops two tightly coupled artifacts: (1) a manually curated benchmark called CUActSpot that evaluates how well vision-language models can perform complex mouse-based computer interactions (clicks, drags, and multi-point drawing) across five visual modalities, and (2) a fully synthetic data generation pipeline that programmatically renders screenshots with precise coordinate annotations, then uses a large language model to produce natural-language instructions paired with executable action traces, yielding ~50M training samples. The core problem both artifacts address is that existing grounding models are trained almost exclusively on simple widget-clicking data and evaluated on widget-clicking benchmarks, yet real computer-use agents fail disproportionately on complex, multi-point interactions like dragging cell boundaries, selecting text spans, tracing object boundaries, and drawing on canvases β€” precisely the types of operations that dominate productivity software workflows. The solution takes the shape of a complementary evaluation protocol that penalizes metric gaming through banned regions and ordered correctness checks, paired with a renderer-first synthesis architecture that decouples visual generation from task specification, enabling an LLM to reason over structured coordinate metadata to construct arbitrarily complex interaction tasks without any human annotation.

3.2 Big-Picture Architecture (Diagram in Words)

The overall system has two independent but mutually informing subsystems:

  1. CUActSpot Benchmark Construction Pipeline β€” A manually executed process that defines interaction taxonomies, designs task templates, renders or captures source screenshots, annotates correct and banned regions, writes instructions, validates with human testers, and evaluates models against rule-based scoring criteria. This is a one-time construction effort producing 206 evaluation samples.

  2. General Action Grounding Data Synthesis Pipeline β€” An automated, scalable pipeline with five modality-specific rendering engines (GUI via Playwright, Text via PyQt5, Table via HTML+CSS, Canvas via matplotlib, Natural Image via SAM), each producing screenshot + structured annotation pairs, followed by a unified LLM-based task generation stage where OpenAI o3 selects elements from the annotations, optionally performs coordinate arithmetic, and produces natural-language instructions paired with PyAutoGUI code traces. This pipeline generates ~50M training samples used to pre-train or mid-train the Phi-Ground-Any-4B model.

Information in the synthesis pipeline flows as follows: modality-specific renderer β†’ screenshot + JSON annotations (bounding boxes, control points, vertices, boundaries, references) β†’ o3 receives screenshot + selected element annotations β†’ o3 reasons over coordinates, designs tasks, performs arithmetic β†’ o3 outputs instruction + PyAutoGUI code with symbol-mapped coordinates + metadata. The output is directly consumable as supervised training data for a VLM to learn instruction-to-coordinate mapping.

The evaluation pipeline for CUActSpot flows: model receives instruction + screenshot β†’ model predicts coordinates for key points β†’ evaluation engine checks predicted points against Correct Regions and Banned Regions using ordered/unordered matching rules β†’ per-sample binary success β†’ aggregate success rate.

3.3 Roadmap for the Deep Dive

  • First, the CUActSpot evaluation framework β€” the rule definitions (Correct Region, Banned Region, rank ordering), the three evaluation rules with their priority hierarchy, and the metric computation β€” because understanding what the benchmark measures and how it prevents gaming is foundational to everything else.
  • Second, the benchmark construction process and the resulting taxonomy β€” the five modalities, the action type breakdown by number of key points, the 12 high-level and 33 detailed task types, and the design philosophy of reducing software-specific knowledge β€” because this defines the target capability space the synthesis pipeline must cover.
  • Third, the data synthesis pipeline architecture β€” the shared LLM-based task generation pattern and the modality-specific rendering engines β€” because the pipeline is the paper's primary technical infrastructure contribution.
  • Fourth, each modality-specific rendering engine in detail β€” GUI via CommonCrawl + Playwright, Text via PyQt5 font/document rendering, Table via HTML+CSS template generation, Canvas via procedural shape library with PPT-style selection markers, Natural Image via SAM with GPT-4o captioning and contour extraction β€” because each modality requires fundamentally different annotation structures and task construction strategies.
  • Fifth, the LLM task generation stage β€” the system prompts, the coordinate computation capabilities, the PyAutoGUI code generation protocols, the symbol mapping mechanism, and the diversity maximization strategies β€” because this is where raw annotations become training data.
  • Sixth, the training setup and data composition β€” including the base model choice (Phi-3.5-VL), hyperparameters, data proportions, and the rationale for including OpenCUA data β€” because this connects the synthesis pipeline to empirical validation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and data engineering paper whose core technical idea is that complex computer-use interactions can be evaluated through a rule-based scoring framework with banned regions and order constraints, and that training data for such interactions can be synthesized at scale by programmatically rendering visual scenes with structured geometric annotations and then delegating task design to a capable LLM that reasons over those annotations.

CUActSpot Evaluation Rules and Metric

The benchmark introduces a scoring framework designed to handle not just single-point clicks but also multi-point drags and multi-point drawing trajectories, while preventing models from gaming the metric by indiscriminately predicting points across the screen.

Region definitions. The evaluation framework defines two types of spatial regions for each sample:

  • Correct Region: A set of one or more areas on the screen where the model's predicted coordinates must fall for the sample to be considered correct. In Figure 3, these are shown as green-shaded regions. A Correct Region may optionally carry a rank attribute, which is an integer used for order-sensitive tasks. For example, in a task requiring the user to drag from the center of a square outward to draw an inscribed circle, the start point must fall within a Correct Region associated with rank 1 (the center area), and the end point must fall within a Correct Region associated with rank 2 (any point on the circle's radius). The paper provides a concrete example: "dragging along an arrow is order-sensitive, whereas dragging to select a span of text is order-insensitive, since the selection can be made by dragging either from front to back or from back to front" (Section 3.1).

  • Banned Region: A set of areas where the model's predicted actions must NOT occur. The purpose is explicitly stated: "prevent metric gaming in tasks with N key points, where a model might otherwise click randomly across the entire screen in an attempt to inflate its score" (Section 3.1). Without banned regions, a model that predicts many points scattered across the screen could achieve a high score simply by having at least one point land in each Correct Region by chance. Banned regions penalize this behavior by marking the entire sample as incorrect if any predicted point falls within them.

A key structural invariant: the paper guarantees that for each sample, either all Correct Regions have a rank attribute, or none of them do. This binary distinction enables the evaluation engine to apply either ordered or unordered matching without per-region branching logic.

Evaluation rules with priority ordering. The three rules are applied sequentially, with Rule 1 taking highest priority. This means if a sample fails Rule 1, Rules 2 and 3 are never checked β€” the sample is immediately marked incorrect.

  • Rule 1 (Banned Region check): "If a sample defines any Banned Region, then the sample is marked as incorrect as soon as any coordinate predicted by the model (e.g., for a drag or a click) falls within a banned region." This is a hard constraint: one violation, one predicted point in the wrong place, and the entire multi-point action is invalidated. This directly counters the metric gaming concern β€” a model that scatters points widely to cover Correct Regions will almost certainly hit a Banned Region and fail.

  • Rule 2 (Ordered Correct Region matching): This applies when Correct Regions carry rank attributes. The matching logic is: "for each rank (where a given rank may correspond to one or more regions), it is sufficient for a key point to fall within any one of the regions associated with that rank; moreover, the sequence of predicted key points must match the order of the ranks." In the upper-right example of Figure 3, dragging from the center outward to draw a circle is an order-sensitive action. The model must predict its first key point (the drag start) somewhere within the center Correct Region (rank 1) and its second key point (the drag end) somewhere on the circle's radius Correct Region (rank 2). The model does NOT need to hit a specific exact point β€” any point within the designated region suffices. But the order of the predicted points must correspond to the rank order: the first predicted key point is matched against rank-1 regions, the second against rank-2 regions, and so on. If a rank has multiple associated regions (e.g., the circle's radius could be represented as several disconnected arc segments), hitting any one of them suffices.

  • Rule 3 (Unordered Correct Region matching): This applies when Correct Regions lack rank attributes. "The prediction is considered correct as long as each correct region contains at least one key point." This is strictly easier than ordered matching β€” the model's predicted key points are treated as an unordered set, and correctness requires only that every Correct Region receives at least one hit from the predicted set. For a text selection task, for example, the model might predict a start point and an end point. The two Correct Regions might correspond to the start-of-selection area and the end-of-selection area. Since the selection can be made in either direction, the evaluation does not enforce which predicted point maps to which region β€” only that both regions are covered.

Metric computation. For each sample, the evaluation engine applies the three rules in priority order. If the sample passes all applicable rules (or reaches the end without failure), it is marked as correct. The overall benchmark score is the sample success rate β€” the fraction of the 206 CUActSpot samples that are marked correct.

Why this design. The three-rule framework with banned regions and optional ordering is designed to accommodate the full spectrum of mouse-based computer interactions without reducing everything to click-point-in-widget matching (which would fail to capture the sequential dependencies in drag operations) or requiring exact coordinate matching (which would be impossibly strict given pixel-level ambiguity). The banned region mechanism is particularly important because it makes the benchmark robust against naive metric gaming β€” a model cannot simply output many points to increase its chance of hitting target regions, because spreading points across the screen increases the probability of hitting a Banned Region and failing the entire sample.

CUActSpot Benchmark Construction and Taxonomy

The benchmark was constructed entirely manually, following a systematic decomposition of computer-use interactions and a deliberate design philosophy to minimize software-specific knowledge requirements.

Modality taxonomy. The authors first categorized GUI interaction targets into five common types (Section 3.2):

  • GUI: Standard GUI widgets such as buttons, checkboxes, search bars, icons, and text labels within application chrome. The paper gives the example of clicking a heart button on sunglasses (a GUI icon click).

  • Text: Operations performed directly on text content, such as insertion (clicking to place a cursor between characters or words), selection (dragging to highlight a span of text), and drag-moving text. The paper clarifies: "Note that clicking a button containing text does not fall into this category" β€” a button with the word "Submit" on it is a GUI element, not a Text element. Text operations are those performed within document-editing contexts like Microsoft Word or Notepad.

  • Table: Spreadsheet-style operations as exemplified by Excel. In addition to clicking individual cells, this category includes dragging cell borders to resize columns or rows, dragging the corner of a cell to extend formulas (the fill handle operation), and dragging to select ranges of cells.

  • Canvas: Operations on graphical objects within presentation or diagramming software, as in PowerPoint. This includes clicking shapes, dragging shapes to reposition them, dragging control points on shapes to resize or reshape them, dragging selection marquees around multiple shapes, and connecting shapes with arrows or lines.

  • Natural Image: Interactions within natural images, as in Photoshop or GIMP. This includes clicking or dragging over specific image regions for operations such as selecting objects for cutout, adjusting curves, applying localized edits, drawing boundaries for image segmentation, and using tools like the eraser or mask brush.

Action type taxonomy by key points. For each modality, the task space was further refined according to the number of key points involved (Section 3.2, Table 1):

  • One point (click): Actions requiring a single coordinate prediction, such as clicking on a specific cell, a specific shape, a specific position within text, or a specific region within an image.

  • Two points (drag): Actions requiring two coordinate predictions (start and end), such as dragging a file icon to a target folder, dragging to select a span of text, dragging a cell corner to extend a formula, dragging a shape from one position to another, or dragging a control point to resize an object. The two points may be ordered (drag along an arrow requires specific start-then-end sequence) or unordered (text selection can be made by dragging in either direction).

  • N points (draw): Actions requiring a sequence of coordinate predictions forming a trajectory, such as drawing a polygon around an object for cutout, tracing a boundary with a freehand selection tool, or zigzagging across an image region with an eraser tool. These are the most complex interaction type and the least represented in existing benchmarks and training data.

Detailed task breakdown. The authors iteratively brainstormed realistic operations commonly performed in various software applications and collected a diverse set of tasks. After collection and annotation, three additional individuals independent of the original annotator attempted all tasks. The authors then "revised any ambiguous task descriptions and removed all tasks that could not be completed by humans" (Section 3.2). The final dataset contains 206 samples spanning 12 high-level tasks and 33 detailed tasks (Table 1, with the full breakdown in Tables 5 and 6 of Appendix A.1).

Tables 5 and 6 in the appendix enumerate every detailed task type. Representative examples include:

  • GUI-1pt-icon: "Click on the heart button of the sun glasses."
  • GUI-2pt-slide bar: "Drag the video progress bar to approximately 10 minutes and 15 seconds."
  • Text-1pt-between text: "Click once at the position before 'and the people' to set the cursor."
  • Text-2pt-select text span: "Drag the mouse to highlight all the text numbered 1-17. (including the numbers)."
  • Table-1pt-content cell: "Please select the cell that contains 'banana' in the table."
  • Table-2pt-corner: "Drag the lower-right corner of cell I7 down to the lower-right corner of cell I13 to apply the formula and calculate the sums for the other six rows."
  • Canvas-2pt-line/arrow: "I have selected node 1 in the diagrams document. Now please drag the blue arrow on its right and connect it to the left side of node 2."
  • Canvas-Npt-point: "Connect the black dots in the figure from smallest to largest."
  • Image-2pt-point: "I am using the image cropping feature to crop out the white boat in the frame without leaving any extra space."
  • Image-Npt-zig-zag mask: "I have now selected the eraser tool. Please drag it over the entire deer to select it."

Design philosophy: reducing the knowledge barrier. A deliberate choice in CUActSpot's construction was to reduce the amount of domain-specific software knowledge required to complete tasks. The paper contrasts this with benchmarks like ScreenSpot-Pro, where "many samples require domain-specific software knowledge to determine the correct target" (Section 3.2). For example, a ScreenSpot-Pro task might ask the model to "click the dodge tool icon button in Photoshop" β€” if the model does not know what the dodge tool icon looks like, it cannot succeed regardless of its grounding ability. The paper argues that "while such expertise is certainly relevant to CUA, it also introduces a potential confound: model performance may be influenced by how well the model is fitting to a particular software environment, rather than reflecting its grounding ability itself." CUActSpot tasks are designed so that the visual referent is identifiable from the instruction and the screenshot content without requiring memorized knowledge of specific software tool icons or menu layouts. This makes the benchmark a purer measure of visual grounding and action execution capability.

Data Synthesis Pipeline: Shared LLM Task Generation Architecture

The data synthesis pipeline is unified by a common pattern: a modality-specific renderer produces a screenshot and a structured JSON annotation file containing precise geometric metadata about every interactive element in the scene; then an LLM (OpenAI o3) receives the screenshot, a selection of annotated elements, and a modality-specific system prompt; the LLM designs tasks by selecting elements from the annotations, optionally performing arithmetic over the provided coordinates to construct non-trivial spatial relationships, and outputs a natural-language instruction paired with executable PyAutoGUI code where all absolute coordinates are replaced with symbolic placeholders (x1, y1, x2, y2, ...) that are resolved through a coordinate_map dictionary.

The symbol-mapping mechanism. This is the central technical pattern that enables the LLM to generate coordinate-precise action traces without hallucinating numbers. In the training data, each sample consists of five components (as defined in the GUI system prompt in Appendix C.1):

  • prompt: A natural-language request for an operation that can be executed on the current screen. The prompt should be diverse in style β€” it can be a direct command ("click the update button"), a first-person contextualized request ("I want to find xxx but I do not see xxx in this screen, can you scroll to find them?"), or a vague request that requires interpretation ("help me enlarge this image" when the screen contains a magnifying glass icon).

  • response: The expected reply from the agent, which includes a chain-of-thought analysis of the prompt followed by a ```python ``` code block containing PyAutoGUI code. All coordinates in the code block are replaced with symbols (x1, y1, x2, y2, ...). The chain-of-thought portion must contain NO actual coordinates β€” this is a critical constraint because "in the future, we may do data augmentation on the coordinates" (system prompt, Appendix C.1). By keeping coordinates out of the reasoning text, data augmentation operations like translation, scaling, or rotation of the scene can be applied without invalidating the natural language portion of the training data.

  • coordinate_map: A dictionary mapping the substitute symbols to their actual values, e.g., {"x1": 0.12345, "y1": 0.67891}. This map can be empty {} if the action does not use any coordinates from the provided elements (e.g., a scroll or keyboard action).

  • used_elements: A list of the indices of elements from the input annotations that were used in constructing this training datum. This can also be empty if no annotated element coordinates were needed.

  • action-type: A string identifying the action category, e.g., "click", "scroll", "combined: moveTo and dragTo", "combined: mouseDown moveTo and mouseUp". This metadata enables filtering and balancing of action types during training.

Action type classes. The system prompts define three classes of action types based on the number of coordinate sets used and the PyAutoGUI action type (Appendix C.1):

  • ZeroSet: Actions requiring no coordinates from the input annotations β€” scroll, typewrite, hotkey. The used_elements and coordinate_map are empty.

  • OneSet: Actions requiring one coordinate β€” moveTo, click, mouseDown, mouseUp. A click task uses one element's center point.

  • TwoSet and combined: Actions requiring two or more coordinates, possibly with multiple function calls chained together β€” combined: moveTo and dragTo, combined: mouseDown moveTo and mouseUp (for selecting text or dragging something), combined: click and type (for focusing a text box and then typing). The paper emphasizes that the LLM is instructed to "maximize the diversity of actions combination, instead of making the most appropriate data, which means even the button cannot be drag, you can drag, even the region can not type, you can also click and type" (system prompt). This is a deliberate choice: the goal is to generate a diverse dataset for model pre-training, not to simulate realistic user behavior. Physically implausible interactions (e.g., dragging a button that is not actually draggable in the source application) are acceptable because they still teach the model to map spatial relationships to coordinate trajectories.

Coordinate computation by the LLM. A critical capability leveraged in the pipeline is the LLM's ability to perform arithmetic over the provided coordinate annotations to construct tasks with non-trivial spatial relationships. The paper provides a concrete example in Figure 4 (bottom), where the LLM reasons: let the center of an arrow be at (x1, y1), the tip of the arrow at (x1, yc), and the top control point of an ellipse be at (x2, yt). To make the arrow tip coincide with the top of the ellipse, the arrow center must be moved from (x1, y1) to (x2, y2), where y2 = yt + y1 - yc. This kind of reasoning β€” translating a relational spatial instruction ("make the arrow tip touch the top of the ellipse") into precise absolute coordinates through arithmetic β€” enables the pipeline to generate tasks that are substantially more complex than simple element-to-element mappings. The LLM is explicitly encouraged to perform such computations: "you are encouraged to do calculation based on the provided info. For example, you can generate task like 'drag the bottom-right scaling control point of xxx to make it a 30px larger in both w and h' and use coordinate (x, y) and (x+30, y+30)" (Canvas system prompt, Appendix C.4).

Diversity maximization in task generation. Each system prompt includes explicit quantity requirements and diversity constraints. For the GUI modality, the LLM is required to generate 10 data points total, with at least 5 being OneSet tasks, and the rest dynamically allocated. For the Table modality, 4 data points are required, with at least 1 being OneSet. For the Canvas modality, 10 different data points are required, with at most 3 using the center point, at most 4 using other control points, and the remainder using calculated coordinates. These constraints are designed to prevent the LLM from defaulting to the easiest task type (click on center point) and to force coverage of the full action space.

GUI Modality Rendering: CommonCrawl + Playwright Pipeline

The GUI modality uses web-based rendering because websites naturally contain the button, icon, link, and form elements that constitute standard GUI widgets. The data construction follows a multi-stage pipeline inherited from Phi-Ground, described in detail in Appendix C.1 and illustrated in Figure 8.

Stage 1: Index and domain deduplication. The pipeline starts with the CC-MAIN-2024-46 crawl from CommonCrawl. After basic URL deduplication (exact match), language filtering (retaining only English), and webpage status filtering (retaining only 2xx, 301, and 302 status codes), approximately 2.6 billion URLs remain. These URLs originate from 45.6 million unique domains. The authors observe that pages from the same domain exhibit strong layout consistency, and the distribution of pages per domain is long-tailed, with the largest domain containing 204K different pages. To prevent the model from overfitting to the layout patterns of high-frequency domains, the authors perform domain-level sampling: "no more than 50 pages were selected from each domain" (Appendix C.1). After this sampling, 475.45 million URLs remain.

Stage 2: Rendering with Playwright. The authors use the Selenium library and Google Chrome Driver (which internally uses the Playwright-like WebDriver protocol) to render webpage screenshots. During rendering, three design parameters are randomized to increase visual diversity: screen resolution (randomly selected from three pixel areas corresponding to 1080p, 2K, and 4K), aspect ratio (randomly chosen between 2:1 and 1:2), and element filtering based on HTML attributes. The element filtering stage is critical: the authors "designed several rules for filtering and retaining them" to preserve elements that are "likely to be interactive components" β€” specifically, elements with relevant tags, event handlers, ARIA roles, or CSS classes suggesting interactivity. At this stage, the pipeline saves webpage screenshots, filtered element information (bounding boxes, types, text content), and layout graphs (where different element types β€” interactive text buttons, interactive icon buttons, images β€” are color-coded). After this stage, 285 million webpages are retained.

Stage 3: Rule-based filtering. Fine-grained filters and deduplication techniques are applied at both the webpage and element levels to eliminate erroneous or overly simplistic webpages. The paper lists specific filter categories (Figure 8): "Incomplete webs, NaΓ―ve layouts, Interactive element [filtering], Icon numbers [filtering], <div> dedup., Scroll page dedup." These are designed to remove pages that would produce low-quality training data β€” pages with no interactive elements, pages whose visual structure is trivially simple, pages that are essentially duplicates of others (e.g., paginated views of the same content), and pages where the interactive elements are not meaningfully distinct. After this phase, 73.5 million webpages remain.

Stage 4: Element selection and GPT-4o labeling. The final selection of elements for annotation considers both spatial distribution (to ensure coverage across the entire canvas) and element type (to balance the representation of text buttons, icon buttons, and other interactive elements). The authors "discretize and uniformly sample across various regions of the canvas" and "prioritize sampling icon elements, as they are less frequent" (Appendix C.1). Ten elements are sampled per screenshot. For each sampled element, GPT-4o is used to generate a natural-language description of the element. The output of this stage is approximately 10.5 million element-description pairs.

Stage 5: o3 task generation. The annotated elements (with their descriptions and center-point coordinates) are provided to o3 along with the screenshot and the GUI-specific system prompt (reproduced in full in Appendix C.1). The system prompt instructs o3 to generate diverse task types β€” ZeroSet (scroll, typewrite, hotkey), OneSet (moveTo, click, mouseDown, mouseUp), and TwoSet/combined (moveTo+dragTo, mouseDown+moveTo+mouseUp, click+type, moveTo+scroll) β€” with explicit instructions to use diverse PyAutoGUI parameters (left vs. right click, number of clicks, scroll amount, drag duration). The prompt samples shown in Figure 9 demonstrate the output diversity: single clicks, double right-clicks, triple-clicks with intervals, click-hold-and-drag sequences, mouseDown-moveTo-mouseUp for text selection, and dragTo with duration parameters.

Scale. The GUI modality contributes the largest share of training data: approximately 30.4 million samples generated, of which 6.8 million are used during training (Table 7).

Text Modality Rendering: PyQt5 Font and Document Rendering

The text modality requires fundamentally different rendering because the interactive elements are individual characters, words, and text spans rather than discrete widgets. The annotation challenge is that if coordinates were recorded for every individual character, the annotation files would become "excessively large" (Appendix C.2). Instead, the pipeline targets a small set of specific interaction scenarios.

Rendering infrastructure. The authors download 2,500 open-source English fonts and manually capture or collect approximately 200 text-background images at different resolutions, including blank Microsoft Word documents and screenshots of Notepad windows. Using the PyQt5 library, textual content sourced from Wikipedia and GitHub is rendered onto the blank regions of these backgrounds with randomly sampled fonts, colors, sizes, and font weights, while recording the coordinates of every individual character. This means the renderer knows, for every character on screen, its bounding box β€” enabling precise annotation of text spans for selection and insertion tasks.

Task scenario coverage. The pipeline covers six scenarios: two data types (code and natural language) crossed with three task types. The three task types are:

  • Drag-selecting a long text span: The task requires highlighting a multi-sentence or multi-line passage. The annotation includes the start and end coordinates of the span. Because long spans are typically unique within the document (the specific combination of starting and ending text is unlikely to appear elsewhere), the reference can be relatively straightforward β€” e.g., "Drag the mouse to select the text 'airline based in Tabubil, Papua ... three Bombardier Dash 8 aircraft'."

  • Drag-selecting a short text span: The task requires highlighting a word or short phrase. Because short spans may appear multiple times within the document, the reference must include surrounding context to disambiguate. The paper gives the example: "Make the selection that corresponds to 'hometown, Takoma' that is surrounded by the context: 'It was named after Fahey's hometown, Takoma Park, Maryland, a suburb of'" (Figure 10).

  • Clicking to place the insertion cursor: The task requires clicking at a precise inter-character position β€” e.g., "Click right at the junction of 'd uses the Granite kernel, but otherwise is a free' / 'standing CAD system.'" The annotation specifies the exact coordinate between two characters.

GPT reformulation. After manually identifying the target regions and their coordinates, the authors use GPT to reformulate the task descriptions for stylistic diversity. This is a lighter touch than the full o3 task generation pipeline used for other modalities β€” the text modality's interaction space is inherently narrower (there are only so many ways to select text or place a cursor), so the main variation comes from the document content, fonts, and background styles rather than from novel task types.

Scale. The text modality contributes approximately 6.1 million samples, of which 5.0 million are used during training (Table 7).

Table Modality Rendering: HTML + CSS Template Generation

The table modality is the most pipeline-intensive of the five, requiring a four-stage process to generate diverse table structures, contents, and visual styles (Appendix C.3).

Stage 1: Seed table collection. The authors collect open-source TableVQA-related datasets from Hugging Face and other sources. These datasets provide tables in Markdown, HTML, or arXiv formats, yielding approximately 16,000 seed tables.

Stage 2: GPT-based table evolution. Using GPT, each seed table is transformed in two ways to create diversity. First, the topic is changed β€” for example, a table of working hours becomes a table of math score statistics. Second, the topology is modified β€” new merged cells are added, new rows or columns are inserted, and the overall structure is altered. Each seed table produces approximately 10 evolved variants, resulting in roughly 160,000 unique table structures.

Stage 3: CSS template generation. Using o3, the authors generate CSS style sheets in diverse visual styles based on various open-source CSS libraries. The critical design property is that the style sheet parameters β€” color, font size, cell size and type, presence or absence of borders β€” are designed to be adjustable and randomly sampled. The authors create approximately 1,000 base templates, and for each template, sample 10 different parameter configurations, yielding 10,000 distinct CSS instances. This two-level generation (template design + parameter randomization) ensures that tables with identical HTML structure can have dramatically different visual appearances.

Stage 4: HTML-CSS combination and cell masking. By randomly pairing HTML tables with CSS instances, the pipeline generates complete table webpages. A key data augmentation: "half of the tables are selected and most of their cells are randomly masked out" (Appendix C.3). This simulates the common real-world scenario of spreadsheets with many empty cells, which "substantially increases the difficulty of both cell grounding and reference generation" because the model must refer to a specific empty cell by its positional context (e.g., "the empty cell in column B under the header 'Apple'") rather than by its content.

Cell annotation structure. For each rendered table, JavaScript extracts the bounding box of every cell. To prevent the LLM from hallucinating positional information, the pipeline programmatically computes additional metadata for each cell (as illustrated in Figure 11): the row index, column index, corresponding row and column headers, the cell content, and the coordinates of the cell boundaries (top, left, bottom, right). Each cell is also associated with the content values of cells above it (in the same column) and to its left (in the same row), enabling relative spatial references. This rich annotation enables the LLM to generate complex tasks like "drag the bottom-right corner of cell I7 down to the bottom-right corner of cell I13" β€” a task that requires understanding the spatial relationship between two cells and computing the target coordinate from their bounding boxes.

o3 task generation with coordinate computation. The table-specific system prompt (Appendix C.3) explicitly instructs o3 to generate tasks that involve computation: dragging cell edges (computing the mid-point of an edge from the bounding box coordinates and then computing a target coordinate offset by some pixel amount), dragging cell corners to extend formulas (computing the bottom-right corner of a source cell and the bottom-right corner of a target cell below it), and adjusting column widths or row heights. The prompt provides hints like: "if in the given screen, cell A (X1, Y1, X2, Y2) is right above the cell B (when the size of cells are similar, you can compute the coordinate of B is (X1, Y2, X2, 2*Y2-Y1)), you can generate task like drag the right-bottom corner of cell A to the right-bottom corner of cell B." This demonstrates the pipeline's core capability: the LLM is not just selecting elements but performing geometric reasoning over the provided annotations to construct tasks with spatial dependencies.

Scale. The table modality contributes approximately 5.2 million samples, of which 2.0 million are used during training (Table 7).

Canvas Modality Rendering: Procedural Shape Simulator with PPT-Style Markers

The canvas modality is the most geometrically rich, requiring a simulator that can procedurally generate arbitrarily complex scenes of shapes, arrows, and connectors with precise annotations of every geometric control point. The full simulator is described in Appendix C.4.

Canvas and element sampling. Each scene begins with a randomly sampled canvas size: width W ∈ [800, 2560] pixels and height H ∈ [600, 1440] pixels, with a background color drawn in HSV color space. Between 3 and 8 elements are then placed on the canvas. Element types are sampled from a registry of 76 primitive types grouped into nine categories β€” rectangles, ellipses, triangles, quadrilaterals, polygons, stars, arrows, lines/connectors, callouts/decorations, and text boxes β€” with "importance weighting that slightly favors common primitives (rectangle, circle, triangle, star, diamond, basic arrows) over the rarer ones to mimic typical slide distributions" (Appendix C.4). Element sizes range from 8% to 40% of the shorter canvas side, with line-like elements allowed to span up to 60%.

Overlap-aware placement. Elements are placed sequentially using a collision avoidance procedure. For each new element, up to 50 random candidate positions are tried. A candidate bounding box is accepted when its "maximum pairwise overlap ratio with all previously placed boxes (relative to the smaller area) is below 0.25; otherwise the lowest-overlap candidate is retained as a fallback" (Appendix C.4). This ensures that shapes are reasonably well-separated on the canvas, preventing occlusion that would make individual shape selection ambiguous. Square-aspect shapes (circle, square, donut, ring, rounded square) are constrained to equal width and height to preserve geometric semantics β€” a circle must look like a circle, not an ellipse.

Color model. Background, fill, and outline colors are drawn from HSV space with rejection sampling against a redmean-weighted Euclidean distance metric. The paper specifies minimum perceptual gaps of 100 between fill and background, and 60 between fill and outline, ensuring that shapes are visually distinct from both the canvas and their own borders. Outlines are randomized between solid and dashed strokes (with 0.2 probability of dashing) and random widths between 1 and 5 pixels.

PPT-style selection markers. This is the most distinctive feature of the canvas simulator, designed to mimic the visual environment of a presentation editor like PowerPoint when a shape is selected. Each element is overlaid with a carefully constructed set of visual markers, and β€” critically β€” the exact pixel coordinates of every marker are recorded in the annotation. The selection chrome consists of:

  • A thin gray bounding box around the element.
  • Eight red control points at the four corners and four edge midpoints (for line-like elements, only the two endpoints are marked, since edge midpoints on a line are geometrically degenerate).
  • Blue diamond vertex markers at every named polygon vertex. These are skipped for shapes whose vertices are dense curve approximations (heart, cloud, moon, wave) because the vertex count would be impractically large.
  • A rotation handle consisting of a short connector line and a 300-degree circular arrow with a small arrowhead tip, randomly anchored to one of the four bounding-box midpoints.

The recording of these marker coordinates is what enables the pipeline to generate tasks like "drag the top-right scaling control point of the rectangle to make it larger" or "hover the pointer over the little circular rotation handle of the dark sun so it shows the rotate cursor" (as shown in the prompt examples in Figure 13). Without these fine-grained control point annotations, the task space would be limited to center-clicking and shape-to-shape dragging.

Reference expression generation. Each shape is paired with a unique English referring expression to ensure the LLM can unambiguously reference any element in the generated tasks. The base expression follows the pattern "<fill-color>-filled <shape> with <outline-color> outline in the <region>". Color words are obtained through nearest-neighbor lookup against a 44-entry named-color palette using redmean distance. The canvas is divided into a 3Γ—3 spatial grid, producing region descriptors like "upper-left area of the canvas." When two elements collide in the base description (e.g., two red-filled rectangles), a cascade of disambiguation strategies is applied in order: relative-size descriptors ("the largest / smallest"), line-style qualifiers ("solid / dashed outline"), refinement to a finer 5Γ—5 region grid, and finally a reading-order ordinal prefix ("the upper, the second, ..., the lower"). This guarantees that every element in every scene possesses at least one unique reference string.

Annotation structure. For each generated image, the simulator emits a JSON file containing, per shape: the unique ID, the symbolic shape_type, the disambiguated reference string, the bounding box, the center_point, the eight named box_points (top_left, top_center, top_right, right_center, bottom_right, bottom_center, bottom_left, left_center), the rotation_handle_center, the full styling dictionary (fill color, outline color, stroke width, line style), and optional vertices or endpoints dictionaries for polygonal and line-like shapes. This rich structured annotation is what enables o3 to generate diverse tasks in the next stage.

o3 task generation. The canvas system prompt (Appendix C.4) provides the same base structure as the GUI prompt but with canvas-specific instructions emphasizing the use of control points (not just centers), coordinate calculation (e.g., computing midpoints between shapes, offsetting by pixel amounts), and diverse action combinations. The prompt requires 10 data points per scene, with explicit constraints: at most 3 using the center point, at most 4 using other control points, and the remainder using calculated coordinates. The examples in Figure 13 illustrate the output diversity: dragging arrow tips to specific points on other shapes, hovering over rotation handles, dragging selection marquees around shapes, and clicking-to-focus-then-type sequences.

Scale. The canvas modality contributes approximately 4.3 million samples, of which 2.0 million are used during training (Table 7).

Natural Image Modality Rendering: SAM + GPT-4o + Contour Extraction

The natural image modality uses real photographs rather than procedurally generated content, because the interaction targets β€” objects, regions, and their boundaries within natural scenes β€” cannot be easily synthesized. Instead, the pipeline leverages the Segment Anything Model (SAM) to provide region annotations and GPT-4o to provide semantic descriptions.

Region sampling from SAM. For each image in the SAM dataset, the pipeline randomly samples five regions. SAM itself provides the bounding box and segmentation mask for each region. The segmentation mask is a binary image indicating which pixels belong to the region.

GPT-4o captioning. Because SAM regions do not come with semantic descriptions, GPT-4o is used to generate fine-grained natural-language captions for each selected region. This transforms a purely geometric annotation (a mask and bounding box) into a semantically grounded reference that can be used in natural-language instructions β€” for example, "the person on the far right in the red-highlighted area," "the white boat," or "the tree's reflection in the water at the bottom right of the image."

Boundary polygon extraction. The authors apply the Suzuki-Abe contour extraction algorithm to each region's segmentation mask, followed by contour sampling, to obtain polygonal boundary curves β€” specifically, 20-point polygons approximating the region's outline. These boundaries are "primarily used to support operations such as object cutout and zigzag-mask editing in Photoshop-like scenarios" (Section 4.4). The boundary polygon enables tasks like "trace the boundary of the deer to select it" or "draw a zigzag path over the entire region to mask it."

Annotation structure. Each of the five regions per image is associated with: a GPT-4o-generated caption, a bounding box, a 20-point boundary polygon, and a center point computed from the bounding box. These annotations are provided to o3 along with the image.

o3 task generation with N-point drawing. The natural image system prompt (Appendix C.5) introduces two N-point task types that are unique to this modality:

  • Boundary tracing: The model is instructed to use the boundary polygon directly as a sequence of waypoints for a drawing action. If the region is large, the full 20-point polygon is used; if the region is small, a subset of points is sampled in order. The task scenario is typically described as a selection or cutout operation, e.g., "I have now selected the eraser tool. Please drag it over the entire deer to select it."

  • Zigzag masking: To simulate paintbrush or eraser strokes that cover an entire region, the model is instructed to reorder the boundary polygon into a zigzag pattern: from the original ordered list [p1, p2, p3, ..., p19, p20], the reordered sequence [p1, p2, p20, p3, p19, p4, p18, ...] creates a back-and-forth sweeping path that covers the interior of the region. The prompt explains that "this will be a left right left right trail to mask the whole image."

The natural image prompt also emphasizes providing clear operational context in the instruction β€” "Now I am using the image erase tool, I need to draw to cover the region of the man in yellow" β€” to explain why a particular action is being performed. Keywords like "the polygon" or "the blue bbox" are explicitly forbidden in prompts because those visual markers are drawn only for the LLM's understanding and do not appear in the original natural image.

Scale. The natural image modality contributes approximately 4.7 million samples, of which 3.0 million are used during training (Table 7).

Training Setup and Data Composition

The synthetic data is used for pre-training or mid-training a vision-language model, following the data composition specified in Table 7.

Base model choice. The authors adopt Phi-3.5-VL, a 4B-parameter VLM, as the backbone. The rationale is that because the synthetic data is "primarily intended for the pre-training or mid-training stages of VLMs, we require a base model that has not been exposed to GUI-related pre-training" (Section 5.1). Starting from a general-purpose VLM rather than a GUI-specialized one ensures that any grounding capabilities observed after training are attributable to the synthetic data rather than to prior GUI-specific knowledge in the base model. Phi-3.5-VL serves as a clean slate.

Training hyperparameters. The paper specifies (Section 5.1 and Appendix B.1): visual input fixed to 16 crops with data augmentation strategies inherited from Phi-Ground (including dynamic image resolution processing), batch size 5120, learning rate 8 Γ— 10⁻⁡, weight decay 0.01, and gradient clipping at 0.1. The authors note that weight decay and gradient clipping are "important for maintaining training stability" (Appendix B.1), suggesting that training on such diverse multi-modal data without these regularizers would lead to instability. Training uses approximately 100 billion tokens total, requiring about 30 hours on 80 NVIDIA H100 GPUs.

Data composition and sampling weights. Table 7 specifies the exact data proportions used during training:

ModalityTotal SamplesUsed SamplesEpochsTraining Weight
GUI30,432,2426,800,0000.220.34
Text6,083,4005,000,0000.820.25
Table5,242,6302,000,0000.380.10
Canvas4,323,2532,000,0000.460.10
Image4,743,6753,000,0000.630.15
OpenCUA340,6651,200,0003.520.06

Several design choices are notable. The GUI modality, despite having the largest raw sample count, is downsampled to 6.8M used samples (a 0.22 epoch fraction), while the smaller modalities are upsampled to 0.4–0.8 epochs. The training weight column represents the fraction of each training batch drawn from each modality β€” GUI dominates at 34%, reflecting its importance for general widget interaction, while Text at 25% and Image at 15% follow. The OpenCUA data, despite being the smallest source at 340K samples, is oversampled to 1.2M used samples (3.52 epochs) because it is "manually annotated and therefore expected to be of higher quality" (Appendix B.1). However, the paper notes in Section 5.3 that "using only OpenCUA yields unsatisfactory performance due to its limited scale," motivating the need for the synthetic data pipeline.

Training strategy. The authors use a best-checkpoint strategy: "we save checkpoints for every 100 training steps, and report the best checkpoint among them" (Appendix B.2). This is noted as applying to all experiment results in the paper. The ablation studies (Section 5.3 and Figure 5) are conducted by training models on progressively accumulated data compositions β€” starting with 2M GUI samples, then adding 1M Text, 1M Table, 1M Canvas, 1M Image, and finally 0.5M OpenCUA β€” and evaluating each intermediate checkpoint to measure the marginal contribution of each modality.

CUActSpot evaluation protocol. During evaluation on CUActSpot, the paper notes that "CUActSpot exhibits fluctuations of approximately Β±3% between adjacent checkpoints during training, while the variation observed during testing with different temperature settings is around 2%" (Appendix A.1). These error bars provide context for interpreting differences between models in Table 2 β€” differences smaller than 3% may not be statistically reliable.

4. Key Insights and Innovations

Innovation 1: Variety Scaling as a Distinct Principle for Vision-Language Action Models

The paper's most conceptually significant contribution is the formulation of variety scalingβ€”the empirical finding that increasing the diversity of tasks and modalities in training data yields substantially greater improvements in model capability than scaling the volume of data within any single modality. This is not merely an observation about data efficiency; it is a challenge to the prevailing "scale is all you need" assumption that has guided language model development and was implicitly imported into vision-language-action research without explicit testing.

What the field assumed before this work. The dominant paradigm in LLM pretrainingβ€”exemplified by the Chinchilla scaling laws and their successorsβ€”holds that performance improvements are primarily a function of total training tokens, with data diversity treated as a secondary or implicit property that emerges naturally from web-scale crawling. This assumption carried over into GUI grounding research, where the standard recipe was to crawl more webpages, extract more widget bounding boxes, and train on more click-to-element pairs. The implicit logic was that if 10M samples produce a certain accuracy, 50M samples of the same type should produce better accuracy. The paper's ablation experiments in Figure 5 directly test this logic and show it breaks down for visuomotor grounding tasks.

What the evidence shows. Figure 5 (specifically subfigure 1-1) demonstrates that independently scaling the training budget for individual modalities produces sharply diminishing returnsβ€”the curves plateau quickly, suggesting that additional examples of the same interaction type (e.g., more button-clicking data) do not translate into continued capability improvements. In contrast, the waterfall plots (Figures 1-2 through 2-3) show that adding entirely new modalitiesβ€”even in relatively small quantities (1M samples each)β€”produces consistent, cumulative gains. The overall CUActSpot score climbs from 14.8% (GUI-only, 2M samples) to 37.1% (all modalities + OpenCUA, 6.5M total samples), with most of the gains coming from adding diversity rather than from the additional volume itself. The +21.9% jump in Table accuracy when adding 1M Table samples (Figure 1-4) is particularly telling: the model had never seen table interactions before, so even a modest amount of table-specific data produces a dramatic capability unlock that 30M GUI samples could never provide.

Why this is a fundamental shift, not an incremental refinement. Variety scaling is not the same as "more data is better" with a diversity footnote. It implies that the learning dynamics for visuomotor grounding are qualitatively different from those for text-based language modeling. In text, the underlying structureβ€”grammar, vocabulary, discourse patternsβ€”is shared across all domains, so scaling within-domain data still transfers across-domain benefits. In visuomotor grounding, by contrast, the mapping from visual context to spatial coordinates is modality-specific in ways that do not transfer automatically. Learning to click on a button teaches the model nothing about how to drag a cell corner in a spreadsheet, because the visual features of buttons and cell corners bear no structural similarity, and the action primitive (single point vs. two-point sequence with spatial dependency) is fundamentally different. The model must learn each interaction primitive separately, and only through exposure to that specific primitive does the capability emerge.

This has direct implications for how training data should be constructed for embodied and agentic AI systems. If variety scaling holds broadlyβ€”and the paper's cross-task generalization results in Table 4, where the model trained on 20 detailed task types succeeds on 27, suggest it mayβ€”then the optimal data strategy is not to maximize total sample count but to maximize the coverage of distinct task types and modalities, even at the expense of per-type volume. This inverts the conventional data collection priority: spend effort on identifying and covering new interaction types rather than on scaling existing ones.

Relationship to cross-task generalization. The compositional generalization evidence in Table 4 supports the variety scaling hypothesis at a mechanistic level. The model trained on 20 distinct detailed task types could successfully complete 27 of the 33 detailed tasks in CUActSpot, including tasks that were never explicitly present in training. The paper speculates that "a model that learns to interact with textual elements and to manipulate visual regions separately may subsequently acquire the ability to operate on text embedded within visual content, such as editing text inside a presentation figure or selecting text from natural images, even when such compositions are not explicitly present in the training data." This is evidence that variety scaling works not just through additive accumulation of independent skills but through enabling compositional recombination of learned primitivesβ€”the more distinct primitives the model possesses, the more novel compositions it can assemble at inference time.

Innovation 2: Diagnosing and Quantifying the Benchmark-Reality Mismatch in Computer-Use Grounding

The paper provides the first systematic empirical evidence that existing grounding benchmarks systematically overestimate real-world grounding capability by testing a narrow, widget-click-centric skill that fails to correlate with agentic performance on complex interactions. This is not merely a critique of specific benchmarks; it is a diagnostic framework for understanding when and why benchmark scores diverge from deployment utility, with implications for how the field should evaluate progress.

The mismatch, quantified. The OSWorld experiment in Table 3 is the paper's clearest demonstration. By controlling for planning (using GPT-5.4 as the planner for all grounders) and varying only the grounding model, the authors isolate grounding capability as the sole variable affecting end-to-end agentic success. The resultβ€”that models with 45-point advantages on ScreenSpot-Pro show no corresponding advantage on OSWorld, and in some cases perform worseβ€”is a direct refutation of the assumption that widget-clicking benchmark scores are a valid proxy for general grounding ability. A model scoring 71.1 on ScreenSpot-Pro (GUI-Owl-1.5-8B-Instruct) achieves 37.7 on OSWorld, while a model scoring 26.3 (Phi-Ground-Any-4B) achieves 42.4. The direction of the gap runs opposite to the benchmark ranking.

Why this matters beyond this specific paper. This finding exposes a construct validity problem in the GUI grounding evaluation ecosystem. Benchmarks measure what they measure, but the research community treats them as measuring something broaderβ€”"grounding ability"β€”without verifying that the measured construct aligns with the real-world capability of interest. The paper's diagnostic contribution is to identify why the misalignment occurs: existing benchmarks test a single interaction type (single-point click on GUI widgets) across software interfaces that require substantial domain-specific knowledge, while real computer use requires diverse interaction types (clicks, drags, multi-point draws) across diverse modalities (widgets, text, tables, canvases, images) with reduced dependence on memorized software knowledge.

The fine-tuning experiment in Section 5.2 provides causal evidence for this interpretation. Adding click-only, app-specific data from Bing Search increased ScreenSpot-Pro and UI-Vision scores but decreased CUActSpot performance. The model was optimizing for the wrong objectiveβ€”it was learning to recognize application-specific icons and layouts rather than learning generalizable action grounding. This is a textbook case of Goodhart's Law applied to benchmark-driven research: when a metric becomes a target, it ceases to be a good metric.

The ScreenSpot-Pro vs. UI-Vision gap as a diagnostic indicator. The paper identifies a pattern in Table 2 that serves as a litmus test for benchmark over-optimization: the performance gap (Ξ”) between ScreenSpot-Pro and UI-Vision has grown substantially in recent models. Older models like OS-Atlas-Base-7B (Ξ” = 9.9) and UGround-V1-7B (Ξ” = 18.2) show modest gaps, while newer models optimized after ScreenSpot-Pro became the dominant benchmark show gaps exceeding 25-33 points. GPT-5.4, which was not optimized for any specific grounding benchmark, shows a gap of only 6.6 points. The paper is appropriately cautious about interpreting this as overfitting, acknowledging that differences in benchmark design and training data coverage could explain the gapβ€”but the temporal pattern (gap size increasing with benchmark age) strongly suggests benchmark-specific optimization is occurring, and the fine-tuning experiment confirms the mechanism is plausible.

A new principle for benchmark design. Implicit in the paper's analysis is a principle for constructing evaluation benchmarks for agentic AI: test the interaction diversity, not the domain knowledge. CUActSpot is designed to require minimal software-specific knowledgeβ€”the visual referent for a task is identifiable from the instruction and screenshot without needing to know what a particular tool icon looks likeβ€”while maximizing the diversity of spatial reasoning primitives (click, drag-ordered, drag-unordered, multi-point draw) across distinct visual modalities. This decouples grounding ability from software memorization, producing a benchmark that the paper demonstrates is more predictive of real-world agentic performance (via the OSWorld correlation) than benchmarks that conflate the two.

Innovation 3: Renderer-First Synthesis with LLM Geometric Reasoning as a Scalable Paradigm for Action Data Generation

The paper's data synthesis pipeline represents a paradigm shift in how training data for visuomotor grounding is generated: rather than crawling real interfaces and retroactively extracting coordinate labels (the dominant approach inherited from web-crawling + accessibility-tree extraction), the pipeline procedurally renders visual scenes with full geometric knowledge and then delegates task design to an LLM that reasons over structured coordinate metadata. This inverts the traditional relationship between visual content and annotationβ€”in the renderer-first paradigm, the annotations are the ground truth that generates the visuals, not an imperfect extraction from them.

The limitation of crawl-and-extract pipelines. Prior data generation approaches (inherited from web agent work and used in datasets underlying SeeClick, OS-Atlas, and others) follow a pattern: crawl webpages or capture application screenshots, extract element bounding boxes from the DOM or accessibility tree, and pair elements with natural-language instructions (either from nearby text, human annotation, or LLM relabeling). This approach has fundamental limitations that the paper implicitly identifies:

  • Coordinate noise: Extracted bounding boxes from DOM elements do not always align precisely with visual rendering, especially for complex CSS layouts, overlapping elements, or dynamically positioned content.
  • Interaction type constraint: The crawl-and-extract paradigm naturally produces click-on-element data because that is what DOM elements represent. Dragging, drawing, and selecting are not represented in the DOM structure and thus cannot be extractedβ€”they must be manually annotated or synthesized through other means.
  • Modality constraint: Web crawling primarily produces GUI widget data. Tables embedded in webpages can be extracted, but text documents, presentation canvases, and natural images with precise region annotations are not naturally produced by web crawling.
  • Annotation cost: Extracted bounding boxes are cheap but imprecise. Human-verified annotations are precise but expensive. The pipeline cannot easily scale to 50M samples with high-quality annotations under either regime.

How renderer-first synthesis resolves these constraints. By rendering visual scenes from code, the pipeline achieves perfect coordinate precision by constructionβ€”every bounding box, vertex, control point, and boundary polygon is known exactly because the rendering code computed it. There is no extraction error, no misalignment between DOM and visual rendering, no ambiguity about what constitutes the "true" bounding box. More importantly, the renderer can generate any interaction type that can be expressed programmatically: dragging control points on shapes (canvas), dragging cell corners (table), selecting text spans between known character coordinates (text), drawing along known boundary polygons (natural image). The interaction space is limited only by what the renderer can simulate and what the LLM can describe, not by what can be extracted from crawled data.

The LLM geometric reasoning component is the second critical innovation. Rather than attempting to procedurally generate task descriptionsβ€”which would require encoding human knowledge about what constitutes a meaningful or natural computer-use task into code, an essentially intractable problemβ€”the pipeline delegates this to a capable LLM (o3) and provides it with structured geometric metadata. The LLM does not need to perceive the visual scene (though it receives the screenshot as additional context); it primarily reasons over the coordinate annotations to construct tasks. The examples in the paperβ€”computing arrow tip positions relative to shape vertices, calculating drag offsets for cell resizing, reordering boundary polygons into zigzag pathsβ€”demonstrate that this is genuine geometric reasoning, not simple template matching. The LLM is performing arithmetic over the annotation coordinates to generate tasks with non-trivial spatial dependencies.

Scalability and generality. The paradigm is scalable because it decouples visual generation from annotation quality. Adding a new shape to the canvas simulator requires writing rendering code and defining its geometric annotation structureβ€”a one-time engineering costβ€”after which the LLM can generate unlimited task variations involving that shape. The pipeline does not require human annotation at any stage beyond the initial system prompt design and the seed data collection (e.g., the initial 16K seed tables for the table modality). The 50M samples are generated entirely automatically from seed data, rendering code, and LLM calls.

The generality of the paradigm extends beyond the five modalities covered in this paper. Any visual environment that can be procedurally rendered with extractable geometric metadataβ€”diagrams, charts, maps, 3D viewports, video timelines, audio waveformsβ€”could be integrated into the same pipeline by writing a modality-specific renderer and a modality-specific system prompt. The paradigm is limited primarily by the coverage of the renderers, not by data availability or annotation cost. This makes it fundamentally more scalable than crawl-and-extract approaches for covering the long tail of computer-use interactions.

Innovation 4: Banned-Region Evaluation as a Principle for Metric-Robust Benchmarking of Sequential Spatial Actions

The CUActSpot evaluation framework introduces a deceptively simple but conceptually powerful mechanismβ€”banned regionsβ€”that addresses a fundamental challenge in evaluating multi-point spatial predictions: how to prevent models from gaming the metric by outputting many points to increase the probability of hitting target regions by chance. This is not merely a practical fix; it is a principle for constructing benchmarks that test genuine spatial understanding rather than reward stochastic coverage strategies.

The metric gaming problem for multi-point actions. In single-point click evaluation (the dominant paradigm in prior benchmarks), metric gaming is difficult because the model outputs exactly one point, and either it falls within the target widget's bounding box or it does not. A model cannot meaningfully "cheat" by outputting many points because the evaluation protocol expects exactly one. But in multi-point actionsβ€”dragging from start to end, drawing along a pathβ€”the number of predicted points scales with the action complexity. A model that predicts many scattered points across the screen would, by random chance, hit more target regions than a model that predicts the minimum required number of points. Without a penalty for incorrect points, the benchmark would reward coverage-maximizing strategies over precision strategies, measuring point-scattering ability rather than spatial understanding.

Why banned regions are a principled solution. The banned region mechanism inverts the scoring logic from "reward for hitting correct regions" to "penalize for hitting incorrect regions." By defining explicit exclusion zones and applying them as the highest-priority evaluation rule (Rule 1, checked before any correctness rules), the benchmark makes it impossible to inflate scores through indiscriminate point generation. The more points a model outputs, the higher the probability that at least one point falls in a banned region and invalidates the entire sample. This creates the correct incentive: models should output the minimum number of precisely targeted points needed to satisfy the instruction.

The paper does not belabor the theoretical justification, but the mechanism connects to established principles in multiple-choice test design (penalizing wrong answers to discourage guessing) and in search evaluation (precision-recall tradeoffs where indiscriminate retrieval damages precision). By adapting this logic to the spatial domain, CUActSpot achieves metric robustness that prior grounding benchmarksβ€”which either evaluated single-point clicks (making gaming a non-issue) or used simple point-in-region checks without exclusion zones (making gaming possible but unaddressed)β€”did not need or did not provide.

The rank-ordering mechanism as a complementary principle. For ordered actions (dragging along a specific direction, drawing vertices in sequence), Rule 2 requires that the sequence of predicted key points matches the rank order of the correct regions. This prevents a different form of metric gaming: a model that correctly identifies the spatial locations but outputs them in the wrong order would fail, even though an unordered evaluation might mark the sample correct. The distinction between ordered and unordered actionsβ€”and the dataset-level guarantee that each sample is either entirely ordered or entirely unorderedβ€”formalizes an aspect of spatial action evaluation that prior work ignored entirely, since click-only benchmarks have no sequential dependency to evaluate.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use the CUActSpot benchmark consisting of 206 manually constructed samples spanning five modalities (GUI, Text, Table, Canvas, Natural Image) and three action types (click, drag, draw). For external comparison, the authors also evaluate on ScreenSpot-Pro (a professional high-resolution GUI grounding benchmark), UI-Vision (a desktop-centric GUI benchmark), and OSWorld (an end-to-end agentic benchmark). The base model training data is the synthetic corpus described in Section 3, totaling approximately 50M samples with the per-modality breakdown in Table 7.

  • Base model(s). The paper uses Phi-3.5-VL (4B parameters) as the pretraining backbone, chosen specifically because it has "not been exposed to GUI-related pre-training" (Section 5.1), making it a clean slate for evaluating the synthetic data's contribution. Baselines include a range of open-source grounding models spanning 2B to 72B parameters (UGround-V1, OS-Atlas-Base, InfiGUI-R1, UI-Venus-Ground, GUI-G2, MAI-UI, GUI-Owl-1.5, InfiGUI-G1, EvoCUA, UI-TARS-1.5, OpenCUA) as well as GPT-5.4 (accessed through Azure OpenAI). The FLOPs comparison is between Phi-Ground-Any-4B and models up to 32B parameters (Table 2) and GPT-5.4.

  • Metrics. The primary metric across all CUActSpot evaluations is sample success rate β€” the fraction of the 206 benchmark samples for which the model's predicted coordinates satisfy the three-rule evaluation framework (Banned Region check β†’ Ordered Correct Region matching β†’ Unordered Correct Region matching). For ScreenSpot-Pro and UI-Vision, the metric is standard click-point-within-widget-bounding-box accuracy. For OSWorld, it is task completion success rate under a maximum of 30 actions per task.

  • Baselines. The paper compares against 17 models in Table 2, including: UGround-V1-2B and -7B (Gou et al., 2024), OS-Atlas-Base-7B (Wu et al., 2024), Phi-Ground-4B-16C (Zhang et al., 2025), InfiGUI-R1-3B (Liu et al., 2025), UI-Venus-Ground-7B and -72B (Gu et al., 2025), GUI-G2-7B (Tang et al., 2025), MAI-UI-2B and -8B (Zhou et al., 2025), GUI-Owl-1.5-8B-Think and -Instruct (Xu et al., 2026), InfiGUI-G1-7B (Liu et al., 2026), EvoCUA-8B and -32B (Xue et al., 2026), UI-TARS-1.5-7B (Qin et al., 2025), OpenCUA-7B and -32B (Wang et al., 2025), and GPT-5.4 (OpenAI, 2026). The paper notes that it reports "only the models for which we were able to successfully reproduce the benchmark scores reported in their papers within a margin of Β±5%" (Section 5.2), implying that other well-known models (such as GTA1) were excluded due to reproduction difficulties.

  • Generation budget / compute accounting. The paper does not use a formal FLOPs-based compute budget for inference comparisons (unlike the training-inference tradeoff analysis in the reference example). Instead, models are compared at inference time through their outputs on the same set of benchmark samples, with the training data scale and composition as the independent variable in the ablation studies. The training budget is approximately 100B tokens, consuming about 30 hours on 80 NVIDIA H100 GPUs (Appendix B.1). The data ablation experiments control for total training samples while varying modality composition (Figure 5, Table 8).

  • Cross-validation / statistical protocol. No formal cross-validation is used for the CUActSpot benchmark evaluation β€” models are evaluated on all 206 samples. The paper notes that "CUActSpot exhibits fluctuations of approximately Β±3% between adjacent checkpoints during training, while the variation observed during testing with different temperature settings is around 2%" (Appendix A.1), providing empirical bounds on result stability. For the OSWorld experiments, GPT-5.4 is uniformly used as the planner to control for planning capability as a confound (Table 3). For the data ablation studies, a best-checkpoint strategy is used: "we save checkpoints for every 100 training steps, and report the best checkpoint among them" (Appendix B.2), with all experiments in the paper following this protocol.

Main Quantitative Results

Benchmark Comparison: CUActSpot versus ScreenSpot-Pro and UI-Vision (Table 2)

The headline result in Table 2 is that Phi-Ground-Any-4B achieves 44.4% overall accuracy on CUActSpot, outperforming all open-source models below 32B parameters. The next best sub-32B models are OpenCUA-7B at 39.8%, UI-TARS-1.5-7B at 28.5%, and EvoCUA-8B at 23.9%. GPT-5.4 achieves 63.6%, and OpenCUA-32B achieves 52.5%, establishing an upper bound for reference.

Critically, the ranking on CUActSpot differs substantially from the ranking on ScreenSpot-Pro and UI-Vision. Models that dominate ScreenSpot-Pro show dramatically weaker performance on CUActSpot:

  • GUI-Owl-1.5-8B-Instruct scores 71.1 on ScreenSpot-Pro (the highest among all models in the table) but only 15.4 on CUActSpot overall.
  • MAI-UI-8B scores 65.8 on ScreenSpot-Pro but only 15.3 on CUActSpot.
  • Phi-Ground-4B-16C, the paper's own prior model, scores 38.0 on ScreenSpot-Pro (second-lowest in the table) and 5.0 on CUActSpot (lowest).

The CUActSpot performance is broken down by modality, revealing that models with strong ScreenSpot-Pro scores are not uniformly weak β€” they are specifically weak on the non-GUI modalities:

  • GUI modality: Several models achieve respectable scores (infiGUI-G1-7B at 44.7, OpenCUA-7B at 42.1, UI-TARS-1.5-7B at 42.1), with Phi-Ground-Any-4B at 44.7 essentially matching the best.
  • Text modality: The performance spread is dramatic. OpenCUA-7B leads at 53.1, followed by Phi-Ground-Any-4B at 68.8 (the highest among all models). Several models score 0.0 on Text entirely (UGround-V1-2B, UGround-V1-7B, OS-Atlas-Base-7B, InfiGUI-R1-3B).
  • Table modality: Phi-Ground-Any-4B achieves 40.6, exceeded only by GPT-5.4 (65.6) and OpenCUA-32B (39.1). Most sub-8B models score between 4.7 and 12.5.
  • Canvas modality: Scores are generally low, with Phi-Ground-Any-4B at 33.3 and GPT-5.4 at 47.6. Multiple models score 0.0.
  • Natural Image modality: Similar pattern β€” Phi-Ground-Any-4B at 33.3, exceeding all open-source models below 32B, with GPT-5.4 at 47.6.

The ScreenSpot-Pro vs. UI-Vision gap (Ξ” column in Table 2) grows substantially in recent models: GUI-Owl-1.5-8B-Instruct shows Ξ” = 33.7 and MAI-UI-8B shows Ξ” = 25.1, while older models like OS-Atlas-Base-7B show Ξ” = 9.9 and GPT-5.4 shows Ξ” = 6.6. The paper interprets this as evidence that "many recent models exhibit a substantial performance gap between the two benchmarks" (Section 5.2), consistent with the hypothesis that benchmark-specific optimization is occurring in models developed after ScreenSpot-Pro became the dominant evaluation target.

Fine-tuning experiment confirming sensitivity to training data distribution. The "+ APP data finetuned" row in Table 2 shows Phi-Ground-Any-4B after incorporating "the common-software data used in Phi-Ground, which was collected through Bing Search and may overlap with both benchmarks. The fine-tuning process included only click-based tasks" (Section 5.2). The effect is stark:

  • ScreenSpot-Pro improves from 26.3 β†’ 41.5 (+15.2 points).
  • UI-Vision improves from 15.8 β†’ 29.7 (+13.9 points).
  • CUActSpot overall declines from 44.4 β†’ 36.5 (βˆ’7.9 points), with specific modality drops in Text (68.8 β†’ 59.4, βˆ’9.4), Table (40.6 β†’ 32.8, βˆ’7.8), Canvas (33.3 β†’ 19.0, βˆ’14.3), and Natural Image (33.3 β†’ 19.0, βˆ’14.3).

This is a causal demonstration that click-only, app-specific training data improves widget-clicking benchmarks while degrading performance on diverse interaction types. The paper explicitly calls this finding a demonstration of "the sensitivity of existing benchmarks to the distribution of the training data" (Section 5.2).

OSWorld Grounding-Only Comparison (Table 3)

Table 3 presents a controlled experiment isolating grounding capability in an agentic setting. The design: all models use GPT-5.4 as the planner (generating single-step natural-language instructions), with the grounding model varying. This controls for planning capability, making grounding the sole independent variable affecting OSWorld success rates (30 action maximum).

The results:

PlannerGrounderScreenSpot-ProOSWorld Success
GPT-5.4GUI-Owl-1.5-8B-Instruct71.137.7
GPT-5.4MAI-UI-8B65.838.2
GPT-5.4GPT-5.444.544.1
GPT-5.4Phi-Ground-Any-4B26.342.4

Despite GUI-Owl-1.5-8B-Instruct and MAI-UI-8B having ScreenSpot-Pro scores 44.8 and 39.5 points higher than Phi-Ground-Any-4B respectively, their OSWorld success rates are actually lower (37.7 and 38.2 vs. 42.4). GPT-5.4 with its own grounding achieves the highest OSWorld score (44.1) despite having a ScreenSpot-Pro score of only 44.5 β€” substantially below both GUI-Owl-1.5-8B-Instruct (71.1) and MAI-UI-8B (65.8).

The paper interprets this as evidence that "the top models on CUActSpot (i.e., GPT-5.4, OpenCUA, EvoCUA, UI-TARS) also happen to report results on OSWorld" and that "research efforts that genuinely focus on agentic settings and explicitly aim to optimize for them are also more likely to collect training data with broader modality coverage and more diverse interaction types" (Section 5.2). The correlation between CUActSpot performance and OSWorld performance is stronger than the correlation between ScreenSpot-Pro performance and OSWorld performance.

Data Ablation: Variety Scaling versus Volume Scaling (Figure 5, Table 8)

Figure 5 is the paper's central empirical contribution, presenting a systematic ablation of training data composition through a series of waterfall plots and a scaling curve.

Subfigure 1-1: The case against pure volume scaling. This figure plots CUActSpot overall accuracy against training data volume for each modality independently. The curves show sharply diminishing returns β€” increasing GUI data from 500K to nearly 3M samples produces only modest and eventually plateauing gains. No single modality's scaling curve shows sustained logarithmic improvement. The paper interprets this as evidence that "increasing the amount of training does not yield sustained improvements" (Figure 5 caption).

Subfigures 1-2 through 2-3: Waterfall plots of incremental modality addition. These eight bar charts show the marginal contribution of adding each data modality to the training mixture, starting from a base of 2M GUI samples and sequentially adding 1M Text, 1M Table, 1M Canvas, 1M Image, and 0.5M OpenCUA samples. The results (with exact numbers from Table 8):

  • CUActSpot-GUI (1-2): 2M GUI: 31.6%. +1M Text: 34.2% (+2.6). +1M Table: 23.7% (βˆ’10.5). +1M Canvas: 28.9% (+5.2). +1M Image: 26.3% (βˆ’2.6). +0.5M OpenCUA: 34.2% (+7.9). Total after all additions: 34.2%.

  • CUActSpot-Text (1-3): 2M GUI: 6.3%. +1M Text: 31.3% (+25.0, the single largest gain in any subfigure). +1M Table: 28.1% (βˆ’3.2). +1M Canvas: 25.0% (βˆ’3.1). +1M Image: 28.1% (+3.1). +0.5M OpenCUA: 31.3% (+3.2). Total: 31.3%.

  • CUActSpot-Table (1-4): 2M GUI: 21.9%. +1M Text: 28.1% (+6.2). +1M Table: 40.6% (+12.5). +1M Canvas: 46.9% (+6.3). +1M Image: 50.0% (+3.1). +0.5M OpenCUA: 56.3% (+6.3). Total: 56.3%.

  • CUActSpot-Canvas (2-1): 2M GUI: 9.4%. +1M Text: 9.4% (+0.0). +1M Table: 10.9% (+1.5). +1M Canvas: 25.0% (+14.1). +1M Image: 29.7% (+4.7). +0.5M OpenCUA: 37.5% (+7.8). Total: 37.5%.

  • CUActSpot-Image (2-2): 2M GUI: 4.8%. +1M Text: 4.8% (+0.0). +1M Table: 9.5% (+4.7). +1M Canvas: 16.7% (+7.2). +1M Image: 23.8% (+7.1). +0.5M OpenCUA: 26.2% (+2.4). Total: 26.2%.

  • CUActSpot-Overall (2-3): 2M GUI: 14.8%. +1M Text: 21.5% (+6.7). +1M Table: 22.5% (+1.0). +1M Canvas: 28.5% (+6.0). +1M Image: 31.6% (+3.1). +0.5M OpenCUA: 37.1% (+5.5). Total: 37.1%.

  • ScreenSpot-Pro (2-4): 2M GUI: 16.4%. +1M Text: 18.1% (+1.7). +1M Table: 19.7% (+1.6). +1M Canvas: 23.1% (+3.4). +1M Image: 22.8% (βˆ’0.3). +0.5M OpenCUA: 24.6% (+1.8). Total: 24.6%.

Several patterns are notable:

  1. Cross-modal transfer is substantial but asymmetric. Adding 1M Text data improves CUActSpot-GUI by only +2.6 points, but adding 1M GUI data improves CUActSpot-Table by +21.9 (the starting point). The transfer is stronger from richer modalities to sparser ones than vice versa.

  2. The "own-modality" jump is largest for the first addition. When a modality's data is first introduced, the accuracy on that modality jumps dramatically: +25.0 for Text when Text data is added (1-3), +12.5 for Table when Table data is added (1-4), +14.1 for Canvas when Canvas data is added (2-1), +7.1 for Image when Image data is added (2-2). Subsequent additions from other modalities produce only incremental gains on that modality.

  3. The Tableβ†’CUActSpot-GUI negative transfer (βˆ’10.5 in 1-2). Adding 1M Table data causes a 10.5-point drop in GUI accuracy, from 34.2 to 23.7. The paper does not directly explain this regression, but it is consistent with catastrophic interference β€” the model's parameters shift to accommodate table-specific spatial reasoning, which interferes with the previously learned GUI-clicking representations. The GUI accuracy partially recovers with subsequent additions (Canvas, Image, OpenCUA), suggesting that broader diversity ultimately helps stabilize the representation.

  4. OpenCUA provides consistent late-stage gains. Despite contributing only 0.5M samples (the smallest volume by far), adding OpenCUA improves every modality: +7.9 GUI, +3.2 Text, +6.3 Table, +7.8 Canvas, +2.4 Image, +5.5 Overall, +1.8 ScreenSpot-Pro. This is consistent with the paper's statement that OpenCUA is "manually annotated and therefore expected to be of higher quality" β€” its high information density per sample makes it disproportionately valuable.

Cross-Task Generalization (Table 4)

Table 4 quantifies the model's ability to generalize beyond its explicit training:

MetricCount
Detailed tasks in CUActSpot33
Detailed tasks in training data20
Detailed tasks the model can complete (β‰₯1 success)27

The model trained on 20 detailed task types succeeds on 27 of the 33 detailed task types in CUActSpot, demonstrating compositional generalization across 7 task types that were never explicitly present in training. The paper provides the example of "text embedded within visual content, such as editing text inside a presentation figure or selecting text from natural images" β€” compositions where skills learned separately (text manipulation + visual region interaction) combine to handle tasks requiring both. This is direct evidence for the variety scaling hypothesis: diverse training across separate primitives enables recombination into novel capabilities.

Ablation Studies and Robustness Checks

  • Single-modality scaling curves (Figure 5, subfigure 1-1): Independently scaling the training budget for each modality (GUI, Text, Table, Canvas, Image) shows that beyond approximately 1–2 million samples per modality, additional within-modality data produces diminishing or plateaued returns on CUActSpot overall accuracy. The curves flatten rather than continuing logarithmic improvement, motivating the shift from volume scaling to variety scaling.

  • Incremental modality addition waterfall (Figure 5, subfigures 1-2 through 2-3, Table 8): Progressively adding 1M samples from each new modality produces cumulative accuracy gains on CUActSpot overall (14.8 β†’ 21.5 β†’ 22.5 β†’ 28.5 β†’ 31.6 β†’ 37.1), with the total gain (+22.3) substantially exceeding what would be achievable by scaling any single modality to the same total volume. The marginal contribution of each modality varies by evaluation target β€” adding Canvas data improves CUActSpot-Canvas (the own-modality effect, +14.1) and has minimal impact on CUActSpot-Text (+0.0), confirming that transfer is modality-specific rather than uniform.

  • ScreenSpot-Pro as a control (Figure 5, subfigure 2-4, Table 8): Adding diverse modalities produces only modest improvements on ScreenSpot-Pro (16.4 β†’ 18.1 β†’ 19.7 β†’ 23.1 β†’ 22.8 β†’ 24.6). The total gain of +8.2 points is substantially smaller than the +22.3 gain on CUActSpot, confirming that ScreenSpot-Pro measures a narrower capability (widget-clicking) that is less sensitive to modality diversity. The negative transfer from adding 1M Image data (βˆ’0.3 on ScreenSpot-Pro) suggests that widget-clicking representations can be slightly disrupted by training on fundamentally different visual contexts (natural images with object boundaries rather than GUI elements).

  • Effect of application-specific fine-tuning (Table 2, "+ APP data finetuned" row): Adding click-only, app-specific data from Bing Search increases ScreenSpot-Pro (+15.2) and UI-Vision (+13.9) while decreasing CUActSpot overall (βˆ’7.9). This is a critical negative result: training on data that improves widget-clicking benchmarks can actively harm performance on diverse interaction types. The specific modality breakdown shows that Text (βˆ’9.4), Canvas (βˆ’14.3), and Natural Image (βˆ’14.3) are the primary casualties, while GUI itself improves (+7.9, from 44.7 to 52.6). This confirms that Click-vs-diverse interaction capability involves partially competing representations.

  • GPT-5.4 as a cross-check on benchmark bias (Table 2, Ξ” column): GPT-5.4, which was not optimized for any specific grounding benchmark, shows a ScreenSpot-Pro vs. UI-Vision gap of only 6.6 points β€” dramatically smaller than the 25–34 point gaps observed in recent benchmark-optimized models. This serves as an existence proof that it is possible to achieve good performance on both benchmarks simultaneously, and that the large gaps in other models are likely attributable to training data distribution choices rather than inherent benchmark conflict.

  • OSWorld grounder comparison with planning controlled (Table 3): Using GPT-5.4 as the planner for all grounders eliminates planning as a confounding variable. The result that grounders with high ScreenSpot-Pro scores (GUI-Owl-1.5-8B-Instruct at 71.1, MAI-UI-8B at 65.8) achieve lower OSWorld success (37.7, 38.2) than Phi-Ground-Any-4B (42.4, with ScreenSpot-Pro of 26.3) is a robustness check confirming that ScreenSpot-Pro scores do not predict agentic grounding capability. The experimental design (fixed planner, varying grounder) is a methodological contribution in itself for isolating grounding quality in end-to-end evaluations.

  • Model scale versus training data diversity: The models in Table 2 span from 2B to 72B parameters. The fact that Phi-Ground-Any-4B (4B parameters) outperforms all models below 32B on CUActSpot, and achieves 42.4 on OSWorld versus 37.7–38.2 for 8B models with higher ScreenSpot-Pro scores, suggests that training data diversity may be a stronger determinant of grounding capability than model scale within this parameter range. However, the comparison is confounded by different training data sources and scales across models, so this is an observation rather than a controlled experiment.

  • Checkpoint stability (Appendix A.1): The authors report that "CUActSpot exhibits fluctuations of approximately Β±3% between adjacent checkpoints during training, while the variation observed during testing with different temperature settings is around 2%." These error bars provide a calibration for interpreting differences between models in Table 2 β€” differences of less than 3% may not be reliably above the training noise floor.

Critical Assessment

This paper makes three central empirical claims: (1) that CUActSpot measures grounding capabilities not captured by existing benchmarks, (2) that variety scaling β€” increasing task and modality diversity β€” is more effective than volume scaling for improving grounding capability, and (3) that Phi-Ground-Any-4B achieves state-of-the-art performance among sub-32B models on complex interaction grounding. Each claim has varying degrees of evidential support.

Claim 1: CUActSpot measures capabilities that existing benchmarks miss.

The evidence for this claim is strong and multi-faceted. The ranking inversion between ScreenSpot-Pro and CUActSpot (Table 2) β€” where models with ScreenSpot-Pro scores above 70 achieve CUActSpot scores below 16 β€” is a clear demonstration that the benchmarks measure different constructs. The OSWorld experiment (Table 3) provides external validation: CUActSpot ranking aligns better with agentic performance than ScreenSpot-Pro ranking. The fine-tuning experiment showing that click-only app-specific data improves ScreenSpot-Pro while degrading CUActSpot is a causal manipulation confirming that the two benchmarks reward different model properties.

However, there are important scope limitations. CUActSpot contains only 206 samples across 33 detailed task types. This is a small evaluation set by modern standards β€” each task type is represented by an average of about 6 samples, and individual modality-action combinations (e.g., Canvas-Npt operations) may be represented by only a handful. The reported Β±3% checkpoint fluctuation means that for small modality subsets, the measurement noise could be comparable to the between-model differences. For example, the Natural Image modality contains a subset of the 206 samples (exact count not reported), and differences of 2–5 percentage points between models on this modality may not be statistically reliable.

Additionally, the claim that CUActSpot "reduces reliance on specialized knowledge" is asserted but not verified through a controlled experiment. The authors state that they removed tasks that could not be completed by humans and revised ambiguous descriptions, but there is no measurement of whether the remaining tasks genuinely minimize domain-specific knowledge dependence. A cleaner test would be to evaluate models with and without software-specific pretraining and measure the performance gap on CUActSpot versus ScreenSpot-Pro β€” expecting a smaller gap on CUActSpot if it indeed requires less specialized knowledge. This experiment was not conducted.

Claim 2: Variety scaling is more effective than volume scaling.

This is the paper's most conceptually significant claim, and the evidence is suggestive but has important limitations. The waterfall ablations (Figure 5, Table 8) clearly show that adding 1M samples from a new modality typically produces a larger accuracy gain on that modality than adding another 1M samples from an existing modality. The single-modality scaling curves (Figure 5, subfigure 1-1) show plateauing behavior that supports the diminishing returns argument.

However, the experimental design has a confound that weakens the variety-versus-volume interpretation: the sequential addition experiments test both increasing diversity and increasing total data volume simultaneously. When moving from "GUI 2M" to "GUI 2M + Text 1M," the total training data increases from 2M to 3M samples. The observed gains could be partially attributable to the 50% increase in total training tokens rather than to the diversity increase per se. A cleaner ablation would hold total data volume constant β€” for example, comparing a model trained on 5M GUI-only samples against a model trained on 1M each of GUI, Text, Table, Canvas, and Image (also 5M total). The paper does not report this experiment. The discussion in Section 5.3 of comparing Figure 1-1 and 2-3 β€” "the key variable driving continual learning is the diversity of task types" β€” is suggestive but observational rather than causally isolated.

The negative transfer from Table data onto GUI accuracy (βˆ’10.5 in Figure 1-2) is an important finding that the paper under-analyzes. This is a form of catastrophic interference where diverse training actually harms previously acquired capabilities. The subsequent partial recovery (+5.2 from Canvas, +7.9 from OpenCUA) suggests that broader diversity can mitigate this interference, but the mechanism is not explained. This complicates the "variety scaling is always beneficial" narrative β€” there appear to be interference effects that depend on the specific modalities being combined and the order in which they are introduced. The paper would be strengthened by an analysis of training dynamics (loss curves per modality during sequential training) and by experiments testing different modality addition orders.

The claim that cross-task generalization occurs through "compositional generalization" (Table 4) is supported only by the observation that the model succeeds on 27 of 33 detailed tasks versus 20 in training. The paper provides one speculative example (text-in-images) but no systematic analysis of which specific task types the model succeeds on and whether they are genuinely compositional combinations of training tasks. A qualitative breakdown of the 7 novel successful task types β€” showing what primitives they combine β€” would substantially strengthen this claim.

Claim 3: Phi-Ground-Any-4B is state-of-the-art among sub-32B models.

This claim is technically accurate given the CUActSpot numbers (44.4 overall, exceeding OpenCUA-7B at 39.8 and all others below 32B at 28.5 or below). However, the "state-of-the-art" framing is narrow. The model's ScreenSpot-Pro score of 26.3 is far below many comparably-sized models (MAI-UI-8B at 65.8, GUI-Owl-1.5-8B-Instruct at 71.1). Whether the model is "state-of-the-art" depends entirely on which benchmark one values as the ground truth for grounding capability. The paper's position is that CUActSpot is a better proxy for real-world performance than ScreenSpot-Pro, and the OSWorld results provide some supporting evidence, but the claim should be understood as "best on CUActSpot" rather than "best overall."

Missing experiments that would strengthen the paper:

  1. Constant-volume diversity ablation. Compare a 5M sample GUI-only model against a 5M sample multi-modality model (1M each of 5 modalities) to isolate diversity from total volume. This is the most direct test of the variety scaling hypothesis and its absence is the largest gap in the experimental design.

  2. Interference analysis of modality combinations. The Table-on-GUI negative transfer (βˆ’10.5) suggests complex interference patterns. Ablations testing different modality addition orders and measuring per-modality loss dynamics would clarify whether this is a fundamental property of multi-modal grounding training or an artifact of the specific training recipe.

  3. Per-detailed-task breakdown for CUActSpot. The 33 detailed tasks are aggregated into 5 modalities and 1 overall score. A finer-grained breakdown showing which specific task types improve with which data additions would illuminate the mechanisms of cross-task transfer and compositional generalization.

  4. Larger-scale training of Phi-Ground-Any. The paper shows plateauing single-modality curves at ~2M samples but the multi-modality model plateaus at 6.5M total samples (Table 8). What happens at 50M samples with full data diversity? Does the composite curve also plateau, or does multi-modality training sustain logarithmic scaling longer? This is directly relevant to whether the synthesis pipeline can support continued scaling.

  5. Direct comparison between crawl-extract and renderer-synthesized data. The paper claims the renderer-first paradigm is superior to crawl-and-extract approaches for covering complex interactions, but there is no experiment where a model is trained on equal volumes of crawl-extracted data versus renderer-synthesized data and compared on CUActSpot. Such a comparison would directly validate the pipeline's claimed advantage.

  6. Human performance on CUActSpot. The benchmark was validated by having "three additional individuals, independent of the original annotator, attempt them" and removing tasks "that could not be completed by humans" (Section 3.2), but no human baseline accuracy is reported. Knowing the human ceiling would calibrate the model scores β€” is 63.6% (GPT-5.4) close to human performance or still far below it? Without human baselines, the room for improvement is unknown.

Conditions under which claims hold:

  • Variety scaling is more effective than volume scaling appears to hold when the base model has already been exposed to a moderate volume of a single modality (2M GUI samples) and additional within-modality data shows diminishing returns. It is untested whether this holds from a cold start (very small per-modality volumes).

  • CUActSpot better reflects real-world agentic performance holds specifically for the OSWorld benchmark with a 30-action budget. It is untested on other agentic benchmarks (Windows Agent Arena, AndroidWorld, WebArena) or on real-world deployment metrics.

  • Phi-Ground-Any-4B outperforms all sub-32B open-source models holds on CUActSpot specifically. It does not hold on ScreenSpot-Pro or UI-Vision. The claim's significance depends on the reader's assessment of CUActSpot's validity as a grounding evaluation.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted For

The assumption or constraint. The compute-optimal allocation framework depends on knowing prompt difficulty before selecting a strategy. The paper's method for estimating difficulty β€” generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) β€” is extraordinarily expensive. The authors explicitly flag this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The paper's headline efficiency claim β€” 4Γ— improvement over best-of-N β€” is computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution. Estimating difficulty via 2048 samples costs more than the largest inference budgets studied (256–512 generations), meaning the true cost could be dominated by the estimation step. The 4Γ— figure is best understood as an upper bound on achievable efficiency β€” the improvement conditional on already knowing the difficulty β€” rather than a realized end-to-end gain. Deploying the compute-optimal policy in practice would require solving the exploration-exploitation tradeoff the paper identifies but does not resolve.

What evidence exists. The paper does not measure this overhead. Figures 4 and 8 plot performance against generation budget for strategy execution only; the cost of generating 2048 samples to bin each question is suppressed. The predicted-difficulty variant (using PRM scores rather than ground-truth labels) still requires those 2048 samples and PRM scoring passes β€” it removes only the label requirement, not the sample cost.

Mitigation status. The paper acknowledges the problem and sketches future directions (Section 8: "pretraining or finetuning models to directly predict difficulty of a question"), but the difficulty predictor is not built or evaluated. Adaptive difficulty estimation β€” starting with a small number of samples and dynamically allocating the remaining budget β€” is mentioned as an analogy to the exploration-exploitation tradeoff (Section 3.2) but is not implemented. Until lightweight difficulty estimation is demonstrated, the compute-optimal gains remain conditional on an oracle that is more expensive than the optimization it enables.

The 14Γ— Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-Versus-Inference Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), where both data and parameters are scaled equally. The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding β€” no majority voting, no best-of-N, no search β€” while the smaller model deploys compute-optimal test-time strategies. This makes the comparison asymmetric: the smaller model gets optimized inference while the larger model gets none.

The consequence. A Chinchilla-optimal model trained with 14Γ— more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining β€” for example, +27.8% relative improvement on easy questions at R β‰ͺ 1 for revisions (Figure 1, upper-right bar chart) β€” may shrink or reverse against a properly compute-optimal larger model. Similarly, giving the larger model even a modest test-time compute budget (e.g., best-of-8) would create a much stronger baseline. The paper's conclusion that "test-time compute can substitute for pretraining under certain conditions" is valid relative to the tested baseline, but the scope of those conditions may be narrower than reported.

What evidence exists. The comparisons are in Figure 9 (scaling curves per difficulty bin at three R values) and the bar charts in Figure 1 (top-right and bottom-right panels). The larger model's accuracy appears as stars on the plots. The paper acknowledges the training methodology caveat in Section 7 but does not test how the results would change with a Chinchilla-optimal baseline or with any test-time compute allocated to the larger model.

Mitigation status. The authors are transparent about this limitation and explicitly defer the compute-optimal pretraining comparison to future work. The choice to fix data and scale parameters is reasonable as a first study β€” it mirrors what many practitioners actually do (the LLaMA paradigm is widely adopted) β€” but it means the findings represent a lower bound on pretraining's competitiveness. Readers should not interpret the result as "test-time compute beats optimally-trained larger models" but rather as "test-time compute can beat a parameter-scaled larger model with greedy decoding."

Hard Problems Remain Essentially Unsolved β€” Test-Time Compute Cannot Create Capability From Nothing

The assumption or constraint. The entire framework assumes the base model can produce correct solutions at some non-trivial rate. When the base model's pass@1 is near zero on a problem class, no amount of search or revision can help β€” there are no correct solutions in the proposal distribution to find (via search) or refine (via revisions). The paper acknowledges this:

"On the hardest questions (bin 5), no method makes meaningful progress β€” the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

The consequence. Across all methods β€” search, revisions, and their compute-optimal combinations β€” the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for all budgets and all R values. The pretraining-versus-inference comparison shows that on hard questions at R ≫ 1, test-time compute underperforms the 14Γ— larger model by βˆ’37.2% (revisions) and βˆ’52.9% (PRM search) β€” negative relative improvements, meaning the larger model is substantially better.

This is a fundamental capability boundary: test-time compute amplifies existing capability but does not create it. If the base model cannot generate a correct solution even once in thousands of attempts, no allocation of search, revision, or verifier optimization will produce one. This means the compute-optimal framework offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining (or fundamentally different base models) remains the only viable path.

What evidence exists. The flat bin-5 curves are visible in Figures 3, 7, and 9. The paper quantifies this in the FLOPs-matched takeaway (Section 7): on hard problems, test-time compute provides minimal gains and pretraining is almost always more effective.

Mitigation status. The authors are candid about this boundary condition and frame it as a key finding rather than a failure. The paper does not attempt to mitigate it β€” fundamentally, it cannot be mitigated within the test-time compute paradigm. The limitation is inherent to the approach: you cannot search for what is not in the proposal distribution. The paper's contribution is precisely characterizing where this boundary lies (the difficulty quintile where pass@1 drops to near zero) and showing that below the boundary, substantial gains are possible.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate β€” Training on Only Incorrect-to-Correct Trajectories Creates a Directional Bias

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target. This is a consequence of the training data construction (Section 6.1): for each training question, the authors identify which of 64 sampled responses are correct and which are incorrect, then construct multi-turn sequences of 0–4 incorrect answers followed by a correct answer. The model never sees examples where the current answer is already correct and should be preserved. The authors report that:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach."

The consequence. At test time, when the revision model produces a correct answer during a revision chain, it may incorrectly "revise" it into a wrong answer in the subsequent step because it has no training signal for "stop revising when correct." This creates a reliability ceiling for sequential revisions: you cannot simply take the last output in a revision chain as the final answer, because the last output may be worse than an intermediate one. The paper mitigates this with within-chain selection (majority voting or verifier-based selection across the entire chain, picking the best answer from any point), but this is an imperfect patch β€” it adds overhead and does not prevent the model from spending compute generating incorrect revisions of previously correct answers. It also means the effective compute budget is partially wasted on regression steps.

What evidence exists. The 38% number is reported in Section 6.1. Figure 6 (left) shows that pass@1 at each step in the revision chain gradually improves but does not monotonically increase β€” there is variance step to step, consistent with occasional reversions. The paper uses within-chain selection in all revision experiments, so the reversion rate is partially compensated for in the reported numbers, but the wasted computation is not quantified.

Mitigation status. The paper applies within-chain selection (majority voting or verifier-based) as a post-hoc mitigation, but this does not address the root cause. A more principled solution β€” such as training the model to recognize when no revision is needed by including correct-to-correct or correct-to-stop trajectories in the training data β€” is not explored. The paper does not suggest future work on this specific issue, though Section 8's call for combining PRM search with revisions could partially address it by using the PRM to detect when a revision chain has peaked and should be terminated early.

Revisions and Search Are Studied Independently β€” The Two Complementary Mechanisms Are Never Combined

The assumption or constraint. The paper studies two modification axes β€” PRM search (optimizing the verifier/selection mechanism) and iterative revisions (modifying the proposal distribution) β€” but analyzes them entirely independently. Section 2 frames them as complementary axes in a unified framework; Sections 5 and 6 present separate experiments for each; and Section 7 tests them separately in the FLOPs-matched comparison. The authors acknowledge this gap explicitly:

"we did not experiment with PRM tree-search techniques in combination with revisions." (Section 8)

The consequence. The paper's strongest results β€” 4Γ— efficiency gains for both search and revisions individually (Figures 4 and 8) β€” represent a lower bound on what a combined system could achieve. The two mechanisms have complementary difficulty-dependent strengths: revisions excel on easy problems (local refinement), PRM search excels on medium problems (global exploration). A system that combines them β€” for example, using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision chains to pursue and when to terminate them β€” could yield gains beyond either mechanism alone, particularly on medium-difficulty problems where both show partial effectiveness.

Additionally, because the mechanisms are never combined, the paper cannot assess whether their benefits are additive, sub-additive (due to overlapping improvements), or super-additive (due to synergy). The compute-optimal policy selects between search and revisions per difficulty bin, but a truly optimal policy might allocate a mixed budget β€” some search, some revisions β€” within a single problem. The current framework cannot evaluate this.

What evidence exists. No experiment combines search and revisions. Figures 4 and 8 show separate compute-optimal scaling curves for search and revisions; a combined curve is absent. The FLOPs-matched comparison in Figure 9 also separates them. Table 2 in Section 5.2 compares aggregate search algorithm performance; Table 4 in Section 5.3 compares revision model sequential-to-parallel ratios; these are independent analyses.

Mitigation status. The authors explicitly call for future work on this combination (Section 8). The absence of combined experiments is a scope limitation β€” the paper already covers substantial ground β€” but it is a significant gap because the complementary strengths are one of the paper's own key insights (Innovation 2 in Section 4). A combined experiment would directly test whether the unified framework's theoretical decomposition into proposal and verifier axes translates to practical gains when both axes are optimized simultaneously.

Single Benchmark, Single Model Family β€” All Results Are on MATH with PaLM 2-S*, and Generalization to Other Domains, Models, and Tasks Is Unverified

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The MATH benchmark consists exclusively of competition-level math problems requiring multi-step symbolic reasoning with clean ground-truth answers that can be checked with exact string matching.

The consequence. Several aspects of the findings could be model-specific or domain-specific:

  • PRM quality and over-optimization behavior depend on PaLM 2-S's output distribution.* A model with different calibration properties or error patterns (e.g., more confident wrong answers, different step-level coherence) might exhibit different difficulty-dependent scaling curves and different over-optimization thresholds for beam search.
  • The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The reported 38% correct-to-incorrect reversion rate may be specific to PaLM 2-S* or to the specific fine-tuning recipe.
  • The MATH benchmark tests symbolic mathematical reasoning. It is unclear whether the difficulty-dependent patterns β€” beam search hurting easy problems (due to PRM over-optimization), revisions helping easy problems but requiring parallel exploration on hard ones β€” generalize to other reasoning domains (code generation, logical reasoning, scientific QA, planning) or to tasks requiring factual knowledge recall rather than multi-step inference.
  • MATH has clean correctness signals β€” answers are either exactly right or wrong. This enables both the PRM training pipeline (Monte Carlo rollout correctness labels) and the difficulty estimation (pass@1 computation). Many important real-world tasks β€” open-ended generation, dialogue, summarization, creative writing β€” lack such binary correctness signals, making it unclear how to adapt the PRM training and difficulty estimation procedures to those domains.

What evidence exists. All figures, tables, and quantitative claims reference only MATH with PaLM 2-S* (and the 14Γ— larger PaLM variant for the FLOPs comparison). No cross-model or cross-domain experiments are reported. The paper does not test on GSM8K (a simpler math benchmark), nor on code generation (HumanEval, MBPP), nor on any non-math reasoning task.

Mitigation status. The authors acknowledge the single-benchmark limitation implicitly by situating their work as "the first systematic scaling analysis" (Section 1) and calling for future work on extension to other domains (Section 8). The choice of MATH is justified (Section 4): it has clear correctness signals, multi-step reasoning, and non-trivial but non-saturating base model performance, making it a good testbed for studying test-time compute scaling. But the generalizability of the findings remains entirely unverified. A practitioner considering deploying compute-optimal test-time strategies on a different model family or task domain has no guarantee that the difficulty-dependent patterns β€” or the specific strategy recommendations (best-of-N on easy, beam search on medium, revisions on easy, balanced ratio on hard) β€” will transfer. The paper's insights should be understood as hypotheses about test-time compute scaling that have been validated on one model and one task, pending broader replication.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper triggers a necessary recalibration of what the field treats as ground truth for GUI grounding progress, revealing that the dominant click-centric benchmarks have systematically overstated model capabilities while ignoring the interaction types β€” dragging, drawing, text selection, table manipulation, image boundary tracing β€” that actually cause real computer-use agents to fail. The magnitude of the shift is best understood as a diagnostic correction, not a paradigm overturn: the paper does not claim that widget-clicking is irrelevant, but rather demonstrates that optimizing for widget-clicking alone is a form of Goodhart's Law in action β€” the metric has become the target, and the resulting models underperform precisely where deployed systems need them most.

The evidence for this recalibration is concentrated in three findings. First, the OSWorld grounding-only experiment (Table 3) directly falsifies the assumption that high ScreenSpot-Pro scores indicate strong general grounding: models with ScreenSpot-Pro scores of 71.1 and 65.8 achieve OSWorld success rates of 37.7 and 38.2, while Phi-Ground-Any-4B with a ScreenSpot-Pro score of 26.3 achieves 42.4. The ranking inverts. A practitioner choosing a grounding model based on the established benchmark would select a model that performs worse in the deployment setting that matters. Second, the fine-tuning experiment (Table 2, "+ APP data finetuned" row) provides causal evidence: adding click-only application-specific training data increases ScreenSpot-Pro by 15.2 points and UI-Vision by 13.9 points but decreases CUActSpot by 7.9 points, with modality-specific drops of up to 14.3 points on Canvas and Natural Image. The benchmarks are not just measuring something different β€” they reward training choices that actively harm complex interaction capability. Third, the ScreenSpot-Pro vs. UI-Vision gap (Ξ” column in Table 2) has grown from ~10 points in older models to 25–34 points in recent benchmark-optimized models, while GPT-5.4 β€” not optimized for any specific grounding benchmark β€” shows a gap of only 6.6 points. This temporal pattern is consistent with benchmark-driven overfitting rather than genuine capability improvement.

These findings reconcile a contradiction in the literature. Prior work on GUI grounding reported steadily climbing benchmark scores, creating an impression of rapid progress. Simultaneously, end-to-end agent benchmarks like OSWorld showed stubbornly low success rates even when using these "high-performing" grounders. The contradiction dissolves when you recognize that ScreenSpot-Pro, SeeClick, and UI-Vision measure a narrow capability β€” single-point click on software-specific widgets β€” that is necessary but far from sufficient for agentic success. The models that dominate these benchmarks have learned to recognize application icons and widget layouts; they have not learned the broader spatial reasoning primitives needed to drag a cell corner, select a text span, or trace an object boundary. The field conflated improvement on the proxy with improvement on the target, and this paper provides the measurement tools to separate them.

The practical implication is a reframing of the GUI grounding research agenda. Before this work, the dominant strategy for improving grounding was to collect more widget-click data from more applications β€” crawl more webpages, render more software screenshots, extract more element bounding boxes. After this work, it is difficult to justify that strategy as the primary path forward. The diminishing returns curves in Figure 5 (subfigure 1-1) and the plateauing single-modality scaling suggest that additional widget-click data yields rapidly declining benefits. The waterfall ablations (Figures 1-2 through 2-3) demonstrate that adding even modest amounts of diverse interaction data produces gains that widget-click scaling alone cannot match. The research direction that becomes more attractive is broad-coverage interaction data synthesis β€” building renderers and task generators for diverse modalities and action types β€” while the direction that becomes less attractive is incremental scaling of widget-click datasets.

The paper also identifies verifier over-optimization as a primary bottleneck for test-time compute scaling (Section 5.3), redirecting research attention from more sophisticated search algorithms toward more robust verifiers. But critically, the paper does not propose a single new verifier or search algorithm β€” it provides a framework for diagnosing when and why complex grounding fails, and a scalable method for generating training data that covers the long tail of interaction types. The implications are methodological: the field now has tools to measure and improve grounding on the interactions that matter, rather than optimizing for a proxy that has been shown to mislead. This is less a paradigm shift than a course correction, but it is a correction that the deployment evidence β€” GPT-5.4 failure distributions, OSWorld results, the fine-tuning experiment β€” suggests is overdue.

Follow-Up Research This Work Enables

Constant-volume diversity ablation to isolate variety scaling from total data volume. The sequential waterfall experiments in Figure 5 add both diversity and total training volume simultaneously (2M GUI β†’ 3M β†’ 4M β†’ 5M β†’ 6M β†’ 6.5M). A cleaner test of the variety scaling hypothesis would hold total volume constant: compare (a) a model trained on 5M GUI-only samples against (b) a model trained on 1M each of GUI, Text, Table, Canvas, and Natural Image (also 5M total). If model (b) substantially outperforms model (a) on CUActSpot, the diversity advantage is isolated from the volume confound. This experiment would directly validate the paper's central theoretical claim and could be run with the existing synthesis pipeline by simply subsampling per-modality data differently. A strong follow-up would also test intermediate diversity levels β€” 2 modalities Γ— 2.5M each, 3 modalities Γ— 1.67M each β€” to map the shape of the diversity-volume tradeoff curve and identify the point where additional modality diversity yields diminishing returns.

Interference analysis of modality combinations and training order. The negative transfer from Table data onto GUI accuracy (βˆ’10.5 in Figure 1-2, from 34.2 to 23.7) followed by partial recovery with subsequent modalities is a finding the paper reports but does not explain. A systematic study varying the order of modality introduction β€” does adding Text before Table prevent the GUI regression? Does starting with Canvas make the model more robust to subsequent modality shifts? β€” would reveal whether this interference is a fundamental property of multi-modal grounding training or an artifact of the specific training recipe. The experiment would train the same base model with different modality ordering permutations while tracking per-modality validation accuracy at each checkpoint. If certain orders consistently prevent interference, this would inform practical training recipes. If interference occurs regardless of order, it would suggest a need for architectural solutions (e.g., modality-specific adapter layers) or training interventions (e.g., replay buffers, elastic weight consolidation). This investigation connects the variety scaling findings to the continual learning and catastrophic forgetting literature.

Human baseline measurement on CUActSpot. The paper validates CUActSpot by having three individuals attempt all tasks and removing those that could not be completed by humans, but no human accuracy baseline is reported. Without this, there is no calibration for interpreting model scores β€” is GPT-5.4 at 63.6% overall close to the human ceiling or still substantially below it? A human study where 3–5 participants complete all 206 CUActSpot samples (similar to the validation procedure already used during benchmark construction) would provide this calibration. If human accuracy is near 95%, then even GPT-5.4 has substantial room for improvement, and the gap quantifies how far current models are from human-level grounding. If human accuracy is notably lower (e.g., 80%) due to residual ambiguity in task descriptions, the benchmark's difficulty is partly driven by instruction clarity rather than purely grounding capability, which would be an important caveat for interpreting model comparisons.

Combined search-and-revision system using PRM-guided revision chains. The paper studies PRM tree-search and iterative revisions as separate mechanisms but explicitly notes they were never combined (Section 8), and the prior analysis identifies them as having complementary difficulty-dependent strengths β€” revisions excel on easy problems, search on medium problems. A natural integration would use the revision model as the proposal distribution within beam search: at each step of the search tree, the model conditions on previous rejected branches as context to produce higher-quality candidate steps. Alternatively, the PRM could guide revision chains dynamically β€” score each revision step and decide whether to continue refining, restart from scratch, or accept the current answer. This experiment would test whether the benefits of the two mechanisms are additive, sub-additive, or super-additive, and would be directly comparable to the individual results in Figures 4, 8, and Table 2. The paper's own framework (Section 2, decomposing test-time compute into proposal and verifier modifications) provides the intellectual scaffolding; this experiment would test whether optimizing both axes simultaneously delivers the gains the framework predicts.

Scaling the synthesis pipeline to 50M samples with full diversity and measuring the composite scaling curve. The paper shows plateauing single-modality curves at ~2M samples (Figure 5, subfigure 1-1) but trains the final model on only 6.5M samples total across all modalities (Table 7), with the overall CUActSpot score reaching 37.1% and potentially still improving (the waterfall shows continued gains at each step). What happens if the full 50M synthesized samples are used β€” ~6.8M GUI plus the full generated volumes of Text, Table, Canvas, and Image? Does the multi-modality composite curve sustain logarithmic scaling, or does it also plateau? This experiment would cost approximately 5Γ— more compute than the current training run (the paper reports ~30 hours on 80 H100 GPUs for the 100B token budget; scaling to the full 50M samples would be roughly 150 hours on equivalent hardware). The result would determine whether the synthesis pipeline is a path to continued capability improvement or whether diminishing returns set in even with diverse data β€” a finding with direct implications for whether resource-constrained teams should invest in building additional modality-specific renderers.

Domain transfer study of compute-optimal strategy recommendations across model families and task types. The paper's findings about which strategies work on which difficulty levels β€” beam search on medium problems, revisions on easy problems, best-of-N on hard (to avoid over-optimization) β€” are derived entirely from MATH with PaLM 2-S*. A replication on code generation (HumanEval, MBPP) using a different base model family (e.g., LLaMA, Qwen) would test whether these strategy recommendations are task-specific, model-specific, or general. The experiment would replicate the difficulty binning (based on pass@1 from 2048 samples), strategy sweep (best-of-N, beam search with M = 4 and M = √N, sequential vs. parallel revisions), and compute-optimal policy selection, then compare the resulting strategy-per-difficulty-bin recommendations against those from MATH. If the recommendations differ substantially β€” e.g., beam search never outperforms best-of-N on code regardless of difficulty β€” it would establish that compute-optimal strategies are domain-dependent and must be calibrated per-task. If the recommendations are consistent, it would strengthen the claim that difficulty-conditioned allocation is a general principle. This experiment is directly enabled by the paper's methodology (difficulty estimation protocol, strategy sweep framework, cross-validation for policy selection) and would be a stress-test of the findings' generality.

Practical Applications and Downstream Use Cases

Training data synthesis for proprietary CUA systems. The renderer-first synthesis pipeline provides a blueprint for generating complex-interaction training data without human annotation. A company building an in-house computer-use agent (e.g., for automating enterprise software workflows) could adapt the pipeline by implementing renderers for their specific target applications β€” generating synthetic screenshots of their CRM, ERP, or design software with ground-truth coordinate annotations β€” and using the same o3-prompting approach to generate instruction-action pairs. The key insight from the paper for this use case is that the LLM can perform geometric reasoning over coordinate metadata to produce non-trivial tasks (computing drag offsets, aligning elements, generating multi-step drawing sequences), meaning the task space is limited by the renderer's geometric annotation richness, not by the complexity of the task templates. The 4Γ— efficiency finding translates here: generating 1M diverse interaction examples across multiple modalities (GUI, table, canvas) requires only the rendering engineering effort, not 1M human annotations.

Evaluation of off-the-shelf grounding models for complex productivity workflows. An organization evaluating which grounding model to integrate into their CUA pipeline β€” currently facing the ScreenSpot-Pro numbers for GUI-Owl-1.5-8B-Instruct (71.1), MAI-UI-8B (65.8), and Phi-Ground-Any-4B (26.3) β€” would, based on established benchmarks, select GUI-Owl or MAI-UI. The CUActSpot results in Table 2 tell a different story: GUI-Owl-1.5-8B-Instruct achieves 15.4 overall and 0.0 on Text, while Phi-Ground-Any-4B achieves 44.4 overall and 68.8 on Text. If the deployment involves document editing, spreadsheet manipulation, or image annotation β€” the work setting that the GPT-5.4 failure study identifies as "the scenario users care about most" β€” the ScreenSpot-Pro leaderboard would direct the practitioner to a model that performs dramatically worse on the interactions that dominate the workflow. CUActSpot provides a direct complement to existing benchmarks for this decision: evaluate candidate models on both a widget-clicking benchmark and CUActSpot, and select based on the interaction profile of the target deployment. The OSWorld correlation (Table 3) provides additional confidence that CUActSpot performance better predicts end-to-end agentic success.

Data curation strategy for fine-tuning VLMs into general-purpose CUAs. The waterfall ablations in Figure 5 provide a concrete curriculum for teams fine-tuning vision-language models into computer-use agents. The finding that adding 1M Text samples improves CUActSpot-Text by +25.0 points (from 6.3 to 31.3, subfigure 1-3) β€” the single largest modality-specific gain β€” but only improves CUActSpot-GUI by +2.6 (subfigure 1-2) implies that modality-specific capabilities are largely modular and must be explicitly covered. A practical data curation strategy: start with a base of GUI widget data for general button/icon clicking, then add targeted volumes of text-selection, table-manipulation, canvas-editing, and image-region data, monitoring per-modality validation performance to determine when each modality has saturated. The negative transfer finding β€” adding Table data caused a temporary βˆ’10.5 regression on GUI β€” suggests that late-stage fine-tuning should maintain a small replay fraction of earlier modalities' data to prevent catastrophic forgetting. The 6-modality data composition in Table 7 (with per-modality weights of 0.34, 0.25, 0.10, 0.10, 0.15, 0.06) provides a starting ratio that could be adjusted based on the target deployment's expected interaction distribution.