GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

ArXiv: 2506.03143

🎯 Pitch

A small 7B VLM outperforms 72B models on GUI grounding by predicting click locations through attention maps rather than generating coordinates, yet only succeeds when the vision backbone’s patch resolution actually captures the target—accidentally revealing a hard limit of patch-level attention for tiniest icons.

---

1. Executive Summary

This paper proposes GUI-Actor, a coordinate-free visual grounding method for VLM-powered GUI agents that replaces text-based coordinate generation with an attention-based action head — a dedicated <ACTOR> token learns to attend directly to relevant visual patch tokens, identifying actionable screen regions in a single forward pass — and introduces a grounding verifier (a lightweight VLM module that evaluates candidate regions by marking them on the screenshot and predicting whether the marked location fulfills the instruction) to select the most plausible action region among multiple candidates. Extensive experiments on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro demonstrate that GUI-Actor-7B achieves scores of 40.7 and 44.6 with Qwen2-VL and Qwen2.5-VL backbones respectively, outperforming UI-TARS-72B (38.1) on ScreenSpot-Pro with significantly fewer parameters and training data. The approach exhibits strong out-of-distribution generalization to unseen screen resolutions and layouts, establishing that explicit spatial-semantic alignment through patch-level attention enables robust grounding only when the underlying vision backbone's patch resolution is sufficient to capture the target element.

2. Context and Motivation

The Core Problem: The Coordinate Generation Bottleneck in Visual GUI Agents

GUIs are the dominant interface through which humans interact with software. Building autonomous agents that can use these interfaces as humans do — by looking at the screen and acting on it — requires connecting language instructions ("click the submit button") to specific screen regions where pixels live. This task, known as visual grounding for GUI agents, is the bridge between high-level reasoning and low-level action execution. An agent that can understand that a screenshot contains a calendar widget but cannot reliably locate the specific "weekly view" button within it is useless in practice.

The paper identifies that the prevailing approach to this grounding problem — text-based coordinate generation — suffers from three deeply intertwined limitations that collectively explain why progress on challenging benchmarks like ScreenSpot-Pro has been slow. These are not surface-level engineering issues but fundamental design flaws that arise from forcing what is inherently a spatial, dense prediction problem through the narrow bottleneck of autoregressive language modeling.

Why This Problem Matters: Beyond Benchmark Numbers

The importance of solving visual grounding robustly extends beyond raising accuracy scores on static benchmarks. It determines whether autonomous GUI agents can function reliably in production environments.

Real-world deployment demands robustness to distribution shift. Professional software interfaces (CAD tools, video editing suites, financial dashboards) look nothing like the web pages and mobile apps that dominate most training datasets. They have higher resolutions, denser layouts, unfamiliar widget styles, and multi-window configurations. An agent that works well on standard web pages but collapses on an unfamiliar IDE screenshot is not ready for deployment. ScreenSpot-Pro was explicitly designed to measure this kind of out-of-distribution generalization, and the paper's focus on it reflects a pragmatic concern: deployment viability requires models that ground correctly even when the interface looks different from anything in the training set.

Cost and scale constraints are real. The paper's Figure 1 (left) makes a pointed argument: UI-TARS-72B, a 72-billion-parameter model trained on massive proprietary datasets, achieves strong results on ScreenSpot-Pro, but this approach is inaccessible to practitioners without the resources to train and deploy such large models. The gap between what a well-resourced industrial lab can achieve and what the broader research community can replicate is significant. A method that achieves comparable or better performance with a 7B model trained on public data — as GUI-Actor claims — directly addresses this democratization problem.

Interactive latency requires efficient inference. GUI agents operate in a closed loop: see screen, decide action, execute action, see new screen. Each grounding step adds latency to this loop. The paper's design — producing multiple candidate regions in a single forward pass, rather than requiring multiple sampling runs or search procedures — is in part motivated by practical concerns about interactive responsiveness. A grounding method that requires 21 separate inference passes (as the authors note is necessary for Aguvis with verifier-based selection, Figure 7) incurs a 20× latency penalty that may be unacceptable in interactive settings.

Prior Approaches and Where They Fall Short

Understanding why coordinate generation became dominant and why it fails is essential to appreciating GUI-Actor's design decisions.

The Coordinate Generation Paradigm

Most existing work formulates GUI grounding as follows: given a screenshot and an instruction, the VLM generates natural language text that happens to include coordinate tokens — for example, producing a string like pyautogui.click(x=0.123, y=0.234). The model's existing language modeling head (an LM head that predicts the next token from the vocabulary) is repurposed to predict numeric coordinate tokens. This approach is appealing because it requires no architectural modification — you can take any VLM that supports autoregressive text generation and fine-tune it on grounding data by simply framing the target as a text string containing coordinates.

The paper cites several prominent examples of this paradigm: Aguvis (Xu et al., 2024) uses pyautogui-style code generation with embedded coordinates; SeeClick (Cheng et al., 2024) generates point coordinates; UGround (Gou et al., 2024) synthesizes diverse GUI grounding examples in text format; and UI-TARS (Qin et al., 2025) generates coordinate-based actions at scale. OS-Atlas (Wu et al., 2024) provides a multi-platform dataset and unified action model built on the same principle.

Limitation 1: Weak Spatial-Semantic Alignment

This is the most fundamental limitation. When a VLM generates x=0.123 as a text token, the following computation happens: the model's hidden state at that position is projected through a linear layer that maps to the vocabulary distribution, and whichever token in the vocabulary corresponds to the subword unit for "0" or "12" or "3" receives high probability. There is no explicit connection between the visual features that represent a button at position (0.123, 0.234) and the coordinate tokens that describe that position. The model must learn this mapping implicitly through the cross-entropy loss during fine-tuning.

This is a spatial-semantic alignment problem. In dense prediction tasks like object detection or segmentation in computer vision, detectors are typically designed so that features at a spatial location predict attributes of whatever object exists at that location — a direct, architecturally-enforced correspondence between "where" in the feature map and "what" in the output. Coordinate generation forgoes this entirely. The spatial location is encoded as discrete tokens with no privileged relationship to the visual feature map. The paper characterizes this as operating "without any explicit spatial inductive bias" and being "inefficient, data-intensive, and prone to errors."

The consequence: the model must expend capacity and training data to learn a mapping that could have been architecturally provided. This is why Figure 3 shows baseline models requiring 80-90% of the training data to plateau, while GUI-Actor reaches final accuracy after ~60% — the baseline is learning a spatial mapping from scratch, while GUI-Actor's action head has it built in.

Limitation 2: Ambiguous Supervision Signals

Consider the instruction "click the submit button." The ground-truth annotation might specify a single pixel as the target click point. But any point within the button's bounding box would be functionally correct — the button responds to clicks anywhere on its surface. A coordinate-based model trained with point supervision treats any deviation from that exact annotated pixel as an error, even if the predicted point would still successfully click the button.

This creates a perverse training dynamic: the model receives a loss penalty for producing a reasonable prediction that happens to be a different valid point within the same element. Over many training examples, this injects noise into the learning signal. The model struggles to distinguish between predictions that are genuinely wrong (clicking on a different element) and predictions that are functionally correct but spatially imprecise (clicking on a different part of the same button). The paper notes that this "ambiguity in GUI interactions, where multiple points within a UI element may all be valid" is intrinsic to the task and fundamentally mismatched with single-point coordinate supervision.

Some prior work attempts to mitigate this by predicting bounding boxes instead of single points (e.g., Aguvis with bounding box supervision), but the paper's ablation study (Table 6) shows that bounding box supervision in a coordinate generation framework does not meaningfully outperform point supervision. The reason: without architectural mechanisms to connect the predicted coordinates to visual features, the additional spatial information in bounding boxes cannot be effectively utilized. The model still learns a text-to-text mapping, and the coordinates remain detached from the visual representation.

Limitation 3: Granularity Mismatch Between Vision and Action Spaces

Modern VLMs use Vision Transformers (ViTs) that process images as a grid of discrete patches — typically 14×14 or 28×28 pixels each. The model's visual understanding is fundamentally patch-aligned: features are extracted and processed at this native resolution. Coordinate generation, however, operates at a much finer granularity. Coordinates are continuous values in the range [0, 1] normalized to the screen dimensions, and the model must predict them to sufficient precision to land within the target element's bounding box.

This forces the model to infer sub-patch precision from patch-level features. A ViT patch covering 28×28 pixels might contain parts of several UI elements, and the model must somehow localize the click point to pixel-level accuracy from this coarsely quantized representation. The paper describes this as forcing the model "to infer dense, pixel-level actions from coarse visual tokens." This is not merely inefficient — it undermines generalization because a model that has memorized specific coordinate-patch mappings may fail when the screen resolution changes, since the same physical element occupies different pixel coordinates and different patch positions at different resolutions.

Some prior work attempts to address resolution issues by using dynamic resolution strategies (as Qwen2-VL does), but this does not resolve the fundamental mismatch: the visual features remain patch-level while the output space is continuous. The model must still bridge this gap through learned, implicit mappings.

How the Paper Positions Itself

The paper draws a specific analogy to motivate its approach: "humans do not calculate precise screen coordinates before acting — they perceive the target element and interact with it directly." This is a reformulation of the problem rather than an incremental improvement. Instead of asking "how do we help the model generate better coordinates?", the paper asks "can the model simply attend to the element it should interact with?"

This reformulation naturally addresses all three limitations simultaneously:

• Spatial-semantic alignment: By having the <ACTOR> token's hidden state compute attention scores directly over visual patch tokens, the spatial alignment is architecturally enforced — the attention distribution is the spatial output, and it operates in the native coordinate frame of the vision backbone's patch grid.

• Ambiguous supervision: By supervising all patches that overlap with the ground-truth bounding box as positive targets (multi-patch supervision), the model learns to attend to the entire spatial extent of the target element, not a single arbitrary point. A prediction anywhere within the element is rewarded.

• Granularity mismatch: By operating at the vision backbone's native patch resolution, the method accepts the granularity of the available visual features and grounds actions at that same granularity. There is no attempt to predict sub-patch coordinates (though the patch size limitation is acknowledged in Appendix A).

The paper positions itself as a complementary approach to the verifier-based validation that has recently gained traction in the LLM reasoning literature. Drawing on the principle that "verification is often easier than generation" (citing Cobbe et al., 2021), the proposed grounding verifier serves as a lightweight decision refinement layer that checks proposed regions against the instruction, analogous to how process reward models verify intermediate reasoning steps. But critically, GUI-Actor's attention mechanism produces a rich set of diverse candidate regions from a single forward pass, providing the verifier with meaningful options to choose among — a marked contrast to coordinate generation methods where resampling tends to produce nearly identical outputs (as shown in Figure 4a and its surrounding analysis).

Available Data Context

The paper compiles its training data from several publicly available GUI datasets totaling approximately 1 million screenshots, summarized in Table 7. These span desktop, mobile, and web domains, with OS-Atlas providing the bounding box annotations that enable multi-patch supervision. Importantly, the paper explicitly notes that it excludes samples from Wave-UI that overlap with downstream test sets, maintaining a clean separation between training and evaluation.

The choice of Qwen2-VL as the backbone VLM is strategic: it is a widely-used open-source model that enables fair comparison with other open-source approaches, and its 7B parameter scale makes the approach reproducible. The paper also reports results with Qwen2.5-VL (Table 4), demonstrating that the method's benefits transfer across backbone generations.

The Broader Research Landscape

While the paper's primary contrast is with coordinate generation methods, it also implicitly positions itself relative to two other strands of work:

Training-free attention-based grounding (Xu et al., 2025): This approach, cited in the related work, proposes using a VLM's internal attention maps for GUI grounding without any fine-tuning. While conceptually related to GUI-Actor's attention-based design, training-free methods are limited by whatever attention patterns emerge from the pretrained model — which were not explicitly optimized for grounding. GUI-Actor's insight is that adding a small, trainable action head that supervises the attention mechanism produces significantly better grounding than relying on emergent attention artifacts.

Structured metadata approaches: Early GUI agents relied on accessibility trees, DOM structures, or view hierarchies to localize elements. The paper acknowledges this tradition but aligns itself with the vision-centric paradigm that has become dominant because metadata is "often noisy, inconsistent, or unavailable across platforms." The grounding verifier's design — requiring only a screenshot and a marked location — reflects this commitment to a pure-vision interface.

3. Technical Approach

3.1 Reader Orientation

GUI-Actor is a VLM-based system that, given a screenshot of a GUI and a natural language instruction describing what to click, identifies the screen region to interact with — without ever generating numeric coordinates as text tokens. The system solves the GUI visual grounding problem by reformulating it from "generate the coordinates of the target element" to "attend directly to the visual patches that constitute the target element," producing an attention heatmap over the screenshot's patch grid from which actionable regions are extracted.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a feedforward pipeline:

1. Backbone VLM (e.g., Qwen2-VL) — a pretrained vision-language model that processes the screenshot and instruction together, producing hidden-state representations for every patch token and every text token in the input sequence. Its role is to fuse visual and linguistic information into a shared representational space.

2. <ACTOR> Token Mechanism — a special token injected into the VLM's output vocabulary whose final-layer hidden state serves as a contextual anchor encoding the entire grounding decision. Rather than generating coordinate tokens, the VLM generates <ACTOR_START><ACTOR><ACTOR_END> at the position where coordinates would normally appear. The hidden state of the middle <ACTOR> token is extracted and routed to the action head.

3. Attention-Based Action Head — a lightweight module (~20M parameters for 2B backbone, ~100M for 7B) that takes the <ACTOR> token's hidden state and the vision encoder's patch features as input, and produces an attention distribution over all visual patches. This attention map is the spatial output — high-attention patches correspond to the target interaction region.

4. Grounding Verifier — a separate, lightweight VLM (fine-tuned from UI-TARS-2B-SFT) that scores candidate regions proposed by the action head by marking them on the screenshot with a visual indicator and predicting whether the marked region correctly fulfills the instruction. It serves as a post-hoc decision refinement layer.

Information flows as follows: screenshot + instruction → backbone VLM → <ACTOR> hidden state extracted → action head computes attention over patch features → attention map thresholded to produce candidate regions → (optional) verifier scores and selects the most plausible candidate → final (x, y) click point returned.

3.3 Roadmap for the Deep Dive

• First, the <ACTOR> token mechanism — how it replaces coordinate generation, what hidden state it produces, and why this single token can serve as a contextual anchor for the entire grounding decision.
• Second, the attention-based action head — the self-attention over patch features, the MLP projections, the attention computation between the <ACTOR> token and patches, and the resulting attention distribution.
• Third, the spatial-aware multi-patch supervision strategy — how ground-truth bounding boxes are converted to patch-level binary masks, how the target distribution is normalized, and how the KL-divergence loss provides dense spatially-structured supervision.
• Fourth, the grounding verifier — its training data construction (positive and negative examples from OS-Atlas), its inference procedure (scoring candidates by marking them on screenshots), and the patch clustering refinement that handles elements spanning patch boundaries.
• Fifth, the training recipe — the two-phase procedure (action-head-only warm-up followed by full fine-tuning), the combined loss function, and the inference-time candidate selection pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that GUI visual grounding can be performed without generating coordinate tokens by instead training the model to attend directly to target visual patches, where the attention distribution itself serves as the spatial output.

---

The <ACTOR> Token as a Contextual Anchor

The paper's most fundamental architectural decision is to replace coordinate tokens with a dedicated attention token. In a standard coordinate-generation model, the VLM produces a sequence like:

pyautogui.click(x=0.123, y=0.234)

where the tokenized x-coordinate (0, ., 1, 2, 3) and y-coordinate occupy specific positions in the output sequence. The model's language modeling head maps each of these positions to vocabulary probabilities, and the model is trained to predict the correct coordinate tokens via standard next-token prediction loss.

GUI-Actor replaces this coordinate span with exactly three tokens:

<ACTOR_START> <ACTOR> <ACTOR_END>

These are new tokens added to the VLM's vocabulary. The surrounding context — e.g., the function call pyautogui.click(...) — remains unchanged, preserving the model's ability to express actions in a natural language-compatible format. Formally, if the original output is a sequence {x_1, ..., x_N} where coordinate tokens occupy positions i through i+m (x-coordinate) and j through j+n (y-coordinate) with a separator between them, GUI-Actor produces:

{x_1:i-1, <ACTOR_START>, <ACTOR>, <ACTOR_END>, x_i+3:N}

where the three special tokens occupy exactly the slot where coordinates would have been. The critical design element is what happens with the <ACTOR> token's hidden state at the final transformer layer. Let h_<ACTOR> denote this hidden state — a vector in ℝ^d where d is the model's hidden dimension. This vector is the contextual anchor that encodes everything the model knows about the grounding target based on the fused visual and textual input. It is the single point through which all grounding information flows from the language side to the action head.

Why a dedicated token rather than a separate module? The <ACTOR> mechanism is not an auxiliary output; it is integrated into the VLM's autoregressive generation. This means the token's hidden state benefits from the full transformer stack — it attends to all input tokens (patches + text), all preceding output tokens, and all other positions through the self-attention layers. The rich contextualization that makes transformers powerful for language tasks is directly inherited by the grounding token. An alternative design — extracting the final hidden state of a special token appended to the input — would not benefit from the autoregressive context of the generated prefix (e.g., the action type pyautogui.click), which may disambiguate whether the grounding target is a clickable button versus a drag destination.

What the <ACTOR> token hidden state contains. The paper does not provide an explicit analysis of what information is encoded in h_<ACTOR>, but the mechanism implies it must encapsulate: (1) the semantic identity of the target element (parsed from the instruction), (2) the spatial constraints implied by the instruction (e.g., "the button next to the search bar"), and (3) a pointer-like signal that can be aligned with visual patch features through the action head's attention mechanism. The fact that a single vector can serve this role is a strong claim about the VLM's capacity to compress grounding-relevant information into a fixed-dimensional representation — a claim the paper's empirical results support.

---

Attention-Based Action Head

The action head is the trainable module that transforms the <ACTOR> token's hidden state into a spatial attention distribution over the screenshot's patch grid. It operates independently of the VLM's language modeling head (which still handles text generation) and produces the grounding output through a dedicated pathway that enforces spatial alignment architecturally.

Step 1: Self-attention over patch features. The input to this stage is the set of visual patch features {v_1, ..., v_M} extracted by the VLM's vision encoder, where each v_i ∈ ℝ^d corresponds to one image patch and M is the total number of patches (equal to the grid width W times grid height H). These features have already been through the vision encoder's internal processing but have not yet been contextualized with each other.

The action head applies a self-attention layer over these patch features:

$ṽ_1, ..., ṽ_M = \text{SelfAttn}(v_1, ..., v_M)$

where ṽ_i ∈ ℝ^d is the contextualized feature for patch i after self-attention.

What this computes: Each patch's feature vector is updated by attending to every other patch's feature vector, producing a representation that incorporates information from spatially distant but semantically related patches. For example, patches covering the top-left and top-right corners of a button can exchange information, allowing the model to represent the button as a coherent visual entity rather than as disconnected patch fragments.

Why this step exists: Without self-attention, each patch feature would only carry information about its local receptive field — the 28×28 pixel region it directly encodes. GUI elements often span multiple patches (a typical button might cover 3×5 patches or more), and treating each patch independently would prevent the action head from learning element-level representations. The self-attention layer provides lightweight spatial context aggregation without requiring access to the VLM's internal cross-attention maps (which the paper explicitly avoids — the action head operates on the output patch features, not intermediate attention weights).

Step 2: Projection into shared embedding space. Both the contextualized patch features ṽ_i and the <ACTOR> token hidden state h_<ACTOR> live in the same ℝ^d space by construction (the VLM's hidden dimension), but they come from different parts of the model and encode different kinds of information. The action head applies separate MLP projections to map them into a shared grounding space:

$z = \text{MLP}_T(h_{<ACTOR>})$ $z_i = \text{MLP}_V(ṽ_i)$

where z ∈ ℝ^d is the projected query vector (from the <ACTOR> token), z_i ∈ ℝ^d is the projected key vector for patch i, and MLP_T and MLP_V are independently parameterized multi-layer perceptrons.

What each MLP does: MLP_T maps the language-grounded representation into a "what to look for" query that encodes the target element's visual and semantic identity. MLP_V maps each contextualized patch feature into a "what is here" key that encodes the local visual content in a format comparable to the query. The use of separate MLPs rather than shared weights is important: the <ACTOR> token's information is fundamentally different from patch information — it carries linguistic semantics and action type context, while patches carry visual appearance — and forcing them through the same projection would entangle these modalities in a way that likely hurts alignment.

Step 3: Scaled dot-product attention. The attention score between the <ACTOR> token and each patch is computed using the standard scaled dot-product formulation:

$\alpha_i = \frac{z^T z_i}{\sqrt{d}}$

$a_i = \frac{\exp(\alpha_i)}{\sum_{j=1}^M \exp(\alpha_j)}, \quad \text{for } i = 1, ..., M$

where α_i is the unnormalized attention score for patch i, a_i is the normalized attention weight, and M is the total number of visual patches.

What this computes: The dot product z^T z_i measures the cosine-similarity-like alignment between the query (what the <ACTOR> token is looking for) and the key (what each patch contains). The scaling factor 1/√d prevents dot products from growing too large in high dimensions, which would push the softmax into a near-one-hot regime where gradients vanish. The softmax normalizes these scores into a probability distribution over patches — a_i can be interpreted as the model's confidence that patch i belongs to the target element.

Why scaled dot-product attention rather than learned attention: Learned attention (e.g., z^T W z_i with a learned weight matrix W) would add parameters but likely provide little benefit over the simpler formulation. The separate MLP projections MLP_T and MLP_V already provide learnable transformations of the input vectors, and adding another linear layer between them would be redundant — z^T W z_i = (W^T z)^T z_i, which is equivalent to modifying the query projection. The simplicity of dot-product attention also makes the attention map directly interpretable: a_i is high when the mapped query and mapped patch feature point in similar directions.

Crucially, what the action head does NOT produce: It does not produce coordinates. The attention distribution {a_1, ..., a_M} is the final spatial output. To convert this to a click point at inference time, the system takes the center of the highest-attention patch (or, when using the verifier, evaluates multiple top-K patches). There is no learned coordinate regression step — the spatial resolution is determined entirely by the vision encoder's patch grid (typically 28×28 pixels per patch for Qwen2-VL).

Parameter count context: The paper notes that the action head comprises approximately 20 million parameters for a 2B backbone and approximately 100 million for a 7B backbone. The self-attention module (which processes patch features of dimensionality d) and the two MLP projections account for most of these parameters. This is small relative to the backbone VLM (2B-7B parameters), meaning the action head can be trained efficiently and added to existing models with minimal overhead.

---

Spatial-Aware Multi-Patch Supervision

The training signal for the action head comes from bounding-box-level annotations converted to patch-level binary masks. This is the mechanism that enables the "coordinate-free" design — the model is never shown explicit coordinate targets; it learns to attend to regions defined by spatial extent rather than precise points.

Step 1: Converting bounding boxes to patch masks. Given a normalized bounding box b = [left, top, right, bottom] where each value is in [0, 1] (fraction of image width or height), the paper scales these coordinates to the patch grid resolution:

$(\lfloor \text{left} \cdot W \rfloor, \lfloor \text{top} \cdot H \rfloor, \lceil \text{right} \cdot W \rceil, \lceil \text{bottom} \cdot H \rceil)$

where W is the number of patches in the horizontal dimension and H is the number in the vertical dimension. The floor and ceiling operations ensure that all patches partially or fully covered by the bounding box are included — the resulting grid-aligned rectangle is the bounding box expanded to the nearest patch boundaries.

What this computes: A binary vector y ∈ {0, 1}^M where y_i = 1 if patch i falls within the grid-aligned bounding box region and y_i = 0 otherwise. For example, a button occupying 3 patches horizontally by 2 patches vertically would produce a mask with 6 positive patches and M - 6 negative patches.

Why floor/ceiling rather than rounding: Rounding to the nearest patch grid coordinate would exclude patches that are only partially covered by the bounding box but still contain a significant portion of the target element. For small elements that span patch boundaries, this could exclude the very patches that carry the strongest visual signal of the element's presence. The floor/ceiling approach is conservative — it labels as positive any patch with any overlap — ensuring the model receives supervision signal on all relevant spatial locations.

Step 2: Normalizing to a target distribution. The binary mask cannot be used directly as a target for the softmax attention distribution because the number of positive patches varies across examples (a large element might have 15 positive patches while a small one has only 2). To make the target distribution comparable across examples, the paper normalizes the mask to sum to 1:

$p_i = \frac{y_i}{\sum_{j=1}^M y_j + \epsilon}, \quad \text{for } i = 1, ..., M$

where ϵ is a small constant for numerical stability (preventing division by zero when all patches are negative, which should not occur in practice since every grounding example has a target). The resulting p_i is the target attention distribution: it assigns uniform probability mass across all positive patches and zero mass to negative patches.

What this computes: If a bounding box covers K patches, then p_i = 1/K for each of those K patches and p_i = 0 for all others. This is a normalized histogram over the patch grid where the target element is treated as a uniform region.

Why uniform rather than weighted by overlap: An alternative would be to weight each patch by the fraction of its area covered by the bounding box, giving more supervision weight to patches that are fully inside the element. The paper's uniform approach is simpler and avoids introducing assumptions about which parts of an element are "more important" — a click anywhere in a button is equally valid, so treating all overlap patches equally is consistent with the underlying ambiguity of the task.

Step 3: KL-divergence loss. The action head is trained to minimize the Kullback-Leibler divergence between the predicted attention distribution a and the target distribution p:

\mathcal{L}_{\text{Action_Attn}} = \sum_{i=1}^M p_i \log \frac{p_i}{a_i}

What this computes: The KL divergence measures how much information is lost when using the predicted distribution a to approximate the target distribution p. When a_i is close to p_i for all i, the ratio p_i / a_i is close to 1 and the log term approaches 0, so the loss is small. When a_i diverges from p_i — e.g., when the model assigns high attention to patches outside the ground-truth region or low attention to patches inside it — the loss increases. The sum over all patches aggregates these per-patch divergences.

Why KL divergence rather than cross-entropy: The cross-entropy -∑ p_i log(a_i) is equivalent up to an additive constant (the entropy of p, which is constant with respect to the model parameters). However, KL divergence provides a more interpretable interpretation: a loss of 0 means the predicted distribution exactly matches the target, and the minimum achievable loss is 0, not the entropy of p. This makes loss values comparable across examples with different numbers of positive patches.

Why this form addresses ambiguous supervision: In a coordinate-based model with point supervision, the model receives zero loss only if it predicts the exact annotated point. Any other point — even one equally valid within the same button — incurs loss proportional to the Euclidean distance from the target. In GUI-Actor's multi-patch supervision, the model receives zero loss if its attention mass is distributed anywhere within the ground-truth bounding box's patch region. A prediction that attends strongly to the left half of a button is equally rewarded as one that attends to the right half, because both produce a_i ≈ 1/K on positive patches and a_i ≈ 0 on negative patches. This directly addresses the supervision ambiguity limitation outlined in the introduction.

Total loss with next-token prediction. The action attention loss is combined with the standard next-token prediction loss used to train the VLM's language generation:

$\mathcal{L} = \mathcal{L}_{NTP} + \mathcal{L}_{Action_Attn}$

The two losses operate on different outputs: L_NTP supervises the language tokens (including the <ACTOR_START> and <ACTOR_END> markers, which the model must learn to generate at the correct positions), while L_{Action_Attn} supervises only the attention distribution derived from the <ACTOR> token's hidden state. There is no hyperparameter weighting between the two losses — they are simply summed, implying that the paper found the natural scale of both losses to be compatible without explicit balancing.

---

Grounding Verifier: Training Data Construction

The grounding verifier is a separate VLM module trained to answer a binary question: given a screenshot with a visual marker at a proposed location and the original instruction, is this the correct element to interact with? Its training data is constructed entirely from the OS-Atlas dataset without additional human annotation.

Source data format. OS-Atlas provides triplets of the form (image, query, bounding_box) — a screenshot, a natural language instruction describing the target element, and the ground-truth bounding box for that element. Each image is paired with multiple queries and their corresponding bounding boxes (e.g., one screenshot of a settings page might have queries for the "WiFi toggle," the "Bluetooth toggle," and the "Airplane mode toggle," each with its bounding box).

Positive example construction. For each triplet, the paper places a visual marker — specifically a hollow red circle — at the center of the ground-truth bounding box. This marked image paired with the query forms a positive training example annotated with the label 'True'. The choice of marker (hollow circle rather than solid dot, red rather than a screen-typical color) is deliberate: it must be visually distinctive enough for the VLM to detect without overwhelming the underlying GUI content, and it must not be easily confused with native UI elements that might use circular or red design elements.

Negative example construction — Strategy 1 (semantic hard negative). For a given query, the paper selects the center of a different bounding box from the same image — one corresponding to a different query. For example, if the query is "click the WiFi toggle," the negative example might use the center of the Bluetooth toggle's bounding box. This creates a semantically plausible but incorrect candidate: it is a real UI element on the same screen, located in a reasonable position, but it answers the wrong instruction. The model must learn to distinguish between "this is a valid element" and "this is the requested element," which requires reading the instruction carefully and comparing it to the visual context around the marker.

Negative example construction — Strategy 2 (random negative). A point is randomly sampled outside the ground-truth bounding box, potentially on an unrelated region of the screen. This creates an easier negative example where the marker is simply not on any semantically relevant element. The combination of both hard and random negatives ensures the verifier learns both fine-grained discrimination (Strategy 1) and basic existence checking (Strategy 2).

Resulting dataset. Each (image, query, bounding_box) triplet yields one positive example and one negative example, formatted as tuples:

(image_with_marked_point, query, 'True') (image_with_wrong_point, query, 'False')

The paper constructs a balanced training set of 730,000 examples — 365,000 positive and 365,000 negative — from the OS-Atlas data. The balance ensures the verifier does not develop a prior bias toward predicting 'True' or 'False' irrespective of the input.

Verifier training objective. The verifier is fine-tuned from UI-TARS-2B-SFT using standard supervised learning with cross-entropy loss:

$\mathcal{L}_{\text{Verifier}} = -\log P_{\theta_v}(y \mid I, x)$

where I is the marked image, x is the instruction, y ∈ {'True', 'False'} is the ground-truth label, and θ_v denotes the verifier's parameters. The model takes the concatenated image and text as input and generates a single token ('True' or 'False') autoregressively; the loss is computed only on this output token.

---

Grounding Verifier: Inference Procedure

At inference time, the verifier is applied as a post-hoc selection layer over candidates proposed by the action head's attention map.

Candidate pool construction. From the attention distribution {a_1, ..., a_M}, the system extracts the top-K patches by attention weight, where K = 20 in the paper's experiments. To avoid evaluating nearly identical candidates (adjacent patches that differ by a single patch position), patches with attention weights below 20% of the maximum attention weight are filtered out. This removes low-confidence patches from consideration while retaining the highest-probability cluster.

Patch clustering refinement. A limitation of pure patch-based selection is that small elements may straddle patch boundaries — the element's visual center might fall between two or four adjacent patches. To handle this, the paper introduces a clustering step: 4-connected neighboring patches (patches sharing an edge, not just a corner) are grouped into clusters, and a weighted center is computed for each cluster based on the verifier scores (not the attention weights). These weighted centers are added to the candidate pool alongside the raw patch centers, providing sub-patch localization precision without modifying the base model's patch resolution.

Candidate scoring. For each candidate position (x, y), the verifier is applied as follows:

1. Image cropping: A square region of size l_crop × l_crop pixels is cropped from the original screenshot, centered at (x, y). The paper sets l_crop = 1000 pixels for all tasks. This crop provides the verifier with both the proposed location (near the crop center) and surrounding context, enabling it to evaluate whether the marked region semantically matches the instruction.

2. Marker placement: A visual marker (the same hollow red circle used during training) is drawn at the candidate position.

3. Verifier scoring: The verifier processes the cropped, marked image together with the instruction and outputs token probabilities for 'True' and 'False'. The score is defined as the normalized 'True' probability:

$\text{s}(I, x) = \frac{P_{\theta_v}(\text{'True'} \mid I, x)}{P_{\theta_v}(\text{'True'} \mid I, x) + P_{\theta_v}(\text{'False'} \mid I, x)}$

where I is the cropped and marked image, x is the instruction, and P_{θ_v} denotes probabilities from the verifier model.

What this computes: The score is the verifier's confidence that the proposed point is correct, normalized so that predictions of ('True': 0.6, 'False': 0.4) and ('True': 0.9, 'False': 0.6) map to approximately 0.6 and 0.6 respectively (in the first case, 0.6/(0.6+0.4)=0.6; in the second, 0.9/(0.9+0.6)=0.6). This normalization makes scores comparable across examples with different levels of output entropy.

Why normalized probability rather than raw P('True'): Raw probability can be uncalibrated — a verifier might systematically output higher probabilities on certain types of screenshots. Normalizing by the sum of both token probabilities mitigates this by converting the two-class output into a relative confidence measure. This is particularly important because the verifier's score is compared against a fixed threshold γ that is the same across all examples.

Early termination with confidence thresholding. Candidates are evaluated in descending order of attention weight. If a candidate achieves a verifier score exceeding a confidence threshold γ, that candidate is immediately returned without evaluating the remaining pool. The threshold differs by benchmark: γ = 0.95 for ScreenSpot-Pro (harder, more out-of-distribution, requiring more careful verification) and γ = 0.8 for ScreenSpot and ScreenSpot-v2 (easier, where the model's top candidate is more likely correct and verification can be less thorough). This is a computational efficiency measure: on easier benchmarks, the verifier is consulted lightly because the action head's top prediction is usually correct; on harder benchmarks, the verifier is more heavily relied upon.

Fallback behavior. If no candidate exceeds the threshold after evaluating all K candidates (up to 20), the system defaults to the candidate with the highest verifier score. This ensures the system always produces an output, even when the verifier is uncertain about all candidates.

Why a separate verifier rather than thresholding on attention weights directly: The attention weights measure visual salience — how strongly the model associates a patch with the instruction — but they do not measure semantic correctness. A patch with high attention might correspond to a visually similar but semantically wrong element (e.g., the "Bluetooth toggle" when asked for the "WiFi toggle"). The verifier provides a second opinion conditioned on the actual marked location, effectively asking "does this specific point, in context, actually answer the instruction?" This verification-then-selection pattern — generating candidates cheaply then spending additional compute only to evaluate the most promising ones — is more efficient than generating diverse candidates through repeated sampling from a coordinate-generation model, as the paper demonstrates in Figure 7 where Aguvis requires 21 inference passes to achieve comparable verification benefits.

---

Training Recipe

The paper's training procedure has two phases, motivated by the observation that the newly introduced action head parameters need to be initialized to reasonable values before the full model is fine-tuned.

Phase 1: Frozen backbone warm-up. All backbone VLM parameters are frozen, and only the newly introduced components are trained:

• The embedding vectors for the three special tokens (<ACTOR_START>, <ACTOR>, <ACTOR_END>).
• The action head, including the self-attention layer over patch features, the two MLP projections (MLP_T and MLP_V), and any normalization parameters.

This phase uses the combined loss L = L_NTP + L_{Action_Attn}. The L_NTP component affects only the new token embeddings (since the language modeling head's output probabilities depend on the embedding, even if the backbone transformer is frozen). The L_{Action_Attn} component affects only the action head parameters.

Why freeze the backbone during warm-up: The action head's parameters start randomly and would produce meaningless attention maps. If the backbone VLM were also being updated, its patch features and hidden states would shift to compensate for the poor action head, potentially degrading the VLM's general-purpose capabilities before the action head converges. Freezing the backbone ensures the action head learns to work with fixed, high-quality representations, after which joint fine-tuning can refine both components together.

Phase 2: Full fine-tuning. All parameters — backbone VLM and action head — are unfrozen and trained jointly with the same combined loss. The paper trains for 1 epoch on the dataset of approximately 1 million screenshots (listed in Table 7), which draws from several public GUI datasets including OS-Atlas, Wave-UI (with test-set-overlapping samples excluded), and others spanning desktop, mobile, and web domains.

Why only 1 epoch: GUI grounding datasets are large but contain repetitive patterns — the same icon appears across many screenshots, the same spatial layout principles recur. Training for multiple epochs would risk overfitting to specific element-location associations rather than learning generalizable grounding skills. The paper's generalization results on ScreenSpot-Pro (an out-of-distribution benchmark) suggest 1 epoch is sufficient.

Extended variant: GUI-Actor-LiteTrain. The paper explores a variant where Phase 1 is performed but Phase 2 (full fine-tuning) is skipped — training stops after the action head has been trained with the backbone frozen. This variant, termed GUI-Actor-LiteTrain, updates only approximately 100M parameters (for 7B backbone) and preserves the backbone VLM's original weights exactly. Table 5 shows that GUI-Actor-LiteTrain, when combined with the grounding verifier, can approach the performance of fully fine-tuned coordinate generation models, demonstrating that substantial grounding capability can be added to a VLM without modifying its pretrained weights.

Data mixture and preprocessing. The training data is formatted as sequences of pyautogui-style operations (following the Aguvis convention) but with coordinates replaced by the <ACTOR> tokens. For example, a training example might look like:

Instruction: Click the submit button. Output: pyautogui.click(<ACTOR_START><ACTOR><ACTOR_END>)

The model learns to generate the <ACTOR> tokens at the appropriate positions in the action syntax while the action head learns to ground the <ACTOR> token to the correct patches.

---

Inference Pipeline

At inference time, GUI-Actor produces a grounded click point through the following sequence:

1. Action generation: The screenshot and instruction are fed to the VLM, which autoregressively generates an action string (e.g., pyautogui.click(<ACTOR_START><ACTOR><ACTOR_END>)). The <ACTOR> token's hidden state from the final transformer layer is extracted.

2. Attention computation: The action head computes the attention distribution {a_1, ..., a_M} over visual patches using the extracted hidden state and the vision encoder's patch features.

3. Candidate extraction: The top-K attention-weighted patches are collected (K=20, filtered to remove patches below 20% of max attention). Patch centers are added to the candidate pool. If the verifier is enabled, connected patches are clustered and weighted cluster centers are added.

4. Final selection (without verifier): The center of the highest-attention patch is returned as the click point. This is the baseline "GUI-Actor (without verifier)" behavior.

5. Final selection (with verifier): Candidates are evaluated sequentially by the verifier as described above. The first candidate with s(I, x) > γ is returned; if none exceeds the threshold, the highest-scoring candidate overall is returned.

Why multiple candidates from a single forward pass: The attention mechanism produces a full distribution over all patches simultaneously. This is fundamentally different from generation-based methods where each candidate requires a separate autoregressive sampling pass. The paper's Figure 4a demonstrates the practical consequence: GUI-Actor's Hit@k improves substantially from k=1 to k=3 (because the top patches are genuinely different proposals covering different parts of the target element), while Aguvis's Hit@k barely improves (because resampling with temperature tends to produce nearly identical coordinate outputs, e.g., shifting from (0.898, 0.667) to (0.899, 0.666)).

Verifier Self-Aggregation (VS) extension. The paper explores a simple test-time augmentation for the verifier: cropping the image at multiple scales (specifically l_crop = 1200 and l_crop = 1400 for ScreenSpot-Pro) and averaging the verifier scores across crops. This provides a more robust score by incorporating both detailed local information (smaller crop) and broader context (larger crop). Table 9 shows this improves performance on ScreenSpot-Pro, demonstrating that even a simple ensembling strategy can enhance verifier reliability — while also highlighting the need for more inherently robust verifiers that do not require multiple forward passes.

4. Key Insights and Innovations

Innovation 1: Reframing GUI Grounding from Coordinate Regression to Region Attention

The paper's most fundamental intellectual contribution is not a specific architectural component but a diagnostic reframing of the visual grounding task itself. Prior work, almost universally, treated grounding as a coordinate generation problem: the model's job was to produce numeric (x, y) values that happen to land inside the target element. This framing is so natural — a click is a coordinate, after all — that the field largely accepted its associated pathologies as inherent difficulty rather than as artifacts of a mismatched formulation.

GUI-Actor argues that these pathologies (weak spatial alignment, ambiguous supervision, granularity mismatch, all detailed in Section 2) are not properties of the grounding task but of the coordinate-as-text representation. The diagnostic move is: if grounding is fundamentally about identifying a visual region, not about producing a numeric value, then the output space should live in the same coordinate system as the visual features. Patch-level attention is not merely a different implementation — it is a different output modality that respects the spatial structure of the input.

This is a fundamental shift in problem formulation rather than an incremental improvement. It is analogous to the distinction between object detection by regressing bounding box coordinates from a fully connected layer versus predicting them from spatial feature maps: both achieve the same task, but the latter provides an architectural inductive bias that dramatically improves sample efficiency and generalization. The paper's finding that GUI-Actor reaches final accuracy with ~60% of the training data required by coordinate-generation baselines (Figure 3) is the empirical signature of this inductive bias — the model does not need to learn that spatial location matters because the output space already encodes it.

What distinguishes this from other attention-based approaches is the deliberate separation of the grounding pathway from the language generation pathway. Xu et al. (2025) showed that pretrained VLMs' internal attention maps could be used for grounding without fine-tuning, but those maps emerge from the model's general-purpose attention patterns, which were never optimized for spatial precision. GUI-Actor's innovation is to supervise the attention directly through a dedicated action head while keeping the language pathway unchanged — the model still generates natural language action descriptions, but the spatial decision routes through a pathway with explicit spatial structure. This dual-output design (language tokens via LM head, spatial attention via action head) is the key architectural insight that enables both interpretable action descriptions and precise localization.

Innovation 2: Multi-Patch Supervision as a Principled Solution to Point Ambiguity

The problem of "where exactly should the model click within a button?" has been an underappreciated source of training noise in GUI grounding. Coordinate-based methods with point supervision penalize all deviations from the single annotated point, even when the alternative click point would be functionally correct. The paper's analysis in Section 3.4 (compared to Section 2) shows that this ambiguity is not an edge case — it is structural, arising from the fact that the annotation granularity (single pixel) is finer than the functional granularity (the entire element's bounding box).

Prior work attempted to mitigate this by predicting bounding boxes instead of points (e.g., Aguvis with bounding-box supervision), but the paper's ablation (Table 6) reveals a surprising negative result: bounding-box supervision in a coordinate-generation framework performs similarly to or worse than point supervision. This is not an obvious outcome — one would expect richer spatial targets to produce better models. The paper's explanation is that without architectural mechanisms to connect the predicted coordinates to visual features, the additional spatial information in bounding boxes cannot be effectively utilized; the model still learns an opaque text-to-text mapping, and predicting four coordinates instead of two merely increases the output complexity without improving the underlying spatial alignment.

GUI-Actor's multi-patch supervision solves this differently by operating at the spatial resolution where the ambiguity naturally lives. All patches overlapping the ground-truth bounding box are treated as equally valid positive targets, and the normalized KL-divergence loss encourages the model to distribute its attention mass across the entire element region. A prediction that attends to the left half of a button is equally rewarded as one that attends to the right half — both satisfy the supervision target. This transforms the training signal from a precise-but-noisy point target to a region-level target where correctness is defined by inclusion, not by exact coincidence.

This is a conceptual innovation in supervision design rather than an architectural one, and it has practical implications beyond GUI grounding. Any task where the target has spatial extent and the precise point is ambiguous — visual question answering about image regions, robotic grasping points, medical image segmentation — could potentially benefit from this density-aware supervision strategy. The key insight is that the supervision granularity should match the functional granularity of the task, not the annotation granularity of the dataset.

The evidence in Figure 4a reinforces this: GUI-Actor's Hit@k improves substantially from k=1 to k=3 because the model learns to attend to the full spatial extent of the element, producing diverse top-K patches that all lie within the ground-truth bounding box. In contrast, coordinate-based baselines produce nearly identical outputs when resampled, even with high temperature — they have learned to predict one specific point and have no mechanism for expressing the full valid region.

Innovation 3: The Verifier-as-Selector Pattern for Dense Visual Outputs

The idea that "verification is easier than generation" (Cobbe et al., 2021) has been productively applied to language reasoning through process reward models and self-consistency checks. GUI-Actor transplants this idea into the visual grounding domain but with a critical distinction: the verifier does not check an external candidate pool but rather re-ranks candidates produced by the same model in the same forward pass.

This is fundamentally different from how verifiers are typically used with coordinate-generation models. When Aguvis is augmented with a verifier (Figure 7), it requires 21 separate inference passes — one deterministic pass plus 20 stochastic samples — because each pass produces essentially one candidate (resampling barely changes the output). The verifier is being used to compensate for the lack of diversity in the proposal distribution, which is expensive.

GUI-Actor's attention mechanism produces a diverse candidate pool as a free byproduct — the attention distribution over all patches exists after a single forward pass, and the top-K patches are genuinely different spatial proposals. The verifier's role shifts from "generate more candidates" to "select among already-diverse candidates." This makes the verification step far more efficient (the paper estimates ~5% of the compute compared to Aguvis's approach) while simultaneously making it more effective (because the verifier is choosing among genuinely distinct options).

This is a practical innovation in inference efficiency with a conceptual underpinning: architectures that produce dense spatial outputs (attention maps, heatmaps, segmentation masks) can decouple candidate generation from candidate selection in a way that sparse outputs (single coordinate predictions) cannot. The design pattern — train a model to produce a rich spatial proposal distribution, then train a lightweight verifier to select from it — could be applied to any dense prediction task where verification is cheaper or more reliable than generation.

The verifier's training data construction is also notable for its simplicity and scalability. By using the existing OS-Atlas dataset and automating positive/negative example generation (center of correct bounding box = positive; center of different bounding box = hard negative; random point = easy negative), the verifier requires no additional human annotation. The hard negatives (semantically plausible but incorrect elements from the same screenshot) are particularly important — they force the verifier to read the instruction carefully rather than simply detecting whether any UI element exists at the marked location. This automated hard-negative mining strategy is a practical contribution that makes the verifier approach reproducible.

Innovation 4: Preserving General-Purpose Capabilities Through Selective Training

A persistent tension in adapting foundation models for specialized tasks is the capability tradeoff: fine-tuning improves task performance but often degrades the model's general abilities. The paper addresses this through an empirical finding that is both surprising and practically significant: the backbone VLM's pretrained representations already contain strong perceptual understanding of GUI elements, and what coordinate-based fine-tuning primarily teaches is coordinate mapping, not visual semantics.

The evidence for this claim comes from the GUI-Actor-LiteTrain variant (Table 5). When only the action head (~100M parameters for a 7B backbone) is trained with the backbone VLM entirely frozen, the resulting model still achieves competitive grounding performance, especially when augmented with the verifier. This implies that the pretrained VLM can already see the button and understand it's the target — it just lacks a mechanism to point to it. The action head provides this pointing mechanism without modifying the representations that support language understanding, visual reasoning, and other general capabilities.

This finding is significant beyond GUI grounding because it suggests a modular capability-augmentation paradigm: rather than fine-tuning a foundation model end-to-end for each downstream task (risking capability degradation), one can attach small, task-specific output heads that leverage the pretrained representations without modifying them. The key property that enables this is the existence of an appropriate intermediate representation — in this case, the <ACTOR> token's hidden state — that encodes task-relevant information in a format that the downstream head can consume. Identifying such representations for other tasks (robot control, image editing, audio generation) could enable similar lightweight adaptation.

This is incremental at the technical level (fine-tuning subset of parameters while freezing the backbone is a standard technique) but fundamental at the conceptual level because it recharacterizes what fine-tuning for GUI grounding actually accomplishes. The paper's ablation (Table 6) showing that bounding-box supervision without architectural mechanisms does not improve over point supervision supports this recharacterization: adding more spatial information to the target format does not help if the model's architecture prevents it from using that information. The bottleneck is not the supervision signal but the output representation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three GUI visual grounding benchmarks: ScreenSpot (Cheng et al., 2024b) — 1,272 single-step instructions across mobile, desktop, and web platforms, with elements categorized as text or icon types; ScreenSpot-v2 (Wu et al., 2024b) — a corrected version of ScreenSpot that fixes annotation errors and ambiguous instructions while maintaining the same sample count; and ScreenSpot-Pro (Li et al., 2025) — 1,581 expert-annotated tasks across 23 professional applications spanning three operating systems, featuring higher-resolution interfaces and substantial domain shift from standard training data (industrial software, multi-window layouts). The last benchmark is explicitly treated as an out-of-distribution generalization test.

• Base model(s). The primary backbone is Qwen2-VL-7B-Instruct (Wang et al., 2024b), chosen because it is a widely-used open-source VLM that supports fair comparison with other methods using the same backbone (Aguvis-7B, UGround-v1-7B, UI-TARS-7B). The paper also reports results with Qwen2-VL-2B-Instruct for a smaller-scale variant and Qwen2.5-VL (Bai et al., 2025) to demonstrate transfer across backbone generations. For the grounding verifier, the base model is UI-TARS-2B-SFT (Qin et al., 2025). The choice of 7B scale is strategic: it enables comparison with the dominant open-source GUI agent models while remaining reproducible without massive compute resources.

• Metrics. The primary metric is Element Accuracy, defined as the proportion of predictions where the predicted click point falls within the ground-truth bounding box of the target element. This is a binary per-example metric: a prediction is correct if and only if the (x, y) coordinate lands inside the annotated bounding box. For online evaluation on OS-World-W, the metric is Task Success Rate — the fraction of multi-step tasks completed successfully (verified by handcrafted scripts). The paper also reports Hit@k for multi-candidate analysis, measuring whether any of the top-k predicted regions contains a point inside the ground-truth bounding box.

• Baselines. The paper compares against three categories: (i) Closed-source models: GPT-4o (OpenAI, 2024), Claude for Computer Use (Anthropic, 2024), and Gemini 2.0 (Google, 2024); (ii) Open-source coordinate-generation models: SeeClick (Cheng et al., 2024b), ShowUI (Lin et al., 2024), Magma (Yang et al., 2025), Aguvis-7B (Xu et al., 2024) with both point supervision and bounding-box supervision variants, UGround-v1-7B (Gou et al., 2024), and UI-TARS-7B/72B (Qin et al., 2025); (iii) Backbone-matched baselines: the unmodified Qwen2-VL (Wang et al., 2024b) evaluated as a zero-shot grounding model, and Jedi (Xie et al., 2025) built on Qwen2.5-VL. For Aguvis baselines, the authors re-implemented both point-supervised and bounding-box-supervised variants using the official source code to ensure identical training data and conditions. All numbers are reported from original papers or from the UI-TARS benchmark (Qin et al., 2025) unless otherwise noted.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time but rather in number of forward passes required per grounding decision. GUI-Actor produces all candidate regions in a single forward pass; the verifier adds one additional pass per evaluated candidate (up to 20, with early termination reducing the average). For the baseline comparison with Aguvis augmented by the verifier, the paper reports that Aguvis requires 21 forward passes (1 deterministic + 20 stochastic samples) versus GUI-Actor's single pass for candidate generation. Figure 7 quantifies this as GUI-Actor requiring "only about 5% of the computation during inference" compared to Aguvis with verifier. Training data scale is measured in number of screenshots (~1M total, detailed in Table 7) and compared across methods in Figure 1.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the compute-optimal scaling example). Instead, it follows standard benchmarking protocol: models are trained once on the full training set (Table 7 datasets, 1 epoch) and evaluated on held-out test benchmarks. The ScreenSpot-Pro benchmark serves as a natural out-of-distribution test since it features professional applications and resolutions not represented in training. For the OS-World-W online evaluation, tasks are drawn from a curated subset of 49 Windows-specific tasks with no training exposure. No confidence intervals or statistical significance tests are reported. The verdict on generalization is based on performance deltas rather than statistical tests.

Main Quantitative Results

ScreenSpot-Pro: Out-of-Distribution Generalization

ScreenSpot-Pro is the paper's most emphasized benchmark because it measures generalization to professional interfaces with higher resolutions and substantial domain shift. Table 1 presents the main results.

Headline result. GUI-Actor-7B (without verifier) achieves a score of 40.7 on ScreenSpot-Pro, outperforming all prior open-source models at comparable scale. With the verifier, performance rises to 44.6, surpassing UI-TARS-72B (38.1) by 6.5 points despite using ~10× fewer parameters (7B vs. 72B) and less training data (Figure 1, left). GUI-Actor-2B (without verifier) achieves 35.2, already exceeding Aguvis-7B (point sup., 29.5) and Aguvis-7B (bbox sup., 26.4) by substantial margins — a 2B model outperforming 7B coordinate-generation baselines by 5.7–8.8 points.

Comparison with backbone-matched baselines (same training data). This is the most controlled comparison because it isolates the method from data and backbone effects. On ScreenSpot-Pro:

• Aguvis-7B (point sup.): 29.5
• Aguvis-7B (bbox sup.): 26.4
• GUI-Actor-7B (no verifier): 40.7 (+11.2 over best Aguvis)
• GUI-Actor-7B (w/ verifier): 44.6 (+15.1 over best Aguvis)

The gain from the coordinate-free approach alone (without verifier) is 11.2 absolute points, representing a ~38% relative improvement. The verifier adds another 3.9 points.

Comparison with larger models and closed-source systems. UI-TARS-7B achieves 25.7 and UI-TARS-72B achieves 38.1. GUI-Actor-7B without verifier (40.7) already exceeds the 72B model. With verifier (44.6), it exceeds the 72B model by a larger margin. Among closed-source models, GPT-4o achieves 50.9 — still ahead of GUI-Actor-7B with verifier by 6.3 points — but GUI-Actor-7B is approaching this level with a much smaller, open-source model. Claude for Computer Use (35.3) and Gemini 2.0 (28.1) are substantially below GUI-Actor-7B.

Domain-level breakdown (Table 1). ScreenSpot-Pro reports performance across three operating system domains (Development, Creative, Office) and an aggregate. The paper reports only the aggregate score in the main table, but the per-domain splits in the full table provide granularity on where gains concentrate. GUI-Actor-7B with verifier achieves the highest scores across all three OS domains compared to other open-source models.

ScreenSpot: Standard In-Distribution Benchmark

Table 2 presents results on the original ScreenSpot benchmark, which measures grounding on standard-resolution interfaces across platforms (mobile, desktop, web) and element types (text, icon).

Headline result. GUI-Actor-7B (no verifier) achieves 77.0 on the overall ScreenSpot metric. With verifier: 77.7. These scores place GUI-Actor in the upper tier of open-source models, competitive with UI-TARS-7B (77.8 without verifier) despite the latter being trained on significantly larger datasets including proprietary data.

Comparison with backbone-matched baselines. The critical comparison is again with Aguvis variants trained on the same data:

• Aguvis-7B (point sup.): 73.7
• Aguvis-7B (bbox sup.): 71.3
• GUI-Actor-7B (no verifier): 77.0 (+3.3 over best Aguvis)

The gain is smaller than on ScreenSpot-Pro (3.3 vs. 11.2 points), which is expected — ScreenSpot is an in-distribution benchmark where coordinate-based methods can partially compensate through memorization of common layouts. The fact that GUI-Actor still maintains an edge suggests the architectural advantages generalize even when distribution shift is minimal.

Element-type breakdown (Table 2). ScreenSpot reports separate accuracy for text elements and icon/widget elements. GUI-Actor-7B (no verifier) achieves 68.3 on text and 53.3 on icon elements (these appear to be per-type breakdowns from the full metric structure, though the paper's Table 2 only reports the aggregate; the described split follows the ScreenSpot benchmark format). Icon elements are typically harder because they lack text cues and rely purely on visual appearance.

Verifier contribution on ScreenSpot. The verifier improves performance from 77.0 to 77.7 — a modest 0.7-point gain. This contrasts with the 3.9-point gain on ScreenSpot-Pro. The paper attributes this difference to the fact that on in-distribution data, the action head's top prediction is already highly reliable, leaving less room for the verifier to add value through candidate selection.

ScreenSpot-v2: Corrected Annotations

Table 3 presents results on ScreenSpot-v2, which fixes annotation errors from the original ScreenSpot while maintaining the same sample count and distribution.

Headline result. GUI-Actor-7B (no verifier) achieves 81.2. With verifier: 81.7. The ~4-point improvement over ScreenSpot (77.0 → 81.2) is consistent with the corrected annotations removing ambiguous or incorrectly labeled examples that penalized reasonable predictions.

Comparison with backbone-matched baselines:

• Aguvis-7B (point sup.): 78.2† († indicates results from the authors' own evaluation of the official model)
• Aguvis-7B (bbox sup.): 77.6
• GUI-Actor-7B (no verifier): 81.2 (+3.0 over best Aguvis)

The ~3-point gap is similar in magnitude to the ScreenSpot gap, reinforcing that the method's advantage is consistent across in-distribution benchmarks.

Cross-benchmark pattern. GUI-Actor-7B achieves 77.0 on ScreenSpot, 81.2 on ScreenSpot-v2, and 40.7 on ScreenSpot-Pro. The large drop on ScreenSpot-Pro (40-point gap from ScreenSpot-v2) reflects the genuine difficulty of professional interfaces and domain shift, not a method-specific weakness — all models exhibit a similar drop, though GUI-Actor's drop is proportionally smaller than Aguvis's (from 78.2 to 29.5, a 48.7-point drop vs. 81.2 to 40.7, a 40.5-point drop).

Qwen2.5-VL Backbone Results

Table 4 reports results when GUI-Actor is built on the newer Qwen2.5-VL backbone instead of Qwen2-VL.

Headline result. GUI-Actor-7B on Qwen2.5-VL achieves 44.6 on ScreenSpot-Pro (without verifier), compared to 40.7 with Qwen2-VL — a 3.9-point improvement from the backbone upgrade alone. GUI-Actor-3B on Qwen2.5-VL achieves 42.2, nearly matching the 7B model on the older backbone (40.7). With Qwen2.5-VL-7B as the backbone and the verifier: 44.6 (the verifier adds no gain on this backbone, presumably because the stronger backbone's grounding is already highly calibrated).

Comparison with Jedi (Xie et al., 2025), which also uses Qwen2.5-VL as backbone and represents a concurrent coordinate-based approach that decomposes interfaces into components. GUI-Actor-7B (Qwen2.5-VL) achieves 44.6 versus Jedi-7B at 36.1 — an 8.5-point advantage.

Cross-backbone generalization. The fact that GUI-Actor's gains transfer from Qwen2-VL to Qwen2.5-VL (both showing 3–11 point advantages over coordinate-based competitors using the same backbone) suggests the architectural benefits are not specific to a particular VLM implementation. The attention-based action head is a modular component that can be attached to any VLM that exposes patch-level features.

Sample Efficiency and Training Dynamics

Figure 3 provides accuracy progression curves over training steps for both GUI-Actor and Aguvis baselines on the three benchmarks.

Key finding on sample efficiency. GUI-Actor-7B reaches its final accuracy on ScreenSpot and ScreenSpot-v2 using only approximately 60% of the training data (Figure 3a, 3b). Aguvis variants (both point and bbox supervision) plateau only after 80–90% of the data. The paper attributes this to GUI-Actor's explicit spatial-semantic alignment through the action head, which reduces the amount of data needed to learn effective grounding by providing architectural structure that coordinate-based models must learn from scratch.

Key finding on out-of-distribution robustness. Figure 3c shows accuracy on ScreenSpot-Pro over training steps. Aguvis-7B (point sup.) peaks early (~step 200) at approximately 30% and then declines steadily to below 28% by the end of training — a classic overfitting pattern where the model initially transfers some capabilities to out-of-distribution data but then adapts to training-distribution-specific patterns that hurt generalization. GUI-Actor-7B shows a qualitatively different trajectory: it dips initially (likely as the model adapts from pretrained representations to GUI-specific grounding), then recovers and gradually increases, stabilizing above 38% without the sustained decline observed in the baseline. GUI-Actor-2B follows a similar pattern at lower absolute accuracy. The paper characterizes this as "no sustained overfitting" and attributes it to the action head's operation at the vision backbone's native patch resolution, which avoids learning resolution-specific coordinate mappings that fail under distribution shift.

Multi-Region Prediction and Hit@k Analysis

Figure 4a reports Hit@1 and Hit@3 for GUI-Actor and Aguvis baselines.

Key finding on candidate diversity. GUI-Actor-7B shows a substantial improvement from Hit@1 to Hit@3 (the exact numbers are visible in the figure but not explicitly quoted in the text — the text describes the gap as "substantial"). Aguvis-7B (point sup.) shows minimal improvement from Hit@1 to Hit@3. The paper's qualitative analysis reveals why: when Aguvis is sampled multiple times with temperature, the outputs are nearly identical — "shifting slightly from (0.898, 0.667) to (0.899, 0.666)." GUI-Actor's attention mechanism, in contrast, naturally produces mutually exclusive candidate regions (different patches or different weighted cluster centers) because attention is distributed across the full spatial extent of the target element. This diversity is what makes the verifier effective — it has genuinely different options to choose among.

Online Evaluation on OS-World-W

Table 10 reports task success rates on the OS-World-W subset (49 Windows-specific multi-step tasks). GPT-4o serves as the planner for all grounding methods, isolating the grounding component's contribution.

Headline result. GUI-Actor-7B achieves 12.2% task success rate, compared to:

• OmniAgent (Lu et al., 2024b): 10.2%
• NAVI (Bonatti et al., 2024): 10.2%
• Aguvis-7B (point sup.): 4.0%

GUI-Actor outperforms the best baseline (OmniAgent/NAVI) by 2.0 absolute points (20% relative improvement) and outperforms Aguvis-7B by 8.2 points (3× improvement). These are multi-step tasks where grounding errors compound — an error at any step typically causes task failure — so the grounding method's per-step accuracy has amplified effects on end-to-end success.

Caveats. The absolute success rates are low across all methods (10–12%), reflecting the difficulty of OS-World tasks which require complex multi-step reasoning, handling of pop-ups and unexpected states, and precise action execution. The paper notes that these evaluations serve as "quick validation" and that GUI-Actor had "no exposure to OSWorld-W tasks during training," so the results primarily measure generalization rather than task-specific optimization. The small sample size (49 tasks) means individual task successes or failures can meaningfully shift percentages.

Lightweight Training: Preserving General Capabilities

Table 5 reports results for GUI-Actor-LiteTrain, where only the action head and special token embeddings are trained (backbone VLM frozen), compared to the fully fine-tuned GUI-Actor and baselines.

Headline result (ScreenSpot-Pro). GUI-Actor-7B-LiteTrain without verifier achieves 24.7, substantially below fully fine-tuned GUI-Actor-7B (40.7) but above the unmodified backbone Qwen2-VL-7B (12.1). With the verifier, performance jumps to 37.8 — within 2.9 points of fully fine-tuned GUI-Actor with verifier (44.6) and competitive with or exceeding fully fine-tuned coordinate generation models like Aguvis-7B (point sup., 29.5) and UI-TARS-7B (25.7).

Key finding on the verifier's role in lightweight training. The verifier provides an outsized benefit for LiteTrain: +13.1 points on ScreenSpot-Pro, compared to +3.9 points for the fully fine-tuned model. This suggests that when the grounding model is weaker (LiteTrain), the verifier's candidate selection has more room to correct errors; when the grounding model is already strong (full fine-tuning), the verifier primarily confirms the already-correct top prediction.

Key interpretation. The paper argues that the backbone VLM's pretrained representations already contain strong perceptual understanding of UI screenshots, and that coordinate-based fine-tuning primarily teaches coordinate mapping rather than visual semantics. The fact that LiteTrain (which preserves the backbone weights exactly) can approach fully fine-tuned coordinate models when augmented with the verifier supports this interpretation: the visual understanding needed for grounding largely exists in the pretrained VLM, and the action head merely provides a mechanism to express spatial decisions without disrupting those representations.

Verifier Self-Aggregation

Table 9 reports results for verifier self-aggregation (VS) on ScreenSpot-Pro, where multiple crop scales are used and their verifier scores are averaged.

Result. VS with two crop sizes (1200 and 1400 pixels) improves performance over single-crop verification. The exact numerical improvement is shown in Table 9. The paper presents this as evidence that multi-scale verification is a simple yet effective enhancement, while also noting that it points to the need for more inherently robust verifiers that do not require multiple forward passes.

Ablation Studies and Robustness Checks

• Coordinate generation vs. coordinate-free (Table 6, Aguvis comparison rows): Models trained with coordinate generation (both bounding-box and point supervision) consistently underperform GUI-Actor-7B across all three benchmarks. On ScreenSpot-Pro: Aguvis-7B (point sup.) 29.5, Aguvis-7B (bbox sup.) 26.4 vs. GUI-Actor-7B 40.7 — a gap of 11.2–14.3 points. On ScreenSpot: 73.7 and 71.3 vs. 77.0 (gap of 3.3–5.7 points). On ScreenSpot-v2: 78.2 and 77.6 vs. 81.2 (gap of 3.0–3.6 points). The larger gap on ScreenSpot-Pro directly supports the claim that coordinate-free grounding improves out-of-distribution generalization.

• Bounding-box supervision without architectural mechanisms (Table 6, Aguvis bbox vs. point): Aguvis-7B trained with bounding-box supervision (26.4 on ScreenSpot-Pro) performs similarly to or worse than point supervision (29.5). This is a non-obvious negative result: providing richer spatial targets does not improve grounding when the model lacks architectural mechanisms to connect coordinates to visual features. The paper interprets this as evidence that "without architectural mechanisms or spatial inductive bias, these coordinate generation based methods remain disconnected from the underlying visual representation."

• Lightweight training (Table 5, LiteTrain rows): Freezing the backbone VLM and training only the action head and special token embeddings (~100M parameters) yields 24.7 on ScreenSpot-Pro without verifier — substantially below full fine-tuning (40.7) but far above the unmodified backbone (12.1). Adding the verifier recovers most of the gap, reaching 37.8 (vs. 44.6 fully fine-tuned with verifier). This demonstrates that: (a) the backbone VLM's pretrained representations already contain substantial GUI understanding; (b) the action head can be trained modularly without modifying the backbone; and (c) the verifier is particularly valuable when the grounding model is weaker.

• Verifier contribution across benchmarks (Tables 1, 2, 3, 5): The verifier improves GUI-Actor-7B by +3.9 on ScreenSpot-Pro, +0.7 on ScreenSpot, +0.5 on ScreenSpot-v2. For LiteTrain, the gains are +13.1 on ScreenSpot-Pro. This pattern confirms that the verifier's value is largest when (a) the task is hard (ScreenSpot-Pro) or (b) the grounding model is weak (LiteTrain). On easy, in-distribution benchmarks with a strong grounding model, the verifier provides marginal benefit because the top attention patch is already correct.

• Candidate diversity (Figure 4a, Hit@k analysis): GUI-Actor's Hit@k improves substantially from k=1 to k=3, while Aguvis baselines show minimal improvement. The paper's qualitative analysis (Section 5, Multi-Region Prediction) confirms that Aguvis resampling produces nearly identical coordinate outputs, while GUI-Actor's attention map naturally produces spatially distinct candidates. This validates a key design claim: the attention-based approach produces genuinely diverse proposals in a single pass, enabling efficient verifier-based selection.

• Training dynamics and overfitting (Figure 3): Aguvis-7B exhibits declining accuracy on ScreenSpot-Pro after an early peak (~step 200), indicating overfitting to training-distribution patterns. GUI-Actor shows recovery after initial dips and stabilization without sustained decline. The paper attributes this to operating at the vision backbone's native patch resolution, which avoids learning resolution-specific coordinate mappings that fail under distribution shift. This is a robustness check on the generalization claim.

• Backbone transfer (Qwen2-VL vs. Qwen2.5-VL, Tables 1 vs. 4): GUI-Actor's advantages transfer across backbone generations. On Qwen2.5-VL-7B, GUI-Actor achieves 44.6 on ScreenSpot-Pro vs. 40.7 on Qwen2-VL-7B — a 3.9-point backbone-driven improvement, with both substantially ahead of coordinate-based competitors using the same backbone (Jedi-7B: 36.1). The modular action head design successfully generalizes to different VLM architectures.

• Verifier applicability to baselines (Appendix G.2, Figure 7): When the same verifier is applied to Aguvis (requiring 21 forward passes for candidate generation + verification), GUI-Actor still achieves "considerably higher grounding accuracy" while using ~5% of the compute. This demonstrates that the verifier's effectiveness is amplified by GUI-Actor's diverse single-pass candidate proposals, not solely by the verifier's standalone discriminative power.

• Verifier self-aggregation (Table 9): Multi-scale verification (1200 + 1400 pixel crops) improves performance on ScreenSpot-Pro over single-scale verification. This is presented as a simple test-time augmentation that enhances robustness, while the paper acknowledges it as a direction for future improvement rather than a core contribution.

Critical Assessment

Claim 1: GUI-Actor outperforms prior state-of-the-art methods across multiple benchmarks. This claim is supported for the specific comparison class the paper targets: open-source models at comparable scale (~7B parameters) trained on public data. The evidence is clearest on ScreenSpot-Pro (Table 1), where GUI-Actor-7B (40.7) exceeds all backbone-matched baselines (Aguvis-7B: 29.5) and even UI-TARS-72B (38.1). On ScreenSpot (Table 2) and ScreenSpot-v2 (Table 3), the margins are smaller (3.0–3.3 points over Aguvis) but consistent. However, the claim of "outperforming prior state-of-the-art" requires qualification: GPT-4o achieves 50.9 on ScreenSpot-Pro, 6.3 points above GUI-Actor-7B with verifier. GUI-Actor is state-of-the-art among open-source, reasonably-sized models, but not overall. The paper generally makes this distinction clear but the unqualified "outperforms prior state-of-the-art methods" in the abstract and conclusion could be read as claiming overall superiority.

Claim 2: GUI-Actor exhibits greater robustness to unseen screen sizes and resolutions. This claim is supported by the ScreenSpot-Pro results (Table 1), which explicitly tests higher-resolution professional interfaces with substantial domain shift. The 11.2-point gap over Aguvis on ScreenSpot-Pro versus the 3.3-point gap on ScreenSpot is the empirical signature of improved out-of-distribution generalization — the advantage widens as distribution shift increases. The training dynamics in Figure 3c provide additional support: Aguvis overfits (declining accuracy after early peak) while GUI-Actor stabilizes. However, the paper does not systematically vary resolution or screen size as a controlled experimental factor — it relies on ScreenSpot-Pro's natural distribution shift as a proxy. A direct experiment testing the same model on the same screenshots at multiple resolutions would provide stronger evidence.

Claim 3: Multi-patch supervision addresses ambiguous supervision targets. The evidence for this is primarily structural (the method labels all overlapping patches as positive) combined with the ablation showing bounding-box supervision without architectural mechanisms does not improve over point supervision (Table 6, Aguvis-7B bbox sup. 26.4 vs. point sup. 29.5 on ScreenSpot-Pro). This is a negative result that supports the claim: providing richer targets doesn't help unless the model architecture can use them. However, the paper does not directly ablate multi-patch supervision against single-patch supervision within GUI-Actor — there is no experiment showing GUI-Actor trained with only the center patch labeled as positive. This is a missing ablation that would directly test the claim.

Claim 4: The verifier improves performance and is easily integrated. Supported across all benchmarks (Tables 1, 2, 3, 5, 9, 10). The verifier consistently adds value, with the largest gains on hard benchmarks (+3.9 on ScreenSpot-Pro) and for weaker models (+13.1 on LiteTrain). The integration with Aguvis (Figure 7) demonstrates portability. However, the verifier adds inference cost (up to 20 additional forward passes, though early termination reduces this) and requires training a separate model on 730K examples — it is not "free" in terms of either compute or data. The claim of "lightweight" should be contextualized against the training cost of a 2B-parameter VLM on a large synthetic dataset.

Missing experiments and analyses. Several experiments would strengthen the paper's claims:

• Resolution scaling study. Systematically evaluating GUI-Actor on the same screenshots rendered at multiple resolutions (e.g., 720p, 1080p, 1440p, 4K) would directly test the claim that operating at patch resolution improves robustness to resolution changes. The current reliance on ScreenSpot-Pro's natural variation conflates resolution changes with domain shift (professional software vs. consumer apps).

• Single-patch vs. multi-patch supervision ablation. Training GUI-Actor with only the center patch of each bounding box labeled as positive would directly test whether multi-patch supervision contributes to performance or whether the attention mechanism alone drives the gains.

• Verifier threshold sensitivity analysis. The paper uses different confidence thresholds for different benchmarks (0.95 for ScreenSpot-Pro, 0.8 for ScreenSpot and ScreenSpot-v2) without reporting how performance varies with threshold choice. A sweep across threshold values would reveal whether the method is robust to this hyperparameter.

• Statistical significance. With test sets of 500–1,581 examples, the differences between methods (particularly the smaller gaps on ScreenSpot and ScreenSpot-v2, ~3 points) may not be statistically significant. The paper does not report confidence intervals or perform significance testing.

• Alternative verifier architectures. The verifier is trained from UI-TARS-2B-SFT with a specific data construction strategy. Ablating verifier model size, training data scale, or the hard-negative mining strategy would clarify which factors matter most for verifier effectiveness.

• Failure mode analysis. Beyond the qualitative attention map visualizations (Figure 5, Appendix C), the paper does not provide a systematic analysis of when and why GUI-Actor fails. On the ~55% of ScreenSpot-Pro examples where GUI-Actor-7B with verifier does not succeed, what goes wrong? Does the attention map fail to highlight the correct region? Does the verifier select an incorrect candidate? Do small element sizes (the limitation acknowledged in Appendix A) account for a significant fraction of failures? A structured error taxonomy would both strengthen the paper and guide future work.

• Combined search and revision paradigm from the analogous paper. The paper introduces a verifier-as-selector pattern but does not explore iterative refinement: using the verifier's feedback to adjust the attention map (e.g., suppressing high-attention but verifier-rejected regions and re-normalizing). This would parallel the search-revision combination that the analogous paper identifies as a promising direction.

Real-world evaluation limitations. The OS-World-W evaluation (Table 10) provides welcome real-world validation but has important limitations: 49 tasks is a small sample (each task is a binary success/failure, so one task represents ~2 percentage points); GPT-4o serves as the planner for all methods, meaning failures due to planning errors are shared across methods and mask differences in grounding quality; and the low absolute success rates (10–12%) mean the signal-to-noise ratio is limited. The paper appropriately treats this as "quick validation" rather than definitive evidence.

Data contamination concerns. The paper excludes Wave-UI samples that overlap with test sets but does not discuss whether other training datasets (OS-Atlas, other GUI datasets) contain screenshots or queries that overlap with ScreenSpot or ScreenSpot-v2 test examples. Given that these benchmarks are constructed from publicly available GUI screenshots, some degree of overlap is plausible and would inflate in-distribution performance estimates. The ScreenSpot-Pro results are less susceptible to this concern because the benchmark explicitly targets professional applications not represented in standard training data.

Granularity limitation. Appendix A acknowledges that the base model's fixed 28×28 pixel patch size poses challenges for very small interface elements. This is a genuine limitation — if a button is smaller than 28×28 pixels (e.g., small toolbar icons at high resolutions), it may be represented by only one or two patches, reducing the precision of the attention map. The paper's patch clustering refinement partly mitigates this but cannot overcome the fundamental resolution limit of the visual encoder. The paper does not report what fraction of ScreenSpot-Pro targets are below this size threshold, which would help readers assess the practical impact.

6. Limitations and Trade-offs

The Patch Resolution Ceiling

The assumption or constraint. GUI-Actor grounds actions at the vision backbone's native patch resolution — typically 28×28 pixels per patch for the Qwen2-VL models used throughout the paper. The method makes no attempt to predict sub-patch coordinates; the finest localization unit is a single patch center or a weighted average of connected patch centers. The paper acknowledges this explicitly in Appendix A:

"the backbone VLM (e.g., Qwen2-VL) adopts a Naive Dynamic Resolution strategy with a fixed patch size of 28×28 pixels. This poses challenges when dealing with very small interface elements (e.g., icons smaller than 10×10 pixels), as such fine-grained details may be insufficiently represented."

The consequence. For very small UI elements — toolbar icons at high display resolutions, tiny close buttons, compact toggle switches — the target element occupies only a fraction of one patch or straddles the boundary between two patches. In these cases, the attention map's spatial precision is fundamentally limited: the model can at best identify which patch or patch boundary the element lies near, but the resulting click point may fall outside the element's bounding box if the element is smaller than half a patch width (~14 pixels). The paper does not characterize how frequently this occurs in practice. On professional interfaces at high resolutions (exactly the regime tested by ScreenSpot-Pro), many UI elements are rendered at small physical sizes. If a significant fraction of ScreenSpot-Pro failures are attributable to sub-patch target sizes, then resolving this limitation would be the most direct path to closing the remaining gap with closed-source models like GPT-4o (50.9 vs. GUI-Actor-7B's 44.6).

What evidence exists in the paper. No direct measurement. The paper does not report the size distribution of target elements in ScreenSpot-Pro or any other benchmark, does not bucket performance by target element size (in pixels or patches), and does not analyze what fraction of failures involve targets below the patch size threshold. The paper's qualitative visualizations (Figure 5, Appendix C) show attention maps over elements that span multiple patches — these are relatively large targets. The patch clustering refinement (Section F.2) is presented as a mitigation but is not evaluated in isolation to show whether it specifically helps on small targets.

Mitigation status. The paper introduces patch clustering (grouping 4-connected neighboring patches and computing weighted centers) as "a simple yet effective refinement" that "enables the generation of candidate points that lie between adjacent patches." This helps when an element straddles a patch boundary but does not address the case where an element is smaller than a single patch. The paper acknowledges that "fully addressing this limitation may require more substantial advancements in the future, such as improving the visual encoder's perceptual resolution or incorporating offset-based spatial refinement." There is no proposal for how to achieve this within the current architecture.

---

Difficulty Estimation Cost: The Hidden Overhead of Candidate Verification

The assumption or constraint. The grounding verifier evaluates up to K = 20 candidate regions per query, with each evaluation requiring a full forward pass of a 2B-parameter VLM (fine-tuned from UI-TARS-2B-SFT). While early termination via the confidence threshold γ limits the average number of verifier calls, the worst case is 20 additional forward passes per grounding decision. For ScreenSpot-Pro, the paper uses a high threshold (γ = 0.95), meaning candidates are less likely to trigger early termination, and the average verifier calls per query may be relatively high. The headline performance numbers (Table 1: 44.6 with verifier) include this computational cost only implicitly — the metric is accuracy, not accuracy-per-FLOP or accuracy-per-second.

The consequence. In latency-sensitive interactive settings — which is the target use case for GUI agents that operate in a closed perception-action loop — the verifier adds substantial wall-clock time. Each verifier call requires running a 2B-parameter transformer on a 1000×1000 pixel cropped image. For 20 candidates, this is roughly equivalent to running the 2B model on 20 separate inputs sequentially (since candidates are evaluated in order until one exceeds γ). While the paper notes that GUI-Actor's candidate generation requires "only about 5% of the computation during inference" compared to Aguvis with verifier (Figure 7), this comparison is to an even more expensive baseline. The absolute cost — a 7B backbone VLM's forward pass plus potentially 20× 2B verifier forward passes — is not benchmarked in terms of milliseconds, frames-per-second, or FLOPs. A practitioner deciding whether to deploy the verifier has no quantitative basis for estimating the latency penalty.

Furthermore, the verifier requires training an entirely separate VLM on 730,000 synthetically-constructed examples (Section 4, Data & Training). This training cost — fine-tuning a 2B model on a large vision-language dataset — is not included in the paper's training efficiency analysis or data efficiency claims. The verifier is not a lightweight classifier head; it is a full VLM that must be fine-tuned on a domain-specific verification dataset.

What evidence exists in the paper. Figure 7 reports the relative compute comparison with Aguvis (GUI-Actor uses ~5% of the computation) but does not report absolute latency, FLOPs, or the average number of verifier calls per benchmark. The confidence threshold γ is reported (0.95 for ScreenSpot-Pro, 0.8 for ScreenSpot and ScreenSpot-v2) without an ablation showing how this threshold affects the speed-accuracy tradeoff. Table 9 (Verifier Self-Aggregation) adds additional verifier calls through multi-scale evaluation, further increasing the cost, but reports only accuracy.

Mitigation status. The paper acknowledges the cost implicitly through its design choices — the early termination mechanism and the per-benchmark threshold adaptation are both attempts to manage it — but does not treat the verifier's computational overhead as a limitation to be systematically characterized. The paper presents the verifier's performance benefits (Tables 1, 2, 3, 5) without a corresponding analysis of the cost-benefit tradeoff. The verifier self-aggregation extension (Table 9) acknowledges the need for "more robust verifiers in the future" that do not require multiple forward passes, but this is framed as a future improvement rather than a current limitation.

---

Single Backbone Family, Single Task Paradigm

The assumption or constraint. All experiments in the paper use Qwen2-VL or Qwen2.5-VL as the backbone VLM — both members of the same model family developed by the same organization (Alibaba). The training data, evaluation benchmarks, and the verifier are all built around pixel-level element grounding in 2D GUI screenshots following pyautogui-style action syntax. The paper does not evaluate on other VLM families (e.g., LLaVA, InternVL, Phi-Vision), on 3D or augmented-reality interfaces, or on action types beyond point-and-click (e.g., drag-and-drop trajectories, gesture paths, scroll regions).

The consequence. The claim that GUI-Actor's architectural advantages (attention-based grounding, multi-patch supervision) are general properties of the coordinate-free approach cannot be separated from the specific VLM backbone used. If Qwen2-VL's patch features are particularly well-suited to attention-based grounding — perhaps due to the specific resolution strategy, training data, or architectural details — then the performance advantages may not transfer to other VLM families. The paper does demonstrate transfer from Qwen2-VL to Qwen2.5-VL (Table 4), but these share the same architectural lineage. An evaluation on a VLM with substantially different visual encoding (e.g., one using a ViT with different patch size, a CNN-based encoder, or dynamic patch merging) would be needed to establish generality.

Similarly, the task scope is limited to single-step element identification: given an instruction, locate the target element. This is the foundational primitive for GUI grounding, but real GUI agents must also perform multi-step sequences where grounding decisions are interleaved with state changes and error recovery. The OS-World-W evaluation (Table 10) tests this to some extent (multi-step tasks with a GPT-4o planner), but the low absolute success rates (10–12%) and small sample size (49 tasks) mean this evaluation primarily demonstrates feasibility rather than providing a reliable measurement of grounding quality in realistic multi-step settings. A grounding error at step 3 of a 10-step task typically causes task failure, so the relationship between per-step grounding accuracy and end-to-end task success is highly nonlinear and sensitive to the specific task distribution.

What evidence exists in the paper. The Qwen2-VL and Qwen2.5-VL results (Tables 1, 4) show consistent advantages for GUI-Actor over coordinate-based baselines using the same backbone, and the Qwen2.5-VL results demonstrate cross-generation transfer. But no other VLM family is tested. The OS-World-W evaluation (Table 10) shows GUI-Actor at 12.2% vs. Aguvis at 4.0%, but with only 49 tasks and an external planner (GPT-4o), this difference — while large in relative terms — rests on a small absolute number of task completions.

Mitigation status. Not addressed. The paper does not discuss the generalizability of its findings to other VLM families or to task paradigms beyond single-step grounding. The choice of Qwen2-VL is practical (it is widely used, open-source, and enables fair comparison with existing work) but the paper does not characterize the extent to which conclusions depend on this choice.

---

The Pretraining-vs-Adaptation Tradeoff Is Measured Only Indirectly

The assumption or constraint. The paper's GUI-Actor-LiteTrain variant (Section 5, Table 5) is positioned as evidence that the backbone VLM can be augmented with GUI grounding capabilities "without compromising its general-purpose strengths." The evidence for this claim is the grounding accuracy achieved by LiteTrain (24.7 on ScreenSpot-Pro without verifier, 37.8 with verifier) relative to unmodified Qwen2-VL-7B (12.1). However, the paper does not actually measure whether the backbone VLM's general-purpose capabilities are preserved after LiteTrain fine-tuning. There is no evaluation on standard VLM benchmarks (MMBench, MME, ScienceQA, TextVQA, etc.) before and after training the action head.

The consequence. The claim of capability preservation is an inference from the fact that the backbone weights are frozen, not an empirical measurement. In practice, adding new special tokens to a VLM's vocabulary and training their embeddings on a domain-specific dataset can affect the model's behavior even if the backbone transformer weights are unchanged. The new token embeddings interact with the existing vocabulary through the transformer's attention and feedforward layers; the VLM's language generation distribution may shift because the <ACTOR> token now competes for attention with existing tokens in certain contexts. A practitioner who needs both strong GUI grounding and general VLM capabilities (e.g., for an agent that must both ground UI elements and engage in open-ended dialogue about screenshots) has no direct evidence about whether GUI-Actor-LiteTrain degrades the latter.

More broadly, the paper does not measure what the standard fully fine-tuned GUI-Actor loses relative to the unmodified backbone. Fine-tuning a VLM on ~1M domain-specific screenshots for 1 epoch likely shifts its visual and linguistic representations toward GUI-centric patterns. Whether this impacts performance on general vision-language tasks is unknown. The paper's argument that coordinate-based fine-tuning primarily teaches "coordinate mapping" rather than visual semantics (Section 5, LiteTrain interpretation) implies that GUI-Actor's fine-tuning might cause less capability degradation than coordinate-based methods, but this is a hypothesis, not a measured result.

What evidence exists in the paper. Table 5 compares LiteTrain to fully fine-tuned GUI-Actor on grounding benchmarks only. There is no general-purpose evaluation. The claim about capability preservation appears in the abstract ("effective grounding capabilities without compromising its general-purpose strengths"), the relevant discussion section, and the LiteTrain analysis, all without supporting measurements.

Mitigation status. Not addressed. The paper treats the LiteTrain variant's existence as sufficient evidence for capability preservation, but the inference logic (frozen weights → preserved capabilities) is incomplete without empirical verification. This is a gap that could be closed with a controlled evaluation on standard VLM benchmarks.

---

Hard Problems Remain Unsolved, and No Mechanism Exists for Breaking Through

The assumption or constraint. Across all benchmarks, there exists a subset of problems where GUI-Actor fundamentally fails — the attention map does not highlight the correct region, or the verifier cannot identify the correct candidate even when it is in the candidate pool. On ScreenSpot-Pro, the best model (GUI-Actor-7B with verifier) achieves 44.6, meaning it fails on 55.4% of examples. On OS-World-W, all methods achieve 10–12% task success rates. The paper does not provide a systematic analysis of why these failures occur, what they have in common, or whether they are addressable through architectural improvements versus being fundamental limitations of the patch-level attention approach.

The consequence. A practitioner cannot predict whether GUI-Actor will work for their specific use case. If failures are concentrated on particular element types (small icons, text-heavy elements, occluded targets, elements with unusual aspect ratios), that suggests targeted improvements to the action head or verifier. If failures are randomly distributed, that suggests a more fundamental performance ceiling. The paper's qualitative examples (Figure 5, Appendix C) show successful cases; failure cases are not systematically analyzed. Without understanding the failure modes, a practitioner has no basis for estimating the model's reliability in their domain or for implementing fallback strategies when grounding fails.

This limitation is particularly consequential because GUI-Actor, unlike generation-based methods, has no natural "retry with different parameters" mechanism for failed groundings. In coordinate generation, if a predicted coordinate is wrong, the model can be resampled with higher temperature or a different random seed. In GUI-Actor, the attention map is deterministic (given the same input, the same attention distribution is produced), and the candidate pool is fixed. If the correct region is not among the top-20 patches, no amount of verifier re-evaluation will recover it — and there is no mechanism for the model to "look again" or adjust its attention based on verifier feedback.

What evidence exists in the paper. The ScreenSpot-Pro accuracy of 44.6 (Table 1) directly implies a 55.4% failure rate, but no breakdown by failure cause is provided. The low OS-World-W success rates (Table 10) are acknowledged as reflecting task difficulty but not decomposed into grounding errors vs. planning errors vs. execution errors. The Hit@k analysis (Figure 4a) shows that even at k=3, a substantial fraction of targets are not captured, meaning the correct region is not among the top attention patches for many queries. The paper does not analyze what these unfound targets look like.

Mitigation status. The paper acknowledges the patch resolution limitation (Appendix A) but does not frame the broader failure-to-ground problem as a limitation requiring systematic analysis. The verifier self-aggregation extension (Table 9) addresses verifier uncertainty but not the case where the correct region is absent from the candidate pool entirely. There is no proposal for iterative refinement or attention map adjustment based on verifier feedback.

---

Training Data Overlap Risk and Benchmark Contamination

The assumption or constraint. The paper constructs its training data from several public GUI datasets (OS-Atlas, Wave-UI, and others, totaling ~1M screenshots, Table 7). It explicitly notes that "we exclude samples from Wave-UI that overlap with downstream task test sets" (Appendix D). However, the other datasets — particularly OS-Atlas — may contain screenshots or interface layouts that overlap with ScreenSpot and ScreenSpot-v2, which are constructed from publicly sourced GUI screenshots. The paper does not explicitly describe any deduplication or overlap-checking procedure between the non-Wave-UI training data and the test benchmarks.

The consequence. If the training data contains screenshots that also appear in ScreenSpot or ScreenSpot-v2 (even with different queries or bounding box annotations), the in-distribution benchmark numbers (77.0 on ScreenSpot, 81.2 on ScreenSpot-v2 for GUI-Actor-7B) may be inflated by memorization of specific interface layouts rather than reflecting generalizable grounding capability. This is the standard data contamination concern for benchmark evaluation, and it is particularly relevant for GUI grounding, where screenshots of popular applications (e.g., Google Calendar, YouTube, common settings pages) are likely to appear in multiple independently collected datasets. The paper's relatively larger advantage on ScreenSpot-Pro (a benchmark explicitly designed to avoid overlap with standard training data by using professional applications) compared to ScreenSpot (40.7 vs. Aguvis's 29.5, an 11.2-point gap, vs. 77.0 vs. Aguvis's 73.7, a 3.3-point gap) is consistent with this concern — GUI-Actor's advantage is largest on the genuinely out-of-distribution benchmark. However, this pattern is also consistent with the architectural benefits genuinely improving generalization, so the interpretation is ambiguous without explicit overlap analysis.

What evidence exists in the paper. The paper explicitly mentions excluding Wave-UI samples overlapping with test sets. It does not describe a similar procedure for other datasets. Table 7 lists the training data sources but does not report deduplication statistics. The ScreenSpot-Pro benchmark is explicitly framed as out-of-distribution, mitigating the concern for the paper's strongest results, but the ScreenSpot and ScreenSpot-v2 results (which show smaller but consistent gains) are more susceptible.

Mitigation status. Partially addressed for Wave-UI; not addressed for other training datasets. The paper's reliance on ScreenSpot-Pro as the primary evidence for generalization capabilities partially mitigates the concern, since ScreenSpot-Pro targets professional applications that are extremely unlikely to appear in general GUI training datasets. However, the ScreenSpot and ScreenSpot-v2 numbers — which contribute to the overall claim of "outperforms prior state-of-the-art methods across multiple benchmarks" — would benefit from a more rigorous deduplication protocol or, at minimum, a discussion of overlap risk.

7. Implications and Future Directions

How This Work Changes the Landscape

GUI-Actor does not merely propose a new grounding method — it reframes what the grounding problem is. The dominant paradigm in the field has treated GUI visual grounding as a text generation problem: the model produces coordinate tokens autoregressively, and the spatial output is an epiphenomenon of language modeling. This framing is so deeply ingrained that the community has largely accepted its associated pathologies — weak spatial alignment, ambiguous supervision, resolution brittleness — as inherent difficulty rather than as artifacts of representational mismatch. GUI-Actor's core move is to argue that these pathologies are not properties of grounding but of the coordinate-as-language-token bottleneck. By shifting the output modality from a sequence of numeric tokens to a spatial attention distribution, the method dissolves all three limitations simultaneously: spatial alignment becomes architecturally enforced (the attention distribution is the spatial output), supervision ambiguity is resolved by training against region-level targets rather than point targets, and the granularity mismatch disappears because the model operates at the vision backbone's native patch resolution.

This reframing changes the research landscape in several concrete ways:

It makes spatial inductive bias a first-class design consideration for VLM adaptation. Prior work on adapting VLMs for spatial tasks (grounding, detection, segmentation) has largely taken one of two approaches: either generate coordinates as text (simple, compatible with any VLM, but spatially naive) or add entirely separate detection heads that bypass the language pathway (architecturally complex, risks degrading general capabilities). GUI-Actor demonstrates a third path: the <ACTOR> token mechanism provides a lightweight, modular interface — a single vector extracted from the VLM's autoregressive generation — that connects language understanding to spatial output without modifying the VLM's core representations. This is not just an implementation detail; it is a design pattern that could generalize to any task where a VLM needs to produce dense spatial outputs informed by language reasoning (pointing to parts of an image during visual QA, identifying grasp points in robot manipulation, selecting edit regions in image generation interfaces). The key property that enables this pattern is the existence of a contextualized hidden state that encodes task-relevant information at a specific position in the autoregressive sequence — the <ACTOR> token — which can serve as a query for spatial attention. Identifying analogous "query points" in other task formulations would extend this pattern to new domains.

It reconciles conflicting narratives about whether verification helps for spatial tasks. The field has received mixed signals about verifier-based approaches for visual tasks. On one hand, self-consistency and process reward models have been transformative for language reasoning (Cobbe et al., 2021). On the other hand, applying these ideas to coordinate generation has been computationally prohibitive: resampling coordinate-based models produces nearly identical outputs (Figure 4a), meaning verification requires many expensive forward passes to get any diversity, and even then the gains are marginal (Figure 7: Aguvis + verifier requires 21 passes and still underperforms single-pass GUI-Actor). GUI-Actor shows that this failure is not about verification per se — it is about the diversity of the candidate pool. When the architecture naturally produces genuinely distinct spatial proposals in a single forward pass, verification becomes both cheap and effective. This finding redirects the conversation from "do verifiers help for visual grounding?" to "how do we design models that produce verifiable proposal distributions?" and places candidate diversity — not verifier quality — as the primary bottleneck for verification-based spatial reasoning.

It shifts the bottleneck narrative from "more data and larger models" to "better spatial representations." The paper's Figure 1 (left) is a pointed visual argument: UI-TARS-72B, trained on massive proprietary datasets with an elaborate multi-stage pipeline, reaches 38.1 on ScreenSpot-Pro; GUI-Actor-7B, trained on ~1M public screenshots with standard supervised fine-tuning, reaches 40.7 and 44.6 with the verifier. Scaling parameters and data — the default playbook for improving deep learning systems — provides diminishing returns when the output representation is fundamentally mismatched to the task. GUI-Actor's 7B model outperforming a 72B coordinate-generation model by ~6.5 points is strong evidence that representational efficiency (choosing the right output space) can substitute for scale. This does not mean scale is unimportant — the gap between GUI-Actor-7B (44.6) and GPT-4o (50.9) likely reflects genuine capability differences that scale could address — but it means that for open-source, academic-scale research, architectural innovation can close much of the gap with industrial-scale systems. The implication for resource allocation is clear: improving spatial grounding mechanisms may have a higher return on investment than scaling parameters or curating larger training sets.

It establishes ScreenSpot-Pro as a de facto standard for measuring generalization, not just accuracy. The paper's analysis of training dynamics (Figure 3c) reveals a pattern that should concern anyone building GUI agents: coordinate-based models overfit to training distributions, with accuracy on out-of-distribution benchmarks peaking early and then declining as training continues. This pattern is invisible if one only evaluates on in-distribution benchmarks (ScreenSpot, ScreenSpot-v2), where models continue to improve or plateau. GUI-Actor's training curve — an initial dip followed by recovery and stabilization — represents a qualitatively different generalization behavior. The paper's emphasis on ScreenSpot-Pro as the primary evaluation benchmark, and its explicit framing of it as an out-of-distribution test, pushes the field toward a standard that measures what matters for deployment: robustness to unfamiliar interfaces. A paper that reports strong ScreenSpot numbers without corresponding ScreenSpot-Pro results is, by this standard, providing an incomplete picture of its method's practical value.

It makes certain research directions less attractive. The paper's ablation showing that bounding-box supervision without architectural mechanisms does not improve over point supervision (Table 6, Aguvis-7B bbox sup. at 26.4 vs. point sup. at 29.5 on ScreenSpot-Pro) is a negative result with direct implications for research prioritization. Efforts to improve coordinate-based grounding through richer supervision formats — predicting bounding boxes, segmentation masks, keypoint heatmaps — are unlikely to succeed unless the architecture provides mechanisms to connect those outputs to visual features. Simply adding more spatial information to the text output does not help if the model cannot use it. This finding suggests that research on better coordinate representations (different normalization schemes, relative vs. absolute coordinates, coordinate compression techniques) is likely a local optimum — the gains are bounded by the fundamental representational mismatch, not by the specific coordinate format.

Follow-Up Research This Work Enables

Systematic failure mode analysis of attention-based grounding on ScreenSpot-Pro. GUI-Actor-7B with verifier fails on approximately 55% of ScreenSpot-Pro examples (Table 1: 44.6 accuracy). Understanding why is the most direct path to improvement. A strong follow-up would annotate a random sample of failures (200–300 examples) into categories: (a) correct patch attended but verifier selected wrong candidate, (b) attention map missed the target entirely, (c) target element smaller than one patch (~28×28 pixels), (d) instruction ambiguous or underspecified, (e) visual occlusion or unusual rendering. For each category, report what fraction of failures it accounts for. If category (c) dominates, the bottleneck is the vision backbone's patch resolution, and follow-up work should focus on higher-resolution encoders or offset prediction. If category (b) dominates, the bottleneck is the action head's ability to align language semantics with visual features, and follow-up work should explore stronger language-vision fusion before attention computation. If category (a) dominates, the bottleneck is verifier calibration, and follow-up should explore better verifier training (adversarial negatives, larger verifier models, multi-image input rather than single-crop). Without this taxonomy, any proposed improvement is speculative. The paper's existing qualitative attention maps (Figure 5, Appendix C) show only successes — a failure counterpart is essential.

Does multi-patch supervision matter independently of the attention mechanism? The paper claims that multi-patch supervision addresses ambiguous supervision targets, but the evidence is indirect — the ablation showing Aguvis with bounding-box supervision does not improve over point supervision (Table 6) suggests that richer targets help only when the architecture supports them, but does not isolate whether multi-patch supervision is necessary given the attention-based architecture. A controlled experiment would train GUI-Actor variants with (a) multi-patch supervision as described, (b) single-patch supervision (only the center patch of the bounding box labeled positive), and (c) a soft variant where patches are weighted by overlap fraction with the bounding box. If (b) performs similarly to (a), then the attention mechanism alone drives the gains, and the multi-patch justification is incorrect — this would be an important negative result for guiding future architecture design. If (a) substantially outperforms (b), then multi-patch supervision is genuinely load-bearing, and understanding why (reduced overfitting? better calibration? more robust attention maps?) would inform supervision design for related dense prediction tasks.

Combining GUI-Actor with iterative refinement via verifier feedback. The paper's verifier operates as a one-shot selector: evaluate candidates, take the best one, stop. But the attention map and verifier scores provide richer information that could drive iterative improvement. Specifically: if the verifier rejects all top-K candidates (score below threshold), the system could use the rejected candidates as negative signals — masking out those patches, re-normalizing the attention map, and producing new candidates from previously lower-ranked patches. This is analogous to constrained decoding where already-explored regions are suppressed. Alternatively, the verifier's per-candidate scores could be used to compute a weighted combination of attention maps (soft selection rather than hard thresholding). A follow-up would implement these iterative refinement strategies, measure how many additional candidates need to be evaluated to recover from an initial miss, and report the accuracy vs. average-verifier-calls tradeoff curve — the key metric for deployment viability. The paper already provides the foundation: the verifier's score distribution across candidates can indicate whether the correct region is likely in the pool (high variance in scores) or absent entirely (all scores low and similar). Exploiting this signal for dynamic compute allocation — spending more verification effort on uncertain cases and less on confident ones — would extend the verifier from a fixed-cost component to an adaptive one.

Cross-backbone generalization: does GUI-Actor's advantage depend on Qwen's specific visual representations? The paper demonstrates transfer from Qwen2-VL to Qwen2.5-VL (Tables 1, 4), but these share architectural lineage. A rigorous test of generality would implement GUI-Actor on a substantially different VLM family — for example, InternVL2 (which uses a different dynamic resolution strategy and vision encoder), LLaVA-NeXT (which uses a different language-vision connector), or a model with a fundamentally different patch size (e.g., ViT-L/14 at 14×14 pixels vs. Qwen's 28×28). The key measurement is the relative advantage over coordinate-generation baselines using the same backbone: is GUI-Actor's gain consistently 3–11 points regardless of backbone, or does it depend on properties like patch size, feature dimensionality, or resolution handling? If the advantage transfers universally, the coordinate-free approach is a robust contribution that practitioners can adopt with any VLM. If the advantage is backbone-dependent, that would identify important boundary conditions — perhaps smaller patch sizes reduce the need for attention-based grounding because coordinate generation becomes more precise, or perhaps certain vision-encoder architectures produce patch features that are inherently more amenable to dot-product attention.

Extending the coordinate-free paradigm to non-click actions. GUI-Actor currently handles point-and-click actions (and implicitly, the start point of drag-and-drop). Real GUI interaction involves richer action spaces: drag-and-drop trajectories (sequence of points defining a path), scroll regions (axis-aligned rectangles rather than single points), text selection spans (two points defining start and end of a selection), and gesture paths (curved trajectories for touch interfaces). Each of these could be reformulated in a coordinate-free manner within the same architectural framework: for trajectories, the attention map could become a 2D distribution where the model predicts multiple <ACTOR> tokens sequentially (each attending to a waypoint on the path); for regions, the attention map could be trained to highlight all patches within the rectangular scroll area; for selections, two <ACTOR> tokens could attend to start and end patches respectively. A follow-up would implement one of these extensions (scroll regions being the simplest), construct appropriate training data from existing GUI datasets (scrollable regions are often annotated in DOM-based datasets), and measure whether the same architectural advantages (sample efficiency, out-of-distribution generalization) transfer to spatial outputs beyond single points. A negative result — finding that coordinate-free is only beneficial for point prediction — would delineate the method's scope and motivate hybrid approaches.

Training a difficulty-aware compute scheduler for the verifier. The paper uses fixed confidence thresholds γ that differ by benchmark (0.95 for ScreenSpot-Pro, 0.8 for ScreenSpot, ScreenSpot-v2). This is a coarse, hand-tuned adaptation that a learned policy could improve. A concrete follow-up would train a lightweight classifier (a small MLP on top of the backbone VLM's pooled features, or using the <ACTOR> token's hidden state) to predict, before any verifier calls, how many candidates the verifier will need to evaluate to find a passing one. This is essentially a verification difficulty estimator: given the attention map's entropy (is attention concentrated on one patch or spread across many?), the top patch's attention weight (is the model confident or uncertain?), and possibly the instruction's complexity, predict the expected number of verifier calls. The training signal is derived from actual verifier runs on the training set (number of candidates evaluated before exceeding γ for each example). At inference time, this scheduler could allocate verification budget proportionally to estimated difficulty: easy cases (concentrated attention, high top score) get 1–2 verifier calls, hard cases get up to 20. The evaluation metric would be accuracy vs. average verifier calls, with the goal of matching full-verification accuracy at reduced average cost. This directly parallels the "compute-optimal" theme from test-time scaling work, applied to the verifier rather than the search strategy, and would transform the verifier from a fixed overhead into an adaptive resource.

Practical Applications and Downstream Use Cases

Desktop agent frameworks requiring robust grounding across diverse applications. The paper's OS-World-W evaluation (Table 10) demonstrates GUI-Actor operating as the grounding component within a larger agent architecture (GPT-4o as planner, GUI-Actor as action executor). This is a realistic deployment pattern: large commercial VLMs provide high-level reasoning and planning, while smaller, specialized grounding models handle the precise spatial decisions. The key value proposition for such frameworks is GUI-Actor's robustness to unfamiliar interfaces (ScreenSpot-Pro results: 40.7–44.6 vs. Aguvis's 26.4–29.5 for backbone-matched training) combined with its efficiency (single forward pass for candidate generation, vs. 21 passes for Aguvis with comparable verification). A desktop agent framework like UFO (Zhang et al., 2024a) or OS-Copilot (Wu et al., 2024a) that currently uses a general-purpose VLM for both planning and grounding could decouple these responsibilities: keep the existing VLM for understanding screenshots and generating action plans, but replace the coordinate-generation step with GUI-Actor for the actual spatial decisions. The paper provides evidence that this would improve grounding accuracy on professional software (the ~11-point gap on ScreenSpot-Pro) while likely reducing per-step latency (since GUI-Actor's grounding is a single forward pass rather than autoregressive coordinate generation). The main integration cost is training the action head on the target domain's screenshots — the paper's data recipe (Table 7) and training procedure (Section 3.4, Phases 1 and 2) provide a concrete template.

Automated GUI testing and data generation pipelines. Many organizations maintain automated testing infrastructure that navigates desktop or web applications to verify functionality. These systems often rely on brittle UI selectors (DOM IDs, accessibility labels, XPaths) that break when the interface changes. Replacing coordinate-based selectors with a vision-based grounding model would make tests robust to layout changes, but only if the grounding model itself generalizes across interface versions. The paper's ScreenSpot-Pro results are directly relevant: professional software interfaces change appearance across versions, and a grounding model that overfits to training layouts (as Aguvis does in Figure 3c) would require retraining for each version. GUI-Actor's improved out-of-distribution generalization — combined with the LiteTrain variant that can be trained on only ~100M parameters without modifying the backbone — offers a practical path: deploy a pre-trained GUI-Actor-LiteTrain model as the grounding component, and for each new application version, fine-tune only the action head on a small set of screenshots from the new interface. The verifier could serve double duty: during testing, it selects the correct candidate; during training data generation, it could automatically verify proposed grounding annotations for new screenshots, reducing the need for manual bounding-box labeling.

On-device mobile agents with limited compute budgets. The paper's 2B model results (GUI-Actor-2B: 35.2 on ScreenSpot-Pro without verifier, Table 1) are notable because they surpass several 7B coordinate-generation models (Aguvis-7B point sup.: 29.5, UI-TARS-7B: 25.7). A 2B VLM is within the deployment envelope of modern flagship smartphones (which increasingly ship with dedicated neural processing units capable of running billion-parameter models locally). The use case: a mobile assistant that observes the screen and grounds actions entirely on-device, without network latency or privacy concerns from cloud-based processing. GUI-Actor's single-pass grounding (no multiple samples needed) and the verifier's optional nature (can be disabled for latency-critical interactions) are key enablers. The LiteTrain variant is particularly relevant here: a mobile deployment could freeze the backbone VLM (preserving its general-purpose assistant capabilities) and add only the ~20M-parameter action head for grounding, resulting in a single model that both conducts dialogue and interacts with the UI. The paper's 2B-ScreenSpot-Pro results provide a performance baseline; the main missing piece for deployment is latency benchmarking on actual mobile hardware, which the paper does not provide.

When to Prefer This Method

The paper explicitly positions GUI-Actor against coordinate-generation methods (Aguvis, UI-TARS, etc.) and provides controlled comparisons with matched backbones and training data. Based on the paper's evidence, the decision conditions are:

• Prefer GUI-Actor over coordinate-generation methods when:

  • The deployment environment involves diverse or unfamiliar interfaces (professional software, multi-platform applications, user-generated content) where out-of-distribution generalization matters — ScreenSpot-Pro results show a ~11-point advantage over coordinate baselines with matched training.
  • Training data is limited to publicly available datasets (~1M screenshots) — Figure 3 shows GUI-Actor reaches final accuracy with ~60% of the training data required by coordinate baselines.
  • Inference efficiency matters for the grounding step specifically — GUI-Actor produces diverse candidates in a single forward pass vs. ~21 passes for coordinate methods with comparable verification (Figure 7).
  • The backbone VLM's general-purpose capabilities must be preserved — GUI-Actor-LiteTrain (Table 5) adds grounding without modifying backbone weights, though the paper does not directly measure capability preservation.
  • The target elements are typically larger than ~28×28 pixels (the standard patch size of Qwen2-VL) — Appendix A acknowledges degraded performance on very small interface elements.

• Coordinate-generation methods may still be preferable when:

  • The deployment environment involves only standard, training-distribution-similar interfaces where the generalization gap is small (ScreenSpot gap: ~3 points vs. ~11 points on ScreenSpot-Pro).
  • The action space includes complex structured outputs beyond point-and-click (e.g., generating executable scripts with embedded coordinates for multiple interdependent actions) where the autoregressive language pathway provides compositionality benefits that a single attention map cannot capture.
  • The available VLM backbone does not expose patch-level features or has incompatible resolution handling, making the action head architecturally infeasible without modifying the backbone.
  • Very small target elements (<28 pixels) dominate the use case — in this regime, coordinate generation's sub-patch precision (learned through the language head) may outperform patch-level attention despite the granularity mismatch.

The paper does not provide evidence for choosing between GUI-Actor and closed-source commercial models (GPT-4o, Claude) — the comparison in Table 1 shows GPT-4o still leads on ScreenSpot-Pro (50.9 vs. 44.6), but the cost, latency, and privacy tradeoffs are context-dependent and not quantified in the paper.