UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning

ArXiv: 2510.20286

🎯 Pitch

Existing GUI grounding models are trained on narrow instruction styles, but a manual audit reveals that 23.3% of instructions in common datasets are flawed, and simply exposing a model to diverse instruction perspectives can yield up to a 76% relative accuracy boost. This paper treats instructions as selectable reasoning pathways rather than static input, training models to choose and compose analytical perspectives—a strategy that sets new state-of-the-art on five grounding benchmarks while also driving a 74.1% success rate on live Android tasks.

1. Executive Summary

This paper proposes the Instruction-as-Reasoning paradigm, which reframes natural-language GUI grounding instructions from static inputs into dynamic reasoning pathways that the model can selectively deploy. The authors first conduct a systematic manual inspection of 1,909 samples from existing grounding datasets—OS-Atlas, AMEX, and Widget Captioning—revealing a 23.3% flaw rate and demonstrating that inference-time exploitation of instruction diversity (appearance, functionality, location, intent perspectives) yields up to a 76% relative performance improvement. Building on these findings, they introduce a two-stage SFT+GRPO framework that first teaches Qwen2.5-VL models to generate diverse instruction perspectives as explicit reasoning chains, then uses reinforcement learning to optimize which analytical pathway the model selects for each GUI scenario. The resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art grounding accuracy across five benchmarks—87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2—while the 7B model drives a 74.1% task success rate on the AndroidWorld online agent benchmark when paired with GPT-5 as planner, establishing that structured, perspective-based reasoning enhances rather than hinders grounding performance only when the reasoning format is explicitly taught during supervised fine-tuning rather than left as free-form exploration during RL.

2. Context and Motivation

The Core Problem: Instructions Are the Neglected Variable in GUI Grounding

GUI grounding—the task of mapping a natural language instruction to the correct (x, y) coordinate of an actionable UI element on a screenshot—is foundational to all GUI agents. Every click, tap, or long-press a GUI agent executes begins with a grounding decision. The quality of that decision depends on two inputs: the screenshot (what the interface looks like) and the instruction (what the user wants clicked). The field has devoted enormous attention to better visual encoders, more sophisticated coordinate prediction heads, and stronger pretraining recipes for the visual modality. The instruction, by contrast, has been treated as a static, transparent carrier of user intent—a string you tokenize and feed into the model, with no further analysis of its role.

This paper's central claim is that this neglect is a significant blind spot. The authors argue that the natural language instruction is not a passive input but an active variable that fundamentally shapes grounding difficulty and accuracy. Through careful empirical investigation (detailed in Section 2 of the original paper), they establish that two properties of instructions—their analytical perspective and their quality—have been systematically underestimated in prior work.

Why This Matters: Real-World Deployment and Capability Ceilings

The practical stakes are high. GUI agents are increasingly deployed in production settings: virtual assistants that navigate mobile apps, robotic process automation systems that interact with desktop software, and accessibility tools that translate screen content into actionable commands. In all these scenarios, the instruction is the only conduit for expressing user intent. If the instruction is ambiguous ("click the button"), refers to nothing visible on screen, or forces the model into a suboptimal analytical lens, the grounding failure cascades into an agent failure—the wrong element is clicked, the task derails, and the user's trust erodes.

The problem is also theoretically significant. The grounding task sits at the intersection of vision-language understanding, spatial reasoning, and goal inference. An instruction like "close the file manager window" forces the model to:

• Identify which of potentially dozens of UI elements qualifies as "the file manager window" (semantic understanding)
• Determine where the close control for that window is located (spatial reasoning)
• Distinguish between the minimize, maximize, and close buttons if multiple similar controls exist (visual discrimination)
• Apply a common-sense understanding of window management conventions (world knowledge)

Different instruction perspectives—describing the target by its appearance ("click the red X"), its function ("close the file manager"), its location ("the button in the top-right corner"), or the user's goal ("get rid of this screen")—emphasize different facets of this reasoning chain. Humans fluidly switch between these perspectives depending on context, choosing the one that most efficiently resolves ambiguity. The paper's core contention is that current models lack this adaptive capacity because they are trained on datasets where each instruction is rendered in a single, fixed style.

Prior Approaches: Treating Instructions as Fixed Inputs

The grounding literature has evolved rapidly, but the treatment of instructions has remained remarkably static across several research generations:

SFT-based grounding models (OS-Atlas by Wu et al., 2024a; Aguvis by Xu et al., 2025; Uground by Gou et al., 2025; Aria-UI by Yang et al., 2024b; JEDI by Xie et al., 2025) train models to directly predict coordinates from (screenshot, instruction) pairs. The instruction is treated as an immutable input token sequence. These models are data-hungry—they need thousands of (screenshot, instruction, coordinate) triples—but the data they consume inherits whatever instruction style the dataset creators happened to use. The models learn a brittle mapping from that specific style to coordinates, with no mechanism for adapting to alternative phrasings of the same intent.

RL-based grounding models (GUI-R1 by Luo et al., 2025; GUI-Actor by Wu et al., 2025; GTA1 by Yang et al., 2025) have pushed performance further by framing grounding as a reinforcement learning problem, using point-in-box accuracy as a reward signal. These methods represent a significant advance in optimization strategy but still treat the instruction as a fixed given. The RL process optimizes how the model processes a given instruction, not which instruction it should reason about.

Instruction augmentation approaches represent the closest prior work to what this paper does, but with critical limitations. Aria-UI (Yang et al., 2024b) and Phi-Ground (Zhang et al., 2025) use advanced MLLMs to paraphrase original instructions into varying styles at the input level—effectively doing data augmentation on the instruction string. The paper identifies three shortcomings of this approach: (1) it treats paraphrased instructions as independent training samples rather than as alternative reasoning pathways for the same sample; (2) it lacks any analysis of why instruction diversity matters or which perspectives work when; and critically, (3) it fails to produce consistent, significant performance gains—the augmentation helps in some cases but not others, with no framework for understanding when or why.

The Free-Form Reasoning Failure Mode

A particularly instructive failure in prior work involves attempts to incorporate free-form reasoning (FFR) into grounding. Several recent methods (GUI-G1 by Zhou et al., 2025; GTA1 by Yang et al., 2025; GUI-G2 by Tang et al., 2025; UI-R1 by Lu et al., 2025) experimented with letting the model generate unstructured "thinking" tokens before predicting coordinates during RL training. The hope was that, as with mathematical reasoning tasks (DeepSeek-R1 by Guo et al., 2025), the model would discover useful reasoning strategies through exploration and reward shaping.

The results were uniformly negative. As the paper documents in Table 8:

• Applying FFR during RL decreased UI-Tars-1.5-7B's ScreenSpot-Pro accuracy by 6.4% (from 50.1% to 46.9%)
• Applying FFR during RL produced zero gain for Qwen2.5-VL-7B (36.4% in both conditions)

This failure is not a footnote—it reveals something fundamental about GUI grounding. Unlike mathematical reasoning, where the space of valid reasoning steps is constrained by formal logic and where intermediate computations can be verified, GUI grounding's reasoning space is poorly structured. A model exploring "what should I think about?" without guidance will generate reasoning that meanders, introduces spurious correlations, or worse—becomes a source of noise that corrupts the coordinate prediction. The paper positions this finding as a central motivation: reasoning can help grounding, but only if the reasoning format is explicitly taught during supervised fine-tuning rather than left as an undirected exploration problem.

Some works (InfiGUI-G1 by Liu et al., 2025d; InfiGUI-R1 by Liu et al., 2025c; GUI-R1 by Luo et al., 2025) did incorporate reasoning components into their training pipelines. However, the paper notes that these works did not provide ablation studies isolating the reasoning component's contribution. Without such studies, it is impossible to determine whether the reasoning helped, was neutral, or was actively harmful but masked by other architectural improvements. GUI-R1's own observation—that model performance improved as the reward weight for the "thinking" format was decreased—is a telling signal that unstructured reasoning was more burden than benefit.

The SFT+RL Policy Collapse Problem

A second technical challenge that motivates this paper's approach is the well-documented but poorly understood phenomenon of policy collapse when applying RL after SFT in grounding tasks. The standard pipeline—fine-tune on (screenshot, instruction, coordinate) triples, then optimize further with RL—seems straightforward but frequently fails. The paper's Table 9 quantifies this:

• Standard SFT on Qwen2.5-VL-7B achieves 37.0% on ScreenSpot-Pro, but subsequent RL training degrades performance to 34.9% (a 5.7% relative drop)
• The same pattern holds for JEDI-7B: 39.5% zero-shot, dropping to 34.5% after RL (a 12.7% relative drop)

The mechanism underlying this collapse becomes clear when examining what standard SFT produces. A model trained solely to output coordinates learns to generate highly uniform, low-diversity responses—essentially, the training objective pressures it toward memorizing the mapping rather than developing flexible processing strategies. When RL training begins, this uniform policy provides no exploratory diversity. All rollouts look similar, the advantage estimates are noisy, and the policy update either does nothing useful or shifts the model in a random direction that happens to increase reward on a few samples at the cost of general performance.

Phi-Ground (Zhang et al., 2025) independently noted this limitation, observing that "models fine-tuned via SFT using only coordinates as ground truths often exhibit highly uniform responses, leading to ineffective exploration and policy collapse in RL." This paper cites Phi-Ground's observation as corroborating evidence and positions its Instruction-as-Reasoning framework as a direct solution: by training the model during SFT to generate diverse instruction perspectives, the policy initialization is inherently more exploratory, producing distinct rollouts during RL and enabling stable optimization.

Data Quality: The Dirty Foundation

Beyond the strategic questions of instruction diversity and reasoning format lies a more basic problem: the instructions in existing grounding datasets are often wrong. The paper's manual inspection of 1,909 samples from OS-Atlas, AMEX, and Widget Captioning—three prominent, widely-used datasets—revealed that 23.3% of instructions contain substantive flaws. These flaws take two forms:

• Ambiguous match (4.4%): The instruction could refer to multiple UI elements on the screen. For example, "The red square indicates the active slide" might match several red-colored UI components, making the ground-truth association unreliable.
• Mismatch (18.9%): The instruction refers to something that does not exist on the screen at all. For example, "Check how many people have interacted with the article" when no such interaction count is visible. These are likely artifacts of automated dataset construction where captioning models hallucinate content or where instructions are paired with screenshots from different moments than when the instruction was valid.

This 23.3% error rate means that nearly one in four training samples teaches the model a wrong or ambiguous association. The impact is not theoretical—Figure 2c shows that training Qwen2.5-VL-7B on cleaned data produces consistent improvements of 2-7 percentage points across MMBench-GUI, ScreenSpot-Pro, and UI-I2E-Bench compared to training on the original noisy data. The paper summarizes this finding bluntly:

"flawed instruction data can significantly degrade downstream performance when used for training. [...] data cleaning is not optional niceties but necessary prerequisites for meaningful training."

This is not a claim that prior work was negligent—it is a claim about the inherent difficulty of constructing large-scale grounding datasets. Many datasets are built by scraping app interfaces and pairing screenshots with automatically generated descriptions. Without explicit verification that each description uniquely identifies exactly one element, noise accumulates. The paper's contribution here is not the observation that data is noisy (which is broadly true across many ML datasets) but rather the systematic quantification of that noise's prevalence and its demonstrable impact on downstream grounding accuracy.

Instruction Diversity: The Untapped Performance Reservoir

The paper's most striking preliminary finding is captured in Figure 2a. On ScreenSpot-Pro, the authors evaluated Qwen2.5-VL-7B zero-shot on instructions rewritten from four different analytical perspectives:

• Original instructions: 24.4% accuracy
• Appearance perspective ("click the red X"): 43.1%
• Function perspective ("close the file manager"): 35.2%
• Location perspective ("the button in the top-right corner"): 29.8%
• Intent perspective ("get rid of this screen"): 21.3%
• Optimal combination (per-sample oracle choosing the best perspective): 26.1%

Several insights emerge from these numbers. First, no single perspective dominates. Appearance achieves the highest average score (43.1%), but the 76% relative improvement in the "Combined" bar—representing an oracle that always picks the best perspective for each sample—reveals that there is enormous latent performance across perspectives that no single perspective captures. Different GUI scenarios call for different analytical lenses. A visually distinctive icon is best described by appearance; a standard control in a conventional location is best described by spatial relationships; a task requiring domain knowledge ("mute the system") is best described by function or intent.

Second, this performance is unlocked without any training. The model was not fine-tuned on these diverse instructions—it was simply prompted with them at inference time. This means the capability to process multiple instruction perspectives already exists in the pretrained model; what is missing is (a) training data that exposes the model to diverse phrasings of the same intent and (b) a mechanism for the model to select the optimal perspective based on context.

Third, the "Original" instruction set performs worse than several individual perspectives. This is not because the original instructions are poorly written per se, but because they represent a single, arbitrary style that may be suboptimal for many samples. The dataset creators made implicit choices about instruction perspective that are now baked into the training data, and models trained on this data inherit those choices' limitations.

How This Paper Positions Itself

The paper synthesizes these observations—data quality issues, instruction diversity's latent potential, free-form reasoning's failure, SFT+RL policy collapse—into a coherent research agenda:

1. Data foundation: Clean the existing data and systematically augment it with diverse instruction perspectives to create a training corpus specifically designed for multi-perspective reasoning.

2. Paradigm shift: Move from treating instructions as static inputs to treating them as dynamic reasoning pathways. Different instruction perspectives are not alternative phrasings of the same thing—they encode distinct analytical angles for solving the grounding task, and an intelligent system should actively select the best angle per instance.

3. Training framework: A two-stage approach where SFT teaches the model how to generate diverse perspectives as reasoning (preventing the policy collapse problem by instilling exploratory diversity), and RL teaches the model which perspective to choose when (optimizing the selection policy through reward feedback).

The paper's positioning relative to prior work is explicit and adversarial in the best sense: it identifies a neglected variable (instruction perspective), quantifies its importance through careful ablation, shows that naive attempts to exploit it (free-form reasoning) fail, and proposes a structured alternative that succeeds. The conceptual contribution—Instruction-as-Reasoning—is not a new model architecture or a new RL algorithm, but a reconceptualization of what an instruction is in the grounding pipeline, with concrete implications for data construction, training, and inference.

3. Technical Approach

3.1 Reader Orientation

This paper builds a training framework and data pipeline that teaches vision-language models to use diverse natural-language descriptions of UI elements as reasoning pathways for GUI grounding. The core problem is that existing grounding models treat instructions as fixed, single-style inputs—e.g., always describing a button by its function rather than its appearance—which ignores the reality that different GUI scenarios are best resolved through different analytical lenses. The solution's shape is a two-stage training protocol (SFT then RL) operating on purpose-built data that systematically augments each grounding instance with instructions from four distinct perspectives (appearance, function, location, intent), then trains the model first to generate these perspectives as intermediate reasoning and subsequently to select the optimal perspective per instance through reinforcement learning with a point-in-box reward.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components arranged in a pipeline:

1. Data Pre-processing Module — Takes raw grounding datasets (OS-Atlas, AMEX, Widget Captioning, Omniact, Android Control, AgentNet) and cleans their bounding box annotations using OmniParser V2 detection plus IoU-based filtering, producing a clean corpus of (screenshot, instruction, validated bounding box) triples.

2. Multi-Perspective Instruction Augmentation Module — Uses GPT-4.1 prompted with the screenshot (target element highlighted) to generate four new instruction variants per sample—appearance-based, function-based, spatial-based, and intent-based—followed by a verification step where GPT-4.1 confirms each generated instruction unambiguously maps to exactly one UI element, yielding a high-quality corpus where each grounding instance has 4–5 valid instruction perspectives.

3. Supervised Fine-Tuning Stage — Trains the base VLM (Qwen2.5-VL-7B or Qwen2.5-VL-32B) on approximately 283k instances for one epoch, where each training example explicitly shows the model a randomly chosen instruction perspective as the "reasoning" text followed by the ground-truth coordinate, teaching the model to generate perspective-based reasoning chains before predicting coordinates.

4. Reinforcement Learning Stage (GRPO) — Fine-tunes the SFT model on approximately 33k instances (expanded to ~100k by generating one sample per perspective) using Group Relative Policy Optimization, where the prompt simply asks the model to "think" without specifying which perspective to use, and the reward is 1 if the predicted coordinate falls inside the ground-truth bounding box and 0 otherwise, incentivizing the model to discover and select the most effective reasoning pathway per context.

5. Inference Module — At test time, the model receives a (screenshot, instruction) pair, generates an internal reasoning chain (which may involve a single perspective, a composition of multiple perspectives, or an entirely novel analytical angle), and outputs the predicted coordinate.

Information flows sequentially: raw datasets → cleaned annotations → augmented multi-perspective instructions → SFT training on perspective-conditioned reasoning → RL training on unconstrained perspective selection → deployed model that reasons before grounding.

3.3 Roadmap for the Deep Dive

• First, the formal task definition—what GUI grounding is, its input-output specification, and the point-in-box reward that drives RL—because this establishes the mathematical substrate for everything that follows.

• Second, the data pipeline (pre-processing and multi-perspective augmentation), since the quality and diversity of training data is the paper's foundational claim and the SFT+RL framework depends on having high-quality, perspective-rich examples.

• Third, the SFT stage in full detail, including the training objective, the construction of training instances with explicitly designated reasoning perspectives, and the mechanism by which SFT instills exploratory diversity to prevent later RL policy collapse.

• Fourth, the RL stage (GRPO), including the reward function, the advantage normalization scheme, the policy optimization objective, and how the open-ended "think" prompt enables emergent reasoning beyond the four predefined perspectives.

• Fifth, the design choices that connect these components—why SFT before RL, why structured perspectives rather than free-form reasoning, why point-in-box reward rather than alternatives, and how the two stages complement each other to produce models that can strategically select, compose, and invent reasoning pathways.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that GUI grounding instructions should be treated as dynamic reasoning pathways rather than static inputs, and that a two-stage SFT+RL framework operating on multi-perspective training data can teach models to select the optimal analytical lens for each grounding instance.

Task Definition: GUI Grounding as Point Prediction

The paper defines GUI grounding as a coordinate regression task with a specific input-output contract. Formally, given a GUI screenshot $S$ (an RGB image of arbitrary resolution, though in practice the model processes it at a fixed input size determined by the vision encoder) and a natural language instruction $I$ (a text string such as "click the close button"), the model $f$ must predict a single coordinate point:

$p = (x_p, y_p)$

where $x_p$ and $y_p$ are pixel coordinates indicating the center of the target UI element on the screenshot. This is a two-dimensional continuous regression problem embedded within a vision-language understanding task—the model must jointly comprehend the visual layout of the interface, parse the linguistic intent expressed in the instruction, and spatially localize the element that satisfies both constraints.

What it computes: the model consumes a multimodal input (image + text) and produces a spatial localization decision—a single $(x, y)$ point that represents the model's best estimate of where the user intends to interact.

Why this form: the point-prediction formulation is the standard interface for GUI agent action execution. When an agent decides to click a button, it needs a screen coordinate to send to the operating system. The alternative—predicting a bounding box and then computing its center—adds complexity without changing the fundamental prediction problem, since the final action is always a point. The paper's error metric reinforces this choice: accuracy is measured by whether the predicted point falls inside the ground-truth bounding box, making the box a tolerance region rather than a prediction target.

The formal grounding accuracy over a test set of $N$ samples is defined as:

$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(p_i \in b_i)$

where $p_i = (x_p, y_p)$ is the model's predicted point for sample $i$, $b_i = (x_l, y_l, x_r, y_r)$ is the ground-truth bounding box defined by its top-left and bottom-right corners, and $\mathbb{I}(\cdot)$ is the indicator function returning 1 if the point lies within the box and 0 otherwise. This metric treats the bounding box as a spatial tolerance—the prediction does not need to be exactly at the center; it only needs to land anywhere within the UI element's extent. This is a practical choice: clicking anywhere on a button typically activates it, so exact centering is unnecessary.

Data Pipeline: Cleaning and Multi-Perspective Augmentation

The data pipeline serves two objectives that Section 2 of the paper established as necessary prerequisites: (1) cleaning annotation noise from existing datasets, and (2) systematically augmenting each instance with diverse instruction perspectives to create a corpus designed for multi-perspective reasoning.

Data sources. The authors collect training data from several public datasets chosen for their coverage of diverse operating systems and interface types: OS-Atlas (Wu et al., 2024a), Omniact (Kapoor et al., 2024), Android Control (Li et al., 2024), AMEX (Chai et al., 2025), and AgentNet (Wang et al., 2025b). These datasets collectively span Windows, MacOS, Linux, iOS, Android, and Web interfaces, ensuring that the trained model encounters a wide variety of visual styles, layout conventions, and interaction patterns. The total number of raw samples is not explicitly stated, but after cleaning and augmentation, approximately 283k instances are used for SFT and 33k (expanded to ~100k via perspective replication) for RL.

Pre-processing: bounding box cleaning via OmniParser. The paper's manual analysis (Section 2.2) revealed that 23.3% of samples in major grounding datasets contain instruction flaws—either ambiguous matches (instruction could refer to multiple elements) or outright mismatches (instruction refers to nothing on screen). To address this, the pipeline first runs OmniParser V2 (Lu et al., 2024) on each screenshot to detect all UI elements and their bounding boxes. OmniParser V2 is a vision-only UI parsing model that identifies interactive elements (buttons, icons, text fields, etc.) and outputs their bounding boxes without requiring any platform-specific accessibility metadata. The pipeline then applies a simple Intersection-over-Union (IoU) based refinement: for each training sample, it computes the IoU between the original ground-truth bounding box and each OmniParser-detected box, and either refines the ground-truth box to better align with the detected element or filters out the sample entirely if no reasonable match exists. This step ensures that every instruction retained in the training corpus is associated with a reliable spatial anchor—a bounding box that corresponds to a real, detectable UI element.

This pre-processing step is the paper's answer to the "dirty foundation" problem identified in Section 2.2. Rather than accepting the 23.3% error rate as an unavoidable cost of large-scale data collection, the authors invest in cleaning as a prerequisite. The ablation in Figure 7b validates this investment: models trained on cleaned data outperform those trained on original data across multiple benchmarks.

Multi-perspective instruction augmentation: GPT-4.1 as a perspective generator. With clean bounding boxes established, the core of the data pipeline is the systematic generation of diverse instruction perspectives. The authors leverage GPT-4.1 (OpenAI, 2025a) as the generation engine, prompted with a carefully designed template (reproduced in full in Appendix A.1). For each training instance, GPT-4.1 receives:

1. The screenshot with the target UI element visually highlighted—the ground-truth bounding box is overlaid as a distinct marker (a red circle or rectangle) so the language model knows exactly which element to describe.
2. The original instruction from the dataset, which serves as a semantic anchor ensuring the generated variants express the same intent.

The prompt then instructs GPT-4.1 to generate four distinct types of instructions, one from each of the following analytical perspectives:

• Appearance-Based: A direct description of the element's visual characteristics—its text content, icon, color, shape, or combination thereof. The prompt explicitly requires that the description be "completely unique" by combining multiple features as needed. Example: "click the red X button in the top-right corner."

• Function-Based: A description of what the element does or what happens when the user interacts with it—its purpose or the immediate outcome of activation. Example: "close the file manager window."

• Spatial-Based: A description that identifies the element through its position relative to other prominent, easily identifiable UI landmarks. The prompt requires that "the described spatial relationship must lead to a unique location." Example: "the button to the right of the File menu."

• Goal-Based: A description of the user's ultimate objective or intent, requiring the system to infer which UI element fulfills that goal. Example: "get rid of this screen."

The prompt also includes an important safeguard: GPT-4.1 is explicitly told that the visual annotation (the red marker) is a ground-truth reference for the language model only, and the generated instructions must never refer to the annotation itself. This prevents the model from generating instructions like "click the element with the red circle on it"—a degenerate shortcut that would teach the grounding model nothing useful.

Why these four perspectives? The choice is not arbitrary. These perspectives correspond to the four major ways humans describe objects in context (Figure 3 of the paper illustrates this mapping with concrete examples). Appearance and location are perceptual—they rely on the model's ability to visually parse the screenshot and match descriptive features to pixels. Function and intent are semantic—they rely on the model's understanding of what UI elements do and what goals users typically pursue. Covering both perceptual and semantic perspectives ensures the training data exercises the full range of reasoning capabilities needed for robust grounding.

Verification: ensuring one-to-one mapping. A known failure mode of LLM-based data generation is hallucination—the model might generate an instruction that sounds plausible but does not actually correspond to a unique element on the screen, or worse, corresponds to a different element than the ground-truth target. To mitigate this, the pipeline includes a second GPT-4.1 call for verification (Appendix A.2 provides the full verification prompt). The verification prompt shows GPT-4.1 the screenshot with the ground-truth bounding box (marked with a red box) and the ground-truth center point (marked with a blue hollow circle), along with the generated instruction, and asks it to assess:

1. Instruction uniqueness: Does this instruction unambiguously describe exactly one element on the screen, or could it match multiple elements?
2. Bounding box appropriateness: Does the ground-truth box tightly enclose the target element without excessive empty space or cutting off parts of the element?

Only instructions that pass both checks are retained. The paper reports (Figure 7a) that this pipeline achieves a 93.5% precise match rate on a manually inspected sample of 1,542 generated instances—a dramatic improvement over the 76.7% rate in the original data and a reduction of the overall error rate from 23.3% to below 8%.

Scale and composition. The final SFT training corpus contains approximately 283k instances. Each original sample is represented multiple times with different instruction perspectives, ensuring the model encounters diverse phrasings of the same grounding intent during training. For the RL stage, a subset of 33k instances is used, but each instance is expanded by generating one sample per instruction perspective, yielding approximately 100k training samples. This expansion ensures the RL training data retains the perspective diversity that the SFT stage instilled.

Supervised Fine-Tuning Stage: Teaching Diverse Reasoning Pathways

The SFT stage is the first half of the two-stage training framework. Its purpose is to explicitly teach the model to use instruction perspectives as intermediate reasoning before predicting the grounding coordinate. This is not merely data augmentation—it is a deliberate training protocol designed to instill a specific cognitive behavior: look at the screenshot, rewrite the instruction into a chosen analytical perspective, reason within that perspective, and then output the coordinate.

Training instance construction. Each SFT training instance is constructed from the multi-perspective corpus as follows. Given a (screenshot $S$, original instruction $I$, ground-truth coordinate point $p_{\text{gt}}$) triple, the authors have 4–5 valid augmented instructions (one per perspective, plus the cleaned original). From this pool, they randomly select two distinct instruction perspectives. One is designated as the user instruction $I_{\text{user}}$ (what the model receives as input), and the other is designated as the reasoning perspective $R_{\text{gt}}$ (what the model should generate as intermediate reasoning). The target sequence $Y_{\text{gt}}$ is then constructed as the concatenation of the reasoning text and the ground-truth coordinate:

$Y_{\text{gt}} = R_{\text{gt}} \oplus p_{\text{gt}}$

where $\oplus$ denotes sequence concatenation. Concretely, the training example takes this form (reproduced from Appendix B.1, with formatting simplified):

System: You are a GUI agent. Your task is to accurately locate a UI element based on 
the user's instructions. First, you should carefully examine the screenshot and analyze 
the user's instructions from one of the following different perspectives:
- Appearance Perspective: [description of how to reason from appearance]
- Function Perspective: [description of how to reason from function]
- Spatial Perspective: [description of how to reason from spatial relationships]
- Goal Perspective: [description of how to reason from user goals]

You should translate the user's instruction into the corresponding instruction, and 
then provide the final action.

User Instruction: Click on the CSDN bookmark in the bookmarks bar to access the CSDN website.

Assistant:  thinking
I will analyze this instruction from Appearance-Based perspective, the user's instruction 
can be represented as: Click the bookmark with the red 'C' icon and the label 'CSDN' in 
the bookmarks bar.
 response
<tool_call> {"name":"grounding","arguments":{"action":"click","coordinate":[588,67]}} </tool_call>

The system prompt explicitly enumerates all four perspectives and provides guidance on how to reason from each. The user instruction is presented as a standard grounding query. The assistant's response contains two key components: (1) the reasoning text in thinking... thinking tags, which explicitly names the chosen perspective and produces a rewritten instruction, and (2) the structured tool call containing the predicted coordinate.

Why randomly select the reasoning perspective? The random selection of which perspective serves as the reasoning text is a deliberate design choice. If the model always saw the same perspective as reasoning for a given instruction, it would learn a fixed mapping rather than a flexible reasoning capability. By varying which perspective appears as the reasoning, the model learns that any perspective can serve as a valid reasoning pathway and develops the ability to generate reasoning in whatever perspective is requested (or, in the RL stage, whatever perspective it independently selects).

Training objective. The SFT stage optimizes the standard autoregressive language modeling objective over the target sequence. Given the model with parameters $\theta$, the training objective is:

$\max_{\theta} \sum_{(S, I, Y_{\text{gt}}) \in \mathcal{D}} \log P(Y_{\text{gt}} \mid S, I; \theta)$

where $\mathcal{D}$ is the SFT training corpus containing approximately 283k instances, $S$ is the screenshot, $I$ is the user instruction presented in the prompt, and $Y_{\text{gt}}$ is the concatenated target sequence of reasoning text and coordinate.

What it computes: the standard next-token prediction loss summed over all tokens in the target sequence. For the reasoning tokens, the model learns to produce coherent perspective-based instruction rewrites. For the coordinate tokens, the model learns to predict spatial coordinates conditioned on both the original inputs and its own self-generated reasoning.

Why this form: the autoregressive objective unifies two skills that must be co-optimized. First, the model must learn to generate a valid reasoning chain—it cannot predict coordinates correctly if the reasoning is incoherent or semantically misaligned with the screenshot. Second, the model must learn to use that reasoning chain to inform coordinate prediction—the reasoning is not a separate output but a conditioning context for the final grounding decision. Training these jointly, rather than as separate modules, ensures that the reasoning the model learns to generate is the reasoning that is most useful for grounding, and vice versa.

Training hyperparameters. The SFT stage uses the following configuration (Section 3.3.1 and surrounding text):

• Training data: approximately 283k instances, one epoch of training
• Base models: Qwen2.5-VL-7B and Qwen2.5-VL-32B (Bai et al., 2025)
• Learning rate: 5e-6
• Global batch size: 256
• The paper does not explicitly state the optimizer, sequence length limits, or gradient accumulation steps for the SFT stage, though these would follow standard practices for Qwen2.5-VL fine-tuning.

The exploratory diversity effect. A critical but non-obvious benefit of this SFT protocol is that it produces a model with inherent exploratory diversity. Because the training data exposes the model to multiple reasoning pathways for the same grounding intent, the SFT-trained model does not collapse to a single, uniform response pattern. Given a screenshot and instruction, it can generate several plausible reasoning chains from different perspectives, each leading to (potentially) different coordinate predictions. This diversity is what prevents the policy collapse that plagues standard SFT+RL pipelines (quantified in Table 9, where standard SFT models degrade during RL, but Instruction-as-Reasoning SFT models improve by 24.0% during RL on ScreenSpot-Pro).

Standard SFT training—where the model is fine-tuned purely on (screenshot, instruction, coordinate) triples without intermediate reasoning—teaches the model a deterministic mapping. The model learns to output [x, y] given the inputs, with no variation in its generation process. When this uniform policy enters RL training, all rollouts are nearly identical, the advantage estimates (which depend on comparing rewards across rollouts) are essentially random, and the policy update provides no useful gradient signal, or worse, pushes the model toward a spurious local optimum that maximizes reward on a few samples at the expense of general performance. This is the mechanism behind the 5.7% drop for Qwen2.5-VL-7B and the 12.7% drop for JEDI-7B shown in Table 9.

In contrast, the Instruction-as-Reasoning SFT produces a model that can generate meaningfully different responses for the same input by choosing different reasoning perspectives. During RL, these diverse rollouts produce a genuine spread of rewards (some perspectives work better than others for a given sample), the advantage estimates become reliable, and the policy update learns to favor the perspectives that consistently lead to correct coordinates. The SFT stage thus functions as an exploratory warm-up—it does not teach optimal perspective selection (that is the RL stage's job), but it teaches the model enough about the space of possible reasoning pathways to make exploration during RL productive rather than destructive.

Reinforcement Learning Stage: Learning to Select the Optimal Perspective

The SFT stage equips the model with the ability to reason from multiple instruction perspectives, but it does not teach the model which perspective to use in any given situation—the reasoning perspective during SFT was randomly assigned, not strategically chosen. The RL stage addresses this gap by training the model to discover and select the reasoning strategy that maximizes grounding accuracy for each specific screenshot-instruction pair.

Prompt modification for RL: open-ended thinking. A crucial shift occurs in the prompt between SFT and RL. During SFT, the system prompt explicitly enumerates the four perspectives and instructs the model to analyze the instruction "from one of the following different perspectives." During RL, the prompt is simplified to remove this enumeration. As shown in Appendix B.2, the RL prompt states:

Your task is to accurately locate a UI element based on the user's instructions. 
The screenshot resolution is height {height} and width {width}.
First, you should carefully examine the screenshot and analyze the user's instructions 
in  thinking... thinking tags and then output the coordinate.

The model is instructed to "think" but is not told how to think—no perspectives are listed, no reasoning format is prescribed. This open-ended prompt is an intentional design choice that serves two purposes. First, it prevents the model from being constrained to only the four perspectives seen during SFT, encouraging it to explore novel reasoning patterns (including combining perspectives or inventing entirely new ones). Second, it creates the variation in rollouts that GRPO needs to compute meaningful advantage estimates—if the prompt rigidly specified "use the appearance perspective," all rollouts would be similar and the RL signal would be weak.

Reward function: point-in-box binary reward. The RL stage uses a simple binary reward based on grounding accuracy. For a given rollout $i$, where the model generates reasoning text and a predicted coordinate $p_i$, the reward is:

$r_i = \begin{cases} 1, & \text{if } p_i \in b_{\text{gt}} \\ 0, & \text{otherwise} \end{cases}$

where $b_{\text{gt}}$ is the ground-truth bounding box and $p_i \in b_{\text{gt}}$ means the predicted point falls within the box (the same point-in-box criterion used for evaluation accuracy).

What it computes: a scalar reward of 1 if the model's predicted coordinate lands anywhere inside the ground-truth UI element, and 0 if it misses—even by a single pixel.

Why this form: the point-in-box reward is the most direct alignment between the RL objective and the evaluation metric. Since all benchmarks report point-in-box accuracy as the primary metric, optimizing this reward directly optimizes for the downstream evaluation. The binary nature of the reward (no partial credit for being "close but outside") might seem harsh, but it accurately reflects the deployment reality: clicking outside a button typically does nothing (or worse, activates the wrong element), so near-misses have the same practical consequence as far-misses. This harshness may actually be beneficial during RL—it creates a clear distinction between successful and unsuccessful rollouts, producing strong gradient signals that push the policy away from near-miss strategies.

The choice not to use a continuous reward (e.g., negative distance to the box center) is also motivated by a desire to avoid teaching the model a different objective than what is evaluated. A distance-based reward might encourage the model to aim for box centers even when the box center is not the most visually salient point, potentially creating a misalignment between the learned policy and what generalizes to new interfaces.

GRPO: advantage normalization and policy optimization. The RL stage uses Group Relative Policy Optimization (GRPO) (Guo et al., 2025; Shao et al., 2024), a variant of policy gradient methods designed for language model fine-tuning. For each training sample, the model generates $G$ rollouts (where $G = 8$ as specified in Section 3.3.2), each consisting of a reasoning chain and a predicted coordinate. The rewards $\{r_i\}_{i=1}^G$ are computed for all rollouts, then normalized into advantages using Z-score normalization:

$\hat{A}_{i,t} = \frac{r_i - \frac{1}{G} \sum_{i=1}^G r_i}{\sqrt{\frac{1}{G} \sum_{i=1}^G \left(r_i - \frac{1}{G} \sum_{i=1}^G r_i\right)^2}}$

where $r_i$ is the binary reward for rollout $i$, $G = 8$ is the number of rollouts, and $\frac{1}{G} \sum_{i=1}^G r_i$ is the sample mean of the rollout rewards.

What it computes: for each rollout, the advantage $\hat{A}_{i,t}$ measures how much better (or worse) that rollout's reward is compared to the average reward across all rollouts for the same training sample, expressed in units of standard deviation. A rollout that correctly grounds the target (reward = 1) when most other rollouts fail (mean reward close to 0) receives a large positive advantage. A rollout that succeeds when most others also succeed receives a small positive advantage. A rollout that fails when most others succeed receives a negative advantage.

Why this form: Z-score normalization serves two purposes in GRPO. First, it makes the advantage scale independent of the absolute reward magnitude—since the reward is always 0 or 1, the raw difference $r_i - \text{mean}(r)$ could be very small if most rollouts get the same reward, leading to vanishing gradients. Dividing by the standard deviation amplifies the signal when there is genuine variation in rollout quality. Second, it provides a form of baseline subtraction that is computed entirely within the current batch, eliminating the need for a separately learned value function (as would be required in standard PPO). This makes GRPO simpler to implement and less prone to value function estimation errors, which is particularly important when the reward signal is sparse and binary.

The policy optimization objective then minimizes:

$\mathcal{L} = -\frac{1}{G} \sum_{i=1}^G \frac{\pi(o_i \mid I, S)}{\pi_{\text{old}}(o_i \mid I, S)} \cdot \hat{A}_{i,t}$

where $o_i$ is the full output (reasoning + coordinate) for rollout $i$, $\pi(o_i \mid I, S)$ is the probability the current policy assigns to that output given the screenshot $S$ and instruction $I$, and $\pi_{\text{old}}(o_i \mid I, S)$ is the probability the old policy (before the current update) assigned to that same output.

What it computes: the standard policy gradient objective with an importance sampling correction. The ratio $\pi / \pi_{\text{old}}$ measures how much the policy has changed relative to the data-collection policy—it ensures that we do not over-update toward rollouts that were only sampled because the old policy happened to favor them. The advantage $\hat{A}_{i,t}$ determines the direction and magnitude of the update: positive advantages increase the probability of the corresponding outputs, negative advantages decrease them.

Why this form: the importance sampling ratio is critical for stable RL training. Without it, the policy gradient could push the model to assign high probability to rollouts that were fortunate accidents of the old policy's sampling distribution, potentially causing the policy to collapse to a narrow mode that does not generalize. The ratio constrains the update to be proportional to how much the policy has already changed, providing a form of trust-region regularization that keeps the policy from moving too far from its initialization in any single update step.

RL training hyperparameters. The GRPO stage uses the following configuration (Section 3.3.2):

• Training data: 33k unique instances, expanded to approximately 100k by generating one sample per instruction perspective
• Learning rate: 1e-6
• Number of rollouts (G): 8
• Global batch size: 256 for the 7B model, 128 for the 32B model
• Reward: binary point-in-box (1 if predicted coordinate falls within ground-truth box, 0 otherwise)
• Evaluation frequency and early stopping: not explicitly specified in the paper

The smaller batch size for the 32B model likely reflects GPU memory constraints—the larger model requires more memory per sample, so the batch size is reduced to fit within the available hardware while keeping the number of rollouts per sample constant at 8.

What the RL stage learns: strategic perspective selection. The RL objective incentivizes the model to discover which reasoning pathways produce correct coordinates and which do not. Because the binary reward is applied after the final coordinate prediction, the optimization must propagate credit backward through the reasoning chain—the model learns that certain reasoning perspectives (or certain ways of combining perspectives) are associated with higher coordinate accuracy for certain types of screenshots and instructions, and it adjusts its generation probabilities accordingly.

For example, if a screenshot contains a visually distinctive icon (a bright red button on an otherwise monochrome interface), rollouts that reason from the appearance perspective are more likely to produce correct coordinates than rollouts that reason from the function perspective (which requires understanding what the button does, which may be ambiguous from visual information alone). Over many training samples and RL updates, the model learns to associate "visually distinctive element" with "use appearance-based reasoning" and "functionally ambiguous interface" with "use spatial or goal-based reasoning." This is not a rule that is explicitly taught—it emerges from the reward signal as an implicit policy.

Emergent capabilities: reasoning beyond the four perspectives. A striking finding from the qualitative analysis (Section 4.5, Figure 8) is that the RL-trained model develops reasoning capabilities that were never explicitly taught during SFT. The paper identifies three such capabilities:

1. Strategic selection: The model chooses different reasoning perspectives for different scenarios based on what works. Figure 8b shows the distribution of perspective usage across 1,477 test samples: appearance (1,461 uses), functionality (1,110 uses), intent (815 uses), location (749 uses), and others. The model does not default to a single perspective but distributes its selections across perspectives, with the distribution reflecting which perspectives are most often effective.

2. Compositional integration: The model frequently combines multiple perspectives into a single reasoning chain. For example, as shown in Figure 9: "Click the button with the icon of centered horizontal lines [Appearance] to center the text [Functionality]." This synthesis of appearance and functionality into one coherent reasoning path is not something the SFT stage explicitly taught (SFT training always assigned a single perspective as the reasoning target). It emerges during RL as the model discovers that combining perspectives produces more robust reasoning than using any single perspective alone. Figure 8a quantifies this: of 1,477 test samples on UI-I2E-Bench, the model generated 5,245 total reasoning pathways, with many responses containing 3, 4, 5, or even 6 distinct perspectives in a single reasoning chain.

3. Emergent perspective invention: Most impressively, the model generates reasoning from perspectives entirely outside the four seen during training. The paper's taxonomy of reasoning perspectives (Appendix C.1) reveals that after RL, the model reasons from angles such as UI element state ("click the inactive button"), group affiliation ("in the alignment control group"), sequential position ("the third option in the dropdown"), component type ("the toggle switch"), and prediction of future state ("which will set it as the new exclusive active state"). These perspectives were never included in the SFT training data, which only contained appearance, function, spatial, and goal perspectives. Their emergence during RL is evidence that the model has learned not just which of the four predefined perspectives to select, but a more general capability to construct analytical lenses that are useful for grounding, drawing on its broader pretraining knowledge about how user interfaces work.

This emergent behavior validates a key design choice: the RL prompt's open-ended "think" instruction. If the prompt had continued to enumerate the four perspectives (as in SFT), the model might have remained constrained to those perspectives and never explored novel reasoning angles. By removing the perspective specification, the RL stage allows the model to discover reasoning strategies that the SFT stage's explicit categories could not capture—strategies that may be more effective for certain grounding scenarios than any of the four predefined perspectives.

Why SFT Then RL: The Complementary Roles of the Two Stages

The two-stage design is not an arbitrary sequence—each stage serves a distinct and necessary role that the other cannot fulfill:

SFT teaches format and diversity; RL teaches selection. The SFT stage establishes the capacity to reason from multiple perspectives by exposing the model to explicit examples of perspective-based reasoning during supervised training. Without this stage, the model would have no concept of what "reasoning about an instruction from a perspective" even means—the RL stage, with its open-ended "think" prompt, would be asking the model to explore a space it has no map of. The SFT stage provides that map. The RL stage then optimizes navigation within that map—given a specific screenshot and instruction, which reasoning path should the model take? The SFT stage cannot do this optimization because it never provides feedback on which perspectives work better than others (the reasoning perspective was randomly assigned).

SFT prevents policy collapse; RL pushes beyond SFT's ceiling. The SFT-only model achieves reasonable performance (e.g., 37.1% on ScreenSpot-Pro for the 7B model in Table 9), but it is limited by the fact that it learned to output whatever reasoning perspective was randomly assigned, not to choose the best one. The RL stage can improve on this by learning to select effective perspectives, but only if the SFT initialization provides enough diversity for the RL process to work. Standard SFT (without reasoning) collapses to a uniform policy, making RL destructive. Instruction-as-Reasoning SFT provides the exploratory diversity that makes RL constructive.

SFT operates on clean, curated reasoning; RL operates on open-ended exploration. The SFT training data contains carefully constructed reasoning chains—GPT-4.1 generated perspective-based instructions that were verified to uniquely map to the ground-truth element. The model learns from these high-quality examples what good reasoning looks like. The RL stage, in contrast, learns from the model's own rollouts, which may contain imperfect or exploratory reasoning. By first establishing a strong baseline of reasoning quality through SFT and then refining the selection policy through RL, the two-stage approach combines the precision of supervised learning with the adaptive optimization of reinforcement learning.

4. Key Insights and Innovations

Innovation 1: The Instruction-Is-a-Pathway Paradigm Shift — Instructions as Dynamic Reasoning, Not Static Inputs

The paper's most fundamental conceptual move is reframing what a GUI grounding instruction is. Prior work uniformly treated the instruction as a static, given input—a string you tokenize and feed through the model alongside the screenshot. Under this view, the only degree of freedom is how the model processes that instruction (architecture, training objective, reward function). The paper argues for a different ontology: an instruction is not a singular input but a choice point among multiple reasoning pathways, each encoding a distinct analytical lens (appearance, function, location, intent) for solving the same grounding problem.

This reframing matters because it transforms the instruction from a constraint into a resource. In the old view, an ambiguous or poorly phrased instruction is a liability—it degrades performance and the model must cope with it. In the new view, the existence of multiple valid phrasings for the same intent is an opportunity: each phrasing emphasizes different visual or semantic cues, and an intelligent system can dynamically select the one that most reliably resolves the grounding ambiguity for a given screenshot. The 76% relative performance improvement in Figure 2a's "Combined" bar—representing an oracle that always picks the best perspective per sample—is not just a performance number but a proof-of-concept that this latent selection capability exists and is vastly underutilized.

This is a fundamental rather than incremental advance because it changes the framing at the task definition level. It is not a better way to do the same thing; it is a claim that the thing everyone was doing (treating instructions as fixed tokens) was missing the core structure of the problem. The analogy to MCMC sampling in the reference paper is apt: just as that work reframed test-time compute strategies as modifications to a proposal distribution vs. a verifier, this work reframes grounding instructions as a space of reasoning pathways over which the model should optimize. The parallel is structural—both papers identify a degree of freedom that was previously treated as fixed and show that optimizing over it yields substantial gains.

Crucially, this reframing also explains why prior work on instruction augmentation (Aria-UI, Phi-Ground) achieved limited success. Those approaches treated augmented instructions as independent training samples—more data, but still static. The Instruction-as-Reasoning paradigm treats them as joint reasoning alternatives for the same grounding instance, with the model learning both to generate them and to select among them. This joint optimization is what distinguishes the approach from mere data augmentation and what enables the emergent compositional and novel-perspective reasoning documented in Section 4.5.

Innovation 2: The Free-Form Reasoning Failure as a Diagnostic Result with Positive Implications

The paper's documentation of free-form reasoning (FFR) failure during RL (Table 8) is distinctive not as a novel finding per se—prior works experienced this failure—but as a systematic diagnostic that converts a pattern of negative results into a positive prescription. Multiple prior methods (GUI-G1, GTA1, GUI-G2, UI-R1) observed that unstructured "thinking" during RL degrades or fails to improve grounding performance, but each treated it as a local failure of their specific setup. The paper generalizes this observation: FFR fails not because of hyperparameter choices or model size, but because the reasoning space for GUI grounding is poorly structured relative to domains like mathematical reasoning where FFR succeeds.

The diagnostic logic is: in math, the space of valid reasoning steps is constrained by formal rules—each step's validity can be verified, and intermediate computations reduce uncertainty about the final answer. In GUI grounding, there is no such structure. A model exploring "what should I think about?" without guidance can generate reasoning about irrelevant visual features, spurious correlations, or hallucinated UI elements, and this noise corrupts rather than informs the coordinate prediction. Table 8's numbers—a 6.4% relative drop for UI-Tars-1.5-7B and zero gain for Qwen2.5-VL-7B—quantify the severity of this mismatch.

The positive implication is what makes this an innovation rather than just a cautionary note. The paper shows that moving from free-form to structured reasoning—where the reasoning format is explicitly taught during SFT using predefined perspective categories—converts the FFR failure into a significant gain (9.9% relative improvement for Qwen2.5-VL-7B on ScreenSpot-Pro in Table 8's bottom section). This establishes a boundary condition for reasoning in grounding: reasoning helps, but only when the model is first taught what constitutes a valid reasoning pathway through supervised exposure to diverse perspectives. The RL stage can then optimize which pathway to select, but it cannot discover the concept of a reasoning pathway from scratch.

This finding has implications beyond GUI grounding. It suggests a more general principle for incorporating reasoning into visuospatial tasks: the reasoning ontology must be defined before the reasoning policy can be optimized. In domains where the space of useful analytical perspectives is enumerable (appearance, function, location, intent), SFT can bootstrap the ontology; in domains where it is not, FFR may remain the only viable approach, but its success is not guaranteed. This is a conceptual contribution to the broader literature on reasoning-augmented multimodal models, distinguishing between tasks where reasoning structure can be taught and tasks where it must be discovered.

Innovation 3: SFT as Exploratory Warm-Up — Reframing the SFT+RL Relationship

The standard view of SFT+RL pipelines is that SFT provides a "good initialization" and RL provides "further optimization"—both stages push in the same direction, and RL is expected to improve upon SFT. The paper's Table 9 reveals that this view is empirically false for grounding tasks: standard SFT+RL degrades performance (5.7% drop for Qwen2.5-VL-7B, 12.7% drop for JEDI-7B). The paper's reframing is that SFT and RL serve qualitatively different functions—SFT provides exploratory diversity, and RL optimizes selection within that diverse space. The two stages are not redundant but complementary, and the failure of standard SFT+RL occurs because standard SFT provides no exploratory diversity at all.

This reframing has a specific mechanism behind it. Standard SFT on (screenshot, instruction, coordinate) triples produces a model with highly uniform outputs—given the same input, it generates nearly identical coordinate predictions every time. When this uniform policy enters RL training, all rollouts are similar, the advantage estimates (which depend on reward variation across rollouts) are essentially noise, and the policy gradient either does nothing or pushes the model in a random direction. The Instruction-as-Reasoning SFT breaks this uniformity by teaching the model to generate multiple distinct reasoning chains (one per perspective) for the same input, creating genuine variation in the rollout distribution. During RL, this variation produces a spread of rewards (some perspectives work better than others for a given sample), reliable advantage estimates emerge, and the policy update meaningfully improves the selection policy.

The innovation here is not the observation of policy collapse—Phi-Ground (Zhang et al., 2025) independently noted this—but the mechanistic explanation and the deliberate design of SFT to prevent it. The paper does not just report that policy collapse happens; it shows why (uniform policy → no exploratory diversity → noisy advantage estimates) and demonstrates a constructive solution (multi-perspective SFT → diverse rollouts → reliable RL signal → 24.0% relative improvement on ScreenSpot-Pro in Table 9). This transforms the SFT+RL pipeline from a fragile recipe that frequently fails into a principled two-stage process where each stage has a defined role: SFT teaches how to reason, RL teaches which reasoning to use.

This is a conceptual advance for the broader RL fine-tuning literature. The idea that SFT can serve as an "exploratory warm-up" rather than just a policy initialization is transferable to other domains where standard SFT produces uniform behavior that starves RL of the variation it needs. The key design principle—structure the SFT objective to produce diverse outputs for the same input by conditioning on auxiliary variables (in this case, reasoning perspectives)—is a recipe that could be applied to other tasks where the space of valid responses is multi-modal.

Innovation 4: Emergent Reasoning as Evidence for Genuine Capability Generalization

The paper's qualitative finding that the RL-trained model generates reasoning from perspectives never seen during SFT—UI element state, group affiliation, sequential position, component type, future state prediction—is significant because it distinguishes memorization of a reasoning format from acquisition of a reasoning capability. If the model merely learned to pattern-match the four SFT perspectives (appearance, function, spatial, goal), it would only ever output reasoning in those categories. The emergence of novel perspectives (quantified in Figure 8b, where "Others" categories account for a non-trivial fraction of reasoning usage) indicates that the model has learned something more abstract: that grounding can be approached from multiple analytical angles, and that constructing an appropriate angle for a given screenshot-instruction pair improves accuracy.

This finding is a form of capability generalization—the model transfers the meta-skill of "choose an analytical perspective" to perspectives that were not explicitly instantiated in training. This is stronger evidence for the Instruction-as-Reasoning paradigm than performance numbers alone, because it suggests the model is not just following a recipe but has internalized the principle behind the recipe. The compositional integration finding—where the model combines multiple perspectives into a single reasoning chain (Figure 8a, with many responses containing 3–6 distinct perspectives)—reinforces this interpretation. The model is not selecting from a fixed menu; it is constructing ad hoc reasoning strategies by assembling perspective components as needed.

The intellectual significance extends beyond grounding. The finding provides a case study in how structured supervision followed by open-ended optimization can produce generalization beyond the supervision's explicit scope. The SFT stage defines the initial reasoning ontology (four perspectives), and the RL stage, by rewarding effective reasoning regardless of its form, allows the model to expand that ontology to include perspectives better suited to the data than the original four. This is a middle path between pure supervised learning (which cannot generalize beyond its training categories) and pure reinforcement learning (which may fail to discover useful structure without guidance). The SFT provides the conceptual scaffolding, and RL enables the model to build beyond it.

The paper does not claim this as a theoretical contribution, but it functions as one: it demonstrates that reasoning format specification during SFT creates a basin of attraction from which RL can discover more effective reasoning strategies than were explicitly taught. This principle—teach the structure, then optimize the content—may generalize to other domains where the space of useful reasoning strategies is larger than what can be exhaustively enumerated in training data.

5. Experimental Analysis

Evaluation Methodology

• Datasets. The paper evaluates on five static grounding benchmarks and one online agent benchmark. For grounding: MMBench-GUI L2 (Xuehui Wang et al., 2025) tests hierarchical instruction understanding across six platforms (Windows, MacOS, Linux, iOS, Android, Web) with "Basic" and "Advanced" instruction subsets; UI-I2E-Bench (Liu et al., 2025a) tests explicit vs. implicit instruction understanding grouped by platform (Web, Desktop, Mobile) and implicitness; ScreenSpot-Pro (Li et al., 2025) tests grounding on high-resolution professional software screenshots across five application categories (CAD, Development, Creative, Scientific, Office, OS) with text and icon element types; ScreenSpot-V2 (Wu et al., 2024a) tests across Mobile, Desktop, and Web platforms with text and icon element types; Showdown (Team, 2025) evaluates instruction-following and low-level control capabilities. For online agent evaluation: AndroidWorld (Rawles et al., 2024) requires completing multi-step tasks in a live, dynamically changing Android environment. The paper reports test-set results on these established benchmarks without creating new train/test splits—each benchmark has its own standard evaluation protocol.

• Base model(s). All experiments use Qwen2.5-VL-7B and Qwen2.5-VL-32B (Bai et al., 2025) as backbone vision-language models. The authors do not explicitly justify why Qwen2.5-VL was chosen over alternatives (e.g., InternVL, LLaVA, or proprietary models), but the choice appears motivated by Qwen2.5-VL being a strong open-weight multimodal model with established grounding baselines in prior work, enabling fair comparison against methods that also use Qwen2.5-VL backbones. The 7B and 32B scales span a representative range—the 7B model is competitive with similarly-sized specialized grounding models (UI-TARS-1.5-7B, GTA1-7B, InfiGUI-G1-7B), while the 32B model competes with larger models (Qwen2.5-VL-72B, InternVL3-78B, UI-TARS-DPO-72B) without reaching the largest scale.

• Metrics. The primary metric across all static benchmarks is point-in-box accuracy: a prediction is counted as correct if the predicted coordinate point p = (x_p, y_p) falls within the ground-truth bounding box b = (x_l, y_l, x_r, y_r). Accuracy over a test set of N samples is the fraction of points satisfying p ∈ b, i.e., (1/N) × Σ I(p_i ∈ b_i). This is the standard metric in grounding literature and directly measures whether a click action would hit the intended UI element. For AndroidWorld, the metric is task success rate—the fraction of multi-step tasks completed correctly in the live environment. The paper does not report secondary metrics (e.g., center distance, IoU, precision-recall curves) that would provide finer-grained insight into spatial error patterns.

• Baselines. The paper compares against an extensive set of methods spanning multiple training paradigms. Closed-source models: GPT-4o (OpenAI, 2024), Claude-3.7 (Anthropic, 2024), Qwen-Max-VL (Yang et al., 2024a), Gemini 2.5 Computer Use (DeepMind, 2025), OpenAI CUA-o3 (OpenAI, 2025). SFT-based grounding models: ShowUI-2B (Lin et al., 2024), OS-Atlas-4B/7B (Wu et al., 2024a), Aguvis-7B (Xu et al., 2025), UGround-V1-2B/7B/72B (Gou et al., 2025), Aria-UI (Yang et al., 2024b), JEDI-7B (Xie et al., 2025), Phi-ground-7B (Zhang et al., 2025), UI-TARS-7B/72B (Qin et al., 2025), UI-TARS-1.5-7B (Seed, 2025), OpenCUA-7B/32B (Wang et al., 2025b), CogAgent-18B (Hong et al., 2024), UI-I2E-VLM-4B/7B (Liu et al., 2025a). RL-based grounding models: GUI-Actor-7B (Wu et al., 2025), SE-GUI-7B (Yuan et al., 2025), GUI-G2-7B (Tang et al., 2025), InfiGUI-G1-7B (Liu et al., 2025d), GTA1-7B/32B (Yang et al., 2025), UI-Venus-7B (Gu et al., 2025), UI-R1-3B (Lu et al., 2025), GUI-R1-7B (Luo et al., 2025), UI-AGILE-7B (Lian et al., 2025), ZonUI-3B (Hsieh et al., 2025). Agent frameworks: InfiGUIAgent (Liu et al., 2025b), Ponder&Press (Wang et al., 2024a), Uground (Gou et al., 2025), AgentS2 (Zhou et al., 2024), JT-GUIAgentV2 (China Mobile, 2025), UI-Tars (Qin et al., 2025). Where the paper uses the ∗ notation (e.g., "GUI-Actor-7B∗"), it indicates results evaluated by the authors themselves rather than reported by the original paper, ensuring consistent evaluation conditions.

• Generation budget / compute accounting. For static grounding benchmarks, there is no explicit "generation budget" sweep—models produce a single coordinate prediction per sample, and accuracy is reported as a scalar. The paper does not conduct best-of-N or majority-voting experiments. The training compute is characterized by dataset size (283k SFT instances, 33k RL instances expanded to ~100k), training epochs (1 epoch SFT, RL steps unspecified), batch sizes (256 for SFT and 7B RL, 128 for 32B RL), and learning rates (5e-6 SFT, 1e-6 RL), but no total FLOPs accounting is provided. For AndroidWorld, the budget is implicit in the agent architecture—GPT-5 planner calling UI-Ins-7B executor once per action step—but no step count or inference cost analysis is reported. For the difficulty estimation experiment (Figure 2a), the "Combined" oracle evaluation requires knowing which perspective works best per sample, but this is an analytical tool rather than a deployable method.

• Cross-validation / statistical protocol. The paper does not describe any cross-validation or statistical significance testing protocol. Results are reported as single accuracy numbers per benchmark. The training-stage ablation study (Table 6) reports results from single training runs—standard error bars, confidence intervals, or multi-seed averages are not provided. This makes it impossible to assess whether observed differences (e.g., 83.1% vs. 83.4% between UI-Ins-32B and GTA1-32B on MMBench-GUI L2) are statistically reliable. The paper's two-fold cross-validation description from Section 3.2 is only for the oracle difficulty estimation experiment (Figure 2a, "Combined" bar), not for the main results.

Main Quantitative Results

Grounding Benchmarks: MMBench-GUI L2 and UI-I2E-Bench

The headline result is that UI-Ins-32B achieves state-of-the-art performance on both benchmarks designed to test complex instruction understanding. On MMBench-GUI L2 (Table 1), UI-Ins-32B attains 84.9% overall average accuracy, compared to 83.4% for GTA1-32B and 80.8% for InfiGUI-G1-7B. On UI-I2E-Bench (Table 2), UI-Ins-32B achieves 87.3% overall accuracy, compared to 83.5% for GTA1-32B and 77.4% for InfiGUI-G1-7B.

The gains are not uniform—they concentrate on the more challenging instruction subsets, which is the pattern the Instruction-as-Reasoning paradigm predicts. On MMBench-GUI L2, the benchmark is divided into "Basic" and "Advanced" subsets. Basic instructions provide explicit visual descriptions (e.g., "A rectangular button with a dark purple background"), while Advanced instructions describe inferred purpose (e.g., "Upgrade your current workspace"). For the 7B model, UI-Ins-7B achieves 82.7% on Basic vs. Qwen2.5-VL-7B's 31.4%—a 134.2% relative improvement—while on Advanced, the gap widens: 64.7% vs. 16.5% (a 159.4% relative improvement). For the 32B model, the pattern persists: UI-Ins-32B achieves 84.9% on Basic vs. Qwen2.5-VL-32B's 73.4% (12.3% relative gain), and 68.4% on Advanced vs. 49.3% (24.5% relative gain). The progressive widening of the margin from Basic to Advanced supports the claim that multi-perspective reasoning is most valuable when instructions are semantically indirect and require inference.

On UI-I2E-Bench (Table 2), the benchmark is divided into "Explicit" and "Implicit" subsets. UI-Ins-32B achieves 92.9% on Explicit vs. GTA1-32B's 91.4%—a narrow 1.6 percentage point advantage—but 83.9% on Implicit vs. GTA1-32B's 78.7%—a more substantial 6.6 percentage point advantage. A similar pattern holds for the 7B model: 88.9% vs. GTA1-7B's 87.0% on Explicit (1.9 point gain), 76.3% vs. 72.8% on Implicit (3.5 point gain). The larger margin on implicit instructions aligns with the paper's argument: when the instruction explicitly names visual features, any competent grounding model performs well; when the instruction requires semantic inference ("'Click' to dispatch the email"), the ability to reason from multiple perspectives (considering function: "what dispatches email?" vs. appearance: "where is the send button?") provides a genuine advantage.

Breaking down by platform on MMBench-GUI L2 (Table 1), UI-Ins-32B achieves the highest per-platform scores in most categories: 96.5% on iOS Basic (vs. 96.2% for GTA1-32B), 97.2% on Android Basic (vs. 95.8% for GTA1-32B), and 94.8% on Web Basic (vs. 95.2% for GTA1-32B—a rare instance where UI-Ins is slightly behind). On Linux Advanced, UI-Ins-32B achieves 56.1% vs. GTA1-32B's 52.0%, a 4.1 point advantage on one of the hardest platform-instruction type combinations.

Figures 11 and 12 provide qualitative examples. Figure 11 compares UI-Ins-7B against GTA1-7B on three challenging cases. On "Access the search functionality to find files or text within VSCode workspace," GTA1-7B predicts (1496, 164)—visually off-target—while UI-Ins-7B reasons "I will click the magnifying glass icon in the left sidebar" and correctly predicts (962, 463). On "Create a shareable link to this specific map location showing Portland," GTA1-7B misses at (639, 156), while UI-Ins-7B reasons "I will click the button with the arrow icon pointing right and up" and correctly predicts (2506, 581). On "Make important points stand out in your issue by applying emphasis," GTA1-7B predicts (2394, 804), while UI-Ins-7B reasons "I will click the 'B' button in the formatting toolbar" and correctly predicts (1590, 801). These examples visualize the mechanism: the reasoning chain identifies the correct element by its visual attributes or function, and the coordinate prediction follows from that identification.

Grounding Benchmarks: ScreenSpot-Pro, ScreenSpot-V2, and Showdown

On ScreenSpot-Pro (Table 3), which tests grounding on high-resolution professional software screenshots, UI-Ins-32B achieves 57.0% overall accuracy, establishing a new state of the art. The previous strongest result was 55.3% from OpenCUA-32B, followed by 53.6% from GTA1-32B and 53.3% from Qwen2.5-VL-72B. UI-Ins-7B achieves 52.2%, outperforming all 7B-scale models and competing with models an order of magnitude larger (Qwen2.5-VL-72B at 53.3%, InternVL3-78B at 72.2% on a different benchmark).

The per-application breakdown in Table 3 reveals that UI-Ins-32B's advantage is strongest on OS (operating system interfaces): 70.1% on text elements and 34.8% on icon elements, compared to GTA1-32B's 70.1% / 32.6% and OpenCUA-32B's unreported OS scores. On Office applications: 88.7% on text and 50.9% on icons, compared to GTA1-32B's 80.8% / 43.4%—a notable gap on icon elements (7.5 points). On Creative applications: 69.7% on text and 18.9% on icons. On CAD: UI-Ins-32B's 51.8% on text and 29.7% on icons is actually worse than some baselines on CAD text (InfiGUI-G1-7B achieves 57.4%, GUI-G2-7B achieves 55.8%).

The strong performance on OS and Office interfaces, and the relatively weaker performance on CAD, suggests that the model's reasoning strategies may be better suited to conventional UI layouts (standard buttons, menus, toolbars) than to specialized professional interfaces with domain-specific iconography and non-standard interaction patterns. The paper does not analyze this platform-dependent variation, though it is visible in the data.

On ScreenSpot-V2 (Table 4), UI-Ins-32B achieves 94.9% overall average, compared to 93.2% for GTA1-32B and 93.5% for InfiGUI-G1-7B. UI-Ins-7B achieves 94.0%, surpassing InfiGUI-G1-7B's 93.5% and UI-TARS-7B's 91.6%. The per-platform breakdown shows UI-Ins-32B strongest on Web (97.0% text, 93.1% icons) and Desktop (99.0% text, 87.9% icons). On Mobile, UI-Ins-32B achieves 98.6% text and 90.0% icons—the icon performance on mobile being slightly behind GTA1-32B's 89.1% but still competitive.

On Showdown (Table 4, final column), UI-Ins-32B achieves 73.8%, compared to 71.1% for GTA1-32B and 70.4% for GUI-G2-7B. UI-Ins-7B achieves 73.1%, outperforming GTA1-7B at 67.9% and GUI-G2-7B at 70.4%. Showdown evaluates instruction-following and low-level control—the paper does not provide per-task breakdowns that would reveal which types of control tasks benefit most from multi-perspective reasoning.

A consistent pattern across these three benchmarks is that the 7B model punches above its weight class. On ScreenSpot-Pro (Table 3), UI-Ins-7B's 52.2% surpasses Qwen2.5-VL-72B's 53.3% and is competitive with models 4–10× its size. On Showdown (Table 4), UI-Ins-7B's 73.1% exceeds GTA1-32B's 71.1%—meaning the 7B model with Instruction-as-Reasoning outperforms a 32B model using an alternative training paradigm. The paper does not explicitly highlight this cross-scale comparison, but it is visible in Tables 3 and 4 and represents evidence that the training methodology, not just model scale, drives the gains.

Online Agent Results: AndroidWorld

Table 5 reports results on AndroidWorld, where UI-Ins-7B paired with GPT-5 as planner achieves a 74.1% task success rate. The comparison points are: Gemini 2.5 Computer Use at 69.7%, UI-TARS-2 at 73.3%, and the same GPT-5 planner paired with Qwen2.5-VL-7B (without Instruction-as-Reasoning) at 50.0%. The 24.1 percentage point gap between UI-Ins-7B and its Qwen2.5-VL-7B backbone, when both serve as grounding executors under identical planner conditions, isolates the contribution of the Instruction-as-Reasoning training.

This is a meaningful result because it tests grounding in a realistic setting with UI drift, variable rendering latency, asynchronous state transitions, and stochastic user feedback—conditions that do not appear in static screenshot benchmarks. The fact that the performance gap between UI-Ins-7B and its backbone (24.1 points) is larger on AndroidWorld than the gap on static benchmarks (e.g., 12.6 points on MMBench-GUI L2 Basic in Table 1) suggests that multi-perspective reasoning provides resilience to the distribution shift between static screenshots and live interfaces, though the paper does not analyze this mechanism in detail.

Several caveats apply. The planner is GPT-5, a closed-source model whose capabilities are not fully documented. The paper does not report AndroidWorld results for the 32B model or with alternative planners, so the generalizability of the agent performance to different planning architectures is unknown. The system prompt for the agent framework (Appendix B.3) is relatively simple—"a simple yet effective agent framework" as the paper describes it—so the strong performance likely reflects grounding quality rather than sophisticated agent architecture.

Training Stage Ablation

Table 6 ablates each training stage individually using UI-Ins-7B. Without any training (zero-shot Qwen2.5-VL-7B): 63.4% MM, 56.0% I2E, 43.6% Show, 24.4% Pro, 86.5% V2. RL only (no SFT, but prompted to think): 72.4% MM, 69.2% I2E, 66.6% Show, 37.0% Pro, 88.6% V2—substantial gains from RL alone, suggesting the base model already has some capacity to reason when prompted. SFT only (no RL): 76.3% MM, 70.1% I2E, 67.5% Show, 37.1% Pro, 90.6% V2—SFT outperforms RL alone across all benchmarks, confirming that the structured perspective training is more beneficial than open-ended RL from scratch. Full SFT+RL: 83.1% MM, 81.1% I2E, 73.1% Show, 52.2% Pro, 94.0% V2—the combination substantially exceeds either stage alone, with the largest marginal gains on the hardest benchmarks (Pro: +15.1 points from SFT-only's 37.1% to full's 52.2%).

The most striking number in this table is the ScreenSpot-Pro progression: 24.4% (zero-shot) → 37.0% (RL-only) → 37.1% (SFT-only) → 52.2% (SFT+RL). The SFT-only and RL-only models achieve nearly identical Pro performance (37.1% vs. 37.0%), but combining them yields a 15.1 point jump. This is non-additive—the stages are complementary, not redundant, supporting the paper's claim that SFT provides the reasoning vocabulary while RL teaches optimal selection.

Ablation Studies and Robustness Checks

Data pipeline effectiveness (Figure 7). The paper manually inspected 1,542 samples generated by its data processing pipeline and found an error rate below 8% (Figure 7a), reduced from 23.3% in the original data. The distribution shows 93.5% precise matches, 1.2% ambiguous matches, and 5.3% mismatches. To verify that this cleaning translates to model performance, the paper trained Qwen2.5-VL-7B on 210k original samples vs. the corresponding 180k cleaned samples. Figure 7b reports consistent improvements: MMBench-GUI from 72.3% to 74.3% (+2.0), UI-I2E from 63.5% to 66.3% (+2.8), ScreenSpot-V2 from 88.1% to 90.2% (+2.1). These are modest but consistent gains, confirming the paper's claim that data quality issues actively harm training.

Reasoning format ablation: Instruction-as-Reasoning vs. free-form reasoning (Table 8). This is the paper's most important ablation because it tests the central claim that structured, perspective-based reasoning is necessary for grounding while unstructured free-form reasoning (FFR) is harmful. Applying FFR during RL to UI-Tars-1.5-7B degrades ScreenSpot-Pro performance from 50.1% (RL without FFR) to 46.9% (a 6.4% relative drop). Applying FFR during RL to Qwen2.5-VL-7B produces zero change: 36.4% in both conditions. In contrast, applying Instruction-as-Reasoning during RL improves UI-Tars-1.5-7B from 48.7% to 51.2% (5.1% relative gain) and Qwen2.5-VL-7B from 47.5% to 52.2% (9.9% relative gain). The crossover—FFR hurts one model and is neutral for the other, while IR helps both—provides clean evidence that the reasoning format, not just the presence of reasoning, determines whether RL helps or harms grounding. The paper notes that this ablation was conducted on ScreenSpot-Pro, and does not report FFR-vs-IR comparisons on other benchmarks, leaving open the question of whether the effect is benchmark-specific.

Intermediate reasoning necessity (Table 7). Removing the intermediate reasoning component entirely from both SFT and RL stages (training the model to predict coordinates directly) results in: 79.1% MM (-4.0 from full 83.1%), 70.7% I2E (-10.4 from 81.1%), 66.1% Show (-7.0 from 73.1%), 44.8% Pro (-7.4 from 52.2%), 91.7% V2 (-2.3 from 94.0%). The drop is largest on UI-I2E-Bench (-10.4 points) and substantial on ScreenSpot-Pro (-7.4 points), confirming that the reasoning component is most valuable on benchmarks requiring complex instruction understanding. Interestingly, removing reasoning only from RL while keeping it in SFT produces intermediate results: 81.6% MM, 76.2% I2E, 72.0% Show, 47.5% Pro, 93.1% V2. Removing reasoning only from SFT while keeping it in RL produces: 78.8% MM, 71.6% I2E, 68.4% Show, 48.0% Pro, 92.0% V2. The fact that SFT reasoning is more valuable than RL reasoning on most benchmarks (compare Row 2: RL-with-reasoning only vs. Row 3: SFT-with-reasoning only) supports the paper's argument that SFT teaches the reasoning format, which is the harder and more important skill.

SFT+RL policy collapse mitigation (Table 9). This ablation directly tests the paper's claim that Instruction-as-Reasoning SFT prevents policy collapse during RL. Standard SFT on Qwen2.5-VL-7B achieves 37.0% on ScreenSpot-Pro, but subsequent RL degrades this to 34.9% (5.7% drop). The same pattern with JEDI-7B: 39.5% zero-shot drops to 34.5% after RL (12.7% drop). In contrast, Instruction-as-Reasoning SFT on Qwen2.5-VL-7B achieves 37.1% (nearly identical to standard SFT's 37.0%), but subsequent RL improves performance to 46.0%—a 24.0% relative improvement. This is the paper's strongest piece of evidence for the "exploratory warm-up" interpretation of the SFT stage. The near-identical SFT-only scores (37.0% standard vs. 37.1% IR-SFT) demonstrate that the instruction-as-reasoning format does not itself improve supervised performance—it is specifically the interaction with RL where the benefit materializes, by providing diverse rollouts that enable stable policy optimization.

Revision-based SFT vs. multi-perspective SFT for exploratory diversity (inferred from Tables 7 and 9, not a separately named ablation). The paper does not directly compare its multi-perspective IR-SFT against an alternative SFT approach that also introduces diversity (e.g., training on paraphrased instructions without explicit perspective reasoning). The comparison is between IR-SFT and standard coordinate-only SFT. This leaves open whether any diversity-introducing SFT would prevent policy collapse, or whether the specific structure of perspective-based reasoning is necessary. The paper's FFR-vs-IR ablation (Table 8) partially addresses this—FFR introduces diversity but fails to improve—but that comparison is within the RL stage, not the SFT stage.

Model scale ablation (Tables 1-4, implicit). The paper trains both 7B and 32B models, providing an implicit ablation on model scale. The gains from Instruction-as-Reasoning are not uniform across scales. On MMBench-GUI L2, the improvement from Qwen2.5-VL-7B to UI-Ins-7B is +49.2 points (33.9% → 83.1%), while the improvement from Qwen2.5-VL-32B to UI-Ins-32B is +12.8 points (72.1% → 84.9%). The larger absolute gain for the 7B model suggests that multi-perspective reasoning partially compensates for the smaller model's weaker visual-semantic alignment—the 32B model already performs well on standard instructions, so the reasoning benefit is proportionally smaller. However, the 32B model still benefits (+12.8 points is substantial), indicating that even strong base models have untapped capability that perspective-based reasoning unlocks.

Verifier quality / PRM training (not applicable). Unlike the reference paper, UI-Ins does not use a separate verifier model—the grounding reward is a simple point-in-box binary computed from the ground-truth annotation. There is no ablation on reward shaping, alternative reward functions (e.g., distance-based continuous rewards), or reward model training. The paper does not discuss whether the binary reward's harshness (no partial credit for near-misses) might slow RL convergence or whether a smoother reward landscape would improve sample efficiency.

Critical Assessment

Claim 1: "Instruction diversity unlocks up to 76% relative performance gain even without training." The evidence for this claim is Figure 2a, which shows Qwen2.5-VL-7B zero-shot performance on ScreenSpot-Pro using instructions from different perspectives. The "Combined" bar at 26.1% represents an oracle that selects the best-performing perspective per sample. The "Original" bar is at 24.4%, meaning the absolute improvement is 1.7 percentage points, which corresponds to a 76% relative improvement when computed as the percentage increase from a lower baseline (the calculation appears to use the formula: (optimal_combined - worst_possible_or_original) / original, though the paper does not make the denominator explicit). This is a striking number, but it is a theoretical ceiling, not a realized gain. No practical system can achieve this without oracle knowledge of which perspective works best per sample. The actual achieved gains from the full UI-Ins system (e.g., 52.2% vs. 24.4% on ScreenSpot-Pro, a +27.8 point absolute improvement or ~114% relative) are larger than the theoretical ceiling from instruction diversity alone, suggesting that the SFT+RL training contributes substantially beyond what inference-time instruction rewriting would provide. The 76% figure is best understood as a motivational diagnostic—it shows there is latent capability to exploit—rather than a performance target the system actually achieves at inference time.

Claim 2: "The Instruction-as-Reasoning paradigm establishes SOTA across five benchmarks." This claim is well-supported for the specific benchmarks tested (MMBench-GUI L2, UI-I2E-Bench, ScreenSpot-Pro, ScreenSpot-V2, Showdown). The margins are clear: on the hardest benchmarks (ScreenSpot-Pro), UI-Ins-32B's 57.0% leads the next best result (OpenCUA-32B at 55.3%) by 1.7 points. On UI-I2E-Bench, the margin is larger (87.3% vs. 83.5% for GTA1-32B). However, several caveats apply:

• The benchmarks are all static screenshot grounding tasks. Whether the SOTA translates to dynamic, streaming, or video-based grounding is untested. The AndroidWorld result (74.1%) is promising but uses a GPT-5 planner, making it impossible to isolate how much of the success comes from grounding vs. planning.
• Competing methods from late 2024 / early 2025 are compared, but the field moves rapidly. The paper compares against methods available at the time of writing, but new baselines (beyond the October 2025 submission date) may shift the landscape.
• MMBench-GUI L2 and UI-I2E-Bench are relatively new benchmarks with fewer established results, so the SOTA claim rests partly on comparison against a smaller set of reported numbers than ScreenSpot-V2, which has broader adoption.

Claim 3: "The two-stage SFT+RL framework prevents policy collapse that affects standard SFT+RL pipelines." Table 9 provides clean evidence: standard SFT+RL degrades (37.0% → 34.9% for Qwen2.5-VL-7B; 39.5% → 34.5% for JEDI-7B), while IR-SFT+RL improves (37.1% → 46.0%). The numbers are unambiguous. However, the paper does not explore why the IR-SFT prevents collapse beyond the high-level "exploratory diversity" explanation. Would any SFT protocol that introduces output diversity work, or is the specific structure of multi-perspective reasoning necessary? An ablation using an alternative diversity-introducing SFT method (e.g., training on multiple paraphrased instructions without the explicit perspective labels) would strengthen this claim but is not reported. Additionally, the RL training for the standard SFT model may not have been tuned with the same care as the IR-SFT model—the paper reports performance after 100 RL steps in both cases (Table 9 header), but does not show learning curves that would reveal whether the standard SFT model might recover with different hyperparameters or more steps.

Claim 4: "The model exhibits emergent reasoning beyond the four predefined perspectives." The qualitative evidence in Figures 8 and 9 is convincing: the model generates reasoning from perspectives like "state" ("click the inactive button"), "group affiliation" ("in the alignment control group"), and "prediction of future state" that were never explicitly taught. Figure 8b provides quantitative evidence—the perspective distribution includes "Component Type" (200 uses), "Structural Relationship" (440 uses), and "Others" (815 uses) alongside the four trained perspectives. However, the claim's strength depends on two assumptions that are not fully verified:

• Are these perspectives truly emergent, or were they implicitly present in the GPT-4.1-generated SFT data? The SFT data was generated by GPT-4.1 using a prompt (Appendix A.1) that specifically requested appearance, function, spatial, and goal perspectives. If GPT-4.1 occasionally generated instructions that implicitly contained state or component-type language (e.g., describing a button as "the disabled submit button"), the model might be reproducing rather than inventing these perspectives. The paper's verification step (Appendix A.2) checks for uniqueness and bounding box quality, not for perspective purity. A closer analysis of the SFT training data's perspective content would be needed to rule out data leakage.
• Do the emergent perspectives improve accuracy, or are they merely present? The paper shows that novel perspectives appear in model outputs but does not compare the accuracy of samples where the model uses novel perspectives vs. trained perspectives. If the novel perspectives are associated with lower accuracy (i.e., represent reasoning failures the model is exploring), the "emergent capability" framing would need qualification. The paper's point about emergent capabilities appears in the qualitative analysis section (Section 4.5) where model outputs on the successful cases from Figure 11 are examined, but there is no systematic comparison of success rates by reasoning perspective type.

Weaknesses in experimental design:

• No confidence intervals or statistical testing. All results in Tables 1-9 are reported as point estimates. With test sets of 500 questions (or fewer for per-platform breakdowns), differences of 1-2 percentage points may not be statistically significant. For example, UI-Ins-32B's 84.9% vs. GTA1-32B's 83.4% on MMBench-GUI L2 (1.5 point gap) could plausibly arise from sampling variance on a test set of unknown size. The paper does not report the number of samples per benchmark, making it impossible to compute standard errors.

• Single seed results. The training-stage ablation (Table 6) and the policy collapse experiment (Table 9) appear to be single-run results. RL training is known to be sensitive to random seeds, initialization, and data ordering. Without multi-seed averages, it is impossible to determine whether the observed differences (e.g., 46.0% vs. 34.9% in Table 9) are robust or an artifact of favorable initialization.

• No breakdown of AndroidWorld performance by task type. The 74.1% success rate is a single aggregate number. The paper does not report per-task success rates, average steps to completion, types of grounding failures, or comparison against the Qwen2.5-VL-7B baseline on individual tasks. This makes it difficult to diagnose where the grounding improvement matters most in the online setting.

• The GPT-4.1 data generation pipeline is a black box. While the paper provides the prompts used for instruction augmentation and verification (Appendices A.1, A.2), it does not report the rate at which GPT-4.1's generated instructions pass the verification step, the distribution of verification failures by perspective type, or any analysis of systematic biases in the generated data. If GPT-4.1 tends to produce more accurate appearance-based instructions than function-based instructions (because appearance is easier to verify from a screenshot), the training data would be imbalanced in ways that could affect the learned policy.

• Missing ablation on the number of perspectives. The paper uses four predefined perspectives (appearance, function, spatial, goal). Would three be enough? Would six be better? An ablation varying the number and types of perspectives in the SFT data would clarify whether the specific four are optimal or whether the method is robust to perspective granularity.

• Missing comparison against inference-time instruction rewriting without training. The paper shows (Figure 2a) that simply prompting Qwen2.5-VL-7B with rewritten instructions improves zero-shot performance. A useful baseline would be: take the best zero-shot perspective per sample using the PRM score as a selection criterion (analogous to the reference paper's predicted difficulty bins), without any SFT or RL. This would isolate how much of UI-Ins's gain comes from the training vs. from simply having better instructions at inference time.

• The AndroidWorld result uses GPT-5 as planner. While the comparison against Qwen2.5-VL-7B as grounding executor with the same GPT-5 planner (50.0% vs. 74.1%) isolates the grounding contribution, the overall agent architecture is a two-model system where the planner's quality matters enormously. The paper does not report results with open-weight planners (e.g., Qwen2.5-VL-7B as both planner and executor) that would make the system fully reproducible and self-contained. The choice of GPT-5 also raises cost and latency concerns that are not discussed—the 74.1% success rate may come at a compute cost that makes the agent impractical for many deployment scenarios.

Experiments that would strengthen the paper:

1. Multi-seed RL training with confidence intervals. Running the GRPO stage from 3-5 different random seeds and reporting mean ± std across all benchmarks would address the most basic reproducibility concern.

2. Learning curves for RL training. Showing the performance trajectory over RL steps for both standard SFT and IR-SFT initializations would reveal whether standard SFT+RL is truly "collapsing" (performance monotonically decreasing) or merely "not improving" (performance flat while IR-SFT improves). The paper's Table 9 reports endpoint performance at 100 steps—the trajectory matters for diagnosing the mechanism.

3. Ablation on the SFT reasoning diversity. Compare IR-SFT with: (a) SFT on multi-perspective data but without the explicit "I will analyze this instruction from X perspective" preamble (just diverse instructions as input, no reasoning chain), and (b) SFT on single-perspective data with reasoning chains. This would disentangle whether the benefit comes from data diversity, the reasoning format, or their interaction.

4. Inference-time scaling ablations. The reference paper's key finding is that compute-optimal allocation beats uniform allocation. Does UI-Ins benefit from generating multiple reasoning chains at test time and selecting the one with the highest confidence (or majority vote on the predicted coordinate)? The paper does not explore test-time scaling strategies—all results are single-sample per (screenshot, instruction) pair.

5. Generalization to unseen GUI platforms. The training data covers Windows, MacOS, Linux, iOS, Android, and Web, but do not include specialized platforms like automotive interfaces, TV UIs, or embedded device screens. Testing on an out-of-distribution platform would test whether the multi-perspective reasoning generalizes or is specific to the platforms seen during training.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For, and No Practical Difficulty Estimator Exists

The assumption or constraint. The paper's core diagnostic—that instruction diversity unlocks up to a 76% relative performance improvement without training—depends on knowing which instruction perspective is best for each sample. The "Combined" bar in Figure 2a represents an oracle that always selects the best-performing perspective per sample, requiring ground-truth knowledge of which perspective yields a correct coordinate. The paper does not propose or evaluate any practical method for selecting the optimal perspective at inference time without such ground-truth knowledge. The full UI-Ins system avoids this oracle requirement by training the model to generate and select perspectives internally during the SFT+RL process, but the core motivating claim about instruction diversity's latent potential rests on an oracle that is not deployable.

The consequence. The 76% figure—which the paper prominently features in the abstract and introduction as evidence of untapped potential—is a theoretical ceiling, not a realized or realizable gain. A practitioner reading this number might reasonably expect that the UI-Ins system achieves something close to a 76% relative improvement, but the actual improvement comes from the full SFT+RL training pipeline, which is a fundamentally different mechanism (training the model to internally select perspectives) than the inference-time oracle (having external knowledge of the best perspective per sample). The distinction matters because it conflates two separate claims: (1) instruction diversity could help if we knew which perspective to use, and (2) the model can learn to select the right perspective. The paper's experiments overwhelmingly support claim (2) but treat claim (1) as the primary motivating evidence, which could mislead readers about what the system actually does at inference time.

Furthermore, the paper does not address whether at inference time, a deployed UI-Ins model could benefit from generating multiple reasoning chains with different perspectives and selecting among them (analogous to best-of-N sampling in the reference paper). The model generates a single reasoning chain and coordinate per input—there is no test-time search over perspectives, no majority voting across multiple rollouts, no confidence-based selection. If the model selects a suboptimal perspective for a given sample (which it inevitably will in some fraction of cases), there is no recovery mechanism. The paper does not report the model's perspective selection accuracy—what fraction of the time does the reasoning perspective the model chooses actually lead to a correct coordinate, and how often does choosing the wrong perspective cause a failure that a different perspective would have avoided?

What evidence exists in the paper. The distinction between the oracle "Combined" bar (26.1% in Figure 2a) and the actual UI-Ins-7B zero-shot performance on ScreenSpot-Pro (24.4% for "Original") is a 1.7 percentage point absolute difference. The full UI-Ins-7B system achieves 52.2% on ScreenSpot-Pro (Table 3), far exceeding the oracle ceiling of 26.1% from instruction diversity alone. This indicates that the SFT+RL training contributes substantially beyond what inference-time perspective selection would provide, but the paper never explicitly reconciles these numbers or explains why the trained model so dramatically exceeds the oracle ceiling. The most plausible explanation is that the SFT+RL process teaches the model more than just perspective selection—it teaches better visual grounding, better spatial reasoning, and better alignment between language and visual features—but the paper does not decompose the total gain into perspective-related vs. general grounding improvement components.

Mitigation status. Not addressed. The paper does not propose a practical method for selecting the optimal instruction perspective at inference time without training, does not ablate how performance varies with the number of perspectives explored at test time, and does not report the accuracy of the model's internal perspective selection decisions. The authors position the 76% figure as motivation rather than a deployable method, which is intellectually honest, but the paper would benefit from an explicit acknowledgement that the motivating oracle result and the actual system's mechanism are fundamentally different, and that the gap between them is bridged by training, not by inference-time selection.

The Training Relies on GPT-4.1 as a Proprietary, Non-Reproducible Data Generator

The assumption or constraint. The multi-perspective instruction augmentation pipeline (Section 3.2 and Appendix A.1) depends entirely on GPT-4.1 (OpenAI, 2025a) to generate high-quality instruction variants from the four perspectives (appearance, function, spatial, goal) and to verify that each generated instruction unambiguously maps to the ground-truth element (Appendix A.2). The paper does not provide any analysis of whether alternative models—open-weight models like Llama-3, Qwen, or earlier GPT versions—could achieve comparable data quality. GPT-4.1 is a closed-source, API-accessed model whose behavior, pricing, and availability may change over time, making exact reproduction of the data pipeline dependent on access to a specific proprietary system at a specific point in time.

The consequence. This creates two distinct problems for the research community:

Reproducibility: A researcher seeking to reproduce UI-Ins's results must have access to GPT-4.1 (or a model with equivalent capabilities) at the time of reproduction. If GPT-4.1 is deprecated, its API is modified, or its behavior changes (as has happened with previous GPT models over time), the generated training data will differ in ways that could affect downstream model performance. The paper provides the exact prompts used for generation and verification (Appendices A.1 and A.2), which is commendable, but prompt fidelity alone does not guarantee reproduction when the underlying model is a moving target. The paper does not report generation throughput, cost, or failure rates, making it impossible to estimate the resources required to replicate the data pipeline.

Scientific attribution: The strong performance of UI-Ins is partially attributable to the quality of GPT-4.1's instruction generation. If GPT-4.1 produces particularly good appearance-based descriptions or particularly reliable verification decisions, these properties are baked into the training data and inherited by UI-Ins. The paper cannot fully separate "what the training framework contributes" from "what GPT-4.1's data quality contributes" without an ablation using a different (ideally open-weight) generation model. The paper also does not ablate whether the quality or the diversity of the generated instructions is the active ingredient—would lower-quality but equally diverse instructions from a weaker model produce similar gains?

Cost and scalability for practitioners: A team wanting to adopt the Instruction-as-Reasoning approach for a new domain (e.g., automotive interfaces, medical device UIs, specialized enterprise software) would need to run the full GPT-4.1 pipeline on their own data—generating four perspective variants per sample and verifying each one. For a dataset of 100k+ samples, this represents a substantial API cost and generation time that is not quantified in the paper. The paper does not report the number of API calls, the cost per call, the generation latency, or the failure/retry rate for the GPT-4.1 pipeline. A practitioner cannot estimate whether the data generation step is a minor upfront expense or a prohibitive barrier.

What evidence exists in the paper. Figure 7a reports that the pipeline achieves a 93.5% precise match rate on a manually inspected sample of 1,542 generated instructions, with a total error rate below 8%. This is the only quality metric reported for the GPT-4.1-generated data. The paper does not report:

• The per-perspective generation success rate (does GPT-4.1 produce valid appearance instructions more reliably than goal instructions?)
• The verification pass rate (what fraction of generated instructions are rejected by the verification step, and does this vary by perspective?)
• The inter-annotator agreement between GPT-4.1 verification and human judgment (was the 93.5% rate validated against human labels, or is it GPT-4.1's self-assessment?)
• The cost or time to run the pipeline on the full ~283k SFT instances

The data pipeline ablation (Figure 7b) shows that models trained on cleaned data outperform those trained on original data, establishing that the cleaning step (OmniParser-based bounding box refinement) is beneficial, but this ablation does not isolate the contribution of GPT-4.1's multi-perspective augmentation from the contribution of cleaning. The paper does not train a model on cleaned-but-not-augmented data vs. cleaned-and-augmented data—the comparison is always original vs. cleaned (with augmentation implicitly included in the cleaned pipeline).

Mitigation status. The paper does not address the GPT-4.1 dependency as a limitation. The authors provide the full prompts (Appendices A.1, A.2) and report the final data quality (93.5% precise match), but do not discuss reproducibility concerns, propose alternative open-weight generation models, or estimate the pipeline's computational cost. The commitment to release code and model checkpoints (stated in the abstract) partially mitigates the reproduction barrier for researchers who want to use the trained models directly, but does not help practitioners who need to apply the method to new domains with new data.

Hard Problems Remain Hard: No Improvement on the Most Difficult Grounding Cases

The assumption or constraint. The Instruction-as-Reasoning paradigm assumes that the base model possesses sufficient visual-semantic understanding to ground instructions from multiple perspectives—that the model can identify the correct element when given the right analytical lens, and the challenge is primarily one of selecting that lens. This assumption breaks down when the base model fundamentally lacks the capability to ground certain types of instructions, regardless of which perspective is used. The paper's error analysis (Section 4.6) identifies three failure categories where UI-Ins's multi-perspective reasoning does not help: lack of domain-specific knowledge, lack of layout understanding ability, and visual ambiguity/hallucination.

The consequence. The Instruction-as-Reasoning framework provides no mechanism for acquiring new capabilities that the base model lacks. It can amplify existing grounding ability by helping the model choose the most effective analytical lens, but it cannot teach the model to recognize "MEGA" as the brand associated with "building block toys" (Figure 10a), to understand complex spatial layouts (Figure 10b), or to disambiguate visually near-identical icons (Figures 10c, 10d). These failures require knowledge, reasoning, or visual discrimination capabilities that must be acquired during pretraining—they are not recoverable through perspective selection at inference time or through the SFT+RL training process.

This limitation establishes a capability ceiling that is analogous to the reference paper's finding that test-time compute cannot help on the hardest MATH problems (difficulty bin 5, where pass@1 is near zero). Just as test-time compute amplifies existing capability but does not create it, multi-perspective reasoning helps the model deploy its existing grounding skills more effectively but does not expand those skills beyond what was acquired during pretraining. A model that cannot tell the difference between a green microphone icon and a visually similar green icon under any perspective will not benefit from being able to choose between appearance, function, spatial, and goal lenses—all lenses will fail because the fundamental visual discrimination is missing.

What evidence exists in the paper. Section 4.6 provides qualitative examples of three failure modes. Figure 10a: the model selects "Jazwares" instead of "MEGA" for "the company known for building block toys," demonstrating missing world knowledge. Figure 10b: the model cannot determine the correct clickable area to fulfill a spatial instruction, demonstrating layout understanding failure. Figures 10c and 10d: the model cannot distinguish a green microphone icon from a visually similar distractor, demonstrating visual ambiguity failures. These are presented as qualitative examples only—the paper does not report the frequency of each failure type (what fraction of total errors fall into each category?), the perspective-dependence of failures (are knowledge failures equally common across all reasoning perspectives, or do some perspectives avoid certain failure modes?), or whether the failure rate on these hard cases changes between the SFT-only and SFT+RL models. Without quantitative error categorization, it is impossible to assess whether the Instruction-as-Reasoning training helps on these hard cases (by selecting a perspective that partially mitigates the knowledge gap) or is simply neutral (failures persist regardless).

The paper also does not report performance on the hardest subsets of the grounding benchmarks. MMBench-GUI L2's "Advanced" subset and UI-I2E-Bench's "Implicit" subset are harder than their "Basic" and "Explicit" counterparts, but within these subsets there is presumably a difficulty distribution—some Advanced instructions are only slightly harder than Basic (requiring mild inference), while others are genuinely impossible without external knowledge. The paper reports aggregate performance on these subsets (e.g., 68.4% for UI-Ins-32B on MMBench-GUI L2 Advanced in Table 1) but does not break down by finer-grained difficulty levels within the subsets, so it is impossible to determine whether there is a "hardest bin" analogous to the reference paper's bin 5 where UI-Ins provides essentially zero gain.

Mitigation status. The paper acknowledges these failure modes qualitatively in Section 4.6 but does not quantify their prevalence, analyze their relationship to the Instruction-as-Reasoning framework, or propose mitigation strategies. The error analysis is descriptive ("we identified three primary types of failures") rather than diagnostic ("Instruction-as-Reasoning helps with Type A but is neutral on Type B, and Type C failures could be addressed by X"). The lack of quantitative error categorization makes it difficult to assess how much headroom remains—if 30% of remaining errors are knowledge-based and fundamentally unfixable by better instruction processing, the practical ceiling for perspective-based methods is lower than the current accuracy numbers suggest.

Single Model Family, Single Task Modality, and Single Type of Reasoning Structure

The assumption or constraint. All experiments use Qwen2.5-VL-7B and Qwen2.5-VL-32B as the base vision-language models. The Instruction-as-Reasoning framework is evaluated exclusively on GUI grounding (point prediction from screenshots and instructions), with no experiments on related tasks such as GUI action prediction (what action to take), GUI state identification (what state the interface is in), or non-GUI visual grounding tasks (referring expression comprehension in natural images). The reasoning structure—four predefined perspectives (appearance, function, spatial, goal)—is manually defined by the authors based on their analysis of how humans describe UI elements (Section 2.1, Figure 3), with no exploration of alternative perspective taxonomies.

The consequence. This creates three distinct generalization uncertainties:

Model family generalization: Qwen2.5-VL may have properties that make it particularly amenable to perspective-based reasoning instruction. Its pretraining data mixture, its vision-language alignment procedure, its architecture (e.g., how visual features and text tokens interact in the attention mechanism)—all of these could influence whether multi-perspective SFT produces the exploratory diversity that prevents RL policy collapse (Table 9). A different model family (InternVL, LLaVA, proprietary models like GPT-4V or Claude) might respond differently to the same SFT+RL protocol. The paper demonstrates that the approach works on two Qwen2.5-VL scales (7B and 32B), which provides some evidence of within-family robustness, but does not test on any non-Qwen architecture. The claim implicit in the paper's title and abstract—that Instruction-as-Reasoning is a general paradigm for GUI grounding—would be substantially stronger with evidence from at least one alternative model family.

Task modality generalization: The paper argues that the Instruction-as-Reasoning paradigm is motivated by the specific structure of GUI grounding—where instructions can be naturally decomposed into appearance, function, spatial, and goal perspectives because these correspond to how humans naturally describe UI elements. Whether this perspective taxonomy transfers to other visual grounding tasks is unknown. For example, grounding a natural language expression to an object in a photograph (e.g., "the dog behind the fence") might benefit from different perspective categories (attribute-based, relational, contextual). The paper does not claim generalizability beyond GUI grounding, but it also does not explicitly bound the scope of the paradigm or discuss what properties a task must have for Instruction-as-Reasoning to be applicable. Practitioners working on non-GUI visual grounding tasks cannot determine from this paper whether the approach would transfer.

Perspective taxonomy generalization: The four-perspective taxonomy (appearance, function, spatial, goal) was designed by the authors based on their qualitative analysis of human instruction strategies (Section 2.1). The paper does not ablate alternative taxonomies—would three perspectives work as well? Would six (adding state, component type, as in the emergent categories in Appendix C.1) work better? The emergent perspective finding (Section 4.5, Figure 8b) suggests that the model discovers perspectives beyond the four trained ones during RL, which implies that the explicit taxonomy is partially constraining the model's reasoning space during SFT. An ablation where the SFT stage uses all 10 perspectives from Appendix C.1's taxonomy, or where the number of perspectives is systematically varied, would reveal whether the specific choice of four perspectives is optimal or whether more perspectives would accelerate the emergence of novel reasoning strategies.

What evidence exists in the paper. The paper provides positive evidence for within-family generalization (7B and 32B both benefit from the method, Tables 1-4) and positive evidence that Instruction-as-Reasoning transfers across benchmark formats (static screenshot grounding on five benchmarks, dynamic agent grounding on AndroidWorld). Tables 1-4 show that UI-Ins outperforms Qwen2.5-VL on all benchmarks, and Table 5 shows that UI-Ins-7B outperforms Qwen2.5-VL-7B as a grounding executor under the same planner, providing evidence for generalization across evaluation conditions within the GUI grounding task.

However, the paper provides no negative evidence—no benchmarks where Instruction-as-Reasoning fails to improve performance, no model families where the approach is neutral or harmful, no perspective taxonomies that are less effective—making it impossible to identify the boundary conditions of the method. The paper's strong positive results could reflect either genuine robustness (the method works across many conditions) or implicit selection bias (the method was developed and tuned on the specific benchmarks and model family where it works best, with failures not reported).

Mitigation status. The paper does not discuss generalization limitations. The authors present the results as establishing Instruction-as-Reasoning as an effective paradigm for GUI grounding without qualifying the scope of "GUI grounding" (does it include action prediction? State identification? Multi-step task completion without a planner?) or the scope of applicable model architectures. A brief acknowledgement in the limitations or future work section—specifying that results are demonstrated on Qwen2.5-VL and that transfer to other architectures or to non-GUI grounding tasks is an open question—would appropriately bound the claims.

The SFT+RL Framework Is Validated at a Single Snapshot, Without Learning Dynamics Analysis

The assumption or constraint. The paper's central methodological claim—that Instruction-as-Reasoning SFT prevents policy collapse during RL by providing exploratory diversity—is supported by a single before-and-after comparison in Table 9. Standard SFT+RL degrades Qwen2.5-VL-7B from 37.0% to 34.9% on ScreenSpot-Pro; Instruction-as-Reasoning SFT+RL improves it from 37.1% to 46.0%. Both measurements are taken after 300 RL steps (stated in the Table 9 header: "Scores after 100 RL steps are reported"—note the discrepancy between "100 RL steps" and the text, which suggests the table header may include a typo; the paper text in Section 4.5 states that the standard SFT+RL model "is trained for 300 RL steps").

The consequence. Endpoint comparisons alone cannot distinguish between several alternative explanations for the observed pattern:

• Genuine prevention of policy collapse: The IR-SFT initialization provides exploratory diversity, RL reliably improves performance, and the standard SFT initialization causes monotonic degradation. The learning curves would show IR-SFT performance rising over RL steps and standard SFT performance falling.

• Different optimal stopping points: Both initializations might improve initially and then degrade (a common pattern in RL fine-tuning where the policy first exploits the reward signal productively and then overfits). The standard SFT model might have peaked earlier and degraded further by step 100, while the IR-SFT model might still be near its peak. If the standard SFT model achieved (say) 39% at step 50 and then dropped to 34.9% by step 100, the problem is not "policy collapse" per se but "faster overfitting"—a different mechanism with different solutions (e.g., early stopping rather than SFT redesign).

• Hyperparameter sensitivity: The RL training hyperparameters (learning rate 1e-6, 8 rollouts, batch size 256) were presumably tuned for the IR-SFT initialization (since that is the main method). The standard SFT initialization might require different hyperparameters—a lower learning rate to prevent destructive updates, more rollouts to reduce advantage noise despite limited output diversity, or a different reward normalization scheme. The paper does not report whether the standard SFT+RL model was given the same hyperparameter optimization budget as the IR-SFT model.

• Reward hacking vs. genuine capability loss: The standard SFT model's degradation (37.0% → 34.9%) could reflect the model learning to produce coordinates that score well under the point-in-box reward without actually improving grounding—for example, biasing predictions toward the center of the screen where UI elements are more common, or toward larger bounding boxes that are easier to hit. If this is the mechanism, the model's grounding accuracy degrades even though its reward might be stable or increasing. The paper reports accuracy but not reward during RL training. If the standard SFT model's reward increased while accuracy decreased, that would be evidence for reward hacking; if both decreased, the mechanism would be genuine capability degradation.

What evidence exists in the paper. None that addresses these alternatives. Table 9 reports two numbers per condition: the SFT-only score and the score after RL. There are no learning curves (accuracy vs. RL steps for both initializations), no reward curves, no analysis of what types of errors increase or decrease during RL training, and no hyperparameter sweep for the standard SFT+RL condition to test whether the observed degradation is robust to hyperparameter choice. The paper's text in Section 4.5 states:

"We compare our SFT+RL framework with a standard one in this ablation. The standard SFT training provides a poor policy initialization, often causing the model's performance to degrade during RL, as evidenced in the upper part of Tab. 9."

The phrase "often causing" suggests the authors have observed this degradation across multiple runs or configurations, but the paper reports only a single number per condition. Without learning curves or multi-seed results, the "policy collapse" mechanism—while plausible and consistent with Phi-Ground's independent observation (Zhang et al., 2025)—remains a conjecture rather than a demonstrated mechanism.

Mitigation status. The paper does not provide learning curves, multi-seed results, or hyperparameter robustness analysis for the policy collapse experiment. The authors cite Phi-Ground's independent observation of the same phenomenon as corroborating evidence, which partially mitigates the single-number concern by suggesting the effect is replicable across different experimental setups. However, Phi-Ground's observation is also based on endpoint comparisons rather than learning dynamics analysis (the paper describes it as noting that "models fine-tuned via SFT using only coordinates as ground truths often exhibit highly uniform responses, leading to ineffective exploration and policy collapse in RL"), so it does not resolve the alternative explanations described above. A learning curve comparison—showing the trajectory of both standard SFT+RL and IR-SFT+RL over training steps—would substantially strengthen the mechanistic claim and is a natural candidate for inclusion in the paper's supplementary material or future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing rather than an incremental improvement: it redefines what a GUI grounding instruction is in the inference pipeline. Before this work, instructions were treated as static, given inputs—strings to be tokenized and fed alongside the screenshot, with no analysis of their internal structure or analytical perspective. The paper demonstrates that this treatment was a blind spot: different phrasings of the same intent (appearance-based, function-based, spatial-based, goal-based) activate fundamentally different reasoning pathways in the model, and which pathway is activated dramatically affects grounding accuracy (Figure 2a, where zero-shot performance on ScreenSpot-Pro varies from 21.3% to 43.1% depending purely on which instruction perspective is provided at inference time).

The shift from "instructions are opaque inputs" to "instructions are choices among reasoning pathways" is not merely taxonomic. It opens a new degree of freedom for grounding systems: rather than passively accepting whatever instruction phrasing a user or upstream planner provides, a system can actively rewrite, select, or compose instruction perspectives to maximize the probability of correct grounding. The 76% relative improvement in the Figure 2a "Combined" oracle bar—representing an oracle that always picks the best perspective per sample—quantifies how much latent performance is locked behind this degree of freedom.

This reframing also resolves a contradiction in prior work about whether reasoning helps or harms GUI grounding. Multiple recent methods (GUI-G1, GTA1, GUI-G2, UI-R1) observed that free-form reasoning during RL either degrades performance or produces no gain, leading to a natural conclusion that "reasoning doesn't work for grounding." The paper's Table 8 replicates this failure (FFR causes a 6.4% relative drop for UI-Tars-1.5-7B on ScreenSpot-Pro) but then demonstrates that structured, perspective-based reasoning—taught explicitly during SFT before being optimized during RL—does work, producing a 9.9% relative gain for the same task and model family. The contradiction is resolved by distinguishing between the format of reasoning: unstructured exploration fails because the reasoning space for grounding lacks the formal constraints that make free-form reasoning tractable in mathematics, but structured reasoning succeeds because the SFT stage provides a map of useful analytical perspectives that the RL stage can then optimize over. The takeaway for the field is not "reasoning is harmful for grounding" but "reasoning must be taught before it can be optimized."

The paper establishes that the SFT+RL relationship in grounding is qualitatively different from standard assumptions. The standard view—SFT provides a good initialization, RL provides further refinement—is empirically false for grounding tasks: standard coordinate-only SFT followed by RL degrades performance (37.0% → 34.9% for Qwen2.5-VL-7B on ScreenSpot-Pro in Table 9). The paper demonstrates a different role for SFT: it serves as an exploratory warm-up that instills output diversity, without which RL has no variation to optimize over and either does nothing or actively harms performance. This reframing has implications beyond grounding: any domain where standard SFT produces uniform, low-diversity outputs may suffer from the same pathology when RL is applied, and introducing structured output diversity during SFT (via auxiliary conditioning variables analogous to instruction perspectives) may be a general recipe for preventing RL policy collapse.

The work also shifts the data quality conversation from a generic concern to a quantitative diagnostic. The finding that 23.3% of samples in widely-used grounding datasets contain instruction flaws (ambiguous matches or outright mismatches, Section 2.2) and that cleaning this noise produces consistent 2–3 point accuracy improvements across benchmarks (Figure 7b) establishes that data quality in grounding is not a minor concern—it is a first-order performance determinant. The field can no longer treat dataset noise as an unavoidable cost of scale; the paper demonstrates that a relatively lightweight cleaning pipeline (OmniParser-based bounding box refinement + GPT-4.1-based instruction verification) reduces the error rate from 23.3% to below 8% (Figure 7a) with measurable downstream benefits.

Research directions that become more attractive after this work include: designing perspective taxonomies for other visuospatial reasoning tasks, studying how SFT output diversity affects RL sample efficiency and stability, developing verifier models that explicitly score grounding confidence from multiple reasoning pathways, and investigating whether the Instruction-as-Reasoning paradigm transfers to non-GUI domains where instructions can similarly be decomposed into complementary analytical lenses. Directions that become less attractive include: pure RL-from-scratch grounding without supervised pretraining on the reasoning format (the paper shows this underperforms SFT-only models on most benchmarks), and free-form reasoning augmentation without structured guidance (Tables 7 and 8 show that removing the reasoning format entirely or using unstructured FFR both degrade performance).

Follow-Up Research This Work Enables

Perspective taxonomy optimization: which analytical lenses matter most, and how many are enough? The paper uses a fixed taxonomy of four perspectives (appearance, function, spatial, goal) chosen based on qualitative analysis of human instruction strategies. The emergent perspective finding (Section 4.5, Figure 8b) reveals that the RL-trained model spontaneously discovers perspectives beyond these four—component type, structural relationship, state, sequential position, salience, and others—suggesting the four-perspective taxonomy is underspecified. A natural follow-up would systematically ablate the number and type of perspectives in the SFT training data: train models with 2, 4, 6, and 10 perspectives (using the full taxonomy from Appendix C.1), measure both SFT-only and SFT+RL performance across benchmarks, and determine whether the performance improvement saturates at some number of perspectives or continues to scale. This would also resolve whether the emergent perspectives are genuinely novel reasoning strategies discovered during RL or whether they were implicitly present in the GPT-4.1-generated SFT data (which could be checked by auditing the SFT corpus for the presence of state, component-type, and structural-relationship language). A strong result would show that models trained with richer perspective taxonomies during SFT converge to higher RL performance faster and generalize better to novel GUI platforms.

Inference-time search over reasoning pathways: can the model generate multiple reasoning chains and select the best? The paper's model generates a single reasoning chain and coordinate per (screenshot, instruction) pair with no test-time exploration over perspectives. However, the model's SFT training teaches it to generate reasoning from any of four perspectives, and the RL training teaches it to prefer perspectives that work. A natural extension would implement best-of-N reasoning at inference time: generate N reasoning chains (potentially with different implicit perspectives) for the same input, predict N coordinates, and select the final prediction via majority voting on the final coordinate, confidence-weighted averaging using the PRM-like internal scores, or some other aggregation mechanism. The reference paper demonstrated that compute-optimal test-time scaling (choosing the right search strategy per prompt difficulty) yields 4× efficiency gains over uniform best-of-N. Translating this to grounding: does generating 4 reasoning chains and majority-voting the coordinates outperform a single chain? Does the benefit vary by sample difficulty (larger benefit on hard cases where any single perspective might fail)? Does the optimal number of chains depend on the model's confidence or the instruction's ambiguity? This would test whether the Instruction-as-Reasoning paradigm creates a "generation budget" knob—analogous to the reference paper's test-time compute budget—that practitioners can tune based on latency requirements.

Cross-model-family replication: is Instruction-as-Reasoning specific to Qwen2.5-VL? The paper demonstrates the method on Qwen2.5-VL-7B and Qwen2.5-VL-32B, showing within-family scalability. However, Qwen2.5-VL may have architectural or pretraining properties that make it particularly responsive to perspective-based reasoning instruction. Replicating the full SFT+RL pipeline on InternVL3, LLaVA-NeXT, or a proprietary model (GPT-4V via API fine-tuning if available) would test whether the paradigm generalizes across vision-language architectures. A particularly informative experiment would apply the Instruction-as-Reasoning SFT to a model family that exhibits strong "free-form reasoning failure" (Table 8)—if UI-Tars-1.5-7B, which degrades 6.4% with FFR, instead improves with IR, this would provide strong evidence that the reasoning format, not the model architecture, is the active ingredient. Conversely, if some model families show no benefit from IR (perhaps because their pretraining already internalizes multi-perspective reasoning), this would define a boundary condition identifying which models benefit from explicit perspective training.

Difficulty-conditioned perspective allocation: do different grounding scenarios call for different reasoning strategies? The paper's error analysis (Section 4.6) identifies three failure categories—lack of domain knowledge, layout understanding failures, and visual ambiguity—but does not analyze whether different reasoning perspectives are differentially effective against different failure types. For example, appearance-based reasoning might be most susceptible to visual ambiguity (because it relies on matching visual features), while function-based reasoning might be most susceptible to domain knowledge gaps (because it requires understanding what UI elements do). A systematic study would annotate grounding failures by type across a large benchmark, then measure per-perspective accuracy conditioned on failure type. If appearance-based reasoning achieves 60% accuracy on visually ambiguous cases while function-based reasoning achieves 35%, this would imply that the optimal perspective depends on the nature of the grounding challenge, not just the overall sample difficulty. This could lead to a difficulty-conditioned perspective allocation policy analogous to the reference paper's compute-optimal search strategy: estimate what type of grounding challenge a sample presents (visual, semantic, spatial), then select the reasoning perspective best suited to that challenge type. The practical implementation would require training a lightweight "challenge classifier" on top of base model features, but the paper's existing infrastructure (OmniParser bounding boxes, GPT-4.1-generated instructions) provides the data needed to label challenge types at scale.

Online adaptation: can the model update its perspective selection policy from live interaction feedback? The AndroidWorld result (Table 5, 74.1% success rate with UI-Ins-7B as grounding executor) demonstrates that the model's grounded predictions work in dynamic environments with UI drift and asynchronous state transitions. However, the model's perspective selection policy is frozen after training—it cannot learn from its mistakes during a session. A natural extension would add online RL where the model adapts its perspective selection based on task-level success/failure feedback during AndroidWorld episodes. If the model selects the appearance perspective for a click, misses the target, and the task fails, can it adjust to prefer function-based reasoning on the next attempt? This would transform the Instruction-as-Reasoning framework from a static training recipe into an online adaptation mechanism, where the model continuously refines its perspective selection based on deployment experience. The AndroidWorld environment is well-suited for this because it provides explicit task completion signals that can serve as sparse rewards, and the multi-step nature of tasks means the model has multiple opportunities to observe and correct its grounding failures within a single episode.

Verifier-guided perspective selection: training a "perspective quality estimator" for inference-time routing. The paper's "Combined" oracle in Figure 2a achieves 26.1% on ScreenSpot-Pro by always selecting the best-performing perspective per sample, but this oracle requires ground-truth coordinate correctness to operate. A practical alternative would train a perspective quality estimator—a model that, given a screenshot, an instruction, and a candidate reasoning perspective, predicts the probability that grounding using that perspective will be correct. This is analogous to the reference paper's process reward model (PRM) that estimates per-step correctness probabilities. The estimator could be trained on the SFT+RL model's own rollouts: for each training sample, generate reasoning and coordinates from multiple perspectives, label each with the point-in-box outcome, and train a binary classifier to predict grounding success from the (screenshot, instruction, reasoning-perspective) triplet. At inference time, the system would generate several candidate reasoning perspectives, score them with the quality estimator, and execute grounding using the highest-scoring one. This decouples perspective generation (which the SFT model can do) from perspective selection (which the estimator optimizes) and could improve robustness when the model's internal perspective selection policy is uncertain.

Practical Applications and Downstream Use Cases

GUI test automation for cross-platform software. Organizations developing applications that must run on Windows, MacOS, Linux, iOS, and Android (e.g., enterprise SaaS, productivity tools, communication platforms) face the challenge of maintaining automated UI tests across platforms with different visual conventions, layout engines, and interaction patterns. The Instruction-as-Reasoning model's strong cross-platform grounding performance—UI-Ins-32B achieves 96.5% on iOS, 97.2% on Android, 99.0% on Desktop, and 97.0% on Web (Table 1, Basic subset)—means a single model can serve as the grounding engine for a multi-platform test harness. Rather than writing platform-specific element locators (XPaths, accessibility IDs, or image templates for each platform), a test engineer could write a single natural-language instruction ("click the submit button") and rely on the model to ground that instruction correctly regardless of whether the button appears as a blue rectangle on Windows, a rounded green capsule on iOS, or a text link on Web. The model's ability to reason from multiple perspectives means it can handle the visual variation across platforms—using appearance-based reasoning when the element is visually distinctive on one platform, function-based reasoning when the element's purpose is clearer than its appearance on another.

Accessibility tools with adaptive description generation. Screen readers and accessibility overlays for users with visual impairments need to translate UI elements into natural language descriptions that are both accurate and actionable. The multi-perspective instruction generation pipeline (Section 3.2, Appendix A.1) can be run in reverse: given a detected UI element, generate descriptions from all four perspectives and select the one most appropriate for the user's current context. A user navigating a file manager might benefit from function-based descriptions ("the button to close the current window") when they know what they want to accomplish, while a user exploring an unfamiliar interface might benefit from appearance-based descriptions ("the red circular button in the upper-left corner"). The paper's verification pipeline (achieving 93.5% precise match rate on generated instructions, Figure 7a) provides evidence that GPT-4.1-level models can generate unambiguous, element-specific descriptions at scale, enabling accessibility tools that dynamically adapt their description style to user preferences or task context.

Robotic process automation (RPA) for legacy enterprise software without accessibility APIs. Many enterprise environments rely on legacy software (custom-built internal tools, older versions of commercial applications, specialized industry software) that lacks modern accessibility APIs (UI Automation on Windows, Accessibility on macOS, AT-SPI on Linux). Traditional RPA tools must rely on brittle pixel-based template matching, coordinate-based click recording, or OCR-based text matching to interact with these interfaces—approaches that break when the UI layout changes, the screen resolution differs, or the application is updated. A grounding model trained with Instruction-as-Reasoning can serve as a vision-only universal UI driver: given a screenshot of the legacy application and a natural language instruction describing the desired action, the model predicts a clickable coordinate without requiring any platform-specific accessibility metadata. The model's cross-platform generalization (Tables 1, 3, 4 show competitive performance across diverse OS and software categories) suggests it can handle the visual variety of legacy interfaces, while its ability to reason from multiple perspectives means it can adapt its grounding strategy based on what information is available—using appearance when the legacy interface has distinctive visual elements, using spatial relationships when the layout is conventional, using function when the element's purpose is inferrable from context. The AndroidWorld result (74.1% success rate with UI-Ins-7B, Table 5) provides evidence that the grounding works in live, dynamic environments where interfaces change between actions.

When to Prefer This Method Over Alternatives

The paper implicitly positions Instruction-as-Reasoning against three categories of alternatives: standard SFT-only grounding models, pure RL grounding models, and SFT+RL models without structured reasoning. The following decision rules are not stated explicitly by the authors but can be inferred from the experimental evidence:

• Prefer Instruction-as-Reasoning (SFT+IR+RL) over standard SFT-only grounding when: (1) the test distribution includes instructions with significant semantic variety (implicit, abstract, goal-oriented instructions rather than purely literal descriptions), as evidenced by the larger performance margins on MMBench-GUI L2 Advanced (159.4% relative gain over Qwen2.5-VL-7B, Table 1) and UI-I2E-Bench Implicit (3.5–6.6 point gains over GTA1, Table 2); (2) the deployment setting involves cross-platform generalization where the visual appearance of UI elements varies but their functional roles are consistent; (3) the base model is Qwen2.5-VL—broader model family generalization remains untested.

• Prefer Instruction-as-Reasoning over pure RL grounding (e.g., GUI-R1, GUI-Actor) when: the training budget allows for an SFT stage. The paper's SFT-only model (Table 6) achieves 76.3% MM, 70.1% I2E, and 37.1% Pro—outperforming RL-only (72.4%, 69.2%, 37.0%) on most benchmarks—and the full SFT+RL pipeline substantially exceeds both. If compute is severely limited, RL-only may be viable (it achieves non-trivial performance, and the paper's Table 6 shows it substantially improves over zero-shot), but the structured SFT stage provides better performance per unit of training compute.

• Prefer Instruction-as-Reasoning over SFT+RL without structured reasoning (Figure 7b's "standard SFT+RL") when: the goal is to avoid policy collapse during RL training. The paper demonstrates (Table 9) that standard coordinate-only SFT followed by RL degrades Qwen2.5-VL-7B from 37.0% to 34.9% on ScreenSpot-Pro, while Instruction-as-Reasoning SFT followed by RL improves from 37.1% to 46.0%. If the RL stage is planned, the Instruction-as-Reasoning SFT format is not optional—it is necessary for the RL stage to be constructive rather than destructive.

• The method is not preferable when: (1) the base model fundamentally lacks the capability to ground certain types of instructions—the error analysis (Section 4.6, Figure 10) shows that domain knowledge gaps, layout understanding failures, and severe visual ambiguity persist regardless of perspective selection; (2) the deployment setting requires minimal latency and cannot accommodate the additional tokens generated during the reasoning chain (the paper does not report inference latency or reasoning token counts, making this tradeoff unevaluable from the paper alone); (3) the cost of running the GPT-4.1 data generation pipeline for new domains is prohibitive—the paper does not quantify this cost, but a practitioner adopting the method for a new GUI domain must budget for API calls to generate and verify multi-perspective instructions at scale.